Business continuity & disaster recovery procedure

Business continuity & disaster recovery procedure

Field	Value
Document title	Business continuity & disaster recovery procedure
Version	1.0
Effective date	29 May 2026
Owner	Engineering Lead (engineering@theorahq.com)
Approved by	Theora founders (acting as Executive Sponsor for security)
Review cadence	Annually, or after any material change to production architecture or any DR event

0. Document control history

Version	Date	Author	Change summary
1.0	29 May 2026	Engineering Lead	Initial published version of the BC/DR procedure covering scope, recovery objectives, scenario playbooks, post-incident review, and tabletop exercise cadence.

1. Purpose & scope

This procedure defines how Theora maintains availability and recovers the production service in the event of infrastructure failure, data loss, or supply-chain compromise. It applies to all production systems supporting the Theora application, including:

• The betaapp-api Cloud Run service in us-central1, Google Cloud project tactile-stack-491017-c0, running Django 4.2 on Python 3.14.
• The Cloud SQL for PostgreSQL primary instance, attached over private IP via the Serverless VPC Access connector betaapp-connector.
• The Google Cloud Storage bucket tactile-stack-491017-c0-betaapp-assets, encrypted at rest with a customer-managed encryption key (CMEK) in Cloud KMS on a 90-day rotation.
• Firebase Identity Platform (authentication, MFA via TOTP and SMS, Argon2 password hashes).
• Firebase Hosting fronting Cloud Run.
• The Cloud Build pipeline defined in cloudbuild.yaml, triggered on push to main in LouizaLab/BetaApp, publishing the container image us-central1-docker.pkg.dev/tactile-stack-491017-c0/betaapp/betaapp-api.

Out of scope: corporate productivity systems (e-mail, calendaring) and individual workstations. Those are governed by Theora's Acceptable Use and endpoint policies.

2. Recovery objectives

Objective	Target	Rationale
Recovery Time Objective (RTO)	4 hours	Maximum tolerable time between declaration of a disaster and restoration of the production API to a serving state.
Recovery Point Objective (RPO)	24 hours	Maximum tolerable data loss, measured against the most recent successful Cloud SQL automated backup and GCS object versioning state.
Backup retention	30 days of automated Cloud SQL backups; GCS object versioning with a 30-day non-current retention rule	Provides recovery range for both acute failure and delayed-discovery scenarios such as accidental deletion.

3. Roles & responsibilities

Role	Held by	Responsibilities during a DR event
Incident Commander	Engineering Lead (on-call founder, rotating)	Declares the incident, owns the recovery decision, communicates status, authorises destructive recovery steps such as point-in-time restore.
Recovery Operator	Engineering on-call	Executes the recovery procedure in section 5 under Incident Commander direction.
Customer Communications	Founder on duty	Posts status updates and replies to inbound customer mail to support@theorahq.com.
Scribe	Any available engineer	Maintains a timestamped log of decisions, commands run, and observed outcomes for the post-incident review.

4. Scenarios covered

The following five scenarios are the named disaster scenarios for which Theora maintains a documented recovery procedure. Each has a defined detection mechanism and a defined recovery procedure (section 5).

1. Cloud SQL data corruption or logical loss in the production PostgreSQL instance.
2. Cloud Run regional outage affecting us-central1, where betaapp-api cannot serve traffic.
3. GCS bucket unavailability or impairment of tactile-stack-491017-c0-betaapp-assets.
4. Supply-chain compromise of the Cloud Build pipeline, including unauthorised image push to Artifact Registry.
5. Accidental data deletion by an authorised operator (for example, a destructive mistake in Django shell or an unintended gcloud command).

5. Detection & recovery procedures

5.1 Cloud SQL data corruption or logical loss

Detection. Cloud Monitoring alert policies on Cloud SQL: replication lag, database error log severity ERROR or above, and application-level 5xx rate from betaapp-api. Customer-side reports inbound to support@theorahq.com. Anomalies in the daily delete_expired_queries Django management command output (unexpected row deltas).

Recovery. Restore via point-in-time recovery (PITR) to the most recent uncorrupted timestamp, into a new instance, then promote DNS / connection string after validation:

gcloud sql backups list \
  --instance=betaapp-prod \
  --project=tactile-stack-491017-c0

gcloud sql instances clone betaapp-prod betaapp-prod-restore \
  --point-in-time='YYYY-MM-DDTHH:MM:SS.000Z' \
  --project=tactile-stack-491017-c0

# After application-level validation against betaapp-prod-restore:
gcloud run services update betaapp-api \
  --region=us-central1 \
  --project=tactile-stack-491017-c0 \
  --update-env-vars=DB_INSTANCE=betaapp-prod-restore

5.2 Cloud Run regional outage in us-central1

Detection. Cloud Monitoring uptime check on the Firebase Hosting front door, Cloud Run request error-rate and latency alerts, and Google Cloud Service Health for the us-central1 region. Customer reports to support@theorahq.com.

Recovery. If the outage is partial, roll forward to the last known good Cloud Run revision; the current production revision is betaapp-api-00070-lnq. If the outage is regional, redeploy the container image to a secondary region from Artifact Registry and shift Firebase Hosting rewrites:

# Roll back to a prior known-good revision in-region:
gcloud run services update-traffic betaapp-api \
  --region=us-central1 \
  --project=tactile-stack-491017-c0 \
  --to-revisions=betaapp-api-00070-lnq=100

# Regional failover — redeploy from the same image to us-east1:
gcloud run deploy betaapp-api \
  --region=us-east1 \
  --project=tactile-stack-491017-c0 \
  --image=us-central1-docker.pkg.dev/tactile-stack-491017-c0/betaapp/betaapp-api:stable \
  --vpc-connector=betaapp-connector \
  --allow-unauthenticated=False

5.3 GCS bucket unavailability

Detection. Cloud Monitoring alert policies on GCS request error rate for tactile-stack-491017-c0-betaapp-assets and on Cloud KMS key access errors (since the bucket is CMEK-protected). Application-side failures surface as upload/download errors logged from betaapp-api. Customer reports to support@theorahq.com.

Recovery. If a CMEK access issue is the root cause, verify the Cloud KMS key state and the Cloud Run service account binding before attempting any data action. If objects are missing or corrupted, restore from object versioning:

gcloud kms keys versions list \
  --location=us-central1 \
  --keyring=betaapp-keyring \
  --key=betaapp-data-key \
  --project=tactile-stack-491017-c0

gcloud storage ls --versions \
  gs://tactile-stack-491017-c0-betaapp-assets/PATH/TO/OBJECT

gcloud storage cp \
  gs://tactile-stack-491017-c0-betaapp-assets/PATH/TO/OBJECT#GENERATION \
  gs://tactile-stack-491017-c0-betaapp-assets/PATH/TO/OBJECT

5.4 Supply-chain compromise via Cloud Build

Detection. Cloud Audit Logs on the Cloud Build trigger and Artifact Registry; Cloud Monitoring alert policies on unexpected image pushes outside of a merged-to-main trigger; GitHub branch protection violation alerts on LouizaLab/BetaApp; anomaly in the Cloud Run revision history (revisions not corresponding to a known commit on main).

Recovery. Freeze the pipeline, pin Cloud Run to the last revision built from a verified commit, then rebuild from a trusted baseline:

# Pause the Cloud Build trigger:
gcloud builds triggers describe betaapp-main \
  --project=tactile-stack-491017-c0
gcloud builds triggers update betaapp-main \
  --project=tactile-stack-491017-c0 \
  --disabled

# Pin traffic to the last verified Cloud Run revision:
gcloud run services update-traffic betaapp-api \
  --region=us-central1 \
  --project=tactile-stack-491017-c0 \
  --to-revisions=betaapp-api-00070-lnq=100

# Inspect the image digest currently serving production:
gcloud run revisions describe betaapp-api-00070-lnq \
  --region=us-central1 \
  --project=tactile-stack-491017-c0 \
  --format='value(spec.containers[0].image)'

Following pipeline freeze, the Incident Commander triggers a forensic review of GitHub audit logs, Cloud Build history, and Artifact Registry image digests before any new image is permitted to deploy.

5.5 Accidental data deletion by an authorised operator

Detection. Cloud Audit Logs on Cloud SQL and GCS administrative actions; alert policy on unexpected large drops in row counts surfaced via the daily delete_expired_queries job's metrics; customer reports to support@theorahq.com describing missing data. User query content is intentionally stripped from Cloud Logging by core.logging.RedactingFilter, so detection is metric- and audit-log-based rather than content-based.

Recovery. For database deletions, restore via point-in-time recovery to a timestamp just before the destructive action (see 5.1) and selectively re-import the affected tables. For object deletions, restore from GCS object versioning (see 5.3). The Incident Commander records the operator action, root cause, and any access-control change in the post-incident review.

6. Post-incident review

Within five business days of recovery, the Engineering Lead chairs a post-incident review with all responders. The review produces a written record stored in the source repository under docs/soc2/incidents/incident-YYYY-MM-DD-.md. The record contains:

• Timeline of detection, declaration, recovery actions, and resolution.
• Root-cause analysis, including contributing factors.
• Customer-impact assessment, including the population of affected accounts and the data categories involved.
• Whether RTO (4 hours) and RPO (24 hours) were met, with explanation if they were not.
• Action items with named owners and target dates, tracked to closure.

Action items that affect controls (for example, an alert policy change or a new automated check) are also reflected in the relevant SOC 2 control document and, where appropriate, in docs/soc2/risk-register.md.

7. Tabletop exercise cadence

Theora exercises this procedure quarterly to validate that the playbooks remain accurate against the current architecture and that responders can execute them under time pressure.

Attribute	Value
Frequency	Quarterly (four exercises per calendar year)
Facilitator	Engineering Lead
Attendees	Both founders, the engineering on-call, and (where available) a representative from customer communications. Minimum quorum is two participants in addition to the facilitator.
Scenario selection	One of the five scenarios in section 4, rotated so that all five are exercised at least once every 15 months.
Format	30 to 60 minute discussion-based walkthrough: the facilitator presents a scenario prompt and injects, participants narrate the steps they would take, gaps and assumption breaks are recorded.
Output	A dated exercise report filed in the source repository at docs/soc2/dr-tabletop-YYYY-MM-DD.md, containing scenario, attendees, narrative, gaps identified, and action items with owners.

Action items from tabletop exercises are tracked in the same way as post-incident review action items (section 6) and are reviewed at the start of the next exercise.

8. Dependencies on sub-processors

Recovery for several scenarios depends on availability of third-party sub-processors. Theora tracks these vendors in docs/soc2/vendor-list.md. The sub-processors materially relevant to BC/DR are:

• Google Cloud (Cloud Run, Cloud SQL, GCS, Cloud KMS, Cloud Build, Artifact Registry, Cloud Monitoring, Cloud Logging, VPC, Serverless VPC Access).
• Firebase (Identity Platform and Hosting).
• GitHub (source of truth for the LouizaLab/BetaApp repository and Cloud Build trigger source).
• Vanta (continuous control monitoring).
• Anthropic and OpenAI (model providers; not on the critical recovery path for the application itself, but degraded availability is communicated to customers via support@theorahq.com).

9. Related documents

• docs/soc2/inventory-of-resources.md — authoritative inventory of production resources referenced by this procedure.
• docs/soc2/risk-register.md — risks tied to availability and supply-chain integrity.
• docs/soc2/vendor-list.md — sub-processor inventory.
• docs/soc2/cloud-ids-exception.md — documented compensating controls (VPC Flow Logs on all 42 subnets at 50% sample with include-all metadata; core.logging.RedactingFilter; per-IP rate limiting on authentication endpoints) referenced where relevant in detection design.