> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pylonsync.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Incident response

> Reporting security issues, the first five minutes, common incidents, post-mortem template.

## Reporting

**Security vulnerabilities**: email `security@pylonsync.com`. Do NOT file a public issue. Acknowledgement within 48 hours. Target remediation 7 days for high severity, 30 days for medium.

**Operational incidents** (your own deploy): follow your internal runbook. This page is for anchoring the generic moves.

## First five minutes

1. **Contain.** If the blast radius is unknown, flip the relevant feature flag or route to maintenance mode. A 503 is better than a breach.

2. **Preserve.** Snapshot the database before any destructive recovery:

   ```sh theme={null}
   pylon backup /var/snapshots/incident-$(date +%Y%m%d-%H%M%S)
   ```

3. **Capture signal.** Pull logs, metrics, and recent audit trail:

   ```sh theme={null}
   journalctl -u pylon --since "30 min ago" > /tmp/incident.log
   curl -s http://localhost:4321/metrics > /tmp/incident-metrics.txt
   ```

4. **Declare.** Spin up an incident channel. One person owns coordination, one person drives fixes. Everyone else stays off.

## Common incidents

### Admin token leaked

Follow [Admin token rotation → emergency path](/operations/token-rotation#emergency-suspected-compromise). Revoke every session, rotate the token, audit the window of exposure. If OAuth credentials were stored as admin-level env vars, rotate those too.

### Policy bypass / data exposure

1. Reproduce with a throwaway session token against staging.
2. Check `audit_log` for rows accessed during the window — this is the GDPR notification trigger in the EU.
3. Patch, ship, verify the regression test in `crates/policy/src/lib.rs::tests` covers it.
4. If user data was exposed: initiate breach notification workflow.

### Runaway write loop

**Signal**: write rate pegged above the 70k/sec ceiling, WAL growing unbounded, disk filling.

1. Identify the source: check recent deploys + `audit_log` for the offending `user_id` or IP.
2. Rate-limit or block at the proxy — don't try to fix inside the server while it's under load.
3. Once stable, investigate the trigger. Common cause: a client in an exponential-backoff retry loop without jitter.

### WAL file growth past disk budget

SQLite in WAL mode delays checkpoints. If checkpoint isn't completing:

```sh theme={null}
# Offline checkpoint — needs exclusive DB access.
systemctl stop pylon
sqlite3 /var/lib/pylon/pylon.db "PRAGMA wal_checkpoint(TRUNCATE)"
systemctl start pylon
```

If the WAL is *always* growing, a long-running read transaction is holding it open. Find it via `sqlite3 … ".pragma stats"` and kill the client holding the lock.

### WS fanout storm

**Signal**: ws-broadcast-N threads at 100% CPU, clients reporting missed events, queue-full warnings in the log.

1. Check client count — per-IP cap is 64; if one IP has 64, a client is looping.
2. Check broadcast rate. The event that's being broadcast might not be expected — e.g. a retry loop creating 1000 inserts/sec.
3. Bounce the WS server (`kill -HUP <pid>`) only as a last resort — every client has to reconnect.

### Cloudflare Workers billing spike

**Signal**: CF email saying you've used 80% of your monthly budget in a week. See [Workers costs](/operations/workers-costs) for patterns.

### Magic-link emails not arriving

1. Check the spam folder.
2. Verify the email provider env vars (`PYLON_EMAIL_*`).
3. Hit the provider's dashboard — your account may be paused for high bounce rate.
4. Check `journalctl -u pylon | grep email` for delivery errors.
5. For domain authentication, verify SPF/DKIM/DMARC records via [`mxtoolbox.com`](https://mxtoolbox.com).

### Sudden surge of 401 / 403 errors

1. Check if a deploy went out — a regression in policy expressions can lock everyone out.
2. Check session DB integrity — if `sessions.db` is corrupted, every authenticated request fails.
3. Restore the previous session DB from backup if needed (sessions are recoverable; users just sign back in).

## Post-mortem template

1. **Summary** — one paragraph, what happened, duration, blast radius.
2. **Timeline** — UTC timestamps from first signal to all-clear.
3. **Root cause** — what went wrong, not who.
4. **What worked** — detection, containment, comms.
5. **What didn't** — slow alerts, unclear runbooks, missing graphs.
6. **Action items** — with owners and target dates. Track in the sprint.

Publish internally within 5 business days. External disclosure for user-impacting incidents per your privacy policy.

## On Pylon Cloud

For Cloud workspaces, Pylon's on-call team is paged on infrastructure-level incidents (DB outage, region-wide failure, control-plane bugs). For app-level incidents (your code, your data), Cloud's dashboard surfaces logs and request traces but you own the response.

Status: [status.pylonsync.com](https://status.pylonsync.com) shows region health and incident history.

## Contacts

* Security: `security@pylonsync.com`
* Public issues / feature requests: [github.com/pylonsync/pylon/issues](https://github.com/pylonsync/pylon/issues)
* Cloud support: dashboard → Help → Contact
* Your internal oncall: (fill in your rotation)