Incident response

​

Reporting

​

First five minutes

1. Contain. If you do not know how far the issue has spread, flip the relevant feature flag or route to maintenance mode. A 503 error is safer than a breach.
2. Preserve. Snapshot the database before any destructive recovery:
   pylon backup /var/snapshots/incident-$(date +%Y%m%d-%H%M%S)
   
3. Gather evidence. Pull logs, metrics, and the recent audit trail:
   journalctl -u pylon --since "30 min ago" > /tmp/incident.log
   curl -s http://localhost:4321/metrics > /tmp/incident-metrics.txt
   
4. Declare. Open an incident channel. One person owns coordination and one person drives fixes. Everyone else only observes.

​

Common incidents

​

Admin token leaked

​

Policy bypass / data exposure

1. Reproduce with a throwaway session token against staging.
2. Check audit_log for rows accessed during the window. In the EU, this triggers the GDPR notification requirement.
3. Patch the issue and ship the fix. Verify that the regression test in crates/policy/src/lib.rs::tests covers it.
4. If user data was exposed, start the breach notification workflow.

​

Runaway write loop

1. Identify the source. Check recent deploys and audit_log for the offending user_id or IP.
2. Rate-limit or block at the proxy. Do not try to fix this inside the server while it is under load.
3. Once the system is stable, investigate the trigger. A common cause is a client stuck in an exponential-backoff retry loop with no jitter.

​

WAL file growth past disk budget

# Offline checkpoint; needs exclusive DB access.
systemctl stop pylon
sqlite3 /var/lib/pylon/pylon.db "PRAGMA wal_checkpoint(TRUNCATE)"
systemctl start pylon

​

WS fanout storm

1. Check the client count. The per-IP cap is 64. If one IP reaches 64 connections, a client is stuck in a loop.
2. Check the broadcast rate. The event might come from a retry loop creating 1000 inserts/sec.
3. Restart the WS server (kill -HUP <pid>) only as a last resort. Every client must reconnect.

​

Cloudflare Workers billing spike

​

Magic-link emails not arriving

1. Check the spam folder.
2. Verify the email provider environment variables (PYLON_EMAIL_*).
3. Check the provider’s dashboard. Your account may be paused for a high bounce rate.
4. Check journalctl -u pylon | grep email for delivery errors.
5. For domain authentication, verify SPF, DKIM, and DMARC records with mxtoolbox.com.

​

Sudden surge of 401 / 403 errors

1. Check whether a deploy went out recently. A regression in policy expressions can lock everyone out.
2. Check the session database integrity. If sessions.db is corrupted, every authenticated request fails.
3. If needed, restore the previous session database from backup. Sessions are recoverable; users only need to sign in again.

​

Post-mortem template

1. Summary: one paragraph covering what happened, how long it lasted, and how far it reached.
2. Timeline: UTC timestamps from the first signal to the all-clear.
3. Root cause: what went wrong, not who caused it.
4. What worked: detection, containment, and communication.
5. What did not work: slow alerts, unclear runbooks, and missing graphs.
6. Action items: owners and target dates. Track them in the sprint.

​

On Pylon Cloud

​

Contacts

• Security: [email protected]
• Public issues / feature requests: github.com/pylonsync/pylon/issues
• Cloud support: dashboard → Help → Contact
• Your internal oncall: (fill in your rotation)