Nullify

Under the hood.

It started from an assignment. In September I rebuilt the core so Postgres, not memory, decides what counts as one incident, and tested it against a real database: 328 tests on the backend, 531 on the frontend.

Start a storm.

A simulation of the two rules at the core, with the app’s own priorities, channels, SLAs and refusal message.

In the real thing, four worker processes race for the same component. A unique index on open incidents lets only one exist, and the test suite fires 100 concurrent signals and expects exactly one incident.

Measured, not promised.

The first version’s README claimed 10,000 signals a second. Nothing had measured it. When I did, every signal took its own transaction and queued for the same incident row.

Tag it, ship it, watch it.

A version tag runs the pipeline, and nothing in CI holds a cloud key. On the server, Prometheus watches the app with eleven alert rules.

1. Tag v*Pushing to main runs nothing. A version tag is the release.
2. Backendruff, then pytest against a real Postgres. Under 80% coverage fails.
3. Frontendeslint, vitest, a production build.
4. Securitygitleaks over all history, pip-audit, npm audit, bandit, zizmor, terraform validate, promtool.
5. ImagesBuilt for amd64, Trivy fails any fixable high or critical, tagged with the commit, never latest.
6. ApprovalThe production environment waits for a click.
7. DeployOIDC for a short-lived role, then SSM runs deploy.sh. No SSH, no stored keys.
8. HealthHTTPS /health from outside. A failed pull, migration or check rolls back.

github.com/Aswin-Shine/Nullify-Incident-Management-System · actions

The API exports its own metrics: signals received, rejected and processed, queue depth, open incidents, state changes, database retries and failed notifications, summed across all four workers.

• NullifyApiDown
• PostgresDown
• RedisDown
• IngestQueueFilling
• SignalsDropped
• SignalProcessingFailing
• IngestBlockedOnDatabase
• NotificationsFailing
• DbRetriesSpiking
• HighErrorRate
• SlowRequests

Grafana · Nullify overview

What broke.

1. 28 Sep 2026 · Local

   Two SREs moved the same incident, and both won.

   A status change read the row, checked the move was legal, then wrote by id. Two people at once both passed the check: five requests, five successes. Now the write is compare-and-set, an UPDATE that only lands if the status is still the one it checked, and the loser gets a 409. The first race test passed even on the broken code because its requests never overlapped; it now holds a row lock to force the race.

   The commit, 7b8b917

2. 1 Oct 2026 · Local

   Eight signals, and the live feed stayed empty.

   The API runs four worker processes. The request that stored a signal and the browser’s WebSocket usually sat in different ones, and each worker only told its own sockets. Now every worker publishes events to a Redis channel and every worker passes them on, so any open dashboard sees everything.

   The commit, b14d46b

3. 8 Oct 2026 · Local

   A Postgres outage sent 1,878 signals nowhere.

   I stopped Postgres under load. Compose drops a stopped container’s DNS name, so the driver raised a lookup error the code treated as permanent: 1,878 of 2,000 signals went to the raw log with no incident, and the API answered 500. Now network errors count as transient, a circuit breaker holds signals and answers 503 with Retry-After, and a replay command recovered 1,875 of the lost ones. The rerun stored 1,500 of 1,500 through a 30-second outage.

   The commit, e3d3a32

4. 9 Oct 2026 · CI

   The scanners said all clear.

   Gitleaks ran in a container, tripped git’s dubious-ownership check, scanned 0 commits and still printed “no leaks found”. It now scans every commit. Then Trivy found 44 fixable high and critical CVEs in the nginx base image; moving to 1.30 with an apk upgrade took it to 0.

   The commit, 0129c50

What I’d do next.

• Automated Postgres backups. Today the database lives on the instance’s own disk.
• Alertmanager, so the eleven alert rules reach a person instead of waiting in Prometheus.
• A second instance. The app already scales sideways (stateless workers, Redis fan-out, the one-incident rule in the database); the infrastructure doesn’t yet.
• A durable ingest queue. A signal that was accepted but not yet written is lost if a process crashes hard.
• Terraform state in S3 before a second person touches the infrastructure.

Tech specs.

Frontend

React 19, Vite, HeroUI v3, Tailwind CSS v4, served by nginx

API

FastAPI on uvicorn (4 workers), Pydantic v2, WebSocket

Data

PostgreSQL 16 (SQLAlchemy 2 async, asyncpg, Alembic), Redis 7, a JSONL lake

Edge

Caddy with Let’s Encrypt and HSTS, nginx with a strict Content Security Policy

Infrastructure

Terraform: EC2 c7i-flex.large, Elastic IP, security group, GitHub OIDC role, SSM

Delivery

GitHub Actions, ruff, pytest, vitest, gitleaks, pip-audit, npm audit, bandit, zizmor, Trivy, Dependabot

Observability

Prometheus, Grafana, Postgres and Redis exporters, 11 alert rules

Tests

328 backend against real Postgres (80% coverage gate), 531 frontend

The repository