← work / ca-connect

Built for Uptime

An API-first Django REST Framework service that exists to answer one question reliably, at any hour: where is the nearest meeting. No marketing site, no low-stakes traffic — every request is a mobile client doing something real, which is exactly why zero downtime was the design brief from day one.

LIVE In production, backing the C.A. Meeting Finder app on iOS and Android.

Project type: API-first backend service (no public marketing site — API traffic only)
Role: Sole backend engineer — architecture, infrastructure, API, deploy pipeline, monitoring
Consumes it: C.A. Meeting Finder — iOS and Android, see App Store / Play Store
Status: Live in production, serving hundreds of thousands of requests per month
Codebase: ~38,000 lines of Python across 20 Django apps, 2,000+ automated tests, 94% coverage
Uptime target: Zero scheduled downtime — every deploy and recovery path is designed to be invisible to the client

// 01 — The constraint that shaped everything

Downtime isn't an inconvenience here. It's someone who needed a meeting and the app didn't answer.

This service has no front door of its own — no marketing site, no landing page, nothing that could absorb a quiet outage. It is purely an API, consumed entirely by a mobile app that people open at unpredictable hours, often at the exact moment they need it most. There is no "low-traffic" version of a bad deploy here: every request is a real person trying to find real support.

That reframes almost every infrastructure decision. A maintenance window is unacceptable. A silent background-job failure is as serious as a loud server crash. And because the fellowship this app serves is built on the principle of anonymity, that same principle has to be enforced technically — in the data model and the analytics pipeline — not just promised in a privacy policy.

Constraint	What it forced
Used in moments of crisis, at any hour	No maintenance windows. A deploy, a bad migration, or a wedged process has to recover itself in minutes, not wait for a human to wake up
Donation-funded, volunteer-run	The infrastructure budget is a single small droplet and a small managed database — resilience has to come from design, not from throwing hardware at it
No marketing site, API-only	Every request this service handles is a mobile client doing something — there is no low-stakes traffic to buffer against; load is 100% real usage
Built for a fellowship whose core principle is anonymity	That constraint isn't a privacy-policy footnote — it has to be enforced in the analytics pipeline, the data model, and what gets logged, not just promised in writing

// 02 — What this work covers

→ Designed the API-first architecture: everything behind Django REST Framework, versioned under one namespace, nothing served as a raw Django view.
→ Wrote the Terraform configuration that provisions the entire production environment from nothing — private network, managed PostGIS database, object storage, compute, firewall, DNS — so the infrastructure is reproducible and reviewable like any other code.
→ Designed the two-layer self-healing runtime: an application-level health check plus an independent host-level watchdog that detects and recovers from failure modes the first layer structurally cannot see.
→ Built the Cloudflare edge-caching layer as code — origin-driven Cache-Control policy that decides what the CDN is allowed to cache, reviewed and versioned instead of living only in a dashboard.
→ Replaced a third-party mobile remote-config service with an in-house equivalent: a zero-dependency static endpoint for global feature flags, plus a thin per-user override layer for staged rollouts and support cases.
→ Built the recurring-donation integration: webhook ingestion, signature verification, and a deliberately separate state machine for subscription entitlements versus payment events, so a failed or canceled charge can never silently claw back access a donor already paid for.
→ Set up the CI/CD pipeline: a test gate that blocks every build, containerized builds pushed to a private registry, and an idempotent deploy script that rolls the running container without a maintenance window.
→ Wired up privacy-scrubbing analytics and error monitoring so operational visibility and the project's anonymity principle aren't in tension with each other.
→ Wrote and maintain 2,000+ tests covering models, serializers, views, permissions, and the business logic behind entitlements, scheduling, and location-based queries.

// 03 — The approach

Seven layers, each one earning its place by covering a failure the layer below it can't see.

None of this is exotic. It's an ordinary Django REST Framework service — the discipline is entirely in making sure every layer that could fail quietly has something watching it.

01

API-first, nothing else.

Every piece of data the mobile app needs — meetings, schedules, literature, recordings, account state, feature flags — is served through one versioned Django REST Framework namespace. No server-rendered pages, no raw Django views standing in for an endpoint. One contract, one place to apply auth, throttling, and caching policy.
02

Infrastructure as code, down to the firewall rule.

Terraform provisions the private network, the managed PostgreSQL/PostGIS cluster, object storage, the compute host, and the firewall that allows only what's needed in. Nothing in production exists that isn't declared in version control — a lost or corrupted host is a redeploy, not an incident.
03

Cloudflare sits in front, and the origin tells it what to do.

DNS, TLS termination, and edge caching all live at Cloudflare. But the origin — not a dashboard toggle — decides what's cacheable, via Cache-Control headers emitted per view. Slow-changing public reads get a shared edge TTL; anything time-sensitive is explicitly marked no-store so it always reaches the API, even while it shares a cached URL prefix with something that doesn't.
04

Ship through a gate, not around one.

Every push runs the full test suite against the same database engine used in production. Only a green suite produces a container image, published to a private registry. Deploy is a pull-and-recreate of that image on the host, followed by migrations and static collection — the same sequence whether it's the first boot of a new server or the hundredth deploy of the week.
05

The service watches itself, from two different angles.

The application container reports its own health from inside its own loopback. A second, independent watchdog running on the host checks reachability from outside the container entirely — because a container can report itself perfectly healthy while the network path between the host and the container has stopped carrying traffic. Either failure signature triggers an automatic recreation, no human in the loop.
06

Remote config that can't go down with the database.

The feature-flag payload the app reads on every launch is a plain, import-only constant — no database query, no per-request computation, cacheable at the edge like any other static response. A thin authenticated layer on top lets a single account's flags be overridden for a staged rollout or a support case, without ever touching the shape of the public payload everyone else gets.
07

Observe everything operationally; identify no one.

Error tracking and scheduled-job monitoring page a human the moment something silently stops working. Product analytics capture that an event happened, never who triggered it, scrubbed at the SDK layer, not just filtered on the way out. The two systems answer different questions — is it working, and is it used — and neither one is allowed to answer the other.

Request path, at a glance

C.A. Meeting Finder (iOS / Android)
      │
      ▼
Cloudflare edge ──► cache HIT? ──► yes ──► served from the edge, origin never touched
      │
      no
      ▼
TLS terminated, forwarded to the origin host
      │
      ▼
Containerized Django REST Framework service
      │   ├─► container healthcheck (inward-looking)
      │   └─► host watchdog (outward-looking) ──► unhealthy either way ──► automatic recreate
      ▼
Managed PostgreSQL + PostGIS (private network only)
      │
      ▼
Response ──► Cache-Control set per view ──► tells Cloudflare what it's allowed to cache next time

Edge caching as a contract, not a setting

A Cloudflare cache rule is only as trustworthy as whatever decided to turn it on. Rather than configure caching behavior by hand in a vendor dashboard — invisible to code review, impossible to blame in a postmortem — every view declares its own policy, and the edge rule is set to simply respect whatever the origin says.

Tier	Applies to	Behavior
Public edge cache	Slow-changing public reads (e.g. meeting listings)	200 OK on GET/HEAD gets a public Cache-Control with a short browser TTL and a longer shared edge TTL — the Cloudflare rule is set to respect the origin, so this header is the actual policy, not a suggestion
No-store	Time-sensitive or per-request endpoints sharing a cached path prefix	Every response explicitly marked uncacheable, so it can never be accidentally served stale by a rule written for its neighbor
Static assets	Media and collected static files	Served from object storage behind its own CDN endpoint with a long client-side max-age, decoupled entirely from the API's own cache policy

// 04 — Technical insights

Self-healing needs two watchdogs, not one

A container-level health check is a necessary first layer, but it has a structural blind spot: it can only ever report on what it can see from inside its own loopback. If the application itself is wedged — a worker that's exhausted its resources, a hung request — that check correctly fails and the orchestrator can react. But if the failure is instead in the network path between the host and the container, the container can report itself perfectly healthy while nothing outside it can actually reach it.

The fix was a second, independent watchdog running on the host itself, checking reachability the way an external client actually would — from outside the container's own network namespace. Either signal, on its own, triggers the same recovery: recreate the container, rebuild its network attachment, and let supervision bring it back up. Neither layer trusts the other to catch what it can't see.

Layer	Detects	Can't see
Container health check	The application process itself is wedged — exhausted file descriptors, a hung worker — while its own loopback can no longer be trusted to answer	A failure in the network path between the host and the container; the container can report itself healthy while nothing outside it can reach it
Host-level watchdog	Unreachability from outside the container's own network namespace — the bridge or proxy layer has stopped routing, independent of whether the app inside is fine	Nothing on its own — it exists specifically to cover the first layer's blind spot, and both signals are checked before triggering a recovery

Workers recycle themselves, on purpose

Any sufficiently long-running process accumulates something it shouldn't — a leaked file descriptor, a thread that never joined, memory a library forgot to release. Chasing down every possible leak in every dependency is a losing game. So application workers recycle themselves after a bounded number of requests, with a little randomness added so they don't all restart in the same instant. It's unglamorous, and it bounds an entire class of slow failure to a single recycle window instead of letting it compound for hours.

Infrastructure as code, including the thing that breaks it

Terraform provisions the private network, the managed PostGIS database (with firewall rules that only ever allow the one host that needs it), the object storage bucket and its CDN, the compute host, and the DNS record pointed at it. Everything about the environment a disaster-recovery exercise would need to rebuild already exists in version control. The one deliberate exception: a configuration change that would normally force a destructive rebuild of the running host is explicitly fenced off from triggering one, so editing a boot-time script doesn't silently cost a multi-minute outage for an unrelated change — a rebuild only happens when someone asks for it on purpose.

Remote config without a single point of failure

The mobile app's feature flags used to come from a third-party remote-config service. Replacing it meant not reintroducing the exact risk that justified owning it: the global config payload is a plain static constant with zero database dependency and zero per-request computation, so it's exactly as fast and exactly as reliable as any other cached response, and it's cacheable at the edge. A separate, authenticated, per-user layer can override individual flags for one account — a staged rollout, a support case — but it can only ever return a small delta, never touch the shape of the payload everyone else gets. The base of the system can't go down because of a feature meant for one account.

Payments and entitlements don't share a state machine

A recurring-donation processor reports two conceptually different things through webhooks: that money actually moved, and that a recurring plan changed state — paused, resumed, canceled, or a retry failed. It would be natural, and wrong, to let the second kind of event immediately revoke what the first kind already bought. A donor's subscription-tier access is earned by a completed charge and expires on its own schedule; a cancellation only stops the next charge from happening. Keeping those two code paths from ever writing to each other's state was a deliberate design boundary, backed by tests that specifically assert a canceled or failed plan event leaves existing entitlement untouched.

Observability without identity

Error tracking and scheduled-job monitoring exist so a silent failure — a background job that stops running without crashing loudly — pages a human exactly as fast as one that errors out loud. Product analytics exist to answer a completely different question: is a given feature getting used. The two are kept strictly apart. Analytics events carry a feature name and nothing that could identify who triggered it — no device fingerprint, no IP, no account identifier — enforced by an explicit allowlist at the point where an event leaves the process, not by a dashboard setting that a future change could quietly weaken. For a project whose entire premise is anonymity, that enforcement has to live in code that gets reviewed, not in a configuration screen that doesn't.

// 05 — Results

It stays up, it recovers itself, and it never knows who's asking.

Measure	Result
Monthly request volume	Hundreds of thousands of API requests per month, served from a single small compute host plus the CDN edge
Scheduled downtime	None by design — deploys roll the running container in place; self-healing recovers wedged processes without a human paged first
Test suite	2,000+ tests, 94% coverage, run against the same database engine as production on every push, gating every deploy
Codebase	~38,000 lines of Python across 20 focused Django apps (accounts, meetings, locations, literature, recordings, subscriptions, notifications, and more)
Mobile clients served	iOS and Android, both built on the same versioned API — one backend, two app stores, no divergent contracts
Payment entitlement bugs	Zero clawbacks — entitlement state and payment-event state are structurally separate, so a canceled or failed charge can only stop future billing, never revoke something already paid for

// 06 — What this project reinforced

Uptime is a design discipline, not a monitoring dashboard.

A health check that only looks inward can still lie to you.

A container can report itself perfectly healthy from its own loopback while the path from the host to that container has quietly stopped working. The fix wasn't a smarter health check — it was a second, independently-running check that looks at the system from the outside, because the two failure modes are invisible to each other by construction.

Every long-lived process leaks something, eventually.

Rather than chase down every possible resource leak in every dependency a web worker touches, the workers recycle themselves periodically as a matter of course. It's a cheap, boring defense against an entire category of slow failure that would otherwise take hours or days to resurface and minutes to diagnose at 3am.

Caching policy belongs in code review, not a dashboard click.

Letting an edge cache rule be configured only in a vendor's UI means the policy has no diff, no author, and no history. Emitting Cache-Control from the view itself, with the edge rule set to simply respect it, means a change to what's cacheable goes through the same review as everything else — and is just as easy to revert.

A remote-config endpoint's only job is to never be the reason the app breaks.

The global config payload has zero database dependency on purpose — it's a constant, so it loads as fast and as reliably as anything can on this stack, and it's cacheable at the edge like static content. Personalization lives in a separate, deliberately thin layer that can only add deltas on top, never risk the shape of what everyone else gets.

Payments and entitlements are two different state machines, and merging them is the bug.

A recurring-donation processor reports two different kinds of event: money moved, and a recurring plan's lifecycle changed. The temptation is to let a cancellation or a failed charge immediately revoke access. But the donor already paid for what they have — a failed retry or a cancellation should only ever stop a future charge, never reach backward and take back something already bought. Keeping those two code paths from ever touching each other's state was a deliberate, tested boundary.

Secrets need exactly one correct home, decided by who needs them first.

Infrastructure-level secrets that a server needs before the application even starts live in one store; application-level secrets the deploy pipeline layers on afterward live in another. Picking the store by "who needs this value first" — instead of case-by-case judgment calls — means every new secret has one unambiguous right answer and rotation never becomes an archaeology project.

Anonymity has to be enforced below the dashboard, not just promised above it.

A product analytics SDK will happily fingerprint a device, a client runtime, an IP, and a session unless something actively strips that out before it leaves the process. For a project whose entire reason for existing rests on anonymity, "we configured privacy mode" isn't good enough — the only property worth trusting is an explicit allowlist that drops everything else, verified by inspecting the literal payload that left the building.

Zero downtime is a budget you spend before you need it, not a reaction you have after.

Every layer of redundancy here — the edge, the watchdog, the worker recycling, the database firewall — exists for an incident that, on a good month, never happens. That's the point. The cost of maintaining a boring, uneventful system is small compared to the cost of a single bad hour for someone who opened the app looking for the nearest meeting.

Parity between a laptop and production is the cheapest test you'll ever write.

The same database image, the same pinned timezone database, and the same dependency versions run in local development, continuous integration, and production. A test that passes on a contributor's machine means something, because there's no gap left for "well, it works on mine" to hide in.

// Technologies used

Layer	Technology	Why it was chosen
Language / framework	Python, Django, Django REST Framework	Mature ORM, admin, and migrations; DRF gives one consistent contract layer for every client
Database	Managed PostgreSQL with PostGIS, provisioned on a private network	Geographic meeting search needs real spatial indexing, not approximations; managed service means automated backups and patching without hand-holding
Infrastructure	Terraform	The entire production environment — network, database, storage, compute, firewall, DNS — is declared, reviewed, and reproducible, not clicked together by hand
Compute & runtime	Docker, a single hardened host, self-healing supervision	Small, auditable surface area; resilience comes from design rather than a large fleet
Edge / DNS / TLS	Cloudflare, origin-driven cache policy	TLS termination, DDoS absorption, and edge caching, with the actual caching rules authored and reviewed in application code
Object storage & CDN	S3-compatible object storage with its own CDN endpoint	Media and static assets served from the edge, decoupled from the API's own request path
CI/CD	A hosted CI pipeline with a mandatory test gate, private container registry	No path to production exists that skips the test suite; deploys are a scripted, idempotent container swap
Error & job monitoring	A dedicated error-tracking and scheduled-job monitoring service	A background job that silently stops running is exactly as dangerous as one that crashes loudly — both need to page a human
Product analytics	A privacy-first analytics pipeline with an explicit server-side scrubbing hook	Event counts without any identifying property — enforced in code, not left to a dashboard setting
Email & push	Transactional email API, mobile push notification service	Account and reminder delivery that doesn't depend on the mobile client being open or online
Payments / donations	Webhook-driven recurring-donation processor integration	Funds the project; entitlement tiers are derived from donation events, kept strictly separate from the donor's billing lifecycle

App Store ↗ Google Play ↗ More work →