Charts that no longer take the server down, migrations that run themselves, and a bounded page size
A week on the platform's own health after an incident on 7 October: the analytics rollup collection had never had an index, and the email history chart formatted twelve million dates per load. All of it measured in production and fixed, with the one-off repairs now running on their own after a deploy.
Added
2- Self-hostingMigrations run themselves after a deploy. The worker runs every registered migration the ledger does not record as succeeded, under a lock so two replicas cannot run the same one, and retries one that failed or was cut off by a restart on the next start. Each run is recorded with its timing, host and outcome in a migrations_applied collection, so "did the index build run on production?" has an answer other than memory. Two repairs from this week, the tenant index build and the empty-rollup prune, are the first entries.
- PlatformThe API edge speaks HTTP/2 and resumes TLS sessions. Measured from Ahmedabad: a request that does no work cost 270 ms to connect, 280 ms to handshake and 265 ms to answer, 1.08 s cold against 0.26 s on a warm connection, and a console page of twenty calls spent 4.82 s on HTTP/1.1's six connections. Over one HTTP/2 connection those calls multiplex, and a returning client skips the certificate exchange.
Improved
7- AnalyticsAn idle organization no longer writes a rollup row for every empty bucket. One created in September that had done nothing held 668 rollup rows, 666 of them zero, every one rewritten by the ten-minute cron, about 290 database operations per organization per pass for charts that render an absent bucket as zero anyway. Only a small expiring watermark is written now, and the existing zero rows are pruned by migration.
- PlatformEvery list endpoint bounds its page size. A request for limit=0 went straight to the paginator, which treats zero as no limit, and the endpoint returned the whole collection; limit=100000 did the same. Anything that is not a positive number now becomes the default page, and anything above 500, the largest page any console asks for, is clamped to it.
- PlatformA read can decline a model's default joins. Every email history read also loaded the user, contact and audience behind each row, whether the caller wanted them or not. The chart's read of 5,899 rows, which only counts statuses, took 530 ms that way and 119 ms without.
- PlatformA health check miss during a busy minute is not a dead replica. The checks do no I/O, so a miss meant the host was too busy to answer inside five seconds, three probes running, which on a box that also runs the database and hourly backups happens; twice on 7 October a healthy replica was restarted for it, and every restart cost users about four seconds of first-byte on a cold process. A replica now has to fail for two minutes before it is restarted, and a fresh one gets a minute to warm up uncounted.
- PlatformThe load balancer follows replicas through DNS. It resolved the replica address once at startup, so every restart since 25 September left a dead address in its list, and on 7 October four restarts produced 62 user-facing 502s in one minute. The peer list now follows Docker's DNS, a replica is removed after three failures rather than one, and requests round-robin across both replicas instead of pinning a client to one, so a user whose replica just restarted is no longer served every request from the cold one.
- Self-hostingThe MongoDB log rotates. Nothing rotated it and the package ships no rule, so production's single log file had reached 29 GB. It rotates daily now, fourteen kept, compressed a day late so yesterday's stays readable.
- PlatformDatabase backups keep two tiers: hourly dumps for two days and a daily dump for thirty, instead of 720 hourly archives. At today's size that is the same protection for a fraction of the storage; at ten thousand organizations it is the difference between 780 GB and 7.2 TB.
Fixed
4- AnalyticsThe analytics rollup collection was created with no indexes, in every organization's database. The upsert that brings it into being used a raw model that builds none, so the query it runs on every pass scanned the whole collection: 47,592 documents examined per upsert, and 97,929 of roughly 98,500 slow queries in the log were this one collection. Sent in a batch, that is what crashed the server during email and campaign analytics. Every raw write that can create a collection now builds its indexes, a test fails if one is added that does not, and existing databases get the build from the migration runner.
- AnalyticsWith the index built, the rollup queries still could not use it. Every soft-deletable model queries with deleted not equal to true, and the unique index is partial on deleted equal to false, which the planner cannot prove from the first; a single-bucket upsert examined 6,104 documents through the wrong index. The rollup queries now name deleted: false, and the same upsert examines one.
- EmailThe email history chart re-scanned every email for every tracking event. On one organization's last thirty days that was 2,129 events by 5,897 emails, about 12.5 million date formats on the event loop, after loading 65 MB of rendered email bodies the chart never read. The process answered nothing for over a minute, was read as dead and restarted, eleven times on the morning of 7 October, each one a minute of 502s for every user. It is two passes now, each date formatted once, reading only the fields it counts.
- PlatformThe production database backup never uploaded. The dump worked and then failed to copy, for two bugs that could only show once a bucket was set: a flag that pointed the uploader at the text log as a manifest, and a log line captured as if it were the archive path. Until this week the disk's daily snapshot was the only copy. Verified by a real run into the bucket and by reading the lifecycle rules back.