Columbia Games · CCM v2 / August 2026 26.
The latest deploy is not necessarily the latest good deploy
Change detection and recovery were too willing to treat the most recently attempted production SHA as a valid baseline. The fix was to model the last successful runtime SHA as a separate piece of state.
Situation
A deployment may fail halfway through or during post-deploy QA, yet its commit still appears in deployment history. If the next run uses that attempted commit as its comparison baseline, it can assume certain changes already exist in production even though they were never validated there. The same mistake is even more dangerous during recovery.
Approach
We stored the last successful runtime SHA separately and updated it only after a release had completed and passed verification. Change detection compared against that verified state rather than the last attempted commit. I explicitly prevented an unverified production SHA from being promoted automatically into a known-good state.
Outcome
After a failed deployment, the next run continued to calculate changes from the runtime that was actually active and proven. Recovery and incremental deployment therefore used the same representation of production reality.