Job Search Agent / August 2026 9.
An 82 match score means nothing if we cannot explain where it came from
While tuning ranking, I built a dedicated diagnostic export and AI-health tooling. The final score was not enough; the system needed to reproduce the inputs and intermediate signals behind the decision.
Situation
When a job landed unexpectedly high or low in the ranking, the final score showed only the symptom. The real issue could be scraper data, deduplication, role classification, eligibility, requirement review or cache state. Trying to reverse-engineer all of that from a single number turns every tuning change into guesswork.
Approach
I built a reproducible scoring diagnostics export that exposed the important inputs, score components and review state. AI requirement health received its own diagnostics as well, and these tools were included in lint, typecheck and image-build paths so they would not decay as forgotten developer side scripts.
Outcome
Ranking failures became much faster to localize. We could see when the score itself was correct but the source record was polluted, or when stale AI review came from cache. Tuning changes could then be justified against concrete regression examples.