Job Search Agent / August 2026 9.
The scraper sometimes mistook half a page for a single job description
Some ranking failures were not scoring bugs at all but data-quality problems: aggregator navigation, multiple listings and page chrome had been merged into one supposed job record.
Situation
With No Fluff and other HTML sources, the crawler sometimes received a dense listing blob instead of a clean detail page. Several job titles, navigation text and salary fragments could end up in the same record. The ranking system then scored that garbage with complete confidence. The smarter the scorer becomes, the more confidently it can make a bad decision from bad input.
Approach
I moved the fix to the start of the pipeline: navigation-artifact filtering, dense-listing detection, safe detail hydration and deduplication that prefers a clean detail record. Best Match curation was allowed only when a record had genuine detail-page evidence behind it. Ranking diagnostics also exposed which input fields had contributed to the decision.
Outcome
Many false positives disappeared without aggressively retuning the scoring weights. The system stopped trying to be clever on malformed HTML and became more willing to say that there was not enough clean data for a confident ranking decision.