Job Search Agent / August 2026 9.
The AI was too eager: it turned responsibilities into must-have requirements
A requirement-review regression showed that the model could promote a responsibility into a mandatory qualification. Explicit evidence guards and a deterministic fallback were needed around it.
Situation
A posting may say that the role works with cloud systems without stating that prior AWS experience is mandatory. Under some wording, the LLM interpreted that too strictly and generated a missing must-have that did not actually exist in the posting. A false requirement like that could incorrectly eliminate a strong role.
Approach
I tied must-have classification to explicit requirement evidence. A responsibility by itself could not trigger a hard cap. When the AI output could not support the claim with the required kind of evidence, the deterministic fallback took over. Regression tests distinguished language patterns such as 'you will work with' from wording that genuinely requires prior experience.
Outcome
The review became less aggressive but more trustworthy. The system preferred to leave an uncertain role for human review rather than reject it because of an invented mandatory requirement. The AI's decision authority received a clear evidence boundary.