Work / Validation gate

The QA Gate That Blocked a 63-of-66 Failing Pricing Path Before It Touched a Live Quote

A fuzzy-matching path guessed catalog prices from ambiguous line descriptions and looked as trustworthy as the exact-match path beside it. A QA harness tested both paths against realized prices, and the fuzzy path was pulled before it ever priced a live quote.

Anonymized. Client, employer, project, manufacturer, part, and pricing identifiers are removed; the mechanism and the numbers are real.

The user and their week

A construction quoting team matched free-text line descriptions from takeoff exports against a catalog of previously realized prices, on every job, under deadline. Exact descriptions matched reliably. Ambiguous ones were harder: field language changes between jobs, abbreviations drift, and a description can reference a related product without naming the priced item itself.

Why it mattered

A fuzzy match could look reasonable while pointing at the wrong catalog item, and once that number entered a quote total, a plausible-looking guess became a commercial decision a client would pay against. The team ran this match on every job, under deadline, with no test in place yet that separated a plausible-looking result from a correct one.

The bet, and what was rejected

The bet was to build the ambiguous-description matcher and prove it against real outcomes under a QA harness before trusting it with a dollar figure, with full manual review as the fallback if it failed that test. The harness replayed known description-and-price pairs through the matching logic and compared each result with the price actually realized, scoring the fuzzy path separately from the exact-match path so a failure in one could not discredit the part of the workflow that had already earned trust. What got rejected was the default alternative: shipping a text-similarity score straight into a quote total because it looked plausible, without a test that measured whether it had selected the correct priced item.

Acceptance criteria, set before build

The bar was set before either path touched a live total: whether the automation selected the correct priced item, scored against the price actually realized. That bar applied equally to both paths, exact and fuzzy, replaying known description-and-price pairs through the harness. It held under the pressure to ship: the fuzzy path had already been built when the harness came back with 63 wrong picks out of 66 firings, and it was pulled from production despite the work already sunk into it.

What shipped

An exact-match path that pulled a validated price whenever a description matched cleanly
A fuzzy-matching path for ambiguous descriptions, built and scored separately from the exact-match path
A QA harness that replayed known description-and-price pairs through both paths and checked each result against the price actually realized
A four-rule policy adopted after testing: exact validated matches may populate a price, ambiguous matches may identify where a reviewer should look, uncertain results carry no automatic dollar impact, and a responsible person approves every exception before submission

The human gate: no exception reaches a submitted quote without a named person approving it first; uncertain automation output carries zero automatic dollar impact until someone signs off.

What failed, and what changed

The fuzzy path was wrong in 63 of 66 cases in which it fired. A stricter version of the same path was correct only once in 21 attempts. The worst mismatch exceeded 1,000 times the reference value, and on one live quote a phantom line stood for roughly 40% of the reported total; human review had already flagged that same line on its own, so it never reached the client. The failure was structural. A description can name the item a component attaches to, or the component itself, and text similarity alone cannot reliably tell those two apart, no matter how the fuzzy path was tuned. Fuzzy matches were removed from automatic quote totals before the live quote was submitted, and the workflow changed to the four-rule policy above.

Result

The fuzzy path guessed wrong 63 times out of 66. I pulled it before it ever touched a real quote.

Against that test, the exact-match path validated 20 of 20 reference lines and 91 of 98 high-confidence lines on a job the system had never seen. The fuzzy path, tested the same way, was wrong 63 of 66 times it fired. Every one of these numbers comes from the same adversarial QA harness, each result scored against the price actually realized.

Adoption, and what carries forward

The exact-match path is the piece that shipped into the live workflow, pulling a validated price whenever a description matches cleanly.

What carries forward is the four-rule policy itself: score automation against real outcomes before trusting it, separate paths by confidence and failure mode, give uncertain results zero automatic dollar impact, and route every exception through a named approver. That discipline was named as reusable ahead of estimating QA, document extraction, schedule generation, and specification review.