The Results
The three strategies flagged 52 issues total. Of those, 40 were found by multiple strategies (useful validation), but 12 were unique to a single approach. Five of those 12 would have broken production. Only one was a false positive.
Which Strategy Found What
Here's where it gets interesting: Contrastive prompting found 8 of the 12 unique issues, including 4 of the 5 blocking bugs. Positive found 3 unique issues (1 blocking). Negative found just 1 unique issue (0 blocking).
That's 67% of the unique critical findings from a single strategy.
The 40 overlapping issues are useful validation. When all three strategies flag the same thing, you can be confident it's real. But the unique findings are where strategy choice actually matters.
The 5 Bugs That Would Have Broken Production
Let me walk through the blocking issues Contrastive found that the other strategies missed.
01
Silent Data Corruption
The plan's original SQL:
UPDATE workspaces SET use_case_id = (SELECT id FROM use_cases WHERE name = 'contracts')The problem Contrastive identified:
"A naive review assumes the subquery will always return a result. But if the use case doesn't exist yet (migration order issue) or there's a typo ('contract' vs 'contracts'), the subquery returns NULL. The UPDATE succeeds, setting ALL workspaces to use_case_id = NULL. No error. Silent data corruption."
The fix:
SELECT id INTO STRICT contracts_id FROM use_cases WHERE name = 'contracts'; -- STRICT throws an exception if not exactly 1 rowThis is the kind of bug that passes every test in staging (where the use case exists) and destroys production (where migration order might differ).
02
Migration Timeout Math
The plan called for:
for clause in clauses: # 950 clauses
await db.insert(clause) # Sequential insertsContrastive's calculation:
"950 clauses × 50ms per insert = 47.5 seconds. Alembic's default timeout is 30 seconds. This migration is mathematically guaranteed to timeout. Worse: it'll fail mid-execution, leaving 600 clauses inserted. Retry hits unique constraint violations."
Neither Positive nor Negative caught this. Positive generated a checklist item for "verify migration completes" but didn't do the math. Negative prohibited "long-running transactions" but didn't flag this specific pattern.
The fix: Bulk insert (950 rows in ~5 seconds instead of 47).
03
LLM Schema Hallucination
The system lets users generate extraction schemas via natural language. The plan included security validation (no code injection) and Pydantic validation (valid structure).
Contrastive identified the gap:
"User says 'generate schema for medical records.' LLM returns contract fields (parties, clauses, terms) because it's been trained on lots of contract examples. Security validation passes (valid JSON). Pydantic passes (correct structure). But semantically? Garbage. User extracts 50 medical PDFs, gets contract fields, loses trust in the system."
The fix: Add semantic validation that checks if schema fields match the stated use case.
04
Type Mismatch Across Services
The plan routed old use cases to the legacy service and new use cases to the new service:
if use_case_id == contracts_id:
result = old_service.process(...) # Returns ExtractionResult
else:
result = new_service.extract(...) # Returns ExtractionAuditTrail
return result #