The hardest thing to fake
invoiceINV-LIVE-* · AUTHORISED · $1,035.00
re-syncadopted by reference · no duplicate
A real invoice, in a real Xero developer organisation. Not a mock. A live OAuth connect, an invoice created and read back, and a second sync that adopted the existing invoice rather than duplicating it.
The live run immediately surfaced two defects the in-repo fake could not model: apps created on or after 2026-03-02 reject broad scopes with invalid_scope, and real Xero rejects an AUTHORISED invoice with no DueDate. Both fixed, both re-verified against the live org.
That’s why it leads. The fake was honest, thorough, and still wrong in two ways only real money infrastructure could reveal.
Three defects no test could catch
In each case the code was internally consistent, fully tested, and disagreed with the real-world rule it claimed to implement.
The boil-water notice that would have lifted early
The stand-down gate required consecutive clear lab results after a contamination declaration. It filtered them on the time the lab report was imported.
createdAt > declaredAt✓ counts as a post-incident clearThe import landed after the declaration, so the gate accepted it. The sample predates the contamination entirely.collectedAt > declaredAt✗ excluded — predates the incidentCollection time is the authoritative one. An unknown collection time is excluded fail-closed: unknown is never safe.E. coli culture takes 18–24 hours. So samples collected before a contamination event are routinely imported after it is declared. Both then passed the filter, the gate saw two clear results, advanced the incident toward closure, and wrote an append-only event asserting “2 consecutive clear FINAL results” — when zero post-declaration samples had been collected.
The common case. Not an edge case.
The codebase was already explicit everywhere else that source time governs. This was the one place that reached for the server clock.
Then its own review found a defect in the fix. Correcting the clock imported a hazard the old clock’s precision had been hiding.
The replaced timestamp was server-generated at microsecond precision — ties were impossible. Collection time is operator-entered at minute precision — ties are routine. A single-key sort left tied rows ordered by physical row placement, able to resolve fail-open. Now a total order.
Charging 15% GST to businesses not registered to collect it
taxRegistered: false was frozen onto the invoice and correctly resolved the rate to zero.standard treatment resolved to NZ OUTPUT2 — and the customer was charged.Inland Revenue treats adding GST to prices as itself a registration trigger. An unregistered supplier may not do it.
No test could catch this: every fixture in the repository is taxRegistered: true. The unregistered path was never exercised.
The fix needed no invention — the correct non-GST mapping already existed in every country pack. Nothing had ever routed to it.
Three engines agreeing perfectly on the wrong number
A reconciliation gate proved a TypeScript spec, a pure-Python port and a DuckDB SQL transform all produced an identical contribution margin. Green, always.
3390c2223cthe headline was 52.5% too high
The revenue feeding it was GST-inclusive — the fixture said so in its own comment — and the engine consumed it as though it were ex-GST. GST collected is remitted to Inland Revenue. It was never margin. Meanwhile the repository asserted in roughly eight places that GST is “never mixed into” contribution margin.
The gate was sound as a gate — and it never proved the shared number was right. Three faithful implementations of the same wrong formula agree perfectly.
That is why a permanently-green pipeline never surfaced it. It’s recorded in the evidence document rather than quietly restated.
What these have in common
None was found by testing. All three came from grounding the code against the published standard it claimed — drinking-water rules, GST rules, the ACSC Essential Eight. Those standards are free and precise, and the products had already made checkable promises against them.
Across all ten products in one pass: 34 confirmed findings, 7 HIGH, each citing a fetched primary source and independently re-checked by a skeptic on both axes — is the rule real, and is the code claim real. Eight candidates were rejected on verification.
Equally important, the method refused to manufacture findings. It corrected my own framing where I’d asserted a product claimed standards it never claimed. It declined to assert a vendor permission rule it couldn’t fetch a source for — “I will not assert a requirement I did not verify.” And it gave credit where a product honestly declared its own limitation.
The ten
TradeOps is the deep one — a complete workflow with a verified integration. The other nine model regulated domains at vertical-slice depth. Ten shallow products would be a weakness; one deep product plus nine credible domain models is the actual shape.
| Product | Domain | Integration reality |
|---|---|---|
| TradeOps AI | Trade services · NZ/AU | Live Gmail IMAP · real LLM extraction · official Xero sandbox |
| ControlOps ANZ | Security evidence · M365 | Live Microsoft Graph, consenting dev tenant — 92 observations |
| WaterOps ANZ | Drinking water · NZ | Live cross-service E. coli scenario against real PostGIS |
| GeoRisk ANZ | Spatial exposure · NZ | Real PostGIS · EPSG:2193 exact-area · least-privilege DB role |
| FactoryOps ANZ | Industrial telemetry · OPC UA | Independent OPC UA server → RLS ingestion |
| TrustID ANZ | Verifiable credentials | Out-of-process verifier over real public HTTP |
| OpenFinanceOps ANZ | Consumer Data Right · AU | Real HTTP/PKCE CDR journey vs an in-repo mock holder |
| MarginFlow | Commerce margin · data engineering | Python/DuckDB pipeline reconciling to the tested engine |
| EnergyOps ANZ | EV charging · OCPP | OCPP simulated — labelled as such |
| CareOps ANZ | Health data · NZ | Synthetic |
Rules the code is held to
Every merge gated. Full suite, a Docker multi-scanner in strict mode, adversarial multi-agent review, green CI. Protected mains; immutable release tags — corrections ship as a successor version, never by moving a tag.
Unknown is never safe. A missing cost is excluded from a margin, never zeroed. A partial evidence collection turns checks amber, never green. A sample with no collection time is excluded fail-closed.
Simulated is never reported as integrated. A fixed vocabulary separates implemented / tested / integration-verified / official-sandbox-verified / simulated / demo-ready / externally-reviewed / production-validated — and nothing is promoted without the evidence for that word.
What isn’t true of this work
- Not externally reviewed. No practitioner outside the project has evaluated any of it. Self-review and AI review are not external validation and are never recorded as such.
- Not production-validated. Nothing has run in production or a pilot.
- Not certified. Reading a regulation is not conformance. The conformance work records a cited position with known gaps — 31 findings remain open and registered, four of them HIGH.
- The scanner’s own gate failed open until recently. An unrecognised category matched nothing, so a strict gate looked enabled while protecting nothing. Every prior “gate passed” was re-verified by measurement rather than assumed. They held — because no repository overrode the default. That check is recorded because the alternative, assuming, is how the three failures above happened in the first place.