Most tests check that software does what it should. For systems that move money, sign documents or hold shared records, that is only half the job. The other half is checking that the software refuses what it should refuse, every time, even when someone is trying hard to get past it.
The rule
Before we write a defence, we write a test that performs the attack. The test fails, because the defence does not exist yet. Then we write the defence and watch the test pass. The test stays in the suite forever, so the defence can never quietly disappear in a later change.
Real examples
These are test names from our open-source blockchain node, PlainChain, which has 622 automated tests:
- "rejects a block with an inflated coinbase and keeps its state intact": a miner tries to pay itself more than the rules allow.
- "survives a peer that sends garbage and keeps serving good peers": a hostile computer on the network sends nonsense.
- "a junk block claiming a legitimate block's hash cannot poison that hash": an attacker tries to get a real block rejected by sending a fake one with the same name.
That last one was a real bug we found by writing attacks. The node remembered the name the sender claimed for a block instead of checking it. The fix was one line. Finding it needed the habit of asking "how would I break this?"
A threat model that points at code
Writing attacks one at a time is not enough on its own, so we also keep a threat model: a list of everything that could go wrong, who might cause it, and what stops it. In our version, every row names the code that defends against the threat and the test that proves it. If a row has no test, that is a gap, and gaps get fixed.
Why it matters for your project
- Auditors and reviewers can check each claim against a test instead of trusting a document.
- New team members can see why each rule exists by reading its attack.
- Changes are safer, because a removed defence fails a test immediately.
We apply this to AI systems too: red-team prompts are attack tests for an agent. If you have a system that has to be right, request an audit.