45 minutes, $460 million, and a flag nobody remembered

On 1 August 2012, Knight Capital, then one of the largest traders in US equities, deployed new code for a retail trading programme. Within about 45 minutes of the market opening it had sent millions of unintended orders and lost roughly $460 million. The firm did not survive as an independent company.

What happened

The public accounts agree on the core sequence:

• The new code reused a configuration flag that, years earlier, had switched on a function called Power Peg.
• Power Peg had been out of use since 2003 but was never removed. A later change had disabled the safeguards it relied on.
• The new code was deployed to seven of eight servers. On the eighth, the old code was still there, and the reused flag switched Power Peg back on.

No single step was exotic. A flag was reused because it looked free. Dead code stayed because removing it looked risky. A deployment missed a server.

The question nobody could answer

Strip away the trading details and the failure rests on one unanswered question: what does this flag still control? The people who knew had built Power Peg nearly a decade earlier. Nothing in the system said it was dangerous, so the flag looked like any other unused switch.

Every company has its own Power Peg

Most companies will never lose $460 million in an hour. But most have:

• Flags and config values whose original purpose nobody remembers.
• Code paths that “aren’t used any more”, that nobody dares delete.
• Thresholds whose reason left with the person who set them.

Each is a decision without an owner. The cheapest moment to record why it exists is while someone still knows.

What would have helped

Not more process at deploy time, but an answer at design time: a record that said this flag enables Power Peg; Power Peg is retired; its safeguards are disabled; do not reuse, with an owner who would be asked before anyone changed it. See Keeping answers current.

Sources