Why AI tools get your business rules wrong

A billing service has a hard-coded 3% discount with no comment. In one comparison of five AI coding agents on a legacy codebase, the agents reportedly treated it as a bug and removed it¹. The discount came from a negotiated vendor contract. The contract was a PDF. The repository never mentioned it.

The agent was not careless. It had no way to know. The same gap shows up in every place AI tools now touch business rules.

Where do AI tools go wrong with business rules?

In three places, and all three come from the same gap.

Agents delete rules that look like bugs. A branch with no test, no comment and no requirement pointing at it looks like dead code. Deleting a branch now takes an agent minutes. Finding out why the branch exists takes as long as it always did.

Assistants explain rules with reasons nobody gave. Ask an assistant over your wiki why orders above a certain amount go to manual review and you get a fluent paragraph. Retrieval does not guarantee the answer is grounded: a peer-reviewed study of commercial legal research tools built on retrieval found incorrect or misleading answers 17% to 33% of the time². Inside a company, a plausible reason gets written into a design document, and the next assistant retrieves it as a source.

Generated code adds rules nobody decided. An assistant writes a payout hold from a short prompt, the tests pass, and a busy reviewer approves. Six months later nobody can say why the threshold is that number. No person ever decided it. In a randomized trial, engineers who learned a new library with an AI assistant scored 50% on a later quiz, against 67% for those who coded by hand, and the widest gap was on debugging³.

Why can’t we just give the model the whole codebase?

A bigger context window gives the tool more code. It does not give it more reasons.

The codebase holds the rule, if region == "X": skip_check(), and perhaps a commit message that says “fix”. It does not hold the audit finding behind the exception or the partner agreement that made it necessary. Those sit in a contract, an incident review from four years ago, a deleted chat thread, or the head of someone who has moved teams.

Tests don’t close the gap either. A test suite encodes the behavior someone thought to write down. If nobody recorded why a rule exists, nobody wrote the test that fails when it is removed, and a green run proves nothing.

Code review has the same limit. A reviewer can confirm a condition is written correctly. The diff doesn’t show whether the threshold is right, or whether another team’s reconciliation depends on it.

What does an AI tool need before it touches a rule?

The same facts an engineer needs:

1. What the rule is and where it lives.
2. Why it exists, in the words of the person who decided.
3. Who owns it, so the change reaches them.
4. What depends on it, so the effect elsewhere is visible first.
5. How sure anyone is. A reason the owner confirmed last month is not a guess from a commit message, and a tool that cannot tell them apart treats both as fact.

The fifth fact is the one AI tools lack by default. A model built to be helpful completes the answer. “Nobody has recorded why” is the more useful answer, because it tells the engineer to go and ask.

Where Organization Memory fits

AppThentic reads a company’s code, finds the business rules in it, asks the person who owns each rule why it exists, and keeps the answer with its source, owner, date and status. That record is Organization Memory.

Every rule carries one of seven statuses. Approved means the owner confirmed the reason. Inferred is AppThentic’s own reading. Unknown is recorded as an answer, not filled in. When the code changes, the answer goes Stale and the owner is asked again.

That changes all three failures:

• Agents connect through the Model Context Protocol. Claude Code, Cursor and other MCP-capable assistants look up a rule’s reason, owner and dependants before they edit it, and flag Disputed, Stale or Unknown rules instead of treating the code as the whole truth. See AI coding tools.
• Questions get answers built from Organization Memory, with the status and source of every rule they rely on. With no approved answer, AppThentic says so and offers to ask the owner. See Asking AppThentic.
• Generated rules show up as rules with no recorded reason, so the owner is asked before anyone relies on them.

The agent that found the 3% branch sees the vendor-contract reason and the owner beside it, and the change goes to that person before anything is deleted. Change impact shows what depends on the rule before it moves.

What to do this week

Pick one service where AI tools already make or review changes.

1. List the ten rules in it that involve money, dates, regions or customer types.
2. Ask your assistant why each one exists, then ask the owner separately.
3. Mark every rule where the owner says “nobody wrote that down” or “an assistant suggested it”, and every one where the assistant gave a confident reason the owner doesn’t recognize.

Those are the rules an agent will treat as bugs. The same logic is behind Chesterton’s fence, and it applies at full force when you hand a legacy system to AI to modernize.

To see this on your own systems, book a walkthrough.

Sources