Logging, telemetry and the discipline of post-mortems

Log decisions, not just outcomes

Most systems log fills. Fills tell you what happened and nothing about why. A useful log records the inputs at the moment of the decision — volatility reading, spread, exposure, filter states — so that months later you can answer the only question that matters: given what it knew, was the action correct?

A minimum viable log line

• —Timestamp in broker server time and UTC. Ambiguity here has cost people entire investigations.
• —Symbol, direction, requested price, filled price, and the difference between them.
• —Computed lot size and the equity figure used to compute it.
• —Every guard-rail state: spread at decision, news window active, exposure count.
• —A stable decision identifier so modifications and closes can be linked back to the entry.

The post-mortem habit

Once a month, take the three worst trades and reconstruct them from the log alone. Half the time the system did exactly what it should and the market simply moved. The other half is where every genuine improvement we have shipped originated.

What a decision log should contain

The goal of logging is to make any past moment reconstructable without guessing. That means recording not only what the system did but what it believed at the time: the inputs, the derived state, the rule that fired, and the values that made it fire. A log that records outcomes without inputs answers no interesting question.

One line per decision, structured

• —Timestamp in both broker and UTC time.
• —The rule identifier and version that produced the decision.
• —Every input value the rule consumed, including account state.
• —The intended action and the observed result, separately.

Running a blameless post-mortem on your own system

When something goes wrong, the useful output is not an explanation but a change: either a code change that removes the failure mode or a documented decision to accept it. The failure of most retail post-mortems is that they stop at 'the broker rejected the order' rather than asking why the system had no defined behaviour for that case.

A five-question template

An incident without a change is an anecdote. The only proof you learned anything is a diff.