Audit logging

An application log answers what went wrong. An audit log answers who did what, and when — and it is read by a different person, for a different reason, often years later: a security investigation, a customer dispute, an auditor, a court.

That difference in reader changes almost every design decision. An application log can be sampled, rotated after a week, and lose a line under load. An audit log that can lose a line is not an audit log.

#What an audit event records

For any action that changes something that matters — money, permissions, personal data, configuration — an event carries:

That last row is the one most often left out. An audit log of only successful actions cannot show an attack, because an attacker's first twenty attempts are the ones that fail. The OWASP Logging Cheat Sheet lists authentication and authorization failures among the events that should always be recorded.

#What an audit event must never record

The same care in the other direction. An audit log is kept for a long time, read by many people, and copied into other systems — so it is a terrible place for a secret.

• No passwords, including wrong ones — a mistyped password is usually one character off the real one.
• No tokens, session IDs in full, or API keys.
• No full card numbers — PCI DSS requires them to be unreadable wherever they are stored, and an audit table is storage.
• Personal data only as much as the purpose needs. Record that a customer's address changed and who changed it; think hard before recording the address.

#The write that must not be lost

The subtle failure is not in what you record but when. Consider the obvious code:

@Transactional
public void grantRole(long userId, Role role, Admin by) {
    users.grantRole(userId, role);
}
// ...and the caller, afterwards:
roleService.grantRole(77, Role.ADMIN, admin);
auditClient.send(new AuditEvent(admin, "ROLE_GRANT", 77));   // a separate step

The role change commits. Then the process crashes, or the audit service times out, and the event is never sent. The permission exists and no record says who granted it — exactly the gap an attacker, or an insider, would want.

The fix is to make the audit record part of the same transaction as the change it describes:

@Transactional
public void grantRole(long userId, Role role, Admin by) {
    Role before = users.grantRole(userId, role);
    auditEvents.save(AuditEvent.of(by, "ROLE_GRANT", "user:" + userId, before, role));
}

Now both commit or neither does. If the audit trail must end up in a separate system, write it to a table in the same transaction and ship it from there afterwards — the transactional outbox from the distributed data course, applied to audit.

#Append-only, and why that is not enough

An audit log is append-only: rows are inserted and never updated or deleted. Enforce it where the application cannot argue: the database user that writes audit rows gets INSERT and SELECT on that table, and no UPDATE or DELETE.

That stops the application from rewriting history. It does not stop a database administrator, or an attacker who has become one. For that, you need the log to be tamper-evident: an edit is still possible, but it can be detected.

The standard technique is a hash chain. Each entry stores the hash of the previous entry, and its own hash covers both:

static void append(List<Entry> log, String event) throws Exception {
    String prev = log.isEmpty() ? "0" : log.get(log.size() - 1).hash;
    log.add(new Entry(event, prev, sha256(prev + "|" + event)));
}
 
static int firstBroken(List<Entry> log) throws Exception {
    String prev = "0";
    for (int i = 0; i < log.size(); i++) {
        Entry e = log.get(i);
        if (!e.prevHash.equals(prev) || !e.hash.equals(sha256(prev + "|" + e.event))) return i;
        prev = e.hash;
    }
    return -1;
}

Three events are appended — a refund, a role grant to ADMIN, an export. Then someone edits the second entry in place, to say the role granted was VIEWER:

intact, first broken: -1
edited, first broken: 1

Verification finds it and names the entry. That is the chain doing its job.

Now the attacker is more careful. They do not edit one row; they rebuild the whole chain from the point they changed, recomputing every hash after it:

rewritten, first broken: -1
head matches anchor: false

The rewritten chain verifies perfectly. Every link is consistent, because the attacker computed them all. A hash chain on its own proves only that the log is internally consistent — and anyone who can write the whole table can make it consistent.

What caught the rewrite is the second line. Before the tampering, the hash of the latest entry was copied somewhere the attacker cannot reach, and the rebuilt chain ends in a different hash. That copy is the anchor, and without one the chain is decoration:

• Publish the head hash periodically to a separate system with separate credentials.
• Ship audit events to write-once storage — object storage with a retention lock, for example, which refuses deletes until the retention period ends.
• Or hand the log to a separate team or service entirely, so no single set of credentials can rewrite both the log and its evidence.

#Retention, and reading it

Keep audit logs as long as the obligations that apply to you require — financial and payment regulations commonly specify a year or more — and plan the storage for that. A year of audit events is rarely large; a year of application logs is.

And test that it can be read. The question an audit log exists to answer is something like "everything admin raj did between the 3rd and the 10th", asked under pressure. If that query needs a script someone has to write first, it will not be answered well on the day.