Logs: what to write and how to read them

Why bother

Printing to a console is fine while you're sitting at your own machine. When the program runs on a server and failed at night for a stranger, the only way to know what happened is what it recorded.

A log is a program's memory of what happened to it.

Levels

Five are conventional, from most detailed to most alarming:

• Debug — details needed only during investigation. Switched off in normal operation.
• Info — significant events: started up, processed an order, sent an email.
• Warning — something went wrong but the program coped: a third-party service didn't answer, we retried.
• Error — an operation failed. A user was affected.
• Critical — the program can't continue.

The point of levels is to see only what matters day to day, and to switch on detail during an investigation without changing code.

What must be logged

Startup and shutdown. With the version and key settings. The first thing anyone checks during an investigation is which version is actually running.

Errors in full. With the call chain, not just the text. "Error while saving" without detail is useless.

External calls. To the database, to third-party services: what was requested, how long it took, what came back. Most problems in real systems live at the boundaries.

Key domain events. Order created, payment succeeded, user registered.

What must never be logged

Passwords and keys. Never. Logs are read by more people than you think and kept longer than you'd expect.

Personal data. Full card numbers, identity documents, addresses. If records must be tied to a person, log an identifier rather than a name.

Everything, at every step. A log recording every line of execution is unreadable and takes more space than the program.

Writing usefully

Add a correlation identifier. A request identifier carried through every record of one operation. Then an investigation can pull the whole chain rather than searching by timestamp.

Write structured records. Not plain text but fields: event, identifier, duration, outcome. Such records can be filtered and counted.

Write in one language. A mixture is painful to search.

Record what happened, not how you feel. "Failed to connect to the database, attempt 3 of 5" is useful. "Something went wrong" isn't.

Reading them

Start at the time of failure and work backwards, not from the top of the file.

Look for the first error, not the last. One breakage often causes a chain of consequences, and the final message is the least useful.

Watch for absences. Sometimes what's missing matters more: if "processing started" appears and "processing finished" doesn't, you know where to look.

Useful commands:

tail -f app.log              follow in real time
grep ERROR app.log           errors only
grep -A5 -B5 "text" app.log  with surrounding context

About rotation

Logs grow and one day fill the disk — a classic cause of a server dying at night. Configure size limits and deletion of old records immediately, not when space runs out.