Log Analysis

Start with the question

Before searching, write down what you are looking for: the time window, the affected user, order or request id, and the error you saw. Then work from the known to the unknown.

1. Find the time. Pin down when the problem happened and check which time zone the log uses (often UTC). See Dates, Times and Time Zones.
2. Find an anchor line: an error, a request id or a user id.
3. Read around it. Look at what happened just before the error, since the cause precedes the symptom.
4. Follow the identifier across all lines, and across services if the id is passed on.

Command-line tools

# PowerShell
Get-Content app.log -Tail 100 -Wait          # follow the end of a file
Select-String -Path app.log -Pattern "ERROR" -Context 3,5
Select-String -Path app.log -Pattern "req-4711"

# Bash
tail -n 100 -f app.log                       # follow the end of a file
grep -n -C 3 "ERROR" app.log                 # matches with 3 lines of context
grep "req-4711" app.log | less               # follow one request
grep -c "ERROR" app.log                      # count matching lines

Pipe results through other tools to count and compare, for example grep ERROR app.log | sort | uniq -c | sort -rn shows which lines are most frequent (strip timestamps first if each line is unique). If logs are JSON, a tool such as jq can filter on fields. Larger systems use a central log platform with a query language, but the method is the same.

Interpreting what you find

• Frequency matters. An error seen 3 times in a day differs from one seen 3,000 times in a minute. Check whether it is new: was it present yesterday?
• Some errors are harmless and always there. Do not assume the first ERROR is the cause; look for the first one after things changed.
• Look for patterns: the same user, host, endpoint or time of day.
• In a stack trace, the top shows where it failed and the “caused by” entries show why.
• Missing lines can mean a crash, rotation or a different log file. Clocks on different machines may differ slightly.
• Copy the relevant lines, with timestamps, into your notes or ticket, with sensitive data removed.

Findings feed into Troubleshooting Methodology and Root Cause Analysis. What you can find depends on how well the application logged; see Logging Fundamentals.

Try it

Generate a log file (run a small application, or write a script that appends random INFO, WARN and ERROR lines with timestamps). Use Select-String or grep to count the errors, find the first one, show three lines of context around it, and list the most frequent messages.