Skip to content

Level 1 capstone: Analyze a web server log

Level 1 · Capstone · ⏱️ 2–3 hours · Prerequisites: all of Level 1, especially Pipes and redirection and Text processing

You are the on-call engineer for a small online shop. Yesterday's nginx access log has landed on your desk with ten questions from your team. Answer every one of them with pipelines of the classic tools from this level, typed at the prompt. No spreadsheets, no Python, no text editor.

This is exactly the kind of work the command line is best at, and it is the skill you will use most often in real jobs: someone asks a question about a big text file, and you have the answer before they finish their coffee.

Rules

  • Use only the tools from Level 1: grep, cut, sort, uniq, wc, head, tail, tr, sed, awk, xargs, paste, column, and shell redirection.
  • Each answer should be one pipeline typed at the prompt (two for questions with two parts). No temporary files except access.log itself.
  • Answer from the data. The generator below contains the answers in its code; resist reading it closely until you have finished.

Set up: generate the log

The log is generated by a short awk program, so everyone gets exactly the same 5,000 lines and can compare answers with the solution. It is in the nginx combined format you met in Text processing, and simulates one day of traffic to shop.example.com: real customers, a search-engine crawler, an internal script polling an API, a few bots, and at least one incident.

Create a working directory:

mkdir -p ~/practice/capstone && cd ~/practice/capstone

Then paste this whole block into your terminal:

awk -v seed=42 -v total=5000 '
# A tiny seeded random number generator (Park-Miller). We use it instead of
# rand() because the awk on Mint (mawk) ignores srand(seed), so rand() never repeats.
function rnd()     { seed = (seed * 16807) % 2147483647; return seed / 2147483647 }
function pick(n)   { return int(rnd() * n) + 1 }    # random integer 1..n
function chance(p) { return rnd() < p }             # true with probability p
BEGIN {
    day = "21/Sep/2026"

    # Relative traffic per hour, 00:00 to 23:00: quiet at night, busy by day,
    # with +/-15% random noise so no two days look the same
    split("2 1 2 2 1 2 3 5 7 9 10 10 9 9 10 10 9 8 7 7 8 7 5 3", weight, " ")
    for (h = 1; h <= 24; h++) { weight[h] *= 0.85 + 0.3 * rnd(); wsum += weight[h] }

    # 120 regular visitors, each with a fixed browser
    split("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15|Mozilla/5.0 (X11; Linux x86_64; rv:125.0) Gecko/20100101 Firefox/125.0|Mozilla/5.0 (iPhone; CPU iPhone OS 17_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Mobile/15E148 Safari/604.1|Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Mobile Safari/537.36", browsers, "|")
    split("192.0.2. 198.51.100. 203.0.113.", nets, " ")
    for (i = 1; i <= 120; i++) {
        visitor[i] = nets[pick(3)] pick(254)
        agent[i] = browsers[pick(5)]
    }

    # Pages regular visitors request: "METHOD PATH WEIGHT"
    n = split("GET / 12|GET /products 8|GET /product 15|GET /cart 5|GET /checkout 3|GET /login 3|POST /login 3|GET /search 6|GET /api/products 6|POST /api/cart 5|GET /api/orders 3|POST /api/orders 3|GET /static/css/main.css 8|GET /static/js/app.js 8|GET /images/logo.png 6|GET /favicon.ico 4|GET /about 2", pages, "|")
    for (i = 1; i <= n; i++) { split(pages[i], f, " "); meth[i] = f[1]; path[i] = f[2]; cum += f[3]; cumw[i] = cum }
    split("laptop|headphones|usb-c+cable|monitor|keyboard|desk+lamp|webcam", terms, "|")
    split("/wp-login.php|/.env|/admin|/phpmyadmin/|/.git/config|/xmlrpc.php", probes, "|")

    line = 0
    for (h = 0; h < 24; h++) {
        count = (h < 23) ? int(total * weight[h + 1] / wsum + 0.5) : total - line
        for (k = 0; k < count; k++) {
            sec = int((k + rnd()) * 3600 / count)          # increasing within the hour
            ts = sprintf("%s:%02d:%02d:%02d +0000", day, h, int(sec / 60), sec % 60)
            ref = "-"; bytes = 0

            if ((h == 2 || h == 3) && chance(0.8)) {          # night-time vulnerability scanner
                ip = "198.51.100.23"; ua = "Mozilla/5.0 zgrab/0.x"
                m = "GET"; p = probes[pick(6)]; status = 404; bytes = 162
            } else if (chance(0.05)) {                          # search engine crawler
                ip = "192.0.2.77"; ua = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
                m = "GET"; id = pick(60); p = "/products/" id
                if (id > 50) { status = 404; bytes = 162 } else { status = 200; bytes = 4000 + pick(8000) }
            } else if (chance(0.06)) {                          # internal script polling the API
                ip = "203.0.113.10"; ua = "python-requests/2.31.0"
                m = "GET"; p = "/api/products?page=" pick(5); status = 200; bytes = 800 + pick(2000)
            } else {                                            # a regular visitor
                v = int(120 * rnd() * rnd()) + 1              # skewed: some visit much more
                ip = visitor[v]; ua = agent[v]
                r = rnd() * cum
                for (i = 1; cumw[i] < r; i++) ;
                m = meth[i]; p = path[i]; status = 200
                if (chance(0.5)) ref = "https://shop.example.com/"

                if (p == "/product") {
                    id = pick(60); p = "/products/" id
                    if (id > 50) status = 404
                } else if (p == "/search") {
                    p = "/search?q=" terms[pick(7)]
                } else if (p == "/login" && m == "POST") {
                    if (chance(0.3)) status = 401
                } else if (p ~ /^\/(static|images)\// || p == "/favicon.ico") {
                    ref = "https://shop.example.com/"
                    if (chance(0.2)) status = 304
                }
                if (p ~ /^\/api\//) {
                    if (h == 14 && chance(0.45)) status = (chance(0.65) ? 502 : (chance(0.7) ? 503 : 500))
                    else if (chance(0.01)) status = 500
                } else if (status == 200 && chance(0.004)) status = 500

                if (status == 304) bytes = 0
                else if (status == 404) bytes = 162
                else if (status == 401) bytes = 58
                else if (status >= 500) bytes = 157 + pick(40)
                else if (p == "/static/js/app.js") bytes = 154210
                else if (p == "/static/css/main.css") bytes = 18342
                else if (p == "/images/logo.png") bytes = 7421
                else if (p == "/favicon.ico") bytes = 1150
                else if (p ~ /^\/api\//) bytes = 200 + pick(3000)
                else bytes = 2000 + pick(9000)
            }
            printf "%s - - [%s] \"%s %s HTTP/1.1\" %d %d \"%s\" \"%s\"\n", ip, ts, m, p, status, bytes, ref, ua
            line++
        }
    }
}' > access.log

Check that you got the same file as everyone else:

wc -l access.log
sha256sum access.log
head -n 3 access.log

5000 access.log
834d88d503b0e9d0f79dda5d64bd260d4f1b3ea8bed3d0fb8faf9d055c635cd6  access.log
192.0.2.157 - - [21/Sep/2026:00:00:11 +0000] "GET /images/logo.png HTTP/1.1" 200 7421 "https://shop.example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15"
203.0.113.203 - - [21/Sep/2026:00:01:17 +0000] "GET /cart HTTP/1.1" 200 4954 "https://shop.example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15"
192.0.2.88 - - [21/Sep/2026:00:02:43 +0000] "POST /api/cart HTTP/1.1" 200 1378 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15"

If the checksum differs, the paste was probably cut short. Delete access.log and paste the block again.

Why the generator has its own random number function

awk has a built-in rand(), and srand(42) is supposed to make its sequence repeatable. On Mint, awk is mawk, and its rand() does not repeat even after srand(42): you get different numbers on every run. Try it: run awk 'BEGIN {srand(42); print rand()}' twice. The generator therefore implements a tiny classic algorithm (the Park-Miller "minimal standard" generator) in one line, rnd(). Its arithmetic stays exact in awk's floating-point numbers, so any POSIX awk (mawk, gawk, busybox awk) produces the same file.

The IP addresses come from the ranges 192.0.2.0/24, 198.51.100.0/24, and 203.0.113.0/24, which are reserved for documentation, so they never belong to real people.

Field reference

When awk or cut -d' ' splits a line on spaces, these are the fields you need:

Field Example Meaning
$1 192.0.2.157 Client IP
$4 [21/Sep/2026:00:00:11 Date and time (hour is characters 14–15)
$6 "GET Method, with a leading quote
$7 /images/logo.png Path, including any ?query
$9 200 HTTP status
$10 7421 Response size in bytes

With awk -F'"', the request is $2, the referer $4, and the user agent $6.

Questions

  1. Size of the problem. How many requests are in the log, and how many distinct client IPs made them?
  2. Top talkers. List the top 10 client IPs by number of requests.
  3. Popular pages. List the top 10 requested paths, ignoring query strings, so /search?q=laptop counts as /search.
  4. Status codes. Count each HTTP status code, most common first. What percentage of all requests failed (status 400 or higher)?
  5. Traffic by hour. How many requests arrived in each hour? Which hour was busiest? One pair of night-time hours is suspiciously busy compared with its neighbours: which, and why? (You may need question 7 first.)
  6. The outage. Count server errors (status 5xx) per hour. Which hour had an outage? Which endpoints failed during it, with which status codes, and between which times did the errors start and stop?
  7. Not found. List the top 10 paths that returned 404, and the IP responsible for most 404s. What was that client looking for, and when was it active?
  8. Bandwidth. How much data did the server send in total, in MiB? Which 5 paths account for the most bytes, and what single file dominates?
  9. Who is visiting. List the user agents by number of requests. What share of all requests came from bots and scripts (Googlebot, python-requests, zgrab)?
  10. What customers want. What are the most popular search terms, most popular first, with + shown as a space?

Stretch. Write a single awk program that prints an hourly report: one line per hour, with the number of requests, the number of server errors, and the error rate as a percentage, sorted by hour.

Acceptance criteria

  • access.log has 5,000 lines and the checksum above.
  • Every question is answered by a pipeline typed at the prompt, using only Level 1 tools.
  • Your numbers match the solution exactly.
  • Your pipelines test fields, not text anywhere in the line: a status test cannot match a byte count, and an hour test cannot match a minute.
  • Every pipeline finishes in well under a second.
  • You can explain what each stage of each pipeline outputs, without running it.
  • You can reproduce the frequency-count idiom (sort | uniq -c | sort -rn | head) and an awk associative-array aggregation from memory.
  • You have adapted at least two of your pipelines to a real log in /var/log (see the bonus below).

Hints

Try each question for at least ten minutes before opening its hint.

Hint for question 1

wc -l counts lines. For distinct values, extract the field, then sort -u | wc -l.

Hint for question 2

This is the frequency-count idiom from Text processing: extract, sort, uniq -c, sort -rn, head.

Hint for question 3

Field 7 is the path. cut -d'?' -f1 keeps everything before the first ?, and leaves lines without a ? unchanged.

Hint for question 4

Field 9 is the status. For the percentage, awk can count matching lines in a variable and divide by NR in an END block. printf "%.1f%%" prints a percentage with one decimal.

Hint for question 5

The hour sits between the first and second colons on each line, because IPv4 addresses contain no colons. cut -d: -f2 extracts it. The log is already in time order, so uniq -c works even without sort; add sort -rn | head -n 1 to find the busiest hour.

Hint for question 6

Filter on the status field with awk ($9 >= 500), then extract the hour. Do not filter with a regex like /:14:/, which also matches minute 14 of every hour. substr($4, 14, 2) is the hour exactly. For the time window, print the time of the matching lines and look at the first and last with sed -n '1p;$p'.

Hint for question 7

$9 == 404 selects the not-found responses. Once you know the top IP, filter on $1 == "that.ip" and look at its paths and its hours.

Hint for question 8

Sum field 10 in awk. For bytes per path, use an associative array keyed by the path. split($7, parts, "?") puts the part before ? in parts[1]. 1 MiB is 1,048,576 bytes.

Hint for question 9

The user agent contains spaces, so whitespace fields do not work. Split on the double quote instead: awk -F'"' '{print $6}'. For the bot share, grep -c -E 'bot|python-requests|zgrab', or count in awk and divide by NR.

Hint for question 10

grep -o 'q=[^ ]*' extracts just the query parameter. Then remove q= with cut -d= -f2 and turn + into a space with tr.

Hint for the stretch question

Use two arrays keyed by hour: one counting all requests, one counting server errors. In END, loop over the first and printf both, plus 100 * errors / requests. Hours with no errors have an empty array entry, which awk treats as 0 in arithmetic. Pipe the result to sort, because for (h in array) has no defined order.

Bonus: real logs from your own machine

The same pipelines work on the logs your Mint system writes every day. They are not in nginx format, so you will need to adapt the fields. Some are readable only by members of the adm group; the first user created on Mint is a member, which you can check with id.

File Contents Readable by
/var/log/syslog General system messages adm group
/var/log/auth.log Logins, sudo, SSH adm group
/var/log/dpkg.log Package installs and upgrades Everyone
/var/log/apt/history.log apt commands you ran Everyone

On Mint 22, syslog and auth.log lines start with an ISO 8601 timestamp, then the hostname, then the program:

2026-09-21T10:15:02.123456+00:00 mint systemd[1]: Started session-4.scope - Session 4 of User alex.

Try these questions:

  1. Which 5 programs write the most lines to /var/log/syslog? (Strip the [pid] and trailing colon from field 3.)
  2. How many syslog lines arrived per hour? (The first 13 characters are the date and hour.)
  3. How many syslog lines per hour mention "error", in any case?
  4. Which commands were run with sudo most often, according to /var/log/auth.log? (Look for COMMAND=.)
  5. How many packages were installed, upgraded, and removed according to /var/log/dpkg.log? (The action is field 3.)

Your answers depend on your own machine, so there is no fixed solution, but the solution page shows working pipelines.

Where logs live on modern systems

On systemd-based systems like Mint, the primary log is the journal, read with journalctl. The text files in /var/log are written by a logging service alongside it. You will learn journalctl in systemd and journalctl, and its output pipes into grep, awk, and sort just like a file.

Solution

When you have answers for all ten questions, compare them with the full worked solution, which explains every pipeline stage by stage and points out the traps.