Skip to content
Expert LinuxLesson 27 of 27 4 min read

Performance Troubleshooting

Reading load average, top, and vmstat to find whether CPU, memory, or I/O is the bottleneck.

Skip to the challenge
On this page8 sections

"The server is slow" is not a diagnosis - it's a starting point. This lesson is a systematic way to turn that complaint into an actual bottleneck: CPU, memory, or I/O, in that order of how fast you can usually check them.

Load average: the first number to check

uptime and top both report load average - three numbers, averaged over the last 1, 5, and 15 minutes:

bash
uptime

14:32:05 up 3 days, load average: 4.21, 3.87, 2.10

A load average is the average number of processes wanting CPU time - either running or waiting for their turn. The number alone means nothing without context: it needs to be read relative to CPU count. Check that first:

bash
nproc

A load average of 4.21 on an 8-core box (nproc → 8) means the machine is roughly half-utilized - fine. That same 4.21 on a 2-core box means work is queuing up, over double what the machine can run at once - a real problem. The rising trend across the three windows (4.21, 3.87, 2.10) also matters: climbing means it's getting worse right now; falling means whatever caused it is already easing off.

top: who's actually using it

bash
top

%Cpu(s):  2.3 us,  1.0 sy,  0.0 ni, 96.5 id
MiB Mem :   7898.4 total,   5343.0 free,   1257.4 used

  PID USER   %CPU  %MEM    VIRT    RES  COMMAND
 4821 www    89.2   12.4  820000 380000  nginx
  • %CPU / %MEM per process - the columns you'll look at first to find a runaway process.
  • VIRT - total virtual memory a process has mapped, including memory it hasn't actually touched yet. Usually much bigger than reality, and rarely the number that matters.
  • RES - resident memory: what the process is actually using in RAM right now. This is the number worth watching for a real leak.

free: memory, read correctly

bash
free -h

              total    used    free   buff/cache   available
Mem:          7.7Gi    1.2Gi   5.1Gi      1.4Gi        6.3Gi

The trap almost everyone falls into once: free doesn't mean problem. Linux deliberately uses spare RAM to cache recently-read files (buff/cache), because unused RAM is wasted RAM - that cache is released instantly the moment something else needs it. The number that actually matters is available, not free: it already accounts for how much of that cache could be reclaimed on demand.

vmstat and iostat: is it CPU, memory, or disk

bash
vmstat 1
iostat

vmstat's columns split the story cleanly:

  • r - processes actively waiting for CPU (high and climbing → CPU-bound).
  • si/so - swap in/out (anything above zero, sustained, means you've run out of real RAM).
  • wa (in the cpu section) - percent of time CPU sat idle waiting on disk I/O (high → I/O-bound, not CPU-bound).

iostat breaks disk activity down further, per device - useful for confirming which disk is the one saturated.

The USE method

A compact framework, useful for any resource (CPU, memory, disk, network) when you're not sure where to start:

  • Utilization - how busy is it? (percent time doing work)
  • Saturation - how much work is queued, waiting for it? (r column, load average)
  • Errors - is it failing outright, separate from being merely busy?

Checked in that order, for each resource in turn, it very quickly narrows "slow" down to one specific, provable bottleneck instead of a guess.

When it's not a resource at all: strace

Sometimes nothing is short on CPU, memory, or I/O, and a process is just stuck - blocked on a lock, a slow DNS lookup, a hung network call. strace attaches to a running process and prints every system call it makes in real time, which is usually the fastest way to see exactly what it's waiting on.

Try it

Run these in the practice terminal (opens in a new tab). It opens in a new tab, so you keep your place here.
bash
uptime
nproc
top
free -h
vmstat
iostat
strace -p 1234

Further reading

  • "Systems Performance" by Brendan Gregg (Addison-Wesley) - the definitive book on this entire topic, written by the engineer who popularized the USE method.
  • "How Linux Works" by Brian Ward (No Starch Press) - explains what the kernel is actually tracking behind top, free, and vmstat's numbers.

Challenge

Get a live process snapshot, then check memory and system load in human-readable form. Chain all three in one line with &&.

Want to experiment first? Try things in the practice terminal (opens in a new tab), then enter the command that solves the task here.

Quiz

Lock in the concepts. Score at least 80% to mark the lesson complete.

0 of 3 answered

Done with this one?

Skipped the challenge? You can mark the lesson complete by hand.

Your notes

Kept on this browser only.