What's swapping to disk in linux / OSX

Recently I’ve ran into an instance where several servers where killing services for out of memory yet nothing would show up in our APM alerts that would suggest overt ram usage.

Popping on the box found that things where swapping out to disk, a lot of it to disk.

here’s the following set of commands that one can use to find out the worst offenders.

Lists all pids that have swap usage - use head(1) to get the last ten or so entries for the TOP 10 Offenders:

find /proc -name status -maxdepth 2 -exec awk '/VmSwap/ {split(FILENAME,pid,"/"); print pid[3]":\t"$2,$3}' {} \; | sort -k2 -n -r

Follow the above command with the process id (left column) to discover the program that’s swapping the most to disk

ps -o uid,time,cmd -f ...

Example


$ free -h
              total        used        free      shared  buff/cache   available
Mem:            31G         15G        2.6G        295M         13G         10G
Swap:          2.0G        2.0G         43M

$ find /proc -name status -maxdepth 2 -exec awk '/VmSwap/ {split(FILENAME,pid,"/"); print pid[3]":\t"$2,$3}' {} \; 2>/dev/null | sort -k2 -n | tail -n 4
118236: 41872 kB
114673: 63716 kB
936:    74152 kB
123207: 79872 kB

$ ps -o pid,time,cmd -f  123207
   PID     TIME CMD
123207 03:03:26 dotnet

From here is up to you but `strace(1)` and `lsof(1)` are good ones to look into before hitting it with `kill(1)`.