What’s filling up root? | /dev/null 2>$1

I had a right noodle scratcher today when trying to find the source of the / filesystem filling up on an AIX 6.1 system.

I did all the normal things, like

du -kx | sort -n

find / -xdev -ls

Even went as far as some pretty complex find commands:
To find file names only:

find / ! -name / -prune ! -type l |grep -vwE $(mount|tail +3|awk ‘{if ( /^[a-zA-Z]/ ) {print $3} else {print $2}}’|grep -vE “^/.*/|^/$”|xargs|tr ‘ ‘ ‘|’)

To show all the detail:

find / ! -name / -prune ! -type l -ls|grep -vwE $(mount|tail +3|awk ‘{if ( /^[a-zA-Z]/ ) {print $3} else {print $2}}’|grep -vE “^/.*/|^/$”|xargs|tr ‘ ‘ ‘|’)

It is also useful when you try to check what is filling up root(/) filesystem:

du -sk $(find / ! -name / -prune ! -type l |grep -vwE $(mount|tail +3|awk ‘{if ( /^[a-zA-Z]/ ) {print $3} else {print $2}}’|grep -vE “^/.*/|^/$”|xargs|tr ‘ ‘ ‘|’))|sort -k1rn

Only when I went back to the beginning did I spot the issue:
find / -xdev -ls |grep “Nov” listed the files that had recently written to /. Only then did I notice a 168M file:
# find / -xdev -ls |grep “Nov 2”
11 8 drwxrwxr-x 5 root system 8192 Nov 27 12:23 /dev
4174 4 drwxrwx— 2 root system 4096 Nov 27 12:41 /dev/.SRC-unix
261 0 crw——- 1 root system 10, 0 Nov 27 12:23 /dev/__vg10
265 0 crw–w–w- 1 root system 6, 0 Nov 22 15:56 /dev/error
272 0 brw-rw—- 1 root system 10, 4 Nov 27 12:23 /dev/hd4
273 0 brw-rw—- 1 root system 10, 1 Nov 27 12:23 /dev/hd5
281 0 crw-rw-rw- 1 root system 2, 2 Nov 27 12:43 /dev/null
844 172764 -rw-r–r– 1 root system 176910336 Nov 22 15:55 /dev/null 2>&1

Turns out this is a bug with one of the IBM agents (cas agaent, I think).

Solution

You can just delete this file. There is a bug fix that will address the issue in the next ML. Until then, this file will re-appear, so a cron job might be a good measure.

 

Additional commands:

If /var is the issue

 # df -g /var
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/bos_hd9var      0.50      0.12   76%     8452    22% /var

# cd /var

find . -xdev -type f -name *core* -exec file {} \; |grep “AIX core file fulldump” | awk -F : ‘{print $1}’ | xargs rm

# du -smx * | sort -nr | head -5
436.13  was
302.21  opt
42.05   teamquest
18.94   log
17.38   adm

# du -smx was/* | sort -nr | head -5
418.46  profiles
13.07   jdbc
4.60    scripts
0.00    lost+found
0.00    heapdump

# du -smx opt/* | sort -nr | head -5
302.21  tivoli
0.00    freeware
0.00    csm

# find . -type f  -xdev | xargs ls -l | sort -rnk5,5 | head -10

TC