Saturday, August 6, 2011

Recover deleted /var/log/messages

I had to handle a recent escalation where /var reportedly got filled up - Someone with 'dbuser' userid deleted /var/log/message file on this server

[root@vapdb01 ~]# last

dkataki pts/0 10.152.208.19 Sat Aug 6 20:46 still logged in

dbuser pts/0 10.152.208.19 Sat Aug 6 20:14 - 20:22 (00:08)

dbuser pts/0 10.152.208.19 Fri Aug 5 04:17 - 10:08 (05:51)

This is how I would tackle this situation

[root@vapdb01 ]# df -h | grep /var

/dev/xvda3 3.0G 2.5G 282M 91% /var

[root@vapdb01 ]# cd /var

[root@vapdb01 var]# du -csh .

347M .

347M total

Looks like some file got deleted but some pid is still holding this file --

[root@vapdb01 var]# lsof | grep -i deleted | grep "/var/"

syslogd 1404 root 2w REG 202,3 2230328164 503883 /var/log/messages (deleted)

gpnpd.bin 4948 oracle 17uw REG 202,3 0 31522 /var/tmp/.oracle/ora_gipc_GPNPD_vapdb01_lock (deleted)

ocssd.bin 5013 oracle 17uw REG 202,3 0

That means someone tried to clear /var partition and deleted unknowingly or accidentally a crucial file from the server - and size of the file was - 2.1G+ , good catch but wrong measure by the user i would say!!!

Anyway - time to avert this situation now -

[root@vapdb01 var]# cat /proc/1404/fd/7 > /root/messages.1

[root@vapdb01 var]# /etc/init.d/syslog restart

[root@vapdb01 var]# cd /root/

[root@vapdb01 var]# gzip messages.1

[root@vapdb01 ~]# mv messages.1.gz /var/log/

[root@vapdb01 ~]# df -h

Filesystem Size Used Avail Use% Mounted on

/dev/xvda2 14G 4.5G 8.2G 36% /

/dev/xvda3 3.0G 434M 2.4G 16% /var

..and a happy Ending -

This only can someone do who has little knowledge on how UNIX works. Shame!! Is not that why we SA folks are very careful while giving root privilege to normal users?

-DK

1 comment:

RCA - Root Cause Analysis

An important step in finding the root causes of issues or occurrences that happen within a system or organization is root cause analysis (RC...