Tuesday, August 23, 2011

Our new little angel..

We've added another color to our family's rainbow... we have named her 'Srishti' . She may be small when it comes to size, but she's the queen of the cutie pies! And very naughty alrerady... :-)

"A Precious one,
So small,
and So cute..

Dancing in
her angel feet
Straight from Heaven's
brightest star..

What a miracle
you are! "



- Delighted Parents
Debajit & Nandita

Saturday, August 6, 2011

Recover deleted /var/log/messages

I had to handle a recent escalation where /var reportedly got filled up - Someone with 'dbuser' userid deleted /var/log/message file on this server

[root@vapdb01 ~]# last

dkataki pts/0 10.152.208.19 Sat Aug 6 20:46 still logged in

dbuser pts/0 10.152.208.19 Sat Aug 6 20:14 - 20:22 (00:08)

dbuser pts/0 10.152.208.19 Fri Aug 5 04:17 - 10:08 (05:51)

This is how I would tackle this situation

[root@vapdb01 ]# df -h | grep /var

/dev/xvda3 3.0G 2.5G 282M 91% /var

[root@vapdb01 ]# cd /var

[root@vapdb01 var]# du -csh .

347M .

347M total

Looks like some file got deleted but some pid is still holding this file --

[root@vapdb01 var]# lsof | grep -i deleted | grep "/var/"

syslogd 1404 root 2w REG 202,3 2230328164 503883 /var/log/messages (deleted)

gpnpd.bin 4948 oracle 17uw REG 202,3 0 31522 /var/tmp/.oracle/ora_gipc_GPNPD_vapdb01_lock (deleted)

ocssd.bin 5013 oracle 17uw REG 202,3 0

That means someone tried to clear /var partition and deleted unknowingly or accidentally a crucial file from the server - and size of the file was - 2.1G+ , good catch but wrong measure by the user i would say!!!

Anyway - time to avert this situation now -

[root@vapdb01 var]# cat /proc/1404/fd/7 > /root/messages.1

[root@vapdb01 var]# /etc/init.d/syslog restart

[root@vapdb01 var]# cd /root/

[root@vapdb01 var]# gzip messages.1

[root@vapdb01 ~]# mv messages.1.gz /var/log/

[root@vapdb01 ~]# df -h

Filesystem Size Used Avail Use% Mounted on

/dev/xvda2 14G 4.5G 8.2G 36% /

/dev/xvda3 3.0G 434M 2.4G 16% /var

..and a happy Ending -

This only can someone do who has little knowledge on how UNIX works. Shame!! Is not that why we SA folks are very careful while giving root privilege to normal users?

-DK

RCA - Root Cause Analysis

An important step in finding the root causes of issues or occurrences that happen within a system or organization is root cause analysis (RC...