Humor - DC Migration

This happened in one of my earlier assignment. We use to maintain a white board just in front of our team. Anonymous people used to write some text, either about priority, failed component, modified target date reminder etc. etc. Already hosed of and tired of all these tiresome work, we hardly had any time to peek into those weird collection of texts. Seeing it flooding one of our cool dude team member wrote his comments on why things are moving slow.

Q. on the white board,
It is said that "God Created Earth in 6 days" , then WTF this moving so slow ?

- “Let God be true, and every man a liar.” - But we are not, he mumbled and wrote the mighty points as below,

1. Hope you well understand that , God probably had an unlimited budget which no boss had to approve.
2. It was also NOT needed to be SOX or ISO compliant.
3. I gues, at that time probably NO OSHA was there.
4. Did GOD implemented DR strategy?
5. I am sure he also did NOT care about Change Mgt.
6. He made it Tier-1, which we cant think of only.
7. Did he documented each step, huh!
8. He was NOT overcautious to meet urgency level.
9.God also probably did not think of ROI(Return of Investment) with a most robust and less prone to failures setup.
10. When we don't plan, it doesn't work; We planned - it did not work either, then why should we plan !

Hence we don't have a concrete plan atm. - so be patience and wait up


--Believe me the white board which saw weird peek sometime, suddenly saw so many enthusiastic flock around, and no one dared to wipe this bold(but meaningful) comments of his! which definitely carried a strong msg. to peoples around.

It was just amazing!!!


Cheers!
D E B U

Recover XEN Guest Root Password

Have you come across a situation when you need to reset the root password of a XEN virtual box.

Logon Host machine run following command to boot guest OS in single usermode and simply recover root user password as normally we do.

#xm create -c /etc/xen/guest_vm_name extra=1

Cheers!

Import Gmail Contact to my Outlook

I worked with a start up employer where it happened that we got an email from out Top Boss! to block all google and yahoo cousins, to promote their own IM probably to pass Beta phase by engaging professional chatters :-) . So guys who were via proxy were blocked! Now in this situation if I had only outlook as my mail client, how would I have imported all my gmail contacts....


Well , it came out to a very simple task, comprising a few clicks..

Login to gmail -> On your left hand side, click on contacts -> Export -> Outlook CSV format (for importing into Outlook or another application) -> Export -> Save the file somewhere in your lap/desk(top) -> Open outlook -> File Menu -> import and export -> (.) import from another program or file -> browse for the CSV you just saved -> Finish.

Done! All your gmail contacts are now part of your outlook contacts.

Cheers!

Tweeting From My Bash Shell

Recently i treid this from a bash command prompt to update my tweeter status -


And see i did it recently successfully..
-Follow me here @ http://twitter.com/debajitkataki

Now this tweet can be a all time favorite for some tweeter crazy system admin to inform you something by integrating this with a linux shell script!

An Example:

#!/bin/bash
load=`awk -F "." '{print $1}' /proc/loadavg`
report=report.txt
mailid="debajit.kataki@gmail.com,techops@<>.com"
if [ $load -ge 10]
then
echo "Warning:: Server load is above 10!! Please look into" >>$report
echo -e "....and current memusage is: \n" >>$report
cat /proc/meminfo | head -n 2 >>$report
mail -s "Warning:: SSO server Load" $mailid < $report elif [ $load -ge 16 ] then echo "Critical: Server load is above 16!! Please look into" >>$report
echo -e "....and current memusage is: \n" >>$report
cat /proc/meminfo | head -n 2 >>$report
mail -s "Critical: SSO Load" $mailid < $report
## Using Curl to update tweeter status
curl -u debajitkataki@gmail.com:bombasticpassword -d status="Dear debu's Follower, Can some one please inform debu to check his email, as one of his server is dieing badly... " http://twitter.com/statuses/update.xml > /dev/null

## End of tweeter update

fi

Am i exploiting tweeter here ;-) I guess Nope! Probably the right attitude and spirit of online collaboration..

Cheers!

DEBU

Log Retention Nightmare

In My previous post I was talking about managing apache web server logs centrally with MySQL, But throughout the post it was constantly rattling my mind thinking about how difficult is it to manage and to reach on a perfect log retention policy. Is it really a nightmare for IT managers?

Log retention in an application cluster or across the server layout is an effort that deal with the issue of maintaining information in your possession for a pre-determined length of time. Different types of logs require different lengths of retention. Retention policies usually describe the procedures for archiving the information, guidelines for destroying the information when the time limit has been exceeded and special mechanisms for handling the information when under litigation.

First, quickly sum up the bottom line on the 'what' and 'why'. Now while trying to understand the definition of "Why" It really comes down to two key points:

1) Legal requirements.
2) Business requirements.

Legal Requirements:

I am not a legal expert nor am I in sync with recent changes in IT laws. But i have come across situations when information regarding certain users last access information like from which IP he accessed the site last, time etc. are sought by various individuals or parties when they sense some sort of e-Crime through our site. To be more precise this happened in one of my recent assignment which happens to be one Social Networking site and sensitive information was quoted to top BU heads from various profiles and we ran out of breath to figure out who this guy is! Luckily we were good to capture them tight. Recently I read it somewhere that Governments in various countries are considering laws that will require communications service providers to capture and archive mobil/telephone and internet traffic data for periods from 6 months to 2 years. These laws have for the most part not yet been enacted.

Business Requirements:

Its tempting to sweep the whole pile into trash. is not it? So, should administrators simply delete everything as quickly as possible? I wish it were that simple. Day to day business activities often dictate the length of time information needs to remain accessible. In addition to legal requirements, individual businesses have their own data retention requirements that can range from contractual obligations with customers or suppliers to administrative or operational information such as policies and procedures that define daily functions. Each business must set their own data retention requirements to sufficiently maintain their business operations.

The crucial point is to identify all the log files that might contain information of interest. Secondly, for each set of logs which i identified as useful one how long I need to have them immediately available on-line. Keeping them online is what is killing me as because, once i decide to make them available off-line say in DVD backup or so, issue stands more of a administrative decision rather than a technical preparation. This can resolved with a meeting with various BU head or stakeholders.


The other aspect of log retention policy for business requirements is to use this information to improve the quality of your services and for other business purposes. For me this is very very important if you mission is continuous improvement od end user experience of your services. Data mining on logs holds the key to uncovering and cataloging the authoritative links, traversal patterns, and semantic structures that will bring intelligence and direction to our Web interactions. Huge volumes of access and usage information—provides a rich and unprecedented data mining sourceand key source of information for interaction designers. Identifying the trends of usage of my site, can really help me to work out on a more business focuss strategy. Hence with the presence of considerable amopunt of logs data mining activities can expand rapidly allowing firms to retrieve highly personalized data about customers, and the exact set if crowd.

Log retention is a complicated balancing act. one calls for a aggressive destruction of electronic data after a short time period. On the other hand is the philosophy that promotes the saving of everything indefinitely. There is no concrete answer when establishing a data retention policy. On one hand you need to save information required by law and your business. On the other hand, you should delete irrelevant, outdated and nonproductive data as quickly as possible. Also you need to plan ahead for potential discovery requests in connection with litigation cases.

Just to conclude what i feel is that - considering all legal aspect, with an intention to improve web experience to our end user, we need to do a detail 'benefit analysis' and publish a suitable retention policy for our logs, and immediately off lining those which dont serve any of these goals.

Does not this call for a real bold IT manager for this bravado act ? :-)


Cheers.
DEBU

Send Apache Logs to MySQL

In one of my previous employer where we had to deal with thousands of server serving web content, space issue was one of the major concern which was filled up very fast by the apache logs. Not only this as you all know Log retention policy definition is never clear to many organizations. I know IT managers dealt with document retention as a way to keep their boss' boss off their boss' back. Hence it was more of administrative concern rather than a technical issue. Anyway, we tried to deal with it with a Apache_MySQL simplee setup with the help of mod_log_sql/mod_log_mysql. we used to push apache logs to a centralized DB server(MySQL MyISAM master-slave) with a very simple config in apache.

The strategy was as below:

Decide on a MySQL DB schema like, based on your apache log files fields.
Decide if you need to additiocal apache log parameter. (We had to add %v, as we had to deal with multiple virtual host per server.
Decide of Database cluster architecture (We chose MySQL MyISAM master-slave)
Decide on how failover will work for database.
Construct DML SQL for data mining on logs to derive various needed stats. like

- How many 404 i have today
- is there any 5xx on my site today
-how much traffic was server by my apache today etc etc,

What we achieved:

This helped us concentrating space maintenance in one specific server when we had a web-cluster in place.
Even failover was easy to either switch to slave DB server or even quick SVN CO of a different apache configuration file which stores logs locally in case if DB cluster itself goes down
Also a quick report like http error code say 5XX etc. can be pull out with a select query as part of troubleshooting.
Seamless integration into the standard Apache logging configuration.
Logs data as it is: e.g. times as SQL DATETIME and n/a items as SQL NULL, with simple config.

How To:

1. Download mod_log_sql tarball -> Compile install
apxs -a -i -c -L/usr/mysql/lib -I/usr/mysql/include -lmysqlclient_r -lz mod_log_mysql.c
2. Edit httpd.conf

LoadModule log_mysql_module [path to your mod_log_mysql.so, usually "modules/mod_log_mysql.so"]

and Most importantly,

LogFormat "INSERT INTO sqlserver SET virtualhost=%v,
remoteip=%a,remoteport=%S,serverid=%{SERVERID}e,
connectionstatus=%X,remoteuser=%u,identuser=%l,
datetime=%t,requestmethod=%m, url=%R,protocol=%H,
statusbeforeredir=%s,statusafterredir=%>s,
processid=%{pid}P,threadid=%{tid}P,seconds=%T,
microseconds=%D,  bytesin=%I,bytesout=%O,
bytescontent=%B,useragent=%{User-Agent}i,
referer=%{Referer}i,uniqueid=%{UNIQUE_ID}e" sqlserver

CustomLog mysql:logwriter!password@ipaddress/apachelog\
sqlserver

DB server settings:

Create database , based on your log parameters or fields.
Grant privileges, user@apache web server ip address (here logwriter is my web mysql user)


Check with restarting apache and validate with failover too!

Cheers.!!!

Windows etc/hosts File Shortcut

When you have a web server cluster behind the LB, and you need to do a timely uni-cast test of each/any server by putting a host entry for some URL, i know you have to play with a lot of host entry / delete.
To quickly access the hosts file go Start -> Run -> Type drivers -> Ok.

This will bring up the folder before the hosts file, just double click into the etc directory from there - > right click on host file and open with your favorite text editor.

There in you put host entry. Syntax : <(host)URL>

e.g.

10.99.1.44 skipfast.blogspot.com

This etc/host file can be put in a short cut - which makes life easier if your opening and closing the same quite frequently -

Right click on the desktop -> Create Shortcut and put the below as target

%windir%\system32\notepad.exe "c:\WINDOWS\system32\drivers\etc\hosts"

Cheers!!! Save time, skip fast...

D E B U