I like cooking too!!

I never stayed in a hostel until i graduated, but time came when I had to relocate to North India(Delhi), during my PG. I was a bit foodie from my school days. Frankly speaking, I did not like North India Food after 2-3 months of my stay. I started feeling, that the dishes i tried until then, were a bit heavy compared to the place from where i belong, Assam(North East, India); where food is very light and refreshing. Hence after much of belt tightening, I decided to start cooking for myself. I started cooking those which is enough to give me mileage for the day until my college gets over. Population started growing in my small room, where I had another two friends joining in. Exam pressure forced us to hire a part time Bengali cook(lady). Her name was Anna. Same time around- Anna Kournikova also started hitting the front page, so she definitly got some edge with her name though!, and that's how we shortlisted her. Well this is not the topic here, so let me come back to point quickly. We discontinued her just after our exams and our experiments with the kitchen started all over again. Sometime for trying out new, sometime for survival.

Days passed(years actually,7 yrs now!!) and I don't know how much I have gleaned on my culinary skills. There was a further boost to my journey when I came to Bangalore and I met Pohar, Dip, Dr. Manas, Samiran(nahar), Shekhar, Dr. Samiran Phukan[skillfully at the tail :-), not accidently ]. Oh buoy! These guys are crazy fun loving kitchen lurkers. First they will make you run behind them, trying to guess when we can grab the dish - with tempting smells of their dishes. I saw them becoming so happy to see a group of family or friends tucking into something they have just prepared. My wife still doesn't know about this blog. So, let me feel free and safe to admit here that we were animals at that time. Crazy weekends, wine, dine, movie, biking all were full on.

So by this time you would have easily get convinced how natural it was for me to have that mighty subject line, when I am married. Folkes from this planet viz.venus actually likes that!!

Anyway with a tribute to the said subject line, let me pen down a quick dish with Brinjal. There is no sexy name we have coined for this dish. So feel free to add on, and let me know too. Its actually a quick Assamese recipe.

Brinjal With Scrambled Egg(For the Name sake!!)

1. Make some cut across the body of the Brinjal. Dont stab her. Put some mustard oil over the body, and kind of do a body massage :-) and make sure the mustard oil actually enters the cut marks. Now Roast the brinjal(s). Make sure it gets lil soften up.
2. De skin the brinjal and mesh it. Mesh it properly without leaving too many lump in it. Little bit is OK.
3. Cut onions in to fine slices
4. Make fine ginger juliennes
5. Crush garlic roughly, alternatively you can also get commercialy available Ginger Garlic paste.
6. Add turmeric powder and salt and mix it well.
7. Heat oil in pan and when its warm add cumin seeds and let it splutter.
8. Fry the onions and when it turns golden add the ginger juliennes and chopped garlic, chilli in to it. Fry till it changes colour.
9. Add the mashed brinjal in to it and fry for some time when its starts to leave oil add the cumin powder.
10. Beat the egg and add a pinch of salt.
11. Spread the fried stuff towards the margin of the pan so as to make a ring and oil gets accumulated in the middle.
12. Add the egg and mix thoroughly and cook for 2-3 mins.
13. Turn off the gas and garnish with fresh coriander leaves and ripe tomato pieces.
14. Enjoy with rice or roti .

A clean kitchen is the sign of a wasted life, so learn some cooking live with honor, turn sideways and always look in the mirror to know if you are overdoing it ;-)


Cheers!

DEBU

PS. Assamese recipes.

Free Online image Manipulation Tools

While trying to give a better look to my website, I always felt I need an easy to use tool, which suffice at least my part, which of course is NOT that complicated, which the existing tools make me feel - whenever i start to work with them. This is probably because -I did not feel at home in with them, since it takes time to master them! Hence I was always constantly looking for something available online, which can better serve my purpose with just a couple of click. And see below, how many of them them is actually available. Easy and that too Free!!!. These tools will enable you to add, manipulate and transform various images, make cartoon and funky looking images and many more. I

The dea here was to make all of them available in one place, so that some novice web thirsty, image crazy lurker saves some time.

RoundPic : This will help you to add rounded corners in you images.
BeFunky: It can help you to give a cartoon look to your images, basically a place of digital Artwork.
GifUP: Online GIF animation tool.
PicBite: A free online tool to host your images, resize them, crop, organize, comment and many more.
ImageReflector: Create customized image reflection.
FotoBabble: Create a Talking Photo
Photo-To-Sketch: Convert Photos To Sketches.
Text-Image: Create cool text images.
Image-Merger: Online Free image merger.
IconsPedia: Free icons.
IconArchive: Tons of icons.
FantasticFace and GrossOut Creating emotions/Fun in a Photo.
InstantMask: Remove backgrounds
Exif-TagRemover: Remove meta-data from your image.



Happy Exploring. Yeah! do come and share below, if you already know some more....

Cheers!
-DEBU

Need For RCA

By definition - Root Cause Analysis(RCA) is the fundamental breakdown or failure of a process which, when resolved, helps us understand why the problem occurred in the first place and prevents a recurrence of the problem time and again.

The whole purpose of Root Cause Analysis or identification is to identify the origin of a problem. It uses a specific set of steps, with associated tools, to find the primary cause of the problem, so that you can determine:

1. What actually happened.
2. How it happened.
3. Why it happened.

RCA assumes that systems and events are interrelated. An action in one area triggers an action in another, and another, and so on. By tracing back these actions, you can discover where the problem started and how it grew into the symptom you're now facing.

Initially RCA happens to be a reactive method of problem detection and solving.- a post analysis This means that the analysis is done after an event has occurred. By gaining expertise in RCA it becomes a pro-active method. This means that RCA is able to forecast the possibility of an event even before it could occur.

Being in Data center operations we too come across repetitive and irritating problems quite often. It is very important to get into the roots. RCA comes in as an ally in such situations. I can understand its 'annoying' when your manager asks for it. But believe me, its equally enjoyable to reveal the root cause, as someone rightly said, Only the inquiring mind solves problems".

In order to deliver high levels of IT infrastructure availability, organizations need tools that help them isolate repetitive problems. If you belong to a team, where alert provisioning is very tight, you might also land up in a situation where multiple alarms fired at the same time. This is the point of root cause analysis -- to dig below the symptoms and find the fundamental, underlying decisions and contradictions that led to the undesired consequences. If you want your problems to go away, your best option is to kill them at the root.YTo identify the root cause, we have to ask “Why?” over and over, until we reach there. ou need to trace back the events in a systematic way by looking at the effects and the causes that created or contributed to those effects. 'Fishbone' diagram may be quite handy at this kind of situation and isolate the issue.

At the end of your analysis, the finding must be willing to probe the data first to determine what happened during the occurrence,second to describe how it happened, and third to understand why.

Once identified the root cause, need to determine to 'Resolve(actionable)' or 'Not To Resolve(Non-actionable)' This is even more crucial if the cost of resolving the same is higher which forces us to consider it as a symptom. Its a very difficult scenario as the cost of the symptom is generally wrapped up in some number of customers satisfaction in addition to the resource costs associated with it. BUT, If cost involved is very minimal, with appropriate failover/backup or downtime this needs to be addressed immediately. And, if its identified as a deeply rooted cause, with higher cost of resolution better to tag it as a known symptom.

Many organizations document a set of procedures to follow on how to tackle this problem if it reoccurs further. Of course, this is now being tagged as a 'known issue', and a considerable amount of time will be saved while addressing the same. What we achieved here is at least a quicker resolution, even though the root of the cause was NOT being removed at all.

As Someone rightly said "Customers don’t expect you to be perfect. They do expect you to fix things when they go wrong.” So be equipped with your tool set, for a quicker resolution, by engaging in a continuous RCA hunt -may be "tomorrow" becomes predictable!, as someone rightly said "It's what you learn after you know it all that counts".

So, dig it big !! AND Don't Skip! :-)

Cheers!
DEBU

Rate Limiting by mod_limitipconn

If you have a relatively busy web site, with lot of juicy information spiders are good fast enough to crawl through your site which sometimes become annoying. Currently there is none in place to stop them on the fly, unless you explicitly looks at the log files and do something.

That’s pretty fair until and unless, you are crawled to death. Hence came the Rate Limiting concept via some means whether tweaking your iptables rule or some other means. We are going discuss mod_limitipconn and how to implement it in order to limit the number of connections per IP on our server. This can be a very useful tool, as it could help in lowering the load on your server due to someone connecting too many times from the same IP.

Cd /usr/local/src/

wget http://dominia.org/djao/limit/mod_limitipconn-0.23.tar.bz2

tar -xjvf mod_limitipconn-0.23.tar.bz2

cd mod_limitipconn-0.23

/usr/local/apache/bin/apxs -i -a -c mod_limitipconn.c

Edit httpd.conf

LoadModule limitipconn_module /usr/local/apache2/modules/mod_limitipconn.so

When this file is still open, lets add in the following lines to the bottom of the file:

ExtendedStatus On

< IfModule mod_limitipconn.c >

MaxConnPerIP 10

NoIPLimit image/*

NoIPLimit image*/*

< /IfModule >

Note: exempting images from the connection limit as your web page might have lots of inline images

Finally Restart Apache.

It is highly recommended that you go through the README file that came with the source, and then keeping a sharp eye on what your webserver does, to see what you need to tweak in the defaults.

Please Note that, This module will not function unless mod_status is loaded and the “ExtendedStatus On” directive is set. Connections in excess of the limit result in a stock 503 Service temporary unavailable response, which can be make more meaningful.

I had sucessfully implemented this with my Prefork MPM apache cluster but i heard people saying that it has some issue while implementing along with mod_cache, do pitch in with your experince on this so that we can discuss further.

The Only disadvantage i have seen with mod_limitconn is that the limits defined by mod_limitipconn.c apply to all IP addresses connecting to your Apache server. Currently there is no way to set different limits for different IP addresses.

Cheers!
DEBU

Rate Limiting by mod_evasive

When there is high traffic on your apache web server, chances are that - are that sooner or later it might come under a DoS attack. So as per my promise in my previous post here is one of the must install tools, implementation steps.

mod_evasive is an Apache module specifically designed to deal with this. From the author’s site:

"mod_evasive is an evasive maneuvers module for Apache to provide evasive action in the event of an HTTP DoS or DDoS attack or brute force attack. It is also designed to be a detection and network management tool, and can be easily configured to talk to ipchains, firewalls, routers, and etcetera. mod_evasive presently reports abuses via email and syslog facilities."

So this is how you go about installing mod_evasive in a linux Box ...

1. cd /usr/local/src

wget http://www.zdziarski.com/blog/wp-content/uploads/2010/02/mod_evasive_1.10.1.tar.gz

3. tar -zxvf mod_evasive_1.10.1.tar.gz
4. cd mod_evasive
5. /usr/local/apache2/bin/apxs -cia mod_evasive20.c
6. Edit httpd.conf file and add -

LoadModule evasive20_module /usr/local/apache2/modules/mod_evasive20.so

DOSHashTableSize 3097
DOSPageCount 2
DOSSiteCount 50
DOSPageInterval 1
DOSSiteInterval 1
DOSBlockingPeriod 120

It is highly recommended that you go through the README file that came with the source, and then keeping a sharp eye on what your webserver does, to see if you need to tweak any defaults. I’d also suggest adding the email alerting option inside the IfModule configuration:

DOSEmailNotify debajitkataki@gmail.com

Explanation Of Various Parameters:

DOSHashTableSize

The hash table size defines the number of top-level nodes for each child’s
hash table. Increasing this number will provide faster performance by
decreasing the number of iterations required to get to the record, but
consume more memory for table space. You should increase this if you have
a busy web server. The value you specify will automatically be tiered up to
the next prime number in the primes list (see mod_evasive.c for a list
of primes used).

DOSPageCount

This is the threshhold for the number of requests for the same page (or URI)
per page interval. Once the threshhold for that interval has been exceeded,
the IP address of the client will be added to the blocking list.

DOSSiteCount

This is the threshhold for the total number of requests for any object by
the same client on the same listener per site interval. Once the threshhold
for that interval has been exceeded, the IP address of the client will be added
to the blocking list.

DOSPageInterval

The interval for the page count threshhold; defaults to 1 second intervals.

DOSSiteInterval

The interval for the site count threshhold; defaults to 1 second intervals.

DOSBlockingPeriod

The blocking period is the amount of time (in seconds) that a client will be
blocked for if they are added to the blocking list. During this time, all
subsequent requests from the client will result in a 403 (Forbidden) and
the timer being reset (e.g. another 10 seconds). Since the timer is reset
for every subsequent request, it is not necessary to have a long blocking
period; in the event of a DoS attack, this timer will keep getting reset.

DOSEmailNotify

If this value is set, an email will be sent to the address specified
whenever an IP address becomes blacklisted. A locking mechanism using /tmp
prevents continuous emails from being sent.

NOTE: Be sure MAILER is set correctly in mod_evasive.c
(or mod_evasive20.c). The default is “/bin/mail -t %s” where %s is
used to denote the destination email address set in the configuration.
If you are running on linux or some other operating system with a
different type of mailer, you’ll need to change this.

DOSSystemCommand

If this value is set, the system command specified will be executed
whenever an IP address becomes blacklisted. This is designed to enable
system calls to ip filter or other tools. A locking mechanism using /tmp
prevents continuous system calls. Use %s to denote the IP address of the
blacklisted IP.

DOSLogDir

You can Choose an alternative temporary directory. By default “/tmp” will be used for locking mechanism.

HOW TO WHITE-LIST THE IP ADDRESSES

IP addresses of trusted clients can be whitelisted to insure they are never
denied. The purpose of whitelisting is to protect software, scripts, local
searchbots, or other automated tools from being denied for requesting large
amounts of data from the server. Whitelisting should *not* be used to add
customer lists or anything of the sort, as this will open the server to abuse.
This module is very difficult to trigger without performing some type of
malicious attack, and for that reason it is more appropriate to allow the
module to decide on its own whether or not an individual customer should be
blocked.

To whitelist an address (or range) add an entry to the Apache configuration
in the following fashion:

DOSWhitelist
DOSWhitelist 127.0.0.*

Wildcards can be used on up to the last 3 octets if necessary. Multiple
DOSWhitelist commands may be used in the configuration.

You will notice that I have a couple of different settings in there than the default. For ex. if you want to ban the IP for 1 hour, you need to tweak the value to 3600 seconds.

That’s it! Time To Catch Some Fish Guys ;-)

Apache Anti-DoS Preparation

I am NOT going to discuss here the definitions and concept. I wanted to put it down, based in what i faced before on the Subject line and how I tackled. I came up with some predictable symptoms which will definitely help some one to get some hint and how to manage this crisis.

I always believed that along with the networks and server admins, - application developer should accommodate a module based on security aspects(necessary but NOT essential), so that the prevention preparation becomes a bit more collaborative. But while trying to meet the application/product deadline line, specially in start-up kinda company, somehow this gets overlooked.

Apache being the Leading web server, its vulnerability has always been worked upon continuously. So there should not be any scope left from our side too, to fine tune this guy so that it is strong enough to combat DoS/DDoS kind of attack.

If you are a web server administrator and see the following symptoms, i would advice you to be more detail about your log analysis:

Symptoms

1. if the sites are suddenly serving slow, despite any new business marketing boost etc.
2. Lots of hanging processes, http://(Your-Apache-Admin-URL)/server-status. Read More.
   
3. You see same set of ip's requesting more than needed -
   netstat -punta | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n.
4. Server resource spike, with top command you see soaring high resource utilization.
5. You are left with very less no. of connection to serve the real traffic, with some intermittent 5xx error.

And Finally You Are Left With,

1. A hang server.
2. Web site down/ intermittent service delivery .
3. Gradually SSH also stops responding. (Lucky if you had already logged in!)
4. Need for a mandatory reboot in order to restore access to the system.

Prevention Preparation:

1. You should have a solid understanding of what is being served from your server, how much average rss memory consumtion per child, how much memory this server has, and based on that only you calculate the MPM settings. For ex.
   #ps -ylC httpd --sort:rss | awk '{ sum = sum + $8 } END { print sum/NR }' ; doing so gave me 28422.2 = approx 28 MB , Hence my MPM settings should be in proportionate to the available share of RAM for this apache web server.
   

   Thus the appropriate setting for MaxSpareServers would help apache not to over consume and ultimately ending up onsuming ethe ntire available memory resource causing a systems OOM error and hang up permanently.

2. Enable SYN COOKIES at the kernel level
   echo 1 > /proc/sys/net/ipv4/tcp_syncookies,
   along with this, lets also do a
   echo 1 > /proc/sys/net/ipv4/tcp_keepalive_probes
   echo 2 > /proc/sys/net/ipv4/tcp_synack_retries
   echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
   
   
3. Enable and Configure iptables to prevent the attack or at least work to identify the attack(this is regardless of apache specific, rather system wide)
   /sbin/iptables -N syn-fld
   /sbin/iptables -A syn-fld -m limit --limit 100/second --limit-burst 150 -j RETURN
   /sbin/iptables -A syn-fld -j LOG --log-prefix "SYN fld: "
   /sbin/iptables -A syn-fld -j DROP
4. Two must deploy tool,
   

   mod_ipconnlimit and mod_evasive module.

5. A fair knowledge of sed/awk/perl and regex shell tricks to do a quick analysis of your access logs.

- and the moment you identify some one(or even slightest suspect) drop all his request(across entire web cluster),

/sbin/iptables -A INPUT -p tcp -s --dport 80 -j REJECT ,
and later on, needless to say, don't forget to white list the same if it is NOT the culprit.

iptables -L INPUT --line-numbers
iptables -D INPUT

Handy Operations Tips:

Tips 1: If your server is behind LB, know if it is stateless or Keep-Alive. if it is a Keep-Alive one, you need to immediately Shutdown the server from LB; this will help stop further traffic to the server. If it is a stateless server , of course you are free to take this guy down and further do the fixing work. And please be quick to do that.

Tips 2: if badly affected, you might also want to increase the MaxClients Limit, to sustain. But its is NOT possible On the fly, since changing of MPM settings needs a apache restart. You need plan this put as a quick crisis Mgt.


I will write more on MPM settings for better performance in coming days. , and steps for setting up mod_evasive and mod_ipconnlimit soon..

PS: I would be more than happy to get the opportunity to discuss your challenges, it will help us to understand various other aspects too.


Thanks & Cheers!
DEBU

Humor - DC Migration

This happened in one of my earlier assignment. We use to maintain a white board just in front of our team. Anonymous people used to write some text, either about priority, failed component, modified target date reminder etc. etc. Already hosed of and tired of all these tiresome work, we hardly had any time to peek into those weird collection of texts. Seeing it flooding one of our cool dude team member wrote his comments on why things are moving slow.

Q. on the white board,
It is said that "God Created Earth in 6 days" , then WTF this moving so slow ?

- “Let God be true, and every man a liar.” - But we are not, he mumbled and wrote the mighty points as below,

1. Hope you well understand that , God probably had an unlimited budget which no boss had to approve.
2. It was also NOT needed to be SOX or ISO compliant.
3. I gues, at that time probably NO OSHA was there.
4. Did GOD implemented DR strategy?
5. I am sure he also did NOT care about Change Mgt.
6. He made it Tier-1, which we cant think of only.
7. Did he documented each step, huh!
8. He was NOT overcautious to meet urgency level.
9.God also probably did not think of ROI(Return of Investment) with a most robust and less prone to failures setup.
10. When we don't plan, it doesn't work; We planned - it did not work either, then why should we plan !

Hence we don't have a concrete plan atm. - so be patience and wait up


--Believe me the white board which saw weird peek sometime, suddenly saw so many enthusiastic flock around, and no one dared to wipe this bold(but meaningful) comments of his! which definitely carried a strong msg. to peoples around.

It was just amazing!!!


Cheers!
D E B U