Friday, April 25, 2014

Ops : Opposite of Luck?



Running effective and efficient Operations is definitely NOT an easy task!! And I do keep on saying that -  “Ops is against Luck!”

Lets consider the below scenarios: 

  • No one is  complaining and  you also don’t have a readily available report on how healthy your systems are , you actually might be in  trouble..
  • Or customers may be too angry , busy or confused to bring up issues - you should not count these days as your lucky ones!!
  • 'No  news' is NOT a good news !! 
  • Assuming  unresolved issues won’t reoccur again..  Also the strategy of duck and cover in the hope that the problem will disappear is a recipe for catastrophe

Someone rightly said - luck is just a thin wire between survival and the worse!!

You may consider them as Luck , while may be so - diligence probably  is what we need along with our willingness to act for consistent efficiency.

But How..?  below may be couple of real life scenarios:

If you own production - negotiate your own production readiness and excellence with ‘Why’, ‘When’ and ‘What’.  It might be overly cautious, BUT we should apply a lens to understand whether the said change ‘now’ will bring real benefit or it can wait for a favorable time. 

Does your PD consider operations as their customers? If NOT speak up.  Set expectations correctly so that you don’t have to struggle to setup monitors and metrics overnight to support production from early next morning.

Also - we should know when to pull the brake on projects , and prepare for actual traffic and earn $

Relying on your firefighting superhero and showering praise on their heroics should be handled with a diff mindset of permanent resolution by getting into the root.

Yep!   Sin kills….!! and Luck is always to blame :-)

Wednesday, April 9, 2014

Why KILL -9 httpd is NOT always a good idea to invoke SSC




Sharing from my personal experience:

One fine day I myself had fallen in to this trap.

Being ‘n-4’  (LTM, BIG IP F5 iRule) is the condition to trigger SSC (static site contingency) on the LTM layer, I could kill apache on most of the web servers but last 7 or 8 ( I use windows laptop so mPuTTY is my guy to trigger them in parallel)  and guess what! all load had eventually had fallen into those few.

I lost control , lost time as well  I was having a superman attitude and never invited  Network (NOC) to do so by logging in to BIGIP and disable the pool members, so whole time was a waste, and finally I had to invite them as a rescue.

LTM was on and off Intermittent SSC and sometimes good but mostly bad Ux.


We can debate - that high load on servers will automatically trigger a SSC, but that will be an worse effect (not an action!) and uncertain scenario where we won’t be  playing any role and we are talking about a human being putting a certain and timely contingency..

Hence I recommend this to go at the LTM layer by someone deliberately …  ALWAYS!

Thanks/-

Friday, March 28, 2014

Curiosity for Subsistence



Most of us hold ourselves back when we should be reaching out and most of the time we never see a need to change. Every such picture tells a story , that - we have somehow landed in a comfort zone and not giving that extra. Extra effort,  extra hour, extra question , or that extra feedback..

This 'extra' thing followed us from our school life itself. Remember ? Yeah ECA :-) (extra curricular activity !! ). These activities helped us to learn the values of teamwork, individual and group responsibility, physical strength and endurance, competition, diversity, and a sense of culture and community. Extracurricular activities provided us a channel for reinforcing the lessons learned in the classroom. Debating was one of them, which found very less participants (at least in my school) - a very few students opted or was leading in this area. Personally I never liked those guys! I felt they got into a habit of 'extra' questioning and was very irritating. Well, that was my own personal perception. 

Now when I look back at those days I believe this  probably helped those kids to inculcate that spirit of questioning at things. Either in open forum or silently under their mind and the leading cause was 'knowledge' - which is what probably they gleaned. Class teacher always used to encourage saying - Q's are Q's, there is not right or wrong Q.

I follow Buddhism as a philosophy where it tells one basic thing - that "everything in the universe embodies the law of cause and effect".  So if "Questioning OR curiosity" is the cause - the effect is "knowledge".

In school,  my sitting choice was the second row bench (even though I make gangs now also proclaiming that I was also always a 'last bench-er' ;-)). Let me admit, I used to hesitate to ask Q's. Fear was - what the other will think about it !! But I was always self-aware about my own personal growth and  used to cover it somehow. But that reverse model took hell lot of time from me, and I did understood it very profoundly that - curiosity is the leading cause of knowledge and that extra curricular activity was part of a well-rounded education.

Today when I look at our youngsters coming and joining us as a freshers or intermediate in the corporates, less or more I do see that tendency there as well.. where they DO  NOT  show enough curiosity to learn that 'extra' thing or question much and tend to settle down. This eventually cost us very heavy and that wrong pick demands lot of your bandwidth and becomes an overhead. This probably because we hire for skills and ignore attitude? .. right hire is always a mystery to me though!


Anyway - lets be curious , admit (an eager confession!) , learn , assess and learn - otherwise the real pressure that keeps us on the wrong path will be our self-generated one!  I am sure this will definitely help us to be creative in our word, action and deed :-)  !!


I am not sure how many 'W's or 'H' can satiate your curiosity while putting your question across, BUT just remember -  curiosity (not that rover!)  can KILL the cat , however we probably just wanted to experience how curiosity freed that cat! :-) 



Tuesday, November 26, 2013

RPO and RTO




RTO = Recovery Time Objective tells how quickly you  will able to resume the normal operations back by connecting users to the data. This measurement is called RTO.

RPO =  Recovery Point Objective describes the point until which data will be recovered when a disaster strikes. Generally the time between last backup and the event.  It tells us how fresh or stale the data is.



These terminologies mostly are pertain to DR (Disaster Recovery) or backup solution. These measurements are mainly based on two things:

  1. The expected loss to the business with the objective. 
  2. The cost of achieving the objective.


Friday, November 1, 2013

Maintanace Calander for IT Operations




Problem Statement:      Multiple maintenance(s)  gives us tough time towards our  availability ,  better manageability and eventually improper resource alignment. 


Trying to solve:    Better communication, better visibility.   With visibility rest will follow.

What we want to Solve:
  1. Because emails aren't just letters anymore, they're tasks.. Specially in Operations.
  2. YOU might have been in the original email BUT got missed in the calendar invite by maintenance driver
  3. We need a ‘one stop’ view to see all the upcoming maintenances.
  4. Easy to analyze (dis)agree/ postpone  these maintenances proposed by multiple team with this single view.
  5. From top of Pyramid view - path from scarcity to visibility and which will facilitate and clarify us the  action part.

How:

Say you already have a kinetic calendar  OR  confluence cal and such…. where multiple teams lists their upcoming maintenance.  But it’s for various teams and looking at multiple places you need to derive your schedule.  Or else someone will have to maintain one of your own which will be nothing but parts/extracts from all those. So will have maintenance overhead.

We can have yet another  tool…  I almost started working on one and stopped by this idea  - 

  1. Let’s have an outlook calendar  for EBS which is shared across all team members.
  2. **Outlook  calendar which will have  “ONLY” maintenance listed.
  3. Forget everyone BUT just add this email id in the calendar invite - maintanace@yourcompany.com
  4. By default it will display with your calendar, OR   you can also Display on a floating TV Screen extended in your teams work area.

What we need:

We are proposing an email id which will give us a calendar also:  maintanace@yourcompany.com
We will  use  JUST  the calendar  part of this mailbox and all of us have a  shared view.



Security:

We will see if sending  email from this id can be restricted ,  if NOT possible we will put strict security disclaimer for this email id. I guess it is possible.
Everyone should have permission to have shared access that calendar.


Enforcement:

If someone is  calling a maintenance, AND any of your  entity is involved ( which attracts risk or needs your attention)  , please  just have this email id included in the calendar invite.  That’s it!! 

Even If someone missed,     one of us just can forward the invite so that it gets listed in the calendar.

Q & Comments:

How does it sound??  

  
-DK



  🔭 First Impression: Exploring Grafana Mimir              ... After Years with Thanos! Not an observability( OE ) purist, but I do appreci...