Why Database CI/CD?

Making the Database Part of Your Continuous Delivery Pipeline

The database, unlike other software code is not a collection of versioned files. Thus, Continuous Integration(CI) and Continuous Delivery(CD) for Database has taken a back seat to source code, leaving database automation to play a game of catch up at a high risk.

In this analogy below, the conveyor belt represents Jenkins ( or such CI tools ) , the chocolates signify the application code releases / release cycle ( powered by CICD currently ) , and Lucy and Ethel are ‘handling’ the database code.

More to come..  :D 

Stay tuned!

--Debajit Kataki

AWS re:Invent summary 2017

As you might know, this year’s AWS re:Invent was held @ Las Vegas 11/27 – 12/1.  

Below links could be a good start to explore all the new features and services announced at AWS re:Invent 2017

Product Announcements :  

https://aws.amazon.com/new/reinvent/

Categorised summary of news and analysis:  

http://searchaws.techtarget.com/conference/Your-guide-to-AWS-reInvent-2017-news-and-analysis

Links to YouTube recordings of AWS re:Invent 2017 sessions : 

https://gist.github.com/stevenringo/108922d042c4647f2e195a98e668108a

Hope this helps!

Architectural Tradeoffs

"In the 1620's Sweden and Poland were at war. Wanting a quick end to this costly war, the King of Sweden commissioned the building of a ship called the "Vasa".

Now, this was no ordinary ship. The requirements for this ship were unlike any other ship of that time; it was to be over 200 feet long, carry 64 guns on two gun decks, and have the ability to transport 300 troops safely across the waters into Poland. The ship architect had never designed such a ship before. Smaller, single gun deck ships were his area of expertise. Nevertheless, the ship's architect extrapolated on his prior experience and set out designing and building the Vasa. The ship was eventually built to specifications, and when the eventful day came for the launch, the ship proudly sailed into the harbor, fired its gun salute, and promptly sank to the bottom of the ocean. The problem with the Vasa was obvious; anyone who has ever seen the deck of a large fighting ship from the 1600's and 1700's knows that the decks on those ships were crowded and unsafe, particularly in battle. Building a ship that was both a fighting ship and a transport ship was a costly mistake. The ship's architect, in an attempt to fulfill all of the kings wishes, created an unbalanced and unstable ship"

( Image Credit : Ancient-origins )

Software architects can learn a lot from this story and apply the learnings from this unfortunate event to the design of software architecture. Trying to fulfill each and every requirement (as with the Vasa) creates an unstable architecture that essentially accomplishes nothing well.

Lesson to remember:

Every software architect should know and understand that you can't have it all. It is virtually impossible to design an architecture that has high performance, high availability, a high level of security, and a high degree of abstraction all at the same time

#ArchitecturalTradeoffs ( Mark Richards )

#97 Things Every Software Architect Should Know  ( book )

8c tips for Freshers / NCG's

I have been having a great company with some of the very talented NCG’s ( New College Hires/ Grads ) last ~2yrs.  I work for Intuit Inc. (India)  and for the group I work for ,  we have this great culture of putting our goals and objectives into a tool called BetterWorks.

What I love most about this tool is -  it makes things very transparent and gives you a picture about how the entire org is moving along in the desired direction , taking the right steps forward - to meet the mission and vision of the organization  in a very effective way.

Now with this background , let me set the context first. Last week , while setting up the objectives for one of my NCG , I had to help her setting up one personal development goal which is strongly encouraged at Intuit.

 The goal I spontaneously suggested  was :  "Please see something around these C’s , or sign-up for some technical online or instructor led course of your interest"    

The C's what I had in mind , and which eventually made a total of 8 C's are as below:

(1) Curiosity 

The first thing , any individual , especially the new college grads , should be fostering is - the element of 'curiosity'. 

Curiosity means having the desire to learn new things. We must ask questions and have an opinion. Try to be non-bias / non-judgmental and try to look at things  through  fresh eyes.  Research shows that - the primary thing which contributes to non-curiosity is  (a) Fear (b) Disapprovals and (c) Absence of caring and guidance.   My suggestion here would be  - exert yourself to read diverse kinds of readings,  don’t take things as granted , and always keep an open mind.

Out of  the 8  operating values we have at Intuit ,  we have these two great values - “Be passionate & Learn fast “, it aligns very much to this quality.

(2) Creativity   

Creativity is a struggle to go beyond what we were yesterday. Creativity or innovation is the process of leaving the known and venturing into the unknown.  Being a new entry to the organization , you must first try to understand and absorb things quickly  and then also wear a customer lens to suggest process change or improvements ( if any)  ,  apply automations or even suggest  newer / smarter ways of doing things. 

Creativity can transform any obstacle into opportunity!

(3) Courage

At Intuit we strive to live  'these values' every day –  and the first one is ‘Be Bold’ which encourages us to create  a vision that inspires and we always strive to think beyond what is accepted as possible.   

 We must always be willing to take a step forward no matter what...

(4) Competency

There is no short cut for learning , we have to upgrade our knowledge on a daily basis. We must keep learning new things ;  something new , day after day.

Please sign-up of for daily feeds on sites like DZone, mashable, techgig, HBR, readwrite.com – and it will point you to something new,  every day.  

(5) Communication

Communication is the key, both verbal and written. While there are myriads  of tips and  writings how to be a better communicator ,   please don’t worry -    communication is an art,  and nobody masters that overnight, at-least I haven’t yet! :))

One thing we must first try to focus on is  – ensure we know how to articulate and we are understood.

(6) Connect

Connect to people and invest time on networking.  This helps to understand industry trends , bouncing ideas, solicit  feedback or getting input,  formulate diverse perspectives , and also learn from others.  So please ensure you have made at-least one professional connect every week , as you start walking into this new world ahead. 

Professional networking is  the fuel that accelerates success!

(7) Conduct  

Your behavior is everything.  Don’t react, don’t push rough shots and always have a ‘Win Together’ mindset.  Conduct is also about Integrity – which  will help you to win trust , earn more responsibilities , gain respect , and build a brand / class which is very important in this journey forward. 

As it is said -   its not about ‘what’ we do , but ‘how’ we do!

(8) Consistency

And finally , whatever we do , consistency in what we say and what we do is very important , and it will  have a multiplier effect -  to generate the trust on you as a professional.

I personally believe that -  these 8C’s will certainty generate lot of ‘awesomeness’ and ‘effectiveness’ in you  and will help you maintain the right ‘attitude’ all the time.

If you are wondering  , why the title is 8c,  well ,  -  it is a mere co-incidence that the count  came to 8 and I know for a reason that -  in Buddhism there is the concept of  Eightfold Path  to achieve Nirvana - the end of suffering :)

Also ‘c’ is one of the letter in English alphabet , which has maximum no. of words in the dictionary and hence the higher probability of having a strong synonym / title for  all the thoughts that I explained above.

Now is this a SMART goal ? Well, yes. The only thing is that – it does not have an ‘end date’ and it’s a journey!

'Optimism Bias' leads to poor decision making!

Effective Time-management and Time-boxing is a concern for 'always ready people'.  Being from a background that I am from,  I am probably one of them!  While doing project planning , factoring in some room for the unknown or unseen is very helpful , but that might not be possible all the time. 

Today I came across  Hofstadter's Law and then it naturally took me to read more about "Planning Fallacy" 

Hofstadter's Law states : 

 It always takes longer than you expect, even when you take into account Hofstadter's Law. - Douglas Hofstadter

The planning fallacy, first proposed by Daniel Kahneman and Amos Tversky in 1979, states it to be 

 -  "a phenomenon in which predictions about how much time will be needed to complete a future task display an optimism bias and underestimate the time needed" ( ref : wikipedia )

 If we analyze the pattern why many of us often tend to miss the project deadlines ,  you will find that - we usually fall prey to this "planning fallacy" , which is nothing but our tendency to underestimate how long it will take to complete a specific task . 

Its because , we have an optimistic bias  ( TED Talk by Tali Sharot ) towards the task that involve us. 

Optimism bias,  or  unrealistic optimism is the tendency of individuals to underestimate the likelihood  that they will experience adverse events, such as - incomplete or delayed projects , serious diseases or road accidents etc. . As a consequence of this bias, some individuals underestimates the need of precautions that might curb such risks. This bias leads us to believe that we are less likely to get affected from misfortune and more likely to attain success , while the reality might be otherwise.

Planning carefully and conservatively can save us from lot of consequent / unwanted frustrations.

We must remember -  the risk of  'Optimism bias'  is -   poor decision-making!

Basics of 'escalation'

This will be probably my shortest post ever!

Every project that you handle - there is an escalation mechanism /  matrix defined.  Many of us either do not use it in the fear of conflict,  some overdo this , and many of us do it wrong!

Escalating a problem at the right time, to the right person with the adequate amount of information is the key for successful projects!

My observation:

• Excessive escalation can cause stakeholders and sponsors to get numb - hope you remember the 'cry wolf' story :)
• Not doing it with the fear of conflict is a sign of sheer un-professionalism!!
• Doing it with a business continuity aspect in mind is what gets praised!

Regarding #how to do it right -

The golden rule is -  "Escalate to the right person who can unblock you!" -  and that person need not have to be a leader.

While DACI  is an effective model for decision-making , role clarity, and communications   - crisis management is a different aspect all together!  

Wear your 'TIE' for a great conversation!!

This is NOT a post about how to 'Tie a Tie'.  Rather , just an effort to remember a few fundamentals before we strike a conversation with someone. 

Too often, a conversation results in wasted time.  'TIE' -   I am using here as a ref. to help us remember these fundamentals which can help add meaning to your conversations and ensure  effective outcomes.

  T I E ,  Let's break it down:  

      T= Time and Timing.
      I=  Interest and Information
      E= Energy and Emphasize


Time and Timing ( T ): 

Devoting genuine and sincere 'time' for any conversation is key to ensure a good two way dialogue.

At the same time opening up on a topic requires certain atmosphere or frame of mind which I am referring here as 'timing'.

Interest and Information (I): 

Good conversation is never one-sided.  When we talk to someone , our brain is unconsciously but constantly searching for -  where the individual's interests lies. Think of a Venn diagram. When you find these intersections of interests , these conversations flows with great intensity and becomes very lively. We like people who are like us! 

Some other tips would be -  asking genuine questions in between ,  keep asking yourself how you can add value into this ongoing conversation, endear them!

I personally believe that  -  before going for any planned meeting,  we should always arm ourselves with the topic. The Information element.   And if we could  not - we can just support the ongoing  discussion. For. ex. say “I don’t know a lot, but I do know that…”

Never try yo  dominate a conversation.  Conversational Narcissism is bad!  Really bad!  On the other hand we should NOT be a non-contributor either.

Don’t overshare  - don't  be someone who pours out his life story as soon as you meet him. Within couple of  minutes you know why his girlfriend dumped him..etc. etc.



Energy and Emphasize (E):

By 'energy' I do not mean exercising extremes in autonomy or to dominate a conversation ,  BUT a well balance mix of -  effective body language , gesture , stance , pitch and pace.

This is NOT about exerting overuse of energy which might add noise to the conversation , rather bringing your  'whole being'  into the conversation.

We also must 'emphasize' the important words and effectively underline important points during such conversations.

Above all these ,  let's also remember the saying - "the key to the art of conversation is not in the talking, but in the listening"


Well , that's it for now!  Wear this 'TIE' to add meaning to your  next conversation ,  and see if it helps! :))   Don't forget to share your experience.

CICD Assessment

( A story on -  how we formulated a process around measuring and achieving CICD, and  how these journey lines has now become a means to cherish and celebrate our collective achievement as 'one team', with a very transparent and clearly defined objectives )
----------------------------------------------------------------------------------------------------------------------------------------

CICD is the automation of Agile!

Last year we had setup a dedicated team to improve the CICD abilities for various teams across our organization.  The overall objective was to - 'Improve the speed of software delivery by implementing CICD practices and processes".

Primarily we focused on :

• Implementing repeatable and reliable software release and deployment processes
• Automate testing(s), builds, and deployments
• Get everything in source control (not just code! Including config and control files)
• No post deployment manual steps!  (Enables one click rollback when needed)
• Software is NOT Done until “released and accepted”
• And all these , with a shared vision that  - all stakeholders  (i.e. Dev, QE, OPS, RM, RE) will demonstration  equal responsibility for the release process!

When we set out on this  transforming  journey ,  we also wanted to measure this progress for various teams. My friend  'Alison Gabriel-Reilly' reminded of this phrase  : “You don’t know when you’ve arrived, if you don’t know your destination”

We looked around and could not find any such tool which we could use  to measure this CICD initiative  and also to capture  this progressive journey line for diff teams.

Essentially , our  need was : 

• We should have a comprehensive list of all applications , which has an active SDLC track. ( worthy to mention  - we are an IT organization and many apps are in contained mode or waiting for sunset )
• Consistent, holistic view of the initial overall status of various  applications for CICD
• An ongoing means to measure and its subsequent improvement ( Continuous Improvement)

We quickly designed and exposed an evaluation URL , basically an internal tool ,  with an underlying  process defined , that -- as soon as we take up any of such CICD implementation work for any team or app , starting point would be to assess the current state.

Now talking about this tool, this is what the initial screen looks like:

[ Landing page ]

With this flow , once an app is selected from the drop down , we render an expanded view of four key pillars of CICD i.e. 'Continuous integration' , 'Test Automation' , 'Continuous Deployment' and 'Continuous Monitoring'.


Many people debated -  how can monitoring be an integral  part of CICD. The answer was simple:  if my Pre-Prod is down , my build pipeline is stuck.  And frankly , not many organization pay a focused attention on monitoring their Pre-Prod environments ; and CICD pipeline primarily encompasses your  Pre-prod.  When we have a final production ready release candidate , its a 'release' or  a 'release program charter'  , which solicits  all party approvals for a go/ no-go!  Which is decision driven, rather than an automated build & deployment flow.


Again , coming back to the tool , these compositions are nothing but some very fundamental, inherent elements aligned to those broader individual CICD aspects as said above. And for teams -  this now becomes  clearly defined objective(s) to strive and then achieve them progressively towards that big  uber goal of CICD.

The composition screen looks like this:

[ Calculate CICD score ]

We also introduced a very fun and exciting  element in this whole process , that -   you potentially can earn a 'badge' based on where you stand. and we display it at the organization level.

[ Badges as per the CICD score range ]

We also have a quarter wise QlikView dashboard , which the leaders of respective group(s) can keep a track of  their own team's CICD status , their progressive journey line, and continue to encourage them further.. 

 

[ Org level view ]

[ Journey line graph - Quarter wise , for individual app ]

Here I wish to mention that -  this entire thing has now become a proven and well established process for our entire org towards an ongoing means to measure and then take up the subsequent improvement steps for those teams/apps -  who has been aspiring to be on the CICD orbit ( i.e. Continuous Integration , Continuous Delivery )


With this tool,  and with these underlying scores  , it is really allowing us to understand -     the current CICD status of those teams or apps  ,  clearly envision the end state, track the ongoing progress in the CICD  journey and collectively agree on ( and act!)  -  where to focus our time and effort  towards enabling these team(s) to achieve complete CICD.

With this established process , these CICD journey lines has now become a means to cherish and celebrate our collective achievement as 'one team',  with a very transparent and clearly defined objective.


Be agile , deliver continuously! 
                                                                                    
 Thank you!!