In December of last year, I've joined Wajam, a great social search start-up. Since then, I've been very busy and learned a ton. I've coded more in the last 6 months than in the previous 2 years. I've learned about Information Retrieval, NoSQL databases and Hadoop. I can now program in Scala, write Pig and Map/Reduce jobs. The pace is incredibly fast; I often come home on Friday completely drained. I work with pure geniuses and lightning-fast typing hackers. Nothing can't be done. Ideas are implemented. I love it! At 34, I'm on the old side. I must admit I sometimes wonder how I can hold the pace with small children at home, but somehow, it works.
One of the aspect of start-up mode that I want to discuss is continuous deployment. When I joined, I was scared: new code was deployed many times a day. There was practically no unit tests, no formal QA, no stress tests. Initially, I thought the system would be continuously unstable and that most developers would spend time managing production fires. It happened...sometimes...but not as often as I thought.
If you know me or if you've been reading some of my posts, you know that I'm a test-oriented developer. So I've started adding unit tests and integration tests and made sure they were run every time new code was added. Then we've added code reviews before pushing to production using the github development flow. As of now, I've never seen a system as stable as the one we have at Wajam...and the best part is that the team still pushes code in production many times a day. I'm not scared anymore.
Unit tests, code reviews...they all help keeping the system stable. But I think the real key to a stable system is deploying often. I know it sounds counter intuitive so let me explain. In the beginning, I would write a feature, then write some more code. The more code I wrote, the more I was scared of deploying it, which somehow led me to add more fixes and small improvements before deploying?! The problem is: the bigger the code change, the more likely the change will break the system and the hardest it will be to troubleshoot. On the opposite, if small chunks of code are deployed continuously, it is easier to validate the code during the code review and easier to find the bug if the change breaks something because the difference between the stable system and the now unstable system is small.
Deploying code often keeps the adrenaline high and the developers alert. There is no sloppy commit since you know you are the last frontier between implementation and production. Developers take a second careful look at the code they write which results in better code in general and the fact that you know your code will be reviewed pushes you to be more thorough.
There is also a motivational aspect to continuous deployment. The reward is instantaneous. You see the new feature live a few minutes after you finished it, you feel the relief when the CPU usage drops by 20% after the code optimization is deployed, etc. It keeps the momentum going.
I remember working for months on a release that would spend some more months in QA...then spending weeks identifying and fixing stress test problems because the diff between the two releases was a set of hundreds of files. Now, I wonder how this could possibly work.