It Shouldn't Be This Hard

One nice thing about working from home and living close to the school is that I can slip away for a bit for things like a talent show my daughter, who is in first grade, performed in. I think 3 of the 15 talents demonstrated involved hula-hoops. I was reminded that it is amazing how some people really get it and they can stand there and make slight movements and the hoop will just glide around there body with very little effort.

Here is a picture of a little girl (not my daughter) that stood almost motionless with her arms folded as the hoop went around and around. I'm sure she could have maintained this trick for hours before she stopped from boredom rather than exhaustion.

Others (like me) will shake their whole body and still only get the hoop to go around a few times before it falls to the ground.

Being the geek that I am I thought about how software development is very similar. I see many people use brute force to build software only to have things fall apart very quickly while other times projects can go very smoothly and can successfully building software for a long time.

The last 2 years has been an interesting experience for me. I joined a team that was disciplined in practicing eXtreame Programming (XP). We still had challenges and issues to overcome but overall it was a productive team that produced a lot of functionality (notice I didn't say code) very quickly. We were sheltered by a management team that understood how to build software and not only allowed us to be agile but expected it.

Eight months ago this team was transferred to work with a startup our company acquired. This company does not believe in XP and doesn't understand the principles and values that guide these practices. Oh yes they claim to be "trying" agile but they really don't understand what it means otherwise they would know they are not. We knew there was going to be a clash of culture but felt like we could still be good software craftsmen because hey, we've been doing it as a team for years.

Wow what a difference! We have been shaking more than our waist but the hoop just won't spin. It seems like we have accomplished very little in the 8 months working on this project. It was the same people with the very similar requirements and technology so what was different? Here are a few that I can think of:

• Automated integration tests - Yes the first thing we did for our project was to setup an continuous integration (CI) server to make sure code we wrote maintained a high level of quality. The problem is that we have to integrate with many other systems and developers that don't believe in the same level of quality. One of the practices of CI is that if the build breaks you stop new development and go back and fix it. The pain comes because other systems that we are integrating with are very unstable and the build is constantly breaking because of external forces instead of code we were responsible for. This would lead to anywhere from 10-40% of our time troubleshooting what was broken. It got so bad at one point people were talking about throwing away our integration tests! The lesson learned here is that for CI to be effective it needs to be done throughout the system, not just one application.
• Communication - Doing agile across many timezones with multiple distributed teams has it's challenges but we were actually successful doing it for a few years. We would make a special effort to do things like weekly brown bags, acceptance meetings, retrospectives, scrum of scrum calls, intense planning sessions where we would come from all over the world to one hotel and plan the next 4 months and other things that enabled lots of open communication. In this new team I know more about what's going on in the kitchen (unused sugar packages, when the pizza is ready, etc.) at the office in CA than I do with commitments people make on behalf of our team. We had a planning meeting in February where they informed us that the CEO had committed to the board that we would release the public 1.0 version of our product on May 5th. Not only did I not know about the date before the commitment but I had no idea what was expected to be deliver. The funniest part of it was that we had absolutely no input as far as estimates and ability to deliver this yet unknown functionality. By the way it's now May 15th and we aren't even close to delivering a 1.0 release. It's clear that we are considered "programmers" that are on a need-to-know basis and we don't need to know. The lesson learned here is that in order to gain commitment and enthusiasm of a team management needs to communication clearly the vision and get input from those that need to carry it out.
• Must-build-it-here syndrome - This company has a very strange choice in what it chooses to build verses what it builds. One example is a system they built themselves that is responsible for message passing. The system requires the applications that need to talk on the transport system to manage tcp sockets, a custom handshake protocol and send malformed xml messages to various systems. As an team developing to this architecture we spend about 20% of our time the first 3 months to deal with this technology. WHY! Messaging systems have been around for a very long time. There are very large commercial packages, there are lightweight open-source packages, there are many standards already in place. Why do we have to write custom software for this? The very first retrospective we had I questioned this decision by asking, "why are we writing our own messaging system when there are so many existing messaging solutions out there?" They didn't even understand my question at first. After explaining my question they said that this system was already in place. The amount of time spent on this proprietary messaging system just baffles me. The lesson learned here is know when to leverage existing technologies so you can focus on what differentiates you.
• Over-engineering - One of the core differences of opinion is the idea of building functionality that you might need someday. On an agile team you may hear the acronym YAGNI every so often. You Ain't Gonna Need It is a short way to describe the practice of building only what you know you need right now and nothing more. It's based on the principle that first your ability to predict what the future needs are usually very wrong. Secondly, your not working on the most important thing which is what you know you need right now. Thirdly, it often leads to extra flexibility which comes at a price in maintenance overhead. There are many resources that talk about the YAGNI principle but it is not something the founders of the startup believe in. We constantly here that something was coded a certain way "in case someday we need it." In the process we have built an incredibly complex system to do some very simple things. I'd give an example but this blog is running on and I don't think anyone will ever get this far.

I really think these last 2 years make a great case study that has really confirmed my believe that Agile is not just a buzz word but it's based on principles and values that really really do work when used correctly.