Why are software delivery estimates so hard to get right?

A day after Mountain Lion came out for the Mac, I upgraded.  It works great and I am very happy with it.  If you noticed, Apple didn't let anyone know the exact release date,  until the day before it was released. We still don't know when iOS 6 will be released by Apple - only that it is coming out sometime in the fall.  This is now the norm with software vendors.

When I was getting my graduate degree in CS, I spent a lot of time thinking about software complexity,  which was the subject of my master's thesis and project.  I built a cool software model around AJ Albrecht's work for measuring complexity called Function Points.

Software complexity is important, because understanding complexity helps you understand how long it may take you to build something.  If you know that a software system has say, n complexity, before you start building it and you know how long it takes to build something of n complexity (based on historical data), then you know how long it takes to build that software system.

The fundamental problem is that you never really know upfront how complex a software system is, and hence how long it will take you to build.  This is something that people with production-minded backgrounds have a really difficult time getting their minds around.  It's not like manufacturing printers or building houses - software is not constrained by the laws of physics, has many more moving parts, and can be changed at the very last minute.  As a matter of fact, in a good software development process, you delay as many decisions as possible - because you are always learning things as you are putting product together.  Some people say developing software is more like writing a book or producing a movie, and that sounds about right to me - you can and should change things at last minute, as long as you understand the costs.

Also - the larger the project, the bigger the estimate risk.  I personally like breaking up large projects into smaller deliverables,  which gives you insight on how well the product development is going.  In our current product, I like (and use) two week sprints and 10 week release cycles.  That means that every 10 weeks we deliver new product like clock work.  If something will take more than 10 weeks, we break it up so that deliverables fit into that window.  If a 10 week release cycle deliverable it not useful to the end user, we keep it in-house - but interestingly that is rarely the case. We can usually find value to deliver every 10 weeks.

A time-constrained release cycle also gives you a time box for delivering value.  Many times we will constrain the delivery of a feature to a release cycle.  This forces us to make decisions about what is really import.   There is a lot that can get left behind, but we often find those things were not that important.

It also really helps to have senior software developers with deep experience.  Nonetheless, it's non-trivial to estimate how complex a software system is before you build it, but if you break up your product into small deliverables, and time constrain those deliverables, you have a good chance of hitting delivery estimates.

It's an app!

This week we delivered an app for iOS devices.  It's a mobile portal to our enterprise operations platform, and it actually is pretty cool, robust and functions well.  It connects to our EC2-based infrastructure over a secure connection.  You wouldn't use it unless you were a customer.

Our app is non-trivial - it interacts with our cloud-based systems to do schedule calculations and manage complex tasks.  It took us a while to develop and I think we got the first pass right.   Like any other piece of software,  there is things we can improve on, bugs we will find, and work flow improvement we will make over the next few release cycles,  but I am proud of what our team has accomplished.

This is not my first iOS app, but I learned somethings again in the process.

First - having a good iOS developer in-house to prime the pump is crucial.  Once the pump is primed, good software developers with deep experience can jump in, pick it up, and contribute.  We did the real thing - Objective C using Xcode.  I have not been impressed with third party iOS development packages and they lock you in to their framework.

Second - the app store approval process from Apple is still a pain.  They filter out incompetence by the complexity of the process.  Maybe it's on purpose.

Third - Mobile user interfaces are very different from web portals (no surprise).  We didn't look to re-invent the wheel here.  We looked at what others were doing and followed similar UI patterns. No need to complicate things.

The iOS ecosystem is cool.  I don't mind objective C and Xcode - I just wish approval and deployment were easier.

Agile: dropping the Scrum training wheels

Many of us have been using parts of the Agile Manifesto for many years.  What you find in the agile movement,  is a rollup of good practices that many of us long time software development practitioners have found work.  I am a big fan of agile principles, specifically short development iterations, the focus on working code, and the focus on human interactions over documentation.

A couple of years ago we introduced scrum at my current company. Scrum enforces good practices that agile talks about, by introducing daily stand ups, sprints, demos, etc... It also introduces a Scrum Master - a facilitator that enables communications and makes sure that decisions are made.  A good Scrum Master also keeps the process wheels well lubricated to make sure everything moves along well.

We went all "in" on scrum - Story boards, story pointing, estimates - everything pretty much by the book.  We bought Jira + green hopper to help manage everything.  All went pretty well first year.  Every sprint and release cycle we got better at the process. People acted like adults and owned things.  Estimates were made, product managers were brought into the loop.  It also helped tremendously that I hired a Scrum Master who had lots of deep experience in Scrum, and who is a fantastic communicator and facilitator.

Then, we started dropping parts of scrum. Scrum helped us stick to agile principles, but over time, we started dropping some scrum things that we grew out of.  Scrum purists would scream foul, but I have always believed that process should adapt to your people and circumstances, not the other way around.  I believe we do agile, but not scrum anymore.

Here is some things we dropped:

• A full time Scrum Master.  Our Scrum Master is now our director of Software Development.  He still helps make sure that the process works well, but by and large the role of scrum master has been taken over by the software development teams themselves.  We have four development teams of about two developers and one test engineer.  They know enough about the process to move things along themselves.  If they need help, they ask and we get it for them.
• Story Point estimating. Story point estimating blows.  It's hard to get estimates right. You have to keep a history, but when people move around teams, the estimates get all messed up.  Now, we still develop stories and talk about them upfront. We still do exit criteria on stories, we prioritize stories, but we just don't story point them.  How do we estimate how much can get done in a release cycle or sprint? We do rough estimates with everyone involved ("I think we will get this far").  You know what? those work as well as all the time we put in story pointing.  Sometimes we are off, more often then not we pretty much on.
• Sprint retrospectives.  We do retrospectives, but not that often. If the teams are always talking (and they should be), there should be no surprises.

We have kept stand ups, demos, sprints (two-weeks) and we like Jira.  The key is that we are free to evolve the process as we need to, and don't necessarily feel we need to be held up by the scrum model.

There are plenty of things I wish we could do better.  We struggle communicating business workflows well, some developers try to be too heroic and do too much,  and have a tendency to leave the test team behind.  But, here is the key: we talk about these things and we try to do better every sprint.  

I know this is heretical to many of my scrum-devoted friends, but I say: make the process work for you and drop the scrum training wheels if you feel you need to.

Software Development: You don't have a technology problem, you have a people problem

"There is nothing new under the sun"  - Ecclesiastes 1:9

We rarely invent anything new in commercial software development.  Most of what we do is to apply well known patterns and technologies to the domain we are working in.  Now, this does not mean what we do is trivial or easy, but generally, we are not inventing anything new.

So then, why are so many software projects, late, bloated, over budget or failures?  According to Tom Demarco, in his classic book, Peopleware, it's not because of technology issues, its because of people issues.

That's right, you don't have a technology problem, you have a people problem.

Bad communication, egos, ignorance, office politics, inexperience, weird personalities all are bigger problems than technology issues. These things all get in the way of success. Unfortunately, the vast majority of technology leadership struggles with dealing with people issues.

It's easier to deal with a build problem, or to fix a bug,  then it is to figure out why Sally and Mike are not getting along on their project. Technology problems are discrete and deterministic, people are messy and emotional.

I have found that finding leadership that is both technical and people-oriented is very hard, and when you have someone who has both, hold on to them, but have to build these soft skills in your existing staff.  I do this following some loose principles:

1. Technology leadership needs to be technical.  This is critical - you can't lead people unless you know what they do so that  you can participate, lead and provide input.  You have to know how to code.
2. Focus on "doing the right thing", not pleasing some internal stake holder.  When you focus on doing the right thing, politics and bickering takes a back seat.
3. Don't worry about who get the credit.  This is easy to say, hard to do, but when you have a team that really believes this, great things can happen.
4. Remember that software development is a team sport.  Empower the team to make decisions and allow them to be responsible for their actions.  Treat them like adults.
5. Take time to talk and listen.  Spending an hour listening to a person who is struggling with a decision, giving input (not telling them what to do), and helping them make a decision is a critical management responsibility.  I make sure I have time every day to talk to people and listen.
6. Treat people like adults, not like children.  Management is not like parenting.  The people that work for you are adults, treat them as such and respect what they do.

It is true, there is a lot malpractice in this industry related to technology, but the issues all start with people.  People design, implement and test software.  Focus on your people problems first and your technology problems second, and you will have a lot more success.