Yes, you can forecast software delivery dates with agile

Agile is often mis-used by product development teams to mean, “what ever we end up delivering”. That just isn’t right. It gives Agile a bad name. It’s incorrect and mis-applied by those that don’t understand what Agile is.

At VitalSource, we use Scrum as a way to apply Agile principles to the way we develop, deliver and maintain software product. Scrum gives discipline to the way we do Agile. The artifacts and ceremony around Scrum are important and keep us moving forward. Daily stand-ups, sprint demos, story grooming sessions, retrospectives gives us a cadence that insures we are doing the right thing, delivering value, removing impediments, and communicating with stakeholders and with each other.

Can we make commitments to delivery dates with Scrum? The answer is absolutely yes. We do it successfully all the time. But, before I jump into that, let me first say that because software has no constraints set by the physical world, estimating software complexity is a notoriously difficult thing to do, and you have to understand complexity before you can have accurate estimating. By complexity I mean - how much work will it take to build a software product, which implies, how hard will it be to do? There was a time when there was a lot of work put into figuring out software complexity, but by and large, in commercial software development no one does anymore, but rather the focus has been on better techniques to deliver software product.

To be clear, waterfall models were (and still are) terrible at software completion estimates. Large specs were written, which inevitably couldn’t cover all nuances and use-cases, the specs were handed over to user interface designers, who would design interfaces no one could implement, which was then turned over to developers, who would scramble to write code that matches the specs and user designs, and the dates would slip because the developers had to go back to the spec writers for clarity, and the spec writers would have to re-write the spec, and maybe consult with an end-user, while at the same time the business owners would be harping about the delivery date that could not be missed, developers would have to squeeze in more time, rush to the end, hand it over to QA, who would rush through testing (because you were already late), and developers and testers would fight over bugs, and then everything would be turned over for acceptance testing, which you never really had to time to do, and finally you get the product out the door, most often late, buggy and incomplete, only to find that the delivered product did not end up looking like the spec which was written months before. It was a nightmare, which pitted people against each other.

Scrum takes a radically different approach to software development by putting product managers, QA engineers, and software developers on the same team, working together on a sprint-by-sprint bases to incrementally deliver business value in the product that can be seen and used. Each increment, or sprint, gets you closer to the end game - the product delivery date. The beauty of the model is that after each sprint, the team reassess what is left to do, measures risk, re-prioritizes work, and moves on to the next sprint. Users stories, which represent business value, get completed each sprint and so the product gets incrementally done - and that includes closing bugs so they don’t accumulate until the very end. During the sprint demo everyone gets to see the product evolving. Every sprint represents an opportunity for the team to do a course correction on the path to successful product delivery.

At VitalSource, we deliberately use the term “forecast” when talking about delivery milestones. Because that is what it is, a forecast. The farther out in time it is, the less trustworthy a forecast is, the closer, the more accurate it is. A team can forecast that, “we believe it will take us ten 2-week sprints to have that feature complete”. They base that on their understanding of the their cadence and the complexity of the features (user stories). We have found that our forecasts have been pretty accurate, and have several successes that demonstrate so. We will naturally have time where forecasts will be off, but the beauty of Scrum is that we find out sooner rather than later, because after each sprint we can see if we believe we are on target.

Can we commit to “firm” dates? Yes, we can, and we often do. This is particularly true when we have commitments with external customers or partners. This is essentially “time boxing” the work. In order for a time-boxed effort to work, the team has to develop a prioritized list of work and agree on the minimal needed, and agree that the minimum can comfortably fit into that time-box. If there is time left, they can work down the prioritized backlog of features. Again, we have had success doing this, and I have seen it work may times in my career.

One of the best feelings in the world is to get product out the door. Even better: an awesome product that our customers are excited about and solve their real problems.

Making Product and Engineering work

One of the tenants of the Agile Manifesto is that we value “Individuals and interactions over processes and tools”.   That doesn’t mean we don’t value process and tools, but rather that teams of highly motivated people, working together, and empowered to make decisions in order to get product out the door, can do amazing things.  I have seen it many times in my career.  We get better product, better people, better teams and better retention.

Almost paradoxically, I also really believe Kristof Kloeckner’s insight that:

“Agile without discipline won't scale. Discipline without Agile can't compete.”

Too many software product teams define agile as “whatever we end up delivering”.  That’s doesn’t work and that’s where the discipline part helps.   That’s also why I like Scrum.  Done properly, Scrum adds enough discipline to make agile work.  Sprints, backlog grooming, daily stand-ups, sprint demos, all add structure to the way we do agile.

It’s been my experience, both here at Vital Source and in my previous companies, that the roles of Product Manager and Engineering team in Scrum can have conflict, unless we have some understanding upfront of who owns what.  The way we have chosen to explain ownership is by what questions each role answers:

Product Manager answers: “why” and “what”
Engineer Team answers:  “how” and “when”

Even though we choose to define roles this way, hopefully this doesn’t mean that we don’t listen and get feedback and input when answering these questions.   Here at Vital Source we have Product Managers who have a good understanding of technology, and Engineers who have a good understanding of the Products we deliver, which facilitates discussion and consensus.

To clarify the roles some more, the Product Manager owns the priority of the backlog, and the message of why things are in the backlog and why they are in that order.  PMs spend time talking to sales, account managers, stake holders and customers in order to prune and manage the backlog.  PMs spend a lot time in both outward and inward communications about the products they manage.

The engineering team is responsible for the execution of the backlog.  By using estimating techniques, the engineering teams estimates how many sprints something will take and when coding and testing will be done in order to deliver a solid product.  When considering implementation, the engineering team will make time for technical debt, and make decisions on tools, languages and software architectures.

If would not be appropriate for the engineering team to change the backlog order without sign off from the product manager, neither would it be appropriate for the product manger to commit to a delivery date or technology implementations with out sign off from the engineering team.  When I personally see these things happen, I speak up.  Hopefully others do as well.

That’s not to say we won’t make mistakes along the way.  We are all at different stages of learning the process and coming up to speed on what we do.  I hope we can be tolerant of mistakes and helpful when things go wrong.

If we can truly learn to value "Individuals and Interactions" and worked towards being a jelled team, we can deliver great product and have fun in the process.

Technology teams should own customer-facing products, not technology

In a business that is responsible for delivering a technology product, I think something incredible wrong happens when a group of engineers feel they are responsible for delivering technology and not for delivering a product that gets delivered to market.  The engineering team starts to focus on the wrong thing - building more technology instead of building business value.

This is a core value that drives the decisions I make and the way I approach technology leadership, and drives the product delivery model I implement.  This is one of the reasons I like Scrum teams - properly done, they encourage this style of ownership.

Engineering teams should be aligned with business units that deliver customer value.  They should be strong participants in and contributors of the culture.  They should understand the business.  They should know and interact with the business teams, they should have a tight relationship with the product managers.  You will get better product, you get happier engineering teams, more retention.

I love it when technology teams feel like the "own" a product and are passionate about it, rather than being soulless cogs in a machine processing tasks assigned to them.   As a big added bonus, if you have ownership, process becomes a lot easier.

If you want better technology product, make your engineering teams responsible for the product they deliver, not for the technology they build.  You will get better product AND better technology,

Engineers: Four ways to increase your communication skills

Engineers are generally pretty good at communicating technical details of their work to other engineers, but things start to fall apart pretty quick when an engineer needs to talk to non-engineers about their work.

Over my career, I've learned how important it is to be able to communicate to non-technical people.  You have to be able to talk to product managers, sales staff, CEOs, CFOs and the like, in a way which clearly explains issues, features, and technology without loosing them in detail.  It's an important skill which takes time to develop.

Here is the key: great engineers know how to articulate technology vision, issues and details clearly.

If you want to take lead technical roles, you will need to know how to communicate to people.

So with that in mind, here are my suggestions for improving your communication skills.

(1) On a regular basis, explain to your significant other why what you do is important.  

Don't explain what you do, but explain why what you are doing is important.  The difference here is subtle, but crucial - you need to be able to explain to non-engineers what value you are adding to the product you are developing.   Do this on a regular basis.  If you are not used to this it can be really hard at first, but it is worth it.

Besides developing a better relationship with your significant other, you will learn to talk business value  about the technology you develop - a great skill to have.

(2) Give presentations.

Is there a need to update sales staff,  demo to a customer, or present a trade show?  Volunteer to do it.  At first it may be hard, but you get better at it with practice.  Talk to your manager and let him/her know that you want to present.  Solicit feedback on how you did.  Does your company offer classes or courses on presentation? Take one.  Another great place to present - user groups.  You can even look for non-technival venues to teach or present like your church or local non-profits.  The more you get used to being in-front of people the easier it becomes and the better you become at communicating.

(3) Listen.

That's right: to become a better communicator become a better listener.  When someone comes to see you at your desk, take your hands off the keyboard, and eyes off the screen.  Turn towards the person who is speaking to you and look at them.  Listen to what they are telling you - don't be raising ahead thinking about your response.  Don't interrupt.  

Again, for many people this is really hard to do and it takes practice.  In order to communicate to people you need to be able to listen, process, and understand what they are telling you even if you disagree with them. 

(4) Read the classics.  

I know, I know -  all you guys hated English and Chaucer.   That's not what I am talking about.  There is some really good stuff out there, so put aside your collection of Battlestar Galactica fan novels for a while.  The classics will not only help you see the world differently, but will dramatically increase your English skills.

I love science fiction and have probably read more of it than most people, but I also have grown to love the classics.  I have read almost everything by Dickens and have read all of Les Miserables by Victor Hugo (unabridged!)   I read them because I thoroughly enjoyed them, but I realized in the process of reading the classics that my command of the English language increased and I got a deeper appreciation of my fellow human beings.  I believe all this helped me become a better communicator and I believe it can help you too.

Adventure?  The Count of Monte Cristo by Dumas.
A cool story about over coming the odds against you? David Cooperfield by Dickens.
A story about pride, greed and good guys overcoming them? Daniel Deronda by George Elliot.

Give it a try.

-----

What about you, do you have any suggestions on how to increase communication skills?

Shine a light on software errors to increase quality

My team develops, tests and operates a fairly complex SaaS product.  We have customers who run their business on our product.  They pay us money and they expect it to work.

I am proud of my team - they deliver a high quality product.  We have a release every 10 weeks with minor point releases in between.  For a complex SaaS, we do a lot of releases.

I believe that the quality of the product you deliver goes up by not hiding errors.  In our application, every time user action generates an error, whether it is visible to the user or not, our application generates an email that goes to the DevOps team and myself.  Every time.  Very few of the errors block or affect user activity and almost all are recoverable, but we know they happening and then we look for patterns, we investigate and we prioritize fixing them.  If we believe a user is blocked or having trouble we reach out them and ask if we can help.

I am a big believer in exposing errors and bugs in a software application.  If your code is silently logging to some log file then you never know they are there, you may ignore them as "expected" and no one ever gets around to fixing them.

Here is a cool side affect of exposing errors - everyone becomes more focused and aware of them.  The QA team makes sure that we test thoroughly and known errors are captured.  The software developers want to make sure their code works - the quality of the entire applications goes up.

If you want to quality of your product to go up, put in mechanisms that expose errors that users are seeing to your technology team and make it clear that minimizing them is a high priority.

Jelled Software Teams

A truly jelled software development team is a beautiful thing to see in action. They talk to each other, not around each other.  They hold each other accountable.  They push towards a goal together.  They deliver very cool product that they are passionate about getting right.  They take pride in their work. They occasionally goof-off together. They have one goal, and they all know what that goal is.

If the team is functioning right, they care more about getting the product right for the customer than what an internal stake holder (like the CTO) thinks. They are willing to fight and standup for what they believe the right thing is to do. Like I said, it really is cool to watch these people in action.

But, here is the thing: not everyone recognizes that software development is a team sport, and as a result teams are often mismanaged.  Computer Science and Engineering degrees teach the discipline as a solo effort.  In college, you work on projects by yourself and rarely collaborate - and then when you get into private industry, you are thrown into a team to work on a product. Stake holders and senior management in many companies come from sales backgrounds, where they rise through the ranks competing against other people in their own teams, and generally struggle with what it means to build a well-functioning team.  People with accounting and liberal arts backgrounds might not get it get it either. Heck, a lot of technology leadership struggles with this too.

I could write a book on what it takes to build great software development teams (and some day I might), but if I could give someone some quick advise there two things: form small teams and lose the command-and-control leadership style.

Any software team with more than 3-5 people on it makes it difficult to jell. Meetings take forever, you loose track on who is working on what, and use loose some of the closeness that comes from working with a small group of people.  This small team is not a "chief surgeon model", where one person is the principle and everyone else is there to help him or her.  This is a team of peers where everyone has an equal voice at the table.

The right sized team is about 5 people and consists of 2-3 developers, 1 QA person and a product person.  A technology group may consist of many of these kind of small teams all working on different tracks during a release cycle to get something out the door.

To have strong jelled teams, you really have allow them function as peers to technology leadership.  In the 21st century, especially when dealing with knowledge workers like technology people, it isn't unusual for an someone to know more than their boss about something.  And, that is OK.  It shouldn't come as a challenge to authority, but should be encouraged. Mistakes will happen, but a good team will recover quickly and learn from them.

Some great benefits from jelled teams: they are happier, enjoy working together and deliver better product. Turn over and discontent also go down. People stop complaining to their manager and start talking to each other to solve problems.

It is worth the effort focus on teams.  Like I mentioned in a previous post, most teams have people problems not technology problems, and having jelled teams goes a long way to solving them.

Yes, Software Architecture really does matter

Software Architecture really does matter and, apparently, this is not a commonly held belief.

All too often, software developers are hired to "program" an app and get it out the door.  Bad patterns from previous gigs are carried over and perpetuated. Whatever technology leadership is in place just reacts to business pressures, and puts pressure on the team to deliver while trusting the developers to get the architecture right.

At some point architecture decay sets in, the code gets brittle, new features take forever to add, new hires can't really figure out the code,  the software gets bloated and runs slow, management gets frustrated and demands to know who is responsible, someone gets fired or quits, and new leadership is brought into "fix" the problem.  I've observed this pattern several times.

Here are my general guidelines for software architecture:

Don't be special!

Software architectures get into trouble when a couple of developers think they have a special set of problems and decide to be special in the way they apply well known patterns or technologies.  I've seen this many times, and have had to pay the price to fix it.  There is a reason why patterns develop over time, and why the community of developers apply something in a particular way.  Use the technology the way it was intended to be used. Otherwise, the next group of people won't understand your "special" techniques and you won't be able to upgrade or evolve properly and then someone will have to undo what you did. Don't be special.

Listen, pause, think, code, repeat.

Before you write the fist line of code, pause and think.  A one hour or one-half day discussion about where everything fits and what the proper architecture should be pays very large dividends.  Find an architect or senior developer, sit with them and bounce around ideas.  Look at what others have done.  Read articles, blog entries or books.  Avoid inventing something new, use something that already works.

Don't go dark

One of the worst things you can do when coding is go dark to the rest of your team.  You need to share and they need to know what you are doing (daily stand ups really help here).  Review your architecture and code with your team often.  Tell them what you are doing often. Keep running your ideas past them.  Doing all of this will minimize the chance that you will do something dumb.

Technology leadership should really lead

Software architecture should be a top-level item for technology leads and managers.  Talk about it, espouse it and be familiar with the architecture that the coders are developing.  Too often, technology leadership has no idea about the software architecture.  Some managers manage by Gantt charts and feature lists -  that's not leadership, that project management.  Lead and get in front of architecture issues. Make sure senior management is aware that this is important.  Add time to the project for it.  Schedule meetings to discuss it.  Give senior and lead developers time to address it.  

Seriously, architecture is important stuff.  Don't just hope it happens right.