How do you respond to surprises?

I've been involved in several start-ups during my career and the one thing I have learned is that things don't go as expected.  A feature doesn't work well, the software doesn't scale, unexpected bugs, or customers who are not happy with some aspect of the product.

What have I learned?  Keep cool.  Don't start blaming people or look for someone to shoot.  It happens to the best of organizations - mistakes happen and things don't go as expected.

My approach for handling unexpected urgent events is something like this:
1) handle the situation in front of you - don't worry about how you got there - just fix it and get past the issue
2) do a blameless post-mortum.  Organizations make mistakes,  but if you are so bent on blaming someone, loosing your cool and throwing a tantrum - the organization becomes risk averse, non-cooperative and plays CYA like a professional sport.
3) don't feel like you need add a layer to the process to prevent whatever from happening again.  A process artifact is usual in place because someone made a mistake years before.  It drives me crazy.  Fix the problem, learn and move on.  You don't have to modify the process every time.

Loosing your cool, throwing tantrums or becoming moody is the sign of immaturity in an organization or individuals.  Amateurs.  Anyone who has been doing technology long enough knows that things happen, and knows enough to learn and move on.

On "Working 40 hours, and mailing it in"

A good friend of mine, who I have a lot of respect for, asked me a rhetorical question on Friday that has bothered me all weekend.  Referring to someone we both know, he asked "What's wrong with working 40 hours a week and just mailing it in?"  The funny thing is that my friend is one of the most passionate and involved guys in technology and technology processes I know.  A family man, who balances his work and life well, nevertheless he does not "just mail it in" when it comes to his career.

Once in my career a few years ago, I was in a "just mail it in" position.  I had a hard time filling 40 hours a week.  I was not challenged.  I was bored. Oh, I had a nice office, was well treated and respected, but not being challenged led me to spend a time finding more interesting things to do - and initiate quiet a few office pranks.  I left for something more challenging, that certainly required more hours - but, and here is the important point: it was a lot more fun!


As a technologist and software developer I have always looked for people in my organization who have a passion and are deeply interested in what they do.  People who write code on the side for fun, are the kind of people I want to work with.  Don't get me wrong, I am a deeply committed family man and have tried hard to spend all the time I could with my children when they were small, and now have a very good relationship with them now that they are college age.  I love spending time with my wife too.  But, my family knows how enthusiastic and interested I am in technology.  I love to build it and use.  I can't imagine having a job and career that I just clocked in for.  That would really blow.

Henry B. Eyring, the son of famed theoretical chemist Henry Eyring tells a story which I can relate to:

'Henry Eyring encouraged his sons to study physics and to prepare for a career in the sciences. Hal dutifully majored in physics at the University of Utah, but one day when he asked his father for help with a complex mathematical problem, it became apparent to Henry that Hal did not share his passion.

“My father was at a blackboard we kept in the basement,” Henry B. Eyring recalls. “Suddenly he stopped. ‘Hal,’ he said, ‘we were working at this same kind of problem a week ago. You don’t seem to understand it any better now than you did then. Haven’t you been working on it?’ ”

Hal said he had not. He then admitted to his father that physics was not something he constantly thought about. His father paused a moment and then, in tender words that released his son to pursue his own professional passion, he said, “You ought to find something that you love so much that when you don’t have to think about anything, that’s what you think about" '  (source)

That is great career advice I share often.  Technology for me is something I think about when I don't have anything else to think about.  It's cool and fun, and that's why I don't just work 40 hours a week and "mail it in".

Hiring Good Coders

"Why the new guy can't code?", a very good article from TechCrunch (no less), got me to thinking about my hiring practices when looking for new programmers.  My favorite and best programmers are those that do it for fun, not as a way to get the bills paid.  The best ones always have a project on the side, because they can't get away from it.  They have cool projects that they can brag about and are eager to share specifics.  During interviews, I look for that.  Those are the guys (and gals) who have the passion to do something right and figure out complex problems.  

As an example, a while ago I hired a new grad out of NC State, here local to RTP.  NC State has a terrific CompSci program, but what convinced me about this fellow was his personal web page and links to applications and algorithmic problems he was working on for fun, on his own.  I hired him, and never regretted him.  That was a job ago, but I still keep in touch for him and would love to have him on my current team, if the opportunity ever presents itself.

No need to Panic: Why we are sticking with Amazon Web Services.

Yes, the outage was very bad for Amazon Web Services.  Essentially, it was a multi-zone failure in the US East region for AWS that was not supposed to happen.  Amazon claimed that each zone in a region was independent from other zones, and that failures in one zone would not affect another.  The failure over last weekend showed that did not happen.  Failures spanned availability zones. That was bad.

Luckily, none of our production systems were affected.  Don’t know why, luck of the draw. Our stage systems went offline, and eventually we got tired of waiting for them to recover, so we copied the EBS volumes, and spun up new systems with the data of the old systems and everything worked fine.

Am I happy with the failure? No.  Especially since EC2 makes it impossible to migrate running instances to other zones or regions for that matter.  Also, the lack of transparency on how these availability zones work is very disturbing.  It’s hard to take their word for it until they prove how they work and why this won’t happen again.

Are we sticking with Amazon Web Services? Yes, but with some changes to our strategy going forward.  Cloud computing models are still the cheapest way to bring up services, platforms and infrastructure without a lot of upfront costs.  Our company could not be where it is now without AWS.  I figure that this failure will make Amazon Web Services better, and increase transparency.  This is not unlike some of the massive Internet service failures that were very common in the late 90’s, but as technology and platforms matured, and as we understood how to engineer better and stronger services, these outages have become very rarer.  Web sites and web-based services are more robust, and handle large volumes of traffic much better than they did 10 years ago.  Similarly, cloud computing will only get better and better.  The cost efficiencies alone are too great for it to go away.

What changes will we make to how we use Amazon Web Services?

1. We will be duplicating our services to other AWS regions (US West and US East),  not relying on availability zones within a region to take care of us.
2. We will be moving our system backups not only to S3, but now we will be moving them to an AWS competitor as well (RackSpace).  
3. I am going to run our team through a fire-drill.  How do we recover our services in another AWS region in case of a disaster?  How do we recover our services somewhere else completely? 

At the end of the day, for our company to leave AWS would require a large capital commitment to hardware and infrastructure we are not prepared or really able to take on. Besides, I don’t want to manage another large set of IT infrastructure anymore.  It sucks to do that.  I got spoiled by letting AWS do that for me,  and frankly I like it that way.  With some refinement to the way we do things, I am just happy to keep paying AWS to do that for me, and that allows me to concentrate on delivering a cool and solid application.

Using EC2 as a playground

One of the neatest things about an infrastructure-as-a-service cloud computing model (like Amazon's EC2), is that you can use it as a playground in order to play with services and operating systems before you make a commitment.  This kind of thing used to be really hard to do as little as 10 years ago.  You needed to buy or find old hardware,  then install an OS (plus patches) and then you could play with it, and then repeat for a new OS.

Using EC2 as a playground is particularly helpful for playing with all the Linux variants.  A micro-instace on EC2 only cost pennies an hour (or about $20 a month if you keep in running 24/7), and you can install all sorts of linux OS's on it.  In the last few weeks our team has tried out Ubuntu server, Fedora 14, CentOS, and the new Amazon Linux AMI.  Pretty cool.  Our beta systems are currently running on Fedora,  but at some point we are going to migrate to the Amazon Linux AMI.  We spent lots of time trying them out before deciding,  and we could do it on the cheap without having to worry about installs and such.

Testing out Amazon Web Service's platform products has been very neat also. Again, since you are only paying for what you use, it can be done economically.  We have tested and are currently integrating S3, but are also testing and considering integrating Elastic Load Balancing and RDS.

The interesting trend here is the commoditization of platform and infrastructure, which allows engineering teams to leverage the commodity pricing in order to build better and more flexible applications.  This trend also puts the value of a software product squarely back on the intellectual property, as it should be, and not on the infrastructure, where it does not belong - but that is a subject for another post.

On Doing the right thing

There are two truths about being a commercial software developer:

1) You are never really done

2) There is always someone who is not happy with you.

On the first item,  there never is really a time when you can put a bow on a product, frame it and hang it up, step back and admire and claim - "it is done!"  There are always more requirements to fulfill, refactoring to do,  defects to fix,  and crud to finish up.  Really.  You are never done.  You continuously fight architectural decay,  requirements burdens and general entropy.

The second item is what has been on my mind recently.  You really can't and don't make everyone happy in a software product.  There is always an opportunity to fault the defects, the process, the requirements gathering, the product management,  the user interface, the performance, the testing, the software architecture.  The bigger and more complicated the product, the more this is true.  

So what should motivate a commercial software development team?  Some teams or individuals are motivated by the coolness factor,  bragging rights.  Telling your buddies what cool algorithm, hack, stack or complexity you developed.  I know that as I look back on my career, I get a smile on my face when I think about some cool code I have slung in some very short amount of time that did some very neat things.  Bragging rights.  

Bragging rights are cool,  but those opportunities don't come very often as you are taking care of the 100th refactor, bug or UI tweak this month.

So, you know what conclusion I have come to over my career?  My motivation is to do the right thing, not please every person.  This means that when things aren't moving as quickly as you want, or you are faced with a problem, or client services wants something quick, I ask myself and my team "what is the right thing to do?"  Not, "what will make us look best?", or "what is easiest?".  If you focus on doing the right thing and not the political thing, or the thing that will make you look best, you develop pride in your work and this in turn gives you a reason to keep working hard and enjoy what you do.

It's not as easy as it sounds.  Doing the right thing means making a hard choice, or admitting you did wrong. It means courage to speak up and advocate, rather than being a passive participant.  It means being able to make a commitment and stick to it.  It means doing something you feel good about - not something that will make you look good in front of others.  Doing the right thing means being able to admit you may be wrong, and adopting a better practice that may exist in the industry or that someone else may by suggesting.

The great thing about doing the right thing is that you don't have to worry about who is happy with you or not, and you don't have to worry about office politics or petty things that creep into teams.  It means you can sleep good at night knowing you have done what you believe is the right thing.

Taking the Google Laptop for a Spin

So, for the last couple of months I have been beta-testing the new google laptop (Officially, Google CR-48 Chrome).  Engadget has a pretty good set of pictures and overview here.  

My 17 year old tech-savvy daughter's assessment? - "It's just the internet."  Well, that's kind of the point.

I have been a Macbook Pro user for the last several years,  and on the development side I favor unix-like OSs, so I have a bias.  I love what the google laptop represents and it's potential - an easy to use thin end-user device where everything you do and work on is saved automatically for you in Google's cloud.  This laptop is all-google-all-the-time: Chrome brower,  gmail, google docs, google calendar, etc...  If you are a power google user you will love the model.

The hardware itself leaves some to be desired.  I recognized that this is beta-model and not a final production laptop, but there are some things that make it wonky to use.  The speed and response is good enough for surfing the web and using all the various google products.  Video playback is OK, but at times choppy (underpowered graphics processor).  The keyboard is OK - not great.  The mouse pad is weird - there is some bug where suddenly it will select a bunch of text and delete or be too sensitive and jump the cursor around.  The built in Verizon 3G access is pretty nice.

Chrome OS itself is pretty solid.  It looks like the Chrome browser, with a simple settings tab.  The OS is very thin.  There is not much to mess-up, and given the complexity of modern desktop OSs with gajillion settings and what not, it is pretty cool.

Like I said, the power of this machine is in the potential and what it says about modern computing - simple end user devices,  cloud-based infrastructure storing your pictures, settings, documents and such, and the "post-pc" world.  If Google sets the price point of these right,  say $300-400, they could sell a bunch.

In the meantime, I will keep using it and downloading the patches to the OS.  Let's see how it continues to evolve.  

To Infinity and Beyond!

Man, I really dig cloud-based computing infrastructure.  It was made my life as CTO of a SaaS lots easier.  The ability to grow new infrastructure with out having to order equipment, service and wait for install has made us more agile.

We are currently using Amazon's EC2.  One of the neatest things for me is how this really enables smooth upgrades of our production software.  When we are ready to do a release, we grow new "candidate" systems - front end serves and databases.  The candidate systems are populated with the new code and "smoke" tested by QA and development.  When cut over time comes around,  we simply shutdown previous version front end servers, move data over, move the dynamic IP addresses to point to the new systems and we are off and running.   We keep around the old systems just in case,  and decommission them about a week later.

Very cool.