Software Development: Moving from Art to Science

Have you read Moneyball? The book is about how Billy Beane, the general manager of the Oakland Athletics, revolutionized the way we evaluate baseball players in order to create better teams with less money (maximize quality while minimize cost). The premise is that the collective wisdom in baseball is based on antiquated statistics that were defined a hundred years ago based on what was easy to measure rather than what could be measured; baseball management was living in the 19th century. Billy Beane believed that by using 21st century statistical analysis, he could create a far better team with far less money. When he put his ideas to the test, his team became one of the best in baseball with one of the lowest team salaries in the game.

I believe in software development, we are still living with antiquated and immature metrics and that there has to be a better way. Let me give you an example. Take project estimation and staffing. In a professional services firm you often have to bid on projects without any idea of who will be on the development team if you win the bid. So what do you do? You decompose the project into discrete tasks, assign time estimates for task completion, do a resource leveling and determine project duration and cost all while using some notion of an 'average' developer. Weeks or months go by before you are notified that you have won the bid and then you begin staffing the project. Here is where the challenge begins.

When you did your task estimates, you assumed you would use an average developer but what metric(s) did you use for the classification? Assume for a minute that you have a team that specializes in the technology you bid, is an average developer a Junior Developer or a Senior Developer? We all know that there is a great range in developer skills yet how do we rank these skills? Is a developer a 10 or a 1 and what would a 10 or a 1 mean? If you bid the project for an average person, is average a 5, a 7 or an 8? Does it take into account quality, clarity and supportability of code or the maturity of the developer?

Actually, how we typically judge a developer is highly subjective, it is with titles like Junior Developer, Developer or Senior Developer. Within each job class, we may have salary bands and the bands may reflect relative skill level or they may reflect different managerial judgment. What is needed is a more accurate measure of a developers skill set. What is needed is a productivity rank for each developer.

Let me walk you through an albeit contrived example to show you the complexity of the problem and the need for a better solution. Let's assume that you have devised a way to grade a developer and have determined that there are 10 different productivity levels. I've often heard that a great developer could be as much as 10 times more efficient than a bad developer, so let's assume that the levels are relatively incremental and that an average developer is defined as a 5. This looks something like:


Now, let's overlay on top of this a typical classification scheme ranging from a junior developer to a senior architect. I'll add a second Y axis for costs an will add in a notional curve for salary and billing rates.


So for a constant level of effort for a project, you can then staff the project in many ways. For example:


Here, the base resource, ranging in capabilities from a 4 to an 8, is augmented with additional staff to complete the work. Mix 1 adds four additional junior resources (rated 3) while Mix 2 leverages five trainees (level 1's), and two junior resources (level 2 and level 4). Mix 3 suggest that the effort could be completed by 2 slightly better than average developers (level 6), while Mix 4 and 5 look at using more senior resources teamed with a junior resource. As the different mixes imply, there are many different ways to staff the same level of effort. A challenge for project managers is that the task breakdown likely lists this effort as 2.5 FTE (Full Time Equivalent). Without knowing the skill set of the staff, 2.5 is a meaningless number. If you put 3 resources on to the project who were all 3's you would not have enough ability to complete the project. Of course, we typically blame the manager in this situation for clearly (s)he had a enough resources (3 being great than 2.5) to deliver on schedule.

Without knowing the productivity factor for your developers, it is dubious to say a task is 2.5FTE and even harder to staff it simply as 2.5 FTE. It is even harder to assure you have the right mix to optimize your gross margins vs. code quality. In the above example, I've put some relative numbers to the task based on the graph:


For this cost structure, I've created a nominal Fixed Bid as the median of all the bid prices. Gross margin is calculated as a percentage of the bid price. I've also computed relative average bill rates and total man hours (as a percent of the 'standard' which I've used as Mix 3).


For the same project and the same level of effort, you can have a 4x increase in man hours and average bill rates. Gross margins are optimal with senior resources though as Mix 4 indicates, there are tradeoffs that help with average bill rates and man hours vs. gross margin.

Though it pains me to say it, software development is still more art than science. Without accurate productivity ratings for each developer, the industry will continually struggle to fill projects defined by FTE.

How does your company resolve this? What metrics do you use to ensure accurate staffing?