Incompetent People Really Have No Clue is an old article but is just as relevant today as it was 7 years ago.

My wife is a public high school science teacher and she will periodically ask the students to submit what they think that their grade is. Her findings match those of the article: poorer performing students tend to believe they are earning a grade significantly higher than is the case. This results in the odd implication that one cannot expect these students to contribute more effort to achieving a higher grade since they already believe that they're done what is necessary to achieve that higher grade. Without constant vigilance, one-on-one instruction, and continual family support to effectively force the student to exert the necessary effort to achieve the higher grade, the student will never know where the various rungs of the ladder lie and therefore will never know what subject mastery means.

I too have witnessed the same phenomenon working with developers over the years. Developers that produce code with the most logical errors that requires the majority of effort to maintain tend to be those that believe highly in their abilities. I have taken a few of these developers by the hand and walked them through a few full life-cycles of development demonstrating what is necessary to produce quality code (and to see the implications of poor quality code). Most of my efforts were greeted with incredulous responses and outright denial of the effort involved but there has been an individual or two that has gained more understanding and used that experience to take their craft to a new level.

The worst has finally happened ... I uttered that phrase without thought. It felt as though it eminated from my very being. We're through the looking glass and we're on the other side. Things are very strange here. May the lord have mercy on my soul.

I have talked a number of times in the past about the CMMI. David J. Anderson has written the paper Stretching Agile to fit CMMI Level 3. It's an interesting read. For more information on this you can check out MSF for CMMI Process Improvement as well as a webcast by Mr. Anderson.

When in a competitive business environment under the strains of the triple constraint (time, resources (cost) and scope), you have to minimize the risk and minimize the TCO (total cost of ownership) while maximizing the ROI (return on investment). But how do you determine the risk?

I commonly use a "bets" model for helping developers work with and understand risk. I ask: "What bets are you making to use this technology and what bets does the company make when using this technology?" It's common that developers will not be able to list the large bets. That's OK. If you're not familiar and accustomed to thinking in these terms then you can't be expected to do it well.

Let's try an example exercise using a fictional product called WonderWidgets. What are some of the bets that you're making if you use WonderWidgets in your project? (This list is by no means exhaustive. If you think of a bet that I'm missing, let me know.)

• Does WonderWidgets meet the needs (requirements) for your project? If your needs are not well known then this bet may become very large. Is there sufficient documentation (traditional docs, blog entries, forums, etc) for you to know if WonderWidgets meets your needs? All of this needs to be added up and a bet made.

  You can certainly divide this one big bet up into its constitutients if you feel more comfortable.

• Is there anyone on the project (or at the company) that's familiar with WonderWidgets? If not, big bet. This ropes in the "hit by a bus principle", the time to learn, the total learning curve, etc.

• Following along the same lines as the previous point: the company would be making a big bet by using a technology at the core of its application that is likely to be understood by only a single developer (i.e. the "bit by a bus principle").

• Are there working examples available (anywhere) that outline most of what you're trying to do with WonderWidgets? If not, big bet. This touches on some of the same points as the previous bets: how do you know if WonderWidgets can do what you need it to do? Having examples will help mitigate learning time and determining applicability.

  Notice that I didn't just say "are there examples of code using WonderWidgets". There are two common siutations that you have to be aware of:

  • Are there examples that run under the current version of WonderWidgets?
  • Are there examples that address your particular project's needs?

• Do you understand the impact that WonderWidgets will make on the rest of the software / architecture / team? If not, you're making a bet. This component has a lot of factors that come into play such as understanding how invasive WonderWidgets will be on the rest of the software.

• Take a look at the time between releases and the number of defects per release of WonderWidgets. If the time's too short then things are changing too fast to possibly keep up. If the time's too long then you may have a defect that wont be fixed for a long time. Same goes for defect rates. Bet accordingly.

• How active is the community around WonderWidgets? If there's a large active community with lots of people that can and do answer questions then this may help mitigate one of the previous bets. If not, you've got a bet.

• It's possible for open source software to just dissolve. You're always making a bit of a bet with these projects. (Yes, the same can be said for commercial software. And yes, there are ways to determine how big of a bet this is based on the size of the open source communitiy surrounding the project (ditto with commercial software)).

  For example, I was on a project where a bet was made on jStateMachine. The license changed and the developers got scooped up and are working elsewhere (or whatever actually happened). At the end of the day the team was stuck with a dead piece of software that was critical to the architecture. Luckily there were mitigating factors for this bet (i.e. an alternative).

I should point out that by no means is the list of bets static. It can and will change with time. You may add bets as they come up, remove some if they are proven false, and change the "size" of them (e.g. if the lead developer on an open source project that you're using was hired by a company that consumed most of their time, your bet for using the project just got bigger). Just keep a record of these bets. (No, this doesn't need to be some huge document that you need dedicated resources to maintain. It may be as simple as a "notepad" file that you keep on your desktop if in a small company / project.)

Looking at the bets, you should choose WonderWidgets based on your ability to find enough mitigating factors to make those bets tolerable. What's tolerable? It depends on the situation and the people involed. If you have a highly agile team (not necessarily in the XP sense of the word) then it may be able to adapt to a lost bet effectively in which case you can absorb and replace the loss more easily.

Just remember that you don't want to have too many components with large bets on them (mitigated or not). It may be fine to have a single large bet component if you have no other unmitigated bets. If you have a large number of components with bets (mitigated or not) ... look out! There may not be time for you to absorb losses from a number of lost bets. For example: You choose to use both jStateMachine and WonderWidgets. You mitigated each by having an alternative and by architecting the app such that you could "easily" place in the alternative. Both jStateMachine and WonderWidgets disappear (or have large bugs that are deemed "not important" or whatever). There's time and resources available to replace one but not the other. Now what? You made intolerable bets.

In order to keep teams happy and to keep up "innovation", I typically allow one large bet (with suitable mitigating factors) per project. This is tolerable for an average team.

A daily work-out is an important aspect of my life both physically and mentally. Throughout most of the year, the facility that I use for my work outs has a natural equilibrium; everyone knows each others cycles and there is rarely a collision where two people want to use the same machine at the same time.

There is one time of year that all consistent exercisers dread: the month after new years. January brings a swell of newcomers into the environment. These bright-eyed newcomers arrive headstrong with the resolution that "this year is the year of becoming fit". After no stretching or warming up whatsoever these individuals load up the weight machines and jerk and pull a few times until they look like they're going to pop and then move to the next machine. They then proceed to the various cardio machines and run / cycle as fast as they can for about thirty seconds until their heart reaches the near explosion point and then they leave (again, without stretching or warming down in any way).

Needless to say this injection of soon-to-be-buff induviduals reeks havok on the natural balance of the workout facility. All regulars recognize the motivation and lifetime of these people and some of us even try to help them prevent from killing themselves. But we all recognize that the number of resolutioners will dwindle to about half by the second week, half that again the third week, and by the start of February there may be a single new person left that becomes a regular.

But why do all of these people fail in their goal? It all boils down to two factors: they lack a detailed plan and they lack domain knowledge. If you would ask one of these individuals why they are there they would likely answer "to get fit". So it is clear that they have a goal (albeit quite loosely defined). If you ask them how do they plan on reaching their goal, they would likely say "I'm going to work out for 50 minutes at least three times a week". That's a good guideline to have but it doesn't actually tell them how to reach their goal. It's also lacking technical detail such as how much weight on which machines or how long with what resistance on which cardio machine. To compound this lack of planning is the lack of knowledge of physiology and the proper use of the available equipment.

This same phenomenon can be witnessed on most projects in low maturity organizations: a general goal is set; people attack the task with wild enthusiasm but little regard for how the goal will be met; within a week or so, half of the individuals are off-task or are already demotivated; in just over a month the first milestones have already been missed or are in the process of being missed; and shortly thereafter, management is engaged to bring the project back into line (the involvement of management introduces a quality cost called a transaction cost). If you ask any member of the team what the goal is, it is likely that they could answer correctly (though I'm always amazed to find people that do not even know the goal). If you ask them how they are going to achieve that goal you are likely to hear a very high level description (or, again, you may find individuals that don't actually know and yet are doing something). It's also common for people not to fully understand the tools available to them or how to best apply them to the task at hand.

Whether you are attempting to fulfill that new years resolution or preparing your next project ensure that you have the basic elements in hand: a detailed plan (including ways to montior if you are on course) and enough knowledge of the environment in which you are working.

This is an interesting take on the software development life-cycle (SDLC) for low maturity organizations:

1. Project Initiation
2. Wild Enthusiasm
3. Disillusionment
4. Chaos
5. Search for the Guilty
6. Punishment of the Innocent
7. Promotion of Non-Participants
8. Definition of Requirements

I have witnessed this exact series of steps acted out in grueling detail at a number of organizations. My personal favorite is the punch line of finally defining requirements.

What really makes you think is that it is believed that defining requirements is too time consuming. I have even heard "Why should we define requirements? They're just going to change and we'll have to update all of the docs!"

For me, software development is largely risk management. Not having requirements against which development and testing occurs is a significant risk. Most immature organizations are able to survive due to the presence of a hero which characterizes them as CMM level 1. This hero has enough knowledge of the desired end result that she effectively embodies the requirements. If this hero leaves or if there is an attempt to scale up development beyond the means of the hero, all begins to disintegrate. Identifying the risk associated with requirements and properly understanding its impact is paramount to a successful project.

I have written about the CMMI a number of times. I have always believed that there should have been a level zero -- a step below ad-hoc and chaotic where the nastiest of the nasty occurs.

Capt. Thomas M. Schorsch has gone that extra step and defined a Capability ImMaturity Model (CIMM). The additional levels that he has added are characterized by:

Over the years I have had much experience with the trials and tribulations of forming a company and the impact that this very important yet much overlooked step has on its well-being. My experience comes from forming my own companies (with multiple partners), various M&As, and from working from the VC's side of the table.

I often equate a business partnership with marriage. One marriage site has their top four reasons for a failed marriage as: poor communication, financial problems, a lack of commitment, and a dramatic change in priorities. These four reasons can be easily translated over to a business partnership. How many marriages last beyond one year or five years these days? We've all heard horror stories about a divorced couple fighting over possession of assets. Imagine the difficulty in dividing up a business partnership.

It is common, especially when forming your first company, to think that "everything will just work out" and that any partners that you're forming with will behave rationally and inline with your desires when necessary. This isn't always the case. Rather than rushing into a partnership, take the time to fully vet out all concerns, responsibilities, and expectations. Write everything down so that there is never the case of "well you said".

There are a number of questionnaires available online (which, of course, I cannot find at this time) that all partners should answer and then sit down and discuss. This allows all involved parties to see how well their visions for the company coincide. The questions might include:

• Where will starting capital come from and what assets are you willing to bring in?
• Where is the priority of the company in your life? (An important question in here would be in relation to existing or future marriage and/or children.)
• How will the interests of the company be divided? (It's not always equal between partners as each partner may not bring equal amounts to the table.)
• How are decisions made and, more importantly, how are disagreements handled? (This is very important in 50/50 partnerships. What happens if you both disagree? And yes, it will happen.)
• Is the goal to build the company and then sell (short term) it or grow it over many years (long term)?
• How do you decide when to throw in the towel? (Planning for success requires that you plan for failures.)
• How does a partner leave the company and what ramifications are there?

Don't jump head first into a new partnership. The time that you take in the beginning will more than amply pay itself off with the headaches it will prevent later. Take as few chances as possible and understand what you are getting into. Shop around and find an attorney and an accountant that all the partners like. They can be invaluable in helping you over all of the startup hurdles.

As mentioned in another entry the classic tradeoff is: features vs. quality. A further refinement of this is: do you release a high priority feature even if it is not complete?

It is common for a project bend to the pressures of client expectation and release a feature that may not be complete or is known to have defects. The thought that leads to this is that it is better to meet the expectation of the feature and deal with the quality risks later than it is to miss expectation. In my experience, the reality is the exact opposite. Releasing a feature with defects lowers the overall expectation of the product to a level much lower than that of a single missing feature.

Let's take a look at this in more detail.

Releasing a product with a missing feature defines a single point of contention (i.e. the missing feature). The client will have a focused attack on the omission that tends not to spread to other features (unless of course there are other missing features). I have no theories as to why an omission does not spread -- I only have experience that says that it does not impact the preception of the rest of the product.

Releasing a product with a defect heavy feature tends to lower the overall perception of the product. I believe that this follows a "If this feature has this many problems, then how bad is the rest of it?" progression. On top of this is the onslaught of defect reports (most of which are already known internally to the developers) that accompanies the feature. The amount of man power required to deflect or stop the barrage is sometimes greater than that required to simply fix the bugs. (Developers may be thinking "Just fix the bugs and ignore the client's bug reports. Once the fixed version comes out all of the bug reports can be cleaned up en masse." Unfortunately, it just doesn't work this way unless the fix comes out a very short time thereafter. In most situations, the client hand-holding and expectation resetting combined with handling of the defect reports consumes needed resources.) I have seen projects where this barrage eventually stalls all progress and the entire team ends up in defect triage, changing focus from strategic to immediate tactical.

My recommendation in this situation is to take the omission hit and follow it up with a well planned and executed (and, if possible, ahead of expectated delivery) patch / release that includes the missing feature. If you botch the followup release, well, let's just say that that tends to be worse than had the defective feature been released in the first place.

Let me begin by saying that I have a lot of respect for the Eclipse team. They have maintained tight milestones on a massive project. They have remained responsive to bug reports and feature requests. I could go on for quite a while. Much can be learned from that project.

Unfortunately, they stumbled on the classic tradeoff and that has lead to a classic mistake.

The classic tradeoff is: features vs. quailty. Do you spend more time fixing defects or creating new features? The classic mistake is: releasing a feature with a high defect rate.

Classic mistake

Notice that I said defect rate and not defect count. It may be that a feature has zero defects when it is shipped but in the recent past the number of defects found and resolved was high. Understanding the difference goes back to a firm understanding of QA and a little math. The defect rate is the number of defects found per unit time. The unit of time is commonly determined by the frequency and duration of test / fix cycles. A large defect rate means that there were many defects found in a cycle whereas a small defect rate means that there were few defects found. The defect rate for a complex feature typically increases then decreases. This trend follows the common sense rule that a single test cycle will not reveal all defects. In fact, in early test cycles, defects will block other defects from being seen. As fixes are put in place, the number of defects will eventually decrease.

For those more savvy, the change rate of the defect rate (a defect acceleration of sorts) is also useful. A positive acceleration means that there are more defects found in a cycle than previously. A negative acceleration means that less defects were found than previously. And a zero acceleration does not mean that zero defects were found -- it means that the same number of defects were found and were found previously.

Releasing a feature while the defect acceleration is positive (or zero with large defect count) or while the defect rate is still large will typically result in a feature with a high defect rate. In effect what you're doing is replacing a testing cycle with your client. This is the classic mistake.

Classic tradeoff

If there is one thing that clients are exceptionally bad at it would be understanding that development takes a finite amount of time (and typically longer than they expect). After the process of requesting and specifying a feature has finished and while development is occurring, the client is already thinking about the next set of features. By the time that development releases the features, the client already has expectations beyond that of the release. This results in development always trailing or missing expectations. (If you're a developer reading this, you've probably thought Well, duh!. Trust me, clients really don't understand the basis of these skewed expectations.)

What I've just described always occurs. It's the background noise that developers and project managers have to deal with. Deciding between missing features or having defects while being affected by the background noise is the classic tradeoff.

Eclipse tradeoff and mistake

Late in development of Eclipse 3.0, code folding was introduced. There was a high defect rate (and a positive defect acceleration) associated with the feature. It interacted with a significant number of other features and is highly complex (this adds up to a high quality risk). Lo and behold, code folding is significantly buggy.

Should code folding, a much asked for feature, have been included in the release? Check back for a follow up entry on this subject.

More times than I can count, I have heard someone say "We're trying for XX percent code coverage with our unit tests". If XX is 100% are you guaranteed to have no defects?

The first thing that comes to my mind when people start rambing on about unit tests is: What about integration and system testing? There are a number of blog entries out there that talk about the difference between the types of testing. It suffices to say that pure unit testing, even at 100% code coverage, is only going to reveal a small percentage of an application's defects.

So what are you actually trying to do?

Test for a reason -- don't just test to test. You can get into a cycle where you're writing unit test apon unit test and still releasing applications that are defect riddled. First you need to define a quality metric (a way to measure the quality of the application). "As few bugs as possible" isn't it! Ask yourself questions such as: What's a bug or defect? and How is the priority and severity of a defect determined? (e.g. Are spelling errors bugs? They may not be to a developer, but they may be show stoppers to a client.) Everyone wants to jump in and test up the wazoo but the problem is, if you test the wrong things and you don't know what a bug or defect is then your testing is potentially wasting time.

Let's put this another way: let's say you have 80% coverage with all of your tests. What's preventing that 20% that you're not testing from containing 90% of the defects and 100% of the P1 show-stoppers?

Your goal is to identify the cross product of the areas of highest quality risks and the areas that are most important to the user of the application. These areas must have enough testing to ensure that the desired level of quality is met. For example, what if the 20% that you did not test just so happens to be the login page? The client cannot log into the application! So what's the point of testing or even writing the rest of the app? This is an obvious overtrivialization as the login page is an obvious element to test. But in reality, the show stoppers commonly fall into a trivial category. A number of projects that I've performed triage in the past suffered from this problem to an alarming degree. Months of work and testing when into components of the product that, at the end of the day, were not high on the clients list and were overshadowed by trivial defects.

Plan ahead. Identify what determines quality in your application. Test effectively.

It is a common belief in the industry that programmers are just interchangable cogs. "Programming is programming", right? I believe that as SOAs, MDAs, RTIs and other "standard" architectures become more prevalent that this "interchangable cog" belief will initially become stronger due to perceived simplification of tasks. Unfortunately, this is exactly opposite from what the reality is.

As the nuances, scope and concerns of a specific architecture grows, developers become more and more specialized in that architecture. Specialization limits ones general knowledge (or just pushes it further down the stack) and therefore limits interchangability. This specialization trend can be seen in regards to development languages, platforms and applications. For example, interchanging a C developer fluent on Unix with one fluent on Windows is a recipe for disaster. The same can be true for Java and C#, or PeopleSoft and SAP.

I have recently tasked myself with determining how our universities are meeting the demands of greater specialization and addressing this interchangable cog paradox. At first glance it appears that there is little being done but I am only at an initial phase of my research. In a recent conversation that I had with a professor at my alma mater, I learned that most universities purposefully have more generic and abstract courses to separate themselves from trade or vocational schools.

I'm a man on a mission and I will post more information as I acquire it.

If you're unsure of what the CMMI is, here is a nice introduction.

Most managers I encounter are caught up in the tactical day-to-day activities of management: finish the project plan; prod so-and-so to finish the thing-a-ma-bob; update the timelines; etc. When asked what they are attempting to achieve, most respond tactically: get the product out on time; reduce the quality risks; etc.

Where are the strategic components to all of this? If the managers are only focused on the tactical and the developers are certainly focused on the tactical, who's focused on the strategic? The upper management? Probably not. They're typically heads down in the tactial morass too; reduce the bottom line; increase profits for the board; get better products out; and so on.

(Quick disclaimer: of course there are companies out there that have a strong strategic component to them. But what fun would it be to talk about them? *grin*)

My bother-in-law recently finished his MBA at a top-notch university. Given my interests in management, we have had a number of lively discussions about both the strategic and tactical sides of management. I would relate a number of our discussions back to the CMMI as a blanket way of understanding how the topics would relate to one another. Much to my surpise, he had never heard of the CMMI.

To me, the CMMI is a roadmap or set of guide posts against which you can gauge progress. It presents the over-arching strategic goals that the day-to-day tactical tasks should be geared towards. Without the vision that it provides, it's nearly impossible to judge the effect and quality of the processes that are employed.

If our management producing machines (i.e. universities) are not presenting a strategic road map, how can we expect our companies to extend beyond the tactical?

A thank you goes out to Christoph Cemper for the link to the "What is the CMMI?" article.

Lately I have been preparing the engineering infrastructure for a technology startup. I have been liberally sprinking QA across this infrastructure (remember: QA is a process).

For anyone that is in need of an excellent reference on QA, I recommend: Software Quality Assurance: From theory to implementation by Daniel Galin. It is well worth the heafty price tag.

It has been my experience that most people respond to change as they do to loss or grieving. Elisabeth Kubler-Ross first identified the five stages of grieving: denial, bargaining, anger, despair, and acceptance. (These five stages have been humorously portrayed in the Simpsons.)

Effectively managing a team through change requires recognizing the reaction and guiding individuals though the steps as they occur. In most scenarios providing adequate time and an having an "open door policy" where people are encouraged to discuss their feelings is sufficient. For the more difficult cases, human resources may need to be brought in.