Pay to Play?
The capability maturity model (CMM) was born in the 1980s out of U.S. Air Force and Department of Defense frustration with purchasing the right software. Carnegie Mellon University won the bid to create an organization called the Software Engineering Institute (SEI) to help improve the process of choosing the right software and vendor. In 1986, Watts Humphrey, a former software development chief at IBM, was brought in to lead this effort. As Humphrey explains, "We were focused on identifying competent people, but we saw that all
of the projects the Air Force had were in trouble. It didn't matter who they had doing the work. So we said let's focus on improving the work rather than just the process."
The initial CMM developed in 1987 was a questionnaire designed to identify good software processes practiced by the software vendors bidding on the contracts. Unfortunately, many of these companies learned how to "work the system" in terms of filling out the questionnaire regardless of the quality of their software practices.
In 1991, the SEI refined the CMM to overcome these abuses. A more detailed model of software development best practices was created along with a group of lead appraisers trained and authorized by the SEI to audit and verify that the software vendors were following the practices that they said they were doing. The lead appraisers led a team within the organization being reviewed and attempted to verify that the vendor was implementing the policies, procedures, and practices outlined in the CMM across a "representative" subgroup (about 10 percent to 30 percent) of all of the company's software projects. Over a period of one to three weeks, a number of confidential interviews with project managers and developers were conducted to confirm what was really going on. Since the lead appraiser led a team of the software vendor's own employees, there often could be a conflict of interest for these people to tell the truth. As one anonymous lead appraiser recalled, "It can be very stressful for the internal team. They have conflicting objectives. They need to be objective, but the organization wants to be assessed at a certain level."
.Another lead appraiser, David Constant, also recalls a situation where the software developers were coached by management to say the right things. According to Constant, "I had to stop the interviews and demand to see people on an ad hoc basis, telling the company who wanted to speak to just before each interview began. And the sad part was that they didn't need to coach anybody. They would have easily gotten the level they were looking for anyway. They were very good."
As of 1994, the newer model became more difficult to exploit. Mark Martak of Westinghouse told his management that "This is a different ball game now. If you have a good lead appraiser, you can't fake it out." He was able to lead his group to a Level 4 assessment.
It is widely believed that moving up the CMM levels will allow an organization to better serve its customers. However, a higher CMM level does not guarantee the effectiveness of performance, only that the organization has processes in place for managing and monitoring software development that organizations on a lower level do not yet have. As Jay Douglas, director of business development at the SEI, points out "Having a higher maturity level significantly reduces the risk over hiring a company with a lower level, but it does not guarantee anything.You can be a Level 5 organization that produces software that might be garbage."
CMM can be costly and time consuming for an organization. On average, it takes about seven years for an organization to move from Level 1 up to Level 5. The cost for a CMM assessment alone can be around $100,000, and this cost doesn't even include the expense and disruption of developing repeatable software processes and the training needed to disseminate them throughout the organization.
Therefore, it's not uncommon for organizations to make false claims. Ron Radice, a lead appraiser and former SEI official, recalls a Chicago-based company that was deceived by an offshore company that falsely claimed to have a CMM rating. Not wanting to name the
guilty party, Radice said "They said they were Level 4, but in fact they had never been assessed." In addition, some appraisers may feel pressured in their assessment. For example, Frank Koch, a lead appraiser with a software services consulting firm called Process Strategies, Inc., said that some Chinese consulting firms he worked with promised their clients that they had a certain CMM level and then expected that he would just give it to them.
According to Koch, "We don't do work with certain [consultancies in China] because their motives are a whole lot less than wholesome. They'd say we're certain [certain clients] are a Level 2 or 3 and that's unreasonable, to say nothing of unethical. The term is called selling a rating."
Given the investment in time and resources to achieve a CMM rating, it's not uncommon for organizations "pay to play" or bribe appraisers for a particular rating. Many government agencies in the United States require companies who bid for their business to obtain at least a Level 3 rating, while many CIOs use a CMM in choosing an offshore provider. Will Hayes, a quality manager at SEI, acknowledged one case where an appraiser had his license revoked for improperly awarding a Level 4 rating. The difficulty in knowing whether an organization's claim to a CMM rating is that the SEI does not monitor the organizations that claim to have a CMM rating, nor do they release any information regarding which organizations were assessed or the outcome of their assessment. As Hayes points out, "We weren't chartered to be policemen.We're a research and development group." Moreover, SEI does not have any intention of becoming a governing body like the American National Standards Institute (ANSI), which governs ISO certification in the United States. As Watts Humphrey contends, "No one has asked us to become a governing body,and that's not our mandate. And if we did, what would we solve? It wouldn't excuse anyone from doing their
homework."
1. What is the value of having a CMM rating?
2. As a project manager looking to outsource the programming of your project overseas to a software house claiming a Level 5 rating, what could you do as part of a due diligence to make sure that the claim is not false?