By Djindo Lee, Theron Leishman, Deepti Sharma and Nishi Narula


Introduction

The idea of agility has significantly different connotations depending on personal background and frame of reference. When considering what represents agility, ideas seem to reflect personal interests and perspectives. To an avid hunter, agility may be reflected in the behavior of a deer – quick, nimble, sneaky, allusive, able to quickly change direction. To a cat lover, agility may be reflected in their cat – sneaky, quiet, able to get into tight spaces with relative ease, ability to jump, twist, turn, and hang-on. To a dancer – flexible, flowing, smooth, ability to jump, spin, and twirl. To an NBA fan – a basketball star with speed, strength, the ability to cut, twist, jump, and fake to avoid defenders.

In the world of software development, the discussion of agility can conjure up similar thoughts. We have all likely experienced projects that profess to be following an agile approach when in reality they are allusively sneaky, quick to change direction, jump through hoops to report progress, claw, scratch, and hang-on to deliver something half-baked, not documented and with no responsibility for the final product.

By contrast, mentioning CMMI often brings to mind extremely large projects burdened by detailed processes, plans, procedures, and volumes of documentation and measures that drag on forever resulting in high risk of failure or a product that is not usable.

In reality, both agile and traditional CMMI based approaches have proven effective in delivering high quality products to customers. The question may be asked, is it possible for an organization to use the concepts and principles of the CMMI high maturity process areas to proactively improve agile processes? Is it possible to have agile processes that provide predictable results?

In this article we will share a case-study of our experience combining Scrum and CMMI Level 5, what we refer to as “Agile 5”, to significantly improve the delivery rate of product capabilities while maintaining a high level of quality and employee satisfaction. We will also share the pain experienced during our journey of blending these industry best practices and the lessons we learned.

The Situation

During a period of tight and reducing budgets, sequestration, and threat of government shutdown, one of our large projects was faced with the challenge of continuing to provide essential aviation systems in a safe and efficient manner. The project was following an agile development lifecycle based on a Scrum process. As a CMMI Level 5 organization, Optimal Solutions and Technologies Incorporated (OST Inc.) has extensive experience using CMMI high maturity practices and principles to improve the effectiveness of processes and achieve business goals and objectives. With the intent of proactively improving our agile process and providing greater value to our client, we embarked upon an effort to improve our agile process using the practices of CMMI Level 5. The goal, based on these objectives, was to increase our feature delivery rate to better meet the demand for system updates while maintaining a high level of quality as measured by production defect density, and promote a high level of team satisfaction.

The Pain

However worthy our objectives were, our journey to improve our agile process using CMMI Level 5 was not without struggle, challenge, and pain. Listed below are the major areas of pain we experienced as we attempted to use CMMI to improve our agile process.

How should we proceed? Although we had significant experience using the high maturity processes of CMMI to gain improvement in our traditional processes, methods, and lifecycle, applying them to an agile process was a new experience. We had established process performance baselines and performance models based on traditional methods to provide insight into our supporting processes. The usefulness of these baselines and models in an agile environment was a question that we had to consider.

Agile – does it lend itself to data collection? We had a significant amount of data that had been collected, analyzed, and used to manage our traditional processes. We were uncertain if our agile process provided the opportunity to gather significant and useful data. Confusion as to what measurement data to collect, where to gather it from, what analysis should be performed on the data, what story the data was telling us, and how to use the data in an effective manner were also challenging.

Sacred Cows: The dictionary defines a “Sacred Cow” as an individual, organization, institution, etc., considered to be exempt from criticism or questioning. Figuratively, anything that is beyond criticism may be considered a sacred cow. In our organization we had many Sacred Cows. Existing processes, measures, and culture were so engrained into the organization that no one questioned their value. Our peer review process is an example of a sacred cow. Our traditional lifecycle plans and processes dictated what a peer review was, how it was conducted, who should attend, what a defect was, and what data about a peer review should be gathered, analyzed, and used. The Scrum process did not lend itself well to our traditional definition and approach. We had to be open to thinking outside of our traditional way of doing things; to think of principle/intent, not existing practice/implementation, and consider different ways of accomplishing things.

CMMI –vs- Agile: A question that we continually faced was, do CMMI and Agile even work together? Our effort to actually blend the two caused us to come back to this question time and again. We had to ask serious questions such as: Do we still need CMMI Level 5? Will this add value? What models will we use? Do we throw away our existing models? Will we be able to blend seemingly opposing methodologies of Agile and CMMI Level 5?

The Approach

In an agile environment, clear and distinct separation between lifecycle phases does not exist as in traditional methodologies. This made modelling individual lifecycle phases challenging or impossible. We determined our best approach was to use Discrete Event Simulation (DES) to model our entire sprint process because it provided a holistic approach to capturing and depicting the sprint lifecycle. Our DES model allowed us to account for the entire lifecycle from user stories (requirements), design, development, and test within individual sprints. Baselines of the functional activities of business analysts, developers, and testers were used to populate the model with actual data from project sprints.

Based on these performance baselines, the model predicted the number of story points that could be completed in a given release and/or sprint. With each sprint, additional data was gathered and incorporated into our performance baselines which allowed us to further calibrate and refine the model. The Causal Analysis and Resolution (CAR) process was incorporated into the sprint retrospective to help refine and improve our process. The model allows various factors to be entered, which are used to statistically predict sprint outcomes. The factors include: development time, test time, test case development time, defect density, number of user stories, number of story points, and resource availability. Using this model during release and sprint planning, we were able to predict the number of story points likely to be completed during each sprint and release. By adjusting the factors, we were able to conduct a hypothetical/what-if analysis to determine the optimal team make up and dynamic to achieve results consistent with business goals and objectives. Our DES model allowed us to model wait times and bottlenecks and to change the controllable factors to evaluate what-if scenarios for downstream impacts. Figure 1 below depicts how the model was used.

The model also supported project planning, project monitoring, and control and risk management activities by guiding resource planning, determining if the velocity was improving over time, and adjusting resource levels based on the constraints.

The SEI has defined a valuable list of “Heathy Ingredients” of CMMI-based process performance models [1]. We use this list to ensure that our models adhere to modelling best practices. Table 1 maps our DES model to these ingredients.

The Result

Although the journey was not a simple one, we are pleased to report that our results were well worth the effort. By applying CMMI high maturity principles to our sprint process we were able to achieve the following:

  • Refined and improved our sprint process and exceeded our sprint velocity improvement goal
  • Developed a predictive model that allowed us to more effectively plan and manage Sprint and Release activities
  • Satisfied 20 percent more requirements per release than originally expected
  • Maintained a high level of quality as measured by production defect density of only three percent
  • Realized productivity gain of $600k over three releases
  • Fostered a high level of team satisfaction due to increased sense of accomplishment

Lessons Learned

In addition to our improvement effort yielding tremendous results to us and our client, we also learned many lessons throughout the process. Those deemed most valuable are shared below.

95 percent Analysis Vs 5 percent Modelling: Accurate data is essential to quantitatively understanding process performance. The effort to gather, understand, analyze, and scrub data to ensure it has integrity can be labor intensive, challenging, and time consuming. It is also imperative that the data being used to provide quantitative insight into processes that support the accomplishment of identified business and quality goals and objects is the right data. We found it useful to ask, why are we measuring the things we measure? There needs to be solid correlation between the project goals and objectives, the critical processes that support the goal, and the things being measured to provide insight into the process.

Consistency in data recording and accounting is also key to the ongoing integrity of performance baselines and models. Attention to detail in this area helped to ensure proper calibration and effectiveness of models.

Giving attention to and planning for the 95 percent effort required for good analysis resulted in significant value in the 5 percent modeling (the fun part) and gaining value from our predictive models.

Have a Plan: We treated this improvement effort like a project. We established a plan to guide our efforts, monitored progress consistently, and involved the right people in a forum of open and honest communication. Appropriate guidance was also key to our plan. We involved an experienced and trusted lead appraiser in our effort to provide insight throughout the journey.

Capitalize on the “Power of Failure”: The power of failure arises when we are not afraid of failure. This power comes as we make decisions based on the best information we have at the time and moving forward. Sometimes these decisions will not have the desired result, but do provide valuable data that can add to our knowledge base and help us make better decisions going forward. At OST, Inc., this principle has proven very valuable. As we began our initiative to use measures to guide business decisions we found in some cases, we were either measuring the wrong thing or using the analysis of the data inappropriately. As we began to develop process performance baselines and models, we found that sometimes they did not provide valuable insight into our processes and toward goal achievement. Rather than give up, we learned from these experiences and continued to move forward. The adage, “If at first you do not succeed, try, try again” should be engrained in a good process improvement culture. As we developed a culture where employees were not afraid to move forward and make decisions due to fear of failure and retribution, we have been able to proactively learn from mistakes and have seen great value from these lessons.

Real value in blending Agile and CMMI Level 5: The practices of CMMI Level 5 added more value to our agile implementation than we saw with our traditional methods. There are those who say that CMMI and agile do not work and play well together. We found the opposite to be the case. A major premise of agile methods like Scrum is to have a process that cycles quickly, providing working features and functionality to the customer on regular intervals. These quick cycles provide more data, which can in turn provide greater insight into the processes that support accomplishment of business objectives.

Using this premise, we were able to achieve significant improvement in our agile process by using predictive models, based on good baseline data. By incorporating the practices of CAR, we were able drive improvements into the process and/or capitalize on good things happening within the process, to significantly improve productivity, customer satisfaction, and goal achievement.

Conclusion

Just as an experienced hunter, cat owner, dancer, or basketball team are able to predict to some degree of probability the behavior of their agile subject; so agile development activities via CMMI Level 5 practices can be analyzed, modeled, and improved to provide predictable outcomes. Although agile methods and CMMI are often perceived to be opposite ends of the spectrum, we found that our agile process actually worked more effectively and matured faster when coupled with the high maturity CMMI practices. The frequency with which agile processes cycle results in the generation of data, actually lends itself to the establishment of performance baselines, better modelling, and predictive results better than traditional methods. The bottom line is, OST Inc.’s Agile5 leverages the discipline, comprehensiveness, and sustainment focus of CMMI with the build, speed, and lean focus of agile to bring the best of both worlds to our clients!

Figures and Tables:

Figure 1. Sprint/Release Planning Model ( Click to view image )

Table 1 ( Click to view image )


References and Notes

1. R. W. Stoddard and D. Goldenson, “Approaches to Process Performance Modeling: A Summary from the SEI Series of Workshops on CMMI High Maturity Measurement and Analysis,” Carnegie Mellon Software Engineering Institute CMU/SEI-2009-TR-021, 2010. [Online]. http://www.sei.mcu.edu/library/abstracts/reports/09tr021.cfm


Deepti Sharma

Click to view image

Ms. Deepti Sharma leads the Strategy and Business Performance Group at OST. This includes Corporate Strategy, Certifications, Data Analytics and Business Performance functions. She has led OST’s CMMI L5 and ISO certifications. Her client portfolio includes FAA, DHS, HUD, DoD and Treasury. She has been a speaker at several conferences including SEPG, NDIA and SEI’s High-Maturity Measurement & Analysis Workshops. She has a Master’s Degree in Engineering from Cornell University.

Nishi Narula

Click to view image

Ms. Nishi Narula has over 20 years of experience supporting government agencies in the areas of software development, strategy formulation, change management, governance and process improvement. As OST’s Director of Human Capital Management, Strategy and Business Performance, she blends best practices, quality and performance metrics to create innovative improvements. She has presented at multiple conferences including SEPG, NDIA and SEI’s High-Maturity Measurement & Analysis Workshops. She holds Master’s degrees in Operations Research and Computer Science.

Djindo Lee

Click to view image

Mr. Djindo Lee is an Account Manager currently overseeing OST’s DoD portfolio. He leads the delivery of high end systems and fosters an innovation-oriented, high performance work environment that marries emphasis on creativity, collaboration and end results. Mr. Lee puts into practice solution based on industry best practices such as CMMI, Agile, ITIL and ISO to achieve stellar product quality and development efficiencies.

Theron Leishman

Click to view image

Theron R. Leishman is OST’s Quality Assurance/Process Improvement Manager supporting the F-16 Program Office at Hill Air Force Base, Utah. Mr. Leishman has significant experience coaching and mentoring clients to leverage industry best practices to ensure quality and improve processes for the achievement of optimal results. He has a master’s degree from the University of Phoenix, is a certified Scrum Master, an As9100 Lead Auditor and an authorized CMMI-DEV/CMMI-SVC instructor.


« Previous Next »