By Roger Stewart


Abstract.

Technology exists today that can make a huge improvement minimizing risk along the supply chains and improve delivery of secure, high-quality products, on time and within budget. And no changes to standards, legislation, or acquisition models are needed. Accepting this approach to Supply Chain Risk Management (SCRM) by industry and government means adopting policies to:

1. Impose contract stipulations on the prime contractor, such as common tools and process use, that must also apply to all vendors along their supply chain, and their vendor’s supply chains,

2. Require a common, standardized platform of: tools, process and training along the supply chains for consistently performing ongoing product risk assessments through defect identification and removal on pre-code product definition artifacts (e.g., contracts, requirements, architecture, design, interfaces), where past studies report more than 70% of product defects historically originate [1].

3. Require results of each product risk assessment be made available to both the prime contractor and the government program manager in the form of an automated, tool-generated report with common format and content.

4. Include contract leverage for the government program office based upon their ongoing evaluations of the resulting product risk visibility.

5. Use a tool-enabled, standard-compliant inspection process for defect identification and removal in Product Definition artifacts along the supply chain and for achieving the ongoing product risk assessments and reports.

Introduction

The supply chains in today’s software acquisition world consist of a wide variety of suppliers spread across the world (Figure 1). Each of these suppliers may have their own standards for development and quality assurance. Therefore, the responsibility for software assurance must be shared not only by software suppliers in the supply chain but also by the acquirer in the supply chain who purchase the software [2].

A key to advancing SCRM is through standardized inspections using tool-enabling technology for correction of past inspection perceptions and shortfalls.

The first 4 of the 5 proposed policies (see Abstract) are contract related actions that are dependent upon the viability of standardized inspections to provide sustained results in removing pre-code defects and reporting on the associated product risk assessments inspections can provide. This article will explore the viability of incorporating standardized inspections along the supply chain (Figure 2) for visibility into pre-code product definition activities.

Note: Federal Acquisition Regulation Part 46.202 on quality assurance currently addresses inspection on government contracts [3].

Inspections are a preventative systematic analysis of a work-product or portion thereof, by a team of three to five peer stakeholders to remove defects at or closest to their points of introduction. Inspections provide the best value when applied to up-front, pre-code product definition artifacts where more than 70% of product defects are historically introduced [1], and on change instruments (e.g., test fixes, change requests) which are defect-prone due to their late application to a product.

Using a tool-enabled, standard-compliant1 inspection process implementation2 is a key to removing product definition defects and eliminating past dramatic variances in inspection results and benefits.

For example, recent training in tool-enhanced, standard compliant inspections on a DoD agency project resulted in the six participating teams collectively finding 236 non-trivial defects in the 290 lines of actual requirement text their project had previously generated; a defect density of 814 defects per 1,000 lines of requirements. Afterward on this project’s first post-training inspection using the tools and techniques they had learned in class, the trained inspectors discovered 163 non-trivial defects in another 180 lines of requirement text; a defect density of 906.

Comparing fixing these defects at their points of introduction using a compliant inspection process (as depicted above and in Figure 3), versus randomly discovering and fixing problems throughout development (e.g., requirement reviews, design, code, test) and operations without compliant inspections, revealed potential estimated net project savings of more than 20,000 hours from the training results and an additional 10,000 or more hours from their first post-class inspection! When further considering the resulting quality, security and schedule issues that can spawn from defects not found early, the benefits of uniformly applying compliant inspections throughout the supply chains, shown by the circle indicators in Figure 2, to pre-code activities can be better appreciated.

Standardizing Inspections

Tragically, effective inspections are no longer used by most organizations, despite the perform peer reviews specific goal and practices of CMMI® Maturity Level 3. Reasons why this previous best practice [4] is no longer in favor include the following:

1: Inspection Pitfalls – the 10 most important reasons were captured in Stewart-Priven Group’s article [5] published in January 2008 explaining why organizations experience poor inspection value and inability to sustain early defect removal results due to unknowingly encountering any of the 10 inspection Pitfalls described:

1. Immature development infrastructure

2. Management responsibilities not understood

3. No inspection planning tools

4. Insufficient time allotted

5. No inspector execution tools for consistency— rigor—completeness, and no tools/reports for management monitoring

6. Limited result tracking & analysis

7. No post-training follow up

8. Lack of project-wide facilitation

9. Slow implementation

10. No inspection process capture

Adding to these pitfalls are a lack of education of software engineers, and the generic non-specific meaning that the terms peer-review and inspection have evolved to.

2: Code Inspection vs. Product Definition Inspection - Organizations then and now tend to believe inspection value applies mainly to code, which is not true. Code analyzer companies introduced high-tech static analysis capabilities in the early 2000s with marketing that downplayed inspection value and Fagan inspections in particular, attempting to convince industry and government to employ their automated defect removal technologies. Code analyzers, for the most part, are not succeeding in achieving high-quality software systems. General William Shelton’s opening remarks at the April 2009 DoD Systems & Software Technology Conference focused on the deteriorating state of software-driven systems. More recently, this was a focus of J.M. Gilligan’s article in the February 2012 issue of Software Technology titled “A Roadmap for Reforming the DoD’s Acquisition of Information Technology[6].” How could code analyzers alone remedy a situation where the majority of defects are introduced pre-code, during product definition activity?

3: Inspection Cost vs. Project Savings - Organizations tend to believe inspections are a cost rather than a savings/investment despite huge quantities of data to the contrary. The Stewart-Priven Group introduced savings estimation capability for software-driven projects to easily demonstrate their expected net project savings from inspection across multiple disciplines before actually committing to inspection. Inspection planning tools can guide projects in using their own past development history (or estimates) to perform what-if analysis to pre-determine their projected net savings from using inspection. More visibility is needed into the merits of inspection planning and saving estimation tools.

4: Manual Inspection vs. Tool-Enabled Inspection - Computerized inspection tools bring added value to inspections by enabling adherence to the inspection standard in section six of IEEE Std 1028TM-2008, ensuring consistency, completeness and rigor. Tool-enabled inspections can also provide interim and final one-page automated management reports of inspection process deviations, find/fix defect progress, ROI for individual inspections, accumulated labor and dollar savings, and rolled-up project inspection results. Unfortunately misleading material clouds the perception of inspections or can imply the inspection process can be tailored by organizations; where resulting consequences would actually weaken or undermine the performance criteria upon which inspections depend for success. The establishment of performance criteria that defined and enabled effective inspections was the output of the five-year experiment in the 1970s, envisioned by Lew Priven who hired Michael Fagan to lead the effort to improve IBM software product quality while reducing cost and schedule. The experiment’s result became known as “inspection” or “software inspection,” used initially by IBM internally, and now the foundation of the 2008 Inspection Standard.

5: Streamlined Inspections vs. Compliant Inspections – Using inspections for effective early defect removal and ongoing product risk assessment must employ inspection specific tools to ensure any non-compliance with the inspection standard is identified during an inspection. Without tool-enabled inspections, deviating from the inspection standard/process is inevitable and organizations will fall into the same pitfalls that contribute to the tarnished reputation of traditional manual inspections. Organizations that streamline, tailor, or re-do inspection material to fit their needs, or for their own use under the guise of public domain, do not understand the limits of inspection performance criteria, or the importance of rigor, repeatability and consistency that standard-compliant inspection tools provide for ongoing early removal of defects in product definition artifacts that code analyzers are typically unsuccessful in removing. Inspections are required for early defect removal in product definition artifacts, as recent history seems to demonstrate [7]; plus, there is no current alternative to inspections for ongoing product risk assessment! CMMI for process adherence is necessary but history and DoD studies have shown it not to be sufficient for attaining consistent high-quality, on-time, and within budget deliveries [2]. Without using standard-compliant inspection execution tools in real-time throughout the seven-step inspection process, then effective early defect removal from product definition artifacts and change instruments cannot be sustained leading to project efforts becoming prohibitively expensive or failing.

6: Human Shortfalls: These include resistance to change, preserving the status quo, protecting personal income and influence, and egos that make peers resist detecting errors in their work. All these contribute to late and costly, problem-ridden capabilities.

The Way Forward

Inspection process adherence using standard-compliant, computerized tools must be the prime focus of inspection training and repeatedly reinforced to consistently reap the benefits of pre-code early defect removal, as the results in Figure 3 demonstrate. There are other advantages of an integrated inspection tool set to supply chain risk management such as consistent and complete data collection, result tracking, and accumulated savings. However the overriding value tools provide is in achieving compliance to the standard for adherence to the performance criteria boundaries required for effective inspections, and their ongoing product risk assessments. Product risk visibility derived from each compliant inspection across the supply chain, and uniformly formatted by a contract specified inspection tool can provide the prime contractor and government program office a powerful vehicle to manage product risk.

Awareness of the vendor offerings for standard-compliant, tool-enhanced inspection capabilities that eliminate past inspection shortfalls and misleading perceptions; and awareness that inspection must be used during product definition activity where most defects are introduced, and on change instruments due to their defect-prone nature. As for code, the code analyzers continue to be necessary but are not sufficient. This message needs to stress that tool-enabled, standard-compliant inspections are the most effective alternative before coding. J. M. Gilligan’s February 2012 article [6], with its several references to recent government reports, highlights the inadequate state of government software-driven systems, and that a path forward is available with current technology.

Summary/Conclusion

Consistently attaining on-time, high-quality, cost-effective, and secure software requires agile techniques, process adherence discipline, sophisticated code-analyzers, and rethinking supply-chain risk management. These are all being done today, but they are not enough to end quality, schedule, and cost struggles! Ongoing product risk assessments also need to be the norm, coupled with effective early defect removal from pre-code product definition artifacts. Tool-enhanced, standard-compliant inspection provides both.

Deploying standardized inspections for product definition artifacts along the supply chain shown in Figure 2 would go a long way to mitigating current supply chain risk and improve product quality, security, schedule and cost.

Disclaimer:

CMMI® is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.

Tables and Figures:

Figure 1: Potential Software Supply Chain Paths ( Click to view image )

Figure 2: Standardized Inspection of Product Definition Artifacts along the Supply Chain ( Click to view image )

Figure 3: Example of Compliant Inspections applied to Product Definition Material ( Click to view image )


References and Notes

Notes: 1. Standard: document, established by consensus and approved by a recognized body, that provides, for common and repeated use, rules, guidelines or characteristics for activities or their results, aimed at the achievement of the optimum degree of order in a given context 2. Process (formal): set of interrelated or interacting activities which transforms inputs into outputs; and in the context of this article, implementing a Standard References: 1a. McGraw, Gary. “Making Essential Software Work” Citgal, Inc. Mar. 2003; 1b. Bender, Richard. [Position Papers] “The Business Case for Software Quality” page 23, 2004 2. The Software Assurance (SwA) Acquisition Working Group. “Software Assurance in Acquisition: Mitigating Risk to the Enterprise.” October 22, 2008 3. Federal Acquisition Regulation, “FAR –Sub-Part 46.202, Quality Assurance” (FAC 2005-53), August 2010 4. McConnell, Steve. ‘10 best influences on SW Engineering over the past 50 years’, IEEE Software, January/February 2000 5. Priven, Lew, and Stewart, Roger. “How to Avoid Software Inspection Failure and Achieve Ongoing Benefits.” Crosstalk The Journal of Defense Software Engineering, January 2008 6. Gilligan, John M. “A Roadmap for Reforming the DoD’s Acquisition of Information Technology”’ Journal of Software Technology, February 2012 7a. House Armed Services Committee Panel on Defense Acquisition Reform, Interim Findings and Recommendations, DAR Interim Report, March 4, 2010; 7b. National Research Council, Committee on improving Processes and Policies for the Acquisition and Test of Information Technologies in the Department of Defense, Achieving Effective Acquisition of Information Technology in the Department of Defense, 2010; 7c. Business Executives for National Security (BENS) Task Force on Defense Acquisition Law and Oversight, Getting to Best: Reforming the Defense Acquisition Enterprise, July 2009; 7d. Tech America, Recommendations on Information Technology Acquisition Reform, Presented to the Defense Acquisition Reform Panel of the House Armed Services Committee, February 23, 2010.

Roger Stewart

Click to view image

Roger Stewart is co-founder and Managing Director of the Stewart-Priven Group. He is an experienced Lead Systems Engineer and Program Manager in both government and commercial system development, including Systems Engineering, Software Development, System Integration, System Testing, and Process Improvement.
Previously, Stewart taught the Fagan Defect-Free Process for Michael Fagan Associates (8 years) after spending 31 years with IBM’s Federal Systems Division, (now part of Lockheed-Martin) managing and developing systems for Air Traffic Control, Satellite Command & Control, On-Board Space Shuttle, Light Airborne Multipurpose System (LAMPS Helicopter); and in Commercial Banking, Telecommunication and Networking systems.
Roger has a BS in Mathematics from Cortland University; published other Inspection articles and presented on the topic at the annual DoD Systems & Software Technology Conference, including a plenary session breakfast in 2009.

P.O. Box 2174
Mt. Juliet TN 37121
Phone: (615) 754-6685
E-mail: rstewart025@gmail.com


« Previous Next »