By Steve Bygren, Greg Carrier, Tom Maher, Patrick Maurer, David Smiley, Rick Spiewak and Christine Sweed


Abstract.

Historically, software developed under government contracts often does not stand up under real-world use, and defects frequently result in cost and schedule overruns. While proposed development activities from contractors commonly list measures to improve quality, these descriptions cannot be used to select a winning bidder if they are not part of the evaluation criteria. By making software quality requirements explicit at the proposal stage, contractor selection can be influenced by criteria based on best practices in software development.

If we want to improve the quality of our software, a “Quality in Depth” approach is needed—introducing quality related measures at every stage of software acquisition. In a previous article1, one of the authors provided recommendations for improving software quality at the construction phase. This article discusses how to apply these same principles to the source selection process.

In order to find a way to include software practices as selection criteria, the authors set out to identify and recommend changes to Sections L and M of a government Request for Proposal (RFP) or Instructions for Proposal Preparation (IFPP) and Evaluation Criteria (EC) in an attempt to improve software and system quality. These changes will enable selection teams to identify contractors whose software development processes and compliance with software quality standards are more likely to produce the desired results.

1. Background

What Is Software Quality?

Quality is often thought of as an absence of defects. With many software products however, “defect” does not adequately describe the range of phenomena that affect software quality as perceived by the customers, end users and other stakeholders. Using Crosby’s philosophy2, we define the term “software quality” to mean conformance to the requirements of the software product’s users and other stakeholders. The more closely a software product conforms to these requirements, the higher its quality.

We are particularly interested in software quality as it affects the acquisition process for defense related software. While end user requirements are of prime importance, poor software development and quality monitoring practices in early- and mid-stage acquisition can result in failure to provide the desired results. These failures range from unwanted or missing features to cost and schedule overruns to critical flaws in system security or reliability.

How Do You Measure Software Quality?

Software quality as an outcome is best measured by the number of defects encountered after development is complete as the numerator, divided by the “size” of the software as the denominator. One could also argue that if two different products were to be compared, some sort of “difficulty factor” could be applied, as well as references to the software language or development environment employed, e.g., assembly code versus high order languages, or object-oriented versus functional languages, etc.

Metrics exist which can be used to estimate the potential defects in code. These are based on the use of function points as the measure of “size.” Function points can also be (loosely) correlated with the commonly used measurement, SLOC.

2. Approach

This article is the outcome of a study the authors conducted at MITRE. Our approach was to gather information from Subject Matter Experts (SMEs), contracting officers, and acquisition experts for recommendations for additions to proposal documents. Part of this study was conducted through interviews and SME e-mail group lists. Reference materials from the Air Force and Navy were found which provided recommendations from prior work [1, 2]. We then adapted the suggestions to Sections L and M to more thoroughly describe software quality related criteria for source selection. Some of these criteria are aimed at the technical evaluation team, while some can be used by cost evaluators and past performance evaluators as well as the technical team.

3. Recommendations for Section L (Instructions for Proposal Preparation)

1. The offeror’s proposal shall include a proposed Software Development Plan (SDP) which describes their approach to software development, to include the tools, techniques and standards to be used for development, unit testing and component testing; integration tools and techniques (including configuration management) used to ensure the integrity of system builds; the number and type of reviews that are part of the development process; and the methods and tools used to manage defect reports and analysis, including root cause analysis as necessary. The proposed SDP will form the basis for a completed SDP to be available after contract award as a Contract Deliverable Requirements List (CDRL) item, subject to government review and approval.

2. The offeror shall describe their plan for effective code reuse in order to minimize the amount of new code to be developed. Reused code can come from any origin, including previous efforts by the offeror or as provided by the Government in the bidders’ library.

3. The offeror shall provide a Basis of Estimate (BOE) describing the rationale for the proposed staffing. The detail of the BOE shall include labor hours for each labor category (e.g., system engineering staff versus software engineering staff) for the identified tasks in the Work Breakdown Structure as it relates to the Statement of Work (SOW).

4. The offeror shall describe the process for orientation and training for all project employees (e.g. certification and training in software best practices including information assurance and risk management).

5. The offeror shall describe related systems experience, including a description of previous experience developing software of the same nature, and a description of the extent to which personnel who contributed to these previous efforts will be supporting this effort.

6. The offeror shall describe proposed development practices. For example, if spiral/incremental development, they shall describe the number, duration, and scope of spirals, as well as how the use of your approach would result in improved product quality and user satisfaction over time.3

7. The offeror shall provide an Integrated Master Schedule (IMS) and accompanying narrative that describes all significant program activities that are aligned with the proposed program staffing profile. Include a timeline for completion of each activity identified in the proposed program. Provide details that clearly describe the purpose for and importance of key activities. Identify all critical path elements and key dependencies.

4. Recommendations for Section M (Evaluation Criteria)

The proposed SDP shall show a complete and comprehensive software development process, which incorporates best practices as well as standards such as IEEE 12207-2008. The contractor will be evaluated based on how their processes, as described in the SDP, incorporate the use of software best practices.

Evaluation criteria related to the SDP include the following:

• The number and type of peer reviews.

• The use of automated unit testing including test coverage requirements.

• The use of automated syntax analysis tools and adherence to the rules incorporated by them.4

• The comprehensiveness of integration and test methods, including continuous integration tools if used.

• The use of readiness requirements such as unit test and syntax analysis for code check-in.

• Configuration management and source code control tools and techniques.

• The extent to which root cause analysis of defects is part of the development process.

• The selection of software source code to be reused, replaced or rewritten from previous implementations or other origins, including a description of how it will be ensured that reused code meets or is brought up to the same standards as newly developed code. Risks associated with reused software shall also be discussed. Such software shall include government rights to the source code.

The IMS and accompanying narrative will be evaluated for level of detail and relevance of significant program activities, degree of alignment, the proposed program staffing profile, and integration of the proposed SDP into the IMS. Additionally, critical path elements and key dependencies will be assessed for relevance, completeness and the manner and level of risk containment.

5. Incorporating Software Quality Measures in Contracts

The contract development process includes several steps at which information can be gathered and requirements set to include software quality as a measure of vendor performance.

Sections L & M or equivalent from the RFP

>> Add software quality measures as a discriminating factor in selecting the contractor

>> Enumerate expectations in this area:

• Types of methods used

• Evidence to be provided

Technical Requirements Document, Statement of Objectives, and SOW

Add requirements in the form of deliverable items—as CDRLs or Data Accession List items as appropriate. Examples include the following:

>> Output of automated unit tests showing code coverage at or above required minimum.

>> Output of automated syntax analysis showing conformance to pre-determined rules.

>> Evidence of accomplishing required peer reviews.

>> Itemized list of tools with version numbers used to produce output from each source module.

>> Programmer’s reference manual with examples.

>> Interface definitions.

>> List of all software components with the following information:

• Purpose and function.

• Interfaces provided.

• Language/version for each module.

• Complete source code.

>> Source from architectural design tool where available.

>> Use cases (text and diagrams).

>> Class diagrams where applicable.

>> Complete list of any third-party components with version numbers.

>> Contact information for any outside dependencies.

>> Build procedures, including documentation for building all software components from source code.

>> Test procedures—including any automated unit tests with source code, test scripts.

6. Rationale for Incorporating Recommended RFP Language

The recommended RFP language was derived by the authors from a variety of sources including MITRE acquisition subject matter experts, existing guidance documents from the Navy and Air Force, and also from the authors’ experience. We have tried to provide a succinct rationale as to why the language asks for specific information from the contractor in the RFP:

The SDP is a maturity indicator of the bidder’s development process. By evaluating this, and then putting its provisions under contract, it becomes possible to select a contractor on the basis of development methodology and then obligate them to perform as proposed.

Automated unit tests and comprehensive peer reviews are widely used best practices. Capers Jones6 has noted that these are among the required steps to achieve effective defect removal.

Continuous Integration (CI) often includes the automated invocation of tests and code analysis during the build process. CI and static analysis expose problems earlier in the development process. The earlier problems are discovered, the lower the cost to resolve.

Root cause analysis prevents the introduction of defects and is a recognized best practice in all approaches to process improvement. It is a CMMI® Level 5 practice area. Prevention is more cost effective than detecting and fixing defects after they are introduced.

The BOE helps the evaluator understand the bidder’s cost to compare against industry averages and government cost models. By examining proposed labor categories, this can be checked against predicted labor distributions from government cost models as well.

The IMS can be checked for alignment with required milestone dates, and it supports an independent estimate.

7. Guidance for Evaluation Team Experience

The government’s evaluation team must have relevant software engineering experience. The experience should cover the full life cycle of software development from design to development, integration, testing, and delivery. If the proposal is seeking a particular style of development methodology (e.g., waterfall, spiral/incremental, agile), then the evaluation team should have experience in that methodology in order to evaluate the RFP response.

Since a significant portion of the suggested contract language relates to software quality monitoring, the evaluators should be familiar with unit testing, peer reviews, CI, static code analysis, and metrics. Finally, evaluators should have some knowledge of various practices and approaches of applying these techniques, for example, when it comes to test-driven development.

The field of software engineering is diverse. It is insufficient to simply have general software engineering experience on the evaluation team without further having experience in the applicable domain(s). Examples of these domains include real-time/embedded, kernel/operating systems, numerical/digital signal processing, web applications, SOA, information retrieval/search, security, and human-computer interface.

Finally, the evaluation team should have an understanding of the CMMI process and rating criteria.

8. Guidance for Evaluating Technical Responses

The recommended contract language in this article includes Section M of the RFP, also appearing as Evaluation Criteria. The language is not very specific so as to elicit responses that are more original than simply claiming to do a long list of things that the government is checking for. In this section, we discuss more specific guidance for the evaluation team in evaluating the responses.

In advance, the team should define objectives that are sought after and then define measurable criteria. The more objective the criteria, the better, though it is recognized that coming up with this criteria can be a challenge. After defining criteria, they are prioritized and then weighted in a scheme the team deems appropriate.

Some general evaluation tips are as follows:

• If key staff are identified in the proposal, how likely are they to be available during contract execution?

• In reference to quality assurance processes, does the proposal language favor or at least mention “empowerment” of the quality assurance team over engineering processes?

• Regarding the contractor’s approach to automated unit testing: Does the contractor require that unit tests be passed and cover a reasonable percentage of code before code can be checked in? Does the contractor use test-driven development?

• Regarding the contractor’s approach to automated syntax analysis: Does the contractor require that syntax analysis be performed and that all required rules are followed before code can be checked in?

• Regarding development build and integration: Does the contractor use an automated build process that incorporates syntax analysis and automated unit testing?

You can expect that the response is going to claim appraisal at a specific CMMI maturity level (commonly at least level 3). This can be verified with the Appraisal Disclosure Statement (ADS) document. Another source is the Standard CMMI Appraisal Method for Process Improvement (SCAMPI). For the larger contractors, particularly when work is further sub-contracted out, look for further CMMI level compliance information on the specific division/unit and sub-contractor(s) as applicable.

9. Development Process

If the proposal declares that a development process will be used that will involve multiple iterations/spirals/increments (which is standard practice), then the evaluation team should look for further details on the process to include the following:

• What is the duration and scope of each increment?

• Are lessons and obstacles from one increment reviewed for improvement to a subsequent increment?

• Is user (customer) feedback interaction only up front or do most increments incorporate this? And how is that feedback prioritized?

• Are multiple increments planned in sufficient detail, or are only the present and possibly next increment planned?

10. Software Engineering

One key thing to look for in a proposal is to what degree the contractor has experience in the technology the RFP calls for them to deliver. The more complex the system, the more important applicable contractor experience is.

Many DoD systems have a degree of interoperability and integration required of them. For integration with particular systems, verify if the contractor has experience with that system or has relationships with third parties with integration capabilities that will be used. The contractor should also participate in applicable Communities of Interest.

Testing processes and technologies that support them are important. Look for information on a test plan or strategy. If the proposal is serious about continuous integration and use of supporting tools, then listing the software to be used for this is a promising sign. Information on how the tools are used (e.g., by exception and/or monitored on a periodic basis—and what period) is also telling. If the proposal includes information on the proposed system design, then the evaluators could look to see how “testable” the design is, particularly as it is incrementally built.

11. Conclusions

While it is important to implement quality measures in software construction, this is undertaken after a contractor has been selected. The authors recommend an in-depth approach, beginning with the process of selecting the contractor. It can be easy to overlook the importance of including specific language in the proposal documents in order to be able to select the right contractor from those responding to an RFP. In order to accomplish this goal, it is critical to specify the instructions in Section L (or the IFPP) and the evaluation criteria in Section M (or the EC) so that these can be used to assign strengths or weaknesses appropriately. This is an early, but often neglected, piece of the puzzle involved in building quality software products for defense applications.

Acknowledgement:

The authors wish to thank Tim Aiken and Carole Mahoney for their guidance as we conducted our research. Additionally, we wish to thank the MITRE Systems Engineering Practice Office, the acquisition community, and our various programs for their contributions to our survey.

Disclaimers:

CMMI® is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.

©2011-The MITRE Corporation. All rights reserved. Approved for public release: 11-2921. Distribution unlimited.

Tables and Figures:

Table 1. Sample Rating Scale for SDP Evaluation Criteria5 ( Click to view image )


References and Notes

References: 1. USAF Weapon Systems Software Management Guide, August 2008. . 2. Guidebook for Acquisition of Naval Software Intensive Systems, dated September 2008. . Notes: 1. Spiewak, Rick and Karen McRitchie. “Apply the Fundamentals of Quality in Software Construction to Reduce Costs and Prevent Defects.” CrossTalk, Dec. 2008. 2. Crosby, Philip B. Quality Is Free: The Art of Making Quality Certain. McGraw-Hill Companies, 1979. 3. Note that while not part of the technical evaluation, the government evaluation team will examine Contractor Performance Assessment Reports (CPARs) for relevant performance by the respondent on other contracts. 4. Jones, Capers. Software Engineering Best Practices. McGraw-Hill, 2010 5. Categories suggested by conversation with Jeff Pattee, Chief, Product Definition, Airspace Mission Planning Division, Electronic Systems Center, USAF. 6. Jones, Capers. “Measuring Defect Potentials and Defect Removal Efficiency.” CrossTalk, June 2008. .

Steve Bygren

Click to view image

Steve Bygren is a Principal Information Systems Engineer at The MITRE Corporation, supporting the Electronics System Center (ESC) from Peterson Air Force Base, Colorado, where his focus is on multi-system integration and advanced development. Bygren has 28 years of experience in software and systems engineering, and received his Bachelor of Science in Computer Science from Montana State University.

The MITRE Corporation
1155 Academy Park Loop
Colorado Springs, CO 80910
Phone: 719-659-3794
E-mail: sbygren@mitre.org

Greg Carrier

Click to view image

Greg Carrier is a lead software systems engineer at the MITRE Corporation in Bedford, MA. Greg’s interest in improving software quality stems from his experience managing software-related projects for a variety of government contracts.

The MITRE Corporation
202 Burlington Rd.
MS S355
Bedford, MA 01730
Phone: 781-271-5180
E-mail: gcarrier@mitre.org

Rick Spiewak

Click to view image

Rick Spiewak is a Lead Software Systems Engineer at The MITRE Corporation. Rick works at the Electronic Systems Center at Hanscom Air Force Base as part of the Battle Management Group, concentrating on Mission Planning. He has been focusing on the software quality improvement process, and has spoken on this topic at several conferences. Rick has been in the computer software industry for more than 41 years, and has bachelor’s and master’s degrees in Electrical Engineering from Cornell University. He studied quality management at Philip Crosby Associates.

The MITRE Corporation
202 Burlington Rd.
MS 1614E
Bedford, MA 01730
Phone: 781-225-9298
rspiewak@mitre.org

Tom Maher

Click to view image

Tom Maher is a Senior Software Systems Engineer for the Information and Computing Technologies Division at MITRE with more than 15 years of hands-on experience in applying technology to meet the needs of business. Tom’s employers and clients have ranged from Internet startups to divisions of multi-billion dollar corporations; he has worked across multiple domains including retail, financial services, medical information systems, and homeland defense.

The MITRE Corporation
202 Burlington Rd.
MS 1630T
Bedford, MA 01730
Phone: 781-225-5355
E-mail: tdmaher@mitre.org

Patrick Maurer

Click to view image

Patrick Maurer is a lead communications engineer at The MITRE Corporation. He develops networking and network management technologies for SATCOM and line-of-sight terminals. His current work focuses on standardizing network management interface for terminals. Prior to joining MITRE, Patrick worked in the telecommunications industry developing modems and networking systems. He has a BSEE from Northeastern University, MSEE from Massachusetts Institute of Technology and an MBA from Boston University.

The MITRE Corporation
MS D300
202 Burlington Road
Bedford, MA 01730
Phone: 781-271-4698
E-mail: pmaurer@mitre.org

David Smiley

Click to view image

David Smiley is a lead software developer specializing in search technologies. He has 12 years of experience in the defense industry at MITRE using Java and various web technologies. David is the principal author of “Apache Solr 3 Enterprise Search Server” and has presented at conferences and taught classes about Solr.

The MITRE Corporation
202 Burlington Rd.
MS M330
Bedford, MA 01730
Phone: 781-271-7659
E-mail: dsmiley@mitre.org

Rick Spiewak

Click to view image

Rick Spiewak is a Lead Software Systems Engineer at The MITRE Corporation. Rick works at the Electronic Systems Center at Hanscom Air Force Base as part of the Battle Management Group, concentrating on Mission Planning. He has been focusing on the software quality improvement process, and has spoken on this topic at several conferences. Rick has been in the computer software industry for more than 41 years, and has bachelor’s and master’s degrees in Electrical Engineering from Cornell University. He studied quality management at Philip Crosby Associates.

The MITRE Corporation
202 Burlington Rd.
MS 1614E
Bedford, MA 01730
Phone: 781-225-9298
rspiewak@mitre.org

Christine Sweed

Click to view image

Christine Sweed is a Lead Networking Systems and Distributed Systems Engineer at The MITRE Corporation. She has worked at MITRE for more than 10 years on various software development projects following 10 years of industry experience. She has a B.A. in chemistry from SUNY Potsdam, and a M.S. in Computer Science from Boston University

The MITRE Corporation
202 Burlington Rd.
MS K214
Bedford, MA 01730
Phone: 781-271-3552
csweed@mitre.org


« Previous Next »