AbstractPrevious articles in CrossTalk and the Journal of Software Technology provided practical guidance to improve the quality of Earned Value Management (EVM) information [1, 2, 3, 4]. This update recommends contract language and project monitoring techniques to ensure that contractors integrate technical performance, including software functionality, with EVM. The key enablers are the Integrated Master Plan (IMP) and linkage to Systems Engineering (SE) work products and best practices.
EVM can become an effective program management tool and deliver better value to the taxpayer and warfighter if contractors revised their processes and reports to integrate technical performance and quality with cost and schedule performance. However, there are no contractual requirements within the acquisition regulations or contract data requirements to require that contractors:
1. Tie the technical baseline to the EV Performance Measurement Baseline (PMB).
2. Tie technical progress to the Technical Performance
Measures (TPM) of the program, including progress towards achieving planned functionality.
The guidance in this article meets EVM challenges that were addressed in the DoD report to Congress; DoD EVM: Performance, Oversight & Governance Report that was required by the “Weapon Systems Acquisition Reform Act of 2009.” The challenges concern technical performance and SE, as follows.
• EVM can be an effective program management tool only if it is integrated with technical performance
• The engineering community should establish TPMs that enable objective confirmation that tasks are complete;
• If good TPMs are not used, programs could report 100% of earned value (or credit for work performed), even though they are behind schedule in terms of validating require ments, completing the preliminary design, meeting weight targets, or delivering software releases that meet the requirements.
• The EV completion criteria must be based on technical performance, the quality of work must be verified, and criteria must be defined clearly and unambiguously.
• The PM should ensure that the EVM process measures the quality and technical maturity of technical work products instead of just the quantity of work performed.
SE Process and Products
EVM can be an effective program management tool only if the:
• EVM processes are augmented with a rigorous SE process
• SE products are costed and included in EVM tracking.
If the SE lifecycle management method is integrated with the planning of the PMB, then EVM will accurately measure technical performance and progress.
Contractual Impediments to Effective EVM
Neither the Defense Federal Acquisition Regulation Supplement (DFARS) nor the Data Item Descriptions (DID) require contractors to tie EV to technical performance. The DFARS Earned Value Management System (EVMS) clauses cite compliance with the ANSI-748 EVMS guidelines. However, the use of TPMs is optional per EVMS. Per the defense acquisition program support methodology, “EVMS has no provision to measure quality. Use TPMs to determine whether your percent completion metrics accurately reflect quantitative technical progress and quality toward meeting key performance parameters.”
EVMS focuses on the work scope and is silent on product scope. It also states, “EV is a direct measurement of the quantity of work accomplished. The quality and technical content of work performed is controlled by other processes.” These loopholes create a “quality gap.” The quality gap enables contractors to submit misleading management information. EV and the cost performance may be overstated when it is based on the percentage of drawings or code completed without regard to the technical maturity of the evolving design. As a result, the estimate at completion may be understated.
Useful guidance to link EVM with TPMs, the technical baselines, IMP accomplishment criteria, and SE work products is found in many DoD guides, as summarized at <http://www.pb-ev.com/Pages/DoDGuidance.aspx>. However, acquisition managers are not able to implement this guidance if the contractors fail to provide needed information. Even the IMP is optional in DoD guidance and not contractually required in DFARS.
Better Buying Power: Suppler Incentives
The DoD is striving to deliver better value to the taxpayer and warfighter by improving the way the it does business via the Better Buying Power (BBP) initiatives. BBP 2.0 includes the initiative to, “Institute a superior supplier incentive program.” To support that initiative, the Navy is currently developing a pilot program for DoD with the intent to recognize and reward contractors who demonstrate superior performance by focusing on cost, schedule, performance, quality, and responsiveness.
The following opportunities and solutions should be considered when developing BBP 2.0 supplier incentives.
Opportunities and Solutions
The following guidance seizes four opportunities that underlie the EVM challenges, as shown in Figure 1. Solutions to improve contractual requirements and acquisition management follow.
Base EV on Technical Performance
This opportunity has two components. First, do top down planning that includes defining milestones for achieving technical objectives. Then measure interim progress towards those meeting those objectives.
Top Down Planning
The solution for basing EV on technical performance has two components. First, develop integrated plans from the top down, starting with the technical baseline. Second, track progress towards meeting technical objectives.
The elements of effective, top down planning are:
1. Contractually-required IMP.
2. Use the Integrated Baseline Review (IBR) to reach agreement on IMP accomplishment criteria and to verify that contractor integrates technical performance and SE work products with the Integrated Master Schedule (IMS) and EVM.
3. Use major technical reviews and EVMS compliance reviews to verify that contractor maintains traceability from IMP to IMS to Control Account/Work Packages.
First, make the IMP a contractual requirement with requirements-based accomplishment criteria that are tied to the technical baseline. The criteria should include the completion of performance measures such as Measures of Effectiveness (MOE), Measures of Performance (MOP) and TPMs at key IMP events such as the System Functional Review (SFR), Preliminary Design Review (PDR), and Critical Design Review (CDR). Examples of accomplishment criteria are shown in Figure 2.
Second, use the IBR to forge agreements and to verify the degree of integrated program management. Verify implementation of the following during IBR:
• Requirements traceability from the requirements data base to the IMS and from the IMS to work package completion criteria.
• IMS includes interim and final milestones for development of SE work products with criteria that are consistent with the Contract Work Breakdown Structure (CWBS). The milestones include derived requirements, definition of required functionality and quality attributes, and verification methods and criteria.
• Milestones for establishing product metrics. MOEs and MOPs are defined at the SFR. TPMs are defined at the PDR.
• Milestones with technical maturity success criteria including TPM planned values, meeting requirements, and percent of designs complete.
• Define success criteria for event-driven technical reviews/ IMP events.
• Revise/clarify criteria for CDR and subsequent events based on knowledge of revised and derived requirements to be met and TPM planned values.
• Flow down of SE milestones to work packages.
Measure Interim Performance
The solution to basing EV on interim, technical performance includes the following actions. First, establish objective linkage between TPM planned values and EVM. For physical objectives, use TPMs. For planned functionality, base EV on achieved functional requirements.
Next, compare the EV schedule variance (converted to duration) with the technical performance schedule variance. If the variances are inconsistent, perform a root cause analysis to determine reasons for the inconsistency. Then revise EV to be consistent with technical performance.
If technical performance is behind schedule, perform variance analysis and develop corrective actions. Then, revise the estimate to complete forward for work packages with corrective actions.
Finally, correct EV to reflect the technical performance status. A backwards adjustment to EV is appropriate for work packages with corrective actions. This technique enables the use of EV to track corrective actions to resolution and closure.
Account for Deferred Functionality
In practice, contractors seldom account for deferred functionality when functional requirements are deferred from one build, release, or block to another. Normally, the numbered build and its respective work package are “closed” and 100% of the EV is taken, based on being finished with the build. When this happens, EV fails to disclose the true schedule variance. Also, cost performance is overstated.
The solution is to account for deferred functionality. If the build is released short of its planned functionality, the preferred technique is to take partial EV and close the work package. Then, transfer the deferred scope and Budgeted Cost of Work Remaining (BCWR) to the first month of the work package of the next increment. When this is done, EV mirrors technical performance and the schedule variance is retained.
Track SE Tasks Discretely
SE tasks are sometimes incorrectly planned as level of effort. Even when SE is discretely planned, EV is often based on interim milestones of progress towards completing a document such as a specification. These techniques fail to show objective progress towards completing requirements-based SE tasks such as requirements analysis and validation, definition of technical measures, or completion of trade studies. Getting behind schedule on these tasks is an early indicator that an IMP event, such as SFR, PDR, or CDR, will slip.
The solution for measuring SE tasks discretely has several elements. First, include significant accomplishments and accomplishment criteria for SE tasks and work products in the IMP. Next, show progress towards completing those SE work products in IMS and work packages. Typical SE work products include the system architecture (functional and physical), interface controls, specifications, trade studies, and test procedures.
For SE tasks such as defining and approving the product requirements, including derived requirements and allocated requirements, develop a requirements-based, time-phased budget that is based on the planned schedule for those requirements. Then base EV on the progress towards completing those requirements as recorded in the requirements data base. Typical examples of requirements status include defined, early validated, determined verification method, approved, allocated, and traced to a test procedure.
For work packages that result in SE work products that are technical measures (MOEs, MOPs, and TPMs), base EV on progress towards meeting the IMP criteria for their completion.
Plan Rework and Track it Discretely
Rework is frequently not adequately planned in the PMB and IMS. The rework can include rework of requirements analysis, design, and test tasks. Even if rework is belatedly budgeted from management reserve, it is often measured as level of effort, or if measured discretely, as a percent of the planned iterations. Neither technique reports progress towards developing or meeting the technical requirements.
The solution for better understanding and management of rework begins the proposal and the negotiated contract value. The program should verify realistic that rework assumptions and estimates are included in suppliers’ proposals and negotiated values. The estimates should include productivity/quality measures such as rework percent and defect density.
The program should review the adequacy of budget and schedule for rework in the PMB. Rework should be planned in a separate planning package from the original task. When converted to a work package, it should be measured discretely based on technical maturity targets.
Establish interim milestones for rework with associated TPM planned values or quantified functionality based on meeting requirements. Then take interim EV based on net achieved technical performance. Make a negative adjustment to EV when necessary for accurate status reporting.
If rework is not in a separate work package and if EV had been taken for achieving a technical milestone, correct EV and the IMS when there is subsequent knowledge that the milestone completion criteria are now unmet. The milestone should be re-opened and a negative adjustment should be made to EV. Cumulative EV must reflect net technical progress.
New Contractual Requirements
New contractual requirements should be included in the Statement of Work (SOW) to communicate program needs. Some of the requirements are tantamount to tailoring several of the EVMS guidelines. The primary objective is to refocus management attention from the work scope to the product scope and to provide EV that truly reflects technical performance. Recommendations for acquisition reforms, including a revision to DFARS, are in a Defense AT&L article . However, program offices can accomplish the same objectives by implementing the specific recommendations that follow.
1. For top down planning, make the IMP a contractual requirement and use a tailored CWBS DID.
2. Use tailored EVMS guidelines or specify EVM techniques in the SOW to:
a. Incorporate the product scope or technical baseline in the PMB.
b. Tie EV to technical performance.
c. Account for deferred functionality.
d. Track specified SE tasks discretely.
e. Plan rework and track it discretely.
IMP and SE Work Products
Require that an IMP be a contract deliverable. Start with the DoD IMP and IMS Preparation and Use Guide that is tailored to specify SE work products and accomplishment criteria. The IMP DID should be developed by the program SE organization.
An excellent source for specifying the SE tasks and work products is the Air Force Space and Missile Command Standard, SE Requirements and Products . For example, it states that required SE products are: the SE accomplishments, accomplishment criteria, and narrative in the IMP; tasks in the IMS; and work packages in the EVMS, and such other specific plans (such as tradeoff plans) as may be needed to achieve the attributes required above.
CWBS DID, DI-MGMT-81334C
In practice, the CWBS does not include or point to the quantified technical or functional performance requirements that are in the specifications. Contractors will have to reference the functions at the CI level in the allocated functional baseline and product specifications in the product baseline. The contractual language is: The CWBS Dictionary for the appropriate CWBS elements must be updated to include or reference, at PDR, the functions allocated to one or more system CIs and, at CDR, the product specifications for each CI in the system.
With regard to Guideline 2.1a, authorized work, add contractual language to: “Include the work necessary to produce the product scope of the program, including rework (when applicable). The product scope is the technical baseline. It includes the features and functions that characterize a product or result.”
With regard to Guideline 2.2b, measure performance, add contractual language to specify that “All TPMs that have been identified at PDR shall be used to measure progress in appropriate work packages. Compare product and process metrics data against plans and schedule using trend analysis to determine technical areas requiring management attention.”
With regard to Guideline 2.5b, revisions, add contractual language to specify, “When work scope that is behind schedule is internally re-planned from the work package that is being closed to another open work package, the BCWR in the work package that is being closed shall be transferred to the first open period of the receiving work package The objective is to prevent arbitrary elimination of existing schedule variances. The time-phased estimate to complete of the receiving work package must be based on an analysis of remaining tasks in the IMS and projected resource plan.”
With regard to Guideline 2.1a, authorized work, add contractual language to specify that: “The work scope includes rework. Rework includes corrective actions to hardware/software deficiencies, including deficiencies in the underlying requirements. Rework shall be planned, estimated, and included in the initial PMB. Rework shall be measured discretely and use technical performance goals to measure progress.
DoD has identified challenges to improve the usefulness and validity of EV information by integrating technical performance and systems engineering work products with EVM.
Implementation of the recommended acquisition management processes and new contractual requirements will provide the following benefits:
• Close the EVMS Quality Gap
• Insightful IBRs and technical reviews
• Valid contract performance reports
- Objective technical/schedule status
- Credible EAC
• Early detection of problems
- Program performance
- EV measurement and compliance
Incentives for suppliers to implement these process improvements can be implemented through the BBP 2.0 initiatives.
Tables and Figures:
Figure 1: ()
Figure 2: Specified IMP Reviews, Baselines, Accomplishments/Criteria ()
References and Notes1. Solomon, Paul J. “Practical Software Measurement, Performance-Based Earned Value.” CROSSTALK Sept. 2001: 25-29 . 2. Solomon, Paul J. “Performance-Based Earned Value.” CROSSTALK Aug. 2005: 25-26 . 3. Solomon, Paul J. “Practical Performance-Based Earned Value.” CROSSTALK May 2006: 20-24 . 4. Solomon, Paul J. “Improving the Quality of EVM Information” JOURNAL OF SOFTWARE ENGINEERING Aug. 2011 . 5. Solomon, Paul J. “Path to EVM Acquisition Reform.” DOD AT&L May 2011:25-27 . 6. Air Force Space Command , Space and Missile Systems Center (SMC) Standard SMC-S-001, “SE Requirements and Products.” July 2010.
Paul J. Solomon, PMP, is co-author of the book, Performance-Based Earned Value.® He supported the B-2, Global Hawk, and F-35 programs at Northrop Grumman. He co-authored the EVMS Standard and received the DoD David Packard Excellence in Acquisition Award. He was a Visiting Scientist at the Software Engineering Institute and published “Using CMMI to Improve EVM.” His web site, , contains EVM best practices. He holds a BA and an MBA from Dartmouth College.
« Previous Next »