By Glenn Tolentino, Jeff Tian and Jerrell Stracener


This paper presents a framework for selecting a combination of existing systems to satisfy new, emerging requirements while reusing existing and proven capabilities to ensure mission success. Decision attributes will be considered during the selection process and will be used to measure the networked computer system’s effectiveness to accomplish the mission. This approach will enable system stakeholders to make critical, well-informed decisions to address the continuing evolution of missions, threats, budget and technology.


Defense Computer Systems developed and maintained over the years have resulted in thousands of disparate, compartmented, focused, and mission-driven systems that are utilized daily for deliberate and crisis mission planning activities. The defense acquisition community is responsible for the development and sustainment of these systems over the course of its systems engineering life cycle from conception to utilization and eventually to the decommissioning of these systems. In addition to the cost of these systems' acquisition and development phases, there are associated investment costs that are necessary to sustain these computer systems over their life cycles. While missions are being planned and satisfied by existing computer systems, there are new missions being proposed which cannot be satisfied by a single existing computer system capability. Therefore, this raises the question of whether a Networked Computer System (NCS) is preferred in order to satisfy new capability requirements by using combinations of existing and developmental computer systems. This paper explores an approach to identifying a preferred NCS solution and measuring the NCS's effectiveness in satisfying a mission.

Defense Computer Systems

The United States Department of Defense (DoD) requires new capabilities with new requirements to address the continuing evolution of missions, threats, budget and technology. These new capabilities can be satisfied with existing operational systems, the development of new functionalities into existing systems, or the development of completely new systems. However, another approach is to develop a combination of existing systems for emerging requirements to satisfy new capabilities while reusing existing and proven capabilities with the goal of ensuring mission accomplishment. This "identification and selection" approach will reduce the risk associated with system development and integration efforts while providing a better-informed decision making process in satisfying user requirements while considering cost, schedule, and program execution performance. In addition to the decrease of defense spending, the acquisition community must also seek innovative ways to satisfy the new systems capabilities needed in order to accomplish the DoD operational mission.

The research performed in this paper was based on the United States DoD's need for an operational system capability that can satisfy a defense mission, and specifically seeks to determine if the capability requires a group of computer systems to be developed into a single NCS solution (see Figure 1). Once a preferred NCS solution is identified, the question becomes "How do we measure the effectiveness of these developed and integrated computer systems with the end of goal of satisfying mission success?"

Overview of Methodology Framework

The notional conceptual methodology includes a number of steps that must be accomplished in order to select computer systems in developing the NCS solution and measuring the solution's effectiveness. Table 1 describes the two phases followed by a summary describing each of the steps under each phase.

Phase 1: Selecting Computer Systems for an NCS Solution

This first phase focuses on developing an NCS solution for a given mission based on mission requirements and objectives. This phase will address the development of a NCS solution based on existing computer systems that are either already operational or currently being developed with a known time for capability readiness and acquisition. The following sections describe each of the steps.

Phase 1 - Step 1: Describe the NCS mission:

This step describes the intended overall mission or missions of the NCS. Defining the mission is a high-overview activity that specifies what is to be performed with specific mission objectives. These mission objectives can be translated as a set of activities that must be performed in order to achieve mission success and can be characterized as the mission profile, which translates to specific capabilities. There are a number of capabilities that are required by the NCS in order to satisfy each of the mission objectives and accomplish mission success.

Phase 1 - Step 2: Identify Computer Systems to Satisfy Mission-Required Capabilities

During this step, each of the capabilities required for the NCS solution is identified. Once all of the capabilities are identified, the capabilities objectives are established along with high-level capabilities requirements to satisfy the objectives. The capability requirements are then used to determine whether initial candidate computer systems will be able to satisfy the requirements.

Phase 1 - Step 3: Determine Computer Systems for NCS Consideration

This step in the process assists in determining which computer systems will be under consideration to be part of the NCS solution. It provides a process to help select the systems based on system capability availability, capability readiness, acquisition time, and acquisition cost (see Fig. 2). This process provides a library list of computer systems for each of the capabilities required for the NCS solution.

Phase 1 - Step 4: Determine NCS Solution to Satisfy the Mission

During this step, a library list of computer systems that satisfies each of the capabilities defined by the high level capability requirements will be available as part of a down select process. This step will identify potential computer system candidates to be considered into the NCS solution. The identification process will utilize a process to determine which computer systems are the "best" candidates in accomplishing the NCS capability objectives. A selection process enables the stakeholders to be able to provide a level of balance between objective and subjective decision-making in selecting the computer systems as a component of the preferred NCS solution.

Phase 2: Determining the Measure of Effectiveness of the NCS Solution

The purpose of Phase 2 is to evaluate the NCS solution based on the decision attributes in quantifying the NCS solution's effectiveness. This phase will evaluate the NCS solution based on the decision attributes selected (capability sustainment, mission reliability, and life cycle cost) and measure the effectiveness based on the estimation. The following sections describe each of the steps of Phase 2.

Phase 2 - Step 1: Evaluate NCS Solution Based on the Decision Attributes

The NCS solution will be evaluated based on the decision attributes that are related to the Measure of Effectiveness (MOE) construct. In terms of MOE, the NCS solution will consider capability sustainment (basic reliability), mission reliability, and capability life cycle cost. Each of the decision attributes will be quantitatively estimated and analyzed in determining the MOE of the NCS solution that could be further analyzed and evaluated.

a) Capability Sustainment Definition and Estimation:

Capability sustainment translated as basic reliability is considered to be a measure of sustainability and operations and support of a system. As defined in MIL-STD-785B, [5] "the measures of basic reliability such as Mean-Time-Between-Failures (MTBF) include all item life units (not just mission time) and all failures within the item (not just mission-critical failures of the item itself)." Basic reliability requirements apply to all items of the system.

In terms of computer systems, two primary components can affect basic reliability: software and hardware. The interrelationship between hardware and software is a primary driver that can affect the overall reliability of the system. The hardware's reliability would consist of all hardware elements of the system in terms of failure that are assessed based on failure rates of the hardware configuration items. [7] Similarly, software reliability can also be characterized in terms of the number of software components and its reliability based on the number of software failures that occur over time. As part of the informed decision making process, both hardware and software reliability and their dependencies must be mathematically formulated in order to estimate and calculate the overall reliability of the system.

b) Mission Reliability Definition and Estimation

"Mission reliability" is defined as the estimate of the probability the NCS will perform its required functions during the mission over a certain time period. This definition is based on the assumption that all mission essential items are ready and operational at the start of the mission. Furthermore, mission reliability is a system-level reliability metric that is a function of (1) the mission definition in terms of mission essential functions by mission phase and (2) the configuration and failure rates of the NCS essential items by mission phase. The mission must be defined and described in terms of the duration of each phase and the functions that must be accomplished for the NCS' mission success. The assurance of mission reliability can be attributed to systems with increased levels of redundancies and failovers. However, increasing the probability of mission success by improving the mission reliability affects basic reliability in the form of increased logistics overhead to include support, maintenance and costs. Therefore, there is an underlying dependency between basic and mission reliability considered as part of this research.

c) Life Cycle Cost Definition and Estimation

One of the requirements in the development of systems that are managed and operated by the DoD is a determined cost of its life cycle. [14] Systems developed within the defense acquisition model follow a cost model to support the affordability among all the phases of a system's life cycle to include material solution analysis, technology development, engineering and manufacturing development, production and deployment, and operation and support. [6] It is important to know the program's cost at particular intervals in order to ensure that adequate funding is available to execute the program according to plan. [11] "Affordability must be a performance consideration from beginning throughout the life cycle." [8] Similarly, the NCS solution will also consider a cost model as a measure of affordability in support of the NCS life cycle to satisfy a mission. (See Figure 3.)

Since the NCS solution will only be acquiring existing systems that are in development or systems that have already achieved their initial operating capabilities, the NCS solution will support two cost model components, (1) cost model for each of the constituent computer systems and (2) cost model for the NCS solution. [6] The first component is the costs associated with acquiring and engineering the computer systems specifically in developing, integrating, testing and deploying. These are cost drivers that involve engineering efforts for each of the computer systems that are part of the NCS solution. The second component is the costs associated with managing, utilizing, maintaining and supporting the NCS during its operational life cycle. The cost is a reoccurring cost throughout the NCS life cycle for as long as the operators utilize the NCS solution.

The cost structure and its elements are cost drivers in developing and sustaining an NCS solution throughout its life cycle. These cost drivers can be categorized by the life cycle phases of an NCS solution in the following cost structure elements table:

Phase 2 - Step 2: Determine Effectiveness Based on Decision Attributes

In this step, the NCS solution and the estimated decision attributes will be used to determine the MOE. The previous section determines the decision attributes based on a quantitative approach for measuring the attributes considered to be critical components of the MOE of the NCS. The question is how to balance all of the decision attributes that are considered important to determining a specific measure in determining the MOE of the NCS solution. Since this notional conceptual methodology is based on determining a solution to be considered based on specific decision attributes to calculate the effective measures of the system, the methodology will consider a process that is able to calculate these decision attributes based on weighted priorities. The weighted priorities take into account the importance of each of the decision attributes and prioritizes each of the attributes based on historical information and experiences of the decision stakeholders. Therefore, during this section, a generic hierarchy or ranking process shall be considered in order to provide a solution that relies on the judgments of experts and subject matter experts to provide a priority or weighted factor on area of importance for each of the measuring attributes. For instance, if the mission requires a higher factor in mission reliability, then the process will take into account the importance of the reliability of the mission. This also goes along with the decision attribute of capability life cycle cost having a priority weight over the other decision attributes. In this case, if the life cycle cost requires a higher priority, subject matter experts weight it according to its importance. Further research is required in this area in order to determine the best approach in determining the feasibility of the NCS solution based on the decision attributes considered.

Way Forward

The work being performed in this area will provide a well-defined methodology in which a program office can utilize a decision process to determine the best feasible approach for satisfying an emerging capability. The approach hinges on the utilization of current operational or developmental systems to fulfill user requirements by taking advantage of existing systems. This paper defined a methodology framework to explore the selection of systems that can, when combined, provide a means to satisfy an emerging capability by minimizing the number of systems for development and utilizing current operational system capabilities that are fielded.

In future work as part of the selection process, the NCS solution will be verified and validated by attaining a measurable metric based on selected decision attributes that help determine the NCS effectiveness. The measurement for the NCS effectiveness will provide information to determine whether an investment in developing the NCS solution can be a viable commitment to successfully satisfy the operational requirement for the users. There is continued work to be performed in this area; however, this paper allows us to review a notional conceptual methodology in identifying decision attributes and using them as part of a process to identify an NCS solution for consideration. We will continually strive to identify and to quantify the preferred NCS solution to satisfy operational requirements. This paper will be followed by a detailed methodology, effectiveness models, and applications that will be applied toward an NCS solution to be considered and addressed.

There will be continuing work to be performed in this area to include a detailed methodology, effectiveness models, and application to an existing operational or notional mission. This will be a continued effort in the area of effectiveness measure in identifying and quantifying the preferred NCS solution in satisfying an operational requirement.

Figures and Tables

Figure 1. Notional NCS Concept of Operations ( Click to view image )

Figure 2. Approach for Determining Feasible Candidate Computer Systems ( Click to view image )

Figure 3. NCS Life cycle ( Click to view image )

Table 1. Methodology Framework Phase One and Two ( Click to view image )

Table 2. Cost Structure Element ( Click to view image )

References and Notes

  1. Chelson, P. O. & Eckstein, R. E. (1971.) “Reliability Computation from Reliability Block Diagrams.” Jet Propulsion Laboratory. California Institute of Technology, Pasadena, California.
  2. National Research Council. (2015.) “Reliability Growth: Enhancing Defense System Reliability.” Washington, D.C. National Academy Press.
  3. DAU. (2016). “Measure of Effectiveness (MOE).”
  4. Department of Defense. (2013.) “Defense acquisition guidebook.” In Defense Acquisition Guidebook, Ed., 2013.
  5. Department of Defense. (1980.) “Reliability Program for Systems and Equipment Development and Production.” Vol. MIL-STD-785B. D. o. Defense, Ed., ed. Washington, D.C.
  6. Office of Secretary Defense. (2014.) “Operating and Support Cost-Estimation Guide.” Office of Secretary of Defense, Ed., ed. Washington, D.C.: Cost Assessment and Program Evaluation (CAPE).
  7. Friedman, M. A.; Tran, P. Y., & Goddard, P. I. (1995.) “Hardware/Software System Reliability Modeling.” in Reliability of Software Intensive Systems (Advanced Computing and Telecommunications Series). 1st Edition ed: William Andrew.
  8. Jaynes, R. A. S. C.; Simpson, T.; Mallicoat, D.; Francisco, J.; Mizell, W. & Cikovic, D. (2012.) “Managing O&S Costs - A Framework to Consider.”
  9. Mccallam, D. (2013.) “Improving Enterprise Security through Cybersecurity Architecture Views.”
  10. Office of the Secretary of Defense, C. (2011.) “DoD Financial Management Policy and Procedures DoD 7000.14-R.” D. o. D. F. M. Regulation, Ed., ed. Washington D.C.: Office of the Under Secretary of Defense (Comptroller).
  11. Office, U. S. G. A. (2009.) “GAO Cost Estimating and Assessment Guide.” GAO-09-3SP ed: GAO.
  12. Sproles, N. (2001.) “Establishing Measures of Effectiveness for Command and Control: A Systems Engineering Perspective.” 30.
  13. Under Secretary of Defense for Acquisition, T. a. L., or USD(AT&L). (2003.) “The Defense Acquisition System Directive 5000.01.” Vol. 5000.01. U. A. L. Department of Defense, Ed., DoD Directive 5000.1 ed. Washinton, D.C.
  14. Under Secretary of Defense for Acquisition, T. a. L., or USD(AT&L). (2015.) “Operation of the Defense Acquisition System 5000.02.” Vol. 5000.02. U. A. L. Department of Defense, Ed., ed. Washington, D.C.

Glenn Tolentino

Click to view image

Glenn Tolentino is a senior systems engineer for the Command and Control Department at Space and Naval Warfare Systems Center Pacific located in San Diego, Calif. During the past 23 years, Tolentino has been directly involved as a software and systems engineer in the design, development, integration and deployment of national level systems in the area of Command, Control, Computers, Communication, and Intelligence. Glenn’s research interests include Systems of Systems, Mission Reliability, Capability Sustainment, and Systems Effectiveness. He earned a B.S. degree in applied mathematics from San Diego State University and an M.S. degree in software engineering from Southern Methodist University (SMU).

Jeff Tian

Click to view image

Jeff Tian received B.S., M.S., and Ph.D. degrees from Xi’an Jiaotong University, Xi’an, China; Harvard University, Cambridge, Mass., USA; and the University of Maryland, College Park, Md., USA; respectively. He was with the IBM Toronto Lab from 1992 to 1995. Since 1995, he has been with Southern Methodist University, Dallas, Texas, USA, where he is currently a professor of computer science and engineering. He has been the associate director of the National Science Foundation Industry/University Cooperative Research Center for Net-Centric and Cloud Software and Systems (NSF NCSS IUCRC) since it was founded in 2009. Since 2012, he has also been a Shaanxi 100 Professor with the School of Computer Science, Northwestern Polytechnical University, Xi’an. His current research interests include software quality, reliability, usability, testing, measurement, and Web/service/cloud computing. Dr. Tian is a member of the ACM.

Jerrell Stracener

Click to view image

Jerrell Stracener is Professor of Practice and founding director of the Southern Methodist University (SMU) Systems Engineering Program. He teaches graduate-level courses in engineering probability and statistics, systems reliability and availability analysis, and integrated logistics support (ILS). He performs and directs systems engineering research and supervises Ph.D. student research. Prior to joining SMU full time in January 2000, Dr. Stracener was employed by LTV/Vought/Northrop Grumman where he conducted and directed systems engineering studies and analysis as well as reliability engineering activities. He was also ILS program manager on many of the nation’s most advanced military aircraft. Dr. Stracener was co-founder and leader of the SAE Reliability, Maintainability and Supportability (RMS) Division (G-11) and is an SAE Fellow and an AIAA Associate Fellow. Jerrell served in the U.S. Navy and earned both Ph.D. and M.S. degrees in statistics from SMU and a B.S. in mathematics from Arlington State College (now the University of Texas at Arlington).

« Previous Next »