By Charles Piersall and Franklin Grange


Abstract.

Informal and casual consideration of Intended Use in Modeling and Simulation practice can pose programmatic risks in acquisition, especially in system-of-system contexts, such as the Ballistic Missile Defense System. Leveraging lessons learned from intense reliance on system-of-system simulations, the Missile Defense Agency is formalizing specification of Intended Uses to mitigate those risks and to improve effectiveness and affordability of the Agency’s diverse simulations.

Introduction

The objective of this article is to present an approach in development by the Missile Defense Agency (MDA) for specifying the Intended Use (IU) in Modeling and Simulation (M&S) applied in a system-of-systems (SoS) context. The MDA’s mission is to develop, test, and field an integrated, layered, Ballistic Missile Defense System (BMDS) to defend the United States, its deployed forces, allies, and friends against all ranges of enemy ballistic missiles in all phases of flight. The BMDS is a SoS.

Many practical and legal constraints force SoS acquisition organizations like MDA to depend on M&S for diverse needs, such as assessment, training, exercise and concept development [1]. Typically, SoS M&S relies heavily on the compositions (sometimes called “federations”) of legacy M&S independently developed for the constituent systems of the SoS. Disparities among the original IUs of the constituent systems’ simulations complicate or even confound a SoS M&S IU, which itself usually differs from the constituent simulations’ original IUs.

Suitability of BMDS M&S for an IU is at the core of an Accreditation process [2] incorporating formal analysis of risks to acquisition and warfighting from using a specific simulation supporting essential decisions, such as deployment. Thus, MDA developed a SoS M&S IU approach to improve outcomes of its BMDS M&S systems engineering and accreditation processes. MDA’s SoS M&S IU approach has features and implications useful for SoS M&S engineering in other M&S-reliant domains, such as Command, Control, Communications, Computers, Intelligence, Surveillance and Reconnaissance (C4ISR) and space exploration.

In the following sections, we consider first what M&S IU really means, and how widespread usage of the term without any common definition creates practical difficulties—especially for SoS M&S. We present our lessons learned about programmatic risks that M&S IU misspecification (or outright ignorance) can create. From those lessons emerged MDA’s new practice for BMDS M&S IU specification, which focuses on the needs of the stakeholder analyst as a consumer of M&S. A subsequent case study shows the implications of ignoring a SoS M&S IU specification. Finally, we present our vision for migrating MDA’s BMDS M&S IU specification to a model-centric systems engineering environment that can increase the reliability and timeliness of BMDS M&S development and use.

“Everybody Knows What M&S ‘Intended Use’ Means…Right?”

Numerous M&S authorities describe the importance of specifying an M&S IU but neglect to define “IU” explicitly (e.g., [3], [4]). For example, [5] presents the following circular definition: “An M&S application’s Intended Use refers to the explicitly and clearly defined purpose for which the application is intended for use.” Representative of DoD interest in M&S accreditation for affordable reuse, MIL-STD 3022, “Documentation of Verification, Validation and Accreditation (VV&A) for Models and Simulations,” also does not define “IU” but does require documentation of “the problem to be addressed by the M&S and its associated data, including the system or process being represented and the role it plays in the overall program” [6].

Notwithstanding arguments about inadequacies of M&S requirements engineering, we observe that effective practitioners invariably develop new M&S with one or more definite purposes in mind. Thus, for new M&S development, an unambiguous, testable definition of M&S “IU” might seem unnecessary. Many authorities thus imply that M&S “IU” is an important, essential but undefinable concept akin to “point,” “line,” or “plane” in geometry.1

Working in a SoS M&S enterprise emphasizing affordability and simulation asset reuse, we drew from our “lessons learned” the conclusion that M&S “IU” must be formally defined so that IU specifications may be compared and tested for clarity, completeness and compatibility. The following review of those lessons learned and programmatic risks we encountered motivates our recommendation for formalizing SoS M&S IU definition and specification in a SoS domain.

“Wait a minute…we’re not talking about the same IU?”

Applying M&S to the BMDS, a SoS, we recognized that BMDS M&S is itself a SoS engineering endeavor. In 2006, the MDA established the BMDS M&S Program including a goal of assessing BMDS capabilities with end-to-end, constructive simulations.2 High-resolution, “engineering-grade” M&S previously and independently developed for Element program acquisitions (i.e., the Elements’ overarching IUs) comprise the BMDS simulations of Element interceptors, sensors and battle management interoperating as a BMD SoS. At first, we expected that composing an acquisition-focused BMD SoS M&S from acquisition-focused Element M&S constituents should be hard but straightforward and predictable work. We envisioned that disciplined, collaborative requirements engineering across both the BMDS M&S and the Element M&S was necessary and sufficient for successful implementation of the BMD SoS M&S. What we encountered were conflicts among the Element M&S IUs, as well as Element IU conflicts with the BMDS M&S IU.

• The mission contexts of Element M&S IUs vary substantially; some, such as the BMDS Command, Control, Battle Management and Communications (C2BMC) Element, exist solely for a BMDS mission, while others, such as Aegis BMD with fleet defense, have standalone or additional non-BMD missions

• Element M&S IUs are rarely “Legos™,” as each Element’s acquisition engineering M&S IU may be very specific to an independent Element program; e.g., enemy missile threats to which an Element is engineered might be very specific, and the M&S IU will also be as specific

• In most cases the Element programs, before developing their M&S, did not receive M&S composability requirements to enable future BMDS M&S IU; in result, Element M&S architectures complicate integration and interoperability; an example is organic, tightly integrated Radar Cross Section (RCS) modeling of an enemy missile threat by two Elements that the BMDS M&S IU requires to have a common, consistent RCS measurement (e.g., adjusted, of course, for sensor characteristics, viewing angles, and threat orientation).

These kinds of M&S IU conflicts embodied risks that too often we saw become programmatic issues. Preparing for BMDS M&S integration, the Element M&S developers undertook reengineering their simulations for interoperation with others’ simulations; in their original, standalone versions, the Element simulations represented the effects of interactions happening, but not the full interactive process (e.g., a BMDS battle manager function assigning a missile intercept to one tracking Element versus another). Discovery of the interoperability and integration requirements was an unexpectedly prolonged trial-and-error process, and integration lead times to prepare BMDS assessments were neither predictable nor affordably short to sustain BMDS analysts’ desired work tempos.

In the early BMDS M&S IU, the BMDS architecture was far more loosely coupled than at present, and omitting some Element interactions or common, consistent threat or environment modeling did not necessarily thwart the overall BMDS M&S IU. Nevertheless, root causes of errors discovered during runs-for-record often traced at least in part to previously unrecognized IU conflicts among the Element M&S and the BMDS M&S. As the near-future BMDS architecture becomes more tightly coupled, these kinds of IU-based errors become much harder and costlier to correct.

Like C4ISR, planetary space missions, and many other SoS’, comprehensive testing of the BMDS is generally so unaffordable, impractical, unsafe or illegal (by treaty) that M&S is the only means to provide estimates of the capabilities and limitations of the SoS. However, while nearly always safe and legal, M&S must be affordable, practical and timely. Clarifying, understanding, and aligning M&S IUs mitigates at least some cost, schedule and scope risks of BMDS M&S.

Emerging MDA Practice for Specifying M&S IU in BMDS M&S

An engineering process of selecting a legacy M&S application/tool or developing a new M&S application/tool begins with creating a clear statement of the M&S IU and a description of the associated M&S Capability Needs from the perspectives of stakeholder analysts (see Section 2 of [9]).3 For the inferences stakeholders want to draw about the BMDS, stakeholder analysts need particular kinds of M&S outputs, and typically require results from particular kinds of M&S runs or experimental conditions. These M&S Capability Needs encompass the set of analysis capabilities that M&S should enable in order to adequately achieve its IU (e.g., “The BMDS M&S shall provide output X under experimental conditions Y to enable the stakeholder analyst to calculate BMDS metric Z”).

The position of the IU within the Systems Engineering Requirements Process is shown in Figure 1. The depicted process pertains to both unitary systems and SoS. For both legacy and new M&S, the stakeholders are responsible to articulate a set of M&S Capability Needs that comprise essential parts of the M&S IU. If the stakeholder who utilizes results of an M&S tool misunderstands what IU drove its development, the stakeholder analyst is at risk of misconstruing the M&S results.

MDA currently requires several parts for a well-defined M&S IU. From the stakeholder analyst’s perspective, an M&S IU is not a capability, solution or implementation, but part of a clear, complete analysis problem statement. MDA is establishing a standard IU specification that contains the following components:

1. Title—For ease of communication and reference among stakeholders and developers

2. Description—Five essential elements:

• Narrative identification of the analysis problem

• Narrative description of what stakeholder analyst tasks or processes the IU supports

• Narrative enumeration of specific simulation outputs the stakeholder analyst will use in the analyst’s processes; should identify products or documents that the IU has inputs to or creates directly

• Narrative description of special conditions or experimental designs under which the simulation should run to produce the analyst’s needed specific simulation outputs; should include any needed simulation calibration methods and associated data

• Narrative description of “controls or governances” identifying documents or processes to which the IU conforms, such as stakeholder test objectives memorandum or a test concepts of operations (CONOPS)

3. Key Attributes—For a specific M&S domain, standard aspects and values that stakeholders and developers agree clarify understanding about the M&S IU; for MDA’s BMDS M&S IU, these include the following:4

• Focus: From narrow consideration of a component (e.g., specific radar) to broad scope of the end-to-end BMDS

• Epoch: Current or future version of the BMDS Architecture represented

• Simulation Type: Constructive, Virtual, Live [8]

• Fidelity:5 Degree of simulation faithfulness

• Uncertainty Quantification: Methods for representing uncertainties and propagating them to simulation outcomes

• Tactical Interoperability: Accuracy by which the M&S environment recreates tactical Element or Component external interfaces

• Stimulation Interface: Level of detail contained within stimulation data distributed through the system; includes threat, modeled communication networks, environment, lethality (interceptor-target interaction physics)

• Operator Screens: Degree to which user interfaces can provide an Operator-in-the-loop with a realistic training or exercise experience

4. Identified Stakeholders—Listing of stakeholders who either “own” the M&S IU or make decisions with the simulation results

A Case Study of Catastrophic Risk from Ignoring SoS M&S IUs

The Crandall Canyon Mine case study reveals how fatal consequences may result from inadequate consideration of M&S IUs in safety engineering.

In August 2007, the Crandall Canyon Mine in Utah collapsed and trapped six workers. Three more workers died in an additional collapse during rescue operations that ultimately failed to recover the original six victims’ bodies. Investigating the disaster, the US Mine Safety and Health Administration (MSHA) reviewed not only mine conditions and operations practices, but also the engineering design of the mine [10]. MSHA determined that improper engineering analysis with two stress simulations was one of the root causes of the collapses killing both the miners and the rescuers.

Mine engineering analysis considers both geologic conditions and mining methods. The Crandall Canyon Mine employed methods of “longwall mining” and “retreat mining” not only to maximize coal recovery, but also to ensure mine structural integrity for the safety of workers and equipment. As longwall mining proceeds through coal beds, pillars of valuable coal left by mining operations support the roof in the mined space. When advance through the space completes, retreat mining involves partial or complete removal of some pillars as workers withdraw equipment back from the mined space. While pillar recovery may trigger some roof collapse, proper retreat mining ensures that roof collapse occurs safely and only in the immediate area of a pillar being extracted. Mine design analysis considers such factors as the mechanical properties of the pillar material, the depth of the mining space and the condition of the prospective roof to establish the needed size and spacing of pillars for productive but safe mining.

MSHA found that undersized pillars contributed significantly to the Crandall Canyon Mine collapse. As retreat mining proceeded, excessive rising stress on remaining pillars finally triggered one pillar failure that started a ripple of load shocks overstressing and collapsing other pillars. MSHA focused specifically on the use of two mine engineering simulations, “Stress and Displacement Calculations” (LaModel) and “Analysis of Retreat Mining Pillar Stability” (ARMPS). While MSHA agreed with the suitability of both models for the mine engineering IU, MSHA found that the Crandall Canyon Mine’s engineering contractor failed to apply LaModel and ARMPS together (i.e., effectively as a SoS M&S) in accordance with their M&S IUs, which required their initial calibration with the same real-world geologic data from the Crandall Canyon Mine.

This case study provides a sobering lesson that the associated data and stakeholder analyst processes are an essential part of a SoS M&S IU, and they must be applied consistently to the M&S components involved in the SoS M&S. For large, complex SoS’ like C4ISR systems or the BMDS, document-centric M&S IU analysis and specification rely strongly on systems engineers’ broad understanding of the entire SoS scope. The case study shows how engineers may fail to specify and apply an M&S IU for even a much narrower scope than the BMDS. The following section outlines MDA’s M&S engineering initiatives to increase the reliability and timeliness of SoS M&S IU analysis and specification through automation.

A Hopeful Future: Formalized M&S IUs for Model-Based Systems Engineering

MDA’s BMDS M&S Program plans to update and incorporate an M&S IU specification into Model-Based Systems Engineering (MBSE) [12] of MDA’s BMDS M&S. Implementation of MBSE for BMDS M&S represents a transition from document-based systems engineering to an integrated Systems Modeling Language (SysML) models [13] of requirements, structure, behavior and parametrics (algorithms, quantities-of-interest, units-of-measure, etc.). MDA is currently developing the MBSE infrastructure of engineering processes, methods, standards, training, staffs and tools for Model-Based SoS Engineering (MBSoSE) of BMDS M&S. Clear, complete M&S IUs are essential to sustained MBSoSE of BMDS M&S to support decisions about the best M&S components to use in standalone or compositions to fulfill stakeholder needs.

Some benefits MDA anticipates from MBSoSE including M&S IUs are the following:

• A repository of M&S IU information updated as M&S constituents evolve

• Automated traceability and allocation of M&S IU information to stakeholder requirements; M&S logical standalone or composition design; M&S behavior; and M&S parametrics

• Automated design verification

• Automated detection and analysis of M&S IU errors, contradictions or gaps in addressing stakeholder needs, derived requirements and logical design

• Clear, consistent, traceable, measurable, testable requirements allocated to the Element and overall BMDS M&S developer organizations (e.g., to put on contract)

Complementing MDA’s transition to MBSoSE with M&S IUs are changes in BMDS M&S governance and culture that are not yet fully defined. Realizing some of the above benefits also requires evolution in MBSE tool technology and the SysML language.6 We anticipate those technical changes will occur concurrently with MDA’s BMDS M&S governance and culture evolution. In the future we hope to report on best practices and lessons learned from MDA’s transition to MBSoSE of BMDS M&S.

Summary

MDA has formalized specification of BMDS M&S IU to mitigate programmatic risks and to improve affordability and outcomes of M&S-based acquisition activities. Lessons learned in BMDS M&S informed MDA’s specification of a BMDS M&S IU standard focused on stakeholder analysts as the consumer of M&S results. Disciplined use of MDA’s SoS M&S IU approach helps avoid costly or even catastrophic risks of misapplying M&S. Though the original SoS M&S IU specification is document-centric, MDA anticipates its transition to future model-centric processes with MBSoSE automation increasing both affordability and the tempo of M&S-based acquisition activities.


References and Notes

References:

1. Department of Defense. Ballistic Missile Defense Review Report. Washington, DC, 2010.

2. Department of Defense (DoD) Instruction 5000.61, DoD M&S Verification, Validation, and Accreditation (VV&A). 9 Dec 2009.

3. Turnitsa, Charles, Jose J. Padilla, and Andreas Tolk. “Ontology for Modeling and Simulation.” Proceedings of the 2010 Winter Simulation Conference. Baltimore. IEEE, 2010. 643-51. 4. Cook, David A., and James M. Skinner. “How to Perform Credible Verification, Validation, and Accreditation for Modeling and Simulation.” Crosstalk, the Journal of Defense Software Engineering (2005): 20-24. 5. Balci, Osman. “How to Successfully Conduct Large-Scale Modeling and Simulation Projects.” Proceedings of the 2011 Winter Simulation Conference. Phoenix. IEEE, 2011. 176-182.

6. Department of Defense (DoD) Standard Practice. MIL-STD-3022 with Change 1, Documentation of Verification, Validation, and Accreditation (VV&A) for Models & Simulations. 5 April 2012.

7. “Primitive Notion.” Wikipedia. Wikimedia Foundation, 23 Feb. 2014. Web. 20 May 2014.

8. Page, Ernest H., and Roger Smith. “Introduction to Military Training Simulation: A Guide for Discrete Event Simulationists.” Proceedings of the 1998 Winter Simulation Conference. Washington, DC, IEEE, 1998. 53-60.

9. Caine, Lisa, Bobby Hartway, Danny Thomas, and Joe Hale. “Incremental Quantification of VV&A “Levels” for Integrated Management of Modeling & Simulation Tools.” Proceedings of the Huntsville Simulation Conference HSC 2006. The Society for Modeling and Simulation International. 17-19 Oct. 2006.

10. U.S. Mine Safety and Health Administration, “Report of Investigation: Underground Coal Mine Fatal Underground Coal Burst Accidents, August 6 and 16, 2007, Crandall Canyon Mine, Genwal Resources Inc, Huntington, Emery County, Utah,” ID No. 42-01715, July 10, 2008.

11. Gross, David C. Report from the Fidelity Implementation Study Group. Rep. no. SISO-REF-002-1999. Orlando: Simulation Interoperability Standards Organization, 1999.

12. Weilkiens, Tim. Systems Engineering with SysML/UML: Modeling, Analysis, Design. Amsterdam: Morgan Kaufmann OMG/Elsevier, 2007. 

13. Friedenthal, Sanford, Alan Moore, and Rick Steiner. A Practical Guide to SysML: The Systems Modeling Language. Waltham, MA: Morgan Kaufmann, 2012.

14. Zeigler, Bernard P., and Phillip E. Hammonds. Modeling & Simulation-based Data Engineering: Introducing Pragmatics into Ontologies for Net-centric Information Exchange. Oxford: Academic, 2007.

15. Tolk, Andreas, Saikou Y. Diallo, and Jose J. Padilla. “Semiotics, Entropy and Interoperability of Simulation Systems—Mathematical Foundations of M&S Standardization.” Proceedings of the 2012 Winter Simulation Conference. Berlin, IEEE, 2012.

Notes:

1. “Point,” “line” and “plane” are examples of a primitive notion, a concept undefined by previously defined concepts, and often motivated by intuition, common sense or everyday experience [7]. We find M&S practitioners inclined to specify IUs as “primitive notions” about which all M&S stakeholders should agree. In System- of-Systems (SoS) M&S engineering, diverse stakeholders using the same words with different meanings can disagree about an M&S IU specification without recognizing any disagreement. Thus, for success of M&S SoS engineering, we argue that M&S “IU” in SoS cannot be a primitive notion like those basic ideas of geometry.

2. A “constructive simulation” is a pure software implementation of a model of the system of interest of “real” (or envisioned) people, hardware, software, other facilities, inputs, outputs and processes [8]. While real simulation operators stimulate (i.e., make inputs, initiate runs) to such simulations, the operators are not otherwise involved in determining the simulation outputs (e.g., operators do not interact as “players” or “trainees” with the simulation during a run). Constructive simulations typically support acquisition engineering activities.

3. In DoD practice, stakeholder analysts are typically the users of the results of simulation runs (experiments). Analysts are often not M&S developers and maintainers. In the experience of one of the present authors, commercial practice tends to combine the roles of stakeholder analyst and M&S developer. Commercial practice is also infrequently concerned with simulation reuse or simulation compositions for SoS engineering.

4. MDA has not publicly released Key Attribute values for M&S IU.

5. “The degree to which a model or simulation reproduces the state and behavior of a real world object or the perception of a real world object, feature, condition, or chosen standard in a measurable or perceivable manner; a measure of the realism of a model or simulation; faithfulness” [11].

6. “Pragmatics” is a term used in formal logics of Ontologies and the Semantic Web. Both Zeigler, et al [14], and Tolk, et al [15], provide a formal mathematical definition of “IU” not as a “primitive notion,” but as a pragmatic in terms of other concepts from Ontologies. The benefit of defining an IU as a mathematical pragmatic is machine readability (e.g., in a future, more formal SysML grammar, or with the Web Ontology Language, OWL) and machine decidability about the congruence of an M&S IU to stakeholder needs (e.g., in SysML design verification of an M&S composition). The future MBSE capabilities will slash the lead time for SoS M&S systems engineering and integration, and increase the tempo of M&S-based analyses of large, complex SoSs.


Charles Piersall

Click to view image

Charles H. Piersall III, P.E. is the Deputy Associate Director for Modeling & Simulation Operations at the Missile Defense Agency (MDA). He earned a Bachelor’s Degree in Marine Engineering from the US Naval Academy and a Master’s Degree in Applied Science from the Naval Postgraduate School. He is a licensed Mechanical Engineer and qualified Navy Nuclear Engineer. He is currently serving as the President of the National Society of Professional Engineers (NSPE) Colorado.
Mr. Piersall has extensive experience in leading the development of complex Ballistic Missile Defense (BMD) models and simulations. He pioneered the use of Agile Software Development methods for multiple models and simulations within the MDA. He also led the development and implementation of a complex end-to-end simulation of the Ballistic Missile Defense System that was used on Capitol Hill to inform US Law Makers of the complexities of Missile Defense.

Phone: 719-721-7438
E-mail: chuck.piersall@mda.mil

Franklin Grange

Click to view image

Dr. Franklin E. Grange is a Scientist at ISSAC Corporation in Colorado Springs, CO. His involvement with BMDS M&S systems engineering, development and application spans 29 years. His career in simulation and analytics has addressed problems in defense, intelligence, high-tech, natural resources, banking, telecommunications, manufacturing and supply chain. He holds PhD and MS degrees in Mineral Economics/Operations Research from the Colorado School of Mines (CSM), and a BS in Petroleum Refining Engineering also from CSM.

Phone: 719-721-2228

E-mail: franklin.grange.ctr@mda.mil


« Previous Next »