The February 15, 2012 dinner meeting of the INCOSE Chesapeake Chapter started with dinner, conversation amongst officers, members, and guests, and a series of Chapter announcements. Once the announcements were completed, we transitioned to the auditorium for the night’s guest speaker, Clyde Smithson of JHU/APL. Mr. Smithson has spent the last 27 years performing system engineering, development, analysis, and assessment on military flight simulators and the Missile Defense Agency’s (MDA) Ballistic Missile Defense System (BMDS). Mr. Smithson has performed simulation and analysis on operational, tactical, and strategic military systems ranging from Theater and National Missile Defense sensor and weapon systems, Command and Control centers, to related CONOPS, TTP, and Doctrine.
Clyde began his presentation by taking a moment to define the terms “System,” “System of Systems,” and “Complex System.” The BMDS has characteristics of both a system of systems as well as complex systems on both a technical and a social level. It has characteristics of a system of systems in that its elements were developed independently, can operate independently of one another, are upgraded and enhanced independently and incrementally, and are distributed worldwide. The BMDS has characteristics of a complex system in that it has a large number of elements, has many interactions within and outside of the system, has no predetermined attributes, is loosely organized, has probabilistic system behavior, evolves over time, has subsystems pursuing their own goals, is highly affected by its operators, and is highly affected by its external environment. The MDA considers the BMDS to be neither a system of systems nor a family of systems, but rather a complex, constantly evolving “integrated system.”
Performing a comprehensive end-to-end performance assessment of the BMDS is an extremely difficult challenge. The BMDS is a globally-deployed system with sensors, weapons, and control functions distributed around the world. It is simply impossible to test the fielded system across all potential operating conditions (threat, environment, system configuration, etc.). MDA relies on a variety of venues to assess system performance including flight tests, ground tests, modeling and simulation, and subject matter expert assessments. The flight test venues provide the highest fidelity representations of the real world but only assess points in the overall set of operating conditions that the BMDS can experience. Ground tests utilize a combination of real world hardware and simulated representations and provide a high fidelity assessment over a broader set of conditions than flight test can provide. Modeling and simulation spans a variety of fidelity levels and simulates across a variety of operating conditions depending on the level of the models (engineering-level, mission-level, campaign-level). Finally, subject matter expert assessment provides the lowest fidelity representation but can provide the broadest assessment across the span of operating conditions.
Clyde discussed the methodology employed for the end-to-end BMDS performance assessment. Once the performance assessment problem is properly defined, MDA plans, develops, and executes a variety of events in its performance assessment campaign that comprise the full range of performance assessment venues described above. Finally, the results of the performance assessment campaign are analyzed, assessed, and reported to MDA leadership.
To date, the agency has found the development of an end-to-end BMDS performance assessment that its leadership can collectively stand behind to be a challenging task. On the technical side, there is no fully accredited system-level medium or high fidelity modeling and simulation tool. The problem is as challenging socially as it is technically, as it is difficult to get all of the BMDS’s stakeholders to agree upon and stand behind the assessment’s methodology and results. The agency continues to work to improve and refine its assessment methodology to produce a performance assessment that its many stakeholders deem as technically correct and can stand behind.
Check out Clyde’s slides from his presentation below, and learn the definition of a “Wicked Problem”