A Framework for the Accreditation of Simulation-Based Experiments

Abstract

The Modeling and Simulation Branch (MSB) of the Maneuver Battle Lab (MBL) conducts simulation-based experimentation in support of Army Capabilities Managers (ACMs) for the United States Army maneuver formations and Cross Functional Teams (CFTs). To be credible, experiments must achieve the customer’s experiment objectives and provide valid data that can answer customers questions required to inform decisions. The two critical outputs of the experiments are a ten-day initial insights report, which provides senior leaders with initial findings, and the experiment final report, which answers all of the objectives and learning demands in detail. The MSB has earned multiple Department of the Army (DA) awards for its work. No experimentation partner/customer has ever disputed the findings contained in any reports. Despite the positive results and use of experiment findings to inform Army decisions, the MSB has been asked by peers in the research community or senior visitors “who accredits your experiments?” The MSB has interpreted this as a question of the validity of its work. The DA has no individual or organization that accredits simulation-based experimentation. If such an individual or agency existed, how would they accredit an experiment? If the models and simulations used have an accreditation memorandum, as required by Army Regulations, would that serve the purpose? Would other aspects require accreditation? If so, what individual or organization would be qualified to provide an accreditation? The MSB has developed a process to meet the intent of accrediting its experiments. This paper proposes a framework for the accreditation of simulation-based experiments for use by other Army organizations and Services. It will conclude with other potential uses of the methodology and areas for continued development.

About the Authors

Tom Yanoschik is a graduate of the United States Military Academy and holds a Masters Degree from the University of Texas at Austin. He is a retired Artilleryman from United States Army. His assignments included service in cannon artillery and multiple launched rocket system units and as a Fire Support Officer in the 3rd Ranger Battalion. Tom currently serves as the SAIC Site Manager for the Modeling and Simulation Branch (MSB) of the Maneuver Battle Lab (MBL), Fort Benning, Georgia. He has participated in virtual and constructive experimentation both locally at the MBL and distributed through the Battle Lab Simulation Collaborative Environment (BLSCE). Tom is a certified Program Management Professional (PgMP), Project Management Professional (PMP) and Modeling and Simulation Professional (CMSP). Chris Willis is a graduate of the United States Military Academy and holds a Masters Degree from Long Island University. He is a retired United States Army Infantry Officer. His assignments included service in Afghanistan, Iraq and Korea. Chris, an Army Civilian, currently serves as the Deputy Director of the Maneuver Battle Lab (MBL), Fort Benning, Georgia. He has led virtual and constructive experimentation both locally at the MBL and distributed through the Battle Lab Simulation Collaborative Environment (BLSCE). Chris is a Certified Modeling and Simulation Professional (CMSP) and holds a professional certificate in modeling and simulation from the University of Alabama, Huntsville.

A Framework for the Accreditation of Simulation-Based Experiments Tom Yanoschik Maneuver Battle Lab Fort Benning, Georgia thomas.j.yanoschik@saic.com

Chris Willis Maneuver Battle Lab Fort Benning, Georgia christopher.r.willis2.civ@mail.mil

Introduction

The Modeling and Simulation Branch (MSB) of the Maneuver Battle Lab (MBL) conducts simulation-based experiments (SIMEXp) in support of the US Army Maneuver Capabilities Integration Directorate (MCDID), the Army Capability Managers (ACMs) for Army maneuver formations, and various Army Cross Functional Teams (CFTs). The MSB holds quarterly synchronization meetings with partners to plan experimentation for the next two years; partners brief desired objectives and the MSB looks for synergies among the partners. A government project officer (PO) and a contract PO lead the coordination and synchronization process to ensure not only that the experiment planning is synchronized but also that the planning will result in an experiment that produces outcomes that answer the partner’s experiment objectives. The SIMEXp may not result in an acquisition decision or change to US Army doctrine, but they inform decisions that lead to those actions. Some of the outcomes of recent SIMEXps include:

• Informing the Optionally Manned Fighting Vehicle (OMFV) Request for Proposal (RFP) and Capability Development Document (CDD)
• Support to Cross Functional Team (CFT) Next Generation Combat Vehicle to scope the concept
• Refinement to Robotic Combat Vehicle (RCV) program requirements
• Informing the Multi-Domain Operations (MDO) Concept
• Assessing an Adaptive Command and Control Web (AC2W) ability to execute with Artificial Intelligence (AI)
• Informing AC2W/Human Interface Requirements
• Informing AC2W strike requirements
• Assessing Modeling and Simulation (M&S) capability to model intelligent behaviors for swarm UAS

The MBL has never had a partner refute findings from its experimentation outcomes or dispute a report. The MBL has earned the annual Department of the Army Modeling and Simulation Office (AMSO) award for excellence in modeling and simulation (M&S) four of the last six years. Every year, the MBL has to turn away potential experiments due to an already high operational tempo. Nonetheless, during organizational briefings several visitors have asked “who accredits your experiments?” The question they ask is not, “who accredits the simulations for their intended use in your experimentation?” They want to know what outside agency gives its stamp of approval for the entire experimentation process and the findings that are published in the final report. By Department of the Army (DA) regulation, the M&S used for any purpose must be accredited for the intended use. Or, as Dr. Mikel Petty expressed, that the M&S must be a sufficiently accurate representation of the real world for its intended use. In essence, that “the model is the right one for the job” (Petty, 2015). For purposes of this paper, the term “intended use” is defined as the M&S containing all of the components and interactions at the level of fidelity necessary to achieve the goals and objectives of that use; in this case the experimental objectives. However, that definition excludes the activities associated with execution of the experiment itself, including the data used to drive it (e.g., scenario, human-in-the-loop tactics, techniques, and procedures (TTPs)). To obtain an overall accreditation of an Army maneuver-focused SIMEXp, the accrediting individual or team would require cross-functional expertise to declare that the complete experimental process (not just the M&S) accurately describes the real world sufficiently to provide valid data for the intended use of those findings. Clearly, the accredited M&S is the foundation of a legitimate SIMEXp and its accreditation ensures that coded algorithms, input data, and entity actions and interactions with other entities function appropriately for the intended use. Accreditation of the overall experiment to account for other factors such as the analytical techniques used to interpret the output data and operational employment of the entities in the simulated combat operations is required for the outcomes of the SIMEXp

to be reliable. To be credible, experiments must achieve the customers’ experiment objectives and must provide valid data that can answer customers’ questions that will ultimately inform decisions. There is currently no known accrediting agency for military experiments. Therefore, a cross-functional framework for military simulation-based experiment accreditation is required to ensure the credibility of findings. This paper presents a framework based on the accreditation process used by the MBL and discusses potential uses of the framework by other organizations.

What is Accreditation?

The United States Army defines verification, validation, and accreditation in Department of the Army Pamphlet 5-11 (Verification, Validation, and Accreditation of Army Models and Simulations) as: (1) Verification is the process of determining that an M&S accurately represents the developer’s conceptual description and specifications. Verification evaluates the extent to which the M&S have been developed using sound and established software-engineering techniques. (2) Validation is the process of determining the extent that an M&S is an accurate representation of the real world from the perspective of the intended use of the M&S. Validation methods include expert consensus, comparison with historical results, comparison with test data, peer review, and independent review. (3) Accreditation is the official determination that a model, simulation, or federation of M&S is acceptable for use for a specific purpose. The nature of a military SIMEXp is different in several ways from traditional scientific (laboratory) experiments. (DA PAM 5-11, p. 7) Accreditation is not unique to the military or to SIMEXps. Numerous professions such as medicine, law, teaching, plumbing, watchmaking, and automotive mechanics require individuals to be accredited to work within their profession or perform to a certain level. Colleges and Universities undergo accreditation processes; each has a Staff and Faculty Accreditation process. Sports such as boxing, marathon races, fencing, and automobile racing all have governing bodies who create standards and sanction competition. Seeking medical care, legal defense in court, or repair of an expensive watch from an accredited professional does not guarantee a successful outcome, but it does provide one confidence that the individual whose services you employ has a given level of training and experience certified by a centralized body. It may not matter where you bring an old “junker” automobile for basic repairs, but if the electronic transmission on your Land Rover malfunctions, a technician at the Land Rover dealership assures you a certified level of training and reach-back in addressing the issues your vehicle is experiencing. You can defend yourself in court having not attended law school or passing the bar exam. In fact, some citizens may opt to fight a traffic ticket, but few would fight a murder charge without an attorney. Within certain industries, organizations, and businesses may earn certifications or be required to be certified in order to function. The International Organization for Standardization certifies quality within manufacturing, occupational health and safety, and specific products. Accreditation assures the customer/client of the credibility of services and qualifications. For military use of simulations, the Defense Modeling and Simulation Enterprise (DMSE) states that verification, validation, and accreditation (VV&A) of a model or simulation is to determine whether a model, simulation, or federation should be used in a given situation (US Department of Defense. (2018), p. 8). Therefore, its credibility must be established by evaluating fitness for the intended use. Given that this accreditation is required for the M&S, it follows that the same would be true of the overall experiment. Accordingly, a methodology is necessary by which one can determine the credibility for the entire experiment, not just the model or simulation.

The SIMEXp Accreditation Methodology

The terms accreditation and validity refer specifically to the M&S for use in a specific experiment, which is different than accrediting the entire simulation experiment itself. There is no known overarching methodology for accrediting an entire simulation experiment. Specific to M&S accreditation, Department of the Army Pamphlet 5-11 (Verification, Validation, and Accreditation of Army Simulations) states in Paragraph 4-2: Acceptability criteria. The accreditation process answers the question; “Will this M&S meet my objectives?” The M&S application sponsor appoints an accreditation agent to establish a set of acceptability criteria by which to determine the suitability of the M&S for the intended use. These acceptability criteria are unique to each problem and give key

insights to potential solutions. Acceptability criteria become, therefore, a set of standards that a particular M&S must meet to be accredited for a given use. (DA Pam 5-11, p.40) Most military SIMEXp are based on the future—a weapon, sensor, or military formation that may not currently exist or only exist at a low technical readiness level. Scientific, laboratory-based experiments generally use tangible realworld subjects, materials, or devices on which the experiment is focused. There are organizations and bodies of scientists who direct procedures for the conduct of generally accepted scientific research and experiments. The closest body of knowledge that governs military experimentation is the Command and Control Research Program’s (CCRP) Code of Best Practices, Code of Best Practices for Campaigns of Experimentation, and The Logic of Warfighting Experiments. All three books outline methodologies for ensuring credible military experimentation. None of the books’ outline who can determine accreditation or validity- only that it must exist. For purposes of the SIMEXp, clearly the suitability of the M&S plays a critical role in the overall accreditation of the overall experiment. In The Logic of Warfighting Experiments, Richard A. Kass states: Nonetheless, a valid experiment provides sufficient evidence to make a conclusion about the truth or falsity of the causal relationship between the manipulated variable and its effect….I will continue to use the popular term valid experiment, understanding that a valid experiment is a well-designed experiment that provides sufficient evidence for valid conclusion. Notice that the definition above uses the phrase “sufficient evidence.” This is to reinforce the notion that the evidence is never absolute. All evidence, even in experimentation, retains some judgment. Chapter 8 will discuss the notion that there is no such thing as a perfect experiment and 100 percent validity certainty is unattainable. However, well-designed experiments can provide sufficient evidence to make a reasonable case for causality. The framework in this book presents four logical requirements to achieve sufficient validity in experiments to support causal inferences about capabilities and their effects. Requirement 1: the ability to use the potential cause. Requirement 2: the ability to observe an effect. Requirement 3: the ability to isolate the reason for the observed effect. Requirement 4: the ability to relate the results of the experiment to the real world. (Kass, p.26) In order to achieve the four requirements identified, an accredited SIMEXp must consist of numerous processes planned and executed in accordance with clearly defined best practices, industry standards, and procedural techniques. The summation of the accredited processes is what accredits a SIMEXp. If any of the subcomponents is not to standard, then the overall SIMEXp should not be considered accredited. Although separate, the processes have many interdependencies that can affect the success or failure of a SIMEXp. For example, if a critical area of analysis is observing Mission Command on the Move, but the tactical scenario used is a static defense, analysts may never be able to observe and report on the learning demand. If data collection requires the recording of the sensing of all entities by other entities, but poor network performance results in dropped packets that result in lost data, the SIMEXp cannot be accredited. A catastrophic or even less than optimal subcomponent can in and of itself risk SIMEXp accreditation or the inability of subcomponents to achieve the requirements of other subcomponents can jeopardize the overall experiment accreditation. The four areas that need to be accredited are models and simulations, the tactical/operational scenario, the physical and computational environment, and the analysis plan (See Table 1, next page).

Analysis Measures of Merit Measures of Performance Measures of Effectiveness Data Collection Data Reduction Computational Methods Role Player Surveys Peer Review of Report

Table 1. Components of SIMEXp Accreditation

Modeling & Simulation	Tactical / Operational Scenario	Physical & Computational Environment	Analysis
Modeling	Appropriate scenario and mission	Network	Measures of merit / performance / effectiveness
Interoperability of simulations	Appropriate terrain	Software	Data collection
Verification	Mission command systems	Hardware	Data reduction
Validation	Threat organization & employment	Experiment footprint	Computational methods
Authoritative data	Civilians on the battlefield	Training of participants	Role player surveys
Accreditation	Accreditation	Knowledge management · Accreditation	Peer review of report · Accreditation

Overall accreditation of the SIMEXp is the sum of the individual accreditations.

Models and Simulations

As indicated by the name, the M&S is an important component of the SIMEXp and is arguably the most critical aspect that requires accreditation. The M&S not only must perform correctly with the accurate data and appropriate data collection tools but also must be appropriate for the intended purpose of the SIMEXp. For example, if the hypothesis being tested involves ground maneuver forces, even though a flight simulator may model tanks and dismounted infantry, it is not be the best tool to achieve the experiment objectives. A low fidelity simulation designed for echelons above the brigade combat team may not be the best model to employ in a study of the size of the squad formation. The “intended-use” aspect of accreditation cannot be emphasized enough. As noted above, it is defined as being able answer SIMEXp partner analytical requirements defined in the SIMEXp Analysis Plan resulting in a Final Report with actionable findings / outcomes. Weapons, munitions, sensors, tactical actions, and vehicle movement speeds are all examples of data components that must be verified and validated in the process of eventual accreditation. Given a fully verified, validated, and accredited simulation, it is still critical that the simulation interactors (“pucksters”) operate the simulation or perform tactical operations in accordance with doctrinally correct tactics, techniques, and procedures (TTPs) for the forces in the SIMEXp. For example, a SIMEXp involving a US Army Armor Brigade Combat Team (ABCT) fighting a near-peer threat force in an open, rolling European scenario, it would be inappropriate for simulation interactors to dismount infantry fighting vehicles 10 kilometers prior to assaulting an objective. All actions modeled by the simulation must be executed by simulation interactors who are knowledgeable for the TTPs for the given force in the time period for the SIMEXp, as well as active duty military role players providing mission command for the operation. The same principles hold true for the threat forces. When conducting a human-in-the-loop (HITL) SIMEXp, it is possible with simulations such as OneSAF to capture general schemes of maneuver and TTPs at specific point in the battle and then to enter those actions for use in closed-loop runs. If simulation interactors and software developers create closed-loop runs based on the HITL execution, and the military role players (MRPs) validate that the accuracy / doctrinal correctness of the TTPs used, the multiple executions of the closed-loop runs can serve as another means to accredit the SIMEXp.

Tactical and Operational Scenario

The tactical and operational scenario must be appropriate for achieving the experiment objectives. The specific type of operation is critical as is from where on the range of military operations that it originates is critical to the design of a credible experiment as well as the time of year, nature of the terrain, and the weather for the specific selected region. Consideration must be given to the friendly force structure, unit formation, operational communications, accurate weapons and sensor characteristics, employment techniques, and various other scenario-specific variable for friendly

both friendly and enemy forces. The threat forces must be accurate for the time period to be studied in the SIMEXp. This may require classified intelligence input that may cause the SIMEXp to be classified Secret or Top Secret, thus introducing important security requirements. SIMEXp planners must ensure both the friendly and the threat forces operated in accordance with sound doctrine for their respective countries in the timeframe under consideration. For example, a country may possess and use landmines as part of their doctrine. If the country has adopted a policy change to ban landmines within the next five years, and the focus of the experiment is seven years in the future, it would be inappropriate to allow landmine use as part of the SIMEXp. By the nature and criticality of the tactical and operational scenario, the person who accredits those aspects must have great knowledge of the current state of doctrine, military organizations, and operational concepts (friendly and enemy) to be studied. The knowledge required for M&S accreditation and scenario accreditation rarely resides in the same person. Therefore, accreditation is required by qualified individuals for the separate components; the sum of the separate scenario accreditations results in an overall accreditation of the scenario.

Physical and Computational Environment

In order for an experiment to be credible, the physical environment must be well designed to achieve the experiment objectives. Communications between experiment participants must accurately reflect the real world. If the fidelity of the human in the loop experiment is done with humans role-playing company commanders, and in the real world, they would not be able to see each other face-to-face and share information that way, then that should not be possible in a SIMEXp. If the company commander would only know the happenings of a subordinate platoon’s area of operations by what is reported on by voice or texted on a mission command system, then the SIMEXp should be physically structured to reflect those same conditions. The company commander should not be able to stand next to the platoon leader, see his areas of operations on the computer screen and have perfect situational awareness of the actions of the friendly and enemy forces from viewing the platoon leader’s computer screen. At the same time, all of the tools/systems to which a company commander would have access in a real combat situation should be made available, including unmanned aircraft feeds, chat functions, subordinate unit position/location/information, etc. Additionally, the computational environment must be sufficient for SIMEXp accreditation. The hardware must be robust enough to process all software functions without delay or lag in the execution of events. The network must be sufficient to ensure packet loss does not hamper execution or data collection. Network security must be in place to ensure the security of SIMEXp execution and data archiving.

Analysis Plan

The analysis plan development begins with the initial meeting with the SIMEXp partner organization and ends upon publication of the final report. The analysis plan serves to focus all efforts towards answering the sponsor’s Experiment Objectives and must be developed and executed by credentialed individuals. The analysts for the MBL SIMEXps are Army Operations Research Analysts (ORSAs), or have equivalent training that is certified prior to them serving on an experiment. The foundation of the analysis plan is in the identification and agreed-upon definitions for the SIMEXp measures of merit (measures of effectiveness, measures of performance, and data parameters). Methods for collecting quantitative and qualitative data must be decided up early in the planning process so that SIMEXp planning can support achieving their requirements. It may be necessary to review analytical terms such as “insight” and “finding” to ensure agreement on their use in reporting outcomes. A sound analysis plan is reviewed at all in progress reviews (IPRs) and planning conferences to ensure partner concurrence and senior leader understanding. Critical to the postexecution analytical process is ensuring knowledgeable, well-trained analysts work closely with data collector to gather all required information from the simulation and reduce the data into insights and outcomes that are understandable to the senior leader decision-makers.

Accreditation of the SIMEXp

Once the accrediting individual reviews and accredits the M&S, or following the successful execution of the record runs, it may be tempting to declare that the overall SIMEXp is accredited. This would be a mistake, for it is with the publication of the final report that the SIMEXp is actually accredited. It is at that point that the SIMEXp Director has received, reviewed, and accredited the sum of the individual parts (models and simulations, the tactical/operational scenario, the physical and computational environment, and the analysis plan). Then, if warranted, accredits the overall SIMEXp in its entirety and accredit its findings to answer the experiment objectives, given the scope and authority of the SIMEXp.

How Accreditation Works in Practice: MBL Simulation Experiment Accreditation Process

The MBL simulation experiment accreditation process is a multi-level standards based process that, when conducted properly, results in a credible experiment designed to answer a proposed military problem. The unique nature of military operations requires a unique experiment VV&A process. The steps in this process include: analytic plan development and approval, data accreditation, model face validation, scenario development and approval, M&S accreditation, and peer review (see Table 2, below). Combat is an inherently dangerous endeavor; failure in the design and experimentation of Army maneuver weapons and combat vehicles could ultimately lead to the loss of American lives. Due to the deadly serious consequences of NOT providing valid results to military experimentation, the MBL requires certification or accreditation at every level of its experimentation process. An important part of the accreditation is the partnership between the MBL and various subject matter experts (SMEs) from the US Army’s S&T community, the acquisition community, the Maneuver Capability Development Integration Directorate (MCDID), and the Maneuver Center of Excellence (MCOE). The MBL includes these partners in every step of its accreditation process.

Step

Development & Approval Data VV&A Data Producer (DAC) Data User (MBL) Scenario Development & Approval M&S Accreditation Peer Review

Table 2. MBL Military Simulation Experiment Approval Process

Step	Standard	Expert	Credentials	Reference
Analysis plan development & approval	AACAS	Lead ORSA	ORSA Military Applications Course (MAC); PhD	AACAS
Data VV&A	DA PAM 5-11 (VV&A of Army M&S)	Data producer (DAC); data user (MBL)	DAC engineer; operational and S&T SMEs; military professional	DA PAM 5-11; DODI 5000.61; MSCO RPG
Scenario development & approval	TRAC developed and approved	BCT SMEs; TEFOR SME	Military professional; threat SME	TR 71-20
M&S accreditation	M&S V&V conducted; all steps above conducted & approved	M&S Branch Chief	Military professional; CMSP	DA PAM 5-11; Simprofessional.org
Peer review	Reviewed by analysis review board	SMEs; ORSAs	Military professional; ORSA MAC	CCRP COBP; AACAS

Analysis. Every good simulation experiment begins and ends with a sound analysis plan. The standard-bearers of analysis within the US Army are the Center for Army Analysis (CAA) and the Army Future Command’s (AFC’s). The Research and Analysis Center (TRAC). In 2015, CAA and TRAC published the Army Analytic Community Analytic Standards (AACAS). The AACAS provides common standards to apply when conducting Department of Defense (DOD) and Army analysis based on the principles of the scientific method (TRAC, 2015). The analytic standards for all Army analysis studies include the following steps, which are aligned to standards that are described in the AACAS: 1) Formulate the Problem, 2) Conduct Literature Review, 3) Develop Study Plan, 4) Develop Data Collection Management Plan (DCMP), 5) Develop Methods, Models, and Tools (MMT), 5) Develop Scenario, 6) Conduct Data Development, 7) Conduct Analysis, 8) Interpret Results, 9) Translate/Present Findings, and 10) Follow Up (TRAC, 2015). The requirement for analysis plan development in the MBL is the rigorous application of these AACAS standards. They must be applied by an Army (or Sister Service) trained ORSA and approved by the MBL’s lead ORSA, who is a PhD level mathematician.

Data VV&A. The MBL’s primary tool for quantitative analysis is the One Semi-Automated Forces (OneSAF) simulation, which is the Army’s constructive simulation model program of record (PoR). OneSAF is a brigade- and

below-focused, composable, entity-level computer generated forces simulation. A designated use of OneSAF is futurefocused concepts and requirements determination for the experimentation and analysis communities (OneSAF User’s Guide). As a composable simulation, OneSAF requires parametric data for model composition. The MBL uses only parametric data that has been produced by the US Army’s Data Analysis Center (DAC). The DAC’s mission is to “Deliver…data across the entire life cycle to ensure readiness today and a more lethal future force tomorrow.” (www.dac.ccdc.army.mil, 2021). The MBL depends on DAC, in that role, to provide parametric data that is required to model future maneuver force capabilities. This leads us to the next phase of the MBL accreditation process, which is data VV&A. The Department of the Army (DA) Pamphlet 5-11 (VV&A of Army M&S states that there are two groups that use data: data generators (producers) and those that put the data to use (users) (Department of the Army (DA), 1999). Therefore, the MBL simulation experiment accreditation process depends on both a data producer (DAC) VV&A process and a data user (MBL) VV&A process. As the data producer, DAC is required to conduct the data producer accreditation. Data producer accreditation is a determination by DAC that its data have been verified and validated against documented criteria. As the data user, the MBL’s user data accreditation is the determination that the data have been verified and validated as appropriate for the MBL’s specific M&S usage. According to Department of Defense Instruction (DODI) 5000.61, DOD components have the final authority for validating representation of their forces and capabilities. Therefore, face validity (credibility to expert audiences) is the soundest technique and is valued more than construct validity and empirical validity (Alberts & Hays, 2002, pp. 337-338) for military experimentation. Accordingly, the MBL uses a robust face validation process to ensure that the models used perform in an accurate manner and produce reliable results. The models are tested in a OneSAF tactical scenario that is part of the overall experiment in order to determine if acquisitions and weapon engagements are having the proper effects; both friendly-on-enemy and enemy-on-friendly (MBL Face Validation SOP). The results of these tests are reviewed by MBL data analysts and subject matter experts (SMEs). Terrain validation is also part of this face validation process. SMEs are required at various levels of the data development and validation process. The MBL only conducts experimentation for others (customers), so those customers’ expert knowledge on their weapons and/or combat vehicles is required not only during the face validation but also at the beginning of the process when data is requested from DAC. These SMEs are either tactical experts, technical experts, or both. Most of them are scientists or engineers that work for the US Army’s Combat Capabilities Development Command (CCDC) or the Assistance Secretary of the Army for Acquisition, Logistics, and Technology (ASA (ALT)). They provide the performance specifications for the entities to be modeled, which is then fed to DAC for conversion into OneSAF parametric data. The SMEs are then paired with maneuver experts who, together, serve as the final arbitrators in the MSB’s face validation process. Noted model deficiencies are then corrected by DAC, if time permits, or by the MBL M&S team. Once corrected, the models are adjusted by the MBL right up to the execution of the simulation experiment itself (MCDID, 2020).

Scenario Development and Approval. Simultaneous to the model development and VV&A process, the MBL develops a tactical scenario that is derived from The Research and Analysis Center (TRAC) approved scenarios. Scenarios are graphic, data, and narrative tools that describe the global conditions surrounding a conflict. They describe the friendly and threat forces in enough detail to support concept and capability development experimentation. TRAC has the responsibility to develop standard tactical scenarios in a Joint and future operational context that support Army wide studies, experimentation, and analysis (TRADOC, 2013). The MBL has resident maneuver SMEs that derive experiment specific tactical scenarios that are within the parameters established by those approved TRAC scenarios. These experiment specific scenarios are also within the context of latest Army Operating Concept (AOC) and Movement and Maneuver Functional Concept (MMFC). Regardless of the capability that is the subject of experimentation, that capability must be fought within the context of an US Army unit or formation. As the primary experimentation venue for the Army’s Maneuver Force, the MBL must conduct its experimentation within the context of an operational fighting unit. The MBL relies on the Army Capability Managers (ACMs) for the Armored Brigade Combat Team (ABCT), Infantry BCT (IBCT), Stryker BCT (SBCT), and the Security Force Assistance Brigades (SFABs) to provide operational and organizational (O&O) concepts for their fighting formations (TRADOC, 2013). The operational SMEs for the various ACMs are part of the MBL’s experimentation process from scenario development through experiment report completion. These SMEs serve as part of the MBL accreditation process at both the model face-validation phase and the scenario development phase.

They must also confirm that the friendly order of battle (ORBAT) for the simulation is in accordance with their O&O construct. A key ingredient in scenario development and model validation is the enemy or threat force. The AFC’s Threat Emulation Force (TEFOR) is tasked with developing the threat force structure and tactical plan for the entire Army Modernization Enterprise (AME) (TRADOC, 2013). The TEFOR play three important roles in the MBL accreditation process. First, these threat SMEs derive a threat force structure that supports the simulation experiment’s learning objectives. This derivative scenario is in line with the AFC approved Future Operational Environment (FOE) and represents a near peer threat within the context of the aforementioned TRAC scenario. Second, they serve as the threat force role players during the experiment. Finally, the TEFOR use their subject matter expertise during the model facevalidation.

M&S Accreditation. Once the friendly and threat models have been validated and the terrain has been tested, the M&S environment is tested in the aforementioned tactical scenario. During this validation phase, the MBL validation agent with help from the friendly and enemy SMEs ensures that it is an accurate representation of the real world from the perspective of the intended use of the M&S and its data. The MBL V&V agent, who is also the MBL M&S team lead, provides a V&V report that addresses the confidence and credibility of the M&S according to its intended application (Department of the Army, 1999). The MBL accreditation agent, the MBL M&S Branch Chief, then conducts a holistic evaluation of the entire experimentation development process by asking the following questions: Is the analysis plan in accordance with the AACAS and has it been approved by the lead MBL analyst? Has the data been accredited by DAC? Have the threat and enemy models passed the MBL face validation process? Has the terrain been tested? Is the scenario within the parameters of a TRAC approved scenario and does it meet the learning demands of the experiment? Have the ACM and TEFOR SME’s validated the friendly and enemy ORBATs, respectively? Is the M&S an accurate representation of the world for its intended use? The accreditation agent then makes an overall accreditation decision on the simulation experiment and documents it in an accreditation report.

Peer Review. The final step in the MBL accreditation process is the peer review. Academia has a tradition of peer review for new knowledge. Simulation based experimentation also requires peer review because assertions will be made without the benefit of real-world knowledge (Alberts & Hayes 2002, p. 340). The AACAS also calls for peer review during its Translate and Present Findings phase (TRAC, 2015). The MBL conducts peer participation and review during multiple stages of its accreditation process. First, the previously mentioned MBL partners participate in the simulation experiment itself and provide real-time feedback as the results are output. Second, the partners SMEs and ORSAs that are not involved in the conduct of the experiment review the report during an analysis review board. During this analysis review board, the participating analysts and MBL partners question the lead analyst’s finding that he or she has derived from the experiment’s qualitative and quantitative data. Finally, the primary stakeholder (or study sponsor) will review the experiment report and concur with comment or non-concur prior to the final signature of the MBL Director (MBL SOP). Adjustments to the final report are made throughout this process to ensure that the experiment report meets the study’s learning objectives. This multi-faceted peer review process ensures both analytic rigor and objectivity.

Applicability of the MBL Simulation Experiment Accreditation Process for other organizations

We propose that this military experiment accreditation framework can also be used to accredit other defense and acquisition related experiments. For example, several other organizations, such as Federally Funded Research and Development Centers (FFRDC), DEVCOM and TRAC, conduct experimentation in support of defense modernization. This methodology could also be applied to other Department of Defense (DOD) experimentation venues (e.g., US Navy and Air Force) and Combined/Allied (e.g., NATO) experimentation venues. The unique nature of military concept and capability development experimentation requires a unique and more robust accreditation process, like the MBL accreditation process. A distinct factor of the MBL process is the qualifications required of the accreditation agent. The accreditation agent is the person, group, or organization responsible for conducting the accreditation assessment (US DOD, 2018). We propose that an accreditation agent for a military experiment has a broader responsibility to accredit the entire

experiment NOT just the M&S. We propose that an accreditation agent for a military experiment should be 1) a military professional and 2) a modeling and simulation (M&S) expert. In The Soldier and the State, Samuel Huntington defines a professional as "a peculiar type of functional group with highly specialized characteristics," which he identifies as "expertise, responsibility, and corporateness." He further states that military officers approach the professional level of applying their specialized skills and professional knowledge, which is the management of violence (Huntington, 1957). In order to understand the true validity of an experiment, and therefore have the credentials for accreditation, one must be a military professional. It is this unique expertise and application of special knowledge that allows only a true military professional to conduct the overall accreditation of a military experiment. Since each US military service defines their professional qualifications differently, each Service should then determine exactly the qualifications required to designate an officer as a military professional. The second recommended qualification for an accreditation agent is that he or she is an M&S expert. We therefore propose that an accreditation agent for a military simulation experiment must also be a credentialed Certified Modeling and Simulation Professional (CMSP). The CMSP is the only known professional certification for M&S professionals. The CMSP designates individuals who have attained a significant degree of knowledge and experience in M&S (simprofessionals.org). The resident knowledge required to serve as a military experiment accreditation agent can only be assured through the CMSP certification process. It is the unique combination of military professionalism and M&S certification that ensures that complex military warfighting simulation experiments are accredited. Finally, the standards described in this document are germane to the MBL, the Army Futures Command, and the Army analytic and experimentation communities. Therefore, each military experiment community of practice should develop or adhere to their own standards for the analysis plan development and approval, data V&V, scenario development and approval, M&S accreditation, and peer review processes that are described in this paper. The complex nature of military operations requires a multi-level, standards-based experiment accreditation process in order to ensure that analytically and tactically sound recommendations are made for military based concept and capability focused simulation experiments.

Acknowledgements

The authors would like to acknowledge the use of references listed below, as well as the help of our ‘birddog’, Lisa Bair (especially for her specific wording in helping to characterize and define “specific use of M&S”). Additionally, we would like to thank Dr. Cindy Forgie and Larry Rieger who reviewed drafts of this paper and provided valuable input.

References