System Verification Proving the Design Solution Satisfies the Requirements
Definition: The ability of a system design to meet operational, functional, and system requirements is necessary to accomplishing a system's ultimate goal of satisfying mission objective(s). One way to assess the design's ability to meet the system requirements is through requirements traceability—the process of creating and understanding the bidirectional linkage among requirements (operational need), organizational goals, and solutions (performance). Keywords : assessment, concept of operations (CONOPS), functional requirements, mission and needs, operational requirements, performance verification, requirements, requirements traceability, requirements traceability matrix, system requirements, traceability, verification MITRE SE Roles and Expectations: MITRE systems engineers (SEs) are expected to understand the importance of system design in meeting the government's mission and goals. They are expected to be able to review and influence the contractor's preliminary design so that it meets the overall business or mission objectives of the sponsor and user. MITRE SEs are expected to be able to recommend changes to the contractor's design activities, artifacts, and deliverables to address performance shortfalls and advise the sponsor if a performance shortfall would result in a capability that supports mission requirements whether or not the design meets technical requirements. They are expected to be thought leaders in influencing decisions made in government design review teams and to appropriately involve specialty engineering [1]. In requirements traceability and performance verification, MITRE SEs are expected to maintain an objective view of requirements and the linkage between the system end-state performance and the source requirements and to assist the government in fielding the best combination of technical solution, value, and operational effectiveness for a given capability. A Key Driver in Meeting System Requirements The success of a system rests on how well it meets the users' needs. User engagement during the development of the system is becoming a standard procedure in the delivery of system functionality based on user priorities rooted in meeting their business objectives. The user community has no appetite for waiting long periods to see new/upgraded system capabilities made operational. As a result, techniques/methods such as "agile" development have become very popular. These methods call for the demonstration of system capabilities during the development of the system. System development team members present their progress during short meetings, called "scrums," in which a clear picture emerges of the timeline of the provisioning of system capabilities that meet user requirements. Traceability and Verification Process A meaningful assessment of a design's ability to meet system requirements centers on the word "traceability." Traceability is needed to validate that the delivered solution fulfills the operational need. For example, if a ship is built to have a top speed of 32 knots, there must be a trail of requirements tied to performance verification that justifies the need for the additional engineering, construction, and sustainment to provide a speed of 32 knots. The continuum of requirements generation and traceability is one of the most important processes in the design, development, and deployment of capability. Traceability is also the foundation for the change process within a project or program. Without the ability to trace requirements from end to end, the impact of changes cannot be effectively evaluated. Furthermore, change should be evaluated in the context of the end-to-end impact on other requirements and overall performance (e.g., see the SEG's Enterprise Engineering section). This bidirectional flow of requirements should be managed carefully throughout a project/program and be accompanied by a well-managed requirements baseline. For more effective traceability, verification, and in general requirements management, it is advisable to use tools available in the industry for this purpose. Requirements Flow The planned functionality and capabilities that a system offers need to be tracked through various stages of requirements (operational, functional, and system) development and evolution. Requirements should support higher level organizational initiatives and goals. It may not be the project's role to trace requirements to larger agency goals. However, it can be a best practice in ensuring value to the government. In a funding-constrained environment, requirements traceability to both solutions as well as organizational goals is essential in order to make best use of development and sustainment resources. As part of the requirements traceability process, a requirement should flow two ways: (1) toward larger and more expansive organizational goals and (2) toward the solution designed to enable the desired capability. This end-to-end traceability ensures a clear linkage between organizational goals and technical solutions. Requirements Verification Because the ability to test and verify is a key element of project/program success, requirements tied to operational needs should be generated from the outset and maintained throughout the requirements life cycle with test and verification in mind. Advice on the ability to test requirements can be extremely effective in helping a project or program. Techniques such as prototyping and experimentation can help assess requirements early and provide a valuable tool for subsequent verification and validation. For more information, see the SEG article Competitive Prototyping. Test and verification plan development and execution should be tied directly back to the original requirements. This is how the effectiveness of the desired capability will be evaluated before a fielding decision. Continual interaction with the stakeholder community can help realize success. All test and verification efforts should relate directly to enabling the fielding of the required capability. Developing test plans that do not facilitate verification of required performance are an unnecessary drain on resources and should be avoided. Before a system design phase is initiated, it should be ensured that the system requirements, captured in repositories or a system requirements document, can be mapped to functional requirements, for example, in a functional requirements document (FRD). The requirements in the FRD should be traceable to operational requirements, for example, in an operational requirements document or a capabilities development document. Ensuring all this traceability increases the likelihood that the design of the system will meet the mission needs articulated in a concept of operations and/or the mission needs statement of the program. Design Assessment Considerations The design of a system should clearly point to system capabilities that meet each system requirement. Two-way traceability between design and system requirements enables a higher probability of a successful test outcome of each system requirement and of the system as a whole, as well as the delivery of a useful capability. As the service-oriented architecture (SOA) approach matures, there is increased emphasis on linking system requirements to specific services. Therefore, the artifacts or components of system design should be packaged in a manner that supports provisioning of services offered within SOA (assuming that SOA deployment is a goal of an enterprise). For example, assume that the system requirements can be grouped in three categories: Ingest, Analysis, and Reporting. To meet system requirements within these categories, the design of the system needs to point to each system component in a manner that addresses how it would fulfill the system requirements for Ingest, Analysis, and Reporting, respectively. The design information should include schematics or other appropriate artifacts that show input, processing, and outputs of system components that collectively meet system requirements. Absent a clear road map showing how input, processing, and outputs of a system component meet a given system requirement, meeting that specific system requirement is at risk. Typically, the project team develops a requirements traceability matrix (RTM) that shows linkage among functional requirements, system requirements, and system capabilities of system design components. An RTM that clearly points to system components that are designed to meet system requirements is more likely to result in a well-designed system, all other considerations being equal. Additional linkages can be included in an RTM to show mechanisms to test functionality of system components for testing the design of the system to meet system requirements. An RTM as described above (i.e., one that ranges from statement of a requirement to methodology to test the system component that satisfies the system requirement) will go a long way in successfully assessing a system design to meet system requirements. A traceability matrix is developed by linking requirements with the design components that satisfy them. As a result, tests are associated with the requirements on which they are based, and the system is tested to meet the requirement. These relationships are shown in Figure 1. A sustained interaction is needed among members of a requirements team and those of design and development teams across all phases of system design and its ultimate development and testing. This kind of dialog will help ensure that a system is being designed properly with an objective to meet system requirements. In this way, an RTM provides a useful mechanism to facilitate the much-needed interaction among project team members. Table 1 is a sample RTM that spans "Requirement Reference" to "Design Reference." The matrix can be extended to include testing mechanisms for further assurance that the system design will meet system requirements. The RTM in Table 1 links a system requirement to a design component (e.g., a name of a module). Table 1. Sample RTM Linking System Requirement to Design Component Project Name: Author: Date of Review: Reviewed By: Req. ID Requirement Reference Requirement Description Design Reference System APP 1.1 APP SRS Ver 2.1 Better GUI APP Ver 1.2 Module A APP 1.2 APP SRS Ver 2.1 Send Alert messages APP Ver 1.2 Module B APP 1.3 APP SRS Ver 2.1 Query handling APP Ver 1.2 Module C APP 1.4 APP SRS Ver 2.1 Geospatial Analysis APP Ver 1.2 Module D The RTM in Table 2 links a test case designed to test a system requirement. Table 2. Sample RTM Linking a Test Case Designed to Test a System Requirement Unit Test Case # System Test Case # Acceptance Requirement Type APP_GUI.xls TC_APP_GUI.xls UAT_APP_GUI.xls New APP_MSG.xls TC_APP_MSG.xls UAT_APP_MSG.xls Change Request APP_QRY.xls TC_APP_QRY.xls UAT_APP_QRY.xls New APP_GA.xls TC_APP_GA.xls UAT_APP_GA.xls Change Request The assessment of a system design should consider how well the design team presents the linkage of its design to system requirements (i.e., through documentation, presentations, and/or critical design reviews). A traceability matrix can be an important device in communicating a design's ability to meet system requirements. As part of the system design approach, the design team may develop mock-ups and/or prototypes for periodic presentation to the end users of the system and at design reviews. This approach provides an opportunity for system designers to confirm that the design will meet system requirements. Therefore, in assessing a system design, the design team's approach needs to be examined to see how the team is seeking confirmation of its design in meeting system requirements. In an agile scenario, this interaction with the end user is much more frequent and iterative than in traditional development. The processes and tools used in design must be able to keep pace with these rapid adjustments. Traceability and Verification Development of project/program scope. The overall goals or desired impact for a project/program must be understood and delineated from the beginning of the effort. The solutions and technologies required can and should evolve during the systems engineering process, but the desired capability end state should be well understood at the beginning. "What problem are we trying to solve?" must be answered first. In newer methods such as agile, the end state is even more important because the pathway to the final capability is much more flexible. Quality of written requirements. Poorly written requirements make traceability difficult because the real meaning is often lost. Ambiguous terminology (e.g., "may," "will") is one way requirements can be difficult to scope, decompose, and test. Assist with written requirements by ensuring a common and clear understanding of terminology. Excessive reliance on derived requirements. As the focus of the work moves away from the original requirements, there is a danger of getting off-course for performance. Over-reliance on derived requirements can lead to a loss of context and a dilution of the true nature of the need. This is where traceability and the bidirectional flow of requirements are critical. Unique challenges of performance-based acquisition. Performance-based projects present a unique set of issues. The nature of performance-based activity creates ambiguity in the requirements by design. This can be extremely difficult to overcome in arriving at a final, user-satisfactory solution. As a matter of practice, much greater scrutiny should be used in this environment than in traditional project/program development. Requirements baseline. A requirements baseline is essential to traceability. The need for a baseline must be carefully considered before embarking on newer techniques such agile development—where a baseline is not used. There must be a trail from the original requirements set to the final implemented and deployed capability. All the changes and adjustments that have been approved must be incorporated in order to provide a seamless understanding of the effort's end state. It should also include requirements that were not able to be met. To adequately judge performance, the requirements must be properly adjusted and documented. Project/program risk impacts. The influence of requirements on project/program risk must be evaluated carefully. If a requirement generates sufficient risk, an effective mitigation strategy must be developed and implemented. Eliminating a requirement can be an outcome of this analysis, but it must be weighed carefully. This is where an FFRDC trusted agent status is especially critical. Chasing an attractive yet unattainable requirement is a common element in project/program delays, cost overruns, and failures. See the SEG's Risk Management topic. Watch for requirements that are difficult to test. If requirements are difficult or impossible to test, the requirements can't be traced to results if the results can't be measured. System-of-systems engineering efforts can greatly exacerbate this problem, creating an almost insurmountable verification challenge. The language and context of requirements must be weighed carefully and judged as to testability; this is especially true in a system-of-systems context. See the SEG's Test and Evaluation of Systems of Systems article. Requirements creep. Requirements creep—both up and down the spectrum—is an enduring conundrum. As requirements flow through the systems engineering process, they can be diluted to make achieving goals easier, or they can be "gold plated" (by any stakeholder) to provide more than is scoped in the effort. Increasing capability beyond the defined requirements set may seem like a good thing; however, it can lead to difficulty in justifying program elements, performance, and cost. Adding out-of-scope capability can drastically change the sustainment resources needed to maintain the system through its life cycle. Requirements creep is insidious and extremely detrimental. On the other hand, the evolution of needs and requirements must be accommodated so that flexibility in creating capabilities can match changing operations and missions and provide timely solutions. Interaction with end users. Interaction with end users is critical to the requirements traceability and verification cycle. The ability to get feedback from people who will actively use the project or program deliverables can provide early insight into potential performance issues. Some methodologies, such as agile, are built around frequent interaction with end users. When determining a development approach, carefully consider the amount of end-user interaction required. Bidirectional requirements traceability. There must be a two-way trace of requirements from the requirements themselves to both larger organizational goals and to applicable capability solutions. Verification of test plans. Pay careful attention to the development of the requirements verification test plans. An overly ambitious test plan can portray a system that completely meets its requirements as lackluster and perhaps even unsafe. On the other hand, a "quick and dirty" test plan can miss potentially catastrophic flaws in a system or capability that could later lead to personnel injury or mission failure. Design Assessment Importance of documentation and team commitment. A thorough review of documentation and an evaluation of the design team's commitment to engage with the stakeholders in the design process are key to conducting a meaningful assessment of whether the system design meets the system requirements. Importance of documented and validated findings. Document your assessment and validate your findings. Arkley, P., S. Riddle, and T. Brookes, 2006, Tailoring Traceability Information to Business Needs, University of Newcastle upon Tyne. Chairman of the Joint Chiefs of Staff, January 23, 2015, Joint Capabilities Integration and Development System (JCIDS) , Instruction 3170.01L. Defense Acquisition Guidebook, accessed July 20, 2015. DoD Chief Information Officer, March 2013, DoD Agile IT Handbook: Management best practices to achieve better outcomes in DoD IT programs. FAA Acquisition System Toolset, accessed October 15, 2015. Federal Aviation Administration, NAS Requirements Documents, viewed October 15, 2015. Government Accountability Office, March 2006, Business Systems Modernization, IRS Needs to Complete Recent Efforts to Develop Policies and Procedures to Guide Requirements Development and Management, GAO-06-310. International Council on Systems Engineering (INCOSE), July 2015, INCOSE Systems Engineering Handbook, 4th Edition. NASA, December 31, 2007, NASA Systems Engineering Handbook. Ramesh, B., C. Stubbs, T. Powers, and M. Edwards, April 1995, Lessons Learned from Implementing Requirements Traceability, CrossTalk. Ramesh, B., January 2001, Towards Reference Models for Requirements Traceability. IEEE Transactions on Software Engineering, Vol. 27, Issue 1, pp. 58–93. Tran, Eushiuan, Spring 1999, Requirements and Specifications, Carnegie Mellon University.Background
The Requirements Traceability Matrix: Where the Rubber Meets the Road
Feature
Module Name
Test Case # Best Practices and Lessons Learned
References and Resources
Additional References and Resources
System Verification Proving the Design Solution Satisfies the Requirements
Source: https://www.mitre.org/publications/systems-engineering-guide/se-lifecycle-building-blocks/system-design-and-development/assess-the-designs-ability-to-meet-the-system-requirements