07 INSIGHT - 2023 - Hecht
07 INSIGHT - 2023 - Hecht
FEATURE
SPECIAL
Validation of SysML
Models
MARCH 2O23
VOLUME 26/ ISSUE 1
Myron Hecht, [email protected]; and Jaron Chen, [email protected]
Copyright © 2021 by Myron Hecht and Jaron Chen. Permission granted to INCOSE to publish and use.
ABSTRACT
Model-based systems engineering depends on correct models. However, thus far, relatively little attention has been paid to en-
suring their correctness. This paper describes a methodology for performing verification and validation on models written in
SysML. The methodology relies on a catalog of candidate requirements that can be tailored for a specific project. Both manual
and automated methods are used to verify and validate these requirements. Manual methods are necessary where knowledge of
the domain and other extrinsic characteristics are necessary. Automated methods can be used where the requirements cover the
use of SysML. Examples from a public domain SysML model of a satellite are presented to demonstrate application of automated
requirements verification.
M
INTRODUCTION
odel-based systems engineer- data typing errors are difficult to find requirements and “validation” means
ing (MBSE) will not succeed through manual inspection. assessment of suitability to user needs.
without correct and complete 4. Inadequate documentation: Internal For the methodology described in
system engineering models. and external documentation of mod- this paper, user needs are stated as
Incorrect models can cause wrong design els is necessary to define the intent of requirements.
decisions or integration failures because of the model and to minimize spurious
errors in representation of the architecture, assessments of model validity. PREVIOUS WORK
design, or behavior; and many other rea- A recent literature survey of 579 SysML
sons. Therefore, verification and validation This paper describes an approach that publications between 2005 and 2017
(V&V) of models is essential to successful addresses these difficulties. The next two identified few that addressed verification
MBSE projects. However, system engineer- subsections de- scribe previous work and and validation of SysML models (Wolny,
ing model V&V is not widely practiced. In the conceptual framework of our approach. et al. 2020). What has been published can
a survey conducted by the System Engi- The next section describes model require- be placed in two major categories: model
neering Research Consortium (SERC), ments formulation, and the final section transformation and model inspection.
more than 60% of government respondents describes of verification and validation Model transformation: Model transfor-
disagreed with the statement “Our organi- methods for SysML models. mation involves converting SysML models
zation has defined processes and tools for into other formalisms that could then
V&V of models at appropriate levels and There are many definitions of “verifi– subsequently be analyzed for conformance
program phases” (McDermott, et al. 2020). cation” and “validation.” IEC/ISO/ to specialized properties that could be
Less than 3% strongly agreed with it. The IEEE 24765 “Software and System analyzed with those formalisms. Ahmad,
difficulties of verifying and validating sys- Engineering Vocabulary” lists six Dragomir, et al. proved invariant properties
tem engineering models include: definitions for verification and five by transforming an XMI export of a SysML
1. No definition of what a “correct” model for validation. There is a separate to an executable UML/SysML profile model
is: Without a set of requirements that vocabulary entry for “verification and (OMEGA2) of a diabetes patient and her
define correctness, it is impossible to validation” which combines them. refrigerator (Ahmad, et al. Jul 2013). The
verify a model. The meaning of both terms, either model consisted of two internal block
2. Failure to define model user needs: separately or collectively, is a set of diagrams and a state machine diagram con-
Without a definition of user needs, it activities intended to assess a model sisting of two 2 states. Jarraya and Debbabi
is not possible to validate a model. for correctness and suitability for demonstrated a method of probabilistic
3. Difficulty of finding syntax errors: the user. In this paper, “verification” verification of SysML activity diagrams
Most syntactic, value property, and means assessment of conformance to by transforming them into the PRISM
33
21564868, 2023, 1, Downloaded from https://incose.onlinelibrary.wiley.com/doi/10.1002/inst.12427 by libffg rrtyu - Beijing Normal University , Wiley Online Library on [09/08/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
(probabilistic symbolic model checker) Sponsor
FEATURE
SPECIAL
34
21564868, 2023, 1, Downloaded from https://incose.onlinelibrary.wiley.com/doi/10.1002/inst.12427 by libffg rrtyu - Beijing Normal University , Wiley Online Library on [09/08/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
General: General Requirements cover ■■ Structural Views: Requirements to exiting a decision node must have de-
FEATURE
SPECIAL
the general user and model sponsor’s capture context, logical architecture, fined guards, forks and decisions having
needs rather than a specific project model. functional architecture, and physical at least two outgoing flows, joins and
Lower-level headings include: architecture, and to allocate model merges having at least two incoming
■■ Development Requirements: related to elements among all three architectures flows, and others.
model language, stereotypes, tool selec- to enable traceability. ■■ Sequence diagrams: Requirements that
tion, documentation of model elements, ■■ Interfaces: Requirements that all block sequence diagrams capture behaviors
conformance to style guidelines, and connections must be made through de- on connections, model operations and
use of prescribed modeling patterns. fined interfaces, that interfaces on each receptions between diagrams, and that
MARCH 2O23
VOLUME 26/ ISSUE 1
■■ Model Organization: requirements on side of connection must be compatible, they capture error handling and recov-
consistent and logical model structure that conventional interface control doc- ery; messages on sequence diagrams
and organization, use of model naviga- uments (ICDs) be linked to interface have signatures assigned (signal or
tion aids, and labeling of all packages. blocks that represent them, that all con- operation); and others.
■■ Interface and Data Consistency: require- nections have defined flows, and that ■■ State machine diagrams: Requirements
ments for interfacing and provision or operations and receptions be used when for state machine such as that that
ingesting of data to at specified inter- behaviors are part of the interface. all blocks that change state shall be
faces to other components of the digital ■■ Blocks: Requirements on traceability modeled using state machine diagrams,
engineering ecosystem and consistency between SysML blocks and system re- naming conventions on states and
with model libraries of the sponsoring quirements, block naming conventions, transitions, capturing of signals, events,
organization. documentation including description of message flows as transition triggers,
■■ Views and Exports: requirements spec- pre- and post-conditions on operations representation of error handling and
ifying the information that the model and receptions, allocation of activi- recovery, and others.
provides to non-modeling users and in- ties or actions to blocks, depiction of
clude views into the model and exports. allocation of software to hardware, data Model Data: Requirements for model
Such exports can include tables, “dash- typing, and others. data including documentation, data types
boards”, and even complete documents ■■ Relationships and associations: Require- for all attributes and properties, consistent
such as system and subsystem design ments for roles and multiplicity on part naming in accordance with naming con-
documents, verification and validation ends of composition relationships, pro- ventions, and a glossary.
plans, test procedures, test reports, and hibition against both composition and
technical orders. inheritance to a single block, prohibi- Non-functional Requirements:
tion against cyclic inheritance, justifica- Requirements covering performance,
User needs and system requirements: tion for any block with no relationship reliability/availability, safety, cybersecurity,
These requirements concern modeling the to other elements, and others. and maintainability and sustainment
needs of the organization and users of the ■■ Internal block diagrams: Requirements modeling. Architectural design decisions
system under development (not the model) for all blocks to be connected through can have a large impact these attributes
as would be stated in the Concept Defini- ports, flows to be shown on all (particularly if they are overlooked). This
tion Document and the System Require- connections, multiplicities on both section is responsive to IEEE 15288 (ISO/
ments Document. Topics covered include: ends of the connectors, typing of flow IEC/IEEE, 2015), Appendix F “Architecture
■■ Use Cases: SysML model requirements properties, and others. modeling”, par. F.3.8 “Other model
that Use Cases capture mission objec- ■■ Parametric diagrams: Requirements considerations”.
tives and system operations, trace use for limits on the number of blocks ■■ Performance Modeling: Requirements
cases to requirements and to verifica- and connections to retain readability, for data necessary to analyze and
tion methods, standards for descrip- verification of numerical constraint calculate response time and capacity at
tion and documentation for use cases blocks, visibility of binding connectors, any architectural level; data necessary
and actors, and coverage of exception and populated constraint specifications, to analyze and calculate response time,
conditions. and others. capacity, ability to simulate at any
■■ System Requirements: requirements on architectural level; and others.
system requirements, for example that Behavioral Content: Requirements in ■■ Reliability Modeling: Requirements for
the model shall include of all system this area cover activity, sequence, and state data necessary to analyze and calculate
requirements in the technical baseline, machine diagrams. Topics covered include: quantitative reliability attributes using
that all requirements shall include a ■■ General behavior: Requirements analytical techniques such as reliability
rationale, and that all requirements be to model all dynamic behavior in block diagrams, fault tree analysis, and
related either to other requirements the scope of the model and that all Markov modeling; data necessary to
through contain, derive or define diagrams document the assumptions, analyze and reliability and availability at
relationships or must themselves have design decisions, preconditions, and any architectural level; and others.
satisfy and verify relationships. postconditions (all requiring manual ■■ System Safety Modeling: Requirements
■■ System Requirements Diagrams and verification). that the model allow for the input,
Tables: requirements on the model to ■■ Activity diagrams: Requirements on the storage, and export of data necessary
show requirements relationships – es- correctness of allocation of actions to to support all deliverables required
pecially satisfy and verify relationships. blocks (swim lanes), and the inclu- under the program contract, data
sion of error handling. Automatically for all safety review and certification
Structural content requirements: These verifiable requirements include: Input authorities whose approval is required,
requirements affect SysML blocks, block pins must have an incoming object data input, storage, and export for
definition, internal block, and parametric flow, output pins must have an outgoing all tasks related to explosive safety
diagrams. Subheadings include: object flow, all control and object flows standards, production of reports
35
21564868, 2023, 1, Downloaded from https://incose.onlinelibrary.wiley.com/doi/10.1002/inst.12427 by libffg rrtyu - Beijing Normal University , Wiley Online Library on [09/08/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Table 1. Manual acceptance criteria for model requirements
FEATURE
SPECIAL
36
21564868, 2023, 1, Downloaded from https://incose.onlinelibrary.wiley.com/doi/10.1002/inst.12427 by libffg rrtyu - Beijing Normal University , Wiley Online Library on [09/08/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Table 1. Manual acceptance criteria for model requirements (continued)
FEATURE
SPECIAL
Category Acceptance Criteria Examples
Demonstration
• Demonstration that model data can be exported and executed by external tools and that
results of external tools can be imported into the model.
Non-functional
• Acceptance criteria are that data interchange results are expected by analysis
requirements
• Demonstration that internal tools are capable of execution. Acceptance criteria are the
MARCH 2O23
VOLUME 26/ ISSUE 1
matching of model results with expected results validated by analysis
Test
• Constraint blocks in reliability and availability, or performance models provide the same results
General
as independent calculations
acceptably formatted, and display Manual Methods minority of the requirements, these scripts
the status of activities required for Table 1 summarizes manually evaluated cover many more elements and diagrams
safety certification approval at any model requirement acceptance criteria than manually verified requirements, and
architectural level. grouped by standard verification method (3) verification can be repeated at low cost.
■■ Cybersecurity Modeling: Requirements (inspection, analysis, demonstration, and Implementation of automated verifica-
for the input, storage, and export of test) and the model requirement topic areas tion differs across SysML modeling tools.
data necessary for deliverables for all shown in Figure 2. In Cameo Systems Modeler (3DS/Nomagic
cybersecurity standards or instructions Our experience using manual inspec- 2021), the tool used in this work, every
required under the program contract, tion in SysML models of multiple large rule violation needs to be associated with
all cybersecurity accreditation and projects is that use of checklists derived a model element. The output of the script
certification authorities, reports from the requirements catalog has resulted associated with the rule must be Boolean. If
acceptably formatted for all tasks in all in significantly faster and more thorough the output is false, the error message will be
cybersecurity standards or instructions. evaluations than with ad hoc inspection. displayed with the element that violates the
■■ Maintainability and Sustainability Manual inspections also detect program associated. An output of true would mean
Modeling: Requirements for the input, specific issues (as expressed in the content that there is no violation. The scripts are
storage, and export of data necessary of the model elements) that are not detect- written using the application programming
to support all deliverables for all able using automated methods (discussed interface in that tool to access model prop-
maintainability, sustainability, storage, in the next section). However, there are two erties and navigate to other elements.
repair, packaging, shipping, and disadvantages: (1) manual methods only Figure 3 shows a script to check con-
handling and other activities required inspect what is visible in the model, and (2)formance a requirement that in a state
for all maintainability, sustainability, manual inspection coverage of large models machine diagram, there should be only
storage, repair, packaging, shipping, is difficult to assess. one transition between any two states. As
and handling standards. shown in the figure, the scripting language
■■ Cybersecurity Modeling: Requirements Automated Methods is Jython (Python implemented in the Java
for the input, storage, and export of Automated methods rely on the capabili- Runtime Environment).
data necessary for deliverables for all ties of SysML modeling tools to run scripts The following examples show the
cybersecurity standards or instructions that query models to check SysML models results of application of these automated
required under the program contract, for conformance with model element, verification rules for use case, requirements,
all cybersecurity accreditation and relation, and connection requirements block definition, internal block, and
certification authorities, reports (sometimes called “validation rules”). A to- activity diagrams. The SysML model
acceptably formatted for all tasks in all tal of 127 requirements in the requirements used to demonstrate their application is
cybersecurity standards or instructions. catalog described earlier can be automati- a hypothetical satellite (Friedenthal and
■■ Maintainability and Sustainability cally validated. Benefits of automated rules Oster 2017). The model authors created it
Modeling: Requirements for the input, are that (1) they can identify subtle syntax as public domain example for identifying
storage, and export of data necessary errors far more rapidly and thoroughly than and evaluating alternative spacecraft
to support all deliverables for all manual inspection, (2) while they cover a architectures, and not as a full industrial
maintainability, sustainability, storage,
repair, packaging, shipping, and
handling and other activities required (A) Z7 (B) Z8
{on, off} Z9
for all maintainability, sustainability,
storage, repair, packaging, shipping, {light, no-light}
On Off
and handling standards. Z10
37
21564868, 2023, 1, Downloaded from https://incose.onlinelibrary.wiley.com/doi/10.1002/inst.12427 by libffg rrtyu - Beijing Normal University , Wiley Online Library on [09/08/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
grade model for representing a system. The model was not Parametric diagrams: Figure 7 shows a parametric diagram of
FEATURE
SPECIAL
intended to be comprehensive, complete, or rigorous. However, it a mass analysis. The automated verification rules detected (1) lack
is model is representative of medium sized projects. Because the of value property typing and (2) a parameter without a binding
authors have contributed it to the public domain under a BSD-2 connector.
license, it is also a valuable test and demonstration article. We Activity diagrams: Figure 8 shows an activity in which the
gratefully acknowledge the effort and expertise that were invested output pin outlined in red does not have an outgoing object flow
in its creation. detected by an automated validation script. Figure 9 shows a similar
Use case diagram. Figure 4 shows how the automated verifica- requirements violation for a pin that does not have an output flow,
tion found a nonconformance with the requirement actors must but also has another violation: Decision nodes must have a name.
MARCH 2O23
VOLUME 26/ ISSUE 1
«include»
S Provide Forest
«stakeholder» Firw Data in Near S
Real Time «stakeholder»
Operator
Fire Department
Archive Data
Figure 5. Output of automated verification rules on block Figure 7. Automated verification rules output in parametric
definition diagrams diagrams
38
21564868, 2023, 1, Downloaded from https://incose.onlinelibrary.wiley.com/doi/10.1002/inst.12427 by libffg rrtyu - Beijing Normal University , Wiley Online Library on [09/08/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
act [ Provide Electrical Power-p ]
FEATURE
SPECIAL
«allocate» «allocate» «allocate» «allocate»
Avionics Subsystem Power Subsystem Power Harness Subsystem
«allocate» «allocate» «allocate» «allocate» «allocate»
Solar Array Battery Power Mgmt SW Power Conditioner Power Distribution
:Generate
System :Manage
Commands Power
MARCH 2O23
VOLUME 26/ ISSUE 1
«valueType» v
:Electrical Power
{stream}
Figure 8. Activity diagram requirements violations detected by automated verification rules
:Sense
Spacecraft
Angular Rate
:Solar Radiation :Sense Sun :Spin Up
Angle :Generate
{stream} Reaction Wheel
SpinCommand «valueType»
:Reflected Light :Sense Earth 3 axis : Torque
{stream} Horizon Angle {stream}
:Generate
System :Spin Down
Commands
ManeuverCMD
:Sense
Reaction Wheel
Spin Rate
:Generate :Generate «valueType»
Thruster Thrust 3 axis : Thrust
Command {stream}
Figure 9. Requirements violations detected by automated verification rules displayed in the control attitude activity
REFERENCES
■■ 3DS/Nomagic. 2021. Cameo Systems Modeler. https://www. ■■ Jarraya, Y. and M. Debbabi. 2012. “Formal Specification and
nomagic.com/products/cameo-systems-modeler . Probabilistic Verification of SysML Activity Diagrams.” IEEE
■■ Ahmad, M. et al. 2013. Early Analysis of Ambient Systems Sixth International Symposium on Theoretical Aspects of
SysML Properties using OMEGA2IFx. Reykjavik, IS. https://hal. Software Engineering, Beijing, CN, 4-6 July.
archives-ouvertes.fr/hal-01085410. [Accessed 9 July 2020.] ■■ McDermott, T. et al. 2020. Benchmarking the Benefits and
■■ Baduel, R., M. Chami, I. Bruel, and I. Ober. 2018. SysML Current Maturity of Digital Engineering/Model-Based SE.
Models Verification and Validation in an Industrial Context: https://sercuarc.org/wpcontent/uploads/2020/03/Briefing_
Challenges and Experimentation. Lecture Notes in Computer Benchmarking-the-Benefits-and-Current-Maturity-ofMBSE-3-2020.
Science, 10890(ECMFA 2018: Modeling Foundation and pdf . Systems Engineering Research Center.
applications), pp. 132-146. ■■ Object Management Group. 2017. OMG SysML Version 1.5
■■ Douglas, B. 2017. Harmony MBSE Modeling Guidelines. http:// Specification. http://www.omg.org/spec/SysML/1.5/ .: s.n.
merlinscave.info/Merlins_Cave/Tutorials/Entries/2017/5/26_ ■■ Pettit, R. 2003. UML Inspection Criteria, s.l.: The Aerospace
Harmony_Modeling_ Guidelines_files/harmony%20mbse%20 Corporation, TOR 2003(1465)2412e.
modeling%20guidelines%202.0.pdf: IBM Rational. ■■ Rahim, M., A. Hammad, and M. Boukala-Iuoaleln. 2015.
■■ Friedenthal, S. and C. Oster. 2017. Architecting Spacecraft with Towards the Formal Verification of SysML Specifications:
SysML Web Site github site with BSD-2 clause simplified license Translation of Activity Diagrams into Modular Petri Nets. s.l.,
(permissions for commercial use, modification, distribution, s.n.
and private use). http://sysmlmodels.com/spacecraft/models.html ■■ Selic, B. 2012. “What will it take? A view on adoption of
[Accessed 20 September 2020]. model-based methods in practice.” Software and Systems
■■ Friedenthal, S. and C. Oster. 2017. Architecting Spacecraft with Modeling 11: 513-526.
SysML. s.l.:CreateSpace Independent Publishing Platform. ■■ Vinarcik, M. and H. Jugovich. 2020. Digital Engineering
■■ Friedenthal, S., A. Moore, and R. Steiner. 2015. A Practical Validation Tool Enables Efficiency Gains. https://www.saic.com/
Guide to SysML, 3rd Edition. blogs/digital-engineering-validation-tool-enables-efficiency-gains :
■■ Hecht, M., J. Chen, and G. Pugliese-Rosillo. 2021. Verification SAIC Corporation.
and Validation of SysML Models, El Segundo: Aerospace ■■ Wolny, S., A. Mazak, C. Carpella, et al. 2020. Thirteen years
Corporation. of SysML: a systematic mapping study. Software and Systems
■■ ISO/IEC/IEEE, 2015. Systems and Software Engineering Modeling 19: 111–169. https://doi.org/10.1007/s10270-01900735-y .
– System Life Cycle Processes. Geneva: International
Organization for Standardization (ISO)/International > continued on page 50
Electrotechnical Commission (IEC), Institute of Electrical and
Electronics Engineers. ISO/IEC/IEEE 15288:2015.
39