Correctness of the simulation model is a primary concern of the developers and users of the model, because they want to obtain credible simulation results. Verification and validation are activities conducted during the development of a simulation model with the ultimate goal of producing an accurate and credible model.
Of the two, verification is essentially a software engineering issue, so it can be assisted with tools used for software quality assurance, for example testing tools. Validation is not a software engineering issue.
As mentioned above, software testing techniques can be of significant help during model verification. Testing can also help to ensure that a simulation model that once passed validation and verification will also remain correct for an extended period.
Software testing is an art on its own, with several techniques and methodologies. Here we'll only mention two types that are important for us, regression testing and unit testing.
The two may overlap; for example, unit tests are also useful for discovering regressions.
One way of performing regression testing on an OMNeT++ simulation model is to record the log produced during simulation, and compare it to a pre-recorded log. The drawback is that code refactoring may nontrivially change the log as well, making it impossible to compare to the pre-recorded one. Alternatively, one may just compare the result files or only certain simulation results and be free of the refactoring effects, but then certain regressions may escape the testing. This type of tradeoff seems to be typical for regression testing.
Unit testing of simulation models may be done on class level or module level. There are many open-source unit testing frameworks for C++, for example CppUnit, Boost Test, Google Test, UnitTest++, just to name a few. They are well suited for class-level testing. However, they are usually cumbersome to apply to testing modules due to the peculiarities of the domain (network simulation) and OMNeT++.
A test in an xUnit-type testing framework (a collective name for CppUnit-style frameworks) operates with various assertions to test function return values and object states. This approach is difficult to apply to the testing of OMNeT++ modules that often operate in a complex environment (cannot be easily instantiated and operated in isolation), react to various events (messages, packets, signals, etc.), and have complex dynamic behavior and substantial internal state.
Later sections will introduce opp_test, a tool OMNeT++ provides for assisting various testing task; and summarize various testing methods useful for testing simulation models.
This section documents the opp_test, a versatile tool that is helpful for various testing scenarios. opp_test can be used for various types of tests, including unit tests and regression tests. It was originally written for testing the OMNeT++ simulation kernel, but it is equally suited for testing functions, classes, modules, and whole simulations.
opp_test is built around a simple concept: it lets one define simulations in a concise way, runs them, and checks that the output (result files, log, etc.) matches a predefined pattern or patterns. In many cases, this approach works better than inserting various assertions into the code (which is still also an option).
Each test is a single file, with the .test file extension. All NED code, C++ code, ini files and other data necessary to run the test case as well as the PASS criteria are packed together in the test file. Such self-contained tests are easier to handle, and also encourage authors to write tests that are compact and to the point.
Let us see a small test file, cMessage_properties_1.test:
%description:
Test the name and length properties of cPacket.
%activity:
cPacket *pk = new cPacket();
pk->setName("ACK");
pk->setByteLength(64);
EV << "name: " << pk->getName() << endl;
EV << "length: " << pk->getByteLength() << endl;
delete pk;
%contains: stdout
name: ACK
length: 64
What this test says is this: create a simulation with a simple module that has the above C++ code block as the body of the activity() method, and when run, it should print the text after the %contains line.
To run this test, we need a control script, for example runtest from the omnetpp/test/core directory. runtest itself relies on the opp_test tool.
The output will be similar to this one:
$ ./runtest cMessage_properties_1.test opp_test: extracting files from *.test files into work... Creating Makefile in omnetpp/test/core/work... cMessage_properties_1/test.cc Creating executable: out/gcc-debug/work opp_test: running tests using work.exe... *** cMessage_properties_1.test: PASS ======================================== PASS: 1 FAIL: 0 UNRESOLVED: 0 Results can be found in work/
This was a passing test. What would constitute a fail?
One normally wants to run several tests together. The runtest script accepts several .test files on the command line, and when started without arguments, it defaults to *.test, all test files in the current directory. At the end of the run, the tool prints summary statistics (number of tests passed, failed, and being unresolved).
An example run from omnetpp/test/core (some lines were removed from the output, and one test was changed to show a failure):
$ ./runtest cSimpleModule-*.test opp_test: extracting files from *.test files into work... Creating Makefile in omnetpp/test/core/work... [...] Creating executable: out/gcc-debug/work opp_test: running tests using work... *** cSimpleModule_activity_1.test: PASS *** cSimpleModule_activity_2.test: PASS [...] *** cSimpleModule_handleMessage_2.test: PASS *** cSimpleModule_initialize_1.test: PASS *** cSimpleModule_multistageinit_1.test: PASS *** cSimpleModule_ownershiptransfer_1.test: PASS *** cSimpleModule_recordScalar_1.test: PASS *** cSimpleModule_recordScalar_2.test: FAIL (test-1.sca fails %contains-regex(2) rule) expected pattern: >>>>run General-1-.*? scalar Test one 24.2 scalar Test two -1.5888<<<< actual output: >>>>version 2 run General-1-20141020-11:39:34-1200 attr configname General attr datetime 20141020-11:39:34 attr experiment General attr inifile _defaults.ini [...] scalar Test one 24.2 scalar Test two -1.5 <<<< *** cSimpleModule_recordScalar_3.test: PASS *** cSimpleModule_scheduleAt_notowner_1.test: PASS *** cSimpleModule_scheduleAt_notowner_2.test: PASS [...] ======================================== PASS: 36 FAIL: 1 UNRESOLVED: 0 FAILED tests: cSimpleModule_recordScalar_2.test Results can be found in work/
Note that code from all tests were linked to form a single executable, which saves time and disk space compared to per-test executables or libraries.
A test file like the one above is useful for unit testing of classes or functions. However, as we will see, the test framework provides further facilities that make it convenient for testing modules and whole simulations as well. The following sections go into details about the syntax and features of .test files, about writing the control script, and give advice on how to cover several use cases with the opp_test tool.
The next sections will use the following language:
Test files are composed of %-directives of the syntax:
%<directive>: <value> <body>
The body extends up to the next directive (the next line starting with %), or to the end of the file. Some directives require a value, others a body, or both.
Certain directives, e.g. %contains, may occur several times in the file.
Syntax:
%description: <test-description-lines>
%description is customarily written at the top of the .test file, and lets one provide a multi-line comment about the purpose of the test. It is recommended to invest time into well-written descriptions, because they make determining the original purpose of a test that has become broken significantly easier.
This section describes the directives used for creating C++ source and other files in the test directory.
Syntax:
%activity: <body-of-activity()>
%activity lets one write test code without much boilerplate. The directive generates a simple module that contains a single activity() method with the given code as method body.
A NED file containing the simple module's (barebones) declaration, and an ini file to set up the module as a network are also generated.
Syntax:
%module: <modulename> <simple-module-C++-definition>
%module lets one define a module class and run it as the only module in the simulation.
A NED file containing the simple module's (barebones) declaration, and an ini file to set up the module as a network are also generated.
Syntax:
%includes: <#include directives>
%global: <global-code-pasted-before-activity>
%includes and %global are helpers for %activity and %module, and let one insert additional lines into the generated C++ code.
Both directives insert the code block above the module C++ declaration. The only difference is in their relation to the C++ namespace: the body of %includes is inserted above (i.e. outside) the namespace, and the body of %globals is inserted inside the namespace.
The following ini file is always generated:
[General] network = <network-name> cmdenv-express-mode = false
The network name in the file is chosen to match the module generated with %activity or %module; if they are absent, it will be Test.
Syntax:
%network: <network-name>
This directive can be used to override the network name in the default ini file.
Syntax:
%file: <file-name> <file-contents>
%inifile: [<inifile-name>] <inifile-contents>
%file saves a file with the given file name and content into the test's extraction folder in the preparation phase of the test run. It is customarily used for creating NED files, MSG files, ini files, and extra data files required by the test. There can be several %file sections in the test file.
%inifile is similar to %file in that it also saves a file with the given file name and content, but it additionally also adds the file to the simulation's command line, causing the simulation to read it as an (extra) ini file. There can be several %inifile sections in the test file.
The default ini file is always generated.
In test files, the string @TESTNAME@ will be replaced with the test case name. Since it is substituted everywhere (C++, NED, msg and ini files), one can also write things like @TESTNAME@_function(), or printf("this is @TESTNAME@\n").
Since all sources are compiled into a single test executable, actions have to be taken to prevent accidental name clashes between C++ symbols in different test cases. A good way to ensure this is place all code into namespaces named after the test cases.
namespace @TESTNAME@ {
...
};
This is done automatically for the %activity, %module, %global blocks, but for other files (e.g. source files generated via %file, that needs to be done manually.
Syntax:
%contains: <output-file-to-check> <multi-line-text>
%contains-regex: <output-file-to-check> <multi-line-regexp>
%not-contains: <output-file-to-check> <multi-line-text>
%not-contains-regex: <output-file-to-check> <multi-line-regexp>
These directives let one check for the presence (or absence) of certain text in the output. One can check a file, or the standard output or standard error of the test program; for the latter two, stdout and stderr needs to be specified as file name, respectively. If the file is not found, the test will be marked as unresolved. There can be several %contains-style directives in the test file.
The text or regular expression can be multi-line. Before match is attempted, trailing spaces are removed from all lines in both the pattern and the file contents; leading and trailing blank lines in the patterns are removed; and any substitutions are performed (see %subst). Perl-style regular expressions are accepted.
To facilitate debugging of tests, the text/regex blocks are saved into the test directory.
Syntax:
%subst: /<search-regex>/<replacement>/<flags>
It is possible to apply text substitutions to the output before it is matched against expected output. This is done with %subst directive; there can be more than one %subst in a test file. It takes a Perl-style regular expression to search for, a replacement text, and flags, in the /search/replace/flags syntax. Flags can be empty or a combination of the letters i, m, and s, for case-insensitive, multi-line or single-string match (see the Perl regex documentation.)
%subst was primarily invented to deal with differences in printf output across platforms and compilers: different compilers print infinite and not-a-number in different ways: 1.#INF, inf, Inf, -1.#IND, nan, NaN etc. With %subst, they can be brought to a common form:
%subst: /-?1\.#INF/inf/ %subst: /-?1\.#IND/nan/ %subst: /-?1\.#QNAN/nan/ %subst: /-?NaN/nan/ %subst: /-?nan/nan/
Syntax:
%exitcode: <one-or-more-numeric-exit-codes>
%ignore-exitcode: 1
%exitcode and %ignore-exitcode let one test the exit code of the test program. The former checks that the exit code is one of the numbers specified in the directive; the other makes the test framework ignore the exit code.
OMNeT++ simulations exit with zero if the simulation terminated without an error, and some >0 code if a runtime error occurred. Normally, a nonzero exit code makes the test fail. However, if the expected outcome is a runtime error (e.g. for some negative test cases), one can use either %exitcode to express that, or specify %ignore-exitcode and test for the presence of the correct error message in the output.
Syntax:
%file-exists: <filename>
%file-not-exists: <filename>
These directives test for the presence or absence of a certain file in the test directory.
Syntax:
%env: <environment-variable-name>=<value>
%extraargs: <argument-list>
%testprog: <executable>
The %env directive lets one set an environment variable that will be defined when the test program and the potential pre- and post-processing commands run. There can be multiple %env directives in the test file.
%extraargs lets one add extra command-line arguments to the test program (usually the simulation) when it is run.
The %testprog directive lets one replace the test program. %testprog also slightly alters the arguments the test program is run with. Normally, the test program is launched with the following command line:
$ <default-testprog> -u Cmdenv <test-extraargs> <global-extraargs> <inifiles>
When %testprog is present, it becomes the following:
$ <custom-testprog> <test-extraargs> <global-extraargs>
That is, -u Cmdenv and <inifilenames> are removed; this allows one to invoke programs that do not require or understand them, and puts the test author in complete command of the arguments list.
Note that %extraargs and %testprog have an equivalent command-line option in opp_test. (In the text above, <global-extraargs> stands for extra args specified to opp_test.) %env doesn't need an option in opp_test, because the test program inherits the environment variables from opp_test, so one can just set them in the control script, or in the shell one runs the tests from.
Syntax:
%prerun-command: <command>
%postrun-command: <command>
These directives let one run extra commands before/after running the test program (i.e. the simulation). There can be multiple pre- and post-run commands. The post-run command is useful when the test outcome cannot be determined by simple text matching, but requires statistical evaluation or other processing.
If the command returns a nonzero exit code, the test framework will assume that it is due to a technical problem (as opposed to test failure), and count the test as unresolved. To make the test fail, let the command write a file, and match the file's contents using %contains & co.
If the post-processing command is a short script, it is practical to add it into the .test file via the %file directive, and invoke it via its interpreter. For example:
%postrun-command: python test.py %file: test.py <Python script>
Or:
%postrun-command: R CMD BATCH test.R %file: test.R <R script>
If the script is very large or shared among several tests, it is more practical to place it into a separate file. The test command can find the script e.g. by relative path, or by referring to an environment variable that contains its location or full path.
A test case is unresolved if the test program cannot be executed at all, the output cannot be read, or if the test case declares so. The latter is done by printing #UNRESOLVED or #UNRESOLVED:some-explanation on the standard output, at the beginning of the line.
Little has been said so far what opp_test actually does, or how it is meant to be run. opp_test has two modes: file generation and test running. When running a test suite, opp_test is actually run twice, once in file generation mode, then in test running mode.
File generation mode has the syntax opp_test gen <options> <testfiles>. For example:
$ opp_test gen *.test
This command will extract C++ and NED files, ini files, etc., from the .test files into separate files. All files will be created in a work directory (which defaults to ./work/), and each test will have its own subdirectory under ./work/.
The second mode, test running, is invoked as opp_test run <options> <testfiles>. For example:
$ opp_test run *.test
In this mode, opp_test will run the simulations, check the results, and report the number of passes and failures. The way of invoking simulations (which executable to run, the list of command-line arguments to pass, etc.) can be specified to opp_test via command-line options.
The simulation needs to have been built from source before opp_test run can be issued. Usually one would employ a command similar to
$ cd work; opp_makemake --deep; make
to achieve that.
Usually one writes a control script to automate the two invocations of opp_test and the build of the simulation model between them.
A basic variant would look like this:
#! /bin/sh opp_test gen -v *.test || exit 1 (cd work; opp_makemake -f --deep; make) || exit 1 opp_test run -v *.test
For any practical use, the test suite needs to refer to the codebase being tested. This means that the codebase must be added to the include path, must be linked with, and the NED files must be added to the NED path. The first two can be achieved by the appropriate parameterization of opp_makemake; the last one can be done by setting and exporting the NEDPATH environment variable in the control script.
For inspiration, check out runtest in the omnetpp/test/core directory, and a similar script used in the INET Framework.
Further sections describe how one can implement various types of tests in OMNeT++.
Smoke tests are a tool for very basic verification and regression testing. Basically, the simulation is run for a while, and it must not crash or stop with a runtime error. Naturally, smoke test provide very low confidence in the model, but in turn they are very easy to implement.
Automation is important. The INET Framework contains a script that runs all or selected simulations defined in a CSV file (with columns like the working directory and the command to run), and reports the results. The script can be easily adapted to other models or model frameworks.
Fingerprint tests are a low-cost but effective tool for regression testing of simulation models. A fingerprint is a hash computed from various properties of simulation events, messages and statistics. The hash value is continuously updated as the simulation executes, and thus, the final fingerprint value is a characteristic of the simulation's trajectory. For regression testing, one needs to compare the computed fingerprints to that from a reference run -- if they differ, the simulation trajectory has changed. In general, fingerprint tests are very useful for ensuring that a change (some refactoring, a bugfix, or a new feature) didn't break the simulation.
Technically, providing a fingerprint option in the config file or on the command line (-fingerprint=...) will turn on fingerprint computation in the OMNeT++ simulation kernel. When the simulation terminates, OMNeT++ compares the computed fingerprints with the provided ones, and if they differ, an error is generated.
The fingerprint computation algorithm allows controlling what is included in the hash value. Changing the ingredients allows one to make the fingerprint sensitive for certain changes while keeping it immune to others.
The ingredients of a fingerprint are usually indicated after a / sign following the hexadecimal hash value. Each ingredient is identified with a letter. For example, t stands for simulation time. Thus, the following omnetpp.ini line
fingerprint = 53de-64a7/tplx
means that a fingerprint needs to be computed with the simulation time, the module full path, received packet's bit length and the extra data included for each event, and the result should be 53de-64a7.
The full list of fingerprint ingredients:
Ingredients may also be specified with the fingerprint-ingredients configuration option. However, that is rarely necessary, because the ingredients list included in the fingerprints take precedence, and are also more convenient to use.
It is possible to specify more than one fingerprint, separated by commas, each with different ingredients. This will cause OMNeT++ to compute multiple fingerprints, and all of them must match for the test to pass. An example:
fingerprint = 53de-64a7/tplx, 9a3f-7ed2/szv
Occasionally, the same simulation gives a different fingerprint when run on
a different processor architecture or platform. This is due to subtle
differences in floating point arithmetic across platforms.
fingerprint = 53de-64a7/tplx 63dc-ff21/tplx, 9a3f-7ed2/szv da39-91fc/szv
Note that fingerprint computation has been changed and significantly
extended in OMNeT++ version 5.0.
It is also possible to filter which modules, statistics, etc. are included in the fingerprints. The fingerprint-events, fingerprint-modules, and fingerprint-results options filter by events, modules, and statistical results, respectively. These options take wildcard expressions that are matched against the corresponding object before including its property in the fingerprint. These filters are mainly useful to limit fingerprint computation to certain parts of the simulation.
cFingerprintCalculator is the class responsible for fingerprint computation. The current fingerprint computation object can be retrieved from cSimulation, using the getFingerprintCalculator() member function. This method will return nullptr if fingerprint computation is turned off for the current simulation run.
To contribute data to the fingerprint, cFingerprintCalculator has several addExtraData() methods for various data types (string, long, double, byte array, etc.)
An example (note that we check the pointer for nullptr to decide whether a fingerprint is being computed):
cFingerprintCalculator *fingerprint = getSimulation()->getFingerprintCalculator();
if (fingerprint) {
fingerprint->addExtraData(retryCount);
fingerprint->addExtraData(rttEstimate);
}
Data added using addExtraData() will only be counted in the fingerprint if the list of fingerprint ingredients contains x (otherwise addExtraData() does nothing).
The INET Framework contains a script for automated fingerprint tests as well. The script runs all or selected simulations defined in a CSV file (with columns like the working directory, the command to run, the simulation time limit, and the expected fingerprints), and reports the results. The script is extensively used during INET Framework development to detect regressions, and can be easily adapted to other models or model frameworks.
Exerpt from a CSV file that prescribes fingerprint tests to run:
examples/aodv/, ./run -f omnetpp.ini -c Static, 50s, 4c29-95ef/tplx examples/aodv/, ./run -f omnetpp.ini -c Dynamic, 60s, 8915-f239/tplx examples/dhcp/, ./run -f omnetpp.ini -c Wired, 800s, e88f-fee0/tplx examples/dhcp/, ./run -f omnetpp.ini -c Wireless, 500s, faa5-4111/tplx
If a simulation models contains units of code (classes, functions) smaller than a module, they are candidates for unit testing. For a network simulation model, examples of such classes are network addresses, fragmentation reassembly buffers, queues, various caches and tables, serializers and deserializers, checksum computation, etc.
Unit tests can be implemented as .test files using the opp_test tool (the %activity directive is especially useful here), or with potentially any other C++ unit testing framework.
When using .test files, the build part of the control script needs to be set up so that it adds the tested library's source folder(s) to the include path, and also links the library to the test code.
OMNeT++ modules are not as easy to unit test as standalone classes, because they typically assume a more complex environment, and, especially modules that implement network protocols, participate in more complex interactions than the latter.
To test a module in isolation, one needs to place it into a simulation where the module's normal operation environment (i.e. other modules it normally communicates with) are replaced by mock objects. Mock objects are responsible for providing stimuli for the module under test, and (partly) for checking the response.
Module tests may be implemented in .test files using the opp_test tool. A .test file allows one to place the test description, the test setup and the expected output into a single, compact file, while large files or files shared among several tests may be factored out and only referenced by .test files.
Statistical tests are those where the test outcome is decided on some statistical property or properties of the simulation results.
Statistical tests may be useful as validation as well as regression testing.
Validation tests aim to verify that simulation results correspond to some reference values, ideally to those obtained from the real system. In practice, reference values may come from physical measurements, theoretical values, or another simulator's results.
After a refactoring that changes the simulation trajectory (e.g. after eliminating or introducing extra events, or changes in RNG usage), there may be no other way to do regression testing than checking that the model produces statistically the same results as before.
For statististical regression tests, one needs to perform several simulation runs with the same configuration but different RNG seeds, and verify that the results are from the same distributions as before. One can use Student's t-test (for mean) and the F-test (for variance) to check that the “before” and the “after” sets of results are from the same distribution.
Statistical software like GNU R is extremely useful for these tests.
Statistical tests may also be implemented in .test files. To let the tool run several simulations within one test, one may use %extraargs to pass the -r <runs> option to Cmdenv; alternatively, one may use %testprog to have the test tool run opp_runall instead of the normal simulation program. For doing the statistical computations, one may use %postrun-command to run an R script. The R script may rely on the omnetpp R package for reading the result files.
The INET Framework contains statistical tests where one can look for inspiration.