Test policy#

Test families and test functions#

Tree structure#

Root directory#

Tests are hosted in the XLIFEPP_HOME/tests directory. Here, there are XLIFEPP_HOME/tests/testUtils.hpp and XLIFEPP_HOME/tests/testUtils.cpp, defining a wide range of utility functions.

XLIFEPP_HOME/tests/testUtils.hpp has to be included by every test file.

Reference Data and log#

Reference data are stored in XLIFEPP_HOME/tests/inputs. They are organized in subdirectories whose names are the name of the test they are about. For instance, XLIFEPP_HOME/tests/unit_Domain.

Log files are stored in XLIFEPP_HOME/tests/res. Their names are the same as the test they are about. For instance, XLIFEPP_HOME/tests/res/unit_Domain.res.

About test families#

There are several families of tests:

unit:

Unitary tests are aimed to test each function of each class separately. In fact, it is easier to say it than to do it. But with a very large set of unitary tests, one can avoid most bugs.

sys:

System tests are aimed to test some user applications but with size of the linear systems not too big.

dev:

Developer tests are aimed to test new functionalities someone is developing. These tests are often like system tests, but are supposed to be deleted as soon as functionalities are fully developed. This family can also be used to check error convergence and cputime for larger problems.

ext:

External tests are aimed to test wrappers to external libraries, such as Arpack and UmfPack. These tests will be integrated in the unit family in the end.

Each family correspond to a subdirectory of XLIFEPP_HOME/tests.

Definition of a test function#

A test function is a usual C++ function with a meaningful name prefixed by the family name it belongs to defined in a file with the same name. The test function takes an optional boolean as input argument and returns a string (String class). For instance, for the unitary tests of the class GeomDomain, the file unit_Domain.cpp in the unit directory looks like:

/*! \file unit_Domain.cpp
    \author XXX
    \since YYY
    \date ZZZ
*/
#include "xlife++-libs.h"
#include "testUtils.h"

namespace unit_Domain {
...
void unit_Domain(bool check)
{
  String rootname = "unit_Domain";
  trace_p->push(rootname);
  Number nbErrors = 0;
  String errors;
  ...
  if (check)
  {
    if (nbErrors == 0) { theCout << message("test_report", rootname, 0, ""); }
    else { error("test_report", rootname, nbErrors, ":\n"+errors); }
  }
  else { theCout << "Data updated " << eol; }

  trace_p->pop();
}
} // end of namespace xlifepp

The errors variable has to contain information about the errors occurring during the test in order to easily locate them. When errors occurs, an error message of type test_report displays the number of errors (computed and stored in nberrors variable) and every error, by specifying which part of the tests and which line of the generated data are concerned.

Attention

It is up to developers to write meaningful tests and as many exhaustive tests of the class functionalities as possible.

Development of a test function#

There are mainly two approaches to make a test:

  • Internal value testing: compare results to expected results inside the function (value comparison),

  • external value testing: comparing results to reference results get previously and stored in a file (ASCII comparison).

Both of them work with a wide range of checkValues() or checkValue available functions, overloaded for quite every kind of datatype to be tested. These functions are defined in XLIFEPP_HOME/tests/testUtils.hpp and XLIFEPP_HOME/tests/testUtils.cpp.

For instance, the following internal value testing comes from unit_HMatrix (the computed norm is tested to be equal to 9.03223055164e-17 with a tolerance or 1e-6.):

nbErrors+=checkValue(norm(Bx-B0x), 9.03223055164e-17, 1e-6, errors, "exact HM/LM |Bx-B0x|");

and the following external value testing comes from unit_Geometry (the string to be tested contains a range of unitary tests of member functions of class BoundingBox):

nbErrors += checkValue(theCout, rootname+"/BoundingBox.in", errors, "Test of BoundingBox class", check);

Here, you have the general case of external value testing. By using the theCout stream in tests, you fill the log file and a string. The content of this string will be compared with the content of a reference file (.in extension).

The external value testing is rather dedicated to the tests of classes and functionalities which involve a lot of basic tests.

Warning

checkvalue() and checkValues() routines clears the internal string of variable theCout. If you intend to write a “test” whose output is not meant to be tested but only printed in the log file, you have to use thePrintStream instead of theCout.

Includes#

As you may have noticed in previous listings, a test file, whatever its family always includes 2 headers: xlife++-libs.h and testUtils.hpp.

Test of a new version of the library#

This section addresses the process of global test involved before a minor or major upgrade of the library. It concerns the administrators of the library.

Compilation of tests#

When compiling tests, each individual test is compiled, but also one global test per family. There is also a specific global test running both unit and sys test families. When compiling a global test, CMake sources all files in the corresponding directory prefixed by the family. If a source file does not respect this naming convention, it is ignored.

Running tests#

The administrators have a special tool to run a lot of tests, xlifepp_test_runner.rb. This is a Ruby script that runs one executable without compiling it if it does not exist.

xlifepp_test_runner.rb deals with all automatic tasks such as updating |doxygen|, generating
    commit history and generating snapshots and releases

SYNOPSIS:
    xlifepp_test_runner.rb -c [-n] <test>
    xlifepp_test_runner.rb -e [-n] <test>
    xlifepp_test_runner.rb -h

DESCRIPTION:
    XLiFE++ is a C++ library with a large set of tests. Each test can be launched in 2 modes :
      - the edit mode (default in XLiFE++ executables) makes tests generating results file, so-called res file
      - the check mode (default in xlifepp_test_runner.rb) makes tests comparing results to the current res file

    As XLiFE++ compilation is based on CMake, it is not a good way to define running targets, because
    of generators (to Eclipse, XCode, Visual Studio, `CodeBlocks`_, ...). On the one hand, the lesser targets
    there are the easier XLiFE++ is to compile and run, on the other hand targets in generators are not just
    compilation targets: you can run them, and pass parameters to them.
    However, XLiFE++ developers using cmake in command-line mode must have an easy way to run tests.
    That's what xlifepp_test_runner.rb is for !!!

OPTIONS:
    -h, --help                     shows the current help
    -c, --check                    test will be run in check mode (default)
    -cxx, --cxx-compiler           sets the compiler to use
    -e, --edit                     test will be run in edit mode
    <test>                         name of the test (Examples: all, unit_Mesh_gmsh, sys_1d, ...)
    -n                             activates dry-run mode
    -v                             activates verbose level (same as --verbose-level 1)
    --verbose-level <num>          allows to set the verbose level. Default is 0, none.
On this page