.. _sec-test-coverage: Test Coverage ============= The aim of tests is to get an impression about the quality of a given piece of software. When a HWUT component test reports ``Oll Korrect!`` this shall express a certain trustworthiness. For this to be true, it must be safe to assume that the tests actually test the component sufficiently. Thus, for a reasonable judgement a test result must be related to a measure what 'portion' of the software is actually tested. The quality of a component must therefore be derived from three parameters * A definition of the software component under test, i.e. which functions and modules are under test and which are not. * Some number that tells how much of the software is tested, i.e. a so called *test coverage*. * The test result. There are many different ways to measure test coverage, e.g. by means of *line coverage*, *branch coverage*, etc. In many systems a nominal behavior must be tested against a set of possible *scenarios*. Tests that are oriented versus plain input/output behavior may be measured sufficiently with static measures, such as line or branch coverage. For software components that manage an internal state, different scenarios become important. A meaningful quantitative measure for the test coverage still requires the judgement of a human reviewer. In this section, though, the *coverage* measure is introduced to help the developer of unit tests to set his focus and identify weak points of his tests. .. _sec-test-coverage-gcov: GCov for C/C++ .............. HWUT can interact with the GNU *gcov* utility to measure coverage. It either refers to 'gcov' as it is found in the current 'PATH' variable, or if the environment variable ``HWUT_GCOV_APPL`` is defined it relies on its content. GCov is a tool to measure line coverage, i.e. it can tell to what percentage the lines of a function are actually executed during the tests. In order to use this tool, the code must be compiled with the GNU compiler using the two compile options:: gcc ... -fprofile-arcs -ftest-coverage ... Note, also that for coverage measurement, the compilation should create an object file for each file under consideration. That means, you should *not* simply write make rules like .. code-block:: make my-test.exe: file_0.c file_1.c ... file_77.c gcc file_0.c file_1.c ... file_77.c -o my-test.exe Instead, the process is better split into .. code-block:: make my-test.exe: file_0.o file_1.o ... file_77.o gcc file_0.o file_1.o ... file_77.o -o my-test.exe %.o: %.c gcc -c -fprofile-arcs -ftest-coverage $< -o $@ The coverage of tests can be determined by means of the *gcov* service. That means, > hwut gcov executes a coverage test for all TEST directories in the sub-directory tree. During this process HWUT again interacts with the makefile in order to get some basic information required to do the measurement. There are three make targets that define the details of the coverage measurement: .. describe:: hwut-gcov-obj This target must return a whitespace separated list of directories that contain generated object files. .. describe:: hwut-gcov-info When HWUT calls ``make hwut-gcov-info`` the Makefile must return a whitespace separated list of C/C++ files that are to be considered for coverage. .. describe:: hwut-gcov-funcs The implementation of this target is optional. When it is implemented it shall report the names of functions that are actually tested. This way, functions can be excluded from consideration that are part of the object files required to compile, but which do not have anything to do with the test. Earlier it has been discussed that for a meaningful measurement it is essential that the software under test is sufficiently precise defined. In the case of *gcov* the software under test is defined by the list of object files, i.e. make target ``hwut-gcov-info`` and the list of functions, i.e. make target ``hwut-gcov-funcs``. The following Makefile fragment shows an example .. code-block:: make hwut-gcov-obj: @echo ./ $(MAIN_OBJ_DIR) hwut-gcov-info: @echo $(SRC_TEST) $(SRC_MINE) hwut-gcov-funcs: @echo my_component_do \ my_component_let \ my_component_drop \ my_component_catch It tells that the object files to be considered for coverage measurement are located either in the current directory or the directory referred to by variable ``MAIN_OBJ_DIR``. The source files to be considered are stored in the variables ``SRC_TEST`` and ``SRC_MINE``. Note, that if files are included in test files, such as to access static functions, then the test files *must* be specified in the ``hwut-gcov-info`` target. From all the functions in those files only the four functions reported by ``hwut-gcov-funcs`` are to be considered. The ``@`` at the begin of the echo command tells that the string ``echo`` is not printed on the standard output. This is essential to prevent ``echo`` itself from being considered as an object directory, source file, or function under test. In the result, functions that are omitted are printed in '(' ')' brackets. Due to compilations with different macro settings, the same function may appear in different object files. If this is the case, then the function name is preceeded by a '+' sign. The coverage value in this case is the coverage value of the maximum. .. note:: In some cases, a 'time stamp error' occurs. This may mean, that object files have been compiled multiple times. To avoid this in 'gmake' declare all object files as ``.SECONDARY`` targets. For example:: .SECONDARY: file-x.o file-y.o file-z.o This keeps gmake from repeatedly deleting and rebuilding them.