hwut 1.0 documentation
previous | next | index

Installation

Following stepts guide towards a working HWUT environment:

Python Language

HWUT requires the Python Programming Language to be installed. The website http://python.org <http://www.python.org>` maintains downloadable packages. Please, note that for HWUT, the python version shall be lesser than 3.0.

Unpack HWUT

To install HWUT, a package of the form hwut-10y02m11d.tbz is required. It must be unpacked in an appropriate directory on your machine. The directory where HWUT is unpacked must be written to the environment variable HWUT_PATH, as explained later.

On Windows, copy the file hwut.bat to C:\WINDOWS\SYSTEM32\ or any other place in your system’s path. The file contains a line of the type:

python %HWUT_PATH%\hwut-exe.py ...

If you have multiple versions of Python installed, please, replace python with the exact path to a python version below 3.0, e.g.:

C:\Python26\python %HWUT_PATH%\hwut-exe.py ...

On Unix machines and the likes please, add a symbolic link from a bin directory to hwut-exe.py and set executable rights, i.e.:

ln -s     $HWUT_PATH/hwut-exe.py  /usr/local/bin/hwut
chmod a+x $HWUT_PATH/hwut-exe.py  /usr/local/bin/hwut

You might also ensure executable rights for all subdirectories, so that hwut can write byte compiled files, i.e.:

chmod a+wx -R $HWUT_PATH
Diff Display Application

If the diff-display feature shall work, then HWUT requires a diff-viewer, such as ‘vimdiff’, ‘gvimdiff’, ‘BeyondCompare2’, ‘WinMerge’ or ‘TkDiff’. Those diff-display tools either come with the distribution of your operating system, or are available on the internet. Later in this section it is explained how to setup hwut for a particular diff display application.

By default, HWUT relies on the vimdiff application which is installed on most Unix machines.

Compile Environment

For a solid compile environment at least a version of the make utility shall be installed. Preferably, GNU-make <http://www.gnu.org/software/make/> may be used. For coverage measurements in C-programs gcov must be installed together with a version of the GNU C/C++ Compiler. For Microsoft Windows the MinGW-Suite provides all required applications in one. For Unix based systems one can most probably rely on the distribution CD/DVD.

Environment Variables

Once all applications are present, the HWUT environment variables can be specified, as they are

HWUT_PATH(mandatory)

must contain the directory path to the place where HWUT was installed. Please, use double quotes if the pathname contains whitespaces.

HWUT_DIFF_APPL(optional)

must contain the path and name of the difference display application. By default ‘vimdiff’ is used. Note, that the diff display application can also be specified on the command line.

HWUT_TEST_DIR_NAME(optional)

By default, HWUT checks for directories that are called TEST. If this shall be different, the environment variable above allows to specify customized directory name.

HWUT_MAKE_APPL(optional)

For interaction with Makefiles, HWUT requires a version of make. By default it uses the make that is available through the system’s PATH variable. To choose a different make, set the value of HWUT_MAKE_APPL.

HWUT_GCOV_APPL(optional)

By default HWUT uses gcov as available in the system’s path. To use a specific gcov application set the value of HWUT_GCOV_APPL. This variable needs only to be specified if coverage measurements in ‘C’ are to be performed and an application different from ‘gcov’ is to be used.

HWUT_TERMINAL_WIDTH(optional)

Determines the width of the user’s terminal in characters. If this variable is not defined, a terminal width of 80 characters is assumed. The terminal width can also be specified for each call to hwut by the command line options -w or --terminal-width.

On Unix machines, these variables are best stored in the .bashrc-file (or whatever shell you use), or /etc/profile. On Microsoft Windows (tm) clicking [Start][Settings][ControlPannel]–>[System][Advanced] opens a tab window with a button [Environment Variables]. Click this button and define the environment variables above.

The following sections explain the basic usage of HWUT. They show how with a few commands and concepts software tests on the level on Unit Tests, i.e. tests on the per function-level, and System Tests, i.e. tests on the behavior-level can be performed and managed. In scenarios where the system’s behavior is no longer fully deterministic temporal logic rules need to be applied. Those, however, are explained in later chapters. Again, HWUT supports deterministic tests and temporal logic test. This introductory section only elaborates on deterministic tests.

Basic Concept

The basic idea of HWUT is that a test-application interacts with software to be tested. During this process strings are printed to the standard output. The human tester checks this output until he is of the opinion that it is correct. He invoques HWUT with the command accept and HWUT stores it away for reference. HWUT is now able to compile and execute the test application, compare its output to the reference and judge wether it is the same or not. If it is not the same, the test has failed.

Test observation based on text passed to the standard output.

Judgement based on textual comparison.

What HWUT does is that it look for all test applications in a TEST directory. There are lines which are excluded from comparison: Lines that start or end with a double hash mark ## are not considered. This facilitates the implementation of debug output that is not subject to the test itself. If the reference output is

Init Elements --> assigned 10.
Sort Elements --> list size = 10.
list: 90, 94, 100, 201, 202, 401, 510, 550, 3123, 53123
Destruct list --> pointer set to 0x0.

The this is equivalent to

Init Elements --> assigned 10.
## quicksort split at i = 5
## ...
## quicksort done
Sort Elements --> list size = 10.
list: 90, 94, 100, 201, 202, 401, 510, 550, 3123, 53123
## destructor catches exception!!!
Destruct list --> pointer set to 0x0.

Except for the comment lines, both outputs are the same. Note, also that multiple spaces and tabulator marks are reduced to one single space. Thus,

print "This is an output"

and

print "This is an   \t output"

write equivalent strings to the standard output.

Given a specific TEST directory, HWUT executes them and compares their output with the reference that it stored earlier. The test result is then printed as follows:

2010y02m02d 16h55
(this directory)
make: just-for-fun.exe .....................................[MADE]
make: just-for-gaudi.exe ...................................[MADE]
make: notification-failure-central-registry.exe ............[MADE]
make: notification-failure-central-registry-2.exe ..........[MADE]
-------------------------------------------------------------------

 -- Notification:

    Fail 5x, then CentralRegistry.
        notification-failure-central-registry.exe .......[OK]
    Notification Fail, then CR, but FBlock Vanishes.
        notification-failure-central-registry-2.exe .....[OK]

 -- Via Notification:

    Simple
        just-for-fun.exe ................................[OK]

 -- Via Requests:

    Simple
        just-for-gaudi.exe Expected .....................[OK]
                           UnExpected ...................[OK]

     ___  _ _   _  __                        _     _
    / _ \| | | | |/ /___  _ __ _ __ ___  ___| |_  | |
   | | | | | | | ' // _ \| '__| '__/ _ \/ __| __| | |
   | |_| | | | | . \ (_) | |  | | |  __/ (__| |_  |_|
    \___/|_|_| |_|\_\___/|_|  |_|  \___|\___|\__| (_)

===================================================================

The final Oll Korrect states that all tests in this directory have passed. If a test would not have passed a [FAIL] flag would be reported instead of [OK] and the total result would be Failure.

Directory Structure

Three subdirectories in a TEST directories are used by HWUT for special purposes–as they are:

ADM

In this subdirectory administrative files are located. HWUT keeps track of the history of testing and the current state of testing. User editable files in this directory are:

title.txt

Which contains the name of the tests in the TEST directory. This is usualy the name of the module or function group to be tested.

scripts.txt

This file contains information about how to distinguish between files in TEST that are test applications and those which are not. The default distinction is usually sufficient. Please, consider section List of Test Applications for more details.

OUT

This subdirectory contains the output files of the last run of the test application. The content of this directory can be deleted without any harm.

GOOD

This directory contains the ‘treasured gold’, i.e. the reference output of test runs that were judged to be correct. Those files result from competent human observation of the system behavior and they preserve knowledge about requirements and behavioral details.

With the files in this directory software tests can be performed even if detailed knowledge of this system is lost or forgotten. When using a configuration management system, it must be ensured that those files are kept safe and sound.

The files in OUT and GOOD are under control of HWUT. There is no need for the user to tamper with the files in these two directories.

Test Applications

In a first step HWUT needs to determine the list test applications in the current TEST directory. It does so by collecting executable files in the directory and, optionally, asking a Makefile about applications to be built by make procedure. In a second step those test applications are interviewed, i.e. they are called with a command line argument --hwut-info. HWUT then reads the standard output of the program and interprets it. The first line of the application’s response is interpreted as test title. For example a Python test program mytest.py:

if "--hwut-info" in sys.argv:
    print "My Stuff's Test"
    sys.exit()

tells HWUT that its title is My Stuff's Test. This title is then used when printing the test results, e.g.:

My Stuff's Test
    mytest.py .......................................[OK]

The most important part, though, is still missing: the textual output that tells about the behavior of the software under test. Imagine a simple program that computes Fibonacci numbers and is located in module “main/fibonacci.py”. A simple test file fibo-test.py might look like

import main.fibonacci as fibonacci

if "--hwut-info" in sys.argv:
    print "Fibonacci Test"
    sys.exit()

print 0, "-->", fibonacci.compute(0)
print 1, "-->", fibonacci.compute(1)
print 997, "-->", fibonacci.compute(997)
print 998, "-->", fibonacci.compute(998)

When this program is run, four lines are printed, i.e.:

0 --> 0
1 --> 1
997 --> 497503
998 --> 498501

Once, it is safe to say that the output is correct and covers the intended functionality, HWUT can accept it, i.e. store the text it as a reference in the GOOD directory. This is done on the command line by:

> hwut a fibo-test.py

Provided that the file is executable, HWUT can now execute tests on its own and judge if the test succeeds or fails. It simply compares its current text output and the text that was stored as a reference. To run HWUT on th current TEST directory simply type:

> hwut

on the command line, and all test applications in the current directory are evaluated. If you only want to execute one particular test application special-test.sh, then type:

> hwut special-test.sh

and only this particular test is performed. If an error occurs, the difference between the current output and the reference output can be viewed by typing:

> hwut dd special-test.sh

provided that the environment variable HWUT_DIFF_APPL points to a diff-display program. If the variable is not set, hwut relies on vimdiff so then this program should be installed.

If a GOOD file has to be built incrementally, the ai accept incrementally option may be used. It works like dd only that it creates an empty file if the GOOD file does not exist. The merge tool may then be used to copy the elements of the output which are fine.

Note

Some people write HWUT tests in a DOS/Windows(tm) environment, others in a Unix environment. Pathnames and things may different significantly. So, on larger projects this may cause confusion. For such cases, HWUT provides a bridge from DOS/Windows to CygWin (Unix World inside Windows). That means, it can be called from within Windows on TEST-s that have been written and designed under Unix. To do this, the batch file:

hwut2cygwin

has to be called instead of cygwin. A version to bridge from Unix to Windows files is under consideration at the time of this writing. It will likely be called:

hwut2wine

which insinuates that the ‘wine’ utility will be used to achieve the bridging.

Make Dependent Test Applications

In many cases, test code needs to be compiled before it can executed. This is particularly true for test written in compiled languages such as ‘C’, ‘ALGOL’, and ‘Pascal’. The process of generating executables from test code must be managed by a Makefile. HWUT needs to be able to say:

> make some-test.exe

If ‘some-test.exe’ is a test application to be run in the current directory. HWUT asks the Makefile about the test applications to be made by asking:

> make hwut-info

and parsing the standard output of the call to make. If ‘test0.exe’, ‘test1.exe’, and ‘test2.exe’ are tests to be made in the current directory, then there should be a make rule:

hwut-info:
    @echo test0.exe test1.exe test2.exe

The ‘@’ before echo prevents the string echo to be printed to standard output. The whitespace separated list of ‘words’ is interpreted as the list of test applications that can be built by this Makefile. Consider the following test file written in ‘C’ for testing some fibonacci number computing component:

#include "fibonacci.h"

int
main(int argc, char** argv)
{
    if( argc >= 1 && strcmp(argv[1], "--hwut-info") == 0 ) {
        printf("Fibonacci Test");
        return 0;
    }
    printf("%i --> %i", 0,   fibonacci_compute(0));
    printf("%i --> %i", 1,   fibonacci_compute(1));
    printf("%i --> %i", 997, fibonacci_compute(997));
    printf("%i --> %i", 998, fibonacci_compute(998));
}

The makefile that contains the test should at least tell how to buildt an executable and that HWUT needs to consider it, e.g.

FILES = fibo-test.exe

all: $(FILES)

fibo-test.exe: fibo-test.c  fibonacci.h
     gcc fibo-test.c -o fibo-test.exe

hwut-info:
     @echo $(FILES)

With this makefile, the user can generate his test applications by simply typing:

> make

on the command line. He can then check the output of his program until he is satisfied and tell HWUT to accept it in the same manner as before, i.e.:

> hwut a fibo-test.py

Tests can now be performed in the same way as with scripts that do not need any compilation.

Summary

This section discussed the basic usage of HWUT for script based and compilation based testing. Tests are performed by typing:

> hwut

in the test directory. To accept the output of a test application one needs to pass a as first command line argument and optionally the application name, e.g.:

> hwut a my-application.py

If errors occur, the difference of the output can be viewed by passing dd as first command line argument, e.g.:

> hwut dd my-application.py

Real life examples can be found in the directories demo/scripts/TEST and demo/compiled/TEST that come with the distribution of HWUT. The next two section elaborate on the details of deterministic and temporal logic tests.

Table Of Contents

Search

Enter search terms or a module, class or function name.

previous | next | index
Show Source
© Copyright 2014, Dr.-Ing. Frank-Rene Schaefer. Created using Sphinx 1.2.1.