Scripted Testing

Introduction

To set the scene, the Ada software under test (SUT) forms part of a system. Generally, the system will be constructed using a layered approach, and the other parts of the system that interact with the SUT are higher-level (which can call interface subprograms presented by the SUT) or lower-level (whose interfaces can be called by the SUT, or which can provide callbacks to the SUT).

This package is intended for testing at a level between unit testing (using, for example, AUnit) and integration testing (using real hardware). Unit testing tends to be fragile, requiring a lot of rework for even minor functional changes in the requirements.

The scripting language supported is Tcl, and specifically the Ada binding TclAdaShell. The reason for choosing Tcl rather than Python or Lua is that Tcl's interface is entirely string-based; this is important, considering the need to specifiy values of enumerated types.

The package provides facilities to write new commands in Tcl to

• call the interface subprograms presented by the SUT,
• set up values to be returned by calls the SUT makes to lower-level system components,
• call callbacks provided by lower-level system components,
• delay for appropriate periods, and
• check that the proper calls have been made to lower-level system components, with the required values.

It's assumed that the interface subprograms of the lower-level subsystems are stubbed so that:

• in and in out parameters can be recorded for later checking,
• out (and in out) parameter values and function return values can be provided to be returned to the SUT,
• exceptions can be raised when required,
• the number of calls to the subprogram can be checked.

The stubbing facilities of ColdFrame meet the above requirements.

Description

The components of the package are Commands, Actions, and an Action Queue.

Commands

A Command implements a Tcl command.

It creates an Action to be executed at run time, and posts it on the Action Queue.

Some commands are provided by this package. Other commands are to be provided to support the specific application to be tested: typically, call_procedure param1 param2 ... (where the parameters are those required by procedure) and check_subprogram parameter value (to check the value passed to procedure in parameter on the last call).

Commands have to be Registered with this package, because once Tcl has been started (using Start, which doesn't return) no more Commands can be added. Registration would normally be done during elaboration of the package in which the Command is defined (see test/test-first.adb in the distribution); Start completes the registration with the Tcl interpreter.

Actions

An Action carries the data required to enact the command at run time.

When it is executed, it performs the required action.

• If all is well, it completes normally.
• If some condition fails, it raises an exception (Execution_Failure) with a message stating the problem.
• Any other exceptions are propagated.

Action Queue

When the Action Queue is started (using the go command), it repeatedly picks the next Action and executes it, until either the end of the queue is reached (which implies that the script has succeeded) or an exception is propagated (which implies that the script has failed).

Provided commands

The commands provided by this package are

echo "string"

outputs string to the terminal at run time. Useful to report the script's progress.

wait duration

delays for duration.

mark name

notes the time at which the command was executed.

wait_from_mark name duration

delays until duration after the named mark. It is an error if the indicated time has already passed. The mark can be re-used.

go

start executing the script.

Building

A GNAT Project (GPR) file (scripted_testing.gpr) is provided. To build the library, say

gprbuild -p -P scripted_testing

To install at your compiler's standard place, say this (you may need to do so as root, via e.g.sudo). DO NOT do this if you're using the compiler supplied with Debian-based systems.

make install

To install in (for example) ~/local, say

make install prefix=~/local

(and remember to put ~/local/lib/gnat on your ADA_PROJECT_PATH).

Your own GPR should then include with "scripted_testing";