This document focuses on regression tests, which are an essential part of Shaman development. The intend is to allow the whole system to be tested with a single click in an IDE.
It does not cover performance tests, or other kind of tests.

Shaman relies on JUnit regression test framework. JUnit offers a structure for describing and running tests, which can be unit tests, or integration tests as well.

The build script mentioned several times is the main Ant script of the project : $SHAMAN_HOME/src/build/build.xml.

Unit tests, for in-process, non-visual implementations, do not cause structural problems most of time. Integration tests, with several servers running in different JVMs, generate more requirements :

• Integration tests require servers to be started up (and shut down) automagically.
• In order to avoid code rewriting, and for getting closer to production conditions, build script should be reused for server startup / shutdown.
• No dependancies between tests. A test should find the system in a "clean" state, whatever happened before.
• Exceptions should never "disappear". They must appear on the Java console, or in some log file.
• Tests may configure servers with specific configurations, overriding defaults. These configurations must be stored as Java resources, since they are the easiest way to manage files from an IDE.
• Servers should not be started when running on a remote host, or running in an IDE (making debugging possible).
• Each test implementation should not require additional code, all server lifecycle being handled by a base class.

This leads to define a test lifecycle as follows :

1. The first time a test is ran is the test suite, all servers are rebuilt and locally deployed, using the build script.
2. Each test is ran as follows :

   1. The server is started if it wasn't already.
   2. The test is ran.
   3. The server is shut down if it wasn't started in step 1.

All source files of the test framework implementation are located in $SHAMAN_HOME/src/junit. Most of them are located in fr.paris5.shaman.system package.

See javadoc for more details.

The fr.paris5.shaman.SingletonMap makes possible to keep real Singletons even with JUnit's class reloading. See Shaman Javadoc and JUnit documentation for more detail.

The ServerTestSupport class encapsulates the behavior described above. It also avoids to inherit from a base class. This is especially useful when using JUnit-related frameworks, which require to inherit from a particular class.

The ServerTestSupportUser class is intended to be subclassed by classes which need to use a ServerTestSupport. It allows the ServerTestSupport to access to methods which would be declared virtual if the ServerTestSupport could be the base class. Instead of it, the ServerTestSupportUser delegates their call to the test class. The test base class should also delegate its setUp and tearDown methods implementation to the ServerTestSupport instance it owns.

The doSetUp and doShutDown methods of the test class are called, respectively, after and before the ServerTestSupport ran its own setUp and shutDown methods.

The getRequiredServers method returns a set of ServerKeys instances. ServerKey is an enumeration class for {SPIRIT, LEGEND, INSIGHT}.

The getAntUserProperties method allows to specify properties (in addition to properties defined in the script itself) passed at the command startup.

See following classes source code for better understanding :

Calls all build / deploy targets. It is called at each startup, but an internal flag takes care of executing only once. The SingleDeployHelper is made a Singleton through the fr.paris5.shaman.SingletonMap.

Handles several ServerControllers, one per server.

Manages the state of one server. It is an abstract class, since knowing if the server is down or up is made in a different way, depending of the server type.

The diagram below describes the states of one ServerController instance :

Allows to create an Ant Project handling build file. It is used each time a script target is being called.
It also gives access to (expanded) build script properties.

Provides a default implementation for tests requiring at least one server to be up.

SingleDeployHelper and ServerSystem are meant to be Singletons. Usually, Singletons in Java are implemented with static variables. The problem is that JUnit may reload test classes at any time, making a Singleton being instantiated more than one. The fr.paris5.shaman.SingletonMap class takes care of keeping one Singleton instance per key, across several ClassLoaders.

Ant documentation does not mention how Ant scripts can be called from a Java program, without spawning an additional JVM instance. Launching Ant in a separate JVM has several drawbacks :

• it is really slower (it should be called at least two times for each test),
• it is harder to debug,
• messy things with JDK's tools.jar, and $JAVA_HOME environment variable could happen.

A careful reading of Ant source code shows that an Ant target may be ran using the org.apache.tools.ant.Project.executeTarget method. Specifying user properties for a given target requires to instantiate the org.apache.tools.ant.taskdefs.Ant command.
Refer to fr.paris5.shaman.system.AntProject and fr.paris5.shaman.system.ServerController source code for details.

This has been tested with the bundled version of Ant 1.4.1, but seems compatible with Ant 1.5.

The build script targets for deploying, starting and shutting down servers must be named deploy-<server_name>, start-<server_name>, stop-<server_name> respectively, where <server_name> is the name given by ServerKey.