Introducing Maven and testing

It is well known and accepted that the Maven build system from Apache is very powerful and capable. It is also easy to use and hard to master.

Countless plugins to extend its functionality can be found.

One of the main functionalities offered by Maven is assisting in various levels of testing, made possible by some powerful plugins.

In building and testing software, individual software components and classes should be unit tested using mockup objects and at a level of granularity that is independent of any services being available, both internal and external.

In this case Maven offers very good solutions, such as the ubiquitous Surefire plugin. Using this plugin unit testing with a variety of testing frameworks is quite straightforward and easy.

On the other hand, Integration Testing is done at a higher level, verifying that the integration between components, both internal and external is done correctly and to help identify problematic areas. Once a faulty integration is discovered, its unit test results can be examined in detail or if it is an external system, further diagnostics on it can be run.

Good judgement on the programmer is key to apply testing at the most convenient levels though on large enterprise systems usually a combination of unit testing and integration testing is most appropriate.

Though less well known than Surefire, Maven offers the Failsafe plugin, designed specifically for integration testing.

It is interesting to note that the name has been chosen to emphasize that failures don’t stop the Maven lifecycle allowing for gracefully cleaning up any resources used during the testing. As resources can be external it i risky -and rather inelegant- not to clean them up correctly.

According to the Failsafe documentation, it needs to be attached to the integration-test and verify phases (though it seems to work if attached only to the integration-phase). In any case, once invoked it runs through the relevant goals and relevant tests are run.

Testing FTP and SFTP functionality under Maven

We have discussed instances of problems where powerful, established plugins are available to do what we want (in this case, testing). However, we need to do something in Maven for where there are no plugins or they are not easy to find. Look it up twice as it is quite likely someone else has solved the problem before though it may be the case no solution exists.

One of such problems is testing systems that depend on external services such as FTP or SFTP. Say we have a client that downloads some data off such a server, such as XML files or large binaries and no other interface is available.

A good non-legacy example of such a service is YouTube’s API.

Even though YouTube offers programmatic APIs in Java for instance, to use more advanced functionality and for efficient bulk uploading an SFTP interface needs to be used. There are many other examples of such services, using both FTP and SFTP.

Unit tests on the system can be done with mockup objects as usual. However, to do integration testing things get more interesting.

We could, of course, use our YouTube production service to do some integration tests using dummy files, etc. Using production for testing is usually not a very good idea, with potential for disaster. Therefore, using the YouTube production service is not advisable.

Ideally, we should be able to create a mock but fully functional FTP environment, containing exactly the files we need where we can upload and download whatever is necessary and if needed, a small script (say, groovy or something) that simulates whatever it is that YouTube does with your files behind the scenes.

The basic workflow would look like this:

Therefore, we need a plugin that fires up a FTP server and points users to specific areas of the target folder. Ideally, it fires up during the pre-integration-test phase and shuts down on the post-integration-test. Easy as a cake.
A google search of ‘maven ftp server plugin‘ yields no significant results and looking up on the official list of plugins here. Also doing a google code search is of no use, either. Lots of stuff to perform FTPs and file transfers but no starting up servers.

Instead of giving up or firing up some server using Ant or even worse, manually(!), we go on and do it the Maven way.

Namely, we develop a plugin that fires up an (S)FTP server on the desired integration test phases.

Firstly, we need a FTP backbone written in Java so it can be integrated easily and of a suitable license. The Apache FtpServer is perfect for that purpose. It is a high-performance server based on Apache MINA for I/O and it is ridiculously easy to embed from any Java app.

Creating the Maven plugin basics

Firstly, we create a plugin skeleton project using the Maven Archetype plugin:

mvn org.apache.maven.plugins:maven-archetype-plugin:1.0-alpha-7:create -DgroupId=cat.calidos.maven.ftpserver -DartifactId=ftpserver-maven-plugin -DarchetypeArtifactId=maven-archetype-plugin -DarchetypeGroupId=org.apache.maven.archetypes

This creates an empty Maven plugin project ready to start adding stuff such as code and tests. We also need to run

mvn eclipse:eclipse

to create Eclipse settings for the project. We also need to run it whenever we add new dependencies to the POM file so they are also included in the Eclipse world.

Firstly, we add the dependencies for the Apache Ftp server (*):

	
		org.apache.ftpserver
		ftpserver-core
		1.0.6
	
	
		org.apache.ftpserver
		ftplet-api
		1.0.6
	
	
		org.slf4j
		slf4j-api
		1.6.1
	
	
		org.slf4j
		slf4j-log4j12
		1.6.1
	

We can find the details on dependencies and versions on a Maven repo such as MVNrepository.

In this case, we also add the Simple Logging Facade for Java ‘Log4j’ implementation< so the ftp server can output logs.

This means that whenever we use the maven plugin we should provide a log4j configuration file through the usual methods. Plugin will still work but will complain about the missing configuration. In any case, should we desire to use another logging framework we only need to change the dependency to our chosen implementation artifact.

Adding our first mojo

Next, we need to start adding some ‘mojos‘, or ‘Maven plain Old Java Object’, which are the fine-grained goals we need to run (or what is the same, tasks that we want Maven to execute). For more information, read the Introduction to the Build Lifecycle article on the official documentation which is very clarifying and chock full of valuable information.

Examining the source of other Maven plugins, we observe a common enough pattern which is to have an abstract superclass with some of the attributes needed by our mojos.

public abstract class AbstractFtpServerMojo extends AbstractMojo {

On this class we can add common attributes to our mojos such as the Maven project instance

/** Encompassing maven project
*	@parameter default-value="${project}"
* 	@required
* 	@readonly
*/
protected MavenProject mavenProject;

To the attributes we add the revelant annotations which let the Maven runtime inject following the popular Inversion of Control pattern. More documentation on the annotations can be found on the Mojo API Specification.

Basically, we need one mojo to start the server at the ‘pre-integration-test’ phase and another to stop it at the ‘post-integration-test’ phase, which gives us this a class hierarchy like this:

Ok, so now we need to take a look at the Apache Ftp Server documentation to discover how to embed it.

Fortunately it is pretty straightforward. A server factory instance is created from which a server instance can be created and attached to a specific port. Any such configuration such as adding users, setting up SSL keystores or port and interface attachment are configured using ‘listeners‘, which are configuration classes. Multiple listener instances can be set onto the same instance, to allow for multiple interfaces, etc. In the case of this example plugin we aim to configure a subset of all setup possibilities as it is only for integration testing and shouldn’t be used to deploy production FTP servers. For proper deployment, it can be run and configured fully from the command line or as a Windows service very easily.

Ok, so we add a method to the ‘run’ mojo to create the relevant instances

/** Create a server instance with default values
*///////////////////////////////////////////////////////////////////////////////
private void initServer() {

	// add relevant system property variables
	if (systemPropertyVariables!=null) {
		Set propertyNames = systemPropertyVariables.keySet();
		for (Iterator iterator = propertyNames.iterator(); iterator.hasNext();) {
			String propertyName = (String) iterator.next();
			String propertyValue = (String) systemPropertyVariables.get(propertyName);
			getLog().debug("Setting system variable '"+propertyName+"'");
			System.setProperty(propertyName, propertyValue);
		}
	}

	serverFactory = new FtpServerFactory();
	factory = new ListenerFactory();

	// set the port of the listener
	getLog().debug("Using port "+port);
	factory.setPort(port);
	serverFactory.addListener("default", factory.createListener());
	server = serverFactory.createServer();

}	// initServer

Firstly, we add any system variables that our POM wants to have added to our environment. This is a common pattern on many plugins and allows plugin users to fine tune system properties on that particular plugin environment. A very important possibility is to be able to configure a custom location of the log4j configuration using the ‘log4j.configuration’ system variable. For another example of setting system properties you can look at the excellent Jetty Maven plugin.

As properties are just key-value pairs of strings, we allow for the Maven injection to inject them defining the appropriate attribute parameter.

/** Additional system property variables (to pass onto tests, etc.)
*	@parameter
*/
protected Map systemPropertyVariables;

Secondly, we create the factory, listener and server instance. To make use of the more sophisticated features of the Apache Ftp Server we would only need to modify this method a little bit, for instance to allow to bind to a different network interface, etc.

Adding user management

Another common enough configuration that is definitely needed is user setup, including password setup, write permissions, home location, simultaneous logins, etc.

Examining the documentation we see a simple yet very flexible API to manage users. An interface to an user manager class is provided, ‘org.apache.ftpserver.ftplet.UserManager’ and two ready made implementations are available: a properties-based manager and a database-based one. This is very convenient and we could easily have the mojo provide database connection details or a path to a property file. However, we would rather provide configuration details on the POM file itself so all settings are selfcontained in the Maven world.

This means we need a simple factory, user manager and a trivial user class, where all user details are created in Java code.

We start by creating a skeleton factory:

public class SimpleUserManagerFactory implements UserManagerFactory {

And fill the necessary methods, in this case, only the ‘createUserManager’ method.
We follow by creating a simple User class:

public class User implements org.apache.ftpserver.ftplet.User  {

We shouldn’t really use the Ftp Server User implementation ‘BaseUser’ for two reasons: a) it’s in a private implementation package ‘impl’ on ftpserver-core so it can’t really be used (the interface is public and resides in the ftplet-api library) and b) we want to populate its details through the POM file so we need to add appropriate Maven annotations to its attributes.

It is no big deal as it is an easy enough interface to implement, with several key-value pairs, related to the different settings users can have such as maximum number of logins, username, password, etc. We tag the attributes with the Maven annotations and implement all the interface methods.

/** @parameter
*	@required */
protected String name;

/** @parameter
*	@required */
protected String password;

Creating the user manager is also pretty straightforward, implementing the UserManager interface of the ftplet API:

public class SimpleUserManager implements UserManager {

As a ‘repository’ to hold the User data we employ a simple in-memory Map instance:

private HashMap	users;

The resulting class structure is clear in its intent and purpose:

Testing user management code

Both the manager and the manager factory classes are pretty much self-contained it is fairly easy to create tests for most of that functionality. For example. in the case of the user manager, on the setUp() jUnit method we create a sample environment which we use on each test.

/* (non-Javadoc)
* @see junit.framework.TestCase#setUp()
*///////////////////////////////////////////////////////////////////////////////
protected void setUp() throws Exception {

	super.setUp();

	String adminName = "admin";
	userManager = new SimpleUserManager(adminName,new ClearTextPasswordEncryptor(),false);
	saveUser("demo","demo");
	saveUser("demo2","demo");
	saveUser(adminName,"administrator");

}	// setUp

Note that we use the clear text password encryptor both on the test and on the functional code as the passwords are only stored in memory therefore they do not really need to be encrypted and they are in plain view in the POM file anyway.

Using the sample environment we test functionality of the user manager:

/**
* Test method for {@link cat.calidos.maven.ftpserver.users.SimpleUserManager#authenticate(org.apache.ftpserver.ftplet.Authentication)}.
*///////////////////////////////////////////////////////////////////////////////
public void testAuthenticate() {

	loginShouldWork("demo", "demo");
	loginShouldFail("demo", "FAIL");
	loginShouldFail("DOESNTEXIST", "whatever");
	loginShouldWork("admin", "administrator");
	loginShouldFail("admin", "FAIL");

}	// testAuthenticate

Methods ‘saveUser’, ‘loginShouldWork’ and ‘loginShouldFail’ are private convenience methods to aid in testing and make tests more readable, tests are still code and should be readable and well-structured like regular functional code.

Run server mojo

Next, we add some code to have our configured FTP server instance run:

/** Startup the server and store it on the project properties if possible
*	@throws MojoFailureException
*//////////////////////////////////////////////////////////////////////////////
private void runServer() throws MojoFailureException {

	getLog().debug("About to start FTP server...");

	try {

		server.start();
		getLog().info("FTP server started.");

	} catch (FtpException e) {

		getLog().error("Could not start FTP server...");
		throw new MojoFailureException("Could not start FTP server instance", e);

	}

	Properties properties = null;
	if (mavenProject!=null) {
		properties = mavenProject.getProperties();
		properties.put(FtpServerConstants.FTPSERVER_KEY, server);
	} else {
		throw new MojoFailureException("Can't add ftpserver instance as maven project is null");
	}

}	// runServer

The ‘start’ method does not block and creates a new thread, where the server code will run and listen for upcoming connections. We also add the instance to the project properties so the server can be stopped gracefully on the ‘post-integration-test’ phase.

Also, we do not forget to add the annotation to the ‘FtpServerRunMojo’ class that binds the run mojo to the ‘pre-integration-test’ phase:

/** Mojo to start the apache FTP server as an integration instance
*	@goal run
*	@phase pre-integration-test
*///////////////////////////////////////////////////////////////////////////////

Stopping the server mojo

Once our run mojo starts the server, integration tests can be run on the ‘integration-test’ phase. Please bear in mind that client POMs can still be configured to run the server on whatever phases they need through XML tweaking.

The stop mojo ‘execute’ method is quite straightforward as well:

public void execute() throws MojoFailureException {

	getLog().debug("Stopping FTP server...");
	Properties properties = null;
	if (mavenProject!=null) {
		properties = mavenProject.getProperties();
	} else {
		throw new MojoFailureException("Can't access maven project to stop FTP server (null)");
	}

	if (properties!=null) {

		FtpServer ftpServer;
		try {
			ftpServer = (FtpServer) properties.get(FtpServerConstants.FTPSERVER_KEY);
		} catch (ClassCastException e) {
			throw new MojoFailureException("Context doesn't contain a valid ftp server instance",e);
		}
		if (ftpServer==null) {
			throw new MojoFailureException("Context doesn't contain any ftp server instance");
		}
		if (!ftpServer.isStopped()) {
			ftpServer.stop();
			getLog().info("FTP server stopped.");
		} else {
			getLog().info("FTP server was stopped already");
		}

	} else {
		throw new MojoFailureException("Maven project has null properties",new NullPointerException());
	}

}	// execute

After error and sanity checks we retrieve the ftp server instance and stop it. Easy.
Bear in mind that, exceptions during the integration-test phase should be instances of ‘MojoFailureException’ which do not cause the build to die (please check the appropriate documentation on the Maven reference book).

So, if for instance a single test fails, the other tests can run and the server is gracefully stopped at the end. Using ‘MojoExecutionException’ would cause the whole build to stop and if done before the stop mojo has a chance to run the server is not stopped. All probable exceptions on the Maven ftp server plugin are of the type ‘MojoFailureException’ as we do not want to stop the whole build if integration tests fail, we report them and it will be up to the client developer to decide what to do.

Testing the mojos the right way

Unit testing mojo code is made easier thanks to the ‘AbstractMojoTestCase’

public class FtpServerMojoTest extends AbstractMojoTestCase {

This superclass contains some methods to help in testing mojos, mainly methods to read POM files and run mojos.

We also need an FTP client to connect to the running instance. The excellent Apache Commons Net library is ideal for that purpose.

To use it we add the relevant test dependencies to our project POM with a ‘test’ scope which means they will only be used on the ‘test’ phase and not when the plugin is ran:

	
		org.apache.maven.plugin-testing
		maven-plugin-testing-harness
		1.2
		test
	
	
	    commons-net
	    commons-net
   		 3.0.1
    	test
    

So what we want to do is run the FTP server startup method on each test setup and stop it on each test teardown.

/** Read pom and start FTP server
*	@throws java.lang.Exception
*///////////////////////////////////////////////////////////////////////////////
public void setUp() throws Exception {

	super.setUp() ;

	pom = getTestFile("src/test/resources/unit/test-project/pom.xml");
	assertNotNull("Test pom not found",pom);
	assertTrue("Test pom not found",pom.exists());

	port = getFreePort();

	FtpServerRunMojo runMojo = (FtpServerRunMojo) lookupMojo("run", pom);

	// all this will be set on a real project (it's not on the test environment)
	runMojo.serverRoot = new File(PlexusTestCase.getBasedir()+"/target");
	runMojo.port = port;
	runMojo.mavenProject = new MavenProject();
	runMojo.mavenProject.getModel().setProperties(new Properties());
	runMojo.execute();
	ftpServer = (FtpServer) runMojo.mavenProject.getProperties().get(FtpServerConstants.FTPSERVER_KEY);

    ftp = new FTPClient();
    ftp.connect("localhost",port);    

}	// setUp

First we call the superclass as it configures stuff we need to call the convenience methods, load up a test pom and find a free port to run the server. Next we lookup the ‘run’ mojo and add the relevant data that it needs to start such as the server root, port, container Maven project, etc. All that data is automatically set on a non-test environment but it is not on a test setup. Last on the setup is to connect to the FTP server.

Teardown needs to cleanup the ftp connection and run the stop mojo:

/** Run stop server mojo
* 	@throws java.lang.Exception
*///////////////////////////////////////////////////////////////////////////////
public void tearDown() throws Exception {

	if (ftp.isAvailable()) {
		if (ftp.isConnected()) {
			ftp.disconnect();
		}
	}

	FtpServerStopMojo stopMojo = (FtpServerStopMojo) lookupMojo("stop",pom);

	Properties properties = new Properties();
	properties.put(FtpServerConstants.FTPSERVER_KEY, ftpServer);
	stopMojo.mavenProject = new MavenProject();
	stopMojo.mavenProject.getModel().setProperties(properties);
	stopMojo.execute();

	super.tearDown();	// called last as it dismantles running mojo stuff

}	// tearDown

Note we create a Maven project instance to store the running FTP instance which will be stopped by the ‘stop’ mojo. We also call ‘super.tearDown()’ last to make sure any superclass resources are cleaned up when we are done cleaning up ourselves.

The test POM should contain appropriate test data so significant tests can be run, therefore we add different configuration settings to the XML

	cat.calidos.maven.ftpserver
	ftpserver-maven-plugin
	1.0
		
			
				admin
admin00
			
			
				
					demo
demo
				
				
					disabled
disabled
					false
				
				
					classes-root
classes-root
					./classes
				
				
					write-disabled
write-disabled
					false
				

			
		
	

This setup lets us test the administrator user, a regular user, a disabled one, home folders and write-disabled users. There are far more features exposed by the Apache FTP Server though it is not the aim of this test to test them all, just to make a general set of features work as expected. Obviously, to get higher code coverage more tests can be added as needed.

After the POM is ready, we add some unit tests such as the regular user test:

/** Demo user should be able to login
*	@throws SocketException
*	@throws IOException
*//////////////////////////////////////////////////////////////////////////////
public void testDemoUser() throws SocketException, IOException {

    ftp.login("demo", "demo");
	int reply = ftp.getReplyCode();
    assertTrue(FTPReply.isPositiveCompletion(reply));
    assertTrue("Can't login with demo user",ftp.isConnected());

	String filename = "test-file";
	putFile(filename);
	boolean found = findRemoteItem(filename);
	assertTrue(filename+" can't be uploaded",found);

    ftp.disconnect();

}	// testDemoUser

We connect, add a test file and ensure it has been uploaded successfully. Both ‘putFile’ and ‘findRemoteItem’ are convenience methods created on the test class.

Adding more test cases is pretty straightforward, like in the test for readonly users:

/**	User shouldn't have write permissions
*	@throws IOException
*//////////////////////////////////////////////////////////////////////////////
public void testCannotWrite() throws IOException {

	ftp.login("write-disabled", "write-disabled");
	int reply = ftp.getReplyCode();
	assertTrue(FTPReply.isPositiveCompletion(reply));
	assertTrue(ftp.isConnected());

	String filename = "test-file2";
	putFile(filename);
	boolean found = findRemoteItem(filename);
	assertFalse(filename+" can be put on nonwrite permissions user",found);

	ftp.disconnect();

}	// testCannotWrite

Using the plugin

Doing integration testing with our completed FTP Server Maven plugin is pretty straightforward using the ‘maven-failsafe-plugin’ to fire up the tests and the ‘ftpserver-maven-plugin’ itself to run the FTP server

	cat.calidos.maven.ftpserver
	ftpserver-maven-plugin
	1.0

		
			
				admin
admin00
			
		
			file:target/test-classes/log4j.properties
		
	

	
	
		
			start-ftpserver
			
				run
			
		
		
			stop-ftpserver
			
				stop
			
		
	

Not that, unlike in unit testing and the ‘maven-surefire-plugin’ the standard test location for the log4j configuration file does not work so we need to specify it on the POM. We also need to specify the goals for the FTP server to startup and shutdown, much along the lines of the Maven Failsafe plugin itself.

If any of the integration tests fails the stop goal is still ran and the server shuts down gracefully.

Additionally, if running the FTP server is needed on phases other than the integration testing ones the goals be attached to the relevant phases through our client POM.

Wrapping up: creation of an FTP server plugin

We have identified a need of using a fully functional FTP and SFTP server in integration testing using Maven. Looking up available plugins on the Web does not yield any already available plugins. We have gone ahead and built a plugin from scratch using the Apache FTP Server opensource library.

You can download the source code of version 1.0 of the maven-ftpserver-plugin as well as a sample client project. Enjoy and comments are welcome.

(*) Between completing the code and writing the article Apache FtpServer came out with a new release (1.0.6). Thanks to Maven and the fair amount testing it was ridiculously easy to upgrade the project, just changed the dependency numbers on the pom file and ran a maven build.

A common feature of Web applications is the need to output structured content, be it XML, XHTML markup, JSON or many others. A number of technologies is used to do that but few seem to be that dynamic, usually only reloading templates when they change or loading them from URLs. Surely we can leverage OSGi to make things more interesting…

Therefore we should be following the OSGi dynamic philosophy as much as possible, exploiting the features made available by the framework such as dynamic discovery, services, bundle headers and the lot.

In the case of our cache app we are violating quite a few basic design principles by having the format embedded in the java code. So we should as well use a template separate from the code and if possible reuse some existing well-known templating engine from somewhere.

Given these basic design principles let’s get started.

Firstly, we need a robust and flexible templating engine. We select the mature Freemarker engine which is reasonably speedy and has the power and flexibility we want. Make sure you check out the license, of course.

We could stop at putting the library JAR in a bundle and package it so it can be used by any other bundle and that is what we do to be able to use it in OSGi. That however doesn’t exploit many of the nicer OSGi capabilities so we will create another bundle called ‘com.calidos.dani.osgi.freemarker-loader’.

What we want is to allow bundles to carry templates inside them and have our freemarker-loader templating bundle discover them automagically. This is the same technique that the Spring dynamic modules use and you can find more info here. The mechanism is described in this diagram:

That is easy enough with a BundleTracker and a BundleTrackerCustomizer implementation. The BundleTracker class tracks bundles being added to the environment like this:

tracker = new BundleTracker(context, Bundle.RESOLVED, templateTracker);
tracker.open();

With this snippet the tracker instance will look for bundles in the RESOLVED state (which lets us track fragments). The ‘templateTracker’ object is an instance of BundleTrackerCustomizer and will receive callbacks whenever bundles are added to the environment.

For instance, when a bundle is added we check for a special header in the bundle which tells us what is the relative path of available templates in the bundle being resolved:

public Object addingBundle(Bundle bundle, BundleEvent event) {

// we look for the header and act accordingly
String templatesLocation = (String) bundle.getHeaders().get(TEMPLATE_HEADER);
if (templatesLocation!=null) {

	Enumeration bundleTemplates = bundle.findEntries(templatesLocation, "*.ftl", true);
	HashSet templatesFromAddedBundle = new HashSet();
	while (bundleTemplates.hasMoreElements()) {

		URL templateURL = bundleTemplates.nextElement();
		addTemplate(bundle, templateURL,templatesLocation);
		templatesFromAddedBundle.add(templateURL);

	}

	templatesOfEachBundle.put(bundle.getBundleId(), templatesFromAddedBundle);

}
return null;

}	// addingBundle

An interesting method being used here is ‘findEntries’ which loads all the entries in the provided templates folder and lets us add them to our holding structure. We also take care to implement the methods to remove the templates and update them accordingly whenever bundles are updated or unloaded from the environment.

Having TEMPLATE_HEADER with a value of ‘Freemarker-Templates’ means that bundles having a header such as Freemarker-Templates: /templates will have any templates within that folder (please note that the ‘/templates’ bit is not added to template URLs!).

The next thing we need to do is make the loaded templates available to the environment. To do that we make a freemarker Configuration object accessible as an OSGi service object. That Configuration instance is the main object Freemarker to load and use templates and has an interesting mechanism to override its template loading mechanism we use to make available our OSGi environment templates.

freemarkerConfig.setTemplateLoader( new URLTemplateLoader() {

@Override
protected URL getURL(String url) {
Stack templateStack = templates.get(url);
if (templateStack!=null) {
	TemplateEntry templateStackTop = templateStack.peek();
	if (templateStackTop!=null) {
		return templateStackTop.getTemplateURL();
	}
	return null;
}
return null;
}

});

The service Configuration object is set with a template loader inner class that uses our template holding objects to retrieve templates stored in our OSGi context. Cool.

This also allows us to effectively disable the template refreshing cycles that Freemarker does by default (allegedly making it slightly more efficient). Now we only need to refresh a bundle containing the templates to get the new version. This can be modified by using the appropriate methods on the Configuration service of course. (There is another method explained later).

An interesting feature we can add to exploit the dynamic nature of OSGi is to make templates available in a stack. This means different bundles can dynamically overwrite templates by the same name. Moreover, once a template is removed the previous version becomes available. This can be used to make temporary changes to templates to add or remove diagnostics information, test new templates temporarily, etc.

We do that using a Stack of TemplateEntry objects, TemplateEntry being a helper class to store template entries.

This is all very nice but we have a problem when having multiple versions of the same bundle that hold multiple copies of the same template, this means they will stack and we have no way to access a particular version of a template. To solve this problem we store each template in three different stacks by three different URLs:

• ‘path/file.ftl’
• ‘bundle://bundlename/path/file.ftl’
• ‘bundle://bundlename:version/path/file.ftl’

In this manner we can use the more generic URL in most cases but still can access specific versions when needed. It is important to think about the dynamic nature of OSGi as well as the possibility of several versions of the same bundle coexisting peacefully in the same environment.

From the perspective of any bundle using the service in the simplest case it only needs to look for a service named ‘freemarker.template.Configuration’. For convenience, the property ‘dynamicConfiguration’ is set to ‘true’ to coexist peacefully with other Configuration services (maybe coming from an official Freemarker bundle). For instance, if we know for sure our dynamic Configuration service is the only one present we can do:

context.getServiceReference(Configuration.class.getName());

That will give us the highest ranking Configuration service. If there are several such services we can use a call like this to get the one that has the dynamic OSGi loader:

context.getServiceReferences(Configuration.class.getName(), "dynamicConfiguration=true");

There is one last feature which lets bundle users feed an already configured template configuration object to the bundle by publishing a Configuration service with property ‘preparedConfiguration’ set to ‘true’. This will get picked up by the bundle and its template loader put in sequence with the dynamic OSGi template loader. This means that any original Configuration settings are maintained (For further information on service filtering based on properties please check the BundleContext javadoc.).

Best thing to do is to go and try it by downloading the bundles here. Source is also available.

In the case of integration testing, it is manly used in a pre-production environment, with a valid build that has all unit tests passed. It can even be used in production to just after a deployment is made, taking care not to have destructive checks or massive load tests in the integration test code. YMMV.

To achieve integration testing we need to check the various OSGi components deployed interact in the way that is expected of them. Therefore we need to test the components in a group and not in isolation. To do that in the OSGi world means we need to have access to the OSGi context from within the tests to access services, call them and check their responses, etc.

To allow for this kind of integration testing within the OSGi environment, we make a slight modification to the excellent test.extender we have already patched in the previous installment.

Basically, the basic test.extender seeks out any JUnit test classes within the fragment bundle, creates an instance using an empty constructor and then fires up the tests. This is activated either by default when the fragment is loaded or by using ‘test ‘ in the console. For further information please see the previous post about this subject.

For our integration testing, we add an extra command to test.extender:

public Object _integrationTest(CommandInterpreter intp) {
        String nextArgument = intp.nextArgument();
    	testExtender.integrationTest(Long.parseLong(nextArgument));
    	return null;
}

And we refactor the TestExtender to add the integrationTest method which reuses some of the code to instantiate test cases using a constructor that accepts the OSGi context as a parameter.

Constructor[] constructors = clazz.getConstructors();
boolean foundConstructor = false;
for (int i = 0; i < constructors.length && !foundConstructor; i++) {
	Constructor constructor = constructors[i];
	Class[] types = constructor.getParameterTypes();
	if (types.length==1 && types[0].isInstance(context)) {
		foundConstructor = true;
		EClassUtils.testClass(inspectClass, constructor.newInstance(context));
	}
} // for

The OSGi context is passed onto the constructor and then the test class is run. It is obviously up to the test class to use the context appropriately for its integration testing.

In our cache project setup, we can do some useful integration testing on the cache.controller component, basically checking if the interaction with the provider components is behaving as we expect it. The integration testing is also added to a fragment that can be deployed optionally, of course.

We start by creating the fragment and adding a testing class like this:

Next, we add the constructor that accepts an OSGi context, which is very simple:

public CacheIntegrationTest(BundleContext ctx) {
	super();
	this.context = ctx;
}

In the setup and teardown methods we get and unget the cache service to perform the testing:

public void setUp() throws Exception {
	serviceReference = context.getServiceReference(Cache.class.getName());
	controller = (CacheControllerCore) context.getService(serviceReference);

}

public void tearDown() throws Exception {
	context.ungetService(serviceReference);
	controller = null;
}

In this case we get the controller cache service and store it in an instance used to perform the tests. This is quite simple and fulfills our intended purpose but we still have the flexibility to make more complex integration testing if needed.

Next we create as many test cases as needed:

public void testGet() {
	try {
		controller.init();
		double v = Math.random();
		String k = "/k"+v;
		controller.set(k, v);
		assertEquals(v, controller.get(k));
	} catch (CacheProviderException e) {
		e.printStackTrace();
		fail(e.getMessage());
	}

}

It should be noted that while the code looks like regular testing code, it is actually using real services from the OSGi environment as opposed to mockups. This means we are testing the real integration between components as well as the individual controller component code. The disadvantage here is that if there is an error in the controller we might mistake the problem with an issue with the services used. In conclusion, having integration code doesn’t negate the need to have unit tests.

Once we load the fragment onto the environment, first we need to obtain the bundle id of the integration fragment and then launch the integration testing in this manner:


osgi> integrate 125
Bundle : [125] : com.calidos.dani.osgi.cache.controller.integration
_
CLASS : [com.calidos.dani.osgi.cache.controller.CacheIntegrationTest]
___________________________________________________________________________
Method : [ testInit ] PASS
Method : [ testInitInt ] PASS
Method : [ testSize ] PASS
14:21:43,077 WARN CacheControllerCore Couldn't clear some of the provider caches as operation is unsupported
14:21:43,077 WARN CacheControllerCore Couldn't clear some of the provider caches as operation is unsupported
Method : [ testClear ] PASS
Method : [ testSet ] PASS
Method : [ testGet ] PASS
Method : [ testGetStatus ] PASS
___________________________________________________________________________


The results tell us that all operations are OK but we need to bear in mind that the clear operation is not supported in some backend caches. If this is what is expected by the operator then all is fine.

We take advantage of the new integration testing functionality to make some extensive changes to logging and exception handling of the controller code. By running the integration tests we make sure all seems to work fine (even though we still need some proper unit testing of the controller). Modifications are made quite quickly thanks to the integration tests.

To recap, we’ve added integration testing support to the existing ‘test.extender’ bundle and created integration testing code for the cache controller component. This has allowed us to make code changes quickly with less risk of mistakes.

Here you can find a patch for the test extender project as well as the patched testing bundle already compiled. Enjoy!

Please make sure you read the previous installments before you continue with this article…

To ensure greater flexibility, performance and encapsulation, we will be using OSGi fragments.

http://static.springsource.org/osgi/docs/1.1.3/reference/html/appendix-tips.html

OSGi fragments attach to “host” bundles and can be used to extend functionality, provide configuration and even extra manifest entries.

On the first part, we move our unit tests to fragments. This has the benefit of being able to separate tests from actual code as well as allowing us to deploy using a smaller footprint in situations where we don’t want test code such as production deployments.

If we start with the cache code we’ve got so far we need to separate the test code from two bundles: ‘com.calidos.dani.osgi.cache.provider.memory’ and ‘com.calidos.dani.osgi.cache.provider.memcached’. That is easy enough.

We create a new OSGi bundle fragment project like this:

We specify the bundle the new fragment is attaching to:

The next step is to move the JUnit code from the main bundle onto the fragment. If we run the test independently we see that it passes as the classpath of the host and fragment bundles is merged into one.

We then add the fragment to our existing run configuration and it will be loaded onto our environment. So far so good, but there is no way to run the tests outside our controlled Eclipse.

Slim Ouertani at Javalobby shows us how to run fragment tests from within the OSGi environment in a great article. His code uses the new bundle tracker feature to discover any test fragments and exposing functionality to be able to run the tests.

You can get the source at kenai: http://kenai.com/projects/testosgifragment

We read the article thoroughly, fetch the source and build test.extender using maven:


mvn install


The code works great but has a bug which is triggered when the code looks for the host of the fragment by reading the ‘Fragment-Host:’ header. If the value specifies any kind of version qualifier it fails. It also crashes when trying to test a bundle or a fragment that doesn’t have the ‘Unit-Test’ custom header.

The ‘Unit-Test’ custom header lives on the fragment manifest and is used by test.extender to control who is able to be tested and if tests are done manually or automatically.

If no ‘Unit-Test’ header is present or if its value is empty, test.extender will refuse to run any tests present on the bundle. If the value is ‘true’ then test.extender will run the tests automatically whenever the test fragment bundle is loaded. If the value is anything else the tests won’t run automatically but you can still run them using the ‘test’ and ‘testall’ commands.

I have patched the code to accept a version qualifier for the fragment host as well as giving out an informative error message when attempting to test the wrong bundle.

Once this is done, we play with the test.extender by obtaining the fragment bundle id and issuing ‘test <id>’ on the console:


osgi> test 80
Bundle : [80] : com.calidos.dani.osgi.cache.provider.memory.test
_
CLASS : [com.calidos.dani.osgi.cache.provider.memory.test.MemoryCacheTest]
___________________________________________________________________________
Method : [ testClear ] PASS
Method : [ testInit ] PASS
Method : [ testInitInt ] PASS
Method : [ testSize ] PASS
Method : [ testSet ] PASS
Method : [ testGet ] PASS
Method : [ testGetStatus ] PASS
___________________________________________________________________________
_


We can also use the ‘testall’ command which will look for all the testable fragment bundles and run the test there.

Having tested that functionality, we separate the tests from the memcached bundle and move them onto their own bundle.

Here you can download the two test fragments and the patch to modify test.extender, neat.

Ok then, we move onto the next thing that is looking closely at the Servlet Bridge and its behaviour. What we are interested in is what happens when the container deploys and inits the Web app itself, starting up the servlet bridge. The bridge uses the container temporary folder to deploy the OSGi environment in a subfolder, copying any bundles into that subfolder. If the subfolder exists and there are bundles that have the same names, they won’t be copied. I have found this behaviour to be very confusing for newbies to the platform and very annoying for veterans. I’m not entirely sure if this is deliberate and has performance reasons or what. Calling the framework controls to prevent that is not optimal and manually removing the temporary folder isn’t, either. One possible solution would be to use a building system that increments the micro number every time there is a build and remember to clear up the container temporary folder from time to time. That is not always desirable or possible but fortunately, the servlet bridge implementation provides a way to customize it to our heart’s content.

Basically we have the following code in the bridge init() method:

	framework.init(getServletConfig());
	framework.deploy();
	framework.start();
	frameworkStarted = true;

In this case, ‘framework’ is a instance of the FrameworkLauncher class which is created just before this code snippet. The FrameworkLauncher is the class responsible for deploying and launching OSGi. We note that the object is created by default as a plain FrameworkLauncher instance. However, the class to be used can be customized by specifying it in the web.xml file using the ‘frameworkLauncherClass’ servlet initialization parameter. As long as that class is a subclass of FrameworkLauncher the appropriate contract is fulfilled.

This means we can override some behaviour and delete the framework deployment folders upon startup:

@Override
public void init() {

super.init();

File servletTemp = (File) context.getAttribute("javax.servlet.context.tempdir");
File platformDirectory = new File(servletTemp, "eclipse");

if (platformDirectory.exists()) {
	deleteDirectory(platformDirectory);
}

}

Cool, so we need to export this as a jar file named dani-frameworklauncher.jar into the package project lib folder:

Next we change the servletbridge web.xml configuration parameter to load the new framework launcher:


<init-param>
<param-name>frameworkLauncherClass</param-name>
<param-value>com.calidos.dani.osgi.servletbridge.FrameworkLauncher2</param-value>
</init-param>


To test, we can add a file onto the $APACHE_TOMCAT/work/Catalina/localhost//eclipse folder and restart Tomcat. Before this new launcher was in place, the file would be kept but now it is deleted alongside the rest. Any other methods can also be extended to customize the bridge behaviour without having to modify the launcher class.

Here you can download the framework launcher project, put it on the WEB-INF/lib folder alongside the servlet bridge and enjoy!

Please read the previous installments to get up to speed and see the source used in this post.

Firstly, we need to setup some kind of project to hold all that is going in the WAR archive. This can be done using the WTP (Web Tools Project) or just as a regular project.

In this case we do it using a plain-vanilla resource project but feel free to use WTP as the basics are the same.

So firstly we create a resources project to hold the stuff, named ‘com.calidos.dani.osgi.cache.package’ for instance.
Secondly, we create the folder structure we need to hold all the files that need to reside in the final WAR webapp, so we create a WEB-INF folder to hold all the classes, libraries metadata and configuration.

We also know we need a web.xml file to define all the servlets this webapp declares. This is where the Equinox Servlet Bridge kicks in. It provides a Servlet that loads up the environment as well as forwards any HTTP requests onto our OSGi-managed Servlet instances.

We check the Equinox in a servlet container article and learn what the bridge Servlet is called, what parameters it can take and any other web.xml details we need.

For instance, we declare that the webapp service class is our BridgeServlet class:

<servlet-class>org.eclipse.equinox.servletbridge.BridgeServlet</servlet-class>
<init-param>
<param-name>commandline</param-name>
<param-value>-console</param-value>
</init-param>
<init-param>
<param-name>enableFrameworkControls</param-name>
<param-value>true</param-value>
</init-param>


We also add two parameters (more information and additional options can be found on the FrameworkLauncher servlet bridge class and the Equinox documentation).

Next, we need this special servlet itself. I prefer to check out the servlet from CVS and compile it myself but you can also find it here (from the the Latest Release pick up the org.eclipse.equinox.servletbridge jar). In the case of using the source, the CVS connection data is the following:


Method: pserver
User: anonymous
Host: dev.eclipse.org
Repository path: /cvsroot/rt
CVS Module: org.eclipse.equinox/server-side/bundles/org.eclipse.equinox.servletbridge


Once we import the project into Eclipse it looks like this:

As we can see, there are just three classes which compose the servlet, a special classloader and finally a class that launches OSGi.

If we import the project into our workspace, we have compiled classes in the bin/ folder of the project but we really want them neatly packaged as a jar file. Therefore, we right-click on the project to export as a JAR java package:

We take this JAR archive and save it in our package project WEB-INF/lib folder. We also check the composition of the file to make sure it includes all the classes we need.

This takes care of the plain webapp side of things so we move onto actually loading OSGi and what configuration it needs.

As mandated by Equinox we add a launch.ini file to clear and possibly override System properties (please see the source for details on that).

We then create an eclipse folder, which the servlet bridge expects to find the OSGi platform jar and any bundles (including ours).

We select all our project bundles, right-click and select the ‘Export…:Deployable plug-ins and fragments’ option.

This leaves us with a list of bundles that compose our custom code:

com.calidos.dani.osgi.cache.controller_1.0.0.beta.jar
com.calidos.dani.osgi.cache.frontend.http_1.0.0.beta.jar
com.calidos.dani.osgi.cache.log4jconfig_1.0.0.beta.jar
com.calidos.dani.osgi.cache.provider.memcached_1.0.0.beta.jar
com.calidos.dani.osgi.cache.provider.memory_0.0.1.dev.jar


Good, as we know, there are some standard and some special bundle dependencies we need as well. Let’s go through them by groups. We get a bunch of basic OSGi bundles we should include in most projects:

org.eclipse.equinox.registry_3.4.100.v20090520-1800.jar
org.eclipse.osgi.services_3.2.0.v20090520-1800.jar
org.eclipse.osgi.util_3.2.0.v20090520-1800.jar
org.eclipse.osgi_3.5.1.R35x_v20090827.jar


There are several options to obtain these bundles:

For expediency, we can go to our Eclipse installation folder and look for them in the ‘plugins’ subfolder. We then copy them into our package ‘plugins’ folder and there you go.

The second option is to get them from the Equinox distribution.

Another option is to the prepackaged servlet bridge feature which can be picked from CVS as well:

Method: pserver
User: anonymous
Host: dev.eclipse.org
Repository path: /cvsroot/rt
CVS Module: org.eclipse.equinox/server-side/features/org.eclipse.equinox.servletbridge.feature

Once that is loaded onto our workspace, we open the feature.xml and select the ‘Export Wizard’ form the ‘Exporting’ section. This lets us export these minimum bundles and generate both the deployment ‘feature.xml’ file as well as the bundle listing (even though the wizard actually takes them from our Eclipse platform).

We also have some more dependencies related to doing HTTP and the logging platform:

org.apache.log4j_1.2.13.v200903072027.jar
org.eclipse.equinox.common_3.5.1.R35x_v20090807-1100.jar
org.eclipse.equinox.http.registry_1.0.200.v20090520-1800.jar
org.eclipse.equinox.http.servlet_1.0.200.v20090520-1800.jar
org.eclipse.equinox.registry_3.4.100.v20090520-1800.jar


These can also be obtained from the Eclipse plugins folder or Equinox distribution folder. One should note that modifying the feature.xml taken from the Eclipse CVS repository, adding any required plugins there and then running the Export Wizard.

Finally, we have two dependencies we need to pay special attention to:

org.eclipse.equinox.http.servletbridge_1.0.200.200911180023.jar
javax.servlet_2.4.0.200911240836.jar


In the first case, it’s a special minimal bundle that hooks the plain webapp servlet bridge with the OSGi ‘org.eclipse.equinox.http.servlet’ standard component which publishes a servlet service onto the OSGi platform.

This are the details to get it from CVS:

Method: pserver
User: anonymous
Host: dev.eclipse.org
Repository path: /cvsroot/rt
CVS Module: org.eclipse.equinox/server-side/bundles/org.eclipse.equinox.http.servletbridge

We export it as a deployable plug-in and add it to the mix. We can also download it from the Equinox distribution site as well.

In the second case, this bundle contains the basic servlet classes and interfaces and I have found problems of class signatures when using 2.5 along with the servlet bridge so unless all is compiled against 2.5 the safest bet is to go with 2.4.

(Check http://www.eclipse.org/equinox/documents/quickstart.php for more info).

Next, Equinox requires an XML file to define some metadata about the loaded bundles and make our configuration easier. The file sits in a folder called ‘features’ plus a subfolder with a reverse DNS name and is called feature.xml.

The structure is quite simple (though it can be created using the Eclipse Feature Export Wizard) and mainly holds a list of plugins, their version data and some more fields:

We complete the list with all our bundles, fully knowing that we can extend our environment on the fly once it’s loaded if we need it.

Next, we create the config.ini file which really tells equinox which of the bundles stated in the feature file to start, at which start level and lets us add even more bundles (though we need to specify where the actual file is located and the full filename in that case). It also lets us configure the environment pretty thoroughly.

The resulting config.ini file looks like this:


#Eclipse Runtime Configuration File
osgi.bundles=org.eclipse.equinox.common@1:start, org.apache.log4j@start, org.eclipse.osgi.util@start, org.eclipse.osgi.services@start, org.eclipse.equinox.http.servlet@start, \
org.eclipse.equinox.servletbridge.extensionbundle, \
org.eclipse.equinox.http.servletbridge@start, \
javax.servlet@start, org.eclipse.equinox.registry@start, org.eclipse.equinox.http.registry@start, org.eclipse.equinox.servletbridge.extensionbundle \
org.eclipse.equinox.http.servlet@start, org.eclipse.equinox.common@start, com.calidos.dani.osgi.cache.log4jconfig, \
com.calidos.dani.osgi.cache.provider.memcached@3:start, com.calidos.dani.osgi.cache.provider.memory, com.calidos.dani.osgi.cache.controller@4:start, \
com.calidos.dani.osgi.cache.frontend.http@5:start

osgi.bundles.defaultStartLevel=4


We decide not to start the in memory bundle as it only works reliably in a single instance deployment scenario. Otherwise, in a multiple server setup the in-memory cache data would become inconsistent.

Also, be sure to check the Equinox quickstart guide and documentation for more information.

Once all is is done, the project looks like this:

Time to right-click on the project and select ‘Export:Archive file’ to save it in zip format and rename it to .war. Ready to deploy! Remember to access it using the URL //cache/. In web.xml the ‘-console 6666′ parameter means that doing a telnet on port 6666 of the machine initiates a session within the OSGi console. WARNING: there is absolutely NO SECURITY so disable, firewall or ACL that port NOW.

As usual, here you can find the source and the completed WAR though they are one and the same.

In this post, we will add another cache provider implementation to the mix as well as provide an HTTP front-end so the whole application can be tested.

First of all, let’s present a conceptual diagram of all the bundles and fragments involved so far.

Read on for more…

We start with providing the extra implementation. We will be making a memcached cache provider using the excellent memcached-spy java library (MIT licensed).

As usual, we create a new plug-in project and name it accordingly (com.calidos.dani.osgi.cache.provider.memcached). We firstly create source folders named ‘src’ and ‘test’ to hold any unit tests. Secondly, we download the spy memcached library from here.

We create a ‘lib’ folder on the project to hold the library and modify the classpath of the plug-in so its code is able to see the memcached client classes (but not the bundle clients!). In the runtime section of the MANIFEST.MF editor we select the library like this:

We also add the controller as a dependency so we can properly implement the CacheProvider service interface.

Secondly, we “copy” the unit testing code from the Memory cache bundle as all tests will be quite similar (as both bundles basically fulfil the same contract).

When doing all this work, we discover that the memcached system doesn’t have some functionality other cache systems might have. For instance, the number of elements in the cache can’t be counted (not easily and cheaply). Additionally, the cache can’t really be cleared without restarting all memcached instances.

So it makes sense to add the UnsupportedOperationException exception to the interface, acknowledging the fact that some cache implementations might not provide all the facilities of the service. Other strategies would be to use subclassing, specialise the providers, etc In this case, we opt for adding the exceptions where it makes sense. That is, in the case of the get and set operations we can’t allow a service that doesn’t let you manage its elements. That’s not our definition of “cache”.

We adjust the code and tests in the Memory cache if necessary to adapt to the new interface and implement the mecached client (the unit tests will need some modification).

The memcached bundle in the project view ends looking like this:

We make sure tests pass on this new bundle, to do that we need to tweak the test code a little bit (to capture new exceptions being thrown).

Next, we register the Cache service proper from the core controller tracker like this:

controller = new CacheControllerCore();
controller.addCacheProvider(cacheProvider);

//we have a provider, we can start publishing our service
registration = context.registerService(Cache.class.getName(),controller,null);

Note that we register the Cache service once we have added the first provider backend. This ensures that at least a backend is available as soon as the main service is registered.

At this point, things are looking good on backend implementation bits so we can move to providing a nice HTTP front-end for the cache so it can be used.

Therefore, we create a new bundle and call it something appropriate, such as ‘com.calidos.dani.osgi.cache.frontend.http’. The main function of this bundle is to glue together a HTTP Service being made available in the context to the business logic Cache service (provided by the core controller bundle). In this manner, the controller bundle does not need to know anything about HTTP or Servlet implementation. Moreover, this bundle only uses the Cache and doesn’t know about providers or implementation details.

We need to import a few packages and require the equinox http bundle to be present:

As an HTTP provider we can use the one provided by Jetty (that is bundled with Equinox).

Following the same model that on the controller bundle, we register two trackers one for the http service and another for the Cache service. Once both are made available we can create the servlet and glue them together.

There is a method to register the servlet on a particular URL:

cacheHttpFrontend.setCacheService(cacheService);
httpService.registerServlet("/cache", cacheHttpFrontend, null, null);

From that moment onwards we are sending any ‘/cache’ URLs onto the servlet instance, which will use ‘/cache’ as the key.

The servlet can implement the usual methods ‘doPost’ and ‘doPut’ to add data onto the cache and doGet to retrieve it. These method calls translate into the appropriate Cache calls. Any exceptions or null values can be translated into equivalent HTTP errors or responses. For example, we can return a 500 if an unrecoverable exception is thrown from the cache or a 400 if the path is simply ‘/cache’ (which means no key is provided).

As a result of thinking in HTTP terms, we come to realise we need some metadata about the contents of the cache. At least, we need to store the mime type of the data so we can properly serve multimedia and any content other than text. Moreover, we could store some more metadata to tune up the cache, do automatic refreshing of the content, etc.

One possible strategy is to ask the Cache to store a POJO instance that holds the metadata as well as the cached content itself. This seems the simplest approach at the moment so this is what we do. The Cache is still a generic service and as long as we prefix the HTTP caches accordingly at the app level we will be fine with using it in other clients (and these might revert to MIME as their way to tag content types as well).

We can create a custom class POJO but that means that any backend that does POJO serialisation needs access to this class so it can create one. To avoid that class and encapsulation break we just use a plain vanilla serialisable class (Map). In any case, we need to make a not of this problem of the design so it is addressed in the future.

NOTE: it could be addressed in a simple enough manner. The cache entry object interface could be formalised and exported by the controller. In this manner, the front-end and the providers could make use of it and serialise it properly.

We create the cache entry Map instance in a simple enough manner:

HashMap httpCacheEntry = new HashMap(2);
httpCacheEntry.put(ENTRY_BODY_KEY, body);
httpCacheEntry.put(ENTRY_MIME_KEY, contentType);
cache.set(key,httpCacheEntry);

In this manner, we can recover the appropriate metadata (in this case MIME type) from the entry cache whenever we do a get() operation. This doesn’t mean we expect all requested entries to have that structure (principle of robustness), it just means we will use it if it’s there, setting the HTTP response MIME type accordingly.

With these enhancements and a few more small code refactors we can stabilise our code and call it a day. To summarise what we have done in this post:

• Created a new CacheProvider implementation using memcached
• Added a bundle that provides a frontend of the system using HTTP and a Servlet
• Devised a way to store some metadata on the cache
• Done some code cleanup and refactoring.

As usual, you can download the code here. Thanks!

Though simple and easy to understand, the first example does nothing out of the ordinary. It is far more interesting to start exploiting some of the basic features OSGi gets us “for free”.

For instance, we could begin by moving our first basic implementation into a separate “model” bundle and enhancing the interface so it can throw exceptions. For instance, an exception can be thrown when no implementations are available or cannot be contacted/operated.

Read the rest of the post for the implementation details…

We start by picking up the project as we left it and firstly export the controller service. Open the MANIFEST.MF and on the ‘Runtime’ section we add the ‘com.calidos.dani.osgi.cache.controller’ package. This will enable any bundle clients to use the cache.

Secondly we create and expose the backend interface we want the model bundles to implement. This is the ‘service’ the bundles can offer that our controller is interested in and is able to use as engines to store the cache data. The interface obviously needs to mimic most if not all of the cache’s public service but we also add lower level functionality, etc. A trivial way to do that is to have the declared backend interface inherit from the public service one.

public interface CacheProvider extends Cache {

CacheProviderStatus getStatus();	

}

At the moment we define a method to get the status of the backend implementation, we return an object as opposed to a simple type to be able to enrich it in the future. If we place this interface in its own package we need to export it on the plug-in manifest as well.

We then the enhance the original interface with exceptions, which give information about the cause of the problem yet they shouldn’t break up too much encapsulation.

public interface Cache {
[...]
void init() throws CacheProviderException;
[…]

It is interesting to note that depending on how we leverage the component nature of OSGi some of these errors can actually be recoverable which is what makes exceptions shine in java.

When adding the exceptions all our tests don’t compile anymore so it’s time to fix them by for example adding the throws declaration into the test signature as at the moment there is squat we can do about exceptions being thrown and these should be test failures anyway.

Cool, so next we create a new project to hold the in-memory implementation of the cache on its own. So we create a new plug-in project and name it accordingly:

Good, next we move our code from the controller bundle onto the new one. First of all we need to modify the new plug-in MANIFEST.MF to reflect we depend on the CacheProvider and Cache interfaces:

This means we can move the code from the controller onto this new provider bundle. Code compiles but our original tests don’t as the actual code has moved.
We first need to change the BasicCacheController class so it implements the more low-level provider interface, time for some refactoring (we also rename the implementation class to something more indicative of its function).

The tests need passing and actually we can move the old test code onto the new component as it effectively tests the in-memory implementation perfectly well. The controller component will need more sophisticated code which we will do after we finish the cleanup. Once the test is moved and the refactoring is complete tests on the new plug-in pass.

Summarizing:

• Move BasicCache class to com.calidos.dani.osgi.cache.provider.memory bundle
• Rename BasicCache to MemoryCache
• Move test from old bundle onto com.calidos.dani.osgi.cache.provider.memory
• Fix compiling errors on the test and rename it to MemoryCacheTest
• Make MemoryCache implement the CacheProvider interface
• Modify the MemoryCacheTest class to test the extended CacheProvider functionality

hat’s cool so now it is time to work in the OSGi world to wire the components together. To do that we edit the memory cache provider Activator class to register the service once it is activated like this:

public class Activator implements BundleActivator {

private MemoryCache cacheService;
private ServiceRegistration registration;

/*
* (non-Javadoc)
* @see org.osgi.framework.BundleActivator#start(org.osgi.framework.BundleContext)
*/
public void start(BundleContext context) throws Exception {

cacheService = new MemoryCache();
cacheService.init();
registration = context.registerService(CacheProvider.class.getName(), cacheService, null);

}

/*
* (non-Javadoc)
* @see org.osgi.framework.BundleActivator#stop(org.osgi.framework.BundleContext)
*/
public void stop (BundleContext context) throws Exception {

registration.unregister();
cacheService.clear();

}

}

On the start() method we are creating a memory cache object and then registering it under the CacheProvider interface class name. This means that any other bundle in the OSGi environment that is interested in cache services can use the service provided by this bundle.

This highlights an interesting problem. Is it better to be active like this code and init the service before registering it? Or should we wait for the consumer to do it and follow a more lazy-loading design? That is actually a good question and I have just decided to create self-contained services that are fully operative once published as I think that leverages OSGi better but YMMV.

Next, we need the main controller to wait for cache provider services. To do that we can create a ServiceTracker subclass on the controller component. On that subclass we implement two methods that are called whenever a cache provider is registered into the context and when it is unregistered. We also supply a constructor that stores the controller Activator so we can pass the services back.

public class CacheProviderTracker extends ServiceTracker {

protected Activator activator;
protected static Logger log = Logger.getLogger(CacheProviderTracker.class);

public CacheProviderTracker (BundleContext context, String clazz,
		ServiceTrackerCustomizer customizer, Activator activator) {

super(context, clazz, customizer);
log.debug("CacheProviderTracker built");
this.activator = activator;

}

@Override
public Object addingService (ServiceReference reference) {		

	CacheProvider cacheProvider = (CacheProvider) context.getService(reference);
	log.debug("Obtained a new CacheProvider service");
	return cacheProvider;

}

@Override
public void removedService(ServiceReference reference, Object service) {

	log.debug("A CacheProvider service has been removed from the context");
	context.ungetService(reference);		

}

}

We then modify the callbacks so the services are actually passed onto the Activator for their use. We need to create the appropriate methods in the Activator and we will fill them up next.

public void addCacheProvider(CacheProvider cacheProvider) {
}

public void removeService(CacheProvider service) {
}

Now we need to glue the backend provider with the controller so any Cache interface requests are passed onto CacheProvider services. To do that we create a new Cache interface implementation class in the controller bundle (appropriately named CacheControllerCore). The signature is something like:

public class CacheController implements Cache {

Which means we can offer this class as a Cache service once it’s ready. So when should we create a CacheControllerCore instance and offer it as a service? Well, whenever we have backend providers ready, not before. Therefore we now complete the addCacheProvider in the Activator method to do so:

if (controller==null) {
	controller = new CacheControllerCore();
	controller.addCacheProvider(cacheProvider);
} else {
	controller.addCacheProvider(cacheProvider);
}

This creates the instance only the fist time a provider is registered and after that just simply tells the controller class that an extra one is available. An added bonus of doing on a controller class instead of directly in the Activator is that we don’t want to overload the Activator with lots of logic. We also complete the removeService method. If in the future we deal with more services we will need a refactor as we would be dealing with a lot of handover methods from the activator class onto the core controller.

Okay, so what is the main goal of the CacheControllerCore then? To maintain a list of providers and hand over requests to them. As a convention and for greater flexiblity, it will send all messages to all services except for get, which will basically hand over the first object it finds.

DESIGN WARNING: in doing the implementation of the core controller we realise there are potential thread-safe potential problems in accessing the provider list. This is a cache and not an ACID system and we are looking for speed + 99.99% efficacy when changing providers and 100% efficacy when providers are stable. One should consider that in 99.9999999% of the time we are not changing providers mid-flight we wouldn’t want to compromise the overall efficiency of the architecture by the odd chance that two or three requests out of a million might actually get a miss from the cache where they should have had a hit or a cache set operation needs to be retried. Therefore, we implement a few basic checks on the core to minimize problems and provide graceful degradation of service in edge cases as opposed to meaningless null pointer exceptions.

Once this cautious but hopefully practical implementation is complete we can take a step back and review what we have done so far.

• We have defined a cache backend provider interface
• Also created a simple provider implementation that stores data in memory
• We have also glued the backend example to the core and refactored things a bit
• The core now loops through available backends and is reasonably resilient

Click here to download the code as it stands. More to come!

I have chosen a simple idea: build a volatile cache that stores objects. The interface with it is simple HTTP calls to perform usual operations: get, set, clear and get info. Additionally, Unit Testing will of course be used (but not Acceptance Testing in this case).

We start with building a component that serves as the controller for the system. It encapsulates any implementation details and answers to the system requests.

Read on for a short tutorial on how to setup the a first bundle, add an interface and some logging and fire up the appropriate OSGi environment.

Setting up the project

We fire up Eclipse and switch to the ‘Plug-in Development’ perspective and create a new ‘Plug-in Project’ using the Other option.

As we are bulding the main controller component for the cache, we name it appropriately e.g. ‘com.calidos.dani.osgi.cache.controller’. Remember to select Equinox as the targeted environment as well as create a Java project along with it.

Next step is to specify the plug-in basic data such as ID and minimum requested execution environment.

In this case we select Java SE 1.6 but in production we should select something lower unless there are any specific VM requirements. As this is a controller-like component it is quite likely that some sort of state activation will take place at some point so we have the wizard generate an activator class.

We unselect the template (we will not be using any) and off we go onto the next step.

Adding some logging

The first thing the wizard does after the definition is finished is present the Manifest overview screen and from there we can tweak dependencies, API exposure and any other properties of the bundle. We can also examine the default Activator java class that contains empty stub start and stop methods.

We clearly need some sort of logging so we add log4j support using the ‘dependencies’ tab.

This means that the bundle won’t load unless the official org.apache.log4j bundle is present in the environment. Additionally, we need somewhere to put the log4j configuration file. There are several alternatives for this but one I feel to be quite elegant is to create a bundle fragment to hold the configuration and attach it to wherever needed.

Using the wizard, we can create a Plug-in Fragment project named ‘com.calidos.dani.osgi.cache.log4jconfig’. In this case we don’t need to create a Java project as the fragment doesn’t have any java code. We configure the fragment to be attached to the log4j bundle. We then add our preferred log4j.properties file to the root of that fragment. To make sure we export the configuration file we select it on the ‘Build’ tab of the manifest wizard. When the fragment is loaded unto the log4j bundle, the log4j.properties file will be made visible to the bundle and configure the log4j environment accordingly.

We create the OSGi environment where we can test our bundle by selecting ‘Run:Run Configurations…’. We create a new ‘OSGi’ configuration and we name it something like ‘Basic Cache env’. One quick way to create the environment is to first deselect all bundles, look for the main bundle ID we wat to run (in this case ‘com.calidos.dani.osgi.cache.controller’) and click on ‘Add Required Bundles’. This should automagically add any required bundles you may need. Make sure the log4j fragment is also selected.

We run the ‘Basic Cache env’ and all should load appropriately. Typing ‘ss’ on the console should spit out something like:

Framework is launched.

id	State       Bundle
0	ACTIVE      org.eclipse.osgi_3.4.3.R34x_v20081215-1030
1	ACTIVE      com.calidos.dani.osgi.cache.controller_1.0.0
2	ACTIVE      org.apache.log4j_1.2.13.v200806030600
	            Fragments=4
4	RESOLVED    com.calidos.dani.osgi.cache.log4jconfig_1.0.0
	            Master=2

This is telling us we have only loaded the bundles we want and all are loaded and properly started. Note that the fragment is only ‘RESOLVED’ as it is not a full bundle in itself.

In true test-driven development style, we can write some tests first that help define the expected behaviour of the bundle. There are several options and schools of thought about how to best do this (for instance creating a separate bundle for the tests, etc.). In our case we incorporate the test in the bundle for simplicity.

We edit our MANIFEST.MF and add org.junit4 as an optional ‘Required Plug-in’ to be able to load tests. In this case then we create a separate ‘test’ java source folder for the bundle and add test cases to it. To define the test cases we need to think about the main interface of the cache.

It seems clear that we can initialize and check the status of the cache and ask how many items we have got on it. We can also clear all items present in the cache. We create the interface and an implementation stub and finally add test code. The interface looks like this:

public interface Cache {

/** Initialize with 0 elems and default starting capacity
*/
void init();

/**	Initialize the cache with zero elements
* 	@param startingCapacity hint to make room
*/
void init(int startingCapacity);

/** @return the number of objects at this time
*/
int size();

/** Clear the cache from any elements
*/
void clear();

[...]

}

Obviously all tests fail at the beginning as the default behaviour for them should be failing (and the current JUnit 4 Eclipse integration puts a ‘fail’ on each test). As it is an internal Unit test we expose the implementation on the setup test method but we actually use the interface on the tests themselves, which preserves implementation details from the tests themselves.

Selecting the test class and choosing ‘Run:Run As’ menu option we can run the tests as standalone JUnit tests or within an OSGi environment, shell included.

Here you can download an export of the project with all the source so far. Hope this is useful and complements other similar resources on the Web. The source is available in the Apache 2.0 license so enjoy.

Update: here you can find an archive with the logging configuration fragment

PS: I have an easy and straightforward method to write this post offline. I write the article on TextEdit, pasting the images as I go. This creates a .rtfd folder with the images stored in the TIFF inside. I’ve written a trivial 5-minute Automator script that turns the images into PNGs, numbers, sorts and renames the files. It is quite handy as I finally post this.

A fundamental problem that this technology solves is being able to load an OSGi environment on a servlet server. That is a relatively costly operation that needs to be done only once and be persistent on the server in between client requests. After that point, state is maintained and bundles can be deployed and managed as needed. Creating, loading and destroying the environment for each request would just be unacceptable. One interesting functionality is then being able to serve client requests, specially HTTP requests, uploads, provide REST interfaces, etc.

Eclipse provides a number of projects to load the environment, register servlets, handle http requests, etc. Of particular interest is the ‘org.eclipse.equinox.servletbridge’ project, which initally starts the framework, loads the appropriate framework bundles, etc.

Looking at the code, you can start by checking out the main class, which has a number of interesting responsabilities:

/**
 * The BridgeServlet provides a means to bridge the servlet and
OSGi runtimes. This class has 3 main responsibilities:
 * 1) Control the lifecycle of the associated FrameworkLauncher
in line with its own lifecycle
 * 2) Provide a servlet "hook" that allows all servlet requests
to be delegated to the registered servlet
 * 3) Provide means to manually control the framework lifecycle
 */

I personally like classes that have a small set of responsabilities, but one can argue that the three are so related that they are expressions of the same one. The first responsability described is actually managing the lifecycle of the OSGi framework FrameworkLauncher class.

Basically, the Servlet Container will load the bridge web application, create an instance of ServletBridge and call its init method (defined by the HttpServlet interface which it implements). In that method the OSGi environment will be created and loaded by the following code:

framework.init(getServletConfig());
framework.deploy();
framework.start();
frameworkStarted = true;

The attribute ‘framework’ is an instance of the ‘FrameworkLauncher’ class, which encapsulates the OSGi environment management logic. The ‘init’ method pulls information from the servlet configuration (such as the name of the ‘WEB-INF’ folder) and calls an overloaded empty init method that could be exploited by specialised subclasses (more on this later).

After that, the framework is “deployed”. The method in question declares the following contract:

/**
* deploy is used to move the OSGi framework libraries into a location
   suitable for execution.
* The default behavior is to copy the contents of the webapp's
   WEB-INF/eclipse directory to the webapp's temp directory.
*/

And that’s right, the code is quite straightforward and goes along the lines of:

File servletTemp = (File) context.getAttribute("javax.servlet.context.tempdir");
platformDirectory = new File(servletTemp, "eclipse");
if (!platformDirectory.exists()) {
	platformDirectory.mkdirs();
}
File plugins = new File(platformDirectory, "plugins");
copyResource(resourceBase + "plugins/", plugins);

Which copies the bundles you want to deploy onto the OSGi environment onto the Servlet Container temporary folder (the ‘work’ folder in the case of Apache Tomcat).

After that the framework is started, reading .ini configuration options, command line switches, bundle list, run levels and so forth. We have not been able to get bundles started automatically unless we specify the bundle filename, complete with ‘.jar’ extension and all. One interesting option is the ability to fire up a standard OSGi console that uses the STDIN/OUT of the Servlet Container process.

Using the very same servlet bridge, there are a number of URLs that can be hit do control the framework, start, stop, undeploy, etc.

After that, one can register servlets as extension points or listen for services named “org.osgi.service.http.HttpService” and adding servlets with code going along these lines:

public Object addingService(ServiceReference reference) {
	HttpService httpService = (HttpService) context.getService(reference);
	httpService.registerServlet(url, servlet, null, null);
}

All is well and good. However, in working with the bridge code we have found a glitch in the startup code. Whenever the bridge app is started anew, it does the framework deployment and startup as expected but if there are any bundles that have any changes they will not be read by the environment unless the redeploy url is hit. This is fine in development environments but not on production, where you want to keep things  as smooth as possible.

Behold the power of OSS, we took a look at the code and submitted a bug entry complete with a working patch that fixes the problem. It is not pretty but it works (it has some code duplication).

Server-side OSGi is proving to be the place to be, or at least it is quite promising. Give it a try.