Switching between data sources when using @DataSourceDefinition

21 May 2013, by: Arjan Tijms

Prior to Java EE 6 setting up and configuring data sources had to be done in a proprietary (vendor specific) way. In many cases this meant a data source had to be created inside the application server.

This may make sense when such an application server runs a multitude of applications and/or those applications are externally obtained ones. In such case it’s great that all those applications can share the same data source and applications themselves don’t dictate which database is being used (via hard-coded referenced to a specific driver).

However, when you run one application per server and especially when that one application is your primary in-house developed code, it’s not always that convenient; you will have to store your data source definitions somewhere away from your code (e.g. in a CFEngine managed repository) and changes to the data source won’t be pulled in together with new code when you pull from your SCM.

For those situations, especially when working in agile and devops centered teams, Java EE 6 introduced the @DataSourceDefinition annotation and data-source element for usage in deployment descriptors such as web.xml.

Although some vendors were supposedly slightly reluctant to support this, it now works reasonably well.

A typical example:

web.xml

<data-source>
    <name>java:app/KickoffApp/kickoffDS</name>
    <class-name>org.h2.jdbcx.JdbcDataSource</class-name>
    <url>jdbc:h2:mem:test</url>
    <user>sa</user>
    <password>sa</password>
    <transactional>true</transactional>
    <isolation-level>TRANSACTION_READ_COMMITTED</isolation-level>
    <initial-pool-size>2</initial-pool-size>
    <max-pool-size>10</max-pool-size>
    <min-pool-size>5</min-pool-size>
    <max-statements>0</max-statements>
</data-source>

Source

One small issue remained though. How do you easily change the settings of such a data source for different stages (e.g. DEV, QA, Production)?

The official way in Java EE is by providing different versions of deployment descriptors like web.xml, but this has two problems:

  • The entire file needs to be swapped, even when only a few changes are needed
  • The file is embedded in the .war/.ear archive. Prying it open, changing the file and closing it again is tedious

One solution is to use build tools to swap in different versions of the deployment descriptor and/or use placeholders that are replaced at build time. Although this is certainly an option, it doesn’t always play nice with incremental builds in IDEs such as Eclipse and can be tricky (but not impossible) to fit into CI pipelines, where the build is tested on some local test server and then as-is automatically deployed to another server.

Another solution that I would like to present here is making use of a data source wrapper that loads its settings from a user defined location (which can be parametrized) and passes it on to the real data source.

Design

Creating a wrapper is by itself simple enough, but one challenge lies in the way how properties are set on a DataSource. There are hardly any properties defined via an interface and there’s no universal setter or map available. Instead, the server inspects the DataSource for JavaBeans properties via reflection and calls those via reflection as well. Obviously a wrapper cannot dynamically at run time add properties to itself.

Fortunately, there are some standard properties that are defined in the JDBC spec that we can statically implement. It’s perhaps a question if we really need them, since we’ll be setting most properties ourselves on the wrapped real data source, but it might be convenient to have them anyway.

We’ll start off with a wrapper for the CommonDataSource, which is the base class for the most important data source types, such as plain DataSource and XADataSource. The most important methods of this wrapper are initDataSource, get and set, and setWithConversion, which are discussed below. The full code is given at the end of this article.

In initDataSource we set the wrapped data source and collect its properties. There are many reflection libraries that make it easier to work reflectively with properties, but using the venerable java.beans.Introspector proved to be good enough here. The only extra thing that was needed was storing the obtained properties in a map (JDK 8 lambdas would sure make this particular task even more straightforward).

public void initDataSource(CommonDataSource dataSource) {
    this.commonDataSource = dataSource;
 
    try {
        Map<String, PropertyDescriptor> mutableProperties = new HashMap<>();
        for (PropertyDescriptor propertyDescriptor : getBeanInfo(dataSource.getClass()).getPropertyDescriptors()) {
            mutableProperties.put(propertyDescriptor.getName(), propertyDescriptor);
        }
 
        dataSourceProperties = unmodifiableMap(mutableProperties);
 
    } catch (IntrospectionException e) {
        throw new IllegalStateException(e);
    }
}

The next thing we do is creating a get and set method for the obtained properties. Calling getReadMethod().invoke(…) on a given property isn’t actually that bad, but the multitude of checked exceptions spoil the party a little. It would be really cool if there was just a Property in the JDK with a simple unchecked get and set method, but as shown it’s nothing a little helper code can’t fix:

@SuppressWarnings("unchecked")
public <T> T get(String name) {
    try {
        return (T) dataSourceProperties.get(name).getReadMethod().invoke(commonDataSource);
    } catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException e) {
        throw new IllegalStateException(e);
    }
}
 
public void set(String name, Object value) {
    try {
        dataSourceProperties.get(name).getWriteMethod().invoke(commonDataSource, value);
    } catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException e) {
        throw new IllegalStateException(e);
    }
}

Next we come to setWithConversion. This allows us to set a property with the correct type using a string representation of the value (which is what you typically have when reading such values from a property file). It appears this can be done with just a few lines of code using the java.beans.PropertyEditorManager class. You can obtain a kind of converter called a PropertyEditor from this based on a class type, which you can feed a String and will return a converted value for the target type.

It stands to reason PropertyEditor was originally designed for a rather different environment, seeing that it contains methods for usage with AWT such as paintValue(Graphics gfx, Rectangle box) and a really obscure method called getJavaInitializationString() that generates a Java code fragment :X. It’s a tad scary to realize these methods are present in code that runs deep inside a Java EE server with not a graphics card in sight, but alas, it’s part of the JDK and as it appeared used quite a lot on the server side by code that works with beans (like e.g. expression language).

Anyway, here’s the implementation:

public void setWithConversion(String name, String value) {
 
    PropertyDescriptor property = dataSourceProperties.get(name);
 
    PropertyEditor editor = findEditor(property.getPropertyType());
    editor.setAsText(value);
 
    try {
        property.getWriteMethod().invoke(commonDataSource, editor.getValue());
    } catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException e) {
        throw new IllegalStateException(e);
    }
}

Next we’ll create a sub class that does the actual switching for a CommonDataSource. Unfortunately there is not really any notion of a lifecyle for a DataSource. The application server just creates an instance using a no-arguments constructor, calls setters on it, and then eventually retrieves a connection from it. If we want to be capable of accepting properties from the @DataSourceDefinition annotation or data-source element in addition to the ones we read from our own file, we have to use a little trick.

Initially we collect properties in a temporary map:

public void set(String name, Object value) {
    if (init) {
        super.set(name, value);
    } else {
        tempValues.put(name, value);
    }
}

When we receive the special property “configFile” we start the initialization:

public void setConfigFile(String configFile) {
    this.configFile = configFile;
    doInit();
}

In this initialization method we load our own properties, and from those fetch another special property called “className”, which is the fully qualified class name of the actual data source. Using the setter methods shown above we set the properties that we collected earlier as well as the properties we read our selves:

public void doInit() {
 
    // Get the properties that were defined separately from the @DataSourceDefinition/data-source element
    Map<String, String> properties = PropertiesUtils.getFromBase(configFile);
 
    // Get & check the most important property; the class name of the data source that we wrap.
    String className = properties.get("className");
    if (className == null) {
        throw new IllegalStateException("Required parameter 'className' missing.");
    }
 
    initDataSource(newInstance(className));
 
    // Set the properties on the wrapped data source that were already set on this class before doInit()
    // was possible.
    for (Entry<String, Object> property : tempValues.entrySet()) {
        super.set(property.getKey(), property.getValue());
    }
 
    // Set the properties on the wrapped data source that were loaded from the external file.
    for (Entry<String, String> property : properties.entrySet()) {
        if (!property.getKey().equals("className")) {
            setWithConversion(property.getKey(), property.getValue());
        }
    }
 
    // After this properties will be set directly on the wrapped data source instance.
    init = true;
}

Because of the JDBC distinction between different data source types, the one more thing left to do is to create the sub class that contains the methods specific for that data source type. This is a small nuisance, but otherwise rather straightforward. For an XA data source it contains the two getXAConnection methods, e.g.

public XADataSource getWrapped() {
    return (XADataSource) super.getWrapped();
}
 
public XAConnection getXAConnection() throws SQLException {
    return getWrapped().getXAConnection();
}

Finally via PropertiesUtils.getFromBase(configFile) a config file is loaded from some location based on a system property (-D commandline option). At the end of the article a somewhat hacky example is shown for loading this from the root of an EAR archive. Unfortunately the root of an EAR is not on the classpath and neither is its META-INF, therefor the chosen solution is rather hacky. It works on JBoss AS 7.x though.

For a WAR archive there’s no super convenient location. /conf in the root would be ideal, but unfortunately in a WAR the root is also not on the classpath and instead directly contains the resources that are made available to web clients. WEB-INF/classes/conf or WEB-INF/classes/META-INF/conf would be the most practical location.

Usage

Having created all the classes, the wrapper class can be specified in e.g. application.xml as follows:

<data-source>
    <name>java:app/myDS</name>
    <class-name>com.example.SwitchableXADataSource</class-name>
 
    <property>
        <name>configFile</name>
        <value>datasource-settings.xml</value>
    </property>
 
</data-source>

Specific settings can be put in e.g. an XML properties files as follows:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties>
 
    <entry key="className">org.postgresql.xa.PGXADataSource</entry>
 
    <entry key="user">user</entry>
    <entry key="password">password</entry>
 
    <entry key="serverName">database.example.com</entry>
    <entry key="databaseName">example_db</entry>
    <entry key="portNumber">5432</entry>
 
</properties>

Source

CommonDataSourceWrapper

package com.example;
 
import static java.beans.Introspector.getBeanInfo;
import static java.beans.PropertyEditorManager.findEditor;
import static java.util.Collections.unmodifiableMap;
 
import java.beans.IntrospectionException;
import java.beans.PropertyDescriptor;
import java.beans.PropertyEditor;
import java.lang.reflect.InvocationTargetException;
import java.sql.SQLException;
import java.sql.SQLFeatureNotSupportedException;
import java.util.HashMap;
import java.util.Map;
import java.util.logging.Logger;
 
import javax.sql.CommonDataSource;
 
public class CommonDataSourceWrapper implements CommonDataSource {
 
    private CommonDataSource commonDataSource;
    private Map<String, PropertyDescriptor> dataSourceProperties; 
 
    public void initDataSource(CommonDataSource dataSource) {
        this.commonDataSource = dataSource;
 
        try {
            Map<String, PropertyDescriptor> mutableProperties = new HashMap<>();
            for (PropertyDescriptor propertyDescriptor : getBeanInfo(dataSource.getClass()).getPropertyDescriptors()) {
                mutableProperties.put(propertyDescriptor.getName(), propertyDescriptor);
            }
 
            dataSourceProperties = unmodifiableMap(mutableProperties);
 
        } catch (IntrospectionException e) {
            throw new IllegalStateException(e);
        }
    }
 
    @SuppressWarnings("unchecked")
    public <T> T get(String name) {
        try {
            return (T) dataSourceProperties.get(name).getReadMethod().invoke(commonDataSource);
        } catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException e) {
            throw new IllegalStateException(e);
        }
    }
 
    public void set(String name, Object value) {
        try {
            dataSourceProperties.get(name).getWriteMethod().invoke(commonDataSource, value);
        } catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException e) {
            throw new IllegalStateException(e);
        }
    }
 
 
    public void setWithConversion(String name, String value) {
 
        PropertyDescriptor property = dataSourceProperties.get(name);
 
        PropertyEditor editor = findEditor(property.getPropertyType());
        editor.setAsText(value);
 
        try {
            property.getWriteMethod().invoke(commonDataSource, editor.getValue());
        } catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException e) {
            throw new IllegalStateException(e);
        }
    }
 
    public CommonDataSource getWrapped() {
        return commonDataSource;
    }
 
 
    // ------------------------- CommonDataSource-----------------------------------
 
    @Override
    public java.io.PrintWriter getLogWriter() throws SQLException {
        return commonDataSource.getLogWriter();
    }
 
    @Override
    public void setLogWriter(java.io.PrintWriter out) throws SQLException {
        commonDataSource.setLogWriter(out);
    }
 
    @Override
    public void setLoginTimeout(int seconds) throws SQLException {
        commonDataSource.setLoginTimeout(seconds);
    }
 
    @Override
    public int getLoginTimeout() throws SQLException {
        return commonDataSource.getLoginTimeout();
    }
 
    // ------------------------- CommonDataSource JDBC 4.1 -----------------------------------
 
    @Override
    public Logger getParentLogger() throws SQLFeatureNotSupportedException {
        return commonDataSource.getParentLogger();
    }
 
 
    // ------------------------- Common properties -----------------------------------
 
    public String getServerName() {
        return get("serverName");
    }
 
    public void setServerName(String serverName) {
        set("serverName", serverName);
    }
 
    public String getDatabaseName() {
        return get("databaseName");
    }
 
    public void setDatabaseName(String databaseName) {
        set("databaseName", databaseName);
    }
 
    public int getPortNumber() {
        return get("portNumber");
    }
 
    public void setPortNumber(int portNumber) {
        set("portNumber", portNumber);
    }
 
    public void setPortNumber(Integer portNumber) {
        set("portNumber", portNumber);
    }
 
    public String getUser() {
        return get("user");
    }
 
    public void setUser(String user) {
        set("user", user);
    }
 
    public String getPassword() {
        return get("password");
    }
 
    public void setPassword(String password) {
        set("password", password);
    }
 
    public String getCompatible() {
        return get("compatible");
    }
 
    public void setCompatible(String compatible) {
        set("compatible", compatible);
    }
 
    public int getLogLevel() {
        return get("logLevel");
    }
 
    public void setLogLevel(int logLevel) {
        set("logLevel", logLevel);
    }
 
    public int getProtocolVersion() {
        return get("protocolVersion");
    }
 
    public void setProtocolVersion(int protocolVersion) {
        set("protocolVersion", protocolVersion);
    }
 
    public void setPrepareThreshold(int prepareThreshold) {
        set("prepareThreshold", prepareThreshold);
    }
 
    public void setReceiveBufferSize(int receiveBufferSize) {
        set("receiveBufferSize", receiveBufferSize);
    }
 
    public void setSendBufferSize(int sendBufferSize) {
        set("sendBufferSize", sendBufferSize);
    }
 
    public int getPrepareThreshold() {
        return get("prepareThreshold");
    }
 
    public void setUnknownLength(int unknownLength) {
        set("unknownLength", unknownLength);
    }
 
    public int getUnknownLength() {
        return get("unknownLength");
    }
 
    public void setSocketTimeout(int socketTimeout) {
        set("socketTimeout", socketTimeout);
    }
 
    public int getSocketTimeout() {
        return get("socketTimeout");
    }
 
    public void setSsl(boolean ssl) {
        set("ssl", ssl);
    }
 
    public boolean getSsl() {
        return get("ssl");
    }
 
    public void setSslfactory(String sslfactory) {
        set("sslfactory", sslfactory);
    }
 
    public String getSslfactory() {
        return get("sslfactory");
    }
 
    public void setApplicationName(String applicationName) {
        set("applicationName", applicationName);
    }
 
    public String getApplicationName() {
        return get("applicationName");
    }
 
    public void setTcpKeepAlive(boolean tcpKeepAlive) {
        set("tcpKeepAlive", tcpKeepAlive);
    }
 
    public boolean getTcpKeepAlive() {
        return get("tcpKeepAlive");
    }
 
    public void setBinaryTransfer(boolean binaryTransfer) {
        set("binaryTransfer", binaryTransfer);
    }
 
    public boolean getBinaryTransfer() {
        return get("binaryTransfer");
    }
 
    public void setBinaryTransferEnable(String binaryTransferEnable) {
        set("binaryTransferEnable", binaryTransferEnable);
    }
 
    public String getBinaryTransferEnable() {
        return get("binaryTransferEnable");
    }
 
    public void setBinaryTransferDisable(String binaryTransferDisable) {
        set("binaryTransferDisable", binaryTransferDisable);
    }
 
    public String getBinaryTransferDisable() {
        return get("binaryTransferDisable");
    }
 
}

SwitchableCommonDataSource

package com.example;
 
import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;
 
import javax.sql.CommonDataSource;
 
public class SwitchableCommonDataSource extends CommonDataSourceWrapper {
 
    private boolean init;
    private String configFile;
    private Map<String, Object> tempValues = new HashMap<>();
 
    @Override
    public void set(String name, Object value) {
        if (init) {
            super.set(name, value);
        } else {
            tempValues.put(name, value);
        }
    }
 
    @SuppressWarnings("unchecked")
    @Override
    public <T> T get(String name) {
        if (init) {
            return super.get(name);
        } else {
            return (T) tempValues.get(name);
        }
    }
 
    public String getConfigFile() {
        return configFile;
    }
 
    public void setConfigFile(String configFile) {
        this.configFile = configFile;
 
        // Nasty, but there's not an @PostConstruct equivalent on a DataSource that's called
        // when all properties have been set.
        doInit();
    }
 
    public void doInit() {
 
        // Get the properties that were defined separately from the @DataSourceDefinition/data-source element
        Map<String, String> properties = PropertiesUtils.getFromBase(configFile);
 
        // Get & check the most important property; the class name of the data source that we wrap.
        String className = properties.get("className");
        if (className == null) {
            throw new IllegalStateException("Required parameter 'className' missing.");
        }
 
        initDataSource(newInstance(className));
 
        // Set the properties on the wrapped data source that were already set on this class before doInit()
        // was possible.
        for (Entry<String, Object> property : tempValues.entrySet()) {
            super.set(property.getKey(), property.getValue());
        }
 
        // Set the properties on the wrapped data source that were loaded from the external file.
        for (Entry<String, String> property : properties.entrySet()) {
            if (!property.getKey().equals("className")) {
                setWithConversion(property.getKey(), property.getValue());
            }
        }
 
        // After this properties will be set directly on the wrapped data source instance.
        init = true;
    }
 
    private CommonDataSource newInstance(String className) {
        try {
            return (CommonDataSource) Class.forName(className).newInstance();
        } catch (InstantiationException | IllegalAccessException | ClassNotFoundException e) {
            throw new IllegalStateException(e);
        }
    }
 
}

SwitchableXADataSource

package com.example;
 
import java.sql.SQLException;
 
import javax.sql.XAConnection;
import javax.sql.XADataSource;
 
public class SwitchableXADataSource extends CommonDataSourceWrapper implements XADataSource {
 
    public XADataSource getWrapped() {
        return (XADataSource) super.getWrapped();
    }
 
 
    // ------------------------- XADataSource-----------------------------------
 
    @Override
    public XAConnection getXAConnection() throws SQLException {
        return getWrapped().getXAConnection();
    }
 
    @Override
    public XAConnection getXAConnection(String user, String password) throws SQLException {
        return getWrapped().getXAConnection();
    }
 
}

PropertiesUtils

package com.example;
 
import static java.lang.System.getProperty;
import static java.util.Collections.unmodifiableMap;
import static java.util.logging.Level.SEVERE;
 
import java.io.IOException;
import java.net.URL;
import java.util.HashMap;
import java.util.Map;
import java.util.Properties;
import java.util.logging.Logger;
 
public class PropertiesUtils {
 
    private static final Logger logger = Logger.getLogger(PropertiesUtils.class.getName());
 
    public static Map<String, String> getFromBase(String base) {
        String earBaseUrl = getEarBaseUrl();
        String stage = getProperty("example.staging");
        if (stage == null) {
            throw new IllegalStateException("example.staging property not found. Please add it, e.g. -Dexample.staging=dev");
        }
 
        Map<String, String> settings = new HashMap<>();
 
        loadXMLFromUrl(earBaseUrl + "/conf/" + base, settings);
        loadXMLFromUrl(earBaseUrl + "/conf/" + stage + "/" + base, settings);
 
        return unmodifiableMap(settings);
    }
 
    public static String getEarBaseUrl() {
        URL dummyUrl = Thread.currentThread().getContextClassLoader().getResource("META-INF/dummy.txt");
        String dummyExternalForm = dummyUrl.toExternalForm();
 
        int ejbJarPos = dummyExternalForm.lastIndexOf(".jar");
        if (ejbJarPos != -1) {
 
            String withoutJar = dummyExternalForm.substring(0, ejbJarPos);
            int lastSlash = withoutJar.lastIndexOf('/');
 
            return withoutJar.substring(0, lastSlash);
        }
 
        throw new IllegalStateException("Can't derive EAR root from: " + dummyExternalForm);
    }
 
    public static void loadXMLFromUrl(String url, Map<String, String> settings) {
 
        try {
            Properties properties = new Properties();
            properties.loadFromXML(new URL(url).openStream());
 
            logger.info(String.format("Loaded %d settings from %s.", properties.size(), url));
 
            settings.putAll(asMap(properties));
 
        } catch (IOException e) {
            logger.log(SEVERE, "Eror while loading settings.", e);
        }
    }
 
    @SuppressWarnings({ "rawtypes", "unchecked" })
    private static Map<String, String> asMap(Properties properties) {
        return (Map<String, String>)( (Map) properties);
    }
 
}

Future

It would be great if Java EE had some more support for easily switching between different configurations, without necessarily having to install such configuration on a dedicated server. A while back I created JAVAEE_SPEC-19 for this.

A somewhat cleaner Property in Java SE and a simple Converter that functions like the existing PropertyEditor but doesn’t have the AWT paint related baggage would be a small but still welcome improvement as well.

For programmatic access to configuration files it would be really helpful if the root of an EAR archive or at least its META-INF folder could be put on the classpath. For a WAR, an archive type where the web resources where in a sub folder (e.g. web-resources) and the root or WEB-INF was on the classpath would be great as well, although such a big change is rather unlikely to happen anytime soon.

Specifically for data sources it would also be a sure improvement if the Java EE spec mandated that the data sources referenced by @DataSourceDefinition can be loaded from the WAR/EAR archive. Currently it doesn’t do that. Most vendors support it anyway, but GlassFish doesn’t.

As mentioned above, it would be great if Java EE had support for switching configuration, but for now the switchable data source as presented in this article can be used as an alternative.


Arjan Tijms

One comment to “Switching between data sources when using @DataSourceDefinition”

  1. Miguel Enriquez says:

    I think it would be easier if Java EE had a notion of profiles. Something like Spring 3.1

Type your comment below:

best counter