application-test.yaml seems to be ignored when test makes second call to DatabaseConfig.loadFromProperties

[simontankersley]

Background

Often when testing in Spring, multiple application contexts can be created and destroyed. When using programatic configuration of ebeans this could create a situation where multiple calls are made to DatabaseConfig.loadFromProperties method.

Expected behavior

It doesn't matter how many times the tests call DatabaseConfig.loadFromProperties, the test configuration is found and used each time.

Actual behavior

Only the first call to DatabaseConfig.loadFromProperties succeeds. When trying to use a Database created from the DatabaseConfig that was created by a second call to DatabaseConfig.loadFromProperties we get exception:

    io.ebean.datasource.DataSourceConfigurationException: DataSource user is not set? url is [null]
        at app//io.ebean.datasource.pool.ConnectionPool.<init>(ConnectionPool.java:121)
        at app//io.ebean.datasource.pool.ConnectionPoolFactory.createPool(ConnectionPoolFactory.java:14)
        at app//io.ebean.datasource.DataSourceFactory.create(DataSourceFactory.java:25)
        at app//io.ebeaninternal.server.core.InitDataSource.create(InitDataSource.java:113)
        at app//io.ebeaninternal.server.core.InitDataSource.createFromConfig(InitDataSource.java:104)
        at app//io.ebeaninternal.server.core.InitDataSource.initDataSource(InitDataSource.java:44)
        at app//io.ebeaninternal.server.core.InitDataSource.initialise(InitDataSource.java:33)
        at app//io.ebeaninternal.server.core.InitDataSource.init(InitDataSource.java:24)
        at app//io.ebeaninternal.server.core.DefaultContainer.setDataSource(DefaultContainer.java:220)
        at app//io.ebeaninternal.server.core.DefaultContainer.createServer(DefaultContainer.java:86)
        at app//io.ebeaninternal.server.core.DefaultContainer.createServer(DefaultContainer.java:29)
        at app//io.ebean.DatabaseFactory.createInternal(DatabaseFactory.java:136)
        at app//io.ebean.DatabaseFactory.create(DatabaseFactory.java:85)
        at app//org.example.PetTest.testDouble(PetTest.java:44)

It appears as though the test configuration isn't found.

Steps to reproduce

I've created a sample project with code at https://github.com/simontankersley/programatic-ebean-test

This code fails

        // simulate using a custom object mapper
        ObjectMapper mapper = new ObjectMapper();

        // simulate creating a database configuration twice
        // this often happens when using spring tests where several
        // contexts can be created and destroyed - causing multiple
        // calls to create the database configuration from properties
        DatabaseConfig dbConfig = new DatabaseConfig();
        dbConfig.loadFromProperties();
        dbConfig.setObjectMapper(mapper);
        Database db = DatabaseFactory.create(dbConfig);

        dbConfig = new DatabaseConfig();
        dbConfig.loadFromProperties();
        dbConfig.setObjectMapper(mapper);
        db = DatabaseFactory.create(dbConfig);
        db.save(new Pet());

If you want to see a base line vs a failing test see below

Success when running a test with a single call to DatabaseConfig.loadFromProperties

git clone git@github.com:simontankersley/programatic-ebean-test.git
cd programatic-ebean-test
./gradlew test --tests '*testSingle'

gives

> Task :test

PetTest > testSingle() PASSED

Failure when running a test with a multiple calls to DatabaseConfig.loadFromProperties

./gradlew test --tests '*testDouble'

gives

> Task :test

PetTest > testDouble() FAILED
    io.ebean.datasource.DataSourceConfigurationException: DataSource user is not set? url is [null]
        at app//io.ebean.datasource.pool.ConnectionPool.<init>(ConnectionPool.java:121)
        at app//io.ebean.datasource.pool.ConnectionPoolFactory.createPool(ConnectionPoolFactory.java:14)
        at app//io.ebean.datasource.DataSourceFactory.create(DataSourceFactory.java:25)
        at app//io.ebeaninternal.server.core.InitDataSource.create(InitDataSource.java:113)
        at app//io.ebeaninternal.server.core.InitDataSource.createFromConfig(InitDataSource.java:104)
        at app//io.ebeaninternal.server.core.InitDataSource.initDataSource(InitDataSource.java:44)
        at app//io.ebeaninternal.server.core.InitDataSource.initialise(InitDataSource.java:33)
        at app//io.ebeaninternal.server.core.InitDataSource.init(InitDataSource.java:24)
        at app//io.ebeaninternal.server.core.DefaultContainer.setDataSource(DefaultContainer.java:220)
        at app//io.ebeaninternal.server.core.DefaultContainer.createServer(DefaultContainer.java:86)
        at app//io.ebeaninternal.server.core.DefaultContainer.createServer(DefaultContainer.java:29)
        at app//io.ebean.DatabaseFactory.createInternal(DatabaseFactory.java:136)
        at app//io.ebean.DatabaseFactory.create(DatabaseFactory.java:85)
        at app//org.example.PetTest.testDouble(PetTest.java:44)

[rbygrave]

Thanks for the example project with failing test. I have cloned and reproduced the issue.

Some background:
There are some considerations going on that we need to navigate.

• We are looking to start the appropriate docker containers once and set them up once (create the database, create the user/role etc)
• We are similarly looking to run the DDL once to drop and recreate the database schema

The current issue as I see it is that the "Run Once Marker" is covering those things that we do really only want to run once but it is also covering the function that does the "set up the DataSourceConfig given the test docker container" we want to run our tests against.

So I expect the fix for this issue to be along the lines of:

• moving that "Run Once Marker" to cover only the "setup the test docker containers"
• always executing the function that does the "set up the DataSourceConfig given the test docker container"
• adjust the DDL configuration such that we use DDL mode none for subsequent executions (only run the DDL once)

Q: Is there a workaround in the meantime?

Probably look to use explicit url, username, password in application-test.yaml and programmatically set the appropriate ddl mode (which assumes we can determine when the first Ebean DatabaseFactory.create() is happening and setDdlRun(true) only for that creation).

[rbygrave]

Maybe workaround

A possible workaround is to explicitly specify the dataSource options in application-test.yaml like:

ebean:
  test:
    platform: postgres
    dbName: test
    ddlMode: dropCreate


datasource:
  db:
    username: test
    password: test
    url: jdbc:postgresql://localhost:6432/test

Noting that the tests now pass but we might not be getting what we want in terms of DDL being run. If we add logging of the test output (and adjust a little so we can see when the tests create the Database instances in the logs) we see:

$ ./gradlew clean test

> Task :compileJava
Note: Ebean APT generated 1 query beans, loaded 0 others - META-INF/ebean-generated-info.mf entity-packages: [org.example.model]

> Task :test

PetTest > atestSingle() STANDARD_OUT
    23:23:50.577 [Test worker] TRACE io.avaje.config - load from [resource:application-test.yaml]
    23:23:50.578 [Test worker] INFO  io.avaje.config - Loaded properties from [resource:application-test.yaml] 
    23:23:50.579 [Test worker] DEBUG io.ebean.test - automatic testing config - with ebean.test.platform:postgres name:db environmentDb:null
    23:23:50.583 [ForkJoinPool.commonPool-worker-2] INFO  io.ebean.test - Using jdbc settings - username:test url:jdbc:postgresql://localhost:6432/test driver:org.postgresql.Driver
    23:23:50.586 [ForkJoinPool.commonPool-worker-2] DEBUG io.ebean.test - Docker properties: {postgres.password=test, postgres.driver=org.postgresql.Driver, postgres.extensions=hstore,pgcrypto, postgres.username=test, postgres.version=14, postgres.url=jdbc:postgresql://localhost:6432/test, postgres.port=6432, postgres.dbName=test}
    23:23:50.718 [ForkJoinPool.commonPool-worker-2] TRACE io.ebean.test.containers - sqlRun: select 1 from pg_database where datname = 'test'
    23:23:50.726 [ForkJoinPool.commonPool-worker-2] DEBUG io.ebean.test.containers - Container ut_postgres ready with host:localhost port:6432
    23:23:50.729 [Test worker] DEBUG io.ebean.test - for testing - using FixedEncryptKeyManager() keyVal:simple0123456789
    23:23:50.729 [Test worker] INFO  io.ebean.test - For testing purposes a current user provider has been configured. Use io.ebean.test.UserContext to set current user in tests.
    23:23:50.729 [Test worker] INFO  org.example.PetTest - testSingle ------------------------------
    23:23:50.729 [Test worker] INFO  io.ebean - ebean version: 13.15.0
    23:23:50.777 [Test worker] INFO  io.ebean.datasource - DataSource [db] autoCommit[false] transIsolation[READ_COMMITTED] min[2] max[200] in[40ms]
    23:23:50.908 [Test worker] INFO  io.ebean.DDL - Executing db-drop-all.sql - 1 statements, autoCommit:true
    23:23:50.909 [Test worker] DEBUG io.ebean.DDL - executing 1 of 1 drop table if exists pet cascade
    23:23:50.916 [Test worker] INFO  io.ebean.DDL - Executing db-create-all.sql - 1 statements, autoCommit:false
    23:23:50.917 [Test worker] DEBUG io.ebean.DDL - executing 1 of 1 create table pet (   id                            uuid not null,   constraint p...
    23:23:50.924 [Test worker] INFO  io.ebean.core - Started database[db] platform[POSTGRES] in 192ms
    23:23:50.927 [Test worker] DEBUG io.ebean.SQL - txn[1001] insert into pet (id) values (?); -- bind(ca527473-577c-418c-9028-8e667b58e4c3)
    23:23:50.931 [Test worker] DEBUG io.ebean.TXN - txn[1001] Commit

PetTest > testTDouble() STANDARD_OUT
    23:23:50.934 [Test worker] DEBUG io.ebean.test - automatic testing config - with ebean.test.platform:postgres name:db environmentDb:null
    23:23:50.934 [Test worker] DEBUG io.ebean.test - for testing - using FixedEncryptKeyManager() keyVal:simple0123456789
    23:23:50.934 [Test worker] INFO  io.ebean.test - For testing purposes a current user provider has been configured. Use io.ebean.test.UserContext to set current user in tests.
    23:23:50.934 [Test worker] INFO  org.example.PetTest - testDouble 1 ------------------------------
    23:23:50.967 [Test worker] INFO  io.ebean.datasource - DataSource [db] autoCommit[false] transIsolation[READ_COMMITTED] min[2] max[200] in[32ms]
    23:23:50.972 [Test worker] INFO  io.ebean.core - Started database[db] platform[POSTGRES] in 37ms
    23:23:50.972 [Test worker] DEBUG io.ebean.test - automatic testing config - with ebean.test.platform:postgres name:db environmentDb:null
    23:23:50.972 [Test worker] DEBUG io.ebean.test - for testing - using FixedEncryptKeyManager() keyVal:simple0123456789
    23:23:50.972 [Test worker] INFO  io.ebean.test - For testing purposes a current user provider has been configured. Use io.ebean.test.UserContext to set current user in tests.
    23:23:50.972 [Test worker] INFO  org.example.PetTest - testDouble 2 ------------------------------
    23:23:50.998 [Test worker] INFO  io.ebean.datasource - DataSource [db] autoCommit[false] transIsolation[READ_COMMITTED] min[2] max[200] in[26ms]
    23:23:51.002 [Test worker] INFO  io.ebean.DDL - Executing db-drop-all.sql - 1 statements, autoCommit:true
    23:23:51.002 [Test worker] DEBUG io.ebean.DDL - executing 1 of 1 drop table if exists pet cascade
    23:23:51.005 [Test worker] INFO  io.ebean.DDL - Executing db-create-all.sql - 1 statements, autoCommit:false
    23:23:51.005 [Test worker] DEBUG io.ebean.DDL - executing 1 of 1 create table pet (   id                            uuid not null,   constraint p...
    23:23:51.008 [Test worker] INFO  io.ebean.core - Started database[db] platform[POSTGRES] in 36ms
    23:23:51.009 [Test worker] DEBUG io.ebean.SQL - txn[1001] insert into pet (id) values (?); -- bind(02036faf-fc6c-4f5c-96f2-fac25f06e1d3)
    23:23:51.010 [Test worker] DEBUG io.ebean.TXN - txn[1001] Commit
23:23:51.016 [EbeanHook] INFO  io.ebean.datasource - DataSource [db] shutdown min[2] max[200] free[2] busy[0] waiting[0] highWaterMark[1] waitCount[0] hitCount[4]  psc[hit:0 miss:4 put:4 rem:0]
23:23:51.016 [EbeanHook] INFO  io.ebean.datasource - DataSource [db] shutdown min[2] max[200] free[2] busy[0] waiting[0] highWaterMark[1] waitCount[0] hitCount[2]  psc[hit:0 miss:1 put:1 rem:0]
23:23:51.017 [EbeanHook] INFO  io.ebean.datasource - DataSource [db] shutdown min[2] max[200] free[2] busy[0] waiting[0] highWaterMark[1] waitCount[0] hitCount[4]  psc[hit:0 miss:4 put:4 rem:0]

BUILD SUCCESSFUL in 1s
7 actionable tasks: 7 executed

... and we can see the ddl being executed multiple times. This might be ok but generally it is not good when we have a bigger schema (ddl takes time) or when we desire to run tests in parallel. Generally the desire is to drop and create the db schema once for all tests.