When working with multiple databases, code can become intertangled with records spanning across the systems. Seperating database domain reponsibilities away from each other is challenging but a good end goal. Sometimes this isn’t possible which in the case of testing clean-up can make for some tricky configuration details.

Setting up multiple databases

First you’ll want to create multiple database.yml files within your config/ directory. These should be named appropriately to your external databases. For example:

Unless you’re running Rails 6.0, multiple database support will be a manual process for schema and migration loading. Running SQL commands in the console isn’t sustainable in the long run so I recommend checking out Robert Ostinelli’s post which has a section on setting up Rake tasks for each database.

Back to your database.yml files, they should each follow the best practices set forth from Rails Guides.

Once created, we can use namespaces to segment models based on database. This allows for a discrete namespace to database mapping. I’ve written a brief example below of what this would look like for the statistics_database.yml configuration within the Statistics namespace.

module Statistics
  # Directly loads YAML configuration from filesystem by current Rails environment
  DATABASE = YAML.load_file(File.join(Rails.root, "config", "statistics_database.yml"))[Rails.env]

  class ApplicationRecord < ActiveRecord::Base

    # Ensure that this class:
    # * cannot be instantiated
    # * does not represent an available database table
    self.abstract_class = true

    # Connect to the secondary database
    # NOTE: We use this in the base class and have new models inherit from it
    #       to limit how many database connections are active at one time.
    establish_connection DATABASE
  end
end

ActiveRecord::Base.establish_connection or just establish_connection since we’re inside the ActiveRecord::Base scope, allows for direct connection to a new database based on the supplied configuration.

YAML.load_file(File.join(Rails.root, "config", "statistics_database.yml"))[Rails.env] really just says: Build a filepath, supply it to YAML.load_file, and extract out environment specific configuration values. Here’s an example of it in the console:

YAML.load_file(File.join(Rails.root, "config", "statistics_database.yml"))[Rails.env]
=> {"adapter"=>"postgresql", "host"=>"localhost", "username"=>"postgres", "encoding"=>"unicode", "database"=>"statistics_development", "pool"=>5}

Finally self.abstract_class ensures we never directly use this class but instead have other classes inherit from it. Additionally, it ensures that it doesn’t map to a database table.

Now that we’ve setup a second database connection what about testing multiple databases?

Database Cleaner Configuration

Database Cleaner is a fantastic gem for making test cleanup and teardown simple. We’re going to work with a simple configuration which looks like the following:

RSpec.configure do |config|
  config.before(:suite) do
    DatabaseCleaner.clean_with(:truncation)
  end

  config.before(:each) do
    DatabaseCleaner.strategy = :transaction
    DatabaseCleaner.start
  end

  config.append_after(:each) do
    DatabaseCleaner.clean
  end
end

Now presently this will only connect to the default database for the application. Not a great situation if we want to ensure data remains cleaned up before/after tests. Similar to the ApplicationRecord abstract model above, we’ll need to connect to the database using establish_connection. I’ve made the assumption that you’ve followed the first part of the guide settings up constants in the respective namespaces: Statistics::DATABASE, Events::DATABASE, Core::DATABASE.

RSpec.configure do |config|
  config.before(:suite) do
    ActiveRecord::Base.establish_connection Statistics::DATABASE
    DatabaseCleaner.clean_with(:truncation)
    ActiveRecord::Base.establish_connection Events::DATABASE
    DatabaseCleaner.clean_with(:truncation)
    ActiveRecord::Base.establish_connection Core::DATABASE
    DatabaseCleaner.clean_with(:truncation)
  end

  config.before(:each) do
    ActiveRecord::Base.establish_connection Statistics::DATABASE
    DatabaseCleaner.strategy = :transaction
    DatabaseCleaner.start
    ActiveRecord::Base.establish_connection Events::DATABASE
    DatabaseCleaner.strategy = :transaction
    DatabaseCleaner.start
    ActiveRecord::Base.establish_connection Core::DATABASE
    DatabaseCleaner.strategy = :transaction
    DatabaseCleaner.start
  end

  config.append_after(:each) do
    ActiveRecord::Base.establish_connection Statistics::DATABASE
    DatabaseCleaner.clean
    ActiveRecord::Base.establish_connection Events::DATABASE
    DatabaseCleaner.clean
    ActiveRecord::Base.establish_connection Core::DATABASE
    DatabaseCleaner.clean
  end
end

Verbose no? We have to specify the Database Cleaner configuration for each database connection. We’ll make this better in a second.

One thing that isn’t clear from above is that the database connections above are order dependent. Meaning the last establish_connection call should be to your core application database. I’m not completely certain why this is but it seems like without it the database won’t be setup properly.

Improving Database Cleaner

So the above is really wordy. Let’s imagine that you have 10 external databases to connect to. That is a lot of repetition. Luckily we can lean into the yield keyword here to make it simple. Additionally, we can enforce the order of database connections.

EXTERNAL_DATABASE_CONNECTIONS = [
  Statistics::DATABASE,
  Events::DATABASE
]

RSpec.configure do |config|
  config.before(:suite) do
    connect_to_available_databases do
      DatabaseCleaner.clean_with(:truncation)
    end
  end

  config.before(:each) do
    connect_to_available_databases do
      DatabaseCleaner.strategy = :transaction
      DatabaseCleaner.start
    end
  end

  config.append_after(:each) do
    connect_to_available_databases do
      DatabaseCleaner.clean
    end
  end

  def connect_to_available_databases
    EXTERNAL_DATABASE_CONNECTIONS.push(Core::Database).each do |db_connection|
      ActiveRecord::Base.establish_connection db_connection
      yield
    end
  end
end

From the above we’ve defined a new array to contain all available external database connections (EXTERNAL_DATABASE_CONNECTIONS).

Next, we have a new method #connect_to_available_databases which takes the array and pushes the core (or default application) database onto the end of the array to ensure it it loaded last.

We then establish_connection directly to ActiveRecord::Base like we’ve been doing.

Finally, the yield keyword allows us to use a block syntax and pass in the Database Cleaner configuration to be executed against the currently connected database. Looking at the current code again, the following:

connect_to_available_databases do
  DatabaseCleaner.clean
end

will translate to

ActiveRecord::Base.establish_connection Statistics::DATABASE
DatabaseCleaner.clean
ActiveRecord::Base.establish_connection Events::DATABASE
DatabaseCleaner.clean
ActiveRecord::Base.establish_connection Core::DATABASE
DatabaseCleaner.clean

With this final change Database Cleaner is configured to run for multiple databases. Now go write some tests!

« Previous Post Next Post »

Join the conversation

comments powered by Disqus