Magic configuration in Node.js using module preloading

Configuration management - the one thing that you know should be done right, yet is so easy to just ignore and put off until later.

Here is an easy way to do application configuration for cloud based node apps.

Lazy config management

Ideally an app's configuration would:

1. Not live in source control
2. Be easy to set up in any environment
3. Get set before any code has a chance to run
4. Not have to make special exceptions for production secrets

These are a few quick ways to deal with app configuration that get fragile because they don't hit these three points.

// keeping a config file around seems nice...
// but how do you get it to a new box or container w/o git?
// also, how do you guarantee that the config is loaded before its needed
require('./config.json');

// dead simple way to set our applications port
// what if other parts of the app need this too?
const env = process.env.PORT || 3000;

// using a little application logic seems tempting
// will your test databases always be the same (even on your CI)?
process.env.DATABASE_URL = process.env.NODE_ENV === 'test' ? 'test-url' : process.env.PROD_DATABASE_URL;

Config management only gets harder as you application grows too. How do you have utility scripts, tests, or secondary processes all get configured the same way?

Magic configs

Enter two tools: dotenv and module preloading.

dotenv is a simple configuration loader. Just add a .env file to your root directory that looks like:

#.env
DATABASE_URL=postgres://....

Call dotenv and your process.env object is now populated with the values from your environment file.

// app.js
require('dotenv').config();

const pg = require('pg');
const client = new pg.Client(process.env.DATABASE_URL);

So how do we make this environment specific? With one line we can have .env.development and .env.test files that bootstrap our code in entirely different ways.^[1]

// still in app.js
const node_env = process.env.NODE_ENV || 'development';
require('dotenv').config({ path: `.env.${node_env}` });

Now NODE_ENV=development node app.js and NODE_ENV=test node app.js can run the same code in the same way while pointing to different resources.

The final bit to make this magic is module preloading. Node provides a -r, --require flag that will require a file before anything else is loaded, which makes it perfectly suited for config loading.

// config.js
const node_env = process.env.NODE_ENV || 'development';
require('dotenv').config({ path: `.env.${node_env}` });

// bam! app.js has all of its configurations set without having to load a thing
node -r ./config.js app.js

Integrating into your tool chain

The final step happens when all these commands are wrapped up behind our npm scripts. While each command is a little ugly to look at, we now have start and test commands that will ensure their respective databases are up to date before each execution.

"start": "node -r ./config.js app.js",
"prestart": "npm run migrate",
"migrate": "node -r ./config.js node_modules/db-migrate/bin/db-migrate up",

"test": "NODE_ENV=test ./node_modules/.bin/istanbul cover node_modules/.bin/_mocha -- --recursive test",
"pretest": "npm run migrate-test",
"migrate-test": "NODE_ENV=test node -r ./config.js node_modules/db-migrate/bin/db-migrate up"

And thats all the magic - multi environment configuration management controlled, exposed to an app via environment variables, controlled with a single file, and executed with our favorite npm start and npm test.

Note on working with istanbul & mocha

Using istanbul & mocha for testing does not allow for node's preload magic, but mocha has a --require option that can be set in test/mocha.opts that will accomplish the same thing

# test/mocha.opts
--require ./config.js

Production

Lets be clear on one point - there should never be a .env.production.

While this post was all about bootstrapping environment variables for you application, this is because we are trying to emulate our production environment as closely as possible.

The reality is that when your app is deployed this environment variables should already be set in one way or another. Variables such as DATABASE_URL are already provided to your container on heroku, and AWS provides similar options for RDS on elasticbeanstalk. By keeping your production secrets out of your container and instead in the management infrastructure you can be assured of safe and efficient configuration deployment.

Happy building!

---

1. Its worth noting that dotenv explicitly advises to not do this in their README.md. The rational is that in doing so it is possible to have .envs that are not orthogonal. In practical use I've found it frustrating to not have a way to not have a simple way to separate databases during local testing & development. A thread going a little deeper into this can be found here. OSSFTW. ↩︎