Fastest way to warm an AWS Redis Elasticache with JSON data

After creating an a Elasticache instance for Redis and loading it with data dynamically, it became apparent that it would be useful to warm the cache with a large dataset to further improve performance.  Redis recommends using their Luke protocol to accomplish this by creating a text file with the data and then use the redis-cli to import that into Redis directly. This approach is significantly faster than doing it programmatically. Here I’ll elaborate on the steps I took to do that.

The first step is to create a text file with the data you want to import.  The basic structure is:

set [key] "[JSON string]"

set [key] "[JSON string]"

set [key] "[JSON string]"
...

However, the devil is in the details. First, you must escape all the quotes in your JSON string. Also, you must be sure to have a carriage-return/line-feed after each line or your data won’t import. This is required even after the last line in your file.

Once you have your file, you then need to create (or use an existing) EC2 instance to call your redis instance and import the data.  On your EC2 instance you first need to install gcc to build the redis-cli and then make the redis-cli code.  Here are the steps:

sudo yum install gcc

wget http://download.redis.io/redis-stable.tar.gz
tar xvzf redis-stable.tar.gz
cd redis-stable
make

Once you do this, you can then pipe your text file into your Redis instance using the following command:

redis-stable $ cat import.txt | ./src/redis-cli \
-h [elasticache hostname from replication group] -p 6379 --pipe

That’s it! If you want to schedule this on interval, you can simply create a cron job to refresh your cache as needed or use an event driven architecture to update the cache based on changes from the source system.