Redis Pipelines with Python

Redis is an in memory datastore mostly used as a cache. Clients can send commands to server using TCP protocol and get the response back. So, usually a request works like this.

• Client sends a request to server and waits for the response to be sent back.
• Server processes the command and writes the response to the socket for client to read.

Sometimes, in application flow we have to retrieve multiple keys at once. In this case, for each request there will be a network overhead for the round trip between server and client. We can reduce this network overhead by sending commands to the Redis server in a batched manner and then process all the responses at once. For eg., using Python we can use pipeline like this:

• Connect to Redis server first. You can change the host parameter to the IP address/DNS of your Redis server.

import redis
r = redis.Redis(host='localhost', port=6379, db=0)

• Create a pipeline instance using the pipeline method

pipe = r.pipeline()

• Send the commands to be executed in batch

pipe.set("key1", "value1")
pipe.get("key2")
pipe.hgetall("key3")
pipe.set("key4","value4")

• Call pipe.execute to execute all the buffered commands

responses = pipe.execute()

• To iterate through server responses

for response in responses:
    print(response)

This way you can take advantage of pipeline capacity provided by Redis and save significant time on network round trips.