Musings on Tech

In this age of web scale architecture, Golang has become the language of choice for many developers to implement high throughput micro services. One of the key components of running and maintaining these services is to be able to measure the performance. Prometheus is a time series based database which has become very popular for monitoring micro services. In this blog, we will see how to implement monitoring for your Go micro service using prometheus. We will be using the official Prometheus library github.com/prometheus/client_golang/prometheus/promhttp to expose the go metrics. You can use Promhttp library’s HTTP handler as the handler function to expose the metrics. package main import ( "github.com/gorilla/mux" "github.com/prometheus/client_golang/prometheus/promhttp" "net/http" ) func main() { router := mux.NewRouter() router.Handle( "/metrics" , promhttp.Handler()) http.ListenAndServe( ":8080" , router ) }...

Redis is an in memory datastore mostly used as a cache. Clients can send commands to server using TCP protocol and get the response back. So, usually a request works like this. Client sends a request to server and waits for the response to be sent back. Server processes the command and writes the response to the socket for client to read. Sometimes, in application flow we have to process multiple keys at once. In this case, for each request there will be a network overhead for the round trip between server and client. We can reduce this network overhead by sending commands to the Redis server in a batched manner and then process all the responses at once. This can be achieved using pipelines as well as transactions. Pipelining in Redis is when you send all the commands to the server at once and receive all the replies in one go. It does not provide any guarantees like no other commands will be processed between your pipelined commands. Transactions in Redis are meant to be ...

Caching is a very useful tool to increase the performance of your service. Done properly, caching can reduce significant load on your underlying databases. There are several different patterns like Read Through Cache, Write through Cache etc. Each of them has its own benefits and pitfalls that need to be kept in mind. Read Through Cache : Read through cache is simplest caching pattern. In read-through caching, data is always served by the cache. If data is not present in cache, the cache hits the underlying database and serves the data to the requester. Read Through Caching      2. Read Aside Caching : Read aside caching is similar to Read Through Cache. The key difference is           that, the client tries to get the data from cache first. If its a cache miss, then the client requests the                underlying database and loads the data into Cache. Read Aside Caching ...

Monitoring is one of the most important parts of any production setup. Good monitoring is critical to detect any issues before they impact your systems and eventually the users. Prometheus is an open source time series data store. It works on a pull based model where you have to expose an endpoint from where Prometheus can pull. We can use prometheus_rabbitmq_exporter plugin to expose /api/metrics endpoint in the context on RabbitMQ management API. For installation and setup instructions, we can follow this article on RabbitMQ website. Once the setup is done, read this article to learn more about which metrics you should monitor and how to do that using Prometheus. First we will look at system metrics: Node Load Average: This metric indicates the Average load on CPU. It should be less than the number of cores on the node CPU. You should setup alerts if this goes higher than number of CPU cores available. Query to setup the graph for this is: node_load1{instance=~”rabbit-clu...

RabbitMQ stores contents of a queue on a single node by default. We can optionally mirror these contents across different nodes. Each mirrored queue has one master and one or more mirrors. Any operation that takes place on the queue like adding a message, consuming a message etc. happens on the master node first. Since all the messages are replicated across all mirrors and all the operations are also propagated to them, adding mirrors does not reduce the load on the queue. However, it provides us with high availability. In case, the node on which master for your mirror is located goes down, one of the mirrors can take over without impacting the availability of the queue. We can setup various different mirroring policies to suit our availability requirements. RabbitMQ provides 3 policies out of the box: Exactly: Using this you can specify exact number of replicas(master + mirrors) for the queue in the cluster. All: Queue will be replicated across all nodes on cluster. Use this ...

I have worked with RabbitMQ a long time. Its easy to make various mistakes which prevent you from getting the best out of it. Each application has its own performance requirements, like some of them require really high throughput while others process batch jobs where its fine if there are some delays.  RabbitMQ queues are single threaded: Since RabbitMQ queues are single threaded, one queue can handle max 50k messages/sec (also depends on available RAM and swap size). Don’t let your queues grow too long: Since RabbitMQ stores all unconsumed messages in RAM, if your queues start to grow too much it will lead to memory and performance issues on broker. RabbitMQ will start throttling your publishers using Flow Control . If your memory consumption breaches a preset threshold, broker will stop accepting any new messages altogether. Use lazy queues for batch publishing jobs : Lazy queues automatically store all the messages to the disk. They are loaded into the memory only when re...

Redis is an in memory datastore mostly used as a cache. Clients can send commands to server using TCP protocol and get the response back. So, usually a request works like this. Client sends a request to server and waits for the response to be sent back. Server processes the command and writes the response to the socket for client to read. Sometimes, in application flow we have to retrieve multiple keys at once. In this case, for each request there will be a network overhead for the round trip between server and client. We can reduce this network overhead by sending commands to the Redis server in a batched manner and then process all the responses at once. For eg., using Python we can use pipeline like this: Connect to Redis server first. You can change the host parameter to the IP address/DNS of your Redis server. import redis r = redis.Redis(host='localhost', port=6379, db=0) Create a pipeline instance using the pipeline method pipe = r.pipeline() Send the c...