Skip to main content

RabbitMQ best practices

I have worked with RabbitMQ a long time. Its easy to make various mistakes which prevent you from getting the best out of it. Each application has its own performance requirements, like some of them require really high throughput while others process batch jobs where its fine if there are some delays. 

  • RabbitMQ queues are single threaded: Since RabbitMQ queues are single threaded, one queue can handle max 50k messages/sec (also depends on available RAM and swap size).
  • Don’t let your queues grow too long: Since RabbitMQ stores all unconsumed messages in RAM, if your queues start to grow too much it will lead to memory and performance issues on broker. RabbitMQ will start throttling your publishers using Flow Control. If your memory consumption breaches a preset threshold, broker will stop accepting any new messages altogether.
  • Use lazy queues for batch publishing jobs: Lazy queues automatically store all the messages to the disk. They are loaded into the memory only when requested by the consumer. This helps with the memory consumption issues in broker. You can use this option, if your consumers are fine with somewhat higher latencies.
  • Set TTL/max-length for all the queues to limit size: This will help you ensure that your queues don’t grow beyond a certain size and cause memory issues on the broker.
  • Limit Number of queues: Since RabbitMQ management interface tracks and calculates several metrics for all the queues on the cluster, having too many queues might cause performance issues.
  • Always have separate connections for Publishers and Consumers: RabbitMQ applies flow control on publishers if consumers can’t keep up with them, which limits the bandwidth of connection. If your consumers and publishers share the same connection, then your consumers will also be throttled by RabbitMQ, which might lead to a cascading effect bringing down the broker.
  • Persistent messages and Durable queues: If you want your queues and messages to survive broker restarts and crashes, then declare your queues as durable and set delivery mode to persistent for messages. However, performance would be impacted by persistent messages as they have to be always stored in the disk.
  • Prefetch count: Prefetch count determines how many messages will be delivered to the consumer at any point of time before receiving acknowledgement. Having too small of a prefetch count would hurt performance, as broker will be waiting for the acknowledgement most of the time. If you have a very high prefetch count, it might lead to all messages being delivered to a single consumer, resulting in other consumers sitting idle. You need to figure out the correct prefetch value for your use case based on number of consumers, network latency, consumer latency.

Comments

Post a Comment

Popular posts from this blog

Monitoring Spring Boot API response times with Prometheus

Prometheus is a very useful tool for monitoring web applications. In this blog post, we will see how to use it to monitor Spring Boot API response times. You have to include following dependencies in your build.gradle file: compile group: 'io.prometheus', name: 'simpleclient_hotspot', version: '0.0.26' compile group: 'io.prometheus', name: 'simpleclient_servlet', version: '0.0.26' compile group: 'io.prometheus', name: 'simpleclient', version: '0.0.26' Now you will have to expose a Rest Endpoint, so that Prometheus can collect the metrics by scraping it at regular intervals. To do that, you would have to include these Java configuration classes: @Configuration @ConditionalOnClass(CollectorRegistry.class) public class Config { private static final CollectorRegistry metricRegistry = CollectorRegistry. defaultRegistry ; @Bean ServletRegistrationBean registerPrometheusExporterServlet() { retu...
1 comment
Read more

Monitoring Go micro services using Prometheus

In this age of web scale architecture, Golang has become the language of choice for many developers to implement high throughput micro services. One of the key components of running and maintaining these services is to be able to measure the performance. Prometheus is a time series based database which has become very popular for monitoring micro services. In this blog, we will see how to implement monitoring for your Go micro service using prometheus. We will be using the official Prometheus library github.com/prometheus/client_golang/prometheus/promhttp to expose the go metrics. You can use Promhttp library’s HTTP handler as the handler function to expose the metrics. package main import ( "github.com/gorilla/mux" "github.com/prometheus/client_golang/prometheus/promhttp" "net/http" ) func main() { router := mux.NewRouter() router.Handle( "/metrics" , promhttp.Handler()) http.ListenAndServe( ":8080" , router ) }...
Post a Comment
Read more

Monitoring RabbitMQ with Prometheus and Grafana

Monitoring is one of the most important parts of any production setup. Good monitoring is critical to detect any issues before they impact your systems and eventually the users. Prometheus is an open source time series data store. It works on a pull based model where you have to expose an endpoint from where Prometheus can pull. We can use prometheus_rabbitmq_exporter plugin to expose /api/metrics endpoint in the context on RabbitMQ management API. For installation and setup instructions, we can follow this article on RabbitMQ website. Once the setup is done, read this article to learn more about which metrics you should monitor and how to do that using Prometheus. First we will look at system metrics: Node Load Average: This metric indicates the Average load on CPU. It should be less than the number of cores on the node CPU. You should setup alerts if this goes higher than number of CPU cores available. Query to setup the graph for this is: node_load1{instance=~”rabbit-clu...
Post a Comment
Read more