Apache Kafka is designed to scale up to handle trillions of messages per day.
Kafka is well known for it’s large scale deployments
but it has an efficient implementation and can be configured to run surprisingly well on
systems with limited resources for low throughput use cases as well. Here
are the key settings you’ll need to change to get Kafka running on your low
end VPS or Raspberry Pi:
Java heap settings
The default java heap sizes for zookeeper and kafka are 512Mb and 1Gb respectively. With
appropriate configuration properties, you can take these down to as low as 4Mb and 32Mb:
KAFKA_HEAP_OPTS="-Xmx4M -Xms4M" bin/zookeeper-server-start etc/kafka/zookeeper.properties
KAFKA_HEAP_OPTS="-Xmx32M -Xms32M" bin/kafka-server-start etc/kafka/server.properties
Kafka config settings
The most important configuration setting to tweak is
For Kafka > v0.9.0, this is set to 128Mb by default, pre-allocating 128Mb of Java heap
space. You can reduce this to as low as ~11Mb. You could also disable log-compaction altogether
false but you won’t want to do this if you’re using
Kafka to keep track of consumer offsets as these are stored in compacted topics.
Finally, there are other parameters that could potentially be tuned down as well, for
background.threads (threads have an associated memory cost), however I’ve never
bothered. I’ve successfully deployed and run Kafka as part of a hobby project on a low-end
VPS with just the above two changes.