1, Use cases

1.RabbitMQ:

Suitable for scenarios with high requirements for ease of use and flexibility

• Asynchronous task processing：RabbitMQ provides a reliable messaging mechanism suitable for processing asynchronous tasks. For example, time-consuming tasks can be placed in a message queue and then processed asynchronously by consumers, improving the system's responsiveness and scalability.

• Decoupling system components: By using RabbitMQ as message middleware, different system components can be decoupled through messaging, creating a loosely coupled architecture and improving the system's maintainability and flexibility.

• Event-driven architecture: RabbitMQ's publish-subscribe model can be used to build an event-driven architecture by publishing events in the system as messages to the corresponding topics. Different consumers can subscribe to topics they are interested in and process them accordingly.

2.RocketMQ:

Suitable for large-scale data processing and high-throughput scenarios

• Distributed transactions: RocketMQ supports distributed transactional messages, making it suitable for distributed transaction scenarios involving multiple business systems. It ensures message consistency and reliability while providing high-throughput messaging.

• Real-time log processing: With its high throughput and low latency, RocketMQ can be used for real-time log processing, including log collection and analysis, log aggregation, and similar scenarios.

• Stream processing: RocketMQ supports a stream processing mode that passes generated data streams through message queues to stream processing frameworks such as Flink and Spark Streaming, enabling real-time data processing and analysis.

3.Kafka：

Suitable for high-throughput real-time data stream processing and stream processing scenarios

• Data pipelines and real-time data processing: Kafka's high throughput and persistent storage make it an ideal choice for building reliable data pipelines and real-time data processing systems to collect, transmit, and process streaming data.

• Log and event stream processing: Kafka’s partitioning and ordering guarantees make it an ideal solution for log and event stream processing, including application log collection, event sourcing, and business monitoring.

• Real-time stream analytics and machine learning: Kafka works with stream processing frameworks (such as Flink and Spark Streaming) to support real-time stream analytics and machine learning tasks, processing real-time data streams to obtain immediate insights and support decision-making.

2, Service architecture

• RabbitMQ

1. Channel: A Channel is a virtual connection between RabbitMQ and an application. By creating multiple channels over a physical connection, an application can perform messaging operations concurrently. Channels send and receive messages and perform related operations, such as declaring queues, defining exchanges and creating bindings. A channel can be viewed as a lightweight session that interacts with RabbitMQ through a single physical connection.

2. Exchange（交换机）：An exchange is the hub for receiving and forwarding messages. When messages are sent to RabbitMQ, they are routed through an exchange. Based on its type and binding rules, the exchange routes messages to one or more queues. Common exchange types include direct, topic, fanout, and headers.

3. Queue: A queue is a buffer RabbitMQ uses to store messages. When messages cannot be routed to consumers immediately, they are stored in a queue, waiting for consumers to retrieve and process them. Each queue has a unique name, and messages are delivered and consumed in FIFO (first in, first out) order.

4. Virtual Host: A virtual host is a logically isolated environment used to divide a RabbitMQ server into multiple independent sections. Each virtual host has its own exchanges, queues, bindings, and permission settings. Virtual hosts help isolate different applications or services and provide controls for security and resource management.

5. Broker: A Broker is an instance of the RabbitMQ message queue server, responsible for receiving, storing, and routing messages. It acts as an intermediary, delivering messages from producers to consumers. A RabbitMQ instance can contain multiple Brokers, each of which can host multiple virtual hosts and queues.

• RocketMQ

1. NameServer (naming server): NameServer is RocketMQ’s naming service component, used to manage and provide Broker routing information. It serves as a metadata hub, maintaining Broker registrations, Topic routing information, and more. Before sending and receiving messages, Producers and Consumers need to interact with NameServer to obtain the correct Broker information.

2. Controller: Controller is RocketMQ's controller component, responsible for coordinating and managing the operation of the entire RocketMQ cluster. It monitors Broker status changes, handles cluster scaling up and down, performs load balancing, and more. Controller is one of the core components of a RocketMQ cluster, ensuring reliable cluster operation and automated management.

3. Broker (message broker): A Broker is a message storage and processing node in RocketMQ. It receives messages from Producers and stores them on disk. It also handles message retrieval requests from Consumers and pushes messages to Consumers for consumption. A RocketMQ cluster can contain multiple Brokers, each responsible for storing some of the message data for Topics.

• KafkaMQ

1. Broker (message intermediary): A Broker is a node in a Kafka cluster responsible for storing and processing messages. Each Broker is an independent Kafka server instance. It can be a standalone physical server, a virtual machine, or a container. A Kafka cluster can contain multiple Brokers, which work together to provide highly available, high-throughput messaging.

2. Topic: A Topic is a logical category or subject for messages. It is the unit of message publication and subscription. Producers publish messages to specified Topics, while Consumers subscribe to Topics of interest to receive messages. A Topic can be viewed as a container for messages, used to categorize and group related messages.

3. Partition: A Topic can be divided into one or more partitions, each a subset of the Topic. A partition is the smallest unit of message storage and delivery. Each partition physically corresponds to an independent log file, and these files are distributed across different Brokers. Partitions enable Kafka to scale horizontally and process messages in parallel while providing higher throughput. Messages within each partition are stored and delivered in first-in, first-out order.

3, Persistence and Reliability:

• RabbitMQ: RabbitMQ uses message persistence, storing messages on disk to ensure their reliability. It supports multiple message acknowledgment mechanisms and transactions to ensure reliable message delivery.

• RocketMQ: RocketMQ offers strong persistence and reliability features, supporting synchronous disk flushing and asynchronous replication to provide highly reliable message delivery guarantees.

• Kafka: Kafka stores messages persistently, writing them to log files on disk. Its partitioning and replication mechanisms provide high reliability and persistent storage.

4, Throughput

• RabbitMQ: RabbitMQ generally has lower throughput and is suitable for small to medium-scale applications. RabbitMQ suits small to medium-scale applications and can typically handle message volumes in the tens to hundreds of thousands. It primarily focuses on reliable message delivery and message persistence; meeting high-throughput requirements may require optimization and tuning.

• RocketMQ: RocketMQ offers high throughput, with the capacity to process millions of messages. It excels in distributed transactions and large-scale messaging scenarios.

• Kafka: Kafka is a message queue system renowned for its high throughput, capable of handling millions of messages or more. Kafka is suited to high-throughput scenarios such as large-scale data processing, real-time stream processing, and log processing.

5、Response Time

6, Community and ecosystem

• RabbitMQ: RabbitMQ has a rich ecosystem of plugins and tools, supported by a broad developer community.

• RocketMQ: RocketMQ is widely used in China, with extensive support from Alibaba’s ecosystem and the community.

• Kafka: Kafka has an active open-source community and a broad ecosystem, and is widely adopted by many large companies.

• Github Star

• Github Contributors

Overall: rabbitmq<rocketmq<kafka

7, Design philosophy

• RabbitMQ: RabbitMQ is open-source messaging middleware based on AMQP (Advanced Message Queuing Protocol), emphasizing ease of use and flexibility, with support for multiple messaging patterns and reliable message delivery.

• RocketMQ: RocketMQ is an open-source distributed messaging middleware developed by Alibaba, originally designed to meet Alibaba's internal needs for massive-scale data processing. It features high throughput and low latency. In 2016, Alibaba donated RocketMQ to the Apache Software Foundation.

• Kafka: Kafka is a distributed stream processing platform developed by LinkedIn, primarily used for high-throughput processing of real-time data streams, storing and processing data persistently. Kafka became an Apache open-source project in 2011.

8、Data Model

• RabbitMQ: RabbitMQ uses a queue (Queue) model: messages are sent to a queue, and consumers receive messages from the queue and process them.

• RocketMQ: RocketMQ uses a Topic and Tag model. Messages are published to topics, and consumers can subscribe to and filter messages by topic and tag.

• Kafka: Kafka uses a publish-subscribe model. Messages are published to topics, and multiple consumers can subscribe to the same topic and consume messages independently.

9, Web Management Tools

• RabbitMQ

RabbitMQ Web Management Interface: RabbitMQ comes with a web management interface accessible through a browser. It provides an intuitive user interface for viewing and managing various aspects of the RabbitMQ server, including queues, exchanges, bindings, user permissions, and more. By default, the web management interface runs on port 15672.

• RokcetMQ

RocketMQ Console: RocketMQ officially provides a web console called RocketMQ Console. It offers a visual interface for managing and monitoring various aspects of a RocketMQ cluster, including topics, consumer groups, message queries, and performance statistics. You can use this tool by visiting the console's URL.

• Kafka

Kafka Manager: Kafka Manager is an open-source graphical management tool developed by Yahoo for managing Apache Kafka clusters. It provides real-time monitoring of cluster status, topic and partition management, consumer group management, and other features.

10、Messaging Models

The transmission models used by mainstream message middleware are primarily the point-to-point model and the publish-subscribe model. Specifically:

1. RabbitMQ:

• Publish-subscribe pattern：RabbitMQ supports the publish-subscribe pattern, in which a producer publishes messages to an exchange（Exchange）, which then delivers the messages to multiple queues bound（Binding）to it. Consumers can receive messages from the queues independently.

• Point-to-point mode: RabbitMQ also supports point-to-point messaging, in which a producer sends messages to a designated queue and consumers receive messages from that queue. Each message can be received and processed by only one consumer.

2.RocketMQ:

• Publish-subscribe model: RocketMQ supports the publish-subscribe model, in which producers send messages to a specified topic (Topic), and consumers subscribe to topics that interest them. RocketMQ supports multiple subscription modes, such as broadcasting and clustering, enabling message multicast or load-balanced consumption.

• Queue mode: RocketMQ also supports queue mode, in which producers send messages to a specified queue and consumers receive messages from that queue. Multiple consumers can consume messages from the same queue in parallel.

3.Kafka：

• Publish-subscribe model: Kafka uses a publish-subscribe model, in which producers publish messages to designated topics, and consumers subscribe to topics of interest. Kafka’s subscription model supports multiple consumer groups, each of which can consume messages independently, enabling high throughput and horizontal scaling.

• Partitioning model: Kafka divides a topic into multiple partitions, each of which physically corresponds to a separate log file. Producers send messages to specified partitions, and consumers can consume messages from different partitions in parallel. This partitioning model enables Kafka to scale horizontally and achieve high throughput.