On November 28, Alibaba announced it would donate the open-source distributed messaging middleware RocketMQ to the Apache Foundation, making it an Apache Incubator project. Upon successful incubation, RocketMQ is poised to become the first top-level middleware project from China at the Apache Foundation and a new member of the distributed messaging engine family globally, joining ActiveMQ and Kafka. This donation signifies that internet middleware represented by MQ (Message Queue) will play an increasingly significant role in emerging IoT and big data fields, benefiting more developers.
RocketMQ is Alibaba’s third-generation distributed messaging middleware, open-sourced in 2012 and maintained by the Alibaba Middleware team. Its commercial version, MQ, is available for purchase on Alibaba Cloud. During every Double 11 shopping festival, RocketMQ has handled all message flows for Alibaba’s production systems, delivering stable and outstanding performance on core transaction paths. It is one of the core foundational products handling peak transaction volumes (170,000 transactions per second in 2016). This year’s Double 11 saw it achieve precise, low-latency delivery of trillions of messages.
Q: Can you briefly introduce the history of RocketMQ?
A: As early as 2007, the Alibaba Middleware team independently developed the Notify messaging middleware, which was widely used in production systems. To meet the massive message backlog demands of online applications, in 2011, the team independently developed the RocketMQ messaging middleware, featuring a single-machine capacity for hundreds of millions of message backlogs and support for strict message ordering. As of this year, all messages within Alibaba online systems are forwarded and delivered through RocketMQ. During Double 11 2016 alone, the message volume reached trillions, with peak TPS in the tens of millions, setting a message flow record in China and even globally.
Q: What are the product features of RocketMQ?
A: RocketMQ is a low-latency, highly reliable distributed messaging middleware. It is designed with a long-polling pull model, supporting traditional point-to-point and publish/subscribe models. A single topic supports a message backlog of tens of millions. For offline computing, especially handling Double 11 peak data floods, it plays an excellent role in peak shaving and valley filling. For enterprise applications, it effectively provides asynchronous decoupling. Within Alibaba, using RocketMQ for loosely dependent decoupling has become the unequivocal choice.
As a data pipeline, RocketMQ also boasts efficient throughput capacity for transactional data replication (globally ordered) and real-time computing. In Alibaba’s data application scenarios during this year’s Double 11, RocketMQ’s end-to-end average latency was reduced to under 100ms.
Furthermore, in virtualization and container deployment, RocketMQ supports rapid start-stop and possesses very strong horizontal scalability.
Q: We’ve heard there is a commercial version of RocketMQ. Can you briefly introduce it?
A: The commercial version of RocketMQ is Alibaba Cloud’s Message Queue MQ. Both products are developed by the Alibaba Middleware Messaging team. Besides supporting TCP access, the commercial MQ version also supports HTTP and IoT MQTT protocol access. Functionally, it enhances operations management and control capabilities, including visualized message traces, resource report statistics, and monitoring alarms. On the public cloud, the commercial MQ version inherently features multi-datacenter deployment with intra-city high-availability disaster recovery, ensuring enterprise-grade high reliability and performance while significantly reducing enterprise customer O&M costs.
Q: Having both a commercial and an open-source version, how will the Middleware team maintain them?
A: How to maximize the synergy between open-source and commercial offerings? This is a global challenge. Unfortunately, we are currently facing this very problem and are racking our brains to solve it. The approach we’ve found to be a more reliable and viable path is one centered on open-source, supplemented by commercial offerings. In other words, we will open-source all core distributed messaging features. On the commercial side, especially regarding cloud platform construction, operations, management control, security authorization, and in-depth training will be top commercial priorities. Following the operational philosophy of the community, especially the Apache open-source community, we will increase investment in community building, documentation, discussions and exchanges, and talent acquisition.
Q: Why does Alibaba engage in open source?
A: Alibaba is a company that places great emphasis on gratitude. In its early days, it was precisely through open-source software that the prototype of the entire e-commerce platform was rapidly built. Even after all these