Many people have discussed Mesos, but still don’t quite understand what problems it actually solves or what its use cases are. Zhou Weitao (one of the early adopters of Docker, Mesos, and related technologies in China) described it in one sentence: Mesos manages resources like CPU and memory on each machine, allowing you to operate the entire data center as if it were a single resource pool.
Zhou Weitao is currently the head of the cloud platform at Shurenyun Technology (whose main product, Shurenyun, is a cloud operating system based on Mesos and Docker). Previously, he worked at Red Hat, an international open-source solutions provider. He is a Red Hat Certified Engineer, a Mesos Contributor, and a senior Python developer. He is among the early group of developers in China who started working with Docker and Mesos.
Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications or frameworks, capable of running Hadoop, MPI, Hypertable, and Spark.
13 Questions to Deepen Your Understanding of Mesos
(Q&A from OSChina Open Source Community’s 100th Expert Session — Apache Mesos)
Q1: For most people who still don’t know what Mesos is, could you introduce what it does, its purpose, and how to use it?
A1: Hi there. Although there currently aren’t many Chinese-language resources on Mesos, a quick Baidu or Google search will still yield some results. Let me use an example to explain Mesos. Suppose a company frequently needs to perform big data computations. The task requires a large amount of CPU and memory at runtime. To meet this demand, we have two approaches:
Approach 1: Use a mainframe鈥攁 single machine that can provide sufficient resources for the task.
Approach 2: Distributed computing鈥攑rovide a group of normally configured machines (compute nodes), i.e., a cluster, split the computing task across these machines, and then aggregate the results.
Approach 2 is the currently popular method, and its advantages need no further elaboration. To realize Approach 2, we need to build a data center (cluster). Furthermore, to fully utilize the data center’s (cluster’s) resources (e.g., allocating different resources to different tasks, allocating based on task priority, etc.), we need a tool for managing and allocating resources across the entire data center. That tool is Mesos. A similar tool to Mesos is YARN.
In addition, Mesos not only offers resources for computing tasks, but it also supports running long-running services (such as web applications). Many internet companies abroad are currently using Mesos as their cluster management tool. Here is a Powered by Mesos list: https://mesos.apache.org/documentation/latest/powered-by-mesos/
Q2: We currently use the Cloudera suite. Could you briefly explain the differences between Mesos and Cloudera?
A2: Mesos’s primary goal is to help manage cluster resources across different frameworks (or application stacks). For example, there is a business need to run Hadoop, Storm, and Spark simultaneously on the same physical cluster. In this case, existing schedulers cannot achieve such fine-grained resource sharing across frameworks. Hadoop’s YARN scheduler is a monolithic scheduler that allows multiple frameworks to run on a single cluster.
However, it becomes very difficult to use framework-specific algorithms or scheduling policies because there is only one scheduling algorithm across multiple frameworks. For instance, MPI uses gang scheduling, while Spark uses delay scheduling. Running both on the same cluster can lead to conflicts in supply and demand. Another approach is to physically partition the cluster into multiple smaller clusters and run different frameworks independently on these smaller clusters. Yet another method is to allocate a set of virtual machines for each framework. As noted by Regola and Ducom, virtualization is considered a performance bottleneck, especially in High-Performance Computing (HPC) systems. This is precisely the scenario where Mesos fits—it allows users to manage cluster resources across frameworks.
Mesos is a two-level scheduler. At the first level, Mesos offers certain resources (in the form of containers) to the corresponding framework. At the second level, after receiving the resources, the framework runs its own scheduling algorithm to assign tasks onto the resources provided by Mesos. Compared to Hadoop YARN’s monolithic scheduler, this might be slightly less efficient in terms of overall cluster resource usage. But it brings flexibility—for instance, multiple framework instances can run within a single cluster. This is something existing schedulers cannot achieve. Even Hadoop YARN is merely trying to support third-party frameworks like MPI on the same cluster. More importantly, as new frameworks emerge鈥擲amza, for example, was recently open-sourced by LinkedIn—Mesos allows these new frameworks to be experimentally deployed onto existing clusters and coexist peacefully with other frameworks.
Q3: Hello, what are the typical application scenarios for Mesos? From some introductions, I’ve seen it can serve as a Docker orchestration service. How does it differ from cloud platforms like OpenStack managing physical machine CPU and memory, or Cloudera Manager managing Hadoop cluster services?
A3: Mesos has a wide range of application scenarios now, such as:
1) Spark on Mesos (this is a standard combo)
2) Jenkins on Mesos
3) Mesos as a Docker orchestration service, etc.
Compared with OpenStack: First, both physical machines and virtual machines can serve as Mesos cluster nodes. Second, the granularity differs. Mesos’s basic compute unit is the container (LXC), while OpenStack’s is the VM (I’ve heard it now supports Docker container technology as well); the former achieves higher resource utilization. Finally, lightweight: Mesos only handles offering resources to Frameworks and does not schedule resources. OpenStack is closer to the IaaS layer, whereas Mesos sits above IaaS. That’s why some call it a DCOS, or Distributed Operating System.
Q4: Where are the boundaries, and what are the pros and cons? Thank you.
A4: Advantages
Resource management strategy Dominant Resource Fairness (DRF). This is the core of Mesos and the fundamental reason we liken Mesos to the kernel of a distributed system. In simple terms, Mesos ensures that all users within the cluster have equal opportunities to use cluster resources鈥擟PU, memory, disk, etc. Many compare Mesos to k8s. I don’t have deep knowledge of k8s, but I believe these two have different focuses and shouldn’t be directly compared. k8s is only responsible for container orchestration, not cluster resource management. Just because both can manage Docker doesn’t mean we should conflate them.
Lightweight. Relative to YARN, Mesos only offers resources to Frameworks and doesn’t schedule them. Theoretically, this allows us to let all sorts of things use Mesos cluster resources, unlike YARN which is mainly confined to the Hadoop ecosystem. What we need to do is develop a scheduler (Mesos Framework).
Improved resource utilization for distributed clusters: This is a generic advantage. In some respects, all cluster management tools aim to improve resource utilization. The emergence of VMs gave birth to IaaS; the emergence of containers gave birth to K8s, Mesos, etc. Simply put, given the same amount of resources, breaking them down into containers using K8s/Mesos clearly yields higher resource utilization than breaking them down into VMs using IaaS. (I’m not discussing security here; we assume an internal subnet environment doesn’t need to consider this.)
Disadvantages
High barrier to entry. Deploying just Mesos alone accomplishes nothing. To use it, you need different Mesos Frameworks like Marathon, Chronos, Spark, etc., or you need to write your own Framework to schedule the resources Mesos offers. This discourages many people.
Currently insufficient support for Stateful Services. Mesos clusters currently cannot handle data persistence. Version 0.23 introduced Persistent resources and Dynamic reservation, which will improve the data persistence situation.
The dirty and tiring work doesn’t disappear. The team was very optimistic in the early stages of using Mesos, thinking that once we mastered Mesos, our ops team would have a much easier time. However, that wasn’t the case at all. Optimizing cluster nodes, configuring disks and networks鈥擬esos won’t do any of that for you. In the initial usage phase, the ops workload not only didn’t decrease but actually increased. The Mesos project is still under intense development, and many features are not yet complete.
Q5: I’d like to ask, if one wants to build a cloud service platform, how should one choose between Mesos and Kubernetes?
A5: The current situation is that both the Mesos and K8s ecosystems are developing quite well on their own; discarding either would be a loss. It’s better to follow your personal preference, pick one, and dive in to start using it. For example, Shurenyun offers direct one-click deployment, which is very convenient. You can quickly experience the benefits of Mesos.
It really depends on your specific requirements. As far as I know, K8s currently only supports Docker and has few production-environment use cases; whereas Mesos does not require your application to be packaged into Docker and has been battle-tested in production environments. On the other hand, however, the K8s community is more active and developing rapidly, with very bright prospects. Of course, the above is not the key point; a good cloud platform is more about having a strong product philosophy. Please refer to Shurenyun.
Q6: For long-running tasks, are there any good scheduler algorithms or strategies?
A6: Long-running tasks rely on the Marathon framework. For Docker, Mesos + Marathon is basically the most mature distributed runtime framework available today. Long-running tasks rely on the Marathon framework. For Docker, Mesos + Marathon is basically the most mature distributed runtime framework available today.
Q7: Regarding Mesos and Docker integration, does Mesos only solve the resource allocation problem?
A7: Correct. Mesos is responsible for resource allocation. Something else needs to handle Docker task scheduling so that Docker instances can be automatically deployed into the cluster to run. That component is called Marathon. Mesos + Marathon is currently the most stable Docker cluster scheduling framework.
Q8: Can Mesos be gradually applied to production environments now?
A8: Mesos has been ready for production environments for a long time. Companies like Airbnb, Apple, Uber, Twitter abroad, and Ctrip, iQiyi domestically, as well as our company Shurenyun, are all using Mesos in production. You can see the list of Mesos users here: https://mesos.apache.org/documentation/latest/powered-by-mesos/
Q9: Is there any connection between Mesos and Zookeeper?
A9: Zookeeper is a software that provides consistency services for distributed applications, and Mesos is a distributed application. Therefore, in a production environment, we need to use Zookeeper to provide consistency services for Mesos.
Q10: Is there a competitive relationship among Mesos, Swarm, and Kubernetes? Although all three claim to support each other, wouldn’t doing so be too convoluted?
A10: Swarm and K8s have a lot of overlap. Mesos focuses more on resource management and just happens to be able to use containers for resource isolation. Whether there is competition depends on the direction the communities take.
Q11: Hello, looking at this framework, I have a few questions:
1. Does this framework come with a built-in log collection module?
2. Can this framework perform performance statistics?
3. Can this framework automatically switch when a node’s resources are exhausted? If all nodes’ resources are exhausted, is it prone to crashing, and how is its self-recovery capability?
4. Can this framework be configured with load balancing?
Thank you 🙂
A11:
1. It has a log module, but the functionality is relatively simple and lacks a global display.
2. Yes, it can perform performance statistics.
3. Mesos decides how many resources to offer to the scheduler based on the current cluster resource statistics. Resource exhaustion only prevents new applications from being deployed; it does not affect running services.
4. Load balancing can be configured. Moreover, Mesos itself has a multi-Master mechanism.
Q12: How does Mesos decide how many resources to allocate? When are allocated resources reclaimed?
br