Today we’ll discuss Mesos, Marathon, and Swan — a recently open-sourced Mesos Framework by DataMan.
What is Mesos?
After the rise of microservices, many large Internet companies needed to abstract resources (thousands of machines) for broader use. Previously, teams would partition hundreds of machines per department/project and run programs directly on hardware or VMs, resulting in poor resource utilization.
On the other hand, adding machines was slow and provisioning was difficult. Clever engineers asked: can we abstract all resources and allocate them on demand? This gave birth to Mesos, a resource scheduler inspired by Google’s Omega paper, first pushed by Twitter.
Having spent time with Mesos APIs and code recently, I’ll share my understanding of its internals.
Mesos Internal Architecture
Mesos master&slave
Mesos is a distributed architecture with master and slave nodes. The source code (C++) heavily uses ‘process’ — not traditional processes but an abstract libprocess, its core library. If you know Erlang, libprocess is an abstraction of Erlang’s IO and communication model.
libprocess (called ‘process’ in Erlang) is a virtual process running on the Erlang VM. Its key feature is message passing between processes — messages are buffered, not sent directly.
This is ideal for distributed systems. Traditional network programming listens on a port, has a module parse packets, then passes to a handler process (possibly IO-bound). The bottleneck is the dependency between parsing and processing — slow parsing blocks everything.
Later came the idea of separating everything — each message gets its own process/thread. Since threads are heavy, Go uses channels and Erlang uses processes. libprocess handles IO this way, heavily used by Mesos.
Zookeeper
Mesos relies on Zookeeper for distributed eventual consistency. All runtime data ends up in Zookeeper. With multiple masters, as long as one is alive, the service is guaranteed (HA).
protobuf
Mesos uses protobuf for internal communication — it’s popular, has good multi-language support, and strong structural semantics.
That covers Mesos internals. Now let’s introduce Mesos concepts.
Mesos Concepts
Mesos master&slave
Mesos is distributed, with master handling task distribution and resource scheduling, and slave executing tasks. Mesos uses two-level scheduling — slaves are the lower level, allowing dynamic addition/removal without affecting the task pool.
Executor
Executor runs the actual task logic. Mesos gives users flexibility to write different Executors — e.g., for containers or MapReduce. An Executor is a binary that slaves pull from a remote URL.
Scheduler
Scheduler: after master/slave collect resources, tasks are handed to the Scheduler which decides what Tasks to run.
Framework
Framework: the upper layer of two-level scheduling, deciding what tasks to run and how many.
Offer
Offer: an abstraction of Mesos resources (CPU, memory, disk) packaged for a Framework to decide how to use.
Task
Task: actual running units. Two types: Long Running Task (e.g., Docker process) and Batch tasks (e.g., MapReduce, cron jobs).
How Mesos Works Internally
Let’s explain how each of these components works.
Mesos master & slave
Mesos master is the cluster core. Multiple masters can run for HA. Its responsibilities include: Framework lifecycle management, slave lifecycle (add/remove/task assignment), Task Management (tracking which tasks run on which slaves), and Resource Allocation/Offer.
Slaves can be dynamically added or removed. If a slave is lost, master notifies the Framework. Slave code contains extensive Executor logic — pulling Executor from remote, launching tasks, maintaining task lifecycle, handling failures.
Mesos Framework
Mesos focuses on resource scheduling, delegating task scheduling to Frameworks. Different teams can write their own Frameworks and submit them to the Mesos cluster. Mesos only knows to give Offers to Frameworks and let them decide which Executors to run.
Two Popular Frameworks
Marathon Framework focuses on Long Running tasks with ‘application’ as its core concept. While Mesos focuses on individual small tasks, Marathon handles applications with multiple instances.
Chronos handles Job/scheduled tasks, especially suited for Docker-based cron jobs. Traditional Cron Jobs run on single hosts with resource limits — distributing them across multiple machines is complex, but Chronos simplifies this.
Scheduler/Scheduler Driver
Scheduler handles task distribution. It receives Offers from master, decides to accept/reject, and assigns Tasks to accepted Offers. Scheduler Driver handles messaging; the Scheduler itself is a Java abstract class/interface that Frameworks implement.
Executor/Executor Driver
Executor corresponds to Scheduler — it’s the execution part. Mesos provides a default container Executor for Docker tasks, but custom Executors can be implemented for specific needs.
For example, an image/file processing Executor is a small-task binary that slaves pull from a URL and execute. It’s an independent binary communicating with master/slave via RBC.
Marathon
Marathon as a Long Running scheduler layer provides significant value. Mesos alone can’t do much with its fine-grained tasks — it focuses on resource management, while Marathon handles tasks.
Marathon’s core concept is ‘application’ (a group of Tasks). A Rails/NodeJS app deploys many Tasks, which Marathon calls ‘instances’. You can scale up/down instances, and Mesos schedules them across slaves.
Marathon offers ‘Group’ — a set of applications. For example, multiple microservices serving the same goal with inter-service dependencies. Groups work well here, though in production, services have specific dependency assumptions.
Marathon supports rolling updates — multiple versions can coexist. If the new version is fine, continue; if not, rollback. Concepts: Deployment and Version. Deployments cannot overlap — they queue to avoid conflicts.
Marathon’s scale and rolling update features are very useful — e.g., Sina Weibo can scale up instantly during major events, or rollback to a healthy version if needed.
While Mesos abstracts resources, sometimes you need scheduling affinity — e.g., CPU-heavy tasks on better machines. Marathon provides ‘Constraints’ (labels, hostname patterns) for this.
Marathon’s interface is user-friendly (HTTP APIs). Mesos APIs are not end-user oriented. Marathon’s UI is also beautiful, especially the new version.
Mesos Scheduler Swan
Finally, let’s introduce Swan (https://github.com/Dataman-Cl…), a Mesos Framework by DataMan for General Purpose Long Running Tasks. After prolonged Marathon use, they found customization difficult. Swan aims to combine Marathon’s functionality with controllability and desired features.
Swan must support HA (inspired by Swarmkit’s Raft protocol — as long as one manager is alive, service continues).
Marathon doesn’t do service discovery — it exposes ports/IPs via API. Swan has built-in service discovery with a service listing.
DNS is another service discovery approach. Major Frameworks put DNS at the core. Swan’s DNS module acts as both DNS and proxy, enabling seamless internal/external access.
Proxy also handles load balancing. Common tools: HAProxy and Nginx. Both are excellent but lack controllability. HAProxy has poor programmability; Nginx is better (Sina Weibo’s upsync-module). Integrating both was challenging.
That’s all for my presentation. Thank you.
Original: https://segmentfault.com/a/1190000007723430