Marathon Mesos FAQ 运维之家

Issue 1: Inconsistent Mesos Leader Election

Use the command below inside the mesos-master container to check its logs and see which server it considers the Mesos leader.
Different servers may elect different leaders!!
The cause of this issue is currently unknown, but it is suspected that time discrepancies between machines are to blame.

Issue 2: Marathon Leader Election Error!

After searching for a long time, I found no useful information! The suggestion is to delete the /marathon directory in ZooKeeper and then restart the Marathon service!!
I also don’t know where to check who the leader is. In any case, first use NTP to ensure the server times are consistent, then use journal to check the logs, and restart whichever one has the problem.

Issue 3: An App Does Not Leave "Waiting"

This means Marathon is not receiving “Resource Offers” from Mesos. The simplest possibility is that there are not enough resources, or another framework is holding all the resources. You can check the Mesos UI to see the available resources.

Issue 4: App Requires a Specific Port

Generally, it is not recommended to configure your app to rely on a specific port, as it restricts scheduling.

Set hostPort to a non-zero value in portBindings.
Set requirePorts to true.

Issue 5: When a server running mesos-slave restarts or the mesos-slave service is shut down directly, the task status in Marathon changes to Unscheduled — due to TASK_LOST. At this point, killing the task will fail.

The solutions found so far are:
(1) For Marathon 1.4.3 and later, add the following to each app’s definition file:

"unreachableStrategy": {
"inactiveAfterSeconds": 30,
"expungeAfterSeconds": 60
}

inactiveAfterSeconds: If a task instance is unreachable for longer than this value, it will be marked as inactive and a new instance will be launched.
At this point, the unreachable task has not yet been expunged.
The default value is 300s, and the minimum is 1s.

expungeAfterSeconds: If a task instance is unreachable for longer than this value, it will be expunged.
An expunged task will be killed if it does not come back.
A task instance is generally marked as unreachable before being expunged.
This value must be greater than inactiveAfterSeconds.
The default value is 600s.

(2) Add the following to the mesos-slave startup command line:

recovery_timeout=1mins

Actual results pending (the current issue is the inability to automatically launch new task instances)
[Results were unsatisfactory]

Issue 6: Marathon Tasks Cannot Start, Deployments Cannot Complete

The actual symptom is that Marathon shows the task status as "Staged", but when you check the target machine, the Docker container has not started.
Checking the mesos-slave logs reveals that the slave ignored the task command because it believes the master that issued the command is not the leader.
However, after this, the task enters a deadlock state; it cannot be killed, and even the appid cannot be stopped.
This is because Marathon receives no ACK for requests regarding the task status or kill requests for that task.
No solution has been found yet.
The only option is to forcefully kill the deployment (or forcefully kill the task; this is untested, but the API exists, so I don’t know if it works).

(1) Forcefully kill the deployment:

marathonAddr="http://abcd:1234";
deployment_id="14eed6d3-6f72-41f4-8329-ce6a7583eb32";
apiPath="/v2/deployments/${deployment_id}"
curl -X DELETE ${marathonAddr}${apiPath}?force=true

(2) Forcefully kill the task:

# Tested, this method is useless

marathonAddr="http://abcd:1234";
app_id="autodeploy"
task_id="autodeploy.8c873ce2-ad63-11e7-a70d-36d30528411f"
apiPath="/v2/apps/${app_id}/tasks/${task_id}"
curl -X DELETE ${marathonAddr}${apiPath}?force=true&scale=false&wipe=false

Author: 鍧氭寔鍒板簳v2
Link: https://www.jianshu.com/p/eb6659cdeda8
Source: 绠€涔?br />Copyright belongs to the author. For commercial reprints, please contact the author for authorization. For non-commercial reprints, please indicate the source.

Marathon Mesos FAQ