Fixes #509 - Flag for reviveOffers and the duration for which to reject offers

[gkleiman]

@ConnorDoyle @elingg @brndnmtthws

NOTE: I plan to add some tests tomorrow, but I would really appreciate if I could get any feedback in the meantime. This has already been done to Marathon, please see mesosphere/marathon#1931.

Add the following command line flags:

--revive_offers_for_new_jobs if true, revive offers is called when a new job
is added to the TaskManager.

Also note, that Mesos only filters offers that are a strict sub set of a
rejected offer.

--decline_offer_duration allows configuring the duration for which unused
offers are declined.

--min_revive_offers_interval if --revive_offers_for_new_jobs is specified,
do not call reviveOffers more often than this interval. It defaults to 5
seconds.

[ConnorDoyle]

One important thing to add is to reviveOffers on registration and re-registration. See: mesosphere/marathon@d3560b4...marathon-0.9.1-backports#diff-27dd4ebda88590d649eae0d05e3a6b47R34

[gkleiman]

@ConnorDoyle: thanks for reviewing this.

We do that here: https://github.com/mesos/chronos/pull/510/files#diff-3b3cdf7fd755c23f1c70baa2d7c90368R49

[ConnorDoyle]

Oops, I missed it. Great!

[gkleiman]

Added some unit tests.

[brndnmtthws]

Maybe add a counter here? From the metrics registry.

[brndnmtthws]

Code looks good. Can you comment on how this has been tested? Aside from the unit tests (thanks for those, btw).

[gkleiman]

@brndnmtthws

Thanks for having taken the time to review this.

The new actor debounces reviveOffers calls, so I added two counters: reviveOffersRequest and reviveOffers. The first one counts the number of times a reviveOffers was requested and the latter the number of times reviveOffers was actually called on the Mesos driver.

I added a couple more unit tests and also did some manual testing:

Reproducing the starvation

• Start Mesos locally: MESOS_RESOURCES="cpus(*):1;mem(*):512;disk(*):100" mesos local --ip=127.0.0.1
• Start 10 Chronos frameworks without the new flags:

#!/bin/sh

i=0;

while [[ $i -le 9 ]]; do
  bin/start-chronos.bash --master 127.0.0.1:5050 \
    --graphite_host_port 192.168.99.100:2003 \
    --graphite_reporting_interval 5 \
    --graphite_group_prefix "chronos_${i}" \
    --http_port "808${i}" \
    --zk_path "/chronos${i}/state" &>"/tmp/chronos_logs/${i}.log"&

  i=$((i+1));
done;

• Add the following jobs to one of the just-started Chronos frameworks:

{
   "name": "test",
   "command": "sleep 500000",
   "schedule": "R//PT10M",
   "runAsUser": "gaston",
   "disk": "35",
   "mem": "1"
}

{
   "name": "starved",
   "command": "sleep 20",
   "schedule": "R//PT1M",
   "runAsUser": "gaston",
   "disk": "2",
   "mem": "1"
}

• Watch in the logs how that framework stops getting offers after having scheduled test and is unable to start starved.

Trying out the new flags

• Stop all the processes and clean up the ZK state.
• Start Mesos as before.
• Start the 10 Chronos frameworks, this time with the following extra parameters: --revive_offers_for_new_jobs --decline_offer_duration 3000000.
• Add the same jobs as before.
• Verify that the framework is able to start both the test and the starved job.

[brndnmtthws]

Looks great to me, nice work. 👍

Merge away.