Node/resource reservation (GPU, etc.)

[IT Expert]

Hi. Have read in docs next features:

* node reservation (mesos)
* experimental resource reservation (GPU, etc.)

Is it mean resource limitation per job, when create job, we limit resource usage, like 1CPU, 1GPU, 1GB RAM. Or its mean we can reserve some resource in cluster for user/project, and it always be free until this project/user will start job ?

If it is reservation it will be really cool, how to configure it ?

[Olivier Sallou]

for GPU, it is the fact that user can ask, in a job, to get 1 ou 2 GPU. When I started this, native GPU reservation was not available in mesos, like for CPU and RAM, so an additional mechanism was set in place. It still exists, but is of no use as GPU is natively managed now (depending on your mesos version).

For node reservation, it is possible to put in compute node, via its config, the label "reservation" to restrict node to specific users or projects (reservation=me,you,myproject). Other user, projects won"t be scheduled on this compute node.

So it is more limitation than reservation....

[IT Expert]

To realize reservation of resources like described in mesos docs:

http://mesos.apache.org/documentation/latest/reservation/

Is it possible to run additional go-docker scheduler, change role name in config, and then add reservation to this role on mesos slave nodes ? If yes, is this second scheduler must be ran with another databases or am I need only to change databases names in config, or left the same names ?

I tried to run second scheduler in new container with all same options except mesos role name. Looks likes it's wrong. Because mesos doesn't register new framework, but changed role name between this two names. (Framework Id was the same, but role name changing every page refresh)

[Olivier Sallou]

scheduler makes use of a mongo db and a redis db. Both can be configured in go-d.ini to use different db name (mongo) and redis prefix. Different schedulers would need different configs. However web makes also use of scheduler (for job identifiers, get job status info etc...) , you cannot, for the moment, have 1 web , for 2 or more scheduling system (using different roles or configuration). You would need 2 "godocker" (but database instances could be the same, using different connection/prefix names)

[Olivier Sallou]

I plan, for the future, to be able to run multiple schedulers , independently while using a single database instance. But this a lot of work.

[Olivier Sallou]

how would you however limit some users of using a scheduler vs the other? (for your case of reservation). You would need to assign in config the rights to a set of users/projects to a specific scheduler as mesos is only aware of the scheduler role, not "godocker" users.

[IT Expert]

As far as I understand, I need second go-docker scheduler and second secret go-docker-web for the selected users. This is wrong direction. But how we could guarantee resources for some project/user? Is there another way ?

[Olivier Sallou]

Second web would not have to be secret. You could use a different/custom auth plugin, limiting acceds to some user groups for example. But i agree this is not an ideal situation. As i said, i expect to provide the possibility to run multiple "same" schedulers (mesos in your case) but no date for the moment

But what do you mean by guaranting resources? Could describe your use case?

If you limit access to a node to a list of user/project, with the godocker reservation label, you have limited access to those resources.

The fairshare scheduler gives an other resource access kind , with the promise to be scheduled in a fait way vs other users, with possible user or project different priority. But running jobs won't be killed to place jobs with a higher priority.

Node reservation (via godocker or mesos roles) will ensure that only some jobs will use a node, but if node is full, you will still need to wait for the resources to place new jobs.

[IT Expert]

But what do you mean by guaranting resources? Could describe your use case?

Situation like this. Computing node owner wants to share his server with others. Like to attach his server to cluster. Because he doesn't run tasks 24/7. But he wants to be sure that when he need to run task, he will get his resources right now(or maybe with minimal configurable time of wait).

I understand to realize this, scheduler must be very clever, it must keep this node free all the time or based on statistics, it must only run as short as possible tasks from other users to provide node owner guarantee the launch of his task, if it suddenly appear, with minimal wait if the node is already busy.

Or maybe node owner should have possibility to reserve his node in advance(Dynamic Reservation), when he need it.

[Olivier Sallou]

I'd like to implement this with a kind of "best effort" task. If cluster is full, and job allows for best-effort, then place job on other reserved nodes. When new jobs for reserved nodes arrive, kill best-effort jobs and reschedule them. But this is quite complex to setup, and not in short time roadmap.