- removed comment
Don't overload machines while building
Certain machines are easily overloaded by building Cactus; building multiple configurations at the same time is not possible there. For example, LONI does not have sufficient compiler licences, or Orca (Sharcnet) does not allow sufficiently many user processes. On LONI, things will be very slow (also for other users), on Orca building will fail with strange error messages.
I suggest a mechanism that automatically serialises building Cactus configuration on certain sets of machines. Ideally, one would extend the "make -j" mechanism for this; practically, an implementation via locks may be simpler. Note that this has to work for sets of machines, not just single machines.
Keyword:
Comments (6)
-
-
reporter - removed comment
This works only on a single system; there is no way to ensure that builds on several separate systems are serialised. It also seems to take the whole system load into account, not only the processes started by one user.
-
- removed comment
Why would you want to ensure that builds on several separate systems are serialised?
Why would you want to limit the number of processes started by one user if the whole system can accommodate more?
-
reporter - removed comment
LONI provides a certain number of compiler licences shared across several systems. Building on several systems simultaneously slows down all builds, and also slows down builds of other users.
Orca does not allow too many processes per user. If one user tries to have more processes, things will fail with strange symptoms.
-
As of git hash 6576596 "wheeler: use makejobs" of simfactory2 most (all?) machines in simfactory support a
-j
switch on the level of simfactory making it easy for users to choose how many parallel make jobs to use.It does not quite automatically pick the correct value but makes it straightforward to control this. Since
-l
seems to not really work and implementing locks will only allow per user limits (and not take into account activity of other users) and is typically fragile, I will close this as wontfix. -
- edited description
- changed status to wontfix
- Log in to comment
GNU make has a -l option to specify a load threshold above which further tasks will not be spawned. In principle this would solve the problem. In practice I have observed it to severely underutilise the machine, with the load on the machine never approaching what I set it to.
From man make:
-l [load], --load-average[=load] Specifies that no new jobs (commands) should be started if there are others jobs running and the load average is at least load (a floating-point number). With no argument, removes a previous load limit.