- changed status to open
- removed comment
support make parallel options in simfactory
Pull request https://bitbucket.org/simfactory/simfactory2/pull-requests/11/parallel-make/diff adds a --jobs option to sim build that sets a @MAKEJOBS@ replacement so that one can use make -j @MAKEJOBS@ in machine definitons files and let the user choose the number of make jobs when compiling
Keyword:
Comments (16)
-
reporter -
- removed comment
Currently, Simfactory's machine database contains the preferred number of make jobs. Do you want to override this manually on a case-by-case basis? Is there a case where the default settings are inconvenient?
-
- removed comment
I could imagine that, from time to time, another than the default number of cores might be of interest. Say, if a build isn't time critical, and shouldn't take a large portion of a machine's cores. I didn't look at the patch yet, but I think the request is understandable. I would go as far as to see if a short "-j" option would be possible.
-
reporter - removed comment
Right now nothing changes since none of the ini files use @MAKEJOBS@. One can add makejobs to the ini files and use this as the default which can be overridden by the user. Eg.
[zwicky] makejobs = 6 make = make -j @MAKEJOBS@
which can then be overridden by the user on the command line (not tested yet).
The defaults are often inconvenient on the OSX and generic Linux builds (since one cannot now in advance whether the build is a raspberry pi or a 16 core corei7 workstation), see https://bitbucket.org/simfactory/simfactory2/commits/e6784d678365/ . I occasionally use a different number of make jobs on clusters but mostly for debugging or compile time error fixes when I want "-j1" so that make stops right after the error and error messages are not mixed with regular output from other make jobs.
-
reporter - removed comment
Replying to [comment:3 knarf]:
I could imagine that, from time to time, another than the default number of cores might be of interest. Say, if a build isn't time critical, and shouldn't take a large portion of a machine's cores. I didn't look at the patch yet, but I think the request is understandable. I would go as far as to see if a short "-j" option would be possible.
-j is already implemented.
-
- removed comment
If I read the code correctly, '-j' (without argument) would imply '-j 1'. Is this correct? If so, is this intended? Usually it means '-j infinity' (for make).
-
reporter - removed comment
-
- removed comment
If I don't specify an option, then Simfactory should use a system-specific default, not 1.
-
reporter - removed comment
Sure. The system specific default must be set in the machine.ini file for the machine by specifying a value of makejobs (same as the system specific default number of threads when running the simulation). If none is specified (and make is actually "make -j @MAKEJOBS@" rather than "make -j 12" or so) then @MAKEJOBS@ defaults to 1 unless either overridden by a setting in the machine.ini file or by a -j or --jobs option on the command line.
Obviously if you just hard-code "make" as "make -j 12" in the machine.ini file then no value for
--jobs
will do anything. -
reporter - changed status to open
- removed comment
Replying to [comment:9 rhaas]:
Sure. The system specific default must be set in the machine.ini file for the machine by specifying a value of makejobs (same as the system specific default number of threads when running the simulation). If none is specified (and make is actually "make -j @MAKEJOBS@" rather than "make -j 12" or so) then @MAKEJOBS@ defaults to 1 unless either overridden by a setting in the machine.ini file or by a -j or --jobs option on the command line.
Obviously if you just hard-code "make" as "make -j 12" in the machine.ini file then no value for
--jobs
will do anything.However it currently does not do this yet. Needs to be fixed.
-
- removed comment
Replying to [comment:7 rhaas]:
-j
without an argument should abort with an error stating that-j
requires an argument.That should be fine. Even 'make -j' (no number specified) is hardly ever useful.
-
reporter - changed status to open
- removed comment
Now it does handle makejobs in ini files and --jobs on the command line properly I believe. This is in git hash 88f3ddb which is the new head of the pull request.
-
reporter - removed comment
This should still be reviewed. Note that I implemented the requested changes.
-
- changed status to open
- removed comment
-
reporter - changed status to resolved
- removed comment
Applied as git hash 85946a3 of simfactory.
-
reporter - edited description
- changed status to closed
- Log in to comment