Optimize the scaling of some of the multicore-supported functions

[Xinqiu Yao]

See the attached figure for the scaling performance of some functions. cmap() has a very good scaling, rmsd() and dccm() are so so, and fit.xyz() is not so good.

Also consider simplifying the usage of multicore-supported functions. For example, auto-detect the number of availabe cores instead of input by user. The argument nseg.scale is sometimes mysterious; may be removed from function input but estimated in function body.

[Barry Grant]

nice to see this plot, thanks Xinqiu! I agree with internalizing default values for nseg.scale and ncores but perhaps have them still user settable also?

[Xinqiu Yao]

That's what I thought. The user set values can override the default values. I am looking for a command to get the number of idle CPU cores and it should work on Linux, Mac, and windows. Any suggestion is appreciated!

[Lars Skjærven]

Looks like package 'parallel' is part of R's base as of 3.0. This should have similar functionality as package 'multicore' (?) I've always used 'multicore' myself, but perhaps we should consider moving to 'parallel' with the advantage that users don't have to install a separate package?

Not sure how to fetch the number of idle CPU's. However, I think it's fair to do as most other software packages: use the number of available cores as default, and add user argument to adjust.

cores <- multicore:::detectCores()

[Xinqiu Yao]

Thanks! That is news to me. If parallel supports mclapply() it would be easy to switch from multicore to parallel; otherwise, we should be careful. I will dig a little bit in the parallel package.

Sounds like the simplest way is to use all availabe cores instead of idle cores. I know some way to get idle cores by calling system() and linux command but I wonder if it works on Mac or Windows.

[Lars Skjærven]

Jepps. parallel includes function mclapply. apparently it builds on the work from the multicore and snow packages.

[Xinqiu Yao]

Issues discussed here have been partially resolved, e.g. the implementation of setup.ncore() to automatically detect available cores and using 'parallel' package to replace 'multicore'. Optimization of functions is still undergoing and details have been moved to the wiki page: Optimizing.