Poor rebdsim performance

[Former user]

I recently upgraded from BDSIM 0.922 to 1.10 on my workstation and noticed the significantly reduced rebdsim performance. I ran some sampler analysis scripts over 0.922 generated data (10^7 primary particles), which generate mostly 2-dim. Plots with 1000x1000 bins (beam phasespace, profile). This before took about half a day and now takes about 4-5 days. BDSIM 1.10 generated data is processed in a similar pace. Reduction of the number of plots to be generated doesn't yield significant performance enhancement. Reduction of 1000x1000 to 100x100 bins on every plot speeds rebdsim up, but not to the formerly known pace (and reduces resolution). A quick look into resources monitor and iotop indicates, that the process is CPU limited (100% core usage, no IO).

[Laurie Nevay]

Hello,

In BDSIM v1.0, we changed the default histogram created by rebdsim from a simple total of all entries to an event-by-event histograms. This makes 1 histogram for each event and averages these histograms to give you a rate per event. This is inevitably more intensive, however, is the only way to ensure the correct statistical uncertainties. The uncertainties for each bin beforehand would not be representative of event to event variation.

The old style histograms can still be be made using "SimpleHistogram1D" (for example) instead of "Histogram". This is the sum of all entries and not the mean per event. However, I recommend the regular per-event "Histogram" one.

Note, an event here is 1 primary particle fired in the simulation.

As you correctly identify, the process is CPU limited rather than data input-output limited. We have taken some care to make this as efficient as possible, but the shear number of events will ultimately result in this timescale for a single process. For example, for 5 days, the analysis is 43ms per event and that includes several large histograms. It is unlikely this can be faster.

The solution is to parallelise the analysis. You can execute rebdsim multiple times - perhaps 1 instance of rebdsim per data file. Afterwards you can combine the rebdsim output histogram files using rebdsimCombine. This method produces an identical result but in a much faster time. Even a single workstation will support multiple processes. If you have access to a computing cluster, this would significantly speed things up.

I myself have just done an analysis on a simulation of 10^7 primaries. I use 200 cores on our high throughput computing cluster. Each job executes rebdsim and analyses 1x 100MB file, and it took about 20 minutes to analyse 50000 events. In total the complete analysis including merging was under 30 mins. I repeated this several times in one day as I decided on the exact histograms I wanted.

There is some discussion of this in the manual here: http://www.pp.rhul.ac.uk/bdsim/manual/output_analysis.html#scaling-up-parallelising-analysis

[Ben Ledroit]

Hi again,

thanks for your response, you are of course right.

I read the chapter, but did not completely get the point of it. I assumed that the successive calculation would increase performance, which is true compared to iterating twice over the data set, but not compared to averaging all events and then calculating error from bin content in the end like in prior versions.

I investigate on beam dynamics and am interested in the beam particles in masses, so the SimpleHistogramND appears to be suitable for my case and for now i can't see bigger impacts of the simple error calculation approach on my research.

Unfortunately, i don't have access to a computing cluster and have to stick with the 8 threads on my workstation. The former simulations i want to analyze further are generated with a single output file, so parallelization is no option so far. I'll have the opportunity for future simulations.

Thank you for your help!

[Laurie Nevay]

Hello Ben,

I will read over the analysis description in the manual again and try to make it clearer for future.

Having thought about your situation, I can make the analysis tool rebdsim analyse just a range of events in the output instead of the whole file. That way, you could copy the one large data file 8 times and analyse only bits of it in 8 separate processes. This is a relatively simple change and will generally be useful. I expect another release of BDSIM within a few weeks to a month.

In the mean time, if it's just a 2D histogram you need without per-entry errorbars (ie beam distributions), you can, as you say, just use the SimpleHistogram2D that should be a good bit quicker. Also, you could have a look at ROOT's TDataFrame which is able to implicitly parallelise analysis on a Tree. I don't have direct experience of this yet, but it looks promising. It would be able to parallelise it on the one file I believe. Our tool is a thin interface to TTree->Draw() method. Something like:

TFile* f = new TFile("yourfile.root");
TTree*  evt = (TTree*)f->Get("Event.");
TH2D* hresult = new TH2D("nameOfHistogram", "nameOfHistogram",   ..... );
evt->Draw("Primary.y:Primary.x >> nameOfHistogram");

would be the manual equivalent for a simple histogram.

I will close this issue now and create a new task for the range analysis in rebdsim.

[Laurie Nevay]

Agreed closed. New task to improve feature of analysis tool.