BDSIM is hanging between event 4000 and 5000

Issue #178 resolved
A Schuetz created an issue

I did several tests today, trying to run BDSIM 0.9.5 for larger numbers of events, but for some reason it hangs before reaching event 5000. I tried it with 'rootevent', 'root', and 'rootdetailed'.

This one for example is running now for almost one hour already, without any change of the output file size which stays at ~30MB. Screenshot_BDSIM_hangs.png

The memory usage stays at ~1%

Comments (7)

  1. Laurie Nevay

    The most common reason for long running events is that we have very low energy particles stuck in dipole fields and they don't loose any energy, so continue to spiral indefinitely. We've put some provision to deal with this in develop just now and have some better fixes being implemented.

    Are you using the same seed in each case?

    You can set:

     option, printModuloFraction=0.0001;
    

    which will ensure print out of every number. If you're using the same seed each time, it'll be getting to the same physical outcome each time. If not set, the seed will be based on the local time of the computer so it will tend to vary after a few minutes.

    Do you see it at the exact same event number?

    Unfortunately, this will show up with high statistics in v0.95.

  2. Laurie Nevay

    If you hit Ctrl+C once, it should kill bdsim (may take a minute or so), and it should write out the data and close the file properly. This only really works in batch mode as the visualiser is hard to kill.

  3. A Schuetz reporter

    I let the test runs run over night, and none of them finished. They are all still stuck between event 4000 and 5000.

    I will try it out with

    option, printModuloFraction=0.0001;
    

    and also set different seeds. I had set the same seed for all of them.

  4. Laurie Nevay

    Specifying the seed as an executable option like:

    bdsim --file=lattice.gmad --outfile=t1 --batch --ngenerate=10 --seed=123
    

    should always override any seed set in an input gmad file with:

    option, seed=123;
    

    As it's repeatable, this is undoubtedly due to the particular outcome of this event and is most likely due to a low energy particle being stuck in a dipole.

    The one practical way around this (until the next release in a few weeks when this will be eliminated) is to use the maximumTrackingTime options (in seconds). This limits the tracking time (ie time of flight) for any track in the simulation.

    Other than that, you'll likely see this problem occur at a similar frequency with other seeds. They way to scale up your statistics would be to run several instances of bdsim with different seeds and a lower number of events. The rootevent output can be easily combined in rebdsim by using '*'.

    InputFilePath                   *.root
    

    A general example of rebdsim is in bdsim/examples/features/io/3_rootevent/analysisConfig.txt

  5. Laurie Nevay

    Yes, I believe this is resolved. We've added specific logic to prevent spiralling particles in magnetic fields for all integrators.

    My recent studies of the LHC have shown around 2 million events with a very smooth distribution in event duration with a hard cut off. No event in that sample was seen to overrun uncontrollably.

    We've also introduced options to limit the maximum tracking time and user limits are attached to all volumes with magnetic field now.

  6. Log in to comment