Memory usage increasing continuously for ever

[Helena Lefebvre]

When running LHC model with 1 particle for many turns (over 1000), the memory usage increases ~linearly until the process is killed. After about 3000 turns, the memory used is about 60 GB.

Attaching the bdsim model, to run:

bdsim --file=lhc.gmad --batch --circular --ngenerate=1 --outfile=maxturns_10000

[Laurie Nevay]

I'm pretty sure this is a repeat of Issue #242. The memory usage will increase linearly with primary trajectory track length as we store the primary trajectory during the event always - even if it isn't written out - this is so we can record the hit and loss positions.

The memory usage is too high though. @stewartboogert is working on reducing this transient memory usage.

@pikharha Are you purposively storing trajectories in the output?

@stewartboogert We could consider reimplementing the accrual of the hit and loss point to something we do as we go along rather than having to save the whole primary trajectory by default.

Generally, though, we should look at reducing the transient memory usage if possible. So for 3000 turns, and for this lattice, I'd guess around 10k steps / turn, comes to 3e7 entries -> ~2kb / trajectory point which isn't crazy. Still a few variables can be removed we think.

Do you require the trajectory information or only say, sampler information?

[Helena Lefebvre]

Having had a quick conversation with @stewartboogert he suggested that there could be an option added to flush the trajectory information (maybe after each turn). The reason I came across this bug was that I was trying to find how many turns it took for the BDSIM LHC model to lose a particle, but I found it hits a limit in memory long before that when I have tried.

[Laurie Nevay]

I have fixed this by reimplementing how the primary first hit and last hit are found. Instead of accruing the whole primary trajectory and processing it at the end, we now accrue only the two points we need as we go along.

This was implemented through a new derived class of BDSTrajectory called BDSTrajectoryPrimary. This can optionally store the points as it goes along (so in the case of storing trajectories or visualisation, things work as normal).

Memory usage is now completely constant with indefinite tracking. Of course, if you have samplers on, these will continue to accrue information (say on each turn), but the memory increase over time is much much smaller.