Sourceloop Stats Race Condition
Issue #45
resolved
problem: the list printed to stdout during raytracing (shown below) is missing the brightest sources because the threads that are raytracing those sources still have counterLog->totalPhoton = 0 which leads to them being skipped in counterCheck in counter.cpp. Also in our case most of the time was spent on a really bright source and the raytracing spent most of its time in lines 188-190 of sourceloop.cpp. However, the printed output makes it seem like the raytracing is done and like something is blocking or deadlock.
'------------------------------------------------------------------------------------------ Type Sources Photons (Sat,Rem,Rej,Acc)% Time (s) Photons/s
Astro Object m=16.5 12 64,157,289 ( 0, 64, 26, 10) 76.67 836,674 Astro Object m=17.0 14 220,273,642 ( 0, 63, 27, 10) 40.70 5,411,025 Astro Object m=17.5 7 88,694,242 ( 0, 63, 30, 7) 18.18 4,876,789 Astro Object m=18.0 17 150,553,451 ( 0, 62, 28, 10) 16.43 9,158,192 Astro Object m=18.5 19 68,570,846 ( 0, 54, 37, 9) 20.69 3,312,576 Astro Object m=19.0 15 35,470,350 ( 0, 60, 28, 12) 13.11 2,702,647 Astro Object m=19.5 44 94,370,571 ( 0, 53, 37, 10) 16.20 5,820,095 Astro Object m=20.0 58 115,153,252 ( 0, 47, 43, 10) 10.55 10,904,195 Astro Object m=20.5 80 199,452,115 ( 0, 56, 37, 7) 12.43 16,027,707 Astro Object m=21.0 122 178,250,592 ( 0, 52, 39, 9) 11.60 15,356,086 Astro Object m=21.5 194 102,350,700 ( 0, 38, 54, 8) 9.55 10,700,678 Astro Object m=22.0 318 106,329,039 ( 0, 37, 55, 8) 8.07 13,156,761 Astro Object m=22.5 450 102,681,552 ( 0, 37, 55, 8) 6.62 15,486,273 Astro Object m=23.0 697 58,497,536 ( 0, 15, 81, 5) 6.75 8,653,991 Astro Object m=23.5 899 58,463,128 ( 0, 21, 75, 4) 5.46 10,689,557 Astro Object m=24.0 1,439 95,335,318 ( 0, 37, 55, 8) 5.34 17,803,592 Astro Object m=24.5 1,872 78,934,220 ( 0, 35, 56, 8) 3.84 20,480,382 Astro Object m=25.0 2,363 76,986,815 ( 0, 42, 50, 8) 2.82 27,245,623 Astro Object m=25.5 3,051 28,149,096 ( 0, 13, 83, 4) 2.40 11,681,872 Astro Object m=26.0 3,687 21,815,578 ( 0, 14, 82, 4) 1.93 11,273,491 Astro Object m=26.5 4,175 63,781,580 ( 0, 49, 40, 11) 1.42 44,612,795 Astro Object m=27.0 4,144 53,724,924 ( 0, 54, 36, 10) 0.95 55,973,135 Astro Object m=27.5 3,331 6,336,556 ( 0, 15, 81, 4) 0.52 11,906,591 Astro Object m=28.0 2,172 2,849,903 ( 0, 15, 81, 4) 0.24 11,274,733 Astro Object m=28.5 1,261 1,144,205 ( 0, 14, 82, 4) 0.11 9,453,888 Astro Object m=29.0 865 526,467 ( 0, 16, 77, 7) 0.07 6,937,341 Astro Object m=29.5 741 269,230 ( 0, 15, 78, 7) 0.04 5,619,497 Astro Object m=30.0 568 137,379 ( 0, 14, 78, 7) 0.02 4,396,832 Astro Object m=30.5 438 56,562 ( 0, 9, 90, 2) 0.01 2,288,111 Astro Object m=31.0 262 22,717 ( 0, 9, 90, 1) 0.01 1,108,420 Astro Object m=31.5 108 6,174 ( 0, 8, 90, 1) 0.00 460,505 Astro Object m=32.0 66 2,030 ( 0, 3, 96, 1) 0.00 170,500 Astro Object m>32.0 65 770 ( 0, 2, 97, 1) 0.00 61,585
Electron to ADC Image Converter ------------------------------------------------------------------------------------------'
reproduce: run raytracing on a machine with lots of threads and have a few bright sources that dominate the runtime.
fix: not sure there is a simple fix to this, perhaps changing the logging for now to make it clear that the bright sources are being accounted for and that the printed stats are misleading with regards to runtime. Also probably worth warning the user that code is still waiting for threads to finish in lines 188-190 in sourceloop.cpp.
Comments (3)
-
-
The source splitting feature is fully operational in v5.0.X, so this is resolved.
-
- changed status to resolved
- Log in to comment
we are finishing up working on a feature that is coming soon that splits the brighter sources among the threads, so this won't occur in the future. in other words, all threads will be working roughly equally on the brightest sources before proceeding. until then, I agree that the output is confusing, but you can trust that it is still working on the bright sources as this is well-tested in massive data challenges. Also, this output confusion i believe in practice only occurs when you turn background off, as when it is on then it will be working on background during this time.