Source

mana-core-athenamp / ChangeLog

Full commit
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
2011-09-26  Vakho Tsulaia  <tsulaia@mail.cern.ch>

	* Sebastien's fixes for non-linux platforms

2011-09-23  Vakho Tsulaia  <tsulaia@mail.cern.ch>

	* Switch to iService for determining geo version in the merge_pool_files_fast

2011-09-05  Robert Harrington <R.Harrington@ed.ac.uk>
	* tagging AthenaMP-00-03-14
	* MpMergeUtils.py: added merge_hit_files method for HITS merging using Merging_trf.py

2011-09-02  Sebastien Binet  <sebastien.binet@cern.ch>

	* tagging AthenaMP-00-03-13
	* remove old AthenaInternal::to_ievent_seek hack
	* M python/PyComps.py

2011-08-30  Vakho Tsulaia  <tsulaia@mail.cern.ch>

	* Fix ATN failures caused by problems with openning certain files in AthFile

2011-08-19  Vakho Tsulaia  <tsulaia@mail.cern.ch>

	* Fix merging for the histograms made through THistSvc

2011-08-15  Vakho Tsulaia  <tsulaia@mail.cern.ch>

	* Fix AthenaMP crash when NumEvents > NumWorkers

2011-08-05  Vakho Tsulaia  <tsulaia@mail.cern.ch>

	* Fix hybrid megrer crash when AthenaMP is activated using ATHENA_PROC_NUMBER environment

2011-07-27  Vakho Tsulaia  <tsulaia@mail.cern.ch>

	* Fix for LD_LIBRAY_PATH hack in the hybrid merger

2011-07-27  Vakho Tsulaia  <tsulaia@mail.cern.ch>

	* Multiple updates and fixes
	1. Replace the old fast pool merger with the hybrid merger and switch to the latter by default
	2. Use Round Robin mode only if explicitly requested, disable automatic failover
	3. By default process 1 event before forking
	4. Re-start the event selector after I/O reinit in workers to fix the missing 1st event problem

2011-07-04  Vakho Tsulaia  <tsulaia@mal.cern.ch>

	* More fixes for the merger selection mechanism

2011-07-01  Vakho Tsulaia  <tsulaia@mail.cern.ch>

	* Fix the merger selection mechanism 

2011-06-30  Vakho Tsulaia  <tsulaia@mail.cern.ch>

	* Several fixes in MpMergeUtils
	1. Do merging even if the number of files =1
	2. Move away from the mechanism of choosing the merger based on file names
	3. Use CollAppend.exe instead of CollAppend for TAG merging

2011-06-22  Vakho Tsulaia  <tsulaia@mail.cern.ch>

	* Patch IoUtils to deal with castor access problems

2011-06-07  Vakho Tsulaia  <tsulaia@mail.cern.ch>

	* Fix few typos in PyComps. Critical for running AthenaMP with transforms

2011-06-03  Vakho Tsulaia  <tsulaia@mail.cern.ch>

	* Apply AthenaMP-00-02-20-01 changes to the trunk

2011-06-02  Charles Leggett
	* move IoComponent stuff to GaudiMP
	* tag AthenaMP-00-03-00

D       python/IoRegistry.py
D       python/FdsRegistry.py
D       src/IoComponentMgr.cxx
D       src/IoComponentMgr.h

M       python/IoUtils.py
M       python/PyComps.py
M       python/MpMergeUtils.py
M       src/components/AthenaMP_entries.cxx

	
2011-05-30  Vakho Tsulaia  <tsulaia@mail.cern.ch>

	* Don't crash if some of output streams have no OutputFile attribute

2011-05-27  Vakho Tsulaia  <tsulaia@mail.cern.ch>

	* Fire EndRun at the end of worker processes

2011-05-13  Vakho Tsulaia  <tsulaia@mail.cern.ch>

	* Change the way MpMergeUtils decides whether or not a given file is POOL file 

2011-05-10  Wim Lavrijsen <WLavrijsen@lbl.gov>
	* Remove the exception catching around PoolFileCatalog uses,
	now that the coding error is fixed in PyUtils-00-10-70 and
	collected in 17.0.1. The exception handling may hide other,
	more serious errors otherwise.

2011-05-10  Wim Lavrijsen <WLavrijsen@lbl.gov>
	* AthenaMP-00-02-17 for the benefit of a possible branch-point

2011-05-03  Wim Lavrijsen <WLavrijsen@lbl.gov>
	* cleanup of idea that neither harms nor works

2011-05-04  Paolo Calafiura  <calaf@calaf-laptop>

	* python/Utils.py (hack_copy): minor cleanup. Not worth a tag...

2011-05-03  Wim Lavrijsen <WLavrijsen@lbl.gov>
	* AthenaMP-00-02-16
	* cleanup global variable, place IoRegistry pickling with IoCompMgr

2011-05-02  Wim Lavrijsen <WLavrijsen@lbl.gov>
	* tagging AthenaMP-00-02-15
	* Allow pickling of MpEventLoopMgr and its assorted spread out
	and otherwise obscurely hidden properties

2011-04-26  Paolo Calafiura  <calaf@calaf-laptop>

	* tagging AthenaMP-00-02-14
	* python/Utils.py (hack_copy): skip files already in destination 
	directory, allowing hack_copy to be called twice in a job (needed 
	for simulation)

2011-04-08  Paolo Calafiura  <calaf@calaf-laptop>

	* tagging AthenaMP-00-02-13
	* python/PyComps.py: call hack copy before initialize to catch
	SimParams.db

2011-03-04  Paolo Calafiura  <calaf@calaf-laptop>

	* tagging AthenaMP-00-02-12
	* python/AthenaMPFlags.py (TmpDir): switched default value to CWD
	which makes more sense for production

2011-01-16  Paolo Calafiura  <calaf@calaf-laptop>

	* tagging AthenaMP-00-02-11
	* python/IoUtils.py (reopen_fds): fix syntax error...

2011-01-14  Paolo Calafiura  <calaf@calaf-laptop>

	* tagging AthenaMP-00-02-10
	* python/IoUtils.py (reopen_fds): HACK ALERT: in ATN there seems to be 
	temporary files in the IoRegistry that have disappeared by the time
	we are ready to reopen them. While we understand what is going on, we
	skip these files and hope for the best...

2010-12-20  Mous Tatarkhanov  <tmmous@berkeley.edu> 

	* tagging AthenaMP-00-02-08
	* changed how CollAppend is used in @f merge_pool_collection_files after 
	    Peter v G's correct instruction.
	* @f write_merging_log is added for writing log files from outputs of meregers.
	* Changed the way merger is picked for each output type

2010-12-17  Mous Tatarkhanov  <tmmous@berkeley.edu> 

	* tagging AthenaMP-00-02-07
	* Added  merge_pool_collection_files function for TAGs merging
	* List of files for merging extended by adding all *.root
	    files from workers dir
	* Changed the way merger is picked for each output type
	* M python/MpMergeUtils.py

2010-12-02  Mous Tatarkhanov  <tmmous@berkeley.edu> 

	* tagging AthenaMP-00-02-06
	* Added the mechanism to choose worker run function
	    round_robin  or queue run using doRoundRobin job property.
	    evtmax=-1 also switches to round_robin scheme
	* minor cleanup with logs and comments
	* M python/PyComps.py

2010-12-02  Mous Tatarkhanov  <tmmous@berkeley.edu> 

	* tagging AthenaMP-00-02-05
	* Added AthenaMP_jobProperties that
	    lists, explains and gives examples of all properties usage.
	* A share/AthenaMP_jobProperties.py

2010-12-01  Mous Tatarkhanov  <tmmous@berkeley.edu> 

	* Added @properties: doFastMerge and doRoundRobin
	* M python/AthenaMPFlags.py

2010-12-01  Mous Tatarkhanov  <tmmous@berkeley.edu> 

	* Added @fn merge_pool_files_fast for merging pool files fast,
	    that utilizes PoolMerge tool from Peter van Gemmeren
	* Added "sort of" plugin feature to choose the merger @fn 
	    based on output being merging
	* Plugin @fns avaialbe and used:
	    merge_pool_files - from PyUtils.PoolFile 
	    merge_pool_files_fast - local, uses PoolMerge
	    merge_root_files - local, uses hadd
	    merge_pmon_files - local, blank - TBImplemented
	* M python/MpMergeUtils.py 

2010-11-30  Mous Tatarkhanov  <tmmous@berkeley.edu> 

	* Added @fn merge_root_files
	* Added @fn merge_pmon_files
	* Skip the perfmon output merging for now
	* M python/MpMergeUtils.py 

2010-11-30  Yushu Yao

	* tagging AthenaMP-00-02-04
	* A share/bs2esd.py: added example joboption for bytestream2esd reconstruction

2010-11-29  Yushu Yao

	* tagging AthenaMP-00-02-03
	* M python/MpMergeUtils.py: fix poolfilecatalog problem when using bytestream

2010-11-25  Mous Tatarkhanov  <tmmous@berkeley.edu> 

	* Additions and modifications on behalf of Charles Leggett:
	* Charles created VetoFirstEvent algorithm and corresponding test jobo
	* A python/VetoFirstEvent.py 
	* A share/tests/test_VetoFirstEvent.py
	* M python/PyComps.py

2010-11-20  Mous Tatarkhanov  <tmmous@berkeley.edu> 
	
	* Added mp example version of fast and full reco 
	* (mp_rdotoesd_id and mp_rdotoesd)
	* A share/mp_rdotoesd_id.py
	* A share/mp_rdotoesd.py

2010-11-15  Paolo Calafiura  <calaf@calaf-laptop>

	* tagging AthenaMP-00-02-02
	* M python/PyComps.py 
	* (MpEventLoopMgr.__init__): check that EvtMax is defined before we try to use it

2010-11-13  Mous Tatarkhanov  <tmmous@berkeley.edu> 

	* Fix to PoolFileCatalog name error for ByteStream input files
	* Fallback to dummy_pfc for ByteStream jobs, 
	* when PoolFileCatalog?()  raises naming exception 
	* catch an error from PyUtils.PoolFileCatalog and 
	* provide dummy replacement function that returns unchanged name 
	* M python/PyComps.py

2010-10-19  Mous Tatarkhanov  <tmmous@berkeley.edu> 

	* Adding python script to monitor PSS, RSS and VMEM for AthenaMP
	    that usese smem tool underneath
	* A share/tests/smem_mon.py

2010-10-14  Mous Tatarkhanov  <tmmous@berkeley.edu>

	* tagging AthenaMP-00-02-01
	* Close and join workers asynchronously
	* M python/PyComps.py

2010-09-15  Mous Tatarkhanov  <tmmous@berkeley.edu> 

	* Adding essential job-properties to AthenaMP in AthenaMPFlags
	* M python/PyComps.py
	* M python/Utils.py
	* A python/AthenaMPFlags.py
	* A share/mp_reco_fast.py

2010-09-12  Mous Tatarkhanov  <tmmous@berkeley.edu> 

	* tagging AthenaMP-00-02-00
	* First major attempt to refactorize the AhenaMP code and IO related bug fixes
	* Code related to Memory-status, CPU-status, affinity or Timer-log moved to Utils.py
	* Code related to IO-operations moved to IoUtils.py
	* Code related to Merging moved to MpMergeUtils.py
	* File Descriptors snapshot is taken in mother process as FdsDict class in FdsRegistry.py
	* FdsDict class deals with files created/used outside Athena IO framework
	* In case EvtMax is -1 athenaMP falls back to "round-robin" evt. scheme (run_worker_batch) 
	    verse default "queued" evt. scheme (run_worker_queue)
	* M python/PyComps.py
	* M python/IoRegistry.py
	* M python/Utils.py
	* A python/FdsRegistry.py
	* A python/IoUtils.py
	* A python/MpMergeUtils.py

2010-08-04  Mous Tatarkhanov  <tmmous@berkeley.edu> 

	* tagging AthenaMP-00-01-15
	* tide up PyComps.py from profiling mess and some other comments
	* M python/PyComps.py

2010-07-30  Paolo Calafiura  <calaf@calaf-laptop>

	* tagging AthenaMP-00-01-14
	* python/IoRegistry.py (merge_io_output): protect against running
	multiple merging procs when ATHENA_PROC_NUMBER is set

2010-07-30  Mous Tatarkhanov  <tmmous@berkeley.edu> 

	* tagging AthenaMP-00-01-13
	* Fixes to the trouble of merged file removal by parent process original output at finalization
	* moved merging of output into finalize() from executeRun() to bypass removal of files by mother
	* added the merge_dir arg into IoRegistry.merge_io_output function for merge destination
	* M python/PyComps.py
	* M python/IoRegistry.py

2010-07-30  Mous Tatarkhanov  <tmmous@berkeley.edu> 
	
	* redirected merged files  to "merge_dir" to avoid being deleted by mother process during finalize in curdir.
	* M python/IoRegistry.py

2010-07-22  Mous Tatarkhanov  <tmmous@berkeley.edu> 
	
	* tagging AthenaMP-00-01-12
	* M python/IoRegistry.py: fix LFN to PFN handling

2010-04-21  Mous Tatarkhanov  <tmmous@berkeley.edu> 

	* adding to the scripts list and minor fix of mp_basic_test.py 
	* M share/tests/mp_basic_test.py
	* M cmt/requirements

2010-04-20  Mous Tatarkhanov  <tmmous@berkeley.edu> 

	* adding script to extract workers event by event timing statistics from logs
	* merged output is renamed as mp_merged.$output_name 
	* A python/share/extract_mp_stat.py
	* M python/PyComps.py
	* M python/IoRegistry.py
	* M python/tests/mpMonTools.py

2010-04-20  Mous Tatarkhanov  <tmmous@berkeley.edu> 

	* seeking to last event problem fix 
	* M python/PyComps.py

2010-04-07  Mous Tatarkhanov  <tmmous@berkeley.edu> 

	* tagging AthenaMP-00-01-11
	* queued distribution of events to workers
	* M python/PyComps.py
	* M python/IoRegistry.py 

2010-04-07  Mous Tatarkhanov  <tmmous@berkeley.edu> 

	* seeking fixed moving setupEvtSelForSeekOps() into init() from setDefaults()
	* M python/PyComps.py
	* M python/IoRegistry.py

2010-04-06  Mous Tatarkhanov  <tmmous@berkeley.edu> 

	* worker-cpu affinity settings
	* M python/PyComps.py

2010-04-05  Mous Tatarkhanov  <tmmous@berkeley.edu> 

	* queue implementation of events to workers distribution 
	* M python/PyComps.py

2010-03-17  Mous Tatarkhanov  <tmmous@berkeley.edu> 
	
	* mp-scaling benchmark tools and monitor for athena multiple jobs
	* A share/tests/mjMon.py
	* A python/tests/mjMonTools.py
	* M share/tests/mpMon.py
	* M python/tests/mpMonTools.py

2010-03-17  Mous Tatarkhanov  <tmmous@berkeley.edu> 
	
	* mp-scaling benchmark monitor for AthenaMP
	* A share/tests/mpMon.py
	* A python/tests/mpMonTools.py

2010-02-22  Sebastien Binet  <binet@farnsworth>

	* tagging AthenaMP-00-01-10
	* dummy-out atn tests (they are just too heavy for ATN)
	* M test/AthenaMP.xml
	* M share/tests/mp_basic_test.py

2010-02-20  Sebastien Binet  <binet@farnsworth>

	* tagging AthenaMP-00-01-09
	* use athena-pool-merge instead of pool-fast-merge
	* reduce number of events for basic-test so it can run within atn
	* M python/IoRegistry.py
	* M share/tests/mp_basic_test.py

2010-02-17  Sebastien Binet  <binet@farnsworth>

	* tagging AthenaMP-00-01-08
	* fix utest (not enough karma)
	* add a basic utest (with enough AtlasCore karma)
	* put back the default EventsBeforeFork to 0
	* A share/tests/mp_basic_test.py
	* M test/AthenaMP.xml
	* M python/PyComps.py

2010-02-16  Sebastien Binet  <binet@farnsworth>

	* tagging AthenaMP-00-01-07
	* adding a genevt unit-test
	* A test/AthenaMP.xml

	* tagging AthenaMP-00-01-06
	* first attempt at adding support for g4 jobs
	* M python/PyComps.py

2010-02-03  Mous Tatarkhanov  <tmmous@berkeley.edu> 
	
	* the script for flushing cached memory
	* A share/tests/flush_cache.py
	* added an option to flush cache in mpMon.py
	* M share/tests/mpMon.py 
	* fast merge merge_pool_files added, slow merge commented
	* M pyton/IoRegistry.py

2010-01-25  Mous Tatarkhanov  <tmmous@berkeley.edu> 
	
	* added timer logs for mpMon.py script
	* M python/PyComps.py

2010-01-21  Mous Tatarkhanov  <tmmous@berkeley.edu> 
	
	* mp-scaling benchmark monitor for AthenaMP
	* A share/tests/mpMon.py
	* A python/tests/mpMonTools.py

2010-01-20  Sebastien Binet  <binet@farnsworth>

	* tagging AthenaMP-00-01-05
	* simple little genevt testing script
	* consolidate rankid numbering scheme
	* A share/tests/mp_genevt_test.py
	* M python/PyComps.py

2010-01-19  Sebastien Binet  <binet@farnsworth>

	* tagging AthenaMP-00-01-04
	* create a shared queue to reliably distribute rank-ids
	* fix merging in batch-run mode
	* fix skipping events in batch-run mode
	* M python/PyComps.py
	* M python/IoRegistry.py

2010-01-18  Sebastien Binet  <binet@farnsworth>

	* tagging AthenaMP-00-01-03
	* adding directories for testing infrastructure
	* fixing batch-run mode
	* A test
	* A python/tests
	* A share
	* A share/tests
	* M python/PyComps.py

2009-12-18  Sebastien Binet  <binet@farnsworth>

	* tagging AthenaMP-00-01-02
	* cosmetics
	* M python/IoRegistry.py

2009-12-17  Paolo Calafiura  for Mous Tatarkhanov

	* tagging AthenaMP-00-01-01
	* python/IoRegistry.py (merge_io_output): use merge_pool_file from
	PyUtils

2009-12-16  Paolo Calafiura for Mous Tatarkhanov
	
	* tagging AthenaMP-00-01-00
	* python/IoRegistry.py (merge_io_output): find files to be merged
	* python/PyComps.py: support forking of event workers after n events

2009-09-21  Sebastien Binet  <binet@farnsworth>

	* tagging AthenaMP-00-00-10
	* obo Mous: hacks for the pileupevtloopmgr
	* M python/PyComps.py
	* M python/IoRegistry.py

2009-09-02  Paolo Calafiura  <calaf@lxcalaf.cern.ch>

	* tagging AthenaMP-00-00-09
	* python/IoRegistry.py (update_io_registry): hack to deal with rfio paths

2009-08-28  Sebastien Binet  <binet@farnsworth>

	* tagging AthenaMP-00-00-08
	* better fall-back solution when no $TMP variable defined
	* M python/PyComps.py (_ATHENA_MP_TMPDIR)

2009-07-09 Paolo Calafiura

	* tagging AthenaMP-00-00-07
	* M python/PyComps.py: fix crash when trying to get AthenaEventLoopMgr

2008-12-10  Sebastien Binet  <binet@lxplus237.cern.ch>

	* tagging AthenaMP-00-00-06
	* using the official way of fixing oid2
	* M python/Utils.py

2008-11-25  Sebastien Binet  <binet@lblbox>

	* tagging AthenaMP-00-00-05
	* stream-lined implementation
	* M python/IoRegistry.py
	* M python/PyComps.py
	* M python/Utils.py

2008-10-06  Sebastien Binet  <binet@lblbox.dyndns.cern.ch>

	* tagging AthenaMP-00-00-04
	* checkreq fixes
	* M cmt/requirements

2008-09-29  Sebastien Binet  <binet@lblbox>

	* tagging AthenaMP-00-00-03
	* migration to AthenaBaseComps + updates
	* M cmt/requirements
	* M python/PyComps.py
	* M python/Utils.py
	* M src/IoComponentMgr.cxx
	* M src/IoComponentMgr.h

2008-08-27  Sebastien Binet  <binet@lblbox>

	* tagging AthenaMP-00-00-02
	* preliminary support of POOL-files merging
	* M python/Utils.py

2008-08-15  Sebastien Binet  <binet@lblbox>

	* tagging AthenaMP-00-00-01
	* support for theApp.EvtMax=-1
	* M python/PyComps.py

	* tagging AthenaMP-00-00-00
	* first import