Fix process extremely slow on large collections
Hello,
I use RC4 to manage a lot of different ROM collections. For the ones with a few hundred/thousand files, RC is quite fast.
But my MAME collection is quite large (~30k files), and it takes hours (literally, more than 8 hours) when I launch a fix on the whole collection. The parts of the process that are slow are the "Remove useless files" and "Remove useless roms" fixes. These take a really long time, and the computer doesn't appear to be doing much during that time (max 15% CPU usage, no disk activity).
My MAME collection is in ZIP format, I have a 4 core (8threads) CPU, 16GB of RAM, and fast SSDs
Comments (15)
-
repo owner -
repo owner - changed milestone to 4.0
-
assigned issue to
-
repo owner Can you attach the log ? (C:\ProgramData\romcenter\romcenter_xml.log) Zip it and post it in this issue.
-
Eric,
I'll launch a complete fix process tonight and send you the log tomorrow morning.
-
So, I launched a fix on my clean collection (34052 zip archives, 56 Go total) last night, it is still stuck (approximately 10 hours later...) on the first step : "Remove useless files". Which is weird since according to the counter on top of the file list, there aren't any.
The Romcenter process is using 15% CPU (1 full core), and is accessing my SSD (about 80Mo/s in write, but no read).
I'm probably going to stop this fix, do you still want the log file ? (it weighs more than 1 Go unzipped, I wasn't able to open it in Chainsaw; romcenter_xml.1.log is over 5Go).
-
Log4View is much better than Chainsaw, I was able to open the log file, here are the last lines :
Date Temps Enregistreur Message 08/03/2019 21:26:38,523 Romcenter Command 'All' launched 08/03/2019 21:26:38,559 Romcenter 34052 items selected 08/03/2019 21:26:40,433 Romcenter Rename files 08/03/2019 21:26:40,433 Romcenter Consumer started, max tasks = 6 08/03/2019 21:26:40,437 Romcenter Rename files producer started. 08/03/2019 21:31:53,515 Romcenter Renaming files... 08/03/2019 21:31:53,515 Romcenter 0 files to process 08/03/2019 21:31:53,515 Romcenter Enter locked DatasQueue section for completeAdding 08/03/2019 21:31:53,515 Romcenter Leave locked DatasQueue section for completeAdding 08/03/2019 21:31:53,515 Romcenter Producer Datas Queue for RomCenter.Engine.Operations.RenameFiles completed 08/03/2019 21:31:53,515 Romcenter exception: The operation was canceled. 08/03/2019 21:31:53,515 Romcenter exception: The operation was canceled. 08/03/2019 21:31:53,515 Romcenter Consumer closed. Waiting for operations to finish. 08/03/2019 21:31:53,597 Romcenter Wait for remaining tasks : 0 tasks 08/03/2019 21:31:53,597 Romcenter All operations finished. 08/03/2019 21:31:53,597 Romcenter 0 files updated 08/03/2019 21:31:53,597 Romcenter Consumer for 'RenameFiles' ended. Status: RanToCompletion 08/03/2019 21:31:53,597 Romcenter Consumer started, max tasks = 6 08/03/2019 21:31:53,597 Romcenter Remove useless files 08/03/2019 21:31:53,597 Romcenter Remove files producer started.
Right now it's 11:07 (09/03/2019)...
-
- attached romcenter.7z
Thanks for 7z compression !
-
repo owner Hi The problem comes from the selection. You selected 34000 files. Select the path on the tree view and it will be way faster.
I'm currently improving that part right now.
-
repo owner - changed status to open
-
Well, my path with my FullSet has over 30k files...
Anyway, when I select only specific files, it is somewhat faster, but still, compared to CMP it really is slow. For the moment I removed the removing of useless files/roms from the Fix.
-
repo owner If you select 32000 files, romcenter will query 32000 times the database (x hours). If you select only the full path (1 item), it will do a global query (10s).
-
repo owner CMP does not show the status of each file, and it does a rebuilding of a set. This is why it is faster. It is not the same usage.
-
repo owner Fixed
-
repo owner - changed status to resolved
-
repo owner - removed milestone
Removing milestone: 4.0 (automated comment)
- Log in to comment
Ok, I will check that. Thanks for the info about slow operations.