Fix process extremely slow on large collections

[Former user]

Hello,

I use RC4 to manage a lot of different ROM collections. For the ones with a few hundred/thousand files, RC is quite fast.

But my MAME collection is quite large (~30k files), and it takes hours (literally, more than 8 hours) when I launch a fix on the whole collection. The parts of the process that are slow are the "Remove useless files" and "Remove useless roms" fixes. These take a really long time, and the computer doesn't appear to be doing much during that time (max 15% CPU usage, no disk activity).

My MAME collection is in ZIP format, I have a 4 core (8threads) CPU, 16GB of RAM, and fast SSDs

[Eric Bole-Feysot]

Ok, I will check that. Thanks for the info about slow operations.

[Eric Bole-Feysot]

Can you attach the log ? (C:\ProgramData\romcenter\romcenter_xml.log) Zip it and post it in this issue.

[Benoît Guillemaud]

Eric,

I'll launch a complete fix process tonight and send you the log tomorrow morning.

[Benoît Guillemaud]

So, I launched a fix on my clean collection (34052 zip archives, 56 Go total) last night, it is still stuck (approximately 10 hours later...) on the first step : "Remove useless files". Which is weird since according to the counter on top of the file list, there aren't any.

The Romcenter process is using 15% CPU (1 full core), and is accessing my SSD (about 80Mo/s in write, but no read).

I'm probably going to stop this fix, do you still want the log file ? (it weighs more than 1 Go unzipped, I wasn't able to open it in Chainsaw; romcenter_xml.1.log is over 5Go).

[Benoît Guillemaud]

Log4View is much better than Chainsaw, I was able to open the log file, here are the last lines :

Date    Temps   Enregistreur    Message
08/03/2019  21:26:38,523    Romcenter   Command 'All' launched
08/03/2019  21:26:38,559    Romcenter   34052 items selected
08/03/2019  21:26:40,433    Romcenter   Rename files
08/03/2019  21:26:40,433    Romcenter   Consumer started, max tasks = 6
08/03/2019  21:26:40,437    Romcenter   Rename files producer started.
08/03/2019  21:31:53,515    Romcenter   Renaming files...
08/03/2019  21:31:53,515    Romcenter   0 files to process
08/03/2019  21:31:53,515    Romcenter   Enter locked DatasQueue section for completeAdding
08/03/2019  21:31:53,515    Romcenter   Leave locked DatasQueue section for completeAdding
08/03/2019  21:31:53,515    Romcenter   Producer Datas Queue for RomCenter.Engine.Operations.RenameFiles completed
08/03/2019  21:31:53,515    Romcenter   exception: The operation was canceled.
08/03/2019  21:31:53,515    Romcenter   exception: The operation was canceled.
08/03/2019  21:31:53,515    Romcenter   Consumer closed. Waiting for operations to finish.
08/03/2019  21:31:53,597    Romcenter   Wait for remaining tasks : 0 tasks
08/03/2019  21:31:53,597    Romcenter   All operations finished.
08/03/2019  21:31:53,597    Romcenter   0 files updated
08/03/2019  21:31:53,597    Romcenter   Consumer for 'RenameFiles' ended. Status: RanToCompletion
08/03/2019  21:31:53,597    Romcenter   Consumer started, max tasks = 6
08/03/2019  21:31:53,597    Romcenter   Remove useless files
08/03/2019  21:31:53,597    Romcenter   Remove files producer started.

Right now it's 11:07 (09/03/2019)...

[Benoît Guillemaud]

Thanks for 7z compression !

[Eric Bole-Feysot]

Hi The problem comes from the selection. You selected 34000 files. Select the path on the tree view and it will be way faster.

I'm currently improving that part right now.

[Benoît Guillemaud]

Well, my path with my FullSet has over 30k files...

Anyway, when I select only specific files, it is somewhat faster, but still, compared to CMP it really is slow. For the moment I removed the removing of useless files/roms from the Fix.

[Eric Bole-Feysot]

If you select 32000 files, romcenter will query 32000 times the database (x hours). If you select only the full path (1 item), it will do a global query (10s).

[Eric Bole-Feysot]

CMP does not show the status of each file, and it does a rebuilding of a set. This is why it is faster. It is not the same usage.

[Eric Bole-Feysot]

Fixed

[Eric Bole-Feysot]

Removing milestone: 4.0 (automated comment)