Fix process extremely slow on large collections

Create issue
Issue #144 resolved
Former user created an issue


I use RC4 to manage a lot of different ROM collections. For the ones with a few hundred/thousand files, RC is quite fast.

But my MAME collection is quite large (~30k files), and it takes hours (literally, more than 8 hours) when I launch a fix on the whole collection. The parts of the process that are slow are the "Remove useless files" and "Remove useless roms" fixes. These take a really long time, and the computer doesn't appear to be doing much during that time (max 15% CPU usage, no disk activity).

My MAME collection is in ZIP format, I have a 4 core (8threads) CPU, 16GB of RAM, and fast SSDs

Comments (15)

  1. Eric Bole-Feysot repo owner

    Can you attach the log ? (C:\ProgramData\romcenter\romcenter_xml.log) Zip it and post it in this issue.

  2. Benoît Guillemaud

    So, I launched a fix on my clean collection (34052 zip archives, 56 Go total) last night, it is still stuck (approximately 10 hours later...) on the first step : "Remove useless files". Which is weird since according to the counter on top of the file list, there aren't any.

    The Romcenter process is using 15% CPU (1 full core), and is accessing my SSD (about 80Mo/s in write, but no read).

    I'm probably going to stop this fix, do you still want the log file ? (it weighs more than 1 Go unzipped, I wasn't able to open it in Chainsaw; romcenter_xml.1.log is over 5Go).

  3. Benoît Guillemaud

    Log4View is much better than Chainsaw, I was able to open the log file, here are the last lines :

    Date    Temps   Enregistreur    Message
    08/03/2019  21:26:38,523    Romcenter   Command 'All' launched
    08/03/2019  21:26:38,559    Romcenter   34052 items selected
    08/03/2019  21:26:40,433    Romcenter   Rename files
    08/03/2019  21:26:40,433    Romcenter   Consumer started, max tasks = 6
    08/03/2019  21:26:40,437    Romcenter   Rename files producer started.
    08/03/2019  21:31:53,515    Romcenter   Renaming files...
    08/03/2019  21:31:53,515    Romcenter   0 files to process
    08/03/2019  21:31:53,515    Romcenter   Enter locked DatasQueue section for completeAdding
    08/03/2019  21:31:53,515    Romcenter   Leave locked DatasQueue section for completeAdding
    08/03/2019  21:31:53,515    Romcenter   Producer Datas Queue for RomCenter.Engine.Operations.RenameFiles completed
    08/03/2019  21:31:53,515    Romcenter   exception: The operation was canceled.
    08/03/2019  21:31:53,515    Romcenter   exception: The operation was canceled.
    08/03/2019  21:31:53,515    Romcenter   Consumer closed. Waiting for operations to finish.
    08/03/2019  21:31:53,597    Romcenter   Wait for remaining tasks : 0 tasks
    08/03/2019  21:31:53,597    Romcenter   All operations finished.
    08/03/2019  21:31:53,597    Romcenter   0 files updated
    08/03/2019  21:31:53,597    Romcenter   Consumer for 'RenameFiles' ended. Status: RanToCompletion
    08/03/2019  21:31:53,597    Romcenter   Consumer started, max tasks = 6
    08/03/2019  21:31:53,597    Romcenter   Remove useless files
    08/03/2019  21:31:53,597    Romcenter   Remove files producer started.

    Right now it's 11:07 (09/03/2019)...

  4. Eric Bole-Feysot repo owner

    Hi The problem comes from the selection. You selected 34000 files. Select the path on the tree view and it will be way faster.

    I'm currently improving that part right now.

  5. Benoît Guillemaud

    Well, my path with my FullSet has over 30k files...

    Anyway, when I select only specific files, it is somewhat faster, but still, compared to CMP it really is slow. For the moment I removed the removing of useless files/roms from the Fix.

  6. Eric Bole-Feysot repo owner

    If you select 32000 files, romcenter will query 32000 times the database (x hours). If you select only the full path (1 item), it will do a global query (10s).

  7. Eric Bole-Feysot repo owner

    CMP does not show the status of each file, and it does a rebuilding of a set. This is why it is faster. It is not the same usage.

  8. Log in to comment