7z Archives Getting Corrupted

[Andrew Quesnel]

I've been trying to cleanup my ROM collection and gave Romcenter a try. When I try to fix all my ROMs the majority of them would get corrupted (like 90% or so). Looking at the logs I saw these messages over and over again:

Invalid archive: open/read error! Is it encrypted and a wrong password was provided?
If your archive is an exotic one, it is possible that SevenZipSharp has no signature for its format and thus decided it is TAR by mistake.

Out of curiosity I grabbed a newer version of the 7z.dll from the SevenZipSharp repository, grabbed a clean version of some of the ROMs that got corrupted, and tried again. This is where it gets weird. Fixing a single archive at a time works just fine with the new DLL. However, selecting multiple ROMs fails and corrupts some of the ROMs. The number of ROMs that will work or get corrupted is not consistent. In some of my tests I was able to fix 3-4 without corrupting any but then in other test fixing 2 would corrupt both. Again, if I take clean versions of the ROMs that would get corrupted and fix them one by one they work no problem (as long as I'm using the newer version of the DLL).

[Andrew Quesnel]

I decided to go one by one through all my ROMs. I found that occasionally a file would end up corrupted even doing this one by one. For the files that would fail I would try again until they worked. Sometimes it would work the first re-try, sometimes I would try 4 times, restart Romcenter, then it would work the first time. Very weird.

[Andrew Quesnel]

After more testing this seems to be the same issue as #259

I grabbed the beta copy and it seems to work a whole lot better but I’m still getting a good number of corrupted files, around 20%. I understand the RAM limitations of a 32bit app but looking at task manager while running Romcenter I'm not seeing memory usage over about 300mb

Edit

I decided to try a bunch of random things and setting the number of threads to 1 made a big difference. I’d still occasionally get corrupted archives, but way less often. Romcenter even handled some really large archives without issue. Doing a bit of Googling I think this issue has less to do with RAM and more to do with this: https://docs.microsoft.com/en-us/troubleshoot/windows-client/application-management/0x8007000e-call-objects-from-one-process-to-another-with-complus

[Eric Bole-Feysot]

Hi. Thanks for the work. Can you give a try to the version 4.2 beta here: https://www.romcenter.com/forum/viewtopic.php?f=5&t=3568

[Andrew Quesnel]

Thank you for the suggestion! I gave the beta a try and it seems way more stable but I was still getting quite a few corrupted ROMs. After seeing that document from Microsoft I decided to try changing the number of threads to 1 and that made a pretty big difference. I still get the occasional corrupted ROM but it’s down to something like 5%-10% corruption rate.

One thing I did notice was that the corruption seems to happen when the ROM is being analyzed. It looks like while analyzing the 0x8007000e error will occur and somehow clear the contents of the ROM file. When a ROM gets corrupted I’m left with a 33 byte file that isn’t even an archive any more (I tried opening it with 7Zip and it throws an error)

Thinking more about that MS document and the fact that lowering the thread count makes a significant change I wonder if you could call the 7Zip using PInvoke instead of COM and see if that makes a difference. You would need to write your own wrappers for 7Zip though.

I was planning to go through another set of ROMs later tonight, if you have debug symbols or are willing to share the source code I could hook up a debugger and see what’s going on. I’m a .NET developer so I’d be pretty comfortable giving that a try if it’ll make things easier for you 😃

P.S. This tool is awesome compared to the others I’ve tried, it’s way more user friendly, great work!

[Eric Bole-Feysot]

I can share the source code. It’s quite difficult for me to debug these cases, as the error didn’t occur on my pcs.
Do you have a Bitbucket account ?

[Eric Bole-Feysot]

Also, you could try to compile in 64b and see if it fixes the problem.

[Andrew Quesnel]

Yup the account I’m posting under is my Bitbucket account, the username is amura11

[Eric Bole-Feysot]

After looking at the permissions form, I need the email address…

[Eric Bole-Feysot]

I sent an invitation. Please use the 4.0 branch, as it is the up to date version.

[Eric Bole-Feysot]

Can you tell me where exactly the error occurs and what is the exception type. I will raise it and handle it correctly to avoid zip corruption.

[Andrew Quesnel]

Sorry, just got around to testing this today. I was able to recreate the error, the exception is thrown on line 519 of SevenZipContainer.cs. So it's an issue with SevenZipSharp. I’m going to see if I can recreate the issue using only SevenZipSharp and raise it with them if I can get it to happen consistently.

[Eric Bole-Feysot]

Great. If you manage to throw it consistently, maybe you can create a test method (or explain how to produce the error so I can create the test) to be sure it won’t be back in the future.

[TJ]

I have found that if I have a solid 7z archive compressed as LZMA2:768m, an error will be thrown consistently when trying to fix that single archive file, resulting in the 7z file becoming corrupted. Let me know if you need any help with testing.