Compressed reconstruction successful, but SFV still fails

[Gfy]

http://www.srrdb.com/release/details/Shadowgate_2014_Steam_Edition_FIX_FOR_UAC_READNFO-FLT

Good RAR version detected: 2014-06-10 5.10
2014-06-10_rar510.exe a -m1 -mdB -s- -ds -mt3 -vn -o+ -ep -idcd temp\tmpu89i7p_pyReScene\pyReScene_compressed.rar \fairlight.nfo
Compressing fairlight.nfo...

[Former user]

This seems to happen because for small files, WinRar writes the CRC hash of the uncompressed (instead of compressed) data into the respective file header field FILE_CRC. Since pyReScene relies on this value to find out when it discovered the correct settings, reconstruction doesn't work, because the hash of the uncompressed data is obviously independent of any compression settings.

As an improvement, pyReScene should check if the final output archive really matches the sfv file, and if not, continue with different settings instead of giving up (this will take a lot more time, but since this issue is only related to small files, it should be reasonable).

[Gfy]

I didn't know that and will investigate when fixing the small file support.

This bug however, is about a successful compression (hash and size matches), but the end result is still wrong.

[Da Fox]

Using master branch at 312781f I ran into the same problem with https://www.srrdb.com/release/details/Homeworld.Deserts.of.Kharak.Update.v1.2.incl.DLC-CODEX All files seemingly reconstruct OK, but afterwards the first (.rar) and last (.r38) volumes still failed CRC. By playing around with the -mt-set option I was able to get the last volume to reconstruct correctly, but not the .rar itself (I tried a lot of different combinations, iirc the .r38 should reconstruct OK with e.g. -mt-set 11,12,16). All -mt options that are used by srr without the -mt-set option are 1,2,3,6,11,12,16. At least -mt 11 and -mt 12 seemed to be required for srr to detect a good rar version.

In the end I was able to reconstruct the .rar manually with

2016-02-03_rar531.exe a -m3 -mdG -ds -mt12 -vn -o+ -idcd -v100000000b ...

Looking at the 'bad' .rar that was generated with srr, I did determine that the problem appears to have been with one of the smaller files, not the big setup-1.bin. One of my manual attempts had compressed these files better, so there was more of the data of the setup-1.bin file in the first volume. Comparing this extra data with the data in the next volume (.r00) showed that it was an exact match. I think this indicates that the setup-1.bin file was being correctly compressed. Comparing the two .rar files, I see that the first differences start to show up in CODEX\Win32\Plugins\x86\steam_api.cdx, the third file in the archive. This file is ~187kb, I don't know if that is considered small or not (I wouldn't think so).

[Da Fox]

I ran into this again with another release (https://www.srrdb.com/release/details/Assassins.Creed.IV.Black.Flag.Update.v1.06-RELOADED), with the first .rar file. I was able to work around it again using different -mt settings.

Once I had a good .rar file (matching the .sfv) I investigated a little more, and it seems that dokterkere was indeed correct. Comparing the good and bad .rar, the first file that mismatches is steam_api.ini (2111 bytes). In both .rar files this file was compressed to the same size (670 bytes), but the compressed data differs. The compressed data starts out identically, but after 0x3d bytes it starts to diverge quickly.

Since the checksum in the .rar is that of the uncompressed file, it is the same in both .rars. And since the compressed data is equally big, rescene has no "choice" but to accept it as a good compression, even if it is not.

One requirement that rescene does not seem to take into account yet seems to be that all files should be compressed with the same -mt setting. So if an -mt setting has been ruled out by previous files, rescene should probably not try it for later files. And the reverse also holds, if a later file fails to compress with a -mt setting that worked for other files, those files should probably be recompressed with a different -mt setting. This may help to avoid this scenario, but I'm not sure if it can ever be fully eliminated this way, so the final check that dokterkere mentions is also needed.