Reconstruction fails on files with block.compression_method == COMPR_STORING

[Former user]

Archive reconstruction fails on files with block.compression_method == COMPR_STORING

Reconstruction fails here:

            # we've done files before
            if len(archived_files) >= 1:
                print("Testing with previous file")
                # try compressing with the previous rar file before it
                prev = get_previous_block()
                if prev:
                    prev_file = archived_files[prev.file_name]

archived_files does not contains entries which are written to rar archive directly without compression.

The problem seems to start here:

                        if block.compression_method != COMPR_STORING:
                            _fire(MsgCode.MSG,
                                message="Trying to rebuild compressed file %s."
                                % block.file_name)
                            srcfs = get_rar_data_object(block, blocks, src,
                                        in_folder, hints, auto_locate_renamed)
                        else: # uncompressed file
                            srcfs = open(src, "rb")

Quick reproduce data sample attached to issue (only first files left from release).

---

Logs:

For 428_Fuusa_Sareta_Shibuya_de_JPN_JB_PS3-Caravan.srr:

...
Trying to rebuild compressed file BLJS10057\PS3_GAME\USRDIR\data\rom\a01a.cpk.
Grabbing large enough data piece size for testing.
Trying 2010-11-11 4.00.
Testing with previous file
Traceback (most recent call last):
  File "C:\Tools\pyReScene-0.5.1\rescene\srr.py", line 460, in main
    parser.exit(manage_srr(options, infolder, infiles, working_dir))
  File "C:\Tools\pyReScene-0.5.1\rescene\srr.py", line 246, in manage_srr
    options.rar_executable_dir, options.temp_dir)
  File "C:\Tools\pyReScene-0.5.1\rescene\main.py", line 1146, in reconstruct
    in_folder, hints, auto_locate_renamed)
  File "C:\Tools\pyReScene-0.5.1\rescene\main.py", line 1494, in get_rar_data_object
    in_folder, hints, auto_locate_renamed))
  File "C:\Tools\pyReScene-0.5.1\rescene\main.py", line 1762, in compressed_rar_file_factory
    nblock, followup_src, solid=False)
  File "C:\Tools\pyReScene-0.5.1\rescene\main.py", line 1887, in __init__
    first_block, blocks, thread_count)
  File "C:\Tools\pyReScene-0.5.1\rescene\main.py", line 2142, in search_matching_rar_executable
    prev_file = archived_files[prev.file_name]
KeyError: u'BLJS10057\\PS3_GAME\\USRDIR\\EBOOT.BIN'
Unexpected Error: u'BLJS10057\\PS3_GAME\\USRDIR\\EBOOT.BIN'

---

For The_Idolmaster_Gravure_for_You_Vol_2_JPN_PS3-MOEMOE.srr:

Adding    N:\BLJS10139\PS3_DISC.SFB  OK
Trying to rebuild compressed file BLJS10139\PS3_GAME\LICDIR\LIC.DAT.
Grabbing large enough data piece size for testing.
Trying 2012-01-09 4.10.
Testing with previous file
Traceback (most recent call last):
  File "C:\Tools\pyReScene-0.5.1\rescene\srr.py", line 460, in main
    parser.exit(manage_srr(options, infolder, infiles, working_dir))
  File "C:\Tools\pyReScene-0.5.1\rescene\srr.py", line 246, in manage_srr
    options.rar_executable_dir, options.temp_dir)
  File "C:\Tools\pyReScene-0.5.1\rescene\main.py", line 1146, in reconstruct
    in_folder, hints, auto_locate_renamed)
  File "C:\Tools\pyReScene-0.5.1\rescene\main.py", line 1494, in get_rar_data_object
    in_folder, hints, auto_locate_renamed))
  File "C:\Tools\pyReScene-0.5.1\rescene\main.py", line 1762, in compressed_rar_file_factory
    nblock, followup_src, solid=False)
  File "C:\Tools\pyReScene-0.5.1\rescene\main.py", line 1887, in __init__
    first_block, blocks, thread_count)
  File "C:\Tools\pyReScene-0.5.1\rescene\main.py", line 2142, in search_matching_rar_executable
    prev_file = archived_files[prev.file_name]
KeyError: u'BLJS10139\\PS3_GAME\\ICON0.PNG'
Unexpected Error: u'BLJS10139\\PS3_GAME\\ICON0.PNG'

[Former user]

Alternate link to reproduce data:

COMPR_STORING_test_data.rar (74 KB):

https://mega.co.nz/#!S1lFVBQA!26FSPW3Cmp_wumM1M5JiwWTW_ztXYmgr7MGgnyg2VlE

Checked with pyrescene 0.6-beta1 and pyReScene-0.5.1

---

Also reproduces for Final_Fantasy_XIII_ASiA_MULTi3_JB_PS3-NRP.srr:

...
Adding    R:\BCAS25005\PS3_GAME\TROPDIR\NPWR00774_00\TROPHY.TRP  OK
05f317a7 74e44dd5 BCAS25005\PS3_GAME\TROPDIR\NPWR00774_00\TROPHY.TRP R:\Final Fantasy XIII HK-JAILBREAK_BY_chpeng2010\nrp-ff13cjb.rar
Trying to rebuild compressed file BCAS25005\PS3_GAME\USRDIR\white_data\movie\z000.ps3.wmp.
Grabbing large enough data piece size for testing.
Trying 2010-11-11 4.00.
Testing with previous file
Traceback (most recent call last):
  File "C:\Tools\pyrescene\rescene\srr.py", line 507, in main
    parser.exit(manage_srr(options, infolder, infiles, working_dir))
  File "C:\Tools\pyrescene\rescene\srr.py", line 268, in manage_srr
    options.volume is None, options.volume)
  File "C:\Tools\pyrescene\rescene\main.py", line 1221, in reconstruct
    in_folder, hints, auto_locate_renamed)
  File "C:\Tools\pyrescene\rescene\main.py", line 1588, in get_rar_data_object
    in_folder, hints, auto_locate_renamed))
  File "C:\Tools\pyrescene\rescene\main.py", line 1873, in compressed_rar_file_factory
    nblock, followup_src, solid=False)
  File "C:\Tools\pyrescene\rescene\main.py", line 1998, in __init__
    first_block, blocks, thread_count)
  File "C:\Tools\pyrescene\rescene\main.py", line 2254, in search_matching_rar_executable
    prev_file = archived_files[prev.file_name]
KeyError: u'BCAS25005\\PS3_GAME\\USRDIR\\EBOOT.BIN'
Unexpected Error: u'BCAS25005\\PS3_GAME\\USRDIR\\EBOOT.BIN'

[Gfy]

Compressed reconstruction fails on files with no compression. (KeyError) Fixes issue #44 and #79.

→ <<cset ab0606bc93ef>>