Can't read header with a large number of records

[Former user]

ESM files may contain a large number of records (greater than 2^14) in their header when they override sizable portions of vanilla records. A good example is USKP 2.1.2 where many Skyproc-based users are reporting issues.

As I've written there,

Considering that Skyproc's report on the problem is "skyproc.exceptions.BadRecord: HEDR doesn't know what to do with a XXXX record" and that the number of entries within the Header record have exceeded the 2^14 limit, it's possible that Skyproc is using a data structure that can no longer contain them all. This (temporary) link contains two versions of a test ESM file; the one that works in Skyproc contains less than 2^14 entries, the one that doesn't contains more.

The attached SkyProcDebug file contains the relevant report from the non-working example.

I'm setting this to critical because it's affecting all USKP users which is a pretty sizable number.

[David Tynan]

Thanks for the detailed report. I'll take a look asap.

[David Tynan]

Unfortunately this is buried in code I'm not really familiar with and I'm going out of town in for the next few days so there won't be an immediate fix. However there is a temporary solution. Run the patcher using the version of the USKP then once all patches are complete replace it with the full version. ONAM entries are not used when generating patches (or as far as I'm aware by anything but the skyrim engine itself) so the generated patches should be identical to what would be made if it could parse the full esp.

[David Tynan]

For any contributors following the issue is the very large number of ONAM entries made the size of the ONAM greater than can be stored in a 16bit int which is what subrecords use. So it instead has an XXXX subrecord before the ONAM which contains the size of the ONAM (or more generally XXXX contains the size of whatever record comes after it). So the function to extract the next record from a plugin needs to be updated to handle XXXX subrecords.

[ogerboss]

I can confirm your assessment. I tortured my SkyProc patch with a debugger yesterday to figure this out. (But unfortunately my ISP has problems at presents, so I could not post my findings earlier.)

To be more precise, there are ten additional bytes:

• 4 chars for "XXXX"
• 2 bytes for the size of the XXXX record (yields value 4)
• 4 bytes to form a 32bit integer giving the size of the data block in the follow up code (this excludes the header and size bytes)

The value from the 4 bytes size block is exactly equal to the number of FormIDs stored in the ONAM record multiplied by 4 (byte).

Do you think that this "XXXX" will have a reasonable chance to be used anywhere else in the plugin? I think just hacking it in as a special sub-record of the HEDR block will be much easier than a general change. After all, this new subrecord contradicts SkyProc's current design principle that each (sub)record knows about his own size...

p.s. I will give implementing a fix for it a try this evening. :)

[ogerboss]

Arthmoor mentioned that the XXXX prefix might also appear in context of other records, like e.g. large NavMeshes. So it would be preferable to implement a general change to the parser instead of just adding a specialized hack to the HEDR record parsing.

[ogerboss]

I just created a pull request, please see pull request #6 for more details.