1. Felix Krull
  2. dx-SavegameCompression

Overview

Transparently Compressed Savegames for Deus Ex
https://bitbucket.org/fk/dx-savegamecompression
==============================================
Deus Ex savegames can get rather big. It's even worse for some mods (looking at you, The Nameless Mod [tnm]). This package is intended to make them smaller via two tricks:
1. All (newly-written) savegame files are transparently compressed (using the Deflate algorithm as implemented in Zlib [zlib]).
2. Deus Ex often copies and reuses files from older savegames which can result in a lot of duplicate files under some circumstances. So instead of copying, we'll be creating hardlinks instead [hardlinks]. By sharing data this way, the amount of unique disk space used by a savegame can ultimately be very small. Don't fear though: There's no danger of dangling links or one savegame changing another, unrelated one.

Let's have an example: I had a savegame in Hong Kong with an uncompressed size of ~24.4 MB. Fully compressed, that same savegame had a size of ~7.7 MB -- just about a third of the normal size. But wait, it gets better: That savegame is on the Canals map, the file for which is ~1.5 MB big. Let's assume you'd load that savegame, then immediately quicksave. In this case, the only files that need to be changed for that quicksave are 06_HongKong_WanChai_Canal.dxs -- the state of the Canals map -- and SaveInfo.dxs (compressed about 13 kB); all other files are shared with the original savegame, so ultimately the new quicksave only uses about 1.5 MB of unique diskspace instead of ~24 MB. So in short, you get a ~66% reduction for free and potentially additional savings. It's quite neat, if I may say so.

A few caveats, however:
1. It currently only works for plain Deus Ex (though with mods like Shifter) or TNM. See the readme of the Plugins package for an in-depth explanation, but in short, a (very simple) mod-specific support class would be needed for most mods.
2. For the hardlinking part to work, your Deus Ex savegame folder needs to live on an NTFS volume. If it doesn't, you won't get the benefits of the hardlinks; instead, copies will be created as before. The same goes for people using Kentie's launcher (the one that stores savegames in your My Documents folder): for technical reasons, no hardlinks will be created in that case, everything will be traditional copies.


Installation
============
Extract the contents of this package to your DeusEx\ directory. Then, open DeusEx\System\DeusEx.ini, find the line "DefaultGame=DeusEx.DeusExGameInfo" and change it so that it reads "DefaultGame=Plugins.DeusExPluginLoader".

Additional steps for TNM:
Open DeusEx\TNM\System\TNM.ini and change the line "DefaultGame=TNM.TNMGameInfo" to "DefaultGame=TNMPluginLoader.TNMPluginLoader". Optionally, if you like your Deus Ex installation tidy, move DeusEx\System\TNMPluginLoader.u to DeusEx\TNM\System\.

Additional steps for HDTP Release 1:
If you've installed HDTP Release 1 *before* making the changes explained above to your DeusEx.ini, open DeusEx\System\HDTP.ini and apply the same change as for DeusEx.ini.


Advanced Configuration
======================
If you setup the package as described above, it will, be default, compress and hard-link all newly-written savegame files as appropriate (if you want to know what the hell that actually implies, check out the section about how Deus Ex saves). To tweak that behaviour, you can use a range of additional configuration options; all these go into a "[SavegameCompression.SavegameCompressionPlugin]" section in DeusEx.ini or TNM.ini or HDTP.ini, depending on which options you installed.

- bEnabled: bool, default: true
Set to false to completely disable anything this plugin normally does. Notably, it won't even decompress any compressed savegames you might have. This is pretty much the same as not having the package installed, actually.

- bEnableCompression: bool, default: true
Determines whether savegame files should be compressed; if you set it to false, newly-written files won't be compressed, but old compressed files will still be transparently decompressed.

- OutputFormat: string, default: "ZLIB"
Determines which compression format to use for new files. The only supported format is currently "ZLIB" which simply compresses savegame files using the Deflate algorithm. Note that if additional formats were to be supported, it would always be possible to correctly read any supported format irrespective of this setting.

- bEnableLinking: bool, default: true
Whether to create hardlinks instead of copies of files. You might want to disable this if your Deus Ex savegame directory lives on a volume that doesn't use NTFS.


Utility Scripts
===============
A number of potentially-useful tools have been bundled:

- count-ntfs-hardlinks: Counts all hardlinks to a file. Easiest use is to drag-and-drop a file onto the executable.
- compress-savegames: Compresses savegame files with Deflate. Just drop a savegame folder onto the executable (don't worry, you'll get a backup).
- decompress-savegames: Decompresses savegame files and resolve hardlinks (i.e. the output files won't be shared with any other savegame). Simply drop a savegame folder onto the executable. Useful to cleanse a savegame so you can load it with regular Deus Ex.


Building from Source
====================
(I'm assuming that you have some knowledge about compiling C++ and UnrealScript code on Windows.)

To compile the source code yourself, you'll need:
1. The Deus Ex SDK [dxsdk]
2. My Plugins package for Deus Ex [plugins].
3. For the native part (SavegameCompression.dll):
  a) Microsoft Visual Studio 2010. The free Visual C++ 2010 Express variant should do [vs10].
  b) Boost [boost]. You'll need all the headers as well as DLLs for Boost.Iostreams with Zlib support for 32-bit MSVC2010. You can compile it yourself or use a binary installer [boostbinary].

Put the source code into your Deus Ex installation directory (so that you have DeusEx\SavegameCompression\Classes, DeusEx\SavegameCompression\Inc and so on). Copy the "boost" directory that contains the Boost headers from your Boost installation to DeusEx\SavegameCompression\external\include\ and the required DLL and LIB files into DeusEx\SavegameCompression\external\lib; they should be called something like:
 - boost_iostreams-vc100-mt-1_49.{dll,lib}
 - boost_zlib-vc100-mt-1_49.{dll,lib}
Then open SavegameCompression.sln and choose Build -> Build Solution. This will compile the code and copy all required files to DeusEx\System.

Then just compile the UnrealScript part like any other Unreal package: Add "EditPackages=SavegameCompression" to the "[Editor.EditorEngine]" section in your DeusEx.ini file and run "ucc make". For this to work, you'll either need DeusEx\System\Plugins.u or have the source for that package set up as explained in its accompanying readme. You'll still need to change the "GameInfo" line in DeusEx.ini as detailed above if you haven't done that yet.


How Deus Ex Saves (and Loads)
=============================
As a short disclaimer, the following might not describe *exactly* what happens. But it's what I gleaned from logs and then extrapolated and might be useful to someone.

Every Deus Ex savegame consists of one file per map that you have visited plus a SaveInfo.dxs file that probably stores some metadata about the savegame. These are all stored in a folder inside DeusEx\Save. For each map that you have visited, the state of that map is stored in a file <Mapname>.dxs. The state of old maps is discarded if it isn't needed anymore, i.e. once you can't return to an old map after a transition (I think that is handled according to the mission number, that number in front of the map name; when it changes, the state of older maps is discarded).

Now, when a savegame is loaded, this happens:
1. The contents of the savegame folder are copied to Save\Current.
2. The game checks Save\Current\SaveInfo.dxs to see which map the player is on and loads the map as well as the state for that map as stored in the savegame.

On a map change, this happens:
1. The state of the old map is saved in the appropriate file in Save\Current. I'm not sure if this happens even if the state of that map is discarded afterwards.
2. Optionally, the state of the old maps is discarded. I don't know when exactly this happens, but it doesn't matter for my purposes anyway, so whatever.
3. The new map is loaded; if there's state for that map in Save\Current, it's also loaded.

When saving, this happens:
1. The contents of Save\Current are copied to the newly-created/emptied savegame folder.
2. The state of the currently loaded map is written to the corresponding file in the savegame folder (this might actually happen first, no idea).
3. SaveInfo.dxs is updated accordingly.


Legal stuff
===========
Copyright (c) 2012 Felix Krull <f_krull@gmx.de>
All rights reserved.
This software is distributed under the terms of the 2-clause BSD license. The full license text can be found in the file License.txt. In short, you can distribute copies and derived works so long as you include the copyright notice and the disclaimer.

Boost is distributed under the terms of the Boost Software License [boostlicense].

Zlib is distributed under the terms of the zlib License [zliblicense].


Links
=====
[tnm] http://thenamelessmod.com
[zlib] http://zlib.net
[hardlinks] http://en.wikipedia.org/wiki/Hard_link
[dxsdk] http://www.moddb.com/games/deus-ex/downloads/sdk-v1112fm
[plugins] https://bitbucket.org/fk/dx-plugins
[vs10] http://microsoft.com/visualstudio/express
[boost] http://boost.org
[boostbinary] http://boostpro.com/download
[boostlicense] http://boost.org/users/license.html
[zliblicense] http://zlib.net/zlib_license.html