Feature Proposal: Extraction Cache For Compressed Files

[John Vilk]

Scenario

I would like to use LaunchBox to access moderately large compressed games on a network drive. The network drive may be mounted over a wireless connection with limited throughput.

However, the issue described also persists in environments where the drive is local, as extraction times can be large for particular files.

Current Limitation

Games are re-downloaded and re-extracted every time they are booted up. This process may take multiple minutes or more for larger games.

Users with games on a local hard drive also have this problem, but it is exacerbated when the games are on a network drive.

Current Design

LaunchBox extracts a game into a temporary directory on the user's primary hard drive when the game is launched, and deletes the files once the game closes.

Proposal

LaunchBox lets the user configure an extraction cache folder for compressed files, which defaults to the current location for extracted files, along with a cache size.

Terminology:

• Compressed file: The archive containing the game (e.g. F:\Games\game.7z)
• Extracted files: The contents of a compressed file.
• Cache size: The configured size of the cache.
• Extraction directory: The subdirectory of the cache where a compressed file was extracted.

LaunchBox should store the following metadata for each compressed file in the cache:

• File path of the compressed file
• Last modified date of the compressed file
• The last time the user selected to play the compressed file
  • NOTE: You actually don't have to store the time, and can merely keep an ordered list of last-played items in the cache. But for clarity, I'm describing it like this.
• The total size of the extracted files
• The extraction directory in the cache.

When a compressed file is launched, Launchbox performs the following procedure:

• Checks if the cache has metadata for the compressed file using the file's path.
  • If not, see LaunchBox extracts the file into the cache (see procedure below).
  • If so, check if the modified date for the compressed file matches the modified date in the metadata.
    • If so, update the last time the user selected to play the compressed file in the metadata to the current date and time, and use the cached version.
    • If not, remove the extracted files and the metadata from the cache, and extract the compressed file into the cache (see procedure below).

Procedure for extracting a compressed file into the cache:

• If the total size of all extracted files + size of extracted files for the current compressed file < the configured cache size:
  • Extract the compressed file into the cache, and generate its metadata.
• Else:
  • Until the above is true:
    • Evict the compressed file from the cache that has the earliest last played datetime.

File names:

To avoid file name clashes, every compressed file should be extracted into its own extraction directory. Care must be taken to avoid directory name clashes. You could use something like the SHA1 of the compressed file, which are highly unlikely to clash, or use the filename of the compressed file with numbers attached when there are clashes.

Other considerations:

If cache size > the user's available hard drive space, you may want to consider lowering the cache size and warning the user.

If extracted file size > the configured cache size, you may want to consider rejecting the user's request to play the game until the cache size is amended.

There may be some caveats to relying on modification time. I think, in the worst case, LaunchBox will use a stale version of a game in the cache. But I suspect it is uncommon for users to update their compressed files in the first place! To mitigate these issues, you could also store file size in bytes as a backup comparison mechanism.

[fraganator]

I've created a tool which should perform this exact cache management. I had actually almost completed it before reading this issue submission, so reading this description was a bit like stepping into the twilight zone (right down to the hashes to avoid filename collisions). Anyways I hope it's useful. Link: http://forums.launchbox-app.com/files/file/234-archive-cache-manager/