Support external compression library within PolyVox.

Create issue
Issue #20 resolved
David Williams created an issue

PolyVox currently includes basic Run Length Encoding compression which I implemented, and which is primarily used to compress blocks in the LargeVolume after they haven't been touched for a while. It could also have applications to serialising data to disk.

I have no significant experience with compression algorithms and so it is likely that an external library would provide a better performance and compression ratio (particularly with smooth terrain where RLE won't be very effective). Two candidates have been identified so far: - This one is desirable due to it's public domain license. We can just drop the code into PolyVox. - This one is probably the best performer, but we should think whether the license (New BSD) has any implications.

Such a library would be an external dependency in PolyVox, but it would be shipped as part of the PolyVox source tree so users would never know about it (this is why the licensing matters). It may be desirable to abstract the exact algorithm being used so that custom compressors can be plugged in if desired.

Comments (10)

  1. David Williams reporter

    Right on cue, unit tests have a now identified a problem with the LargeVolume when compression is enabled. It might not be in the actual compression algorithm but it's worth keeping in mind.

  2. David Williams reporter

    I'm having a lot of difficulties with the miniz compression library. I can use the high-level interface OK, but really I would like to use the low-level interface which avoids memory allocation by using user-supplied buffers. But it's not very well documented at this level. There are also some strange implementation choices like the whole library being in a single .c file with no header... this means you have to #include the .c file and fight 'multiply defined symbol' linker problems.

    So, I'm wondering what the implications are if we want to use a different (BSD or MIT) library and include the source code in PolyVox. My main concern is whether we can still say PolyVox is under the zlib license if the repository contains code which is under a different license. What should we do to make this clear? Have a separate subfolder with the compression code and a copy of the BSD/MIT license?

    Let me know if you have any thoughts or information on this - in the mean time I think I'll have another look at the higher level interface for miniz.

  3. illidan

    Hi David, I'm using miniz for my own project and regarding your comment of the library coming without a header, you can create one like this:

    #ifndef MINIZ_H
    #define MINIZ_H
    #include "miniz.c"

    This solved all the linking problems I had myself.

    Can't help with the other stuff but I wish you good luck, can't wait to see how well the new compression in PolyVox wil work :)

  4. David Williams reporter

    Thanks for the comment, I've done something kind of similar except I've wrapped miniz.c in my own .cpp (Compression.cpp) file and then made a header for that (Compression.h) and this seems to work ok. I'm also simplifying the interface so there is a single compress/decompress function to worry about, and it will also be possible for users to implement the interface with a different library if they wish.

    As for the actual miniz implementation, I think I'm going to use the high level version for now and come back to the low level version in the future (if necessary).

    I'm also aware that some kind of streaming compression might be useful in the future (for saving volumes to disk) but I won't worry about that for now.

  5. David Williams reporter

    I've just merged a lot of code from the large-volume-work branch (now deleted) into develop. It includes a base 'Compressor' interface and two implementations based on RLE and miniz. Initial impressions are that RLE is both faster and better on cubic data, but does not perform well on smooth data (as expected) so both compressors are useful.

    I still need to do some more tidying and testing.

  6. David Williams reporter

    I'm assigning this task to Matt because I'm not sure if he get's notifications otherwise (?), not because it's his problem.

    I've merged in some compression changes but I've broken the C# bindings in the process. It's probably as easy fix but I don't know where to start. Matt, can you point me in the right direction or fix this yourself when you get a chance (if it's easy, otherwise leave it to me)?

    CDash error here:

  7. Matt Williams

    Interesting, I indeed hadn't been getting notifications on this thread. I got the original email and then a second one for the above comment but had missed all the intervening discussion. Generally you should be able to explicitly mention someone like @DavidWilliams and it should notify them. However it would be good for us if we were emailed about everything so we can both keep track of goings-on.

    About the C# bindings, it's probably an easy fix but I will look into it myself. Most likely it's just missing a heading include in the SWIG files or something.

  8. David Williams reporter

    Ok, I think this task is basically finished. I got the low-level version of miniz working in the end, and have finally managed to suppress some warning from GCC that I didn't want to fix (inside miniz.c).

    I'm going to leave this task open until I write some documentation though.

  9. Log in to comment