lzjb-erlang /

Filename Size Date modified Message
src
1.1 KB
75 B
3.0 KB
who:   matt@genges.com
what:  erlang driver for lzjb compression (stolen from ZFS).
when:  made January 2009. published January 2009.
where: everywhere
why:   everybody can find a use for a fast compression system.

prereqs:
  - nil

performance: not tested for throughput yet
benchmark versus [something else]:
  not tested yet

installation: build it.  
              edit src/Makefile if erlang isn't in /usr/local/lib/erlang
              put everything in dist/ where you need it to be

usage [APPLICATION]:
% export LD_LIBRARY_PATH=./lzjb-erlang/lzjb-VER/priv/ 
% erl -pz ./lzjb-erlang/lzjb-VER/ebin/

Eshell V5.6.5  (abort with ^G)
1> application:start(lzjb).    % optional
ok.
2> lzjb:start().
<0.37.0>
3>  A = lzjb:compress(<<"some binary data to compress">>).           
{28,
 <<0,115,111,109,101,32,98,105,110,0,97,114,121,32,100,97,
    116,97,0,32,116,111,32,99,111,109,112,...>>}
4> lzjb:decompress(A).
<<"some binary data to compress">>
5>
6> Z = lzjb:compress_term({something, [to, compress]}).
{37,
 <<0,131,104,2,100,0,9,115,111,0,109,101,116,104,105,110,
    103,108,0,0,0,0,2,100,0,2,116,...>>}
7> lzjb:decompress_term(Z).
{something,[to,compress]}


usage [DIRECT]:  % Needs to be refined.  Too raw currently.
% export LD_LIBRARY_PATH=./lzjb-erlang/lzjb-VER/priv/ 
% erl -pz ./lzjb-erlang/lzjb-VER/ebin/

Eshell V5.6.5  (abort with ^G)
1> application:start(lzjb).    % optional
ok.
2> Port = lzjb:make_port().
#Port<0.595>
3> port_command(Port, lzjb:encode_compress(<<"port data to compress">>)).
true
4> receive Data -> Data end.
{#Port<0.595>,
 {data,<<0,112,111,114,116,32,100,97,116,0,97,32,116,111,
          32,99,111,109,0,112,114,101,115,115>>}}
5> {Port, {data, CompressedData}} = Data.
{#Port<0.595>,
 {data,<<0,112,111,114,116,32,100,97,116,0,97,32,116,111,
          32,99,111,109,0,112,114,101,115,115>>}}
6> OriginalDataSize = size(<<"port data to compress">>).
7> port_command(Port, lzjb:encode_decompress(CompressedData, OriginalDataSize)).
true
8> receive DecompressedDataTerm -> DecompressedDataTerm end.
{#Port<0.595>,{data,<<"port data to compress">>}}
9> {Port, {data, DecompressedFoundData}} = DecompressedDataTerm.
{#Port<0.595>,{data,<<"port data to compress">>}}
10> DecompressedFoundData.
<<"port data to compress">>


Notes:
  - The driver must be in your LD_LIBRARY_PATH.
  - The compressed format doesn't contain the original size of the data, so
    we include it manually as an unsigned int.
  - The maximum size of one binary to compress 4GB.
  - Only compress things needing compression.  If your data has little
    to no repetition or is less than few dozen bytes, the compression can't do 
    anything except add markers saying "this isn't compressed" which make
    your "compressed" data larger than you started with by a few bytes.
    Compress largeish text entries or large terms with repeated data.
    Don't compress a secondary index lookup of just a primary key.

TODO:
  - Throughput testing.
  - Testing multithreaded (+A64) versus not (+A0) performance.
  - Add unit tests.
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.