warc-tools /

Filename Size Date modified Message
152 B
Added some more output files to .hgignore.
551 B
Added tag build_success-2012-09-14T16-25-56.483325901 for changeset 741fe327f233
1.1 KB
licensing update & clarification MIT license
2.5 KB
Whitespace change to test build system.
2.2 KB
Added ability to specify output directory in build script.
9.0 KB
Exclude a few worthless messages from pylint.
956 B
adding aliases for scripts
3 B
Changed version tracking to use a version number stored in a file.
7.5 KB
reporting collisions in names generated
setuptools unittest2 python 2.6

hanzo warc tools:

returns 0 if the arguments are all valid arc/warc files non zero on error
warcdump.py - writes human readable summary of warcfiles:

usage: python warcdump.py foo.warc foo.warc.gz autodetects input format when filenames are passed i.e recordgzip vs plaintext, warc vs arc

assumes uncompressed warc on stdin if no args

python warcfilter.py pattern file file file
searches all headers for regex pattern

use -i to invert search use -U to constrain to url use -T to constrain to record type use -C to constrain to content-type

autodetects and stdin like warcdump

prints out a warc format by default.


python warc2warc <input files>

autodetects compression on file args, assumes uncompressed stdin if none

use -Z to write compressed output

i.e warc2warc -Z input > input.gz

should ignore buggy records in input

creates a crappy warc file from arc files on input a handful of headers are preserved use -Z to write compressed output i.e arc2warc -Z input.arc > input.warc.gz
spits out an index like this:

#WARC filename offset warc-type warc-subject-uri warc-record-id content-type content-length warccrap/mywarc.warc 1196018 request /images/slides/hanzo_markm__wwwoh.pdf <urn:uuid:fd1255a8-d07c-11df-b125-12313b0a18c6> application/http;msgtype=request 193 warccrap/mywarc.warc 1196631 response http://www.hanzoarchives.com/images/slides/hanzo_markm__wwwoh.pdf <urn:uuid:fd2614f8-d07c-11df-b125-12313b0a18c6> application/http;msgtype=response 3279474

not great, but a start


arc2warc uses the conversion rules from the earlier arc2warc.c as a starter for converting the headers

I haven't profiled the code yet (and don't plan to until it falls over)

warcvalid barely skirts some of the iso standard:
missing things: strict whitespace, required headers check... mime quoted printable header encoding treating headers as utf8
things left to do (in no order):
lots more testing. supporting pre 1.0 warc files add more documentation support more commandline options for output and filenames s3 urls

-- tef thomas.figg@hanzoarchives.com