warc-tools /

Filename Size Date modified Message
122 B
381 B
1.0 KB
2.4 KB
3.7 KB
8.8 KB
487 B
3.2 KB
1.5 KB
1.6 KB
2.6 KB
1.4 KB
7.4 KB
1.6 KB
	python 2.6

hanzo warc tools:
        returns 0 if the arguments are all valid arc/warc files
        non zero on error - writes human readable summary of warcfiles:
        usage: python foo.warc foo.warc.gz
        autodetects input format when filenames are passed
        i.e recordgzip vs plaintext, warc vs arc

        assumes uncompressed warc on stdin if no args 
        python pattern file file file
            searches all headers for regex pattern
        use -i to invert search
        use -U to constrain to url
        use -T to constrain to record type
        use -C to constrain to content-type
        autodetects and stdin like warcdump

        prints out a warc format by default.
        python warc2warc <input files>

        autodetects compression on file
        args, assumes uncompressed stdin if none

        use -Z to write compressed output

        i.e warc2warc -Z input > input.gz

        should ignore buggy records in input
        creates a crappy warc file from arc files on input
        a handful of headers are preserved
        use -Z to write compressed output
        i.e arc2warc -Z input.arc > input.warc.gz
        spits out an index like this:
#WARC filename offset warc-type warc-subject-uri warc-record-id content-type content-length
warccrap/mywarc.warc 1196018 request /images/slides/hanzo_markm__wwwoh.pdf <urn:uuid:fd1255a8-d07c-11df-b125-12313b0a18c6> application/http;msgtype=request 193
warccrap/mywarc.warc 1196631 response <urn:uuid:fd2614f8-d07c-11df-b125-12313b0a18c6> application/http;msgtype=response 3279474
        not great, but a start

    arc2warc uses the conversion rules from the earlier arc2warc.c
    as a starter for converting the headers

    I haven't profiled the code yet (and don't plan to until it falls over)

    warcvalid barely skirts some of the iso standard:
        missing things: strict whitespace, required headers check...
	mime quoted printable header encoding
	treating headers as utf8

things left to do (in no order):
    lots more testing.
    supporting pre 1.0 warc files
    add more documentation
    support more commandline options for output and filenames
    s3 urls

-- tef

Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.