Issue #17 new

Streaming operation with n-triples

pdxleif
created an issue

First: Thanks for all your work on this!

I know I commented before about being surprised at the non-set behaviour of the graph... However, I wonder if in some cases, when working with large data sets I intend to load into another tool (4store I was thinking), where the "set union" would be done there, maybe I'd prefer to work with just a lazy list of triples (parsing the input file w/ lazy IO or something), and then write that out as n-triples. It'd get de-duped in the destination (aside from bnodes), and I worry about ensuring uniqueness causing the entire graph to be held in-memory, instead of streaming through transfering my input data to n-triples a triple at a time.

Comments (1)

  1. pdxleif reporter

    I suppose this might be a non-issue - if I'm merely transforming a stream of data, all I need to do is format it as n-triples; I know how to do that myself, and don't need to load it into a graph lib to do that.

  2. Log in to comment