Implements a portion of the TF-IDF algorithm. Takes Avro based file as input, calculates TF, DF and D portions in a batch mode and places the results into a cassandra table. This will later be read by a storm-trident DRPC query and combined with the realtime data to form a complete view of the world.


Ensure that both hadoop and cassandra are started, then:

lein deps
lein compile
lein uberjar

hadoop jar ./target/tfidf-cascalog-0.1.0-SNAPSHOT-standalone.jar data/document.avro data/en.stop

Obviously replacing the IP address with the appropriate cassandra IP address.