1. Mike Speriosu
  2. updown

Commits

<spe...@gmail.com  committed b94c057

  • Participants
  • Parent commits 5a5b114
  • Branches default

Comments (0)

Files changed (1)

File Home.wiki

View file
 ***** PER USER EVAL *****
 Number of users evaluated: 0 (min of 3 tweets per user)
 Mean squared error: NaN
+}}}
+
+Point the -g flag to other preprocessed feature files and the -f flag to the corresponding username-username-edges.txt file to run the label propagation on other datasets.
+
+=== Per-Target Evaluation ===
+
+Tweets in the HCR datasets are annotated for target as well as sentiment. To extract the list of targets for one of the HCR datasets (necessary to perform per-target evaluation), add a third argument before the '>' to the HCR preprocessing command, where that argument is a target output filename. For example, this will extract the targets from HCR-dev:
+
+{{{
+$ updown preproc-hcr data/hcr/dev/orig/hcr-dev.csv src/main/resources/eng/dictionary/stoplist.txt data/hcr/dev/hcr-dev-targets.txt > data/hcr/dev/hcr-dev-features.txt
+}}}
+
+Whenever running the above experiments on an HCR dataset for which targets have been extracted, you can point to the appropriate target file with the -t flag and see a breakdown of results per target. For example, this command will run per-target evaluation on HCR-dev after performing the default label propagation:
+
+{{{
+$ updown 8 junto -g data/hcr/dev/hcr-dev-features.txt -m models/maxent-eng.mxm -p src/main/resources/eng/lexicon/subjclueslen1polar.tff -f data/hcr/username-username-edges.txt -r src/main/resources/eng/model/ngramProbs.ser.gz -t data/hcr/dev/hcr-dev-targets.txt 
+}}}
+
+You should see the following output:
+{{{
+***** PER TWEET EVAL *****
+Accuracy: 0.6516854 (348.0/534)
+
+***** PER USER EVAL *****
+Number of users evaluated: 24 (min of 3 tweets per user)
+Mean squared error: 0.12439673091458808
+
+***** PER TARGET EVAL *****
+hcr: 0.6298932384341637 (281)
+gop: 0.5822784810126582 (79)
+other: 0.7446808510638298 (47)
+dems: 0.7567567567567568 (37)
+conservatives: 0.5714285714285714 (35)
+stupak: 0.75 (24)
+obama: 0.8235294117647058 (17)
+teaparty: 0.8181818181818182 (11)
+liberals: 0.3333333333333333 (3)
 }}}