David McClosky avatar David McClosky committed 31b38b2

Initial commit.
This version is based on my latest modifications to the parser from

http://bllip.cs.brown.edu/download/reranking-parserAug06.tar.gz

Comments (0)

Files changed (327)

+# Top-level makefile for reranking parser
+# Mark Johnson, 28th July 2008 -- added EXEC prefix to major commands
+
+########################################################################
+#                                                                      #
+#                                Summary                               #
+#                                                                      #
+########################################################################
+#
+# To build just the reranking parser run-time, execute:
+#
+# make                 # builds reranking parser programs
+#
+# To retrain the reranking parser, run the following steps:
+#
+# make reranker        # builds reranking parser and training programs
+# make nbesttrain      # builds 20 folds of n-best training parses
+# make eval-reranker   # extracts features, estimates weights, and evaluates
+#
+# The following high-level goals may also be useful:
+#
+# make nbestrain-clean # removes temporary files used in nbesttrain
+# make nbest-oracle    # oracle evaluation of n-best results 
+# make features        # extracts features from 20-fold parses
+# make train-reranker  # trains reranker model
+# make train-clean     # removes all temporary files used in training
+#
+# I typically run nbesttrain to produce the n-best parses 
+
+# To run 2 jobs in parallel (e.g. on a multiprocessor) run, e.g.,
+#
+# make -j 2 nbesttrain
+#
+# This really only helps with nbesttrain, since the other time consuming
+# step (reranker feature weight estimation) isn't yet parallelized.
+
+# The environment variable GCCFLAGS can be used to specify
+# machine-dependent optimization flags, e.g.
+#
+# setenv GCCFLAGS "-march=pentium4"
+#
+# or
+#
+# setenv GCCFLAGS "-march=opteron -m64"
+#
+# The top-level make goal builds the reranking parser using a pre-trained
+# model.  To build this parser, just run
+#
+# make 
+#
+# You may need to tweak the following variables to suit your environment
+
+# GCCFLAGS is not set here, so we use the shell environment
+# variable's value.  But you can set it here if you want.
+# Version 4.1 and later gcc permit -march=native, but older
+# versions will need -march=pentium4 or -march=opteron
+#
+# GCCFLAGS = -march=native -mfpmath=sse -msse2 -mmmx -m32
+
+# CFLAGS is used for all C and C++ compilation
+#
+CFLAGS = -MMD -O6 -Wall -ffast-math -finline-functions -fomit-frame-pointer -fstrict-aliasing $(GCCFLAGS)
+LDFLAGS = $(GCCLDFLAGS)
+EXEC = time
+
+# for debugging, uncomment the following CFLAGS, LDFLAGS and EXEC
+#
+# CFLAGS = -g -O -MMD -Wall -ffast-math -fstrict-aliasing $(GCCFLAGS)
+# LDFLAGS = -g -Wall $(GCCLDFLAGS)
+# EXEC = valgrind
+
+CXXFLAGS = $(CFLAGS) -Wno-deprecated
+export CFLAGS
+export CXXFLAGS
+export LDFLAGS
+
+# Building the 20-fold training data with nbesttrain 
+# --------------------------------------------------
+
+# For training the parser and reranker you will need your own copy of the
+# Penn WSJ Treebank.
+#
+# PENNWSJTREEBANK must be set to the base directory of the Penn WSJ Treebank
+#
+PENNWSJTREEBANK=/usr/local/data/Penn3/parsed/mrg/wsj/
+
+# NPARSES is the number of alternative parses to consider for each sentence
+#
+NPARSES=50
+
+# NFOLDS is the number of folds to use, and FOLDS is a list of the numbers
+# from 00 to NFOLDS-1 (I couldn't see how to program this in make).
+#
+NFOLDS=20
+FOLDS=00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19
+
+# SECTIONS is a list of sections from the treebank to n-best parse
+# using Eugene's standard n-best parser (in addition to the folds).
+#
+SECTIONS=22 23 24
+
+# NBESTPARSERBASEDIR is the base directory in which the n-best parser is
+# located.  If you change this, change NBESTPARSERNICKNAME as well.
+#
+NBESTPARSERBASEDIR=first-stage
+
+# NBESTPARSER is the n-best parser.  If you change this, please
+# change NBESTPARSERNICKNAME below as well.
+#
+NBESTPARSER=$(NBESTPARSERBASEDIR)/PARSE/parseIt
+
+# NBESTTRAINER is the program (probably a shell script) for training
+# the n-best parser.  If you change this, please change
+# NBESTPARSERNICKNAME below as well.
+#
+NBESTTRAINER=$(NBESTPARSERBASEDIR)/TRAIN/allScript
+
+# NBESTPARSERNICKNAME is a nickname for the n-best parser.  If you 
+# experiment with several n-best parsers, give each one a different
+# nickname.
+#
+NBESTPARSERNICKNAME=ec
+
+# TMP specifies a temporary directory used while constructing the
+# folds while producing the training parses for the reranker.  You can
+# delete this directory after nbesttrain has finished.  On an NFS
+# system you may want to change this to a local directory.
+#
+TMP=tmp
+
+# Extracting features from 20-fold n-best parses
+# ----------------------------------------------
+
+# VERSION should be either "final" or "nonfinal".  If VERSION is
+# "nonfinal" then we train on folds 00-19, folds 20-21 are used as
+# dev, and sections 22 and 24 are used as test1 and test2
+# respectively.  If VERSION is "final" then we train on folds 00-21,
+# section 24 is used as dev and sections 22 and 23 are used as test1
+# and test2 respectively.
+#
+VERSION=nonfinal
+# VERSION=final
+
+# FEATUREEXTRACTOR is the program that used to extract features from
+# the 20-fold n-best parses.  If you change this, please pick a new 
+# FEATURESNICKNAME below.
+#
+FEATUREEXTRACTOR=second-stage/programs/features/extract-spfeatures
+
+# FEATUREEXTRACTORFLAGS are flags you want to give to the feature extractor
+#
+FEATUREEXTRACTORFLAGS=-l -c -i -s 5
+
+# FEATURESNICKNAME is an arbitrary string used to identify a
+# particular set of extracted features for training the reranker.  You
+# can keep several different sets of feature counts and corresponding
+# models around by giving each a unique FEATURESNICKNAME.  If you
+# develop a new set of features, give them a new FEATURESNICKNAME so
+# they doesn't over-write the existing features.
+#
+FEATURESNICKNAME=sp
+
+# Estimating weights for features
+# -------------------------------
+
+# ESTIMATOR is the program used to estimate feature weights from the
+# feature counts. This is the feature weight estimator that gives best
+# performance.  There are others in the same directory (e.g., weighted
+# perceptron).  If you decide to use a different feature weight
+# estimator you should also change ESTIMATORNICKNAME below.
+#
+ESTIMATOR=second-stage/programs/wlle/cvlm
+
+# ESTIMATORFLAGS are flags given to the estimator
+#
+ESTIMATORFLAGS=-l 1 -c0 10 -Pyx_factor 1 -debug 10 -ns -1
+
+# ESTIMATORNICKNAME is used to name the feature weights file
+#
+ESTIMATORNICKNAME=cvlm-l1c10P1
+
+# ESTIMATORSTACKSIZE is the size (in KB) of the per-thread stacks
+# used during estimation
+#
+# ESTIMATORSTACKSIZE=40960
+
+# ESTIMATORENV is a prefix to the estimator command that can be
+# used to set environment variables, etc.
+#
+# ESTIMATORENV=ulimit -s $(ESTIMATORSTACKSIZE) && OMP_STACKSIZE=`ulimit -s` &&
+ESTIMATORENV=
+
+########################################################################
+#
+# You probably shouldn't need to change anything below here.
+
+# TARGETS is the list of targets built when make is called
+# without arguments
+#
+TARGETS = PARSE reranker-runtime evalb
+
+.PHONY: top
+top: $(TARGETS)
+
+# PARSE builds the n-best first-stage parser (i.e., Eugene's parser).
+#
+.PHONY: PARSE
+PARSE:
+	make -C $(NBESTPARSERBASEDIR)/PARSE parseIt
+
+# TRAIN builds the programs needed to train the first-stage parser.
+#
+.PHONY: TRAIN
+TRAIN:
+	make -C $(NBESTPARSERBASEDIR)/TRAIN all
+
+# reranker-runtime builds the run-time components of the reranker.
+# These include best-parses, which reranks the n-best parses produced
+# by the first-stage parser, and ptb, which is a program that converts
+# Penn Treebank trees into the various formats needed by Eugene's
+# parser, the reranker training programs, EVALB, etc.  
+#
+.PHONY: reranker-runtime
+reranker-runtime:
+	make -C second-stage/programs/features best-parses
+	make -C second-stage/programs/prepare-data ptb
+
+# reranker builds the training and run-time components of the reranker.
+# These include:
+#  ptb, which converts the Penn Treebank parse trees into
+#       the various formats needed by Eugene's parser, the reranker training
+#       program, EVALB, etc., 
+#  extract-spfeatures, which produces feature-count files used to train 
+#       the reranker, 
+#  cvlm, which estimates the feature weights.
+#
+.PHONY: reranker
+reranker: top TRAIN
+	make -C second-stage
+
+# Good old EVALB!
+#
+.PHONY: evalb
+evalb: 
+	make -C evalb evalb
+
+# clean removes object files.
+#
+.PHONY: clean
+clean:
+	(cd $(NBESTPARSERBASEDIR); rm -f PARSE/*.o; rm -f TRAIN/*.o)
+	make -C $(NBESTPARSERBASEDIR)/TRAIN clean
+	make -C $(NBESTPARSERBASEDIR)/PARSE clean
+	make -C second-stage clean
+
+# nbesttrain-clean removes temporary files used in constructing the 20
+# folds of n-best training data.
+#
+nbesttrain-clean:
+	rm -fr $(TMP)
+
+# train-clean gets rid of all data not essential for the runtime reranking parser. 
+#
+.PHONY: train-clean
+train-clean: nbesttrain-clean
+	rm -fr results
+	make -C second-stage train-clean
+
+# real-clean tries to get rid of all object and binary files to
+# produce a version for distribution.  But Eugene writes new programs
+# faster than I can make real-clean clean them up!
+#
+.PHONY: real-clean
+real-clean: clean train-clean
+	(cd $(NBESTPARSERBASEDIR); rm -f PARSE/parseIt)
+	make -C second-stage real-clean
+
+########################################################################
+#                                                                      #
+# nbesttrain -- Preparing the N-best training data for the reranker    #
+#                                                                      #
+########################################################################
+
+# To build the 20-fold n-best data in second-stage/train 
+# for training the ranker, run
+#
+# make nbesttrain 
+#
+# or
+# 
+# make -j 2 nbesttrain
+#
+# on a multiprocessor machine
+
+# TRAIN specifies the location of the trees to be divided into NFOLDS
+# This is defined here to use sections 2-21 of the Penn WSJ treebank.
+#
+TRAIN=$(PENNWSJTREEBANK)/0[2-9]/*mrg $(PENNWSJTREEBANK)/1[0-9]/*mrg $(PENNWSJTREEBANK)/2[0-1]/*mrg
+
+# NBESTDIR is the directory that holds the n-best parses for training
+# the reranker.
+#
+NBESTDIR=second-stage/nbest/$(NBESTPARSERNICKNAME)$(NPARSES)
+
+# NBESTFILES are all of the files in the n-best folds, plus dev and test sections
+#
+NBESTFILES= $(foreach fold,$(FOLDS),$(NBESTDIR)/fold$(fold).gz) $(foreach section,$(SECTIONS),$(NBESTDIR)/section$(section).gz)
+
+.PHONY: nbesttrain
+nbesttrain: $(NBESTFILES) PARSE TRAIN second-stage/programs/prepare-data/ptb
+
+# This goal copies and gzips the output of the n-best parser
+# into the appropriate directory for training the reranker.
+#
+# .PRECIOUS: $(NBESTDIR)/fold%.gz
+.INTERMEDIATE: $(NBESTDIR)/fold%.gz
+$(NBESTDIR)/fold%.gz: $(TMP)/fold%/$(NBESTPARSERNICKNAME)$(NPARSES)best
+	mkdir -p $(NBESTDIR)
+	gzip -c $+ > $@
+
+# The remaining goals in this section are for training and parsing
+# with the n-best parser to produce the folds for training the
+# reranker.
+
+.INTERMEDIATE: $(TMP)/fold%/$(NBESTPARSERNICKNAME)$(NPARSES)best
+$(TMP)/fold%/$(NBESTPARSERNICKNAME)$(NPARSES)best: $(TMP)/fold%/DATA $(TMP)/fold%/yield $(NBESTPARSER)
+	$(EXEC) $(NBESTPARSER) -l400 -K -N$(NPARSES) $(@D)/DATA/ $(@D)/yield > $@
+
+.INTERMEDIATE: $(TMP)/fold%/DATA
+$(TMP)/fold%/DATA: $(TMP)/fold%/train $(TMP)/fold%/dev $(NBESTTRAINER)
+	mkdir -p $@
+	LC_COLLATE=C; cp $(NBESTPARSERBASEDIR)/DATA/EN/[a-z]* $@
+	$(EXEC) $(NBESTTRAINER) $@ $(@D)/train $(@D)/dev
+
+.INTERMEDIATE: $(TMP)/fold%/train
+$(TMP)/fold%/train: second-stage/programs/prepare-data/ptb
+	mkdir -p $(@D)
+	$(EXEC) second-stage/programs/prepare-data/ptb -n $(NFOLDS) -x $(patsubst $(TMP)/fold%,%,$(@D)) -e $(TRAIN)  > $@
+
+.INTERMEDIATE: $(TMP)/fold%/dev
+$(TMP)/fold%/dev: second-stage/programs/prepare-data/ptb
+	mkdir -p $(@D)
+	second-stage/programs/prepare-data/ptb -n $(NFOLDS) -i $(patsubst $(TMP)/fold%,%,$(@D)) -e $(TRAIN)  > $@
+
+# $(TMP)/fold%/DATA: $(TMP)/%/train $(TMP)/%/dev
+# 	mkdir -p $@
+# 	LC_COLLATE=C; cp $(NBESTPARSERBASEDIR)/DATA/EN/[a-z]* $@
+# 	$(NBESTPARSERBASEDIR)/TRAIN/allScript $@ $(@D)/train $(@D)/dev
+
+.INTERMEDIATE: $(TMP)/fold%/yield
+$(TMP)/fold%/yield: second-stage/programs/prepare-data/ptb
+	mkdir -p $(@D)
+	$(EXEC) second-stage/programs/prepare-data/ptb -n $(NFOLDS) -i $(patsubst $(TMP)/fold%,%,$(@D)) -c $(TRAIN) > $@
+
+# .PRECIOUS: $(NBESTDIR)/section%.gz
+.INTERMEDIATE: $(NBESTDIR)/section%.gz
+$(NBESTDIR)/section%.gz: $(TMP)/section%/$(NBESTPARSERNICKNAME)$(NPARSES)best
+	mkdir -p $(NBESTDIR)
+	gzip -c $+ > $@
+
+.INTERMEDIATE: $(TMP)/section%/$(NBESTPARSERNICKNAME)$(NPARSES)best
+$(TMP)/section%/$(NBESTPARSERNICKNAME)$(NPARSES)best: $(TMP)/section%/yield $(NBESTPARSER)
+	$(EXEC) $(NBESTPARSER) -l400 -K -N$(NPARSES) $(NBESTPARSERBASEDIR)/DATA/EN/ $(@D)/yield > $@
+
+.INTERMEDIATE: $(TMP)/section%/yield
+$(TMP)/section%/yield: second-stage/programs/prepare-data/ptb
+	mkdir -p $(@D)
+	$(EXEC) second-stage/programs/prepare-data/ptb -c $(PENNWSJTREEBANK)/$(patsubst $(TMP)/section%,%,$(@D))/wsj*.mrg  > $@
+
+########################################################################
+#                                                                      #
+# touch-nbest marks the n-best parses as up-to-date                    #
+#                                                                      #
+########################################################################
+
+.PHONY: touch-nbest
+touch-nbest:
+	touch $(NBESTDIR) $(NBESTDIR)/*
+
+########################################################################
+#                                                                      #
+# nbest oracle evaluation                                              #
+#                                                                      #
+########################################################################
+
+.PHONY: nbest-oracle
+nbest-oracle: second-stage/programs/features/oracle-score second-stage/programs/prepare-data/ptb $(NBESTFILES)
+	$(EXEC) second-stage/programs/features/oracle-score "zcat $(NBESTDIR)/fold[0-1][0-9].gz" "second-stage/programs/prepare-data/ptb -g $(TRAIN)"
+	$(EXEC) second-stage/programs/features/oracle-score "zcat $(NBESTDIR)/section22.gz" "second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/22/wsj*.mrg"
+	$(EXEC) second-stage/programs/features/oracle-score "zcat $(NBESTDIR)/section24.gz" "second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/24/wsj*.mrg"
+
+.PHONY: nbest-oracle-detailed
+nbest-oracle-detailed: second-stage/programs/eval-beam/main second-stage/programs/prepare-data/ptb $(NBESTFILES)
+	$(EXEC) second-stage/programs/eval-beam/main "zcat $(NBESTDIR)/fold[0-1][0-9].gz" "second-stage/programs/prepare-data/ptb -g $(TRAIN)"
+	$(EXEC) second-stage/programs/eval-beam/main "zcat $(NBESTDIR)/section22.gz" "second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/22/wsj*.mrg"
+	$(EXEC) second-stage/programs/eval-beam/main "zcat $(NBESTDIR)/section24.gz" "second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/24/wsj*.mrg"
+
+########################################################################
+#                                                                      #
+# extract-features extracts feature counts for training reranker       #
+#                                                                      #
+########################################################################
+
+# FEATBASEDIR is the directory in which the feature counts will be saved,
+# minus the $(VERSION) flag.
+#
+FEATBASEDIR=second-stage/features/$(NBESTPARSERNICKNAME)$(NPARSES)$(FEATURESNICKNAME)
+
+# FEATDIR is the directory in which the feature counts will be saved.
+#
+FEATDIR=$(FEATBASEDIR)$(VERSION)
+
+# MODELBASEDIR is the directory in which the features and feature
+# weights are saved, minus the version.
+#
+MODELBASEDIR=second-stage/models/$(NBESTPARSERNICKNAME)$(NPARSES)$(FEATURESNICKNAME)
+
+# MODELDIR is the directory in which the features and feature weights
+# are saved.
+#
+MODELDIR=$(MODELBASEDIR)$(VERSION)
+
+.PHONY: features
+features: $(MODELDIR)/features.gz $(FEATDIR)/train.gz $(FEATDIR)/dev.gz $(FEATDIR)/test1.gz $(FEATDIR)/test2.gz
+
+# This goal does feature extraction for reranker training for the
+# nonfinal case (i.e., train is folds 0-17, dev is folds 18-19, test1
+# is section 22 and test2 is section 24).
+#
+$(MODELBASEDIR)nonfinal/features.gz $(FEATBASEDIR)nonfinal/train.gz $(FEATBASEDIR)nonfinal/dev.gz $(FEATBASEDIR)nonfinal/test1.gz $(FEATBASEDIR)nonfinal/test2.gz: second-stage/programs/prepare-data/ptb $(FEATUREEXTRACTOR) $(NBESTFILES)
+	mkdir -p $(FEATBASEDIR)nonfinal
+	mkdir -p $(MODELBASEDIR)nonfinal
+	$(EXEC) $(FEATUREEXTRACTOR) $(FEATUREEXTRACTORFLAGS) \
+		"zcat $(NBESTDIR)/fold0[0-9].gz $(NBESTDIR)/fold1[0-7].gz" \
+		"second-stage/programs/prepare-data/ptb -g -n 10 -x 9 $(TRAIN)" \
+		$(FEATBASEDIR)nonfinal/train.gz \
+		"zcat $(NBESTDIR)/fold1[8-9].gz" \
+		"second-stage/programs/prepare-data/ptb -g -n 10 -i 9 $(TRAIN)" \
+		$(FEATBASEDIR)nonfinal/dev.gz \
+		"zcat $(NBESTDIR)/section22.gz" \
+		"second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/22/*mrg" \
+		$(FEATBASEDIR)nonfinal/test1.gz \
+		"zcat $(NBESTDIR)/section24.gz" \
+		"second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/24/*mrg" \
+		$(FEATBASEDIR)nonfinal/test2.gz \
+		| gzip > $(MODELBASEDIR)nonfinal/features.gz
+
+# This goal does feature extraction for reranker training for the
+# final case (i.e., train is folds 0-19, dev is section 24, test1
+# is section 22 and test2 is section 23).
+#
+$(MODELBASEDIR)final/features.gz $(FEATBASEDIR)final/train.gz $(FEATBASEDIR)final/dev.gz $(FEATBASEDIR)final/test1.gz $(FEATBASEDIR)final/test2.gz: second-stage/programs/prepare-data/ptb $(FEATUREEXTRACTOR) $(NBESTFILES)
+	mkdir -p $(FEATBASEDIR)final
+	mkdir -p $(MODELBASEDIR)final
+	$(EXEC) $(FEATUREEXTRACTOR) $(FEATUREEXTRACTORFLAGS) \
+		"zcat $(NBESTDIR)/fold*.gz" \
+		"second-stage/programs/prepare-data/ptb -g $(TRAIN)" \
+		$(FEATBASEDIR)final/train.gz \
+		"zcat $(NBESTDIR)/section22.gz" \
+		"second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/22/*mrg" \
+		$(FEATBASEDIR)final/test1.gz \
+		"zcat $(NBESTDIR)/section23.gz" \
+		"second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/23/*mrg" \
+		$(FEATBASEDIR)final/test2.gz \
+		"zcat $(NBESTDIR)/section24.gz" \
+		"second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/24/*mrg" \
+		$(FEATBASEDIR)final/dev.gz \
+		| gzip > $(MODELBASEDIR)final/features.gz
+
+
+########################################################################
+#                                                                      #
+# train-reranker estimates the reranker feature weights                #
+#                                                                      #
+########################################################################
+
+WEIGHTSFILE=$(MODELDIR)/$(ESTIMATORNICKNAME)-weights
+WEIGHTSFILEGZ=$(WEIGHTSFILE).gz
+
+.PHONY: train-reranker
+train-reranker: $(WEIGHTSFILEGZ)
+
+# This goal estimates the reranker feature weights (i.e., trains the
+# reranker).
+#
+# $(WEIGHTSFILEGZ): $(ESTIMATOR)
+$(WEIGHTSFILEGZ): $(ESTIMATOR) $(MODELDIR)/features.gz $(FEATDIR)/train.gz $(FEATDIR)/dev.gz $(FEATDIR)/test1.gz
+	$(ESTIMATORENV) zcat $(FEATDIR)/train.gz | $(EXEC) $(ESTIMATOR) $(ESTIMATORFLAGS) -e $(FEATDIR)/dev.gz -f $(MODELDIR)/features.gz -o $(WEIGHTSFILE) -x $(FEATDIR)/test1.gz
+	rm -f $(WEIGHTSFILEGZ)
+	gzip $(WEIGHTSFILE)
+
+########################################################################
+#                                                                      #
+# eval-reranker evaluates the reranker on the two test data sets       #
+#                                                                      #
+########################################################################
+
+EVALDIR=second-stage/eval/$(NBESTPARSERNICKNAME)$(NPARSES)$(FEATURESNICKNAME)$(VERSION)-$(ESTIMATORNICKNAME)
+
+.PHONY: eval-reranker
+eval-reranker: $(EVALDIR)/weights-eval # $(EVALDIR)/dev-parsediffs.gz
+
+$(EVALDIR)/weights-eval: $(WEIGHTSFILEGZ) $(MODELDIR)/features.gz $(FEATDIR)/dev.gz $(FEATDIR)/test1.gz $(FEATDIR)/test2.gz second-stage/programs/eval-weights/eval-weights
+	mkdir -p $(EVALDIR)
+	zcat $(WEIGHTSFILEGZ) | second-stage/programs/eval-weights/eval-weights $(EVALWEIGHTSARGS) $(MODELDIR)/features.gz $(FEATDIR)/dev.gz > $(EVALDIR)/weights-eval
+	zcat $(WEIGHTSFILEGZ) | second-stage/programs/eval-weights/eval-weights $(EVALWEIGHTSARGS) $(MODELDIR)/features.gz $(FEATDIR)/test1.gz >> $(EVALDIR)/weights-eval
+	zcat $(WEIGHTSFILEGZ) | second-stage/programs/eval-weights/eval-weights $(EVALWEIGHTSARGS) $(MODELDIR)/features.gz $(FEATDIR)/test2.gz >> $(EVALDIR)/weights-eval
+
+$(EVALDIR)/dev-parsediffs.gz: $(WEIGHTSFILEGZ) $(FEATDIR)/test1.gz $(NBESTDIR)/section24.gz second-stage/programs/eval-weights/best-indices second-stage/programs/eval-weights/best-parses second-stage/programs/eval-weights/pretty-print
+	zcat $(WEIGHTSFILEGZ) \
+	 | second-stage/programs/eval-weights/best-indices $(FEATDIR)/test1.gz \
+	 | second-stage/programs/eval-weights/best-parses $(NBESTDIR)/section24.gz \
+	 | second-stage/programs/eval-weights/pretty-print -d \
+	 | gzip > $(EVALDIR)/dev-parsediffs.gz
+# Top-level makefile for reranking parser
+# Mark Johnson, 24th October 2007
+
+########################################################################
+#                                                                      #
+#                                Summary                               #
+#                                                                      #
+########################################################################
+#
+# To build just the reranking parser run-time, execute:
+#
+# make                 # builds reranking parser programs
+#
+# To retrain the reranking parser, run the following steps:
+#
+# make reranker        # builds reranking parser and training programs
+# make nbesttrain      # builds 20 folds of n-best training parses
+# make eval-reranker   # extracts features, estimates weights, and evaluates
+#
+# The following high-level goals may also be useful:
+#
+# make nbestrain-clean # removes temporary files used in nbesttrain
+# make nbest-oracle    # oracle evaluation of n-best results 
+# make features        # extracts features from 20-fold parses
+# make train-reranker  # trains reranker model
+# make train-clean     # removes all temporary files used in training
+#
+# I typically run nbesttrain to produce the n-best parses 
+
+# To run 2 jobs in parallel (e.g. on a multiprocessor) run, e.g.,
+#
+# make -j 2 nbesttrain
+#
+# This really only helps with nbesttrain, since the other time consuming
+# step (reranker feature weight estimation) isn't yet parallelized.
+
+# The environment variable GCCFLAGS can be used to specify
+# machine-dependent optimization flags, e.g.
+#
+# setenv GCCFLAGS "-march=pentium4"
+#
+# or
+#
+# setenv GCCFLAGS "-march=opteron -m64"
+#
+# The top-level make goal builds the reranking parser using a pre-trained
+# model.  To build this parser, just run
+#
+# make 
+#
+# You may need to tweak the following variables to suit your environment
+
+# GCCFLAGS is not set here, so we use the shell environment
+# variable's value.  But you can set it here if you want.
+# Version 4.1 and later gcc permit -march=native, but older
+# versions will need -march=pentium4 or -march=opteron
+#
+# GCCFLAGS = -march=native -mfpmath=sse -msse2 -mmmx -I <path-to-boost-libraries>
+
+# CFLAGS is used for all C and C++ compilation
+#
+CFLAGS = -MMD -O6 -Wall -ffast-math -finline-functions -fomit-frame-pointer -fstrict-aliasing $(GCCFLAGS)
+
+# for debugging, uncomment the following CFLAGS and LDFLAGS
+#
+# CFLAGS = -g -O -MMD -Wall -ffast-math -fstrict-aliasing $(GCCFLAGS)
+# LDFLAGS = -g -Wall
+
+
+# Building the 20-fold training data with nbesttrain 
+# --------------------------------------------------
+
+# For training the parser and reranker you will need your own copy of the
+# Penn WSJ Treebank.
+#
+# PENNWSJTREEBANK must be set to the base directory of the Penn WSJ Treebank
+#
+PENNWSJTREEBANK=/usr/local/data/Penn3/parsed/mrg/wsj/
+
+# NPARSES is the number of alternative parses to consider for each sentence
+#
+NPARSES=50
+
+# NFOLDS is the number of folds to use, and FOLDS is a list of the numbers
+# from 00 to NFOLDS-1 (I couldn't see how to program this in make).
+#
+NFOLDS=20
+FOLDS=00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19
+
+# SECTIONS is a list of sections from the treebank to n-best parse
+# using Eugene's standard n-best parser (in addition to the folds).
+#
+SECTIONS=22 23 24
+
+# NBESTPARSER is the n-best parser.  If you change this, please
+# change NBESTPARSERNICKNAME below as well.
+#
+NBESTPARSER=first-stage/PARSE/parseIt
+
+# NBESTTRAINER is the program (probably a shell script) for training
+# the n-best parser.  If you change this, please change
+# NBESTPARSERNICKNAME below as well.
+#
+NBESTTRAINER=first-stage/TRAIN/allScript
+
+# NBESTPARSERNICKNAME is a nickname for the n-best parser.  If you 
+# experiment with several n-best parsers, give each one a different
+# nickname.
+#
+PARSERNICKNAME=ec
+
+# TMP specifies a temporary directory used while constructing the
+# folds while producing the training parses for the reranker.  You can
+# delete this directory after nbesttrain has finished.  On an NFS
+# system you may want to change this to a local directory.
+#
+TMP=tmp
+
+# Extracting features from 20-fold n-best parses
+# ----------------------------------------------
+
+# VERSION should be either "final" or "nonfinal".  If VERSION is
+# "nonfinal" then we train on folds 00-19, folds 20-21 are used as
+# dev, and sections 22 and 24 are used as test1 and test2
+# respectively.  If VERSION is "final" then we train on folds 00-21,
+# section 24 is used as dev and sections 22 and 23 are used as test1
+# and test2 respectively.
+#
+VERSION=nonfinal
+# VERSION=final
+
+# FEATUREEXTRACTOR is the program that used to extract features from
+# the 20-fold n-best parses.  If you change this, please pick a new 
+# FEATURESNICKNAME below.
+#
+FEATUREEXTRACTOR=second-stage/programs/features/extract-spfeatures
+
+# FEATUREEXTRACTORFLAGS are flags you want to give to the feature extractor
+#
+FEATUREEXTRACTORFLAGS=-l -c -i -s 5
+
+# FEATURESNICKNAME is an arbitrary string used to identify a
+# particular set of extracted features for training the reranker.  You
+# can keep several different sets of feature counts and corresponding
+# models around by giving each a unique FEATURESNICKNAME.  If you
+# develop a new set of features, give them a new FEATURESNICKNAME so
+# they doesn't over-write the existing features.
+#
+FEATURESNICKNAME=sp
+
+# Estimating weights for features
+# -------------------------------
+
+# ESTIMATOR is the program used to estimate feature weights from the
+# feature counts. This is the feature weight estimator that gives best
+# performance.  There are others in the same directory (e.g., weighted
+# perceptron).  If you decide to use a different feature weight
+# estimator you should also change ESTIMATORNICKNAME below.
+#
+ESTIMATOR=second-stage/programs/wlle/gavper
+
+# ESTIMATORFLAGS are flags given to the estimator
+#
+ESTIMATORFLAGS= -a -n 10 -d 10 -F 1 -m 0
+
+# ESTIMATORNICKNAME
+#
+ESTIMATORNICKNAME=gavper-aa
+
+########################################################################
+#
+# You probably shouldn't need to change anything below here.
+
+# TARGETS is the list of targets built when make is called
+# without arguments
+#
+TARGETS = PARSE reranker-runtime evalb
+
+CXXFLAGS = $(CFLAGS)
+export CFLAGS
+export CXXFLAGS
+
+.PHONY: top
+top: $(TARGETS)
+
+# PARSE builds the n-best first-stage parser (i.e., Eugene's parser).
+#
+.PHONY: PARSE
+PARSE:
+	make -C first-stage/PARSE parseIt
+
+# TRAIN builds the programs needed to train the first-stage parser.
+#
+.PHONY: TRAIN
+TRAIN:
+	make -C first-stage/TRAIN all
+
+# reranker-runtime builds the run-time components of the reranker.
+# These include best-parses, which reranks the n-best parses produced
+# by the first-stage parser, and ptb, which is a program that converts
+# Penn Treebank trees into the various formats needed by Eugene's
+# parser, the reranker training programs, EVALB, etc.  
+#
+.PHONY: reranker-runtime
+reranker-runtime:
+	make -C second-stage/programs/features best-parses
+	make -C second-stage/programs/prepare-data ptb
+
+# reranker builds the training and run-time components of the reranker.
+# These include:
+#  ptb, which converts the Penn Treebank parse trees into
+#       the various formats needed by Eugene's parser, the reranker training
+#       program, EVALB, etc., 
+#  extract-spfeatures, which produces feature-count files used to train 
+#       the reranker, 
+#  cvlm, which estimates the feature weights.
+#
+.PHONY: reranker
+reranker: top TRAIN
+	make -C second-stage
+
+# Good old EVALB!
+#
+.PHONY: evalb
+evalb: 
+	make -C evalb evalb
+
+# clean removes object files.
+#
+.PHONY: clean
+clean:
+	(cd first-stage; rm -f PARSE/*.o; rm -f TRAIN/*.o)
+	make -C first-stage/TRAIN clean
+	make -C first-stage/PARSE clean
+	make -C second-stage clean
+
+# nbesttrain-clean removes temporary files used in constructing the 20
+# folds of n-best training data.
+#
+nbesttrain-clean:
+	rm -fr $(TMP)
+
+# train-clean gets rid of all data not essential for the reranking
+# parser. 
+#
+.PHONY: train-clean
+train-clean: nbesttrain-clean
+	rm -fr results
+	make -C second-stage train-clean
+
+# real-clean tries to get rid of all object and binary files to
+# produce a version for distribution.  But Eugene writes new programs
+# faster than I can make real-clean clean them up!
+#
+.PHONY: real-clean
+real-clean: clean train-clean
+	(cd first-stage; rm -f PARSE/parseIt)
+	make -C second-stage real-clean
+
+########################################################################
+#                                                                      #
+# nbesttrain -- Preparing the N-best training data for the reranker    #
+#                                                                      #
+########################################################################
+
+# To build the 20-fold n-best data in second-stage/train 
+# for training the ranker, run
+#
+# make nbesttrain 
+#
+# or
+# 
+# make -j 2 nbesttrain
+#
+# on a multiprocessor machine
+
+# TRAIN specifies the location of the trees to be divided into NFOLDS
+# This is defined here to use sections 2-21 of the Penn WSJ treebank.
+#
+TRAIN=$(PENNWSJTREEBANK)/0[2-9]/*mrg $(PENNWSJTREEBANK)/1[0-9]/*mrg $(PENNWSJTREEBANK)/2[0-1]/*mrg
+
+# NBESTDIR is the directory that holds the n-best parses for training
+# the reranker.
+#
+NBESTDIR=second-stage/nbest/$(PARSERNICKNAME)$(NPARSES)
+
+# NBESTFILES are all of the files in the n-best folds, plus dev and test sections
+#
+NBESTFILES= $(foreach fold,$(FOLDS),$(NBESTDIR)/fold$(fold).gz) $(foreach section,$(SECTIONS),$(NBESTDIR)/section$(section).gz)
+
+.PHONY: nbesttrain
+nbesttrain: $(NBESTFILES) PARSE TRAIN second-stage/programs/prepare-data/ptb
+
+# This goal copies and gzips the output of the n-best parser
+# into the appropriate directory for training the reranker.
+#
+# .PRECIOUS: $(NBESTDIR)/fold%.gz
+.INTERMEDIATE: $(NBESTDIR)/fold%.gz
+$(NBESTDIR)/fold%.gz: $(TMP)/fold%/$(NPARSES)best
+	mkdir -p $(NBESTDIR)
+	gzip -c $+ > $@
+
+# The remaining goals in this section are for training and parsing
+# with the n-best parser to produce the folds for training the
+# reranker.
+
+.INTERMEDIATE: $(TMP)/fold%/$(NPARSES)best
+$(TMP)/fold%/$(NPARSES)best: $(TMP)/fold%/DATA $(TMP)/fold%/yield $(NBESTPARSER)
+	$(NBESTPARSER) -l400 -K -N$(NPARSES) $(@D)/DATA/ $(@D)/yield > $@
+
+.INTERMEDIATE: $(TMP)/fold%/DATA
+$(TMP)/fold%/DATA: $(TMP)/fold%/train $(TMP)/fold%/dev $(NBESTTRAINER)
+	mkdir -p $@
+	LC_COLLATE=C; cp first-stage/DATA/EN/[a-z]* $@
+	$(NBESTTRAINER) $@ $(@D)/train $(@D)/dev
+
+.INTERMEDIATE: $(TMP)/fold%/train
+$(TMP)/fold%/train: second-stage/programs/prepare-data/ptb
+	mkdir -p $(@D)
+	second-stage/programs/prepare-data/ptb -n $(NFOLDS) -x $(patsubst $(TMP)/fold%,%,$(@D)) -e $(TRAIN)  > $@
+
+.INTERMEDIATE: $(TMP)/fold%/dev
+$(TMP)/fold%/dev: second-stage/programs/prepare-data/ptb
+	mkdir -p $(@D)
+	second-stage/programs/prepare-data/ptb -n $(NFOLDS) -i $(patsubst $(TMP)/fold%,%,$(@D)) -e $(TRAIN)  > $@
+
+# $(TMP)/fold%/DATA: $(TMP)/%/train $(TMP)/%/dev
+# 	mkdir -p $@
+# 	LC_COLLATE=C; cp first-stage/DATA/EN/[a-z]* $@
+# 	first-stage/TRAIN/allScript $@ $(@D)/train $(@D)/dev
+
+.INTERMEDIATE: $(TMP)/fold%/yield
+$(TMP)/fold%/yield: second-stage/programs/prepare-data/ptb
+	mkdir -p $(@D)
+	second-stage/programs/prepare-data/ptb -n $(NFOLDS) -i $(patsubst $(TMP)/fold%,%,$(@D)) -c $(TRAIN) > $@
+
+# .PRECIOUS: $(NBESTDIR)/section%.gz
+.INTERMEDIATE: $(NBESTDIR)/section%.gz
+$(NBESTDIR)/section%.gz: $(TMP)/section%/$(NPARSES)best
+	mkdir -p $(NBESTDIR)
+	gzip -c $+ > $@
+
+.INTERMEDIATE: $(TMP)/section%/$(NPARSES)best
+$(TMP)/section%/$(NPARSES)best: $(TMP)/section%/yield $(NBESTPARSER)
+	$(NBESTPARSER) -l400 -K -N$(NPARSES) first-stage/DATA/EN/ $(@D)/yield > $@
+
+.INTERMEDIATE: $(TMP)/section%/yield
+$(TMP)/section%/yield: second-stage/programs/prepare-data/ptb
+	mkdir -p $(@D)
+	second-stage/programs/prepare-data/ptb -c $(PENNWSJTREEBANK)/$(patsubst $(TMP)/section%,%,$(@D))/wsj*.mrg  > $@
+
+########################################################################
+#                                                                      #
+# nbest oracle evaluation                                              #
+#                                                                      #
+########################################################################
+
+.PHONY: nbest-oracle
+nbest-oracle: second-stage/programs/features/oracle-score second-stage/programs/prepare-data/ptb $(NBESTFILES)
+	second-stage/programs/features/oracle-score "zcat $(NBESTDIR)/fold[0-1][0-9].gz" "second-stage/programs/prepare-data/ptb -g $(TRAIN)"
+	second-stage/programs/features/oracle-score "zcat $(NBESTDIR)/section22.gz" "second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/22/wsj*.mrg"
+	second-stage/programs/features/oracle-score "zcat $(NBESTDIR)/section24.gz" "second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/24/wsj*.mrg"
+
+.PHONY: nbest-oracle-detailed
+nbest-oracle-detailed: second-stage/programs/eval-beam/main second-stage/programs/prepare-data/ptb $(NBESTFILES)
+	second-stage/programs/eval-beam/main "zcat $(NBESTDIR)/fold[0-1][0-9].gz" "second-stage/programs/prepare-data/ptb -g $(TRAIN)"
+	second-stage/programs/eval-beam/main "zcat $(NBESTDIR)/section22.gz" "second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/22/wsj*.mrg"
+	second-stage/programs/eval-beam/main "zcat $(NBESTDIR)/section24.gz" "second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/24/wsj*.mrg"
+
+########################################################################
+#                                                                      #
+# extract-features extracts feature counts for training reranker       #
+#                                                                      #
+########################################################################
+
+# FEATBASEDIR is the directory in which the feature counts will be saved,
+# minus the $(VERSION) flag.
+#
+FEATBASEDIR=second-stage/features/$(PARSERNICKNAME)$(NPARSES)$(FEATURESNICKNAME)
+
+# FEATDIR is the directory in which the feature counts will be saved.
+#
+FEATDIR=$(FEATBASEDIR)$(VERSION)
+
+# MODELBASEDIR is the directory in which the features and feature
+# weights are saved, minus the version.
+#
+MODELBASEDIR=second-stage/models/$(PARSERNICKNAME)$(NPARSES)$(FEATURESNICKNAME)
+
+# MODELDIR is the directory in which the features and feature weights
+# are saved.
+#
+MODELDIR=$(MODELBASEDIR)$(VERSION)
+
+.PHONY: features
+features: $(MODELDIR)/features.gz $(FEATDIR)/train.gz $(FEATDIR)/dev.gz $(FEATDIR)/test1.gz $(FEATDIR)/test2.gz
+
+# This goal does feature extraction for reranker training for the
+# nonfinal case (i.e., train is folds 0-17, dev is folds 18-19, test1
+# is section 22 and test2 is section 24).
+#
+$(MODELBASEDIR)nonfinal/features.gz $(FEATBASEDIR)nonfinal/train.gz $(FEATBASEDIR)nonfinal/dev.gz $(FEATBASEDIR)nonfinal/test1.gz $(FEATBASEDIR)nonfinal/test2.gz: second-stage/programs/prepare-data/ptb $(FEATUREEXTRACTOR) $(NBESTFILES)
+	mkdir -p $(FEATBASEDIR)nonfinal
+	mkdir -p $(MODELBASEDIR)nonfinal
+	$(FEATUREEXTRACTOR) $(FEATUREEXTRACTORFLAGS) \
+		"zcat $(NBESTDIR)/fold0[0-9].gz $(NBESTDIR)/fold1[0-7].gz" \
+		"second-stage/programs/prepare-data/ptb -g -n 10 -x 9 $(TRAIN)" \
+		$(FEATBASEDIR)nonfinal/train.gz \
+		"zcat $(NBESTDIR)/fold1[8-9].gz" \
+		"second-stage/programs/prepare-data/ptb -g -n 10 -i 9 $(TRAIN)" \
+		$(FEATBASEDIR)nonfinal/dev.gz \
+		"zcat $(NBESTDIR)/section22.gz" \
+		"second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/22/*mrg" \
+		$(FEATBASEDIR)nonfinal/test1.gz \
+		"zcat $(NBESTDIR)/section24.gz" \
+		"second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/24/*mrg" \
+		$(FEATBASEDIR)nonfinal/test2.gz \
+		| gzip > $(MODELBASEDIR)nonfinal/features.gz
+
+# This goal does feature extraction for reranker training for the
+# final case (i.e., train is folds 0-19, dev is section 24, test1
+# is section 22 and test2 is section 23).
+#
+$(MODELBASEDIR)final/features.gz $(FEATBASEDIR)final/train.gz $(FEATBASEDIR)final/dev.gz $(FEATBASEDIR)final/test1.gz $(FEATBASEDIR)final/test2.gz: second-stage/programs/prepare-data/ptb $(FEATUREEXTRACTOR) $(NBESTFILES)
+	mkdir -p $(FEATBASEDIR)final
+	mkdir -p $(MODELBASEDIR)final
+	$(FEATUREEXTRACTOR) $(FEATUREEXTRACTORFLAGS) \
+		"zcat $(NBESTDIR)/fold*.gz" \
+		"second-stage/programs/prepare-data/ptb -g $(TRAIN)" \
+		$(FEATBASEDIR)final/train.gz \
+		"zcat $(NBESTDIR)/section22.gz" \
+		"second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/22/*mrg" \
+		$(FEATBASEDIR)final/test1.gz \
+		"zcat $(NBESTDIR)/section23.gz" \
+		"second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/23/*mrg" \
+		$(FEATBASEDIR)final/test2.gz \
+		"zcat $(NBESTDIR)/section24.gz" \
+		"second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/24/*mrg" \
+		$(FEATBASEDIR)final/dev.gz \
+		| gzip > $(MODELBASEDIR)final/features.gz
+
+
+########################################################################
+#                                                                      #
+# train-reranker estimates the reranker feature weights                #
+#                                                                      #
+########################################################################
+
+WEIGHTSFILE=$(MODELDIR)/$(ESTIMATORNICKNAME)-weights
+WEIGHTSFILEGZ=$(WEIGHTSFILE).gz
+
+.PHONY: train-reranker
+train-reranker: $(WEIGHTSFILEGZ)
+
+# This goal estimates the reranker feature weights (i.e., trains the
+# reranker).  This is not hard to parallelize, but I haven't actually
+# done that yet.
+#
+# $(WEIGHTSFILEGZ): $(ESTIMATOR)
+$(WEIGHTSFILEGZ): $(ESTIMATOR) $(MODELDIR)/features.gz $(FEATDIR)/train.gz $(FEATDIR)/dev.gz $(FEATDIR)/test1.gz
+	zcat $(FEATDIR)/train.gz | $(ESTIMATOR) $(ESTIMATORFLAGS) -e $(FEATDIR)/dev.gz -f $(MODELDIR)/features.gz -o $(WEIGHTSFILE) -x $(FEATDIR)/test1.gz
+	rm -f $(WEIGHTSFILEGZ)
+	gzip $(WEIGHTSFILE)
+
+########################################################################
+#                                                                      #
+# eval-reranker evaluates the reranker on the two test data sets       #
+#                                                                      #
+########################################################################
+
+EVALDIR=second-stage/eval/$(PARSERNICKNAME)$(NPARSES)$(FEATURESNICKNAME)$(VERSION)-$(ESTIMATORNICKNAME)
+
+.PHONY: eval-reranker
+eval-reranker: $(EVALDIR)/weights-eval # $(EVALDIR)/dev-parsediffs.gz
+
+$(EVALDIR)/weights-eval: $(WEIGHTSFILEGZ) $(MODELDIR)/features.gz $(FEATDIR)/dev.gz $(FEATDIR)/test1.gz $(FEATDIR)/test2.gz second-stage/programs/eval-weights/eval-weights
+	mkdir -p $(EVALDIR)
+	zcat $(WEIGHTSFILEGZ) | second-stage/programs/eval-weights/eval-weights $(EVALWEIGHTSARGS) $(MODELDIR)/features.gz $(FEATDIR)/dev.gz > $(EVALDIR)/weights-eval
+	zcat $(WEIGHTSFILEGZ) | second-stage/programs/eval-weights/eval-weights $(EVALWEIGHTSARGS) $(MODELDIR)/features.gz $(FEATDIR)/test1.gz >> $(EVALDIR)/weights-eval
+	zcat $(WEIGHTSFILEGZ) | second-stage/programs/eval-weights/eval-weights $(EVALWEIGHTSARGS) $(MODELDIR)/features.gz $(FEATDIR)/test2.gz >> $(EVALDIR)/weights-eval
+
+$(EVALDIR)/dev-parsediffs.gz: $(WEIGHTSFILEGZ) $(FEATDIR)/test1.gz $(NBESTDIR)/section24.gz second-stage/programs/eval-weights/best-indices second-stage/programs/eval-weights/best-parses second-stage/programs/eval-weights/pretty-print
+	zcat $(WEIGHTSFILEGZ) \
+	 | second-stage/programs/eval-weights/best-indices $(FEATDIR)/test1.gz \
+	 | second-stage/programs/eval-weights/best-parses $(NBESTDIR)/section24.gz \
+	 | second-stage/programs/eval-weights/pretty-print -d \
+	 | gzip > $(EVALDIR)/dev-parsediffs.gz
+# Top-level makefile for reranking parser
+# Mark Johnson, 8th December 2007
+
+########################################################################
+#                                                                      #
+#                                Summary                               #
+#                                                                      #
+########################################################################
+#
+# To build just the reranking parser run-time, execute:
+#
+# make                 # builds reranking parser programs
+#
+# To retrain the reranking parser, run the following steps:
+#
+# make reranker        # builds reranking parser and training programs
+# make nbesttrain      # builds 20 folds of n-best training parses
+# make eval-reranker   # extracts features, estimates weights, and evaluates
+#
+# The following high-level goals may also be useful:
+#
+# make nbestrain-clean # removes temporary files used in nbesttrain
+# make nbest-oracle    # oracle evaluation of n-best results 
+# make features        # extracts features from 20-fold parses
+# make train-reranker  # trains reranker model
+# make train-clean     # removes all temporary files used in training
+#
+# I typically run nbesttrain to produce the n-best parses 
+
+# To run 2 jobs in parallel (e.g. on a multiprocessor) run, e.g.,
+#
+# make -j 2 nbesttrain
+#
+# This really only helps with nbesttrain, since the other time consuming
+# step (reranker feature weight estimation) isn't yet parallelized.
+
+# The environment variable GCCFLAGS can be used to specify
+# machine-dependent optimization flags, e.g.
+#
+# setenv GCCFLAGS "-march=pentium4"
+#
+# or
+#
+# setenv GCCFLAGS "-march=opteron -m64"
+#
+# The top-level make goal builds the reranking parser using a pre-trained
+# model.  To build this parser, just run
+#
+# make 
+#
+# You may need to tweak the following variables to suit your environment
+
+# GCCFLAGS is not set here, so we use the shell environment
+# variable's value.  But you can set it here if you want.
+# Version 4.1 and later gcc permit -march=native, but older
+# versions will need -march=pentium4 or -march=opteron
+#
+# GCCFLAGS = -march=native -mfpmath=sse -msse2 -mmmx -I <path-to-boost-libraries>
+
+# CFLAGS is used for all C and C++ compilation
+#
+CFLAGS = -MMD -O6 -Wall -ffast-math -finline-functions -fomit-frame-pointer -fstrict-aliasing $(GCCFLAGS)
+
+# for debugging, uncomment the following CFLAGS and LDFLAGS
+#
+# CFLAGS = -g -O -MMD -Wall -ffast-math -fstrict-aliasing $(GCCFLAGS)
+# LDFLAGS = -g -Wall
+
+
+# Building the 20-fold training data with nbesttrain 
+# --------------------------------------------------
+
+# For training the parser and reranker you will need your own copy of the
+# Penn WSJ Treebank.
+#
+# PENNWSJTREEBANK must be set to the base directory of the Penn WSJ Treebank
+#
+PENNWSJTREEBANK=/usr/local/data/Penn3/parsed/mrg/wsj/
+
+# NPARSES is the number of alternative parses to consider for each sentence
+#
+NPARSES=50
+
+# NFOLDS is the number of folds to use, and FOLDS is a list of the numbers
+# from 00 to NFOLDS-1 (I couldn't see how to program this in make).
+#
+NFOLDS=20
+FOLDS=00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19
+
+# SECTIONS is a list of sections from the treebank to n-best parse
+# using Eugene's standard n-best parser (in addition to the folds).
+#
+SECTIONS=22 23 24
+
+# NBESTPARSER is the n-best parser.  If you change this, please
+# change NBESTPARSERNICKNAME below as well.
+#
+NBESTPARSER=first-stage/PARSE/parseIt
+
+# NBESTTRAINER is the program (probably a shell script) for training
+# the n-best parser.  If you change this, please change
+# NBESTPARSERNICKNAME below as well.
+#
+NBESTTRAINER=first-stage/TRAIN/allScript
+
+# NBESTPARSERNICKNAME is a nickname for the n-best parser.  If you 
+# experiment with several n-best parsers, give each one a different
+# nickname.
+#
+PARSERNICKNAME=ec
+
+# TMP specifies a temporary directory used while constructing the
+# folds while producing the training parses for the reranker.  You can
+# delete this directory after nbesttrain has finished.  On an NFS
+# system you may want to change this to a local directory.
+#
+TMP=tmp
+
+# Extracting features from 20-fold n-best parses
+# ----------------------------------------------
+
+# VERSION should be either "final" or "nonfinal".  If VERSION is
+# "nonfinal" then we train on folds 00-19, folds 20-21 are used as
+# dev, and sections 22 and 24 are used as test1 and test2
+# respectively.  If VERSION is "final" then we train on folds 00-21,
+# section 24 is used as dev and sections 22 and 23 are used as test1
+# and test2 respectively.
+#
+# VERSION=nonfinal
+VERSION=final
+
+# FEATUREEXTRACTOR is the program that used to extract features from
+# the 20-fold n-best parses.  If you change this, please pick a new 
+# FEATURESNICKNAME below.
+#
+FEATUREEXTRACTOR=second-stage/programs/features/extract-splhfeatures
+
+# FEATUREEXTRACTORFLAGS are flags you want to give to the feature extractor
+#
+FEATUREEXTRACTORFLAGS=-f splh -l -c -i -s 5
+
+# FEATURESNICKNAME is an arbitrary string used to identify a
+# particular set of extracted features for training the reranker.  You
+# can keep several different sets of feature counts and corresponding
+# models around by giving each a unique FEATURESNICKNAME.  If you
+# develop a new set of features, give them a new FEATURESNICKNAME so
+# they doesn't over-write the existing features.
+#
+FEATURESNICKNAME=splh
+
+# Estimating weights for features
+# -------------------------------
+
+# ESTIMATOR is the program used to estimate feature weights from the
+# feature counts. This is the feature weight estimator that gives best
+# performance.  There are others in the same directory (e.g., weighted
+# perceptron).  If you decide to use a different feature weight
+# estimator you should also change ESTIMATORNICKNAME below.
+#
+ESTIMATOR=second-stage/programs/wlle/cvlm
+
+# ESTIMATORFLAGS are flags given to the estimator
+#
+ESTIMATORFLAGS=-l 1 -c0 10 -Pyx_factor 1 -debug 10 -ns -1
+
+# ESTIMATORNICKNAME
+#
+ESTIMATORNICKNAME=cvlm-l1c10P1
+
+########################################################################
+#
+# You probably shouldn't need to change anything below here.
+
+# TARGETS is the list of targets built when make is called
+# without arguments
+#
+TARGETS = PARSE reranker-runtime evalb
+
+CXXFLAGS = $(CFLAGS)
+export CFLAGS
+export CXXFLAGS
+
+.PHONY: top
+top: $(TARGETS)
+
+# PARSE builds the n-best first-stage parser (i.e., Eugene's parser).
+#
+.PHONY: PARSE
+PARSE:
+	make -C first-stage/PARSE parseIt
+
+# TRAIN builds the programs needed to train the first-stage parser.
+#
+.PHONY: TRAIN
+TRAIN:
+	make -C first-stage/TRAIN all
+
+# reranker-runtime builds the run-time components of the reranker.
+# These include best-parses, which reranks the n-best parses produced
+# by the first-stage parser, and ptb, which is a program that converts
+# Penn Treebank trees into the various formats needed by Eugene's
+# parser, the reranker training programs, EVALB, etc.  
+#
+.PHONY: reranker-runtime
+reranker-runtime:
+	make -C second-stage/programs/features best-parses
+	make -C second-stage/programs/prepare-data ptb
+
+# reranker builds the training and run-time components of the reranker.
+# These include:
+#  ptb, which converts the Penn Treebank parse trees into
+#       the various formats needed by Eugene's parser, the reranker training
+#       program, EVALB, etc., 
+#  extract-spfeatures, which produces feature-count files used to train 
+#       the reranker, 
+#  cvlm, which estimates the feature weights.
+#
+.PHONY: reranker
+reranker: top TRAIN
+	make -C second-stage
+
+# Good old EVALB!
+#
+.PHONY: evalb
+evalb: 
+	make -C evalb evalb
+
+# clean removes object files.
+#
+.PHONY: clean
+clean:
+	(cd first-stage; rm -f PARSE/*.o; rm -f TRAIN/*.o)
+	make -C first-stage/TRAIN clean
+	make -C first-stage/PARSE clean
+	make -C second-stage clean
+
+# nbesttrain-clean removes temporary files used in constructing the 20
+# folds of n-best training data.
+#
+nbesttrain-clean:
+	rm -fr $(TMP)
+
+# train-clean gets rid of all data not essential for the reranking
+# parser. 
+#
+.PHONY: train-clean
+train-clean: nbesttrain-clean
+	rm -fr results
+	make -C second-stage train-clean
+
+# real-clean tries to get rid of all object and binary files to
+# produce a version for distribution.  But Eugene writes new programs
+# faster than I can make real-clean clean them up!
+#
+.PHONY: real-clean
+real-clean: clean train-clean
+	(cd first-stage; rm -f PARSE/parseIt)
+	make -C second-stage real-clean
+
+########################################################################
+#                                                                      #
+# nbesttrain -- Preparing the N-best training data for the reranker    #
+#                                                                      #
+########################################################################
+
+# To build the 20-fold n-best data in second-stage/train 
+# for training the ranker, run
+#
+# make nbesttrain 
+#
+# or
+# 
+# make -j 2 nbesttrain
+#
+# on a multiprocessor machine
+
+# TRAIN specifies the location of the trees to be divided into NFOLDS
+# This is defined here to use sections 2-21 of the Penn WSJ treebank.
+#
+TRAIN=$(PENNWSJTREEBANK)/0[2-9]/*mrg $(PENNWSJTREEBANK)/1[0-9]/*mrg $(PENNWSJTREEBANK)/2[0-1]/*mrg
+
+# NBESTDIR is the directory that holds the n-best parses for training
+# the reranker.
+#
+NBESTDIR=second-stage/nbest/$(PARSERNICKNAME)$(NPARSES)
+
+# NBESTFILES are all of the files in the n-best folds, plus dev and test sections
+#
+NBESTFILES= $(foreach fold,$(FOLDS),$(NBESTDIR)/fold$(fold).gz) $(foreach section,$(SECTIONS),$(NBESTDIR)/section$(section).gz)
+
+.PHONY: nbesttrain
+nbesttrain: $(NBESTFILES) PARSE TRAIN second-stage/programs/prepare-data/ptb
+
+# This goal copies and gzips the output of the n-best parser
+# into the appropriate directory for training the reranker.
+#
+# .PRECIOUS: $(NBESTDIR)/fold%.gz
+.INTERMEDIATE: $(NBESTDIR)/fold%.gz
+$(NBESTDIR)/fold%.gz: $(TMP)/fold%/$(NPARSES)best
+	mkdir -p $(NBESTDIR)
+	gzip -c $+ > $@
+
+# The remaining goals in this section are for training and parsing
+# with the n-best parser to produce the folds for training the
+# reranker.
+
+.INTERMEDIATE: $(TMP)/fold%/$(NPARSES)best
+$(TMP)/fold%/$(NPARSES)best: $(TMP)/fold%/DATA $(TMP)/fold%/yield $(NBESTPARSER)
+	$(NBESTPARSER) -l400 -K -N$(NPARSES) $(@D)/DATA/ $(@D)/yield > $@
+
+.INTERMEDIATE: $(TMP)/fold%/DATA
+$(TMP)/fold%/DATA: $(TMP)/fold%/train $(TMP)/fold%/dev $(NBESTTRAINER)
+	mkdir -p $@
+	LC_COLLATE=C; cp first-stage/DATA/EN/[a-z]* $@
+	$(NBESTTRAINER) $@ $(@D)/train $(@D)/dev
+
+.INTERMEDIATE: $(TMP)/fold%/train
+$(TMP)/fold%/train: second-stage/programs/prepare-data/ptb
+	mkdir -p $(@D)
+	second-stage/programs/prepare-data/ptb -n $(NFOLDS) -x $(patsubst $(TMP)/fold%,%,$(@D)) -e $(TRAIN)  > $@
+
+.INTERMEDIATE: $(TMP)/fold%/dev
+$(TMP)/fold%/dev: second-stage/programs/prepare-data/ptb
+	mkdir -p $(@D)
+	second-stage/programs/prepare-data/ptb -n $(NFOLDS) -i $(patsubst $(TMP)/fold%,%,$(@D)) -e $(TRAIN)  > $@
+
+# $(TMP)/fold%/DATA: $(TMP)/%/train $(TMP)/%/dev
+# 	mkdir -p $@
+# 	LC_COLLATE=C; cp first-stage/DATA/EN/[a-z]* $@
+# 	first-stage/TRAIN/allScript $@ $(@D)/train $(@D)/dev
+
+.INTERMEDIATE: $(TMP)/fold%/yield
+$(TMP)/fold%/yield: second-stage/programs/prepare-data/ptb
+	mkdir -p $(@D)
+	second-stage/programs/prepare-data/ptb -n $(NFOLDS) -i $(patsubst $(TMP)/fold%,%,$(@D)) -c $(TRAIN) > $@
+
+# .PRECIOUS: $(NBESTDIR)/section%.gz
+.INTERMEDIATE: $(NBESTDIR)/section%.gz
+$(NBESTDIR)/section%.gz: $(TMP)/section%/$(NPARSES)best
+	mkdir -p $(NBESTDIR)
+	gzip -c $+ > $@
+
+.INTERMEDIATE: $(TMP)/section%/$(NPARSES)best
+$(TMP)/section%/$(NPARSES)best: $(TMP)/section%/yield $(NBESTPARSER)
+	$(NBESTPARSER) -l400 -K -N$(NPARSES) first-stage/DATA/EN/ $(@D)/yield > $@
+
+.INTERMEDIATE: $(TMP)/section%/yield
+$(TMP)/section%/yield: second-stage/programs/prepare-data/ptb
+	mkdir -p $(@D)
+	second-stage/programs/prepare-data/ptb -c $(PENNWSJTREEBANK)/$(patsubst $(TMP)/section%,%,$(@D))/wsj*.mrg  > $@
+
+########################################################################
+#                                                                      #
+# nbest oracle evaluation                                              #
+#                                                                      #
+########################################################################
+
+.PHONY: nbest-oracle
+nbest-oracle: second-stage/programs/features/oracle-score second-stage/programs/prepare-data/ptb $(NBESTFILES)
+	second-stage/programs/features/oracle-score "zcat $(NBESTDIR)/fold[0-1][0-9].gz" "second-stage/programs/prepare-data/ptb -g $(TRAIN)"
+	second-stage/programs/features/oracle-score "zcat $(NBESTDIR)/section22.gz" "second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/22/wsj*.mrg"
+	second-stage/programs/features/oracle-score "zcat $(NBESTDIR)/section24.gz" "second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/24/wsj*.mrg"
+
+.PHONY: nbest-oracle-detailed
+nbest-oracle-detailed: second-stage/programs/eval-beam/main second-stage/programs/prepare-data/ptb $(NBESTFILES)
+	second-stage/programs/eval-beam/main "zcat $(NBESTDIR)/fold[0-1][0-9].gz" "second-stage/programs/prepare-data/ptb -g $(TRAIN)"
+	second-stage/programs/eval-beam/main "zcat $(NBESTDIR)/section22.gz" "second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/22/wsj*.mrg"
+	second-stage/programs/eval-beam/main "zcat $(NBESTDIR)/section24.gz" "second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/24/wsj*.mrg"
+
+########################################################################
+#                                                                      #
+# extract-features extracts feature counts for training reranker       #
+#                                                                      #
+########################################################################
+
+# FEATBASEDIR is the directory in which the feature counts will be saved,
+# minus the $(VERSION) flag.
+#
+FEATBASEDIR=second-stage/features/$(PARSERNICKNAME)$(NPARSES)$(FEATURESNICKNAME)
+
+# FEATDIR is the directory in which the feature counts will be saved.
+#
+FEATDIR=$(FEATBASEDIR)$(VERSION)
+
+# MODELBASEDIR is the directory in which the features and feature
+# weights are saved, minus the version.
+#
+MODELBASEDIR=second-stage/models/$(PARSERNICKNAME)$(NPARSES)$(FEATURESNICKNAME)
+
+# MODELDIR is the directory in which the features and feature weights
+# are saved.
+#
+MODELDIR=$(MODELBASEDIR)$(VERSION)
+
+.PHONY: features
+features: $(MODELDIR)/features.gz $(FEATDIR)/train.gz $(FEATDIR)/dev.gz $(FEATDIR)/test1.gz $(FEATDIR)/test2.gz
+
+# This goal does feature extraction for reranker training for the
+# nonfinal case (i.e., train is folds 0-17, dev is folds 18-19, test1
+# is section 22 and test2 is section 24).
+#
+$(MODELBASEDIR)nonfinal/features.gz $(FEATBASEDIR)nonfinal/train.gz $(FEATBASEDIR)nonfinal/dev.gz $(FEATBASEDIR)nonfinal/test1.gz $(FEATBASEDIR)nonfinal/test2.gz: second-stage/programs/prepare-data/ptb $(FEATUREEXTRACTOR) $(NBESTFILES)
+	mkdir -p $(FEATBASEDIR)nonfinal
+	mkdir -p $(MODELBASEDIR)nonfinal
+	$(FEATUREEXTRACTOR) $(FEATUREEXTRACTORFLAGS) \
+		"zcat $(NBESTDIR)/fold0[0-9].gz $(NBESTDIR)/fold1[0-7].gz" \
+		"second-stage/programs/prepare-data/ptb -g -n 10 -x 9 $(TRAIN)" \
+		$(FEATBASEDIR)nonfinal/train.gz \
+		"zcat $(NBESTDIR)/fold1[8-9].gz" \
+		"second-stage/programs/prepare-data/ptb -g -n 10 -i 9 $(TRAIN)" \
+		$(FEATBASEDIR)nonfinal/dev.gz \
+		"zcat $(NBESTDIR)/section22.gz" \
+		"second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/22/*mrg" \
+		$(FEATBASEDIR)nonfinal/test1.gz \
+		"zcat $(NBESTDIR)/section24.gz" \
+		"second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/24/*mrg" \
+		$(FEATBASEDIR)nonfinal/test2.gz \
+		| gzip > $(MODELBASEDIR)nonfinal/features.gz
+
+# This goal does feature extraction for reranker training for the
+# final case (i.e., train is folds 0-19, dev is section 24, test1
+# is section 22 and test2 is section 23).
+#
+$(MODELBASEDIR)final/features.gz $(FEATBASEDIR)final/train.gz $(FEATBASEDIR)final/dev.gz $(FEATBASEDIR)final/test1.gz $(FEATBASEDIR)final/test2.gz: second-stage/programs/prepare-data/ptb $(FEATUREEXTRACTOR) $(NBESTFILES)
+	mkdir -p $(FEATBASEDIR)final
+	mkdir -p $(MODELBASEDIR)final
+	$(FEATUREEXTRACTOR) $(FEATUREEXTRACTORFLAGS) \
+		"zcat $(NBESTDIR)/fold*.gz" \
+		"second-stage/programs/prepare-data/ptb -g $(TRAIN)" \
+		$(FEATBASEDIR)final/train.gz \
+		"zcat $(NBESTDIR)/section22.gz" \
+		"second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/22/*mrg" \
+		$(FEATBASEDIR)final/test1.gz \
+		"zcat $(NBESTDIR)/section23.gz" \
+		"second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/23/*mrg" \
+		$(FEATBASEDIR)final/test2.gz \
+		"zcat $(NBESTDIR)/section24.gz" \
+		"second-stage/programs/prepare-data/ptb -g $(PENNWSJTREEBANK)/24/*mrg" \
+		$(FEATBASEDIR)final/dev.gz \
+		| gzip > $(MODELBASEDIR)final/features.gz
+
+
+########################################################################
+#                                                                      #
+# train-reranker estimates the reranker feature weights                #
+#                                                                      #
+########################################################################
+
+WEIGHTSFILE=$(MODELDIR)/$(ESTIMATORNICKNAME)-weights
+WEIGHTSFILEGZ=$(WEIGHTSFILE).gz
+
+.PHONY: train-reranker
+train-reranker: $(WEIGHTSFILEGZ)
+
+# This goal estimates the reranker feature weights (i.e., trains the
+# reranker).  This is not hard to parallelize, but I haven't actually
+# done that yet.
+#
+# $(WEIGHTSFILEGZ): $(ESTIMATOR)
+$(WEIGHTSFILEGZ): $(ESTIMATOR) $(MODELDIR)/features.gz $(FEATDIR)/train.gz $(FEATDIR)/dev.gz $(FEATDIR)/test1.gz
+	zcat $(FEATDIR)/train.gz | $(ESTIMATOR) $(ESTIMATORFLAGS) -e $(FEATDIR)/dev.gz -f $(MODELDIR)/features.gz -o $(WEIGHTSFILE) -x $(FEATDIR)/test1.gz
+	rm -f $(WEIGHTSFILEGZ)
+	gzip $(WEIGHTSFILE)
+
+########################################################################
+#                                                                      #
+# eval-reranker evaluates the reranker on the two test data sets       #
+#                                                                      #
+########################################################################
+
+EVALDIR=second-stage/eval/$(PARSERNICKNAME)$(NPARSES)$(FEATURESNICKNAME)$(VERSION)-$(ESTIMATORNICKNAME)
+
+.PHONY: eval-reranker
+eval-reranker: $(EVALDIR)/weights-eval # $(EVALDIR)/dev-parsediffs.gz
+
+$(EVALDIR)/weights-eval: $(WEIGHTSFILEGZ) $(MODELDIR)/features.gz $(FEATDIR)/dev.gz $(FEATDIR)/test1.gz $(FEATDIR)/test2.gz second-stage/programs/eval-weights/eval-weights
+	mkdir -p $(EVALDIR)
+	zcat $(WEIGHTSFILEGZ) | second-stage/programs/eval-weights/eval-weights $(EVALWEIGHTSARGS) $(MODELDIR)/features.gz $(FEATDIR)/dev.gz > $(EVALDIR)/weights-eval
+	zcat $(WEIGHTSFILEGZ) | second-stage/programs/eval-weights/eval-weights $(EVALWEIGHTSARGS) $(MODELDIR)/features.gz $(FEATDIR)/test1.gz >> $(EVALDIR)/weights-eval
+	zcat $(WEIGHTSFILEGZ) | second-stage/programs/eval-weights/eval-weights $(EVALWEIGHTSARGS) $(MODELDIR)/features.gz $(FEATDIR)/test2.gz >> $(EVALDIR)/weights-eval
+
+$(EVALDIR)/dev-parsediffs.gz: $(WEIGHTSFILEGZ) $(FEATDIR)/test1.gz $(NBESTDIR)/section24.gz second-stage/programs/eval-weights/best-indices second-stage/programs/eval-weights/best-parses second-stage/programs/eval-weights/pretty-print
+	zcat $(WEIGHTSFILEGZ) \
+	 | second-stage/programs/eval-weights/best-indices $(FEATDIR)/test1.gz \
+	 | second-stage/programs/eval-weights/best-parses $(NBESTDIR)/section24.gz \
+	 | second-stage/programs/eval-weights/pretty-print -d \
+	 | gzip > $(EVALDIR)/dev-parsediffs.gz
+~BLLIP/reranking-parser/README
+
+(c) Mark Johnson,Eugene Charniak, 24th November 2005 --- August 2006
+
+We request acknowledgement in any publications that make use of this
+software and any code derived from this software.  Please report the
+release date of the software that you are using (this is part of the
+name of the tar file you have downloaded), as this will enable others
+to compare their results to yours.
+
+MULTI-THREADED PARSING
+======================
+
+NEW!!!  The first stage parser, which uses about 95% of the time, is
+now multi-treaded. The default is two threads.  Currently the maximum
+is four.  To change the number (or maximum) see the README file for
+the first-stage parser.  For the time being a non-threaded version is
+available in case there are problems with threads.  Send email to ec
+if you have problems. See below for details.
+
+COMPILING THE PARSER
+====================
+
+To compile the two-stage parser, first define GCCFLAGS appropriately
+for your machine, e.g., with csh or tcsh
+
+> setenv GCCFLAGS "-march=pentium4 -mfpmath=sse -msse2 -mmmx"
+
+or
+
+> setenv GCCFLAGS "-march=opteron -m64"
+
+Then execute 
+
+make
+
+After it has built, the parser can be run with
+
+> parse.sh <sourcefile.txt>
+
+E.g.,
+
+> parse.sh sample-data.txt
+
+The script parse-eval.sh takes a list of treebank files as arguments
+and extracts the terminal strings from them, runs the two-stage parser
+on those terminal strings and then evaluates the parsing accuracy with
+the U. Penn EVAL-B program.  For example, on my machine the Penn
+Treebank 3 CD-ROM is installed at /usr/local/data/Penn3/, so the
+following code evaluates the two-stage parser on section 24.
+
+> parse-eval.sh /usr/local/data/Penn3/parsed/mrg/wsj/24/wsj*.mrg
+
+
+TRAINING THE RERANKER
+=====================
+
+Retraining the reranker takes a considerable amount of time, disk
+space and RAM.  At Brown we use a dual Opteron machine with 16Gb RAM,
+and it takes around two days.  You should be able to do it with only
+8Gb RAM, and maybe even with 4Gb RAM with an appropriately tweaked
+kernel (e.g., sysctl overcommit_memory, and a so-called 4Gb/4Gb split
+if you're using a 32-bit OS).  
+
+The time and memory you need depend on the features that the reranker
+extracts and the size of the n-best tree training and development
+data.  You can change the features that are extracted by changing
+second-stage/programs/features/features.h, and you can reduce the size
+of the n-best tree data by reducing NPARSES in the Makefile from 50
+to, say, 25.
+
+You will need to edit the Makefile in order to retrain the reranker.
+
+First, you need to set the variable PENNWSJTREEBANK in Makefile to the
+directory that holds your version of the Penn WSJ Treebank.  On my
+machine this is:
+
+PENNWSJTREEBANK=/usr/local/data/Penn3/parsed/mrg/wsj/
+
+You'll also need the Boost C++ and the Petsc/Tao C++ libraries in
+order to retrain the reranker.  The environment variables PETSC_DIR
+and TAO_DIR should all point to the installation directories of this
+software.  I define these variables in my .login file as follows on my
+machine.
+
+setenv PETSC_DIR /usr/local/share/petsc
+setenv TAO_DIR /usr/local/share/tao
+setenv PETSC_ARCH linux
+setenv BOPT O_c++
+
+While many modern Linux distributions come with the Boost C++
+libraries pre-installed, if the Boost C++ libraries are not included
+in your standard libraries and headers, you will need to install them
+and add an include file specification for them in your GCCFLAGS.  For
+example, if you have installed the Boost C++ libraries in
+/home/mj/C++/boost, then your GCCFLAGS environment variable should be
+something like:
+
+> setenv GCCFLAGS "-march=pentium4 -mfpmath=sse -msse2 -mmmx -I /home/mj/C++/boost"
+
+or
+
+> setenv GCCFLAGS "-march=opteron -m64 -I /home/mj/C++/boost"
+
+Once this is set up, you retrain the reranker as follows:
+
+> make reranker 
+> make nbesttrain
+> make eval-reranker
+
+The script train-eval-reranker.sh does all of this.
+
+The reranker goal builds all of the programs, nbesttrain constructs
+the 20 folds of n-best parses required for training, and eval-reranker
+extracts features, estimates their weights and evaluates the
+reranker's performance on the development data (dev) and the two test
+data sets (test1 and test2).
+
+If you have a parallel processor, you can run 2 (or more) jobs
+in parallel by running
+
+> make -j 2 nbesttrain
+
+Currently this only helps for nbesttrain (but this is the slowest
+step, so maybe this is not so bad).
+
+The Makefile contains a number of variables that control how the
+training process works.  The most important of these is the VERSION
+variable.  You should do all of your experiments with VERSION=nonfinal,
+and only run with VERSION=final once to produce results for publication.
+
+If VERSION is nonfinal then the reranker trains on WSJ PTB sections
+2-19, sections 20-21 are used for development, section 22 is used as
+test1 and section 24 is used as test2 (this approximately replicates
+the Collins 2000 setup).
+
+If VERSION is final then the reranker trains on WSJ PTB sections 2-21,
+section 24 is used for development, section 22 is used as test1 and
+section 23 is used as test2.
+
+The Makefile also contains variables you may want to change, such as
+NBEST, which specfies how many parses per sentence are extracted from
+each sentence, and NFOLDS, which specifies how many folds are created.
+
+If you decide to experiment with new features or new feature weight
+estimators, take a close look at the Makefile.  If you change the
+features please also change FEATURESNICKNAME; this way your new
+features won't over-write our existing ones.  Similarly, if you change
+the feature weight estimator please pick a new ESTIMATORNICKNAME and
+if you change the n-best parser please pick a new NBESTPARSERNICKNAME;
+this way you new n-best parses or feature weights won't over-write the
+existing ones.
+
+To get rid of (many of) the object files produced in compilation, run:
+
+> make clean
+
+Training, especially constructing the 20 folds of n-best parses,
+produces a lot of temporary files which you can remove if you want to.
+To remove the temporary files used to construct the 20 fold n-best
+parses, run:
+
+> make nbesttrain-clean
+
+All of the information needed by the reranker is in
+second-stage/models.  To remove everything except the information
+needed for running the reranking parser, run:
+
+> make train-clean
+
+To clean up everything, including the data needed for running the
+reranking parser, run:
+
+> make real-clean
+
+
+NON-THREADED PARSER
+===================
+To use the non-threaded parser instead change the following line
+in the Makefile
+
+NBESTPARSER=first-stage/PARSE/parseIt
+
+It should now read:
+NBESTPARSER=first-stage/PARSE/oparseIt
+
+That is, it is identical except for the "o" in oparseIt
+
+Then run oparse.sh, rather than parse.sh.  
+
+
+INSTALLING PETSC AND TAO
+========================
+
+You'll need to have PETSc and Tao installed in order to 
+retrain the reranker.
+
+These installation instructions work for gcc version 4.2.1 (you also
+need g++ and gfortran).
+
+1. Unpack PETSc and TAO somewhere, and make shell variables point
+to those directories (put the shell variable definitions in your
+.bash_profile or equivalent)
+
+export PETSC_DIR=/usr/local/share/petsc
+export TAO_DIR=/usr/local/share/tao
+export PETSC_ARCH="linux"
+export BOPT=O_c++
+
+cd /usr/local/share
+ln -s petsc-2.3.3-p6 petsc
+ln -s tao-1.9 tao
+
+2. Configure and build PETSc
+
+cd petsc
+FLAGS="-march=native -mfpmath=sse -msse2 -mmmx -O3 -ffast-math"
+./config/configure.py --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --download-f-blas-lapack=1 --with-mpi=0 --with-clanguage=C++ --with-shared=1 --with-dynamic=1 --with-debugging=0 --with-x=0 --with-x11=0 COPTFLAGS=$FLAGS FOPTFLAGS=$FLAGS CXXOPTFLAGS=$FLAGS
+make all
+
+3. Configure and build TAO
+
+cd ../tao
+make all
+
+README.results:
+==============
+
+This file records the results of the n-best parser under VERSION=nonfinal, training on
+sections 2-19, using 20-21 as dev and testing on section 22 (test1) and section 24 (test2).
+
+version of 4th September 2006
+=============================
+
+# Evaluating second-stage/features/ec50spnonfinal/dev.gz
+# 1219273 features in second-stage/models/ec50spnonfinal/features.gz
+# 3984 sentences in second-stage/features/ec50spnonfinal/dev.gz
+# ncorrect = 66991, ngold = 73905, nparse = 73397, f-score = 0.909574, -log P = 18859.3, 1219273 nonzero features, mean w = 0.000210852, sd w = 0.000403686
+# Evaluating second-stage/features/ec50spnonfinal/test1.gz
+# 1219273 features in second-stage/models/ec50spnonfinal/features.gz
+# 1700 sentences in second-stage/features/ec50spnonfinal/test1.gz
+# ncorrect = 28022, ngold = 30633, nparse = 30460, f-score = 0.917356, -log P = 8087.09, 1219273 nonzero features, mean w = 0.000210852, sd w = 0.000403686
+# Evaluating second-stage/features/ec50spnonfinal/test2.gz
+# 1219273 features in second-stage/models/ec50spnonfinal/features.gz
+# 1346 sentences in second-stage/features/ec50spnonfinal/test2.gz
+# ncorrect = 23121, ngold = 25729, nparse = 25327, f-score = 0.905711, -log P = 6257.9, 1219273 nonzero features, mean w = 0.000210852, sd w = 0.000403686
+#! /bin/sh
+
+# This script recompiles the reranker code, rebuilds the nbest trees
+# and retrains and evaluates the reranker itself.
+
+# You can change the flags below here
+
+NBESTPARSERBASEDIR=first-stage
+NBESTPARSERNICKNAME=ec
+
+# NBESTPARSERBASEDIR=first-stage-Aug06
+# NBESTPARSERNICKNAME=Aug06
+
+# FEATUREEXTRACTOR=second-stage/programs/features/extract-spfeatures
+# FEATUREEXTRACTORFLAGS="-l -c -i -s 5"
+# FEATURESNICKNAME=spc
+
+FEATUREEXTRACTOR=second-stage/programs/features/extract-nfeatures
+FEATUREEXTRACTORFLAGS="-l -c -i -s 5 -f splh"
+FEATURESNICKNAME=splh
+
+# ESTIMATOR=second-stage/programs/wlle/cvlm
+# ESTIMATORFLAGS="-l 1 -c0 10 -Pyx_factor 1 -debug 10 -ns -1"
+# ESTIMATORNICKNAME=cvlm-l1c10P1-openmp
+
+ESTIMATOR=second-stage/programs/wlle/cvlm-owlqn
+ESTIMATORFLAGS="-l 1 -c 10 -F 1 -d 10 -n -1 -t 1e-7"
+ESTIMATORNICKNAME=owlqn-l1c10t1e-7
+
+# ESTIMATOR=second-stage/programs/wlle/cvlm-owlqn
+# ESTIMATORFLAGS="-l 1 -p 1 -c 10 -F 1 -d 10 -n -1 -t 1e-7"
+# ESTIMATORNICKNAME=owlqn-l1c10p1t1e-7
+
+# ESTIMATOR=second-stage/programs/wlle/avper
+# ESTIMATORFLAGS="-n 10 -d 0 -F 1 -N 10"
+# ESTIMATORNICKNAME=avper
+
+# ESTIMATOR=second-stage/programs/wlle/gavper
+# ESTIMATORFLAGS="-a -n 10 -d 10 -F 1 -m 999999"
+# ESTIMATORNICKNAME=gavper
+
+# ESTIMATOR=second-stage/programs/wlle/hlm
+# ESTIMATORFLAGS="-l 1 -c 10 -C 10000 -F 1 -d 100 -n 0 -S 7 -t 1e-7"
+# ESTIMATORNICKNAME=hlm2S7
+
+
+###############################################################################
+#
+# You shouldn't need to change anything below here
+#
+FLAGS="NBESTPARSERBASEDIR=$NBESTPARSERBASEDIR NBESTPARSERNICKNAME=$NBESTPARSERNICKNAME FEATUREEXTRACTOR=$FEATUREEXTRACTOR FEATURESNICKNAME=$FEATURESNICKNAME ESTIMATOR=$ESTIMATOR ESTIMATORNICKNAME=$ESTIMATORNICKNAME"
+
+# echo make clean $FLAGS
+# make clean
+
+echo
+echo make reranker $FLAGS
+make reranker $FLAGS
+
+# echo
+# echo make -j 8 nbesttrain
+# make -j 8 nbesttrain
+
+# Avoid remaking the nbest parses.  Warning -- you'll recompute the features if you run this!
+#
+# echo
+# echo make touch-nbest $FLAGS
+# make touch-nbest $FLAGS
+
+# The nonfinal version trains on sections 2-19, uses sections 20-21 as dev,
+# section 22 as test1 and 24 as test2 (this is the "Collins' split")
+#
+echo
+echo make eval-reranker VERSION=nonfinal $FLAGS FEATUREEXTRACTORFLAGS="$FEATUREEXTRACTORFLAGS" ESTIMATORFLAGS="$ESTIMATORFLAGS"
+time make eval-reranker VERSION=nonfinal $FLAGS FEATUREEXTRACTORFLAGS="$FEATUREEXTRACTORFLAGS" ESTIMATORFLAGS="$ESTIMATORFLAGS"
+
+# The final version trains on sections 2-21, uses section 24 as dev, 
+# section 22 as test1 and section 23 as test2 (this is the standard PARSEVAL split)
+# 
+echo
+echo make eval-reranker VERSION=final $FLAGS FEATUREEXTRACTORFLAGS="$FEATUREEXTRACTORFLAGS" ESTIMATORFLAGS="$ESTIMATORFLAGS"
+make eval-reranker VERSION=final $FLAGS FEATUREEXTRACTORFLAGS="$FEATUREEXTRACTORFLAGS" ESTIMATORFLAGS="$ESTIMATORFLAGS"
+

evalb/COLLINS.prm

+##------------------------------------------##
+## Debug mode                               ##
+##   0: No debugging                        ##
+##   1: print data for individual sentence  ##
+##------------------------------------------##
+DEBUG 0
+
+##------------------------------------------##
+## MAX error                                ##
+##    Number of error to stop the process.  ##
+##    This is useful if there could be      ##
+##    tokanization error.                   ##
+##    The process will stop when this number##
+##    of errors are accumulated.            ##
+##------------------------------------------##
+MAX_ERROR 10
+
+##------------------------------------------##
+## Cut-off length for statistics            ##
+##    At the end of evaluation, the         ##
+##    statistics for the senetnces of length##
+##    less than or equal to this number will##
+##    be shown, on top of the statistics    ##
+##    for all the sentences                 ##
+##------------------------------------------##
+CUTOFF_LEN 40
+