Commits

Richard Bairwell committed b490493

Fix SQL Injection issues (moving away from old mysql_* connections to parameterised query strings)
Add more documentation to automated processes for logging
Extract out analytics/advertisements to config.php (now checks for the existence of the functions "advertisement" and "remote_analytics")
Remove PHP short tags
Disable election.php as data not currently available.
Add various XSS protection sections (validate strings such as "house=" a bit more)
Remove .cvsignore files as we're are now on GIT
Update code location to BitBucket as we're moving away from GitHub

Comments (0)

Files changed (73)

+loader/PublicWhip/Config.pm
+website/config.php
+website/google*

build/awstats.publicwhip.org.uk.conf

-# AWSTATS CONFIGURE FILE 5.5
-#-----------------------------------------------------------------------------
-# Copy this file into awstats.www.mydomain.conf and edit this new config file
-# to setup AWStats (See documentation in docs/ directory).
-# The config file must be in /etc/awstats, /etc/opt/awstats or /etc (for
-# Unix/Linux) or same directory than awstats.pl (Windows, Mac, Unix/Linux...)
-# To include an environment variable in any parameter (AWStats will replace
-# it with its value when reading it), follow the example:
-# Parameter="__ENVNAME__"
-# Note that environment variable AWSTATS_CURRENT_CONFIG is always defined with
-# the config value in an AWStats running session and can be used like others.
-#-----------------------------------------------------------------------------
-# $Revision: 1.2 $ - $Author: frabcus $ - $Date: 2005/05/06 02:23:41 $
-
-
-
-#-----------------------------------------------------------------------------
-# MAIN SETUP SECTION (Required to make AWStats work)
-#-----------------------------------------------------------------------------
-
-# "LogFile" contains the web server logfile to analyze.
-# Possible values: A full path, or a relative path from awstats.pl directory.
-# Example: "/var/log/apache/access.log"
-# Example: "../logs/mycombinedlog.log"
-# You can also use tags in this filename if you need a dynamic file name
-# depending on date or time (Replacement is made by AWStats at the beginning
-# of its execution). This is available tags :
-#   %YYYY-n  is replaced with 4 digits year we were n hours ago
-#   %YY-n    is replaced with 2 digits year we were n hours ago
-#   %MM-n    is replaced with month we were n hours ago
-#   %DD-n    is replaced with day we were n hours ago
-#   %HH-n    is replaced with hour we were n hours ago
-#   %NS-n    is replaced with number of seconds at 00:00 since 1970
-#   %WM-n    is replaced with the week number in month (1-5)
-#   %Wm-n    is replaced with the week number in month (0-4)
-#   %WY-n    is replaced with the week number in year (01-52)
-#   %Wy-n    is replaced with the week number in year (00-51)
-#   %DW-n    is replaced with the day number in week (1-7, 1=sunday)
-#                              use n=24 if you need (1-7, 1=monday)
-#   %Dw-n    is replaced with the day number in week (0-6, 0=sunday)
-#                              use n=24 if you need (0-6, 0=monday)
-#   Use 0 for n if you need current year, month, day, hour...
-# Example: "/var/log/access_log.%YYYY-0%MM-0%DD-0.log"
-# Example: "C:/WINNT/system32/LogFiles/W3SVC1/ex%YY-24%MM-24%DD-24.log"
-# You can also use a pipe if log file come from a pipe.
-# Example: "gzip -d </var/log/apache/access.log.gz |"
-#
-LogFile="-"
-#/home/francis/webstats/publicwhip/logs/access_log"
-
-# Enter here your log format (Must match your web server config. See setup
-# instructions in documentation know how to configure your web server to have
-# the required log format).
-# Possible values: 1,2,3,4,5 or "your_own_personalized_log_format"
-# 1 - Apache or Lotus Notes/Domino native combined log format (NCSA combined/XLF/ELF log format)
-# 2 - IIS log format (W3C log format)
-# 3 - Webstar native log format
-# 4 - Apache or Squid native common log format (NCSA common/CLF log format)
-#     With LogFormat=4, some features (browsers, os, keywords...) can't work.
-# "your_own_personalized_log_format" = To use AWStats with any not If your log is a personalized format,
-#   you must use the following syntax keys to define the log format string:
-#   %host             Host client name or IP address
-#   %logname          Authenticated login/user used on protected pages
-#   %time1            Date and time with format: [dd/mmm/yyyy:hh:mm:ss +0000]
-#   %time1b           Date and time with format: [dd/mmm/yyyy:hh:mm:ss]
-#   %time2            Date and time with format: yyyy-mm-dd hh:mm:ss
-#   %methodurl        Method and URL with format: "GET /index.html HTTP/x.x"
-#   %methodurlnoprot  Method and URL with format: "GET /index.html"
-#   %method           Method with format: GET
-#   %url              URL only with format: /index.html
-#   %query            Query string (used by URLWithQuery option)
-#   %code             Return code status (with format for web log: 999)
-#   %bytesd           Size of document in bytes
-#   %refererquot      Referer page with format: "http://from.com/from.htm"
-#   %referer          Referer page with format: http://from.com/from.htm
-#   %uaquot           User agent with format: "Mozilla/4.0 (compatible, ...)"
-#   %ua               User agent with format: Mozilla/4.0_(compatible...)
-#   %gzipin           mod_gzip compression input bytes: In:XXX
-#   %gzipout          mod_gzip compression output bytes & ratio: Out:YYY:ZZZpct.
-#   %gzipratio        mod_gzip or mod_deflate compression ratio: ZZZpct.
-#   %email            EMail sender (for mail log)
-#   %email_r          EMail receiver (for mail log)
-#   %syslog           Syslog-specific time and host stamp with format: Mon dd hh:mm:ss hostname
-#   %virtualname      Web sever virtual hostname. Use this tag when same log
-#                     file contains data of several virtual web servers. The
-#                     SiteDomain will be used to filter the one you want.
-#   If your log format has some fields not included in this list, use
-#   %other            Means another field not used
-#
-# Examples for Apache combined logs (following two examples are equivalent):
-# LogFormat = 1
-# LogFormat = "%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot"
-#
-# Examples for IIS (following two examples are equivalent):
-# LogFormat = 2
-# LogFormat = "%time2 %host %logname %method %url %code %bytesd %other %ua %referer"
-#
-LogFormat=1
-
-
-# If your log field's separator is not a space, you can change this parameter.
-# This parameter is not used if LogFormat is a predefined value (1,2,3,4,5,6)
-# Example: " "
-# Example: "\t"
-# Example: "|"
-# Default: " "
-#
-LogSeparator=" "
-
-
-# "SiteDomain" must contain the main domain name, or the main intranet web
-# server name, used to reach the web site.
-# If you share the same log file for several virtual web servers, this
-# parameter is used to tell AWStats to filter record that contains records for
-# this virtual host name only (So check that this virtual hostname can be
-# found in your log file and use a personalized log format that include the
-# %virtualname tag).
-# But for multi hosting a better solution is to have one log file for each
-# virtual web server. In this case, this parameter is only used to generate
-# full URL's links when ShowLinksOnUrl option is set to 1.
-# If analysing mail log, enter here the domain name of mail server.
-# Example: "myintranetserver"
-# Example: "www.domain.com"
-# Example: "ftp.domain.com"
-# Example: "domain.com"
-#
-SiteDomain="www.publicwhip.org.uk"
-
-
-# Enter here all other possible domain names, addresses or virtual host
-# aliases someone can use to access your site. Try to keep only the minimum
-# number of possible names/addresses to have the best performances.
-# You can repeat the "SiteDomain" value in this list.
-# Use space between each value.
-# This parameter is used to analyze referer field in log file and to help
-# AWStats to know if a referer URL is a local URL of same site or an URL of
-# another site.
-# Example: "www.myserver.com x.y.z.w localhost 127.0.0.1"
-#
-HostAliases="localhost 127.0.0.1"
-
-
-# If you want to have hosts reported by name instead of ip address, AWStats
-# need to make reverse DNS lookups (if not already done in your log file).
-# With DNSLookup to 0, all hosts will be reported by their IP addresses and
-# not by the full hostname of visitors.
-# If you want to set DNSLookup to 1, don't forget that this will reduce
-# dramatically AWStats update process speed. Do not use on large web sites.
-# Note: Reverse DNS lookup is done on IPv4 only.
-# Note: Country detection can work without reverse DNS lookup if plugin
-# 'geoipfree' or 'geoip is enabled (faster and more accurate than reverse DNS
-# lookup).
-# Possible values:
-# 0 - No DNS Lookup
-# 1 - DNS Lookup is fully enabled
-# 2 - DNS Lookup is made only from static DNS cache file (if exists)
-# Default: 2
-# 
-DNSLookup=1
-
-
-# When AWStats updates its statistics, it stores results of its analysis in 
-# files (AWStats database). All those files are written in the directory
-# defined by the "DirData" parameter. Set this value to the directory where
-# you want AWStats to save its database and working files into.
-# Warning: If you want to be able to use the "AllowToUpdateStatsFromBrowser"
-# feature (see later), you need "Write" permissions by web server user on this
-# directory (and "Modify" for Windows NTFS file systems).
-# Example: "/var/cache/awstats"
-# Example: "../data"
-# Example: "C:/awstats_data_dir"
-# Default: "."          (means same directory as awstats.pl)
-#
-#DirData="/var/cache/awstats"
-DirData="/home/francis/webstats/publicwhip/awstats"
-
-# Relative or absolute web URL of your awstats.pl directory.
-# This parameter is used only when AWStats is run from command line
-# with -output option (to generate links in HTML reported page).
-# Default: "/cgi-bin"   (means awstats.pl is in "/mywwwroot/cgi-bin")
-#
-DirCgi="/cgi-bin"
-
-
-# If AWStats used as a CGI, enter relative or absolute web URL of all icons
-# subdirectories.
-# If you build static reports ("... -output > outputpath/output.html"), enter
-# path of icon directory relative to the directory outputpath.
-# Example: "/icon"
-# Example: "../icon"
-# Default: "/icon" (means you must copy icon directories in "/mywwwroot/icon")
-#
-DirIcons="/awstats-icons"
-
-
-# When this parameter is set to 1, AWStats add a button on report page to
-# allow to "update" statistics from a web browser. Warning, when "update" is
-# made from a browser, AWStats is ran as a CGI by the web server user defined
-# in your web server (user "nobody" by default with Apache, "IUSR_XXX" with
-# IIS), so the "DirData" directory and all already existing history files
-# awstatsMMYYYY[.xxx].txt must be writable by this user. Change permissions if
-# necessary to "Read/Write" (and "Modify" for Windows NTFS file systems).
-# Warning: Update process can be long so you might experience "time out"
-# browser errors if you don't launch AWStats enough frequently.
-# When set to 0, update is only made when AWStats is ran from the command
-# line interface (or a task scheduler).
-# Possible values: 0 or 1
-# Default: 0
-#
-AllowToUpdateStatsFromBrowser=0
-
-
-
-#-----------------------------------------------------------------------------
-# OPTIONAL SETUP SECTION (Not required but increase AWStats features)
-#-----------------------------------------------------------------------------
-
-# When the update process run, AWStats can set a lock file in TEMP or TMP
-# directory. This lock is to avoid to have 2 update processes running at the
-# same time to prevent unknown conflicts problems and avoid DoS attacks when
-# AllowToUpdateStatsFromBrowser is set to 1.
-# Because, when you use lock file, you can experience sometimes problems in
-# lock file not correctly removed (when process is killed for example, this
-# requires that you remove the file manualy), this option is not enabled by
-# default (Do not enable this option with no consol server access).
-# Change : Effective immediatly
-# Possible values: 0 or 1
-# Default: 0
-#
-EnableLockForUpdate=0
-
-
-# AWStats can do reverse DNS lookups through a static DNS cache file that was
-# previously created manually. If no path is given in static DNS cache file
-# name, AWStats will search DirData directory. This file is never changed.
-# This option is not used if DNSLookup=0.
-# Note: DNS cache file format is 'minsince1970 ipaddress resolved_hostname'
-# or just 'ipaddress resolved_hostname'
-# Change : Effective for new updates only
-# Example: "/mydnscachedir/dnscache"
-# Default: "dnscache.txt"
-#
-DNSStaticCacheFile="dnscache.txt"
-
-
-# AWStats can do reverse DNS lookups through a DNS cache file that was created
-# by a previous run of AWStats. This file is erased and recreated after each
-# statistics update process. You don't need to create and/or edit it.
-# AWStats will read and save this file in DirData directory.
-# This option is used only if DNSLookup=1.
-# Note: If a DNSStaticCacheFile is available, AWStats will check for DNS
-# lookup in DNSLastUpdateCacheFile after checking into DNSStaticCacheFile.
-# Change : Effective for new updates only
-# Example: "/mydnscachedir/dnscachelastupdate"
-# Default: "dnscachelastupdate.txt"
-#
-DNSLastUpdateCacheFile="dnscachelastupdate.txt"
-
-
-# You can specify specific IP addresses that should NOT be looked up in
-#  the DNS. You may specify partial addresses (ie 163.85. for everything
-#  behind the usual firewall setup, etc)...
-# This option is used only if DNSLookup=1.
-# Note: Use space between each value.
-# Change : Effective for new updates only
-# Example: "163.85. 201.101.51.2"
-# Default: ""
-#
-SkipDNSLookupFor=""
-
-
-# The following two parameters allow you to protect a config file from being
-# read by AWStats when called from a browser if web user has not been
-# authenticated. Your AWStats program must be in a web protected "realm" (With
-# Apache, you can use .htaccess files to do so. With other web servers, see
-# your server setup manual).
-# Change : Effective immediatly
-# Possible values: 0 or 1
-# Default: 0
-#
-AllowAccessFromWebToAuthenticatedUsersOnly=0
-
-
-# This parameter give the list of all authorized authenticated users to view
-# statistics for this domain/config file. This parameter is used only if
-# AllowAccessFromWebToAuthenticatedUsersOnly is set to 1.
-# Change : Effective immediatly
-# Example: "user1 user2"
-# Example: "__REMOTE_USER__"
-# Default: ""
-#
-AllowAccessFromWebToFollowingAuthenticatedUsers=""
-
-
-# When this parameter is define to something, the IP address of the user that
-# read its statistics from a browser (when AWStats is used as a CGI) is
-# checked and must match the IP address range defined by this parameter.
-# Change : Effective immediatly
-# Example: "123.123.123.10-123.123.123.255"
-# Default: ""
-#
-AllowAccessFromWebToFollowingIPAddresses=""
-
-
-# If the "DirData" directory (see above) does not exists, AWStats return an
-# error. However, you can ask AWStats to create it.
-# This option can be used by some Web Hosting Providers that has defined a 
-# dynamic value for DirData (for example DirData="/home/__REMOTE_USER__") and
-# don't want to have to create a new directory each time they add a new user.
-# Change : Effective immediatly
-# Possible values: 0 or 1
-# Default: 0
-#
-CreateDirDataIfNotExists=0
-
-
-# In most case, AWStats is used as a cgi program. So AWStats process is ran
-# by default web server user (nobody for Unix, IUSR_xxx for IIS/Windows,...).
-# To make use easier and avoid permission's problems between update process 
-# (run by an admin user) and CGI process (ran by a low level user), AWStats
-# save its database files with read and write for everyone.
-# If you have experience on managing security policies (Web Hosting Provider),
-# you should set this parameter to 0. AWStats will keep default process user
-# permissions on its files.
-# Change : Effective for new updates only
-# Possible values: 0 or 1
-# Default: 1
-#
-SaveDatabaseFilesWithPermissionsForEveryone=1
-
-
-# AWStats can purge log after processing it. By this way, the next time you
-# launch AWStats, log file will be smaller and processing time will be better.
-# IMPORTANT !!!
-# AWStats is able to detect new lines in log file, to process only them, so
-# you can launch AWStats as soon as you want, even with this parameter to 0.
-# With 0, no purge is made, so you must use a scheduled task or a web server
-# that make this purge frequently.
-# With 1, the purge of the log file is made each time AWStats update is ran.
-# This parameter doesn't work with IIS (This web server doesn't let its log
-# file to be purged).
-# Change : Effective for new updates only
-# Possible values: 0 or 1
-# Default: 0
-#
-PurgeLogFile=0
-
-
-# When PurgeLogFile is setup to 1, AWStats will clean your log file after
-# processing it. You can however keep an archive file (saved in "DirData") of
-# all processed log records by setting this to 1 (For example if you want to
-# use another log analyzer).
-# This parameter is not used if PurgeLogFile=0
-# Change : Effective for new updates only
-# Possible values: 0 or 1
-# Default: 0
-#
-ArchiveLogRecords=0
-
-
-# Each time you run the update process, AWStats overwrite the 'historic file'
-# for the month (awstatsMMYYYY[.*].txt) with the updated one.
-# When write errors occurs (IO, disk full,...), this historic file can be
-# corrupted and must be deleted. Because this file contains information of all
-# past processed log files, you will loose old stats if removed. So you can
-# ask AWStats to save last non corrupted file in a .bak file. This file is
-# stored in "DirData" directory with other 'historic files'.
-# Change : Effective for new updates only
-# Possible values: 0 or 1
-# Default: 0
-#
-KeepBackupOfHistoricFiles=0
-
-
-# Default index page name for your web server.
-# Change : Effective for new updates only
-# Example: "default.htm default.html"
-# Default: "index.html"
-#
-DefaultFile="index.html"
-
-
-# Do not include access from clients that match following criteria.
-# If your log file contains IP adresses in host field, you must enter here
-# matching IP adresses criteria.
-# If DNS lookup is already done in your log file, you must enter here hostname
-# criteria, else enter ip address criteria.
-# The opposite parameter of "SkipHosts" is "OnlyHosts".
-# Note: Use space between each value.
-# Note: ^xxx means hosts starting with xxx
-# Note: xxx$ means hosts ending with xxx
-# Change : Effective for new updates only
-# Example: "^127.0.0.1$ ^192.168. ^10.0."
-# Example: "localhost abcxyz"
-# Default: ""
-#
-SkipHosts=""
-
-
-# Do not include access from clients with a user agent that match following
-# criteria. If you want to exclude a robot, you should update the robots.pm
-# file instead of this parameter.
-# Note: Use space between each value. This parameter is not case sensitive.
-# Change : Effective for new updates only
-# Example: "konqueror"
-# Default: ""
-#
-SkipUserAgents=""
-
-
-# Use SkipFiles to ignore access to URLs that match one of following entries.
-# You can, with this option, add a list of not important frame pages (like 
-# menus, etc...) to exclude them from statistics.
-# For example, to ignore a whole directory tree, just add "directorytoignore",
-# to ignore "users" pages in your stats, you can add "/~".
-# The opposite parameter of "SkipFiles" is "OnlyFiles".
-# Note: This parameter is not case sensitive.
-# Note: Use space between each value and do not remove default values.
-# Note: ^xxx means url starting with xxx.
-# Note: xxx$ means url ending with xxx.
-# Change : Effective for new updates only
-# Example: "robots.txt$ favicon.ico$ badpage.html$ /~"
-# Default: "robots.txt$ favicon.ico$"
-#
-SkipFiles="robots.txt$ favicon.ico$"
-
-
-# Include in stats, only accesses from hosts that match one of following
-# entries. For example, if you want AWStats to filter access to keep only
-# stats for visits from particular hosts, you can add thoose hosts names
-# in this parameter.
-# If DNS lookup is already done in your log file, you must enter here hostname
-# criteria, else enter ip address criteria.
-# The opposite parameter of "OnlyHosts" is "SkipHosts".
-# Note: This parameter is not case sensitive.
-# Note: Use space between each value.
-# Note: ^xxx means hosts starting with xxx.
-# Note: xxx$ means hosts ending with xxx.
-# Change : Effective for new updates only
-# Example: "^123.123.123.123$ ^10.0."
-# Default: ""
-#
-OnlyHosts=""
-
-
-# Include in stats, only accesses to URLs that match one of following entries.
-# For example, if you want AWStats to filter access to keep only stats that
-# match a particular string, like a particular directory, you can add this
-# directory name in this parameter.
-# The opposite parameter of "OnlyFiles" is "SkipFiles".
-# Note: This parameter is not case sensitive.
-# Note: Use space between each value and do not remove default values
-# Note: ^xxx means url starting with xxx.
-# Note: xxx$ means url ending with xxx.
-# Change : Effective for new updates only
-# Example: "marketing_directory"
-# Default: ""
-#
-OnlyFiles=""
-
-
-# Add here a list of kind of url (file extension) that must be counted as
-# "Hit only" and not as a "Hit" and "Page/Download". You can set here all
-# images extensions as they are hit downloaded that must be counted but they
-# are not viewed pages. URLs with such extensions are not included in the TOP
-# Pages/URL report.
-# Note: If you want to exclude your own URLs from stats (No Pages and no Hits
-# reported), you should use SkipFiles parameter instead.
-# Change : Effective for new updates only
-# Example: ""
-# Example: "css js class gif jpg jpeg png bmp zip arj gz z wav mp3 wma mpg"
-# Default: "css js class gif jpg jpeg png bmp"
-#
-NotPageList="css js class gif jpg jpeg png bmp jar txt"
-
-
-# By default, AWStats considers that records found in web log file are
-# successful hits if HTTP code returned by server is a valid HTTP code (200
-# and 304). Any other code are reported in HTTP error chart.
-# However in some specific environment, with web server HTTP redirection,
-# you can choose to also accept other codes.
-# Change : Effective for new updates only
-# Example: "200 304 302 305"
-# Default: "200 304"
-#
-ValidHTTPCodes="200 304"
-
-
-# By default, AWStats considers that records found in mail log file are
-# successful mail transfers if field that represent return code in analyzed
-# log file match values defined by this parameter.
-# Change : Effective for new updates only
-# Example: "1"
-# Example: "0"
-# Default: "1"
-#
-ValidSMTPCodes="1"
-
-
-# Some web servers on some Operating systems (IIS-Windows) considers that a
-# login with same value but different case are the same login. To tell AWStats
-# to also considers them as one, set this parameter to 1.
-# Change : Effective for new updates only
-# Possible values: 0 or 1
-# Default: 0
-# 
-AuthenticatedUsersNotCaseSensitive=0
-
-
-# Some web servers on some Operating systems (IIS-Windows) considers that two
-# URLs with same value but different case are the same URL. To tell AWStats to
-# also considers them as one, set this parameter to 1.
-# Change : Effective for new updates only
-# Possible values: 0 or 1
-# Default: 0
-# 
-URLNotCaseSensitive=0
-
-
-# Keep or remove the anchor string you can find in some URLs.
-# Change : Effective for new updates only
-# Possible values: 0 or 1
-# Default: 0
-#
-URLWithAnchor=0
-
-
-# In URL links, "?" char is used to add parameter's list in URLs. Syntax is:
-# /mypage.html?param1=value1
-# However, some servers/sites have also others chars to isolate dynamic part of
-# their URLs. You can complete this list with all such characters.
-# Change : Effective for new updates only
-# Example: "?;,"
-# Default: "?;"
-#
-URLQuerySeparators="?;"
-
-
-# Keep or remove the query string to the URL in the statistics for individual
-# pages. This is primarily used to differentiate between the URLs of dynamic
-# pages. If set to 1, mypage.html?id=x and mypage.html?id=y are counted as two
-# different pages.
-# Warning, when set to 1, memory required to run AWStats is dramatically
-# increased if you have a lot of changing URLs (for example URLs with a random
-# id inside). Such web sites should not set this option to 1 or use seriously
-# the next parameter URLWithQueryWithoutFollowingParameters.
-# Change : Effective for new updates only
-# Possible values:
-# 0 - URLs are cleaned from the query string (ie: "/mypage.html")
-# 1 - Full URL with query string is used     (ie: "/mypage.html?p=x&q=y")
-# Default: 0
-# 
-URLWithQuery=1
-
-
-# When URLWithQuery is on, you will get the full URL with all parameters in
-# URL reports. But among thoose parameters, sometimes you don't need a
-# particular parameter because it does not identify the page or because it's
-# a random ID changing for each access even if URL points to same page. In
-# such cases, it is higly recommanded to ask AWStats to remove such parameters
-# from the URL before counting, manipulating and storing it. Enter here list
-# of all non wanted parameters. For example if you enter "id", one hit on
-# /mypage.cgi?p=abc&id=Yo4UomP9d and /mypage.cgi?p=abc&id=Mu8fdxl3r
-# will be reported as 2 hits on /mypage.cgi?p=abc
-# This parameter is not used when URLWithQuery is 0.
-# Change : Effective for new updates only
-# Example: "PHPSESSID jsessionid"
-# Default: ""
-# 
-URLWithQueryWithoutFollowingParameters=""
-
-
-# Keep or remove the query string to the referrer URL in the statistics for
-# external referrer pages. This is used to differentiate between the URLs of
-# dynamic referrer pages. If set to 1, mypage.html?id=x and mypage.html?id=y
-# are counted as two different referrer pages.
-# Change : Effective for new updates only
-# Possible values:
-# 0 - Referrer URLs are cleaned from the query string (ie: "/mypage.html")
-# 1 - Full URL with query string is used      (ie: "/mypage.html?p=x&q=y")
-# Default: 0
-# 
-URLReferrerWithQuery=0
-
-
-# AWStats can detect setup problems or show you important informations to have
-# a better use. Keep this to 1, except if AWStats says you can change it.
-# Change : Effective immediatly
-# Possible values: 0 or 1
-# Default: 1
-#
-WarningMessages=1
-
-
-# When an error occurs, AWStats output a message related to errors. If you
-# want (in most cases for security reasons) to have no error messages, you
-# can set this parameter to your personalized generic message.
-# Change : Effective immediatly
-# Example: "An error occured. Contact your Administrator"
-# Default: ""
-#
-ErrorMessages=""
-
-
-# AWStat can be run with debug=x parameter to ouput various informations
-# to help in debugging or solving troubles. If you wand (in most cases for 
-# security reasons) to disable debugging, set this parameter to 0.
-# Change : Effective immediatly
-# Possible values: 0 or 1
-# Default: 1
-#
-DebugMessages=1
-
-
-# To help you to detect if your log format is good, AWStats report an error
-# if all the first NbOfLinesForCorruptedLog lines have a format that does not
-# match the LogFormat parameter.
-# However, some worm virus attack on your web server can result in a very high
-# number of corrupted lines in your log. So if you experience awstats stop
-# because of bad virus records at the beginning of your log file, you can
-# increase this parameter (very rare).
-# Change : Effective for new updates only
-# Default: 50
-#
-NbOfLinesForCorruptedLog=50
-
-
-# For some particular integration needs, you may want to have CGI links to
-# point to another script than awstats.pl.
-# Use the name of this script in WrapperScript parameter.
-# Change : Effective immediatly
-# Example: "awstatslauncher.pl"
-# Default: ""
-#
-WrapperScript=""
-
-
-# DecodeUA must be set to 1 if you use Roxen web server. This server converts
-# all spaces in user agent field into %20. This make the AWStats robots, os
-# and browsers detection fail in some cases. Just change it to 1 if and only
-# if your web server is Roxen.
-# Change : Effective for new updates only
-# Possible values: 0 or 1
-# Default: 0
-#
-DecodeUA=0
-
-
-# LogScreenSizeUrl can be used to make screen size report working.
-# You must define a string value for LogScreenSizeUrl parameter. Then add
-# the following HTML code at the end of your index page (before </BODY>)
-# replacing value of 'logscreensizeurl' with string choosed:
-#   <script language="javascript">
-#   var awstatsscreenlogger="logscreensizeurl";
-#   document.write('<img src="/'+awstatsscreenlogger+'?w='+screen.width+'&h='+screen.height+'" border=0 width=0 height=0>');
-#   </script>
-# If code is not added in index page, or if value of "logscreensizeurl" in
-# this code does not match value defined in LogScreenSizeUrl config parameter,
-# screen size detection won't work.
-# You must also change ShowScreenSizeStats parameter to 1 to report results.
-# Change : Effective for new updates only.
-# Possible value: Any string but same than the one used in added HTML code.
-# Default: "logscreensizeurl"
-#
-LogScreenSizeUrl="logscreensizeurl"
-
-
-
-#-----------------------------------------------------------------------------
-# OPTIONAL ACCURACY SETUP SECTION (Not required but increase AWStats features)
-#-----------------------------------------------------------------------------
-
-# Following values allows you to define accuracy of AWStats entities (robots,
-# browsers, os, referers) detection.
-# Large web sites or overloaded ISP with a lot of hosts should set this
-# parameter to 1 or 0, instead of 2, to gain speed and memory.
-# Possible values:
-#  0 = No detection,
-#  1 = Medium/Standard detection
-#  2 = Full detection
-# Change : Effective for new updates only
-# Default: 2
-#
-LevelForRobotsDetection=2			# 0 will increase AWStats speed by 1%.
-LevelForBrowsersDetection=2			# 0 disables Browsers detection. No speed gain.
-LevelForOSDetection=2				# 0 disables OS detection. No speed gain.
-LevelForRefererAnalyze=2			# 0 will increase AWStats speed by 5%.
-
-
-
-#-----------------------------------------------------------------------------
-# OPTIONAL APPEARANCE SETUP SECTION (Not required but increase AWStats features)
-#-----------------------------------------------------------------------------
-
-# When you use AWStats as a CGI, you can have the reports shown in HTML frames.
-# Frames are only available for report viewed dynamically. When you build
-# pages from command line, this option is not used and no frames are built.
-# Possible values: 0 or 1
-# Default: 0
-#
-UseFramesWhenCGI=1
-
-
-# This parameter ask your browser to open detailed reports into a different
-# window than the main page.
-# Possible values:
-# 0 - Open all in same browser window
-# 1 - Open detailed reports in another window except if using frames
-# 2 - Open always in a different window even if reports are framed
-# Default: 1
-#
-DetailedReportsOnNewWindows=1
-
-
-# You can add in the HTML report page a delay to force browsers to not use cache
-# if page is loaded a second time after this delay (in seconds).
-# This parameter is not used when report are built with -staticlinks option.
-# Example: 3600
-# Default: 0
-#
-Expires=0
-
-
-# To avoid too large web pages, you can ask AWStats to limit number of rows of
-# all reported charts to this number when no other limit apply.
-# Default: 1000
-#
-MaxRowsInHTMLOutput=1000
-
-
-# Set your primary language.
-# Possible value:
-#  Albanian=al, Bosnian=ba, Bulgarian=bg,
-#  Chinese (Taiwan)=tw, Chinese (Simpliefied)=cn, Czech=cz,
-#  Danish=dk, Dutch=nl, English=en, Estonian=et, Finnish=fi, French=fr,
-#  German=de, Greek=gr, Hungarian=hu, Icelandic=is, Indonesian=id, Italian=it,
-#  Japanese=jp, Korean=kr, Latvian=lv, Norwegian (Nynorsk)=nn,
-#  Norwegian (Bokmal)=nb, Polish=pl, Portuguese=pt, Portuguese (Brazilian)=br,
-#  Romanian=ro, Russian=ru, Serbian=sr, Slovak=sk, Spanish=es,
-#  Spanish (Catalan)=es_cat, Swedish=se, Turkish=tr, Ukrainian=ua, Welsh=wlk.
-#  First available language accepted by browser=auto
-# Default: "auto"
-#
-Lang="auto"
-
-
-# Set the location of language files.
-# Example: "/usr/share/awstats/lang"
-# Default: "./lang" (means lang directory is in same location than awstats.pl)
-#
-DirLang="/usr/share/awstats/lang"
-
-
-# You choose here which reports you want to see in the main page and what you
-# want to see in those reports.
-# Possible values:
-#  0  - Topic is not shown at all
-#  1  - Report is shown with default informations
-# XYZ - Report is shown with only informations defined by code X,Y,Z...
-#       X,Y,Z... are code letters among the following:
-#        U = Unique visitors
-#        V = Visits
-#        P = Number of pages
-#        H = Number of hits
-#        B = Bandwith
-#        L = Last access date
-#        E = Entry pages
-#        X = Exit pages
-#        C = Web compression (mod_gzip,mod_deflate)
-#        M = Average mail size (mail logs)
-#
-# Show menu header with report links
-# Default: 1, Possible codes: None
-ShowMenu=1					
-# Show monthly chart
-# Default: UVPHB, Possible codes: UVPHB
-ShowMonthStats=UVPHB
-# Show days of month chart
-# Default: VPHB, Possible codes: VPHB
-ShowDaysOfMonthStats=VPHB
-# Show days of week chart
-# Default: PHB, Possible codes: PHB
-ShowDaysOfWeekStats=PHB
-# Show hourly chart
-# Default: PHB, Possible codes: PHB
-ShowHoursStats=PHB
-# Show domains/country chart
-# Default: PHB, Possible codes: PHB
-ShowDomainsStats=PHB
-# Show hosts chart
-# Default: PHBL, Possible codes: PHBL
-ShowHostsStats=PHBL
-# Show authenticated users chart
-# Default: 0, Possible codes: PHBL
-ShowAuthenticatedUsers=0
-# Show robots chart
-# Default: HBL, Possible codes: HBL
-ShowRobotsStats=HBL
-# Show email senders chart (For use when analyzing mail log files)
-# Default: 0, Possible codes: HBML
-ShowEMailSenders=0
-# Show email receivers chart (For use when analyzing mail log files)
-# Default: 0, Possible codes: HBML
-ShowEMailReceivers=0
-# Show session chart
-# Default: 1, Possible codes: None
-ShowSessionsStats=1
-# Show pages-url chart.
-# Default: PBEX, Possible codes: PBEX
-ShowPagesStats=PBEX
-# Show file types chart.
-# Default: HB, Possible codes: HBC
-ShowFileTypesStats=HB
-# Show file size chart (Not yet available)
-# Default: 1, Possible codes: None
-ShowFileSizesStats=0		
-# Show operating systems chart
-# Default: 1, Possible codes: None
-ShowOSStats=1
-# Show browsers chart
-# Default: 1, Possible codes: None
-ShowBrowsersStats=1
-# Show screen size chart
-# Default: 0 (See LogScreenSizeUrl if set to 1), Possible codes: None
-ShowScreenSizeStats=0
-# Show origin chart
-# Default: PH, Possible codes: PH
-ShowOriginStats=PH
-# Show keyphrases chart
-# Default: 1, Possible codes: None
-ShowKeyphrasesStats=1
-# Show keywords chart
-# Default: 1, Possible codes: None
-ShowKeywordsStats=1
-# Show http errors chart
-# Default: 1, Possible codes: None
-ShowHTTPErrorsStats=1
-
-
-# Some graphical reports are followed by the data array of values.
-# If you don't want this array (to reduce report size for example), you can
-# set thoose options to 0.
-# Possible values: 0 or 1
-# Default: 1
-#
-# Data array values for the ShowMonthStats report
-AddDataArrayMonthStats=1
-# Data array values for the ShowDaysOfMonthStats report
-AddDataArrayShowDaysOfMonthStats=1
-# Data array values for the ShowDaysOfWeekStats report
-AddDataArrayShowDaysOfWeekStats=1
-# Data array values for the ShowHoursStats report
-AddDataArrayShowHoursStats=1
-
-
-# Following parameter can be used to choose maximum number of lines shown for
-# each particular reporting.
-#
-# Stats by domains
-MaxNbOfDomain = 10
-MinHitDomain  = 1
-# Stats by hosts
-MaxNbOfHostsShown = 10
-MinHitHost    = 1
-# Stats by authenticated users
-MaxNbOfLoginShown = 10
-MinHitLogin   = 1
-# Stats by robots
-MaxNbOfRobotShown = 10
-MinHitRobot   = 1
-# Stats by pages
-MaxNbOfPageShown = 10
-MinHitFile    = 1
-# Stats by screen size
-MaxNbOfScreenSizesShown = 5
-MinHitScreenSize = 1
-# Stats by referers
-MaxNbOfRefererShown = 10
-MinHitRefer   = 1
-# Stats for keyphrases
-MaxNbOfKeyphrasesShown = 10
-MinHitKeyphrase = 1
-# Stats for keywords
-MaxNbOfKeywordsShown = 10
-MinHitKeyword = 1
-# Stats for emails
-MaxNbOfEMailsShown = 20
-MinHitEMail   = 1
-
-
-# Choose if you want the week report to start on sunday or monday
-# Possible values:
-# 0 - Week start on sunday
-# 1 - Week start on monday
-# Default: 1
-#
-FirstDayOfWeek=1
-
-
-# List of visible flags with link to other language translations.
-# See Lang parameter for list of allowed flag/language codes.
-# If you don't want any flag link, set ShowFlagLinks to "".
-# This parameter is used only if ShowHeader parameter is set to 1.
-# Possible values: "" or "language_codes_separated_by_space"
-# Default: "en es fr nl es"
-#
-ShowFlagLinks="en fr de nl es"
-
-
-# Each URL, shown in stats report views, are links you can click.
-# Possible values: 0 or 1
-# Default: 1
-#
-ShowLinksOnUrl=1
-
-
-# When AWStats build HTML links in its report pages, it starts thoose link
-# with "http://". However some links might be HTTPS links, so you can enter
-# here the root of all your HTTPS links. If all your site is a SSL web site,
-# just enter "/".
-# This parameter is not used is ShowLinksOnUrl is 0.
-# Example: "/shopping"
-# Example: "/"
-# Default: ""
-#
-UseHTTPSLinkForUrl=""
-
-
-# Maximum length of URL shown on stats page (number of characters). This
-# affects only URL visible text, link still work.
-# Default: 72
-#
-MaxLengthOfURL=72
-
-
-# AWStats can include a link to WhoIs database on hostnames/ip. For this, you
-# must set ShowLinksToWhoIs to 1. Warning, a such feature depends on two next
-# parameters (LinksToWhoIs and LinksToIPWhoIs) and on WhoIs server
-# exhaustivity and availability.
-# For this reason, this feature can't be a reliable feature.
-# Possible values: 0 or 1
-# Default: 0
-#
-ShowLinksToWhoIs=1
-
-
-# Set here the link used to point to Internet WhoIs database for hostnames.
-# This parameter is not used if ShowLinksToWhoIs is 0.
-# Default: "http://www.whois.net/search.cgi2?str="
-# Example: "http://www.ripe.net/perl/whois?form_type=simple&searchtext="
-# Example: "http://ws.arin.net/cgi-bin/whois.pl?queryinput="
-#
-LinksToWhoIs="http://www.whois.net/search.cgi2?str="
-
-
-# Set here the link used to point to Internet WhoIs database for ip addresses.
-# This parameter is not used if ShowLinksToWhoIs is 0.
-# Default: "http://ws.arin.net/cgi-bin/whois.pl?queryinput="
-# Example: "http://ws.arin.net/cgi-bin/whois.pl?queryinput="
-#
-LinksToIPWhoIs="http://ws.arin.net/cgi-bin/whois.pl?queryinput="
-
-
-# You can enter HTML code that will be added at the top of AWStats reports.
-# Default: ""
-#
-HTMLHeadSection=""
-
-
-# You can enter HTML code that will be added at the end of AWStats reports.
-# Great to add advert ban.
-# Default: ""
-#
-HTMLEndSection=""
-
-
-# You can set Logo and LogoLink to use your own logo.
-# Logo must be the name of image file (must be in $DirIcons/other directory).
-# LogoLink is the expected URL when clicking on Logo.
-# Default: "awstats_logo1.png"
-#
-Logo="awstats_logo1.png"
-LogoLink="http://awstats.sourceforge.net"
-
-
-# Value of maximum bar width/height for horizontal/vertical graphics bar
-# Default: 260/90
-#
-BarWidth   = 260
-BarHeight  = 90
-
-
-# You can ask AWStats to use a particular CSS (Cascading Style Sheet) to
-# change its look.
-# Example: "/css/awstats.css"
-# Default: ""
-#
-StyleSheet=""
-
-
-# Those colors parameters can be used (if StyleSheet parameter is not used)
-# to change AWStats look.
-# Example: color_name="RRGGBB"	# RRGGBB is Red Green Blue components in Hex
-#
-color_Background="FFFFFF"		# Background color for main page (Default = "FFFFFF")
-color_TableBGTitle="CCCCDD"		# Background color for table title (Default = "CCCCDD")
-color_TableTitle="000000"		# Table title font color (Default = "000000")
-color_TableBG="CCCCDD"			# Background color for table (Default = "CCCCDD")
-color_TableRowTitle="FFFFFF"	# Table row title font color (Default = "FFFFFF")
-color_TableBGRowTitle="ECECEC"	# Background color for row title (Default = "ECECEC")
-color_TableBorder="ECECEC"		# Table border color (Default = "ECECEC")
-color_text="000000"				# Color of text (Default = "000000")
-color_textpercent="606060"		# Color of text for percent values (Default = "606060")
-color_titletext="000000"		# Color of text title within colored Title Rows (Default = "000000")
-color_weekend="EAEAEA"			# Color for week-end days (Default = "EAEAEA")
-color_link="0011BB"				# Color of HTML links (Default = "0011BB")
-color_hover="605040"			# Color of HTML on-mouseover links (Default = "605040") 
-color_u="FFB055"				# Background color for number of unique visitors (Default = "FFB055")
-color_v="F8E880"				# Background color for number of visites (Default = "F8E880")
-color_p="4477DD"				# Background color for number of pages (Default = "4477DD")
-color_h="66F0FF"				# Background color for number of hits (Default = "66F0FF")
-color_k="2EA495"				# Background color for number of bytes (Default = "2EA495")
-color_s="8888DD"				# Background color for number of search (Default = "8888DD")
-color_e="CEC2E8"				# Background color for number of entry pages (Default = "CEC2E8")
-color_x="C1B2E2"				# Background color for number of exit pages (Default = "C1B2E2")
-
-
-
-#-----------------------------------------------------------------------------
-# PLUGINS
-#-----------------------------------------------------------------------------
-
-# Add here all plugins file you want to load.
-# Plugin files must be .pm files stored in 'plugins' directory.
-# Uncomment LoadPlugin lines to enable a plugin after checking that perl
-# modules required by the plugin are installed.
-
-# Plugin: Tooltips
-# Perl modules required: None
-# Add some tooltips help on HTML report pages.
-# Note that enabled this kind of help will increased HTML report pages size,
-# so server load and bandwidth.
-#
-LoadPlugin="tooltips"
-
-# Plugin: IPv6
-# Perl modules required: Net::IP and Net::DNS
-# This plugin gives AWStats capability to make reverse DNS lookup on IPv6
-# addresses.
-# Note: If you are interesting in having country report, you should use the
-# geoipfree (or geoip) plugin instead of enabled reverse DNS lookup.
-#
-#LoadPlugin="ipv6"
-
-# Plugin: GeoIPfree
-# Perl modules required: Geo::IPfree version 0.2+ (from Graciliano M.P.)
-# Country chart is built from an Internet IP-Country database.
-# This plugin is useless for intranet only log files.
-# Note: You must choose between using this plugin (need Perl Geo::IPfree module)
-# or the GeoIP plugin (need Perl Geo::IP module from Maxmind).
-# This plugin reduces AWStats speed of 9% !
-#
-LoadPlugin="geoipfree"
-
-# Plugin: GeoIP
-# Perl modules required: Geo::IP or Geo::IP::PurePerl (from Maxmind)
-# Country chart is built from an Internet IP-Country database.
-# This plugin is useless for intranet only log files.
-# Note: You must choose between using this plugin (need Perl Geo::IP module 
-# from Maxmind) or the GeoIPfree plugin (need Perl Geo::IPfree module).
-# This plugin reduces AWStats speed of 9% !
-#
-#LoadPlugin="geoip"
-
-# Plugin: UrlAliases
-# Perl modules required: None
-# Add a text (Page title, description...) in URL reports after URL value.
-# A text file called urlalias.myconfig.txt, with two fields (first is URL,
-# second is text to show) separated by a tab char. must be created in plugins
-# directory.
-#
-#LoadPlugin="urlalias"
-
-# Plugin: UserInfo
-# Perl modules required: None
-# Add a text (Firtname, Lastname, Office Department, ...) in authenticated user
-# reports for each login value.
-# A text file called userinfo.myconfig.txt, with two fields (first is login,
-# second is text to show) separated by a tab char. must be created in plugins
-# directory.
-#
-LoadPlugin="userinfo"
-
-# Plugin: HashFiles
-# Perl modules required: Storable
-# AWStats DNS cache files are read/saved as native hash files. This increase
-# DNS cache files loading speed, above all for very large web sites.
-#
-LoadPlugin="hashfiles"
-
-# Plugin: TimeHiRes
-# Perl modules required: Time::HiRes
-# Time reported by -showsteps option is in millisecond. For debug purpose.
-#
-#LoadPlugin="timehires"		
-
-# Plugin: TimeZone
-# Perl modules required: Time::Local
-# Allow AWStats to correct a bad timezone for user of some IIS that use
-# GMT date in its log instead of local server time.
-# This module is useless for Apache and most IIS version.
-# This plugin reduces AWStats speed of 40% !!!!!!!
-#
-#LoadPlugin="timezone +2"
-
-# Plugin: Graph3D
-# Perl modules required: None
-# Supported charts are built by a nice 3D graphic applet.
-#
-#LoadPlugin="graph3d"		# !!! NOT YET AVAILABLE !!!
-
-
-
-#-----------------------------------------------------------------------------
-# EXTRA SECTION
-#-----------------------------------------------------------------------------
-
-# You can define your own charts, you choose here what are rows and columns
-# keys. This feature is particularly usefull for marketing purpose, tracking
-# products orders for example.
-# For this, edit all parameters of Extra section. Each set of parameter is a
-# different chart. For several charts, duplicate section changing the number.
-# Note: Each Extra section reduces AWStats speed by 8%.
-#
-# WARNING: A wrong setup of Extra section might result in too large arrays
-# that will consume all your memory, making AWStats unusable after several
-# updates, so be sure to setup it correctly.
-# In most cases, you don't need this feature.
-#
-# ExtraSectionNameX is title of your personalized chart.
-# ExtraSectionConditionalX are conditions on URL and/or QUERY_STRING and/or
-#   REFERER you can use to count or not the hit. Use "|" for "OR".
-# ExtraSectionFirstColumnTitleX is the first column title of the chart.
-# ExtraSectionFirstColumnValuesX is a Regex string to tell AWStats how to 
-#   extract the value used for first column. Each different value found will
-#   be a different row. Be sure that list of different values is "limited" to
-#   avoid "not enough memory" problems !
-# ExtraSectionStatTypesX are things you want to count. You can use standard
-#   code letters (P for pages,H for hits,B for bandwidth,L for last access).
-# MaxNbOfExtraX is maximum number of rows shown in chart.
-# MinHitExtraX is minimum number of hits required to be shown in chart.
-#
-
-# Example to report the 20 products the most ordered by "order.cgi" script
-#ExtraSectionName1="Product orders"
-#ExtraSectionCondition1="URL,/cgi-bin/order.cgi"
-#ExtraSectionFirstColumnTitle1="Product ID"
-#ExtraSectionFirstColumnValues1="QUERY_STRING,productid=([^&]+)"
-#ExtraSectionStatTypes1=PL
-#MaxNbOfExtra1=20
-#MinHitExtra1=1
-
-
-
-#-----------------------------------------------------------------------------
-# INCLUDES
-#-----------------------------------------------------------------------------
-
-# You can include other config files using the directive with the name of the
-# config file.
-# This is particularly usefull for users who have a lot of virtual servers, so
-# a lot of config files and want to maintain common values in only one file.
-# Note that when a variable is defined both in a config file and in an
-# included file, AWStats will use the last value read.
-#
-
-Include "/etc/awstats/awstats.conf.local"
-

build/clear-caches

 set -e
 #set -x # debugging
 
-DB=pw_live
+DB='publicwhip'
+echo $DB
 SITE=www.publicwhip.org.uk
 CACHE=pwcache-live
-if [ "$1" == "staging" ]
-then
-    DB=pw_staging
-    SITE=staging.publicwhip.org.uk
-    CACHE=pwcache-staging
-fi
-
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] clear-cache: Clearing file cache"
 # Clear file cache (front page only is cached like this -- 2005-11-28)
 rm -fr /home/publicwhip/$CACHE/*
 
 # Clear database table caches
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] clear-cache: Clearing database table caches: dreaminfo"
 echo "delete from pw_cache_dreaminfo;" | mysql $DB
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] clear-cache: Clearing database table caches: dreamreal_distance"
 echo "delete from pw_cache_dreamreal_distance;" | mysql $DB
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] clear-cache: Clearing database table caches: divwiki"
 echo "delete from pw_cache_divwiki;" | mysql $DB
 echo "update pw_cache_dreaminfo set cache_uptodate = 0; " | mysql $DB
-
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] clear-cache: Regenerating cached data"
 # Provoke regeneration of policy cached data, so that Policy Comparison
 # table on MP pages appears correctly.
-for i in `snarf -q http://$SITE/policies.php - | egrep 'href="policy.php' | sed 's/^.*\?\(id=[0-9][0-9]*\)".*/\1/'`
+for house in commons lords scotland
 do
-    for house in commons lords scotland
+    echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] clear-cache: Regenerating cached data for house ${house}"
+    for i in `snarf -q http://$SITE/policies.php - | egrep 'href="policy.php' | sed 's/^.*\?\(id=[0-9][0-9]*\)".*/\1/'`
     do
         snarf "http://${SITE}/dreamplot.php?${i}&display=reverse&size=large&house=${house}" - > /dev/null
     done
 done
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] clear-cache: Completed"

build/dailyupdate

 	exit $?
 fi
 
-FROMDATE=2001-06-01
+#FROMDATE=1999-11-17
+FROMDATE=2012-02-01
 PATH=$PATH:/usr/local/bin:/usr/bin:/bin
 export HOME=/home/publicwhip
 export CVS_RSH=/usr/bin/ssh
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] dailyupdate: Starting with from date of $FROMDATE"
 
 # Grab new scraped XML files from TheyWorkForYou
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] dailyupdate: Fetching divisions"
 rsync --recursive ukparse.kforge.net::parldata/scrapedxml/divisionsonly /home/publicwhip/parldata/scrapedxml
+
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] dailyupdate: Fetching lords"
 rsync --recursive ukparse.kforge.net::parldata/scrapedxml/lordsdivisionsonly /home/publicwhip/parldata/scrapedxml
+
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] dailyupdate: Fetching sp"
 rsync --recursive ukparse.kforge.net::parldata/scrapedxml/sp /home/publicwhip/parldata/scrapedxml
+
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] dailyupdate: Fetching sp-motions"
 rsync --recursive ukparse.kforge.net::parldata/scrapedxml/sp-motions /home/publicwhip/parldata/scrapedxml
+
 chmod -R a+r /home/publicwhip/parldata
 
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] dailyupdate: Updating parlimentary parser"
 # Update members files and the like
 cd ~/parlparse
 svn -q update
 cd ~/publicwhip-live/
 # cvs -Q update -dP
 
-echo "Loading list of members"
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] dailyupdate: Loading list of members"
 # Load new list of members and ministers
 cd ~/publicwhip-live/loader
 ./memxml2db.pl
 
-echo "Loading new divisions"
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] dailyupdate: Loading new divisions"
 # Load new divisions
-#if ! 
-./load.pl --quiet --from=$FROMDATE divsxml 
+#if !
+./load.pl --quiet --from=$FROMDATE divsxml
 #then
-	# echo "There were new divisions, recalculating cached values"
+	echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] dailyupdate: There were new divisions, recalculating cached values"
 
 	# There were new divisions, so run checks
 	./load.pl --quiet --from=$FROMDATE check
-echo "Calculating cache"
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] dailyupdate: Calculating cache"
 	# and update counts of rebels etc.
     ./calc_caches.php
 
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] dailyupdate: Clearing page cache"
 	# clear page cache
 	cd ~/publicwhip-live/build
 	./clear-caches
 
 	# export dumps of data
 	cd ~/publicwhip-live/build
-echo "Export db"
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] dailyupdate: Export db"
 	./export-db
-echo "Votematrix"
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] dailyupdate: Votematrix"
 	./votematrix.pl
 #fi
 
-echo "Copying across ministers"
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] dailyupdate: Copying across ministers"
 # Copy across ministers file (used by minwhirl)
 cp ~/parlparse/members/ministers.xml ~/www.publicwhip.org.uk/docs/data/
 cp ~/publicwhip-live/*.txt ~/www.publicwhip.org.uk/docs/data/
 chmod a+r ~/www.publicwhip.org.uk/docs/data/*
 
-echo "Starting rss feeds"
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] dailyupdate: Starting rss feeds"
 # RSS feeds
 RSSDEST=~/www.publicwhip.org.uk/docs/feeds
 cd ~/publicwhip-live/loader
-echo "Interesting divisions"
-./interestingdivisions2rss.pl > $RSSDEST/interestingdivisions.xml
-echo "All divisions"
-./alldivisions2rss.pl > $RSSDEST/alldivisions.xml
 
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] dailyupdate: Interesting divisions"
+./interestingdivisions2rss.pl > $RSSDEST/interestingdivisions.xml
 
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] dailyupdate: All divisions"
+./alldivisions2rss.pl > $RSSDEST/alldivisions.xml
 
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S) dailyupdate: Complete"
 
 # Exports database for others to download.
 
-MYSQL_DUMP=/usr/bin/mysqldump
-
+MYSQL_DUMP="/usr/bin/mysqldump publicwhip --add-drop-table --tables "
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] export-db: Main tables"
 # Main tables with data from Hansard
-$MYSQL_DUMP --add-drop-table pw_live pw_division pw_moffice pw_mp pw_vote | bzip2 - >~/www.publicwhip.org.uk/docs/data/pw_static_tables.sql.bz2
-
+$MYSQL_DUMP pw_division pw_moffice pw_mp pw_vote | bzip2 - >~/www.publicwhip.org.uk/docs/data/pw_static_tables.sql.bz2
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] export-db: User edited tables"
 # User edited tables
 # DO NOT include pw_dyn_user, pw_dyn_newsletter or pw_dyn_auditlog as they have emails etc. in them
-$MYSQL_DUMP --add-drop-table pw_live pw_dyn_dreammp pw_dyn_dreamvote pw_dyn_wiki_motion | bzip2 - >~/www.publicwhip.org.uk/docs/data/pw_dynamic_tables.sql.bz2
-
+$MYSQL_DUMP pw_dyn_dreammp pw_dyn_dreamvote pw_dyn_wiki_motion | bzip2 - >~/www.publicwhip.org.uk/docs/data/pw_dynamic_tables.sql.bz2
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] export-db: Cache tables"
 # Cache tables.
-$MYSQL_DUMP --add-drop-table pw_live pw_cache_attendrank_today pw_cache_divdiv_distance pw_cache_divinfo pw_cache_divwiki pw_cache_dreaminfo pw_cache_dreamreal_distance pw_cache_mpinfo pw_cache_mpinfo_session2002 pw_cache_partyinfo pw_cache_realreal_distance pw_cache_rebelrank_today pw_cache_whip | bzip2 - >~/www.publicwhip.org.uk/docs/data/pw_cache_tables.sql.bz2
-
+$MYSQL_DUMP pw_cache_attendrank_today pw_cache_divdiv_distance pw_cache_divinfo pw_cache_divwiki pw_cache_dreaminfo pw_cache_dreamreal_distance pw_cache_mpinfo pw_cache_mpinfo_session2002 pw_cache_partyinfo pw_cache_realreal_distance pw_cache_rebelrank_today pw_cache_whip | bzip2 - >~/www.publicwhip.org.uk/docs/data/pw_cache_tables.sql.bz2
+echo "[$(date +\%Y-\%m-\%d\ %H:%M:%S)] export-db: Completed"
 
 

build/source-release.txt

-7

build/source-to-sf

-#!/bin/bash
-set -e
-
-# Makes a source release of the Public Whip as a tarball and uploads to SourceForge FTP
-
-cd ../publicwhip # make sure in sensible place
-
-cvs -Q update -dP
-
-BUILD=`cat build/source-release.txt`
-let BUILD=BUILD+1
-echo $BUILD >build/source-release.txt
-WHO=`whoami`@`hostname`
-cvs commit -m "Autobuild $BUILD" build/source-release.txt
-
-cvs tag -r HEAD source-release-$BUILD .
-
-cd /tmp
-rm -fr publicwhip-sourceforge-temp
-mkdir publicwhip-sourceforge-temp
-cd publicwhip-sourceforge-temp
-
-export CVSROOT=:ext:frabcus@cvs.sourceforge.net:/cvsroot/publicwhip
-cvs export -r source-release-$BUILD publicwhip
-mv publicwhip publicwhip-source-$BUILD
-
-tar cvfz publicwhip-source-$BUILD.tar.gz publicwhip-source-$BUILD
-
-ftp-upload --passive --host upload.sourceforge.net --dir /incoming --user anonymous publicwhip-source-$BUILD.tar.gz
-
-echo publicwhip-source-$BUILD.tar.gz now in sourceforge.net incoming
-

build/votematrix.pl

     print METADATA "file created: " . scalar localtime() . "by http://www.publicwhip.org.uk/\n";
     # Get ids of MPs
     my $limit = "";
-
-    my $mp_query = PublicWhip::DB::query($dbh, "select pw_mp.mp_id, pw_mp.first_name, pw_mp.last_name, pw_mp.party from pw_mp, pw_cache_mpinfo where
+    printf "[%4d-%02d-%02d %02d:%02d:%02d] %s: %s\n",$year+1900,$mon+1,$mday,$hour,$min,$sec,"votematrix","Processing $outfile\n";
+    my $sql="select pw_mp.mp_id, pw_mp.first_name, pw_mp.last_name, pw_mp.party from pw_mp, pw_cache_mpinfo where
         pw_mp.mp_id = pw_cache_mpinfo.mp_id and $where 
-        order by pw_mp.last_name, pw_mp.first_name, pw_mp.constituency $limit");
+        order by pw_mp.last_name, pw_mp.first_name, pw_mp.constituency $limit";
+    my $mp_query = PublicWhip::DB::query($dbh, $sql);
     print METADATA $mp_query->rows . " mps\n";
     my @mp_ixs;
     my %mp_name;

loader/PublicWhip/.cvsignore

-Config.pm

loader/PublicWhip/Parliaments.pm

 use strict;
 
 our @list = ( # put newer Parliaments first
-    { id => '2010', from => '2010-05-18', to => '9999-12-31', name => '2010' },
+    { id => '2010', from => '2010-05-06', to => '9999-12-31', name => '2010' },
     { id => '2005', from => '2005-05-05', to => '2010-04-12', name => '2005' },
     { id => '2001', from => '2001-06-07', to => '2005-04-11', name => '2001' },
     { id => '1997', from => '1997-05-01', to => '2001-05-14', name => '1997' }

loader/calc_caches.php

 require_once "../website/db.inc";
 require_once "../website/parliaments.inc";
 
-$db = new DB();
-$db2 = new DB();
-
-current_rankings($db);
-count_party_stats($db, $db2);
-guess_whip_for_all($db, $db2);
-count_mp_info($db);
-count_div_info($db);
+print '['.date('Y-m-d H:i:s').'] calc_caches: current rankings'.PHP_EOL;
+current_rankings();
+print '['.date('Y-m-d H:i:s').'] calc_caches: counting party stats'.PHP_EOL;
+count_party_stats();
+print '['.date('Y-m-d H:i:s').'] calc_caches: guessing whip for all'.PHP_EOL;
+guess_whip_for_all();
+print '['.date('Y-m-d H:i:s').'] calc_caches: counting mp info'.PHP_EOL;
+count_mp_info();
+print '['.date('Y-m-d H:i:s').'] calc_caches: counting division info'.PHP_EOL;
+count_div_info();
 
 # then we loop through the missing entries and fill them in
-function count_party_stats($db, $db2)
+function count_party_stats()
 {
+    global $pwpdo;
+    global $pwpdo2;
     # TODO: redo this per parliament
-	$db->query("DROP TABLE IF EXISTS pw_cache_partyinfo");
-	$db->query("CREATE TABLE pw_cache_partyinfo (
+    $pwpdo->query("DROP TABLE IF EXISTS pw_cache_partyinfo",array());
+    $pwpdo->query("CREATE TABLE pw_cache_partyinfo (
 						party varchar(100) not null,
 						house enum('commons', 'lords', 'scotland') not null,
 						total_votes int not null
-                        )");
+                        )",array());
 
 	$query = "SELECT party, house, COUNT(vote) AS total_votes
 			  FROM pw_vote
 			  GROUP BY party, house";
 
     #print $query;
-	$db->query($query);
+	$rows=$pwpdo->fetch_all_rows($query,array());
 
-	while ($row = $db->fetch_row_assoc())
+	foreach ($rows as $row)
 	{
-        #print_r($row);
-		$qattrs = "party, house, total_votes";
-		$qvalues = "'".$row['party']."', '".$row['house']."', '".$row['total_votes']."'";
-		$db2->query("INSERT INTO pw_cache_partyinfo ($qattrs) VALUES ($qvalues)");
+		$pwpdo->query("INSERT INTO pw_cache_partyinfo (party, house, total_votes) VALUES (?,?,?)",array($row['party'],$row['house'],$row['total_votes']));
     };
 }
 
 
 # party whip calc everything
-function guess_whip_for_all($db, $db2)
+function guess_whip_for_all()
 {
+    global $pwpdo;
+    global $pwpdo2;
 	# this table runs parallel to the table of divisions
-	$db->query("DROP TABLE IF EXISTS pw_cache_whip_tmp");
-	$db->query("CREATE TABLE pw_cache_whip_tmp (
+    $pwpdo->query("DROP TABLE IF EXISTS pw_cache_whip_tmp",array());
+	$pwpdo->query("CREATE TABLE pw_cache_whip_tmp (
 					division_id int not null,
 					party varchar(200) not null,
 					aye_votes int not null,
 					possible_votes int not null,
 					whip_guess enum('aye', 'no', 'abstention', 'unknown', 'none') not null,
 					unique(division_id, party)
-			    )");
+			    )",array());
 
 	$qselect = "SELECT sum(pw_vote.vote = 'aye') AS ayevotes,
 					   sum(pw_vote.vote = 'tellaye') AS ayetells,
 	$query = $qselect.$qfrom.$qjoin.$qgroup;
 
     #print $query;
-	$db->query($query);
+    $pwpdo->query($query,array());
 
-	while ($row = $db->fetch_row_assoc())
+	while ($row = $pwpdo->fetch_row())
 	{
 		$party = $row['party'];
 		$ayevotes = intval($row['ayevotes']);
 		}
 
 		$qattrs = "division_id, party, aye_votes, aye_tells, no_votes, no_tells, both_votes, abstention_votes, possible_votes, whip_guess";
-		$qvalues = $row['division_id'].", '".$row['party']."', $ayevotes, $ayetells, $novotes, $notells, $boths, $abstentions, $possibles, '".$whip_guess."'";
-        $query = "INSERT INTO pw_cache_whip_tmp ($qattrs) VALUES ($qvalues)";
-        #print_r($row);
-        #print $query;
-		$db2->query($query);
+        $placeheld=array($row['division_id'],$row['party'],$ayevotes,$ayetells,$novotes,$notells,$boths,$abstentions,$possibles,$whip_guess);
+        $query='INSERT INTO pw_cache_whip_tmp ('.$qattrs.') VALUES (?'.str_repeat(',?',count($placeheld)-1).')';
+        $pwpdo2->query($query,$placeheld);
     }
 
-	$db->query("DROP TABLE IF EXISTS pw_cache_whip");
-	$db->query("RENAME TABLE pw_cache_whip_tmp TO pw_cache_whip");
+	$pwpdo->query("DROP TABLE IF EXISTS pw_cache_whip",array());
+	$pwpdo->query("RENAME TABLE pw_cache_whip_tmp TO pw_cache_whip",array());
 }
 
-function count_mp_info($db) {
-    count_4d_info($db, "pw_cache_mpinfo", "pw_mp.mp_id", "mp_id", "votes_attended", "votes_possible");
+function count_mp_info() {
+    count_4d_info("pw_cache_mpinfo", "pw_mp.mp_id", "mp_id", "votes_attended", "votes_possible");
 }
 
-function count_div_info($db) {
-    count_4d_info($db, "pw_cache_divinfo", "pw_division.division_id", "division_id", "turnout", "possible_turnout");
+function count_div_info() {
+    count_4d_info( "pw_cache_divinfo", "pw_division.division_id", "division_id", "turnout", "possible_turnout");
 }
 
-function count_4d_info($db, $table, $group_by, $id, $votes_attended, $votes_possible) {
+function count_4d_info( $table, $group_by, $id, $votes_attended, $votes_possible) {
     #print "Creating table $table\n";
-    $db->query( "DROP TABLE IF EXISTS ${table}_tmp" );
-    $db->query(
+    global $pwpdo;
+    $pwpdo->query( "DROP TABLE IF EXISTS ${table}_tmp",array() );
+    $pwpdo->query(
         "CREATE TABLE ${table}_tmp (
         $id int not null,
         rebellions int not null,
         $votes_possible int not null,
         aye_majority int not null,
         index($id)
-    )");
+    )",array());
     // majority is meaningless in the case of the mp_info -- just how many more ayes than noes in the lifetime of the MP
 
     $scottish_rebellion_condition = "( (pw_division.house = 'scotland') and
             pw_division.division_date < pw_mp.left_house
 
         LEFT JOIN pw_vote ON
-            pw_vote.mp_id = pw_mp.mp_id AND
-            pw_vote.division_id = pw_division.division_id
+            pw_vote.division_id = pw_division.division_id AND
+            pw_vote.mp_id = pw_mp.mp_id
 
         LEFT JOIN pw_cache_whip ON
-            pw_cache_whip.party = pw_mp.party AND
-            pw_cache_whip.division_id = pw_division.division_id
+            pw_cache_whip.division_id = pw_division.division_id AND
+            pw_cache_whip.party = pw_mp.party
 
         GROUP BY $group_by
     ";
 
     #print $query;
-    $db->query($query);
+    $pwpdo->query($query,array());
 
-	$db->query("DROP TABLE IF EXISTS $table");
-	$db->query("RENAME TABLE ${table}_tmp TO $table");
+	$pwpdo->query("DROP TABLE IF EXISTS $table",array());
+	$pwpdo->query("RENAME TABLE ${table}_tmp TO $table",array());
 }
 
-function current_rankings($db) {
+function current_rankings() {
     # Create tables to store in
-    $db->query("drop table if exists pw_cache_rebelrank_today" );
-    $db->query("create table pw_cache_rebelrank_today (
+    global $pwpdo;
+    $pwpdo->query("drop table if exists pw_cache_rebelrank_today",array() );
+    $pwpdo->query("create table pw_cache_rebelrank_today (
         mp_id int not null,
         rebel_rank int not null,
         rebel_outof int not null,
         index(mp_id)
         )"
-    );
-    $db->query("drop table if exists pw_cache_attendrank_today" );
-    $db->query("create table pw_cache_attendrank_today (
+        ,array());
+    $pwpdo->query("drop table if exists pw_cache_attendrank_today" ,array());
+    $pwpdo->query("create table pw_cache_attendrank_today (
             mp_id int not null,
             attend_rank int not null,
             attend_outof int not null,
             index(mp_id)
         )"
-    );
+        ,array());
 
-    do_house_ranking($db, "commons");
-    do_house_ranking($db, "scotland");
-    do_house_ranking($db, "lords");
+    do_house_ranking( "commons");
+    do_house_ranking("scotland");
+    do_house_ranking("lords");
 }
 
 function rebelcomp($a, $b) {
     return ($mpattend[$a] > $mpattend[$b]) ? -1 : 1;
 }
 
-function do_house_ranking($db, $house) {
+function do_house_ranking($house) {
+    global $pwpdo;
     # Select all MPs in force today, and their attendance/rebellions
     $mps_query_start = "select pw_mp.mp_id as mp_id, 
             round(100*rebellions/votes_attended,2) as rebellions,
             party
             from pw_mp, pw_cache_mpinfo 
             where pw_mp.mp_id = pw_cache_mpinfo.mp_id 
-                  and house = '$house' ";
-    $db->query($mps_query_start . "and entered_house <= curdate() and curdate() <= left_house");
-    if ($db->rows() == 0) {
-        $db->query($mps_query_start .  "and left_house = '2011-03-23'");
-        if ($db->rows() == 0) {
+                  and house = ? ";
+    $rows=$pwpdo->fetch_all_rows($mps_query_start . "and entered_house <= curdate() and curdate() <= left_house",array($house));
+    if (count($rows) == 0) {
+        $rows=$pwpdo->fetch_all_rows($mps_query_start .  "and left_house = '2011-03-23'",array($house));
+        if (count($rows) == 0) {
             die("No MPs/MSPs/Lords currently active have been found (house: '$house'), change General Election date in code if you are coming up to one");
             return;
         }
     $mprebel = array();
     $mpsattend = array();
     $mpattend = array();
-	while ($row = $db->fetch_row()) {
-        list( $mpid, $rebel, $attend, $party ) = $row;
+	foreach ($rows as $row) {
+        $mpid=$row['mp_id'];
+        $rebel=$row['rebellions'];
+        $attend=$row['attendance'];
+        $party=$row['party'];
         if ( $rebel ) {
             if (!whipless_party($party)) {
                 $mpsrebel[] = $mpid;
         if ( $mprebel[$mp] != $prevvalue )
             $activerank = $rank;
         $prevvalue = $mprebel{$mp};
-        $db->query("insert into pw_cache_rebelrank_today (mp_id, rebel_rank, rebel_outof)
-            values ($mp, $activerank, ".count($mpsrebel).")"
-        );
+        $pwpdo->query('INSERT INTO pw_cache_rebelrank_today (mp_id, rebel_rank, rebel_outof) VALUES (?,?,?)',array($mp,$activerank,count($mpsrebel)));
     }
 
     # Sort, and calculate ranking for attendance
         if ( $mpattend{$mp} != $prevvalue )
             $activerank = $rank;
         $prevvalue = $mpattend{$mp};
-        $db->query("insert into pw_cache_attendrank_today (mp_id, attend_rank, attend_outof)
-            values ($mp, $activerank, ".count($mpsattend).")"
-        );
+        $pwpdo->query('insert into pw_cache_attendrank_today (mp_id, attend_rank, attend_outof) values (?,?,?)',array($mp,$activerank,count($mpsattend)));
     }
 }
 
 Loads voting lists from XML files into MySQL database for the Public Whip
 website.  Peforms various consistency checks.
 
-scrape.pl [OPTION]... [COMMAND]...
+build.pl [OPTION]... [COMMAND]...
 
 Commands are any or all of these, in order you want them run:
 divsxml - parse divisions from XML files and add them to database

loader/memxml2db.pl

 use PublicWhip::Error;
 use PublicWhip::DB;
 my $dbh = PublicWhip::DB::connect();
+my $sec;
+my $min;
+my $hour;
+my $mday;
+my $mon;
+my $year;
+($sec,$min,$hour,$mday,$mon,$year)=localtime(time);
+printf "[%4d-%02d-%02d %02d:%02d:%02d] %s: %s\n",$year+1900,$mon+1,$mday,$hour,$min,$sec,"memxml2db","Mapping gid to internal mp_ids";
 
 # Map from gid to the pw_mp.mp_id internal Public Whip ids, so we reload
 # table with same new ids
     $last_mp_id = $mp_id if ($mp_id > $last_mp_id);
 }
 
+($sec,$min,$hour,$mday,$mon,$year)=localtime(time);
+printf "[%4d-%02d-%02d %02d:%02d:%02d] %s: %s\n",$year+1900,$mon+1,$mday,$hour,$min,$sec,"memxml2db","Deleting tables";
 # We completely rebuild these two tables
 $sth = PublicWhip::DB::query($dbh, "delete from pw_moffice");
 $sth = PublicWhip::DB::query($dbh, "delete from pw_constituency");
 
 my %membertoperson;
+($sec,$min,$hour,$mday,$mon,$year)=localtime(time);
+printf "[%4d-%02d-%02d %02d:%02d:%02d] %s: %s\n",$year+1900,$mon+1,$mday,$hour,$min,$sec,"memxml2db","Setting up twig";
 my $twig = XML::Twig->new(
     twig_handlers => { 
             'constituency' => \&loadcons, 
             'moffice' => \&loadmoffice 
         }, 
     output_filter => 'safe');
+($sec,$min,$hour,$mday,$mon,$year)=localtime(time);
+printf "[%4d-%02d-%02d %02d:%02d:%02d] %s: %s\n",$year+1900,$mon+1,$mday,$hour,$min,$sec,"memxml2db","Parsing constituencies.xml";
 $twig->parsefile("$members_location/constituencies.xml");
+($sec,$min,$hour,$mday,$mon,$year)=localtime(time);
+printf "[%4d-%02d-%02d %02d:%02d:%02d] %s: %s\n",$year+1900,$mon+1,$mday,$hour,$min,$sec,"memxml2db","Parsing sp-constituencies.xml";
 $twig->parsefile("$members_location/sp-constituencies.xml");
+($sec,$min,$hour,$mday,$mon,$year)=localtime(time);
+printf "[%4d-%02d-%02d %02d:%02d:%02d] %s: %s\n",$year+1900,$mon+1,$mday,$hour,$min,$sec,"memxml2db","Parsing people.xml";
 $twig->parsefile("$members_location/people.xml");
+($sec,$min,$hour,$mday,$mon,$year)=localtime(time);
+printf "[%4d-%02d-%02d %02d:%02d:%02d] %s: %s\n",$year+1900,$mon+1,$mday,$hour,$min,$sec,"memxml2db","Parsing ministers.xml";
 $twig->parsefile("$members_location/ministers.xml");
+($sec,$min,$hour,$mday,$mon,$year)=localtime(time);
+printf "[%4d-%02d-%02d %02d:%02d:%02d] %s: %s\n",$year+1900,$mon+1,$mday,$hour,$min,$sec,"memxml2db","Parsing all-members.xml";
 $twig->parsefile("$members_location/all-members.xml");
+($sec,$min,$hour,$mday,$mon,$year)=localtime(time);
+printf "[%4d-%02d-%02d %02d:%02d:%02d] %s: %s\n",$year+1900,$mon+1,$mday,$hour,$min,$sec,"memxml2db","Parsing all-members-2010.xml";
 $twig->parsefile("$members_location/all-members-2010.xml");
+($sec,$min,$hour,$mday,$mon,$year)=localtime(time);
+printf "[%4d-%02d-%02d %02d:%02d:%02d] %s: %s\n",$year+1900,$mon+1,$mday,$hour,$min,$sec,"memxml2db","Parsing peers-ucl.xml";
 $twig->parsefile("$members_location/peers-ucl.xml");
+($sec,$min,$hour,$mday,$mon,$year)=localtime(time);
+printf "[%4d-%02d-%02d %02d:%02d:%02d] %s: %s\n",$year+1900,$mon+1,$mday,$hour,$min,$sec,"memxml2db","Parsing sp-members.xml";
 $twig->parsefile("$members_location/sp-members.xml");
 
+($sec,$min,$hour,$mday,$mon,$year)=localtime(time);
+printf "[%4d-%02d-%02d %02d:%02d:%02d] %s: %s\n",$year+1900,$mon+1,$mday,$hour,$min,$sec,"memxml2db","Deleting cached";
 # Delete things left that shouldn't be from this table
 foreach my $gid (keys %$gid_to_internal) {
     $sth = PublicWhip::DB::query($dbh, "delete from pw_mp where gid = '$gid'");

website/.htaccess

 </FilesMatch>
 Options ExecCGI
 DirectoryIndex index.php
-
+AddHandler cgi-script .cgi
 # make things safe
 #php_flag register_globals off # doesn't work / not needed for PHP5
 <Files ~ "\.inc$">
 RedirectPermanent /code.php http://www.publicwhip.org.uk/project/code.php
 RedirectPermanent /mp-info.xml http://www.publicwhip.org.uk/feeds/mp-info.xml
 RedirectPermanent /dreammp.php http://www.publicwhip.org.uk/policy.php
-RedirectPermanent /awstats http://seagrass.goatchurch.org.uk/cgi-bin/awstats.pl?config=publicwhip
 
+RewriteCond %{HTTP_HOST} !^(www\.publicwhip\.org\.uk|pw\.live|publicwhip-v1.localdomain)$
+RewriteRule (.*) http://www.publicwhip.org.uk/$1 [R=301,L]
 Options +FollowSymLinks
 RewriteEngine On
+RewriteCond %{QUERY_STRING} (.*)Louise_Bagshawe(.*)
+RewriteRule ^ %{REQUEST_URI}?%1Louise_Mensch%2 [R=302,L]
+
 RewriteRule ^testrewrite$ /index.php [PT]
 RewriteRule ^[Nn]/([^;]+)$                      /newsletters/signup.php?token=$1
 RewriteRule ^[Uu]/([^;]+)$                      /newsletters/signup.php?token=$1&unsub=1
 RewriteRule ^fortytwodays$ /fortytwodays.php [PT]
 
 RewriteRule ^forum.*$ /outageforums.php [R=302,L]
+RewriteRule ^account.*$ /outageforums.php [R=302,L]
 
 # Net strike
 #RewriteCond %{REQUEST_URI} !^/patents/patents\.php

website/ORIGINAL_election_crewe_and_nantwich_2008.php

 print "<h4 id=\"th4b\"><strike>Crewe and Nantwich</strike>$constituency</h4>\n";
 print "<h4 id=\"th4c\">by-election: 6 November 2008</h4>\n";
 
-$vdash = mysql_escape_string(db_scrub($_GET["dash"])); # used to tell if /by-election or /byelection was used
+$vdash = mysql_real_escape_string(db_scrub($_GET["dash"])); # used to tell if /by-election or /byelection was used
 $vpostcode = db_scrub($_POST["vpostcode"]);  # a string of letters (each a-e for strong favour to against) in order of the policies
-$vvote = mysql_escape_string(db_scrub($_POST["vvote"]));
-$vkey = mysql_escape_string(db_scrub($_POST["vkey"]));
-$vcomment = mysql_escape_string(db_scrub($_POST["vcomment"]));
-$vinitials = mysql_escape_string(db_scrub($_POST["vinitials"]));
+$vvote = mysql_real_escape_string(db_scrub($_POST["vvote"]));
+$vkey = mysql_real_escape_string(db_scrub($_POST["vkey"]));
+$vcomment = mysql_real_escape_string(db_scrub($_POST["vcomment"]));
+$vinitials = mysql_real_escape_string(db_scrub($_POST["vinitials"]));
 $vrand = $_POST["vrand"];
 if ($vrand)
     $vrand = (int)$vrand;
 $ipnumber = $_SERVER["REMOTE_ADDR"];
 if (!$referrer)
     $referrer = $_SERVER["HTTP_USER_AGENT"];
-if (!isrobot() and !$vkey and !preg_match("/.*?house=z/", $querystring))
-    $db->query("INSERT INTO pw_logincoming
-            (referrer, ltime, ipnumber, page, subject, url, thing_id)
-            VALUES ('$referrer', NOW(), '$ipnumber', 'crewe_election', '$vkey', '$vdash', $vrand)");
 
 # we've had a posting
 if ($vkey)

website/account/addpolicy.php

         }
     ?>
         <P>
-        <FORM ACTION="<?=$PHP_SELF?>" METHOD="POST">
+        <FORM ACTION="<?php echo $PHP_SELF?>" METHOD="POST">
         <B>Title of policy</B><BR>
-        <INPUT TYPE="TEXT" NAME="name" VALUE="<?=html_scrub($name)?>" SIZE="40" MAXLENGTH="50">
+        <INPUT TYPE="TEXT" NAME="name" VALUE="<?php echo html_scrub($name)?>" SIZE="40" MAXLENGTH="50">
         <P>
         <B>Someone who believes that...</B><BR>
-        <textarea name="description" rows="6" cols="80"><?=html_scrub($description)?></textarea></p>
+        <textarea name="description" rows="6" cols="80"><?php echo html_scrub($description)?></textarea></p>
 
         <p><span class="ptitle">Privacy notes:</span>
         By creating a policy you are making your user name
-        <b><?=user_getname()?></b> and the policy's voting record public.  
+        <b><?php echo user_getname()?></b> and the policy's voting record public.
 
         <p><INPUT TYPE="SUBMIT" NAME="submit" VALUE="Make Policy">
         </FORM>

website/account/changeemail.php

 require_once "../database.inc";
 require_once "user.inc";
 
-$password1=mysql_escape_string($_POST["password1"]);
-$new_email=mysql_escape_string($_POST["new_email"]);
-$change_user_name=mysql_escape_string($_POST["change_user_name"]);
-$submit=mysql_escape_string($_POST["submit"]);
+$password1=mysql_real_escape_string($_POST["password1"]);
+$new_email=mysql_real_escape_string($_POST["new_email"]);
+$change_user_name=mysql_real_escape_string($_POST["change_user_name"]);
+$submit=mysql_real_escape_string($_POST["submit"]);
 $ok = false;
 if ($submit) {
 	$ok = user_change_email ($password1,$new_email,$change_user_name);

website/account/changepass.php

 
 if (user_isloggedin())
 {
-    $new_password1=mysql_escape_string($_POST["new_password1"]);
-    $new_password2=mysql_escape_string($_POST["new_password2"]);
+    $new_password1=mysql_real_escape_string($_POST["new_password1"]);
+    $new_password2=mysql_real_escape_string($_POST["new_password2"]);
     $change_user_name=$user_name;
-    $old_password=mysql_escape_string($_POST["old_password"]);
-    $submit=mysql_escape_string($_POST["submit"]);
+    $old_password=mysql_real_escape_string($_POST["old_password"]);
+    $submit=mysql_real_escape_string($_POST["submit"]);
 
     $ok = false;
     if ($submit) {

website/account/confirm.php

 require_once "../database.inc";
 require_once "user.inc";
 
-$email=mysql_escape_string($_GET["email"]);
-$hash=mysql_escape_string($_GET["hash"]);
+$email=mysql_real_escape_string($_GET["email"]);
+$hash=mysql_real_escape_string($_GET["hash"]);
 
 if ($hash && $email) {
     $return_url = user_confirm($hash,$email);

website/account/lostpass.php

 require_once "../database.inc";
 require_once "user.inc";
 
-$email=mysql_escape_string($_POST["email"]);
-$user_name=mysql_escape_string($_POST["user_name"]);
-$submit=mysql_escape_string($_POST["submit"]);
+$email=mysql_real_escape_string($_POST["email"]);
+$user_name=mysql_real_escape_string($_POST["user_name"]);
+$submit=mysql_real_escape_string($_POST["submit"]);
 
 if (user_isloggedin()) {
 	user_logout();

website/account/register.php