Bitbucket is a code hosting site with unlimited public and private repositories. We're also free for small teams!

Close

Wikipedia edits and views data - scripts for downloading and processing

Edits data from SNAP

Download the data from SNAP; process it as follows:

bzcat enwiki-20080103.main.bz2 | python wikipedia.py | bzip2 > output.txt.bz2

See wikipedia.sh for more on usage. The AWK script wikipedia.awk and C++ program wikipedia are redundant implementations.

The scripts transform the input, which has records like so:

REVISION 4781981 72390319 Steven_Strogatz 2006-08-28T14:11:16Z SmackBot 433328
CATEGORY American_mathematicians
IMAGE
MAIN Boston_University MIT Harvard_University Cornell_University
TALK
USER
USER_TALK
OTHER De:Steven_Strogatz Es:Steven_Strogatz
EXTERNAL http://www.edge.org/3rd_culture/bios/strogatz.html
TEMPLATE Cite_book Cite_book Cite_journal
COMMENT ISBN formatting &/or general fixes using [[WP:AWB|AWB]]
MINOR 1
TEXTDATA 229
[blank line]

into a tab-separated format with 18 fields:

4781981 72390319    Steven_Strogatz 2006-08-28T14:11:16Z    SmackBot    433328  American_mathematicians     Boston_University MIT Harvard_University Cornell_University             De:Steven_Strogatz Es:Steven_Strogatz   http://www.edge.org/3rd_culture/bios/strogatz.html  Cite_book Cite_book Cite_journal    ISBN formatting &/or general fixes using [[WP:AWB|AWB]] 1   229

The first six fields contain the data from the REVISION line split by tabs. Each other line's data is put into a field verbatim; thus, within some of the other fields, there are several elements separated by spaces; e.g., the TEMPLATE field contains "Cite_book Cite_book Cite_journal", and the OTHER field contains "De:Steven_Strogatz Es:Steven_Strogatz".

The output has a total of 116,590,856 records.

Raw page views data

Downloading

The download_pagecounts.sh script retrieves the raw views data from dumps.wikimedia.org, saving it in the current directory. To use it:

bash download_pagecounts.sh

It can be re-run without redownloading any existing files and will check that each month downloaded properly using the MD5 sums.

Aside from MD5 checksumming and putting all files in the current directory, download_pagecounts.sh is easily replaced by

wget -r --no-parent http://dumps.wikimedia.org/other/pagecounts-raw/

if wget is available (it isn't distributed with OS X, while curl is).

Aggregating

The join_pagecounts.sh script combines the pageview files, producing files with the page name and a string of space-separated view counts on each line:

Brown-eared 1 1 1
Brown-eared_Bulbul 1 1 1 1
Brown_eared-pheasant 1
Brown-eared_Pheasant 2
Brown_Eared_Pheasant 2 1 1 1 1 1 1
Brown-eared_Woodpecker 1 1
Brown-eared_Woolly_Opossum 1 1 1 3 1 1

The sum_pagecounts.py script acts as a filter, reducing the string of space-separated counts to their sums:

Brown-eared 3
Brown-eared_Bulbul 4
Brown_eared-pheasant 1
Brown-eared_Pheasant 2
Brown_Eared_Pheasant 8
Brown-eared_Woodpecker 2
Brown-eared_Woolly_Opossum 8

The syntax for join_pagecounts.sh is join_pagecounts.sh [-r|-i] [outfile] [infiles]. Use -r ("raw") if the inputs are the raw pagecounts-????????-??????.gz files. For instance,

./join_pagecounts.sh -r join-20071210.txt pagecounts-20071210-??????.gz

joins the raw pageview files for 2007/12/10 into a file named join-20071210.txt. Use -i ("intermediate") with output files like join-20071210.txt. For instance,

for d in {01..31}; do ./join_pagecounts.sh -r join-200712${d}.txt pagecounts-200712${d}-??????.gz; done

./join_pagecounts.sh -i join-200712.txt join-200712??.txt

first joins the raw pageview files for each day in December, 2007 into files named join-20071210.txt, join-20071211.txt, join-20071212.txt, ..., join-20071231.txt, then joins these files together to produce join-200712.txt (note that December, 2007 lacks raw pageview files prior to 2007/12/10).

At any point, you can aggregate the counts down to their sums by running the file through sum_pagecounts.py. For instance,

cat join-200712.txt | ./sum_pagecounts.py > sum-200712.txt

produces sum-200712.txt, containing the total number of pageviews for each page in December, 2007.

You can also add the optional -s flag to join_pagecounts.sh to tell it to sum the counts before outputting them.

To join and sum an entire year (e.g., 2008), do

for m in {01..12}; do ./join_pagecounts.sh -r -s join-2008${m}.txt pagecounts-2008${m}??-??????.gz; done

./join_pagecounts.sh -i -s join-2008.txt join-2008??.txt

This results in a file, join-2008.txt, containing the total number of pageviews for each page in all of 2008, along with monthly totals in join-200801.txt, join-200802.txt, ..., join-200812.txt.

Recent activity

Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.