1. Scott Zeid
  2. Dilbert Tools

Overview

HTTPS SSH

Dilbert Tools

Copyright (c) 2008-2017 Scott Zeid
https://code.s.zeid.me/dilbert-tools

Download Dilbert strips and maintain a private Dilbert collection.


dilbert-tools consists of two Python command-line scripts:

  • fetch-dilbert downloads individual Dilbert comic strips and their metadata.
  • update-dilbert helps you maintain a private Dilbert collection.

Disclaimer

These scripts are automated scripts that will scrape various pages from dilbert.com. This is a violation of their terms of use. Your use of these scripts is at your own risk. By downloading or using these scripts, you agree to not hold me liable for any action taken against you due to your use or download of them.

Installation

dilbert-tools requires Python 2.7 or later (including 3.x), unless you are using the Windows EXEs.

To install dilbert-tools, you have several options:

  • Extract fetch-dilbert and update-dilbert (or the EXE versions if on Windows) from the appropriate zip file for your platform. If you are not on Windows, make sure fetch-dilbert and update-dilbert are executable.
  • Download the dilberttools-<version>.tar.gz archive and run pip install <archive>.
  • From the root of the repository, run ./setup.py install or pip install ..

Requirements

If you installed dilbert-tools with pip or if you are using the Windows EXEs, then you do not need to install anything else.

Otherwise, you must also install the following:

  • BeautifulSoup 4
  • lxml
  • Pillow
  • PyYAML
  • Requests

To install the requirements on Fedora:

sudo dnf install python python-lxml python-pillow PyYAML python-requests python-pip
sudo pip install -U beautifulsoup4

To install the requirements on Ubuntu and other Debian-based distros:

sudo apt-get install python python-lxml python-pillow python-yaml python-pip
sudo pip install -U beautifulsoup4 requests

To install the requirements on other platforms, with Python 2.7 or later and pip already installed:

pip install -U beautifulsoup4 lxml Pillow PyYAML requests

fetch-dilbert

This script downloads Dilbert strips for a given day or days, or a whole year at a time. It will also save the strips' titles, tags, and transcripts to YAML files.

Each YAML file will be named YYYY-MM-DD.yml and will have the keys date, title, tags, and transcript. If there is no title or transcript for the strip, then the respective key's value will be null.

If a strip has to be downloaded from a source other than dilbert.com, then its YAML file will also have a key called alternate-source containing the URL of either the page that was used or the image URL itself. A warning will also be printed to standard error for each alternate source tried.

Usage

Syntax: fetch-dilbert [-h|--help] [-v|--verbose] [-o|--output|--output-dir <path>] <date/year> [...]

A date or range of dates must be given on the command line as positional arguments in one of the following formats:

  • today
    Download today's strip.
  • YYYY-MM-DD
    Download the strip from the given day.
  • YYYY
    Download the strip from the given year.

Date/year arguments may be separated with spaces, commas, or both, and extraneous spaces and commas will be ignored.

The -o/--output/--output-dir argument is optional, and it is the folder where you want the strip(s) to be saved. It defaults to saving in the current directory.

The old -d/--date, --dates, and -y/--year arguments are deprecated, and they may be removed at a later date. You should transition to using the new positional arguments if you are still using the old ones.

Examples

  • fetch-dilbert today 2008-06-23 1992
    Downloads today's strip, the one from June 23, 2008, and all strips from the year 1992 and saves them to the current working directory.
  • fetch-dilbert 2008-04-15 -o Comics/Dilbert/2008
    Downloads the strip from April 15, 2008 and saves it to the folder Comics/Dilbert/2008 within the current directory.
  • fetch-dilbert 1997 -vo Comics/Dilbert/1997
    Downloads the strips from the year 1997, outputs progress messages to standard error, and saves them to the folder Comics/Dilbert/1997 within the current directory.

update-dilbert

This script helps you maintain a collection of Dilbert strips. It will download all strips that have been published during the current year that you don't already have. Run daily after midnight Pacific time for best results.

Usage

Syntax: update-dilbert [-h|--help] [-v|--verbose] [-p|--path <path>] [-m|--metadata-only]

Run update-dilbert -p <path>, where <path> is where you keep your Dilbert strips, and it will do the dirty work for you. If the -p argument is omitted, it will use the current directory. <path> should contain directories whose names are year numbers. Year subdirectories will be created if necessary.

Building dilbert-tools

To build dilbert-tools, run make from the root of the repository in a Unix-like environment with the below dependencies installed.

To make the distribution zip files, run make dist. The Windows zip file will be created only if EXE files for both scripts exist in dist/<version-number>.

Build dependencies

  • Unix-like environment (e.g. Linux, Cygwin, Windows Subsystem for Linux, O$ X, ...)
  • GNU make
  • zip
  • setuptools

Making Windows EXEs

The official EXEs are created using make exes, which uses ezpyi, which is a wrapper for PyInstaller. Except in Windows Subsystem for Linux, Cygwin, and MSYS, make exes uses Wine via this ezpyi-wine script.

On Unix-like platforms (except WSL, Cygwin, and MSYS):

  1. Install Wine.

  2. Download ezpyi-wine, and preferably put it somewhere on your $PATH.

  3. Set up the environment and install the runtime dependencies into it:
    ezpyi-wine : pip install -U beautifulsoup4 lxml Pillow PyYAML requests

  4. Run make exes, or if necessary, make exes EZPYI_WINE=<path to ezpyi-wine>.

On Windows Subsystem for Linux:

  1. Install Python inside the Windows environment.

  2. In Windows, set the PYTHONIOENCODING environment variable to UTF-8:

    a. Run sysdm.cpl.

    b. Go to Advanced → Environment Variables.

    c. Under System variables (or User variables if necessary), click New.

    d. Set the name to PYTHONIOENCODING and the value to UTF-8.

    e. Click OK → OK → OK.

    f. Restart the WSL environment if it is already running.

  3. In Windows, install ezpyi: pip install -U ezpyi

  4. In Windows, install the runtime dependencies:
    pip install -U beautifulsoup4 lxml Pillow PyYAML requests

  5. Make sure the dilbert-tools Git repository is located in a Windows path (e.g. /mnt/c/...).

  6. In WSL, run make exes, or if necessary, make exes EZPYI=<path to ezpyi>.

On Cygwin or MSYS:

  1. In Windows, install ezpyi: pip install -U ezpyi

  2. In Windows, install the runtime dependencies:
    pip install -U beautifulsoup4 lxml Pillow PyYAML requests

  3. Run make exes, or if necessary, make exes EZPYI=<path to ezpyi>.