1. Moritz Wilhelmy
  2. eggdrop-urlmagic

Overview

HTTPS SSH
urlmagic
========

Written by rojo and ente

Description
-----------

Does following magic things with URLs:

  * Follows links posted in a channel if the udef flag +urlmagic is set
  * If content-type ~ text/html and <title>content exists</title> display title
  * Otherwise, display content-type
  * If url length > threshold, fetch and display tinyurl

Optional features
-----------------

Plugins exist for

  * Optionally recording all URLs to your Twitter page or an SQLite database
  * Flagging links as NSFW if it was mentioned in the text next to the link
  * Displaying the size of a page
  * Displaying the final destination URL in case there was a redirect

In short, what's displayed and what isn't can be influenced by plugins.

This script needs at least the following packages: http, tls and tcllib (for hook)
It is advised to also install either gumbo or arabica to use the more fail-safe
title parser. Your eggdrop needs to be compiled against Tcl 8.5 or higher. Tcl 8.4
and below are at least untested, but probably won't work at all.

Dependencies
------------

Use your distribution's package management system to install the dependencies
as appropriate.

For optional dependencies introduced by specific plugins, check the directory of
the plugin you want to use for a file called README.

### Examples:

#### Debian / Ubuntu:

  * apt-get install mercurial       (optional; makes updating easier)
  * apt-get install tcllib tcl-tls  (no matter what plugins are enabled; "core")
  * apt-get install libsqlite3-tcl  (if sqlite plugin is enabled)

On Debian stretch and above, there is a package for the gumbo title parser:

  * apt-get install libgumbo-dev build-essentials (optional, possible title parser)

#### Red Hat / CentOS:

  * yum install mercurial           (optional)
  * yum install tcllib tcltls       (core)
  * yum install sqlite-tcl          (sqlite plugin)

#### OpenSuSE:

  * zypper install mercurial        (optional)
  * zypper install tcllib tls       (core)
  * (sqlite plugin does not require extra packages to be installed)

#### Gentoo:

  * emerge -v mercurial             (optional)
  * emerge -v tcllib dev-tcltk/tls  (core)
  * emerge -v sqlite                (sqlite plugin)

#### FreeBSD (port names):

  * devel/mercurial                 (optional)
  * devel/tcllib devel/tcltls       (core)
  * databases/tcl-sqlite3           (sqlite)
  * devel/gumbo                     (possible title parser)

#### NetBSD (pkgsrc package names):

  * devel/mercurial                 (optional)
  * devel/tcllib security/tcl-tls   (core)
  * databases/sqlite3               (sqlite)


### Title parsers

The "dumb" regex-based title extraction regexp is not very reliable. Therefore,
there are multiple options for extracting the title out of the HTML documents
downloaded from the server. The most reliable option is using Google's gumbo
library, however it isn't packaged for any distributions I've checked (and
neither is the much less robust arabica library, which has been around for
several years longer). To avoid making each potential user compile their own
gumbo version, I've written a title parser in perl that is invoked via pipe,
because html5 parsers for perl are more easily available than for Tcl or C.

Unfortunately, this introduces a dependency to either Tcl 8.6 or TclX. 

This means that you will want to install the following packages on
debian wheezy if your bot is linked against Tcl 8.5: 

  * apt-get install libhtml-html5-parser-perl tclx8.4

If your bot is linked against Tcl 8.6, you can leave tclx8.4 off and be done.

On OpenSuSE, you can find a repository/package for Perl's html5 parser here:

  * http://software.opensuse.org/package/perl-HTML-HTML5-Parser

Alternatively, you can install the perl packages from cpan (for instance if
you're not on debian and your distribution doesn't have a package). The command
is:

  * cpan HTML::HTML5::Parser


Please report bugs, gripes, missing operating systems in the table above, etc
to ente on IRCnet, and feel free to let me know if you use it, or just contact
me for chit-chat.