Commits

Anonymous committed 8802249

Updated API to v2
Built framework for easier extensibility for other exception modules
Cleaned up code quite a bit

Comments (0)

Files changed (24)

+
+django-hoptoad
+==============
+
+django-hoptoad is some simple Middleware for letting Django_-driven websites report their errors to Hoptoad_.  Now ponies_ can ride the toad too.
+
+.. _Django: http://djangoproject.com/
+.. _Hoptoad: http://hoptoadapp.com/
+.. _ponies: http://djangopony.com/
+
+
+Requirements
+------------
+
+django-hoptoad requires:
+
+* Python_ 2.5+ (preferably 2.6+ as that's what I've tested it with)
+* PyYAML_ (`pip install pyyaml` or `easy_install pyyaml`)
+* Django_ 1.0+
+* A Hoptoad_ account
+
+.. _Python: http://python.org/
+.. _PyYAML: http://pyyaml.org/
+
+
+Installation
+------------
+
+Grab the the django-hoptoad code by cloning the Mercurial_ repository (or just `download the latest version <http://bitbucket.org/sjl/django-hoptoad/get/tip.zip>`_ and unzip it somewhere)::
+
+    hg clone http://bitbucket.org/sjl/django-hoptoad/
+
+There's a git mirror too if you *really* want it::
+
+    git clone git://github.com/sjl/django-hoptoad.git
+
+Once you download it, you can install it in the usual manner::
+
+    cd django-hoptoad
+    python setup.py install
+
+If you'd prefer to be able to update at any time by pulling down changes with Mercurial or git, you can symlink the module into your ``site-packages`` directory instead of using ``python setup.py install``::
+
+    ln -s /full/path/to/django-hoptoad/hoptoad /full/path/to/site-packages/
+
+To make sure it works you can run::
+
+    python -c 'import hoptoad'
+
+.. _Mercurial: http://mercurial.selenic.com/
+
+
+Usage
+-----
+
+To set up a Django project to notify Hoptoad of its errors, you need to do two things in its `settings.py` file.
+
+First, add the ``HoptoadNotifierMiddleware`` as the last item in the ``MIDDLEWARE_CLASSES``` setting::
+
+    MIDDLEWARE_CLASSES = (
+        # ... other middleware classes ...
+        'hoptoad.middleware.HoptoadNotifierMiddleware',
+    )
+
+Next, you'll need to add a ``HOPTOAD_API_KEY`` setting.  You can get the key from the Hoptoad project page::
+
+    HOPTOAD_API_KEY = 'Your Hoptoad API key.'
+
+
+Documentation
+-------------
+
+The documentation for django-hoptoad is at the `project page <http://sjl.bitbucket.org/django-hoptoad/>`_. There's a `Quick Start guide <http://sjl.bitbucket.org/django-hoptoad/quickstart/>`_, `Configuration guide <http://sjl.bitbucket.org/django-hoptoad/config/>`_, `Troubleshooting guide <http://sjl.bitbucket.org/django-hoptoad/troubleshooting/>`_, and a few other things there.
+
+The documentation is stored in the ``docs/`` directory of the repository if you prefer to read it offline.
+
+
+Suggestions
+-----------
+
+This Middleware is a work in progress.  If you have a suggestion or find a bug please `add an issue <http://bitbucket.org/sjl/django-hoptoad/issues/?status=new&status=open>`_ and let me know.
+wiki-name: "django-hoptoad"
+
+static-dir: "static"
+
+hide-prefix: "."
+document-extensions: [.md, .mdown, .markdown]
+generate-listing: always
+listing-filename: "list.html"
+
+use-default-static: false
+
+markdown:
+  safe_mode: false
+  output_format: xhtml1
+  extensions: [codehilite, def_list, toc]
+#!/usr/bin/env bash
+
+markdoc build
+rsync --delete -az .html/ ~/src/sjl.bitbucket.org/django-hoptoad
+hg -R ~/src/sjl.bitbucket.org commit -Am 'django-hoptoad: Update documentation.'
+hg -R ~/src/sjl.bitbucket.org push

docs/static/media/css/layout.css

+/* @override http://localhost:8008/media/css/layout.css */
+body, html {
+    background-color: #092E20;
+    margin: 0;
+    padding: 0;
+}
+
+div#breadcrumbs {
+    border-bottom: 4px dashed #092E20;
+    padding: 0em 1.5em;
+    background-color: #fff;
+    width: 55em;
+    margin: 0em 0em 1em -1.5em;
+}
+div#breadcrumbs p {
+    margin: 0.5em auto;
+}
+div#content {
+    background-color: #fff;
+    height: 100%;
+    margin: 0 auto 3em;
+    overflow: hidden;
+    padding: 0em 1.5em 0em 1.5em;
+    width: 55em;
+}
+
+h1 {
+    margin: 0.3em 0;
+    padding: 0.1em 0;
+}
+h2, h3, h4, h5, h6 {
+    margin-top: 1em;
+    margin-bottom: 0.4em;
+}
+p, ul, blockquote {
+    line-height: 1.8em;
+    margin-top: 0;
+    margin-bottom: 1em;
+}
+a {
+    color: #3E6B00;
+}
+a:hover {
+    color: #F14800;
+}
+
+p#footer {
+    background-color: #fff;
+    border-top: 4px dashed #092E20;
+    width: 55em;
+    padding: 0.75em 1.5em;
+    text-align: center;
+    color: #666;
+    margin-top: 1.5em;
+    margin-left: -1.5em;
+    margin-bottom: 0em;
+}
+
+ul {
+    padding-left: 1.5em;
+}
+ul li {
+    list-style-type: disc;
+    margin-left: 1.5em;
+    margin-top: 0.4em;
+}
+
+div.toc ul {
+    padding: 0;
+}
+div.toc ul li {
+    list-style-type: none;
+    margin: 0;
+}
+div.toc > ul > li {
+    list-style-type: none;
+    margin: 0;
+}
+div.toc > ul ul {
+    margin: 0 3em;
+}
+
+code {
+    background-color: #f6f6f6;
+    border: 1px solid #999;
+    padding: 2px;
+    white-space: nowrap;
+}
+pre code {
+    border: none;
+    padding: 0;
+    background: none;
+    white-space: pre-wrap;
+}
+a code {
+    background: none;
+    border: none;
+    margin: 0;
+    padding: 0;
+    text-decoration: inherit;
+}
+a.code {
+    background-color: #3f3f3f;
+}
+
+dt {
+    font-weight: bold;
+    margin-top: 1em;
+}
+dd {
+    line-height: 1.8em;
+    margin-left: 1.5em;
+}
+blockquote p {
+    margin: 0;
+}
+
+img {
+    background-color: #f6f6f6;
+    border: 1px solid #999;
+    padding: 1em;
+    display: block;
+    margin: 0 auto;
+}
+
+table {
+    margin: 1em;
+}
+table thead {
+    background-color: #f6f6f6;
+}
+table thead th {
+    border: 1px solid #999;
+    padding: 0.5em 1em;
+}
+table tbody tr td {
+    border: 1px solid #999;
+    padding: 0.5em 1em;
+}
+
+pre, blockquote, table.codehilitetable {
+    background-color: #f6f6f6;
+    border: 1px solid #999;
+    display: block;
+    line-height: 1.5em;
+    margin-left: 1.5em;
+    margin-right: 1.5em;
+    padding: 0.5em;
+    padding-left: 1em;
+}
+pre, table.codehilitetable {
+    margin-bottom: 1em;
+}
+table.codehilitetable div.linenodiv {
+    border-right: 1px solid #ccc;
+    margin-right: 1em;
+    padding-right: 1em;
+}
+table.codehilitetable pre {
+    background: none;
+    border: none;
+    margin: 0;
+    padding: 0;
+}
+
+table#pages tr, table#subdirs tr, table#files tr {
+    border-top: 1px solid #999;
+    border-bottom: 1px solid #999;
+}
+table tr td.name a {
+    display: block;
+    padding: 0.5em 1em;
+}
+table tr td.name a:hover {
+    background-color: #f6f6f6;
+}
+table tr td.size {
+    padding: 0.5em 1em;
+    width:  40px;
+}

docs/static/media/css/pygments.css

+.hll { background-color: #ffffcc }
+.c { color: #808080 } /* Comment */
+.err { color: #F00000; background-color: #F0A0A0 } /* Error */
+.k { color: #008000; font-weight: bold } /* Keyword */
+.o { color: #303030 } /* Operator */
+.cm { color: #808080 } /* Comment.Multiline */
+.cp { color: #507090 } /* Comment.Preproc */
+.c1 { color: #808080 } /* Comment.Single */
+.cs { color: #cc0000; font-weight: bold } /* Comment.Special */
+.gd { color: #A00000 } /* Generic.Deleted */
+.ge { font-style: italic } /* Generic.Emph */
+.gr { color: #FF0000 } /* Generic.Error */
+.gh { color: #000080; font-weight: bold } /* Generic.Heading */
+.gi { color: #00A000 } /* Generic.Inserted */
+.go { color: #808080 } /* Generic.Output */
+.gp { color: #c65d09; font-weight: bold } /* Generic.Prompt */
+.gs { font-weight: bold } /* Generic.Strong */
+.gu { color: #800080; font-weight: bold } /* Generic.Subheading */
+.gt { color: #0040D0 } /* Generic.Traceback */
+.kc { color: #008000; font-weight: bold } /* Keyword.Constant */
+.kd { color: #008000; font-weight: bold } /* Keyword.Declaration */
+.kn { color: #008000; font-weight: bold } /* Keyword.Namespace */
+.kp { color: #003080; font-weight: bold } /* Keyword.Pseudo */
+.kr { color: #008000; font-weight: bold } /* Keyword.Reserved */
+.kt { color: #303090; font-weight: bold } /* Keyword.Type */
+.m { color: #6000E0; font-weight: bold } /* Literal.Number */
+.s { background-color: #fff0f0 } /* Literal.String */
+.na { color: #0000C0 } /* Name.Attribute */
+.nb { color: #007020 } /* Name.Builtin */
+.nc { color: #B00060; font-weight: bold } /* Name.Class */
+.no { color: #003060; font-weight: bold } /* Name.Constant */
+.nd { color: #505050; font-weight: bold } /* Name.Decorator */
+.ni { color: #800000; font-weight: bold } /* Name.Entity */
+.ne { color: #F00000; font-weight: bold } /* Name.Exception */
+.nf { color: #0060B0; font-weight: bold } /* Name.Function */
+.nl { color: #907000; font-weight: bold } /* Name.Label */
+.nn { color: #0e84b5; font-weight: bold } /* Name.Namespace */
+.nt { color: #007000 } /* Name.Tag */
+.nv { color: #906030 } /* Name.Variable */
+.ow { color: #000000; font-weight: bold } /* Operator.Word */
+.w { color: #bbbbbb } /* Text.Whitespace */
+.mf { color: #6000E0; font-weight: bold } /* Literal.Number.Float */
+.mh { color: #005080; font-weight: bold } /* Literal.Number.Hex */
+.mi { color: #0000D0; font-weight: bold } /* Literal.Number.Integer */
+.mo { color: #4000E0; font-weight: bold } /* Literal.Number.Oct */
+.sb { background-color: #fff0f0 } /* Literal.String.Backtick */
+.sc { color: #0040D0 } /* Literal.String.Char */
+.sd { color: #D04020 } /* Literal.String.Doc */
+.s2 { background-color: #fff0f0 } /* Literal.String.Double */
+.se { color: #606060; font-weight: bold; background-color: #fff0f0 } /* Literal.String.Escape */
+.sh { background-color: #fff0f0 } /* Literal.String.Heredoc */
+.si { background-color: #e0e0e0 } /* Literal.String.Interpol */
+.sx { color: #D02000; background-color: #fff0f0 } /* Literal.String.Other */
+.sr { color: #000000; background-color: #fff0ff } /* Literal.String.Regex */
+.s1 { background-color: #fff0f0 } /* Literal.String.Single */
+.ss { color: #A06000 } /* Literal.String.Symbol */
+.bp { color: #007020 } /* Name.Builtin.Pseudo */
+.vc { color: #306090 } /* Name.Variable.Class */
+.vg { color: #d07000; font-weight: bold } /* Name.Variable.Global */
+.vi { color: #3030B0 } /* Name.Variable.Instance */
+.il { color: #0000D0; font-weight: bold } /* Literal.Number.Integer.Long */

docs/static/media/css/reset.css

+/*
+Copyright (c) 2009, Yahoo! Inc. All rights reserved.
+Code licensed under the BSD License:
+http://developer.yahoo.net/yui/license.txt
+version: 2.7.0
+*/
+html{color:#000;background:#FFF;}body,div,dl,dt,dd,ul,ol,li,h1,h2,h3,h4,h5,h6,pre,code,form,fieldset,legend,input,button,textarea,p,blockquote,th,td{margin:0;padding:0;}table{border-collapse:collapse;border-spacing:0;}fieldset,img{border:0;}address,caption,cite,code,dfn,em,strong,th,var,optgroup{font-style:inherit;font-weight:inherit;}del,ins{text-decoration:none;}li{list-style:none;}caption,th{text-align:left;}h1,h2,h3,h4,h5,h6{font-size:100%;font-weight:normal;}q:before,q:after{content:'';}abbr,acronym{border:0;font-variant:normal;}sup{vertical-align:baseline;}sub{vertical-align:baseline;}legend{color:#000;}input,button,textarea,select,optgroup,option{font-family:inherit;font-size:inherit;font-style:inherit;font-weight:inherit;}input,button,textarea,select{*font-size:100%;}

docs/static/media/css/typography.css

+html {
+	font-family: "Helvetica Neue", Helvetica, Arial, Geneva, sans-serif;
+	font-size: 10pt;
+}
+
+h1, h2, h3, h4, h5, h6 {
+	font-weight: bold;
+}
+
+h1 {
+	font-size: 2em;
+}
+
+h2 {
+	font-size: 1.6em;
+}
+
+h3 {
+	font-size: 1.3em;
+}
+
+h4 {
+	font-size: 1.1em;
+	font-weight: bold;
+}
+
+em {
+	font-style: italic;
+}
+
+strong {
+	font-weight: bold;
+}
+
+a {
+    font-weight: bold;
+	text-decoration: none;
+}
+
+table#files a:hover, table#subdirs a:hover, table#pages a:hover {
+	color: #a00000;
+}
+
+p#footer a {
+	text-decoration: none;
+}
+
+p code, ul code, ol code, dl code, blockquote code, tbody code, thead code {
+	font-size: 8pt;
+}
+
+pre, code, tt, table#subdirs tr td.name, table#files tr td.name, table tr td.size {
+	font-family: "DejaVu Sans Mono", "Bitstream Vera Sans Mono", Menlo, Inconsolata, Consolas, Monaco, "Courier New", Courier;
+}
+
+table#subdirs code, table#files code {
+	font-size: 10pt;
+}
+
+table.codehilitetable div.linenodiv {
+	color: #777;
+}
+
+blockquote {
+	color: #222;
+	font-style: italic;
+}
+
+pre {
+	overflow-x: auto; /* Use horizontal scroller if needed; for Firefox 2, not needed in Firefox 3 */
+	white-space: pre-wrap; /* css-3 */
+	white-space: -moz-pre-wrap !important; /* Mozilla, since 1999 */
+	white-space: -pre-wrap; /* Opera 4-6 */
+	white-space: -o-pre-wrap; /* Opera 7 */
+	/* width: 99%; */
+	word-wrap: break-word; /* Internet Explorer 5.5+ */
+}
+
+table thead th {
+	font-weight: bold;
+}
+
+.list-crumb {
+	color: #777;
+	font-variant: italic;
+}

docs/wiki/config/index.mdown

+Configuration
+=============
+
+There are a few extra things you can configure if you'd like to tweak the notification process a bit.
+
+[TOC]
+
+Notify Hoptoad While in DEBUG Mode
+----------------------------------
+
+By default the Middleware will **not** report errors to Hoptoad when your project is in `DEBUG` mode.  The idea behind this is that if you can already see the error information right in the browser you probably don't need to see it in Hoptoad too.  If you want to always notify Hoptoad of errors, even while in `DEBUG` mode, add the following setting:
+
+    HOPTOAD_NOTIFY_WHILE_DEBUG = True
+
+Specify a Default Timeout
+-------------------------
+
+By default, the amount of time the notifier will wait before giving up on contacting Hoptoad is Python's "global default timeout setting".  I have no idea what that is because the [documentation][urllib2docs] does not see fit to explain that to me.
+
+If you'd like to change that amount you can use the `HOPTOAD_TIMEOUT` setting.  You **must** be running Python 2.6+ to use this.
+
+    HOPTOAD_TIMEOUT = 5
+
+The number is the number of seconds the notifier will wait before timing out.  Yes, you can use a float like `0.5` to specify fractions of a second.
+
+[urllib2docs]: http://docs.python.org/library/urllib2.html
+
+Track 404 Errors
+----------------
+
+By default Hoptoad will **not** be notified of 404 (page not found) errors.  If you'd like to change this you'll need to add the following setting:
+
+    HOPTOAD_NOTIFY_404 = True
+
+**IMPORTANT**: If you are using Django's `flatpages` app and want to track 404 errors, you need to make sure the `FlatpageFallbackMiddleware` comes *after* the `HoptoadNotifierMiddleware`.  If you don't do this Hoptoad will be notified of 404 errors even if the user actually sees a Flatpage.
+
+To track 404s while using the `flatpages` app your `MIDDLEWARE_CLASSES` setting should look like this:
+
+    MIDDLEWARE_CLASSES = (
+        # ... other middleware classes ...
+        'hoptoad.middleware.HoptoadNotifierMiddleware',
+        'django.contrib.flatpages.middleware.FlatpageFallbackMiddleware',
+    )
+
+A couple of things to note:
+
+* If your website doesn't have a favicon specified most browsers will request it each time.  This will result in one (or possibly two, if it tries to append a slash) 404 errors for every page view.
+* At the moment all 404 errors are grouped together as "similar" errors in Hoptoad.  I am trying to figure out what causes this.
+
+Track 403 Errors
+----------------
+
+By default Hoptoad will **not** be notified of 403 (forbidden) errors.  If you'd like to change this you'll need to add the following setting:
+
+    HOPTOAD_NOTIFY_403 = True
+
+Note:
+
+* At the moment all 403 errors are grouped together as "similar" errors in Hoptoad.  I am trying to figure out what causes this.
+
+Ignore Specific User Agents
+---------------------------
+
+If you'd like to ignore all errors from certain User Agents you can use the following setting:
+
+    HOPTOAD_IGNORE_AGENTS = ['MSIE 6.0', 'Trident']
+
+If any of the strings in the list appear *anywhere* in the User Agent string, Hoptoad will not be notified of the error.
+
+The strings are actually regular expressions, so you can be more specific if you like:
+
+    HOPTOAD_IGNORE_AGENTS = [r'^Mozilla.*compatible; MSIE \d+\.\d+.*$']
+
+One thing this is useful for (aside from hating on IE) is ignoring errors from web crawlers.  Often bots will mangle URLs and if you're tracking 404 errors you'll see a *lot* of errors that you probably don't care about.
+
+This would probably be a good starting point for ignoring crawlers:
+
+    HOPTOAD_IGNORE_AGENTS = ['Googlebot', 'Yahoo! Slurp', 'YahooSeeker']
+
+Problems?
+---------
+
+If you're having trouble you might want to take a look at the [Troubleshooting Guide][troubleshooting].
+
+[troubleshooting]: /troubleshooting/

docs/wiki/index.mdown

+django-hoptoad
+==============
+
+`django-hoptoad` is some simple Middleware for letting [Django][]-driven websites report their errors to [Hoptoad][].  Now [ponies][] can ride the toad too.
+
+[Django]: http://djangoproject.com/
+[Hoptoad]: http://hoptoadapp.com/
+[ponies]: http://djangopony.com/
+
+[Installation][]
+----------------
+
+[Quick Start][]
+---------------
+
+[Configuration][]
+-----------------
+
+[Troubleshooting][]
+-------------------
+
+[Code][]
+--------
+
+[Installation]: /installation/
+[Quick Start]: /quickstart/
+[Configuration]: /config/
+[Troubleshooting]: /troubleshooting/
+[Code]: http://bitbucket.org/sjl/django-hoptoad/
+

docs/wiki/installation/index.mdown

+Installation
+============
+
+django-hoptoad requires:
+
+* [Python][] 2.5+ (preferably 2.6+ as that's what I've tested it with)
+* [PyYAML][] (`pip install pyyaml` or `easy_install pyyaml`)
+* [Django][] 1.0+
+* A [Hoptoad][] account
+
+[Python]: http://python.org/
+[PyYAML]: http://pyyaml.org/
+[Django]: http://djangoproject.com/
+[Hoptoad]: http://hoptoadapp.com/
+
+Grab the the `django-hoptoad` code by cloning the [Mercurial][] repository (or just [download the latest version][tip-dl] and unzip it somewhere):
+
+    hg clone http://bitbucket.org/sjl/django-hoptoad/
+
+There's a git mirror too if you *really* want it.
+
+    git clone git://github.com/sjl/django-hoptoad.git
+
+Once you download it, you can install it in the usual manner:
+
+    cd django-hoptoad
+    python setup.py install
+
+If you'd prefer to be able to update at any time by pulling down changes with Mercurial or git, you can symlink the module into your `site-packages` directory instead of using `python setup.py install`:
+
+    ln -s /full/path/to/django-hoptoad/hoptoad /full/path/to/site-packages/
+
+To make sure it works you can run:
+
+    python -c 'import hoptoad'
+
+[Mercurial]: http://mercurial.selenic.com/
+[tip-dl]: http://bitbucket.org/sjl/django-hoptoad/get/tip.zip
+
+Take a look at the [Quick Start][] guide to learn how to report your Django project's errors to Hoptoad.
+
+[Quick Start]: /quickstart/

docs/wiki/quickstart/index.mdown

+Quick Start
+===========
+
+To set up a Django project to notify Hoptoad of its errors, you need to do two things in its `settings.py` file.
+
+First, add the `HoptoadNotifierMiddleware` as the last item in the `MIDDLEWARE_CLASSES` setting:
+
+    MIDDLEWARE_CLASSES = (
+        # ... other middleware classes ...
+        'hoptoad.middleware.HoptoadNotifierMiddleware',
+    )
+
+Next, you'll need to add a `HOPTOAD_API_KEY` setting.  You can get the key from the Hoptoad project page.
+
+    HOPTOAD_API_KEY = 'Your Hoptoad API key.'
+
+Restart the server and you're all set!  `django-hoptoad` will begin reporting errors to Hoptoad right away.
+
+There are a few options that you can configure to specify exactly what errors will be reported.  Take a look at the [Configuration guide][config] to learn about them.
+
+[config]: /config/

docs/wiki/troubleshooting/index.mdown

+Troubleshooting
+===============
+
+If things don't go smoothly, the first thing to do is run the tests to see if they can determine what's wrong.  To enable the tests you'll need to add django-hoptoad to your `INSTALLED_APPS` setting:
+
+    INSTALLED_APPS = (
+        # ... other apps ...
+        'hoptoad',
+        # ... other apps ...
+    )
+
+Once you've done that you can run the unit tests:
+
+    python manage.py test hoptoad
+
+**NOTE**: The unit tests are very simple at the moment.  I'm working on more, but please feel free to submit ideas (or better yet: patches).
+from django.conf import settings
+
+
+__version__ = 0.3
+VERSION = __version__
+NAME = "django-hoptoad"
+URL = "http://bitbucket.org/sjl/django-hoptoad"
+
+
+def get_hoptoad_settings():
+    hoptoad_settings = getattr(settings, "HOPTOAD_SETTINGS", None)
+
+    if not hoptoad_settings:
+        # do some backward compatibility work to combine all hoptoad
+        # settings in a dictionary
+        hoptoad_settings = {}
+        # for every attribute that starts with hoptoad
+        for attr in itertools.ifilter(lambda x: x.startswith('HOPTOAD'),
+                                      dir(settings)):
+            hoptoad_settings[attr] = getattr(settings, attr)
+
+        if not hoptoad_settings:
+            # there were no settings for hoptoad at all..
+            # should probably log here
+            raise MiddlewareNotUsed
+
+    return hoptoad_settings

hoptoad/api/__init__.py

Empty file added.
+import traceback
+import urllib2
+import yaml
+
+from django.views.debug import get_safe_settings
+from django.conf import settings
+
+
+def _parse_environment(request):
+    """Return an environment mapping for a notification from the given request."""
+    env = dict( (str(k), str(v)) for (k, v) in get_safe_settings().items() )
+    env.update( dict( (str(k), str(v)) for (k, v) in request.META.items() ) )
+
+    env['REQUEST_URI'] = request.build_absolute_uri()
+
+    return env
+
+def _parse_traceback(trace):
+    """Return the given traceback string formatted for a notification."""
+    p_traceback = [ "%s:%d:in `%s'" % (filename, lineno, funcname)
+                    for filename, lineno, funcname, _
+                    in traceback.extract_tb(trace) ]
+    p_traceback.reverse()
+
+    return p_traceback
+
+def _parse_message(exc):
+    """Return a message for a notification from the given exception."""
+    return '%s: %s' % (exc.__class__.__name__, str(exc))
+
+def _parse_request(request):
+    """Return a request mapping for a notification from the given request."""
+    request_get = dict( (str(k), str(v)) for (k, v) in request.GET.items() )
+    request_post = dict( (str(k), str(v)) for (k, v) in request.POST.items() )
+    return request_post if request_post else request_get
+
+def _parse_session(session):
+    """Return a request mapping for a notification from the given session."""
+    return dict( (str(k), str(v)) for (k, v) in session.items() )
+
+
+def _generate_payload(request, exc=None, trace=None, message=None, error_class=None):
+    """Generate a YAML payload for a Hoptoad notification.
+
+    Parameters:
+    request -- A Django HTTPRequest.  This is required.
+
+    Keyword parameters:
+    exc -- A Python Exception object.  If this is not given the
+           mess parameter must be.
+    trace -- A Python Traceback object.  This is not required.
+    message -- A string representing the error message.  If this is not
+               given, the exc parameter must be.
+    error_class -- A string representing the error class.  If this is not
+                   given the excc parameter must be.
+    """
+    p_message = message if message else _parse_message(exc)
+    p_error_class = error_class if error_class else exc.__class__.__name__
+    p_traceback = _parse_traceback(trace) if trace else []
+    p_environment = _parse_environment(request)
+    p_request = _parse_request(request)
+    p_session = _parse_session(request.session)
+
+    return yaml.dump({ 'notice': {
+        'api_key':       settings.HOPTOAD_API_KEY,
+        'error_class':   p_error_class,
+        'error_message': p_message,
+        'backtrace':     p_traceback,
+        'request':       { 'url': request.build_absolute_uri(),
+                           'params': p_request },
+        'session':       { 'key': '', 'data': p_session },
+        'environment':   p_environment,
+    }}, default_flow_style=False)
+
+def _ride_the_toad(payload, timeout):
+    """Send a notification (an HTTP POST request) to Hoptoad.
+
+    Parameters:
+    payload -- the YAML payload for the request from _generate_payload()
+    timeout -- the maximum timeout, in seconds, or None to use the default
+    """
+    headers = { 'Content-Type': 'application/x-yaml',
+                'Accept': 'text/xml, application/xml', }
+    r = urllib2.Request('http://hoptoadapp.com/notices', payload, headers)
+    try:
+        if timeout:
+            urllib2.urlopen(r, timeout=timeout)
+        else:
+            urllib2.urlopen(r)
+    except urllib2.URLError:
+        pass
+
+def report(payload, timeout):
+    return _ride_the_toad(payload, timeout)
+import sys
+import traceback
+import urllib2
+import yaml
+from xml.dom.minidom import getDOMImplementation
+
+from django.views.debug import get_safe_settings
+from django.conf import settings
+
+from hoptoad import VERSION, NAME, URL
+from hoptoad import get_hoptoad_settings
+from hoptoad.api.htv1 import _parse_environment, _parse_request, _parse_session
+from hoptoad.api.htv1 import _parse_message
+
+def _class_name(class_):
+    return class_.__class__.__name__
+
+def _handle_errors(request, response, exc):
+    if response:
+        code = "Http%s" % response
+        msg = "%(code)s: %(response)s at %(uri)s" % {
+                   'code' : code,
+                   'response' : {'Http403' : "Forbidden",
+                                 'Http404' : "Page not found"}[code],
+                   'uri' : request.build_absolute_uri()
+                }
+        return (code, msg)
+
+    excc, inst = sys.exc_info()[:2]
+    if exc:
+        excc = exc
+    return _class_name(excc), _parse_message(excc)
+
+
+def generate_payload(request, response=None, exc=None):
+    """Generate an XML payload for a Hoptoad notification.
+
+    Parameters:
+    request -- A Django HTTPRequest.
+
+    """
+    hoptoad_settings = get_hoptoad_settings()
+    p_error_class, p_message = _handle_errors(request, response, exc)
+
+    # api v2 from: http://help.hoptoadapp.com/faqs/api-2/notifier-api-v2
+    xdoc = getDOMImplementation().createDocument(None, "notice", None)
+    notice = xdoc.firstChild
+
+    # /notice/@version -- should be 2.0
+    notice.setAttribute('version', '2.0')
+
+    # /notice/api-key
+    api_key = xdoc.createElement('api-key')
+    api_key_data = xdoc.createTextNode(hoptoad_settings['HOPTOAD_API_KEY'])
+    api_key.appendChild(api_key_data)
+    notice.appendChild(api_key)
+
+    # /notice/notifier/name
+    # /notice/notifier/version
+    # /notice/notifier/url
+    notifier = xdoc.createElement('notifier')
+    for key, value in zip(["name", "version", "url"], [NAME, VERSION, URL]):
+        key = xdoc.createElement(key)
+        value = xdoc.createTextNode(str(value))
+        key.appendChild(value)
+        notifier.appendChild(key)
+    notice.appendChild(notifier)
+
+    # /notice/error/class
+    # /notice/error/message
+    error = xdoc.createElement('error')
+    for key, value in zip(["class", "message"], [p_error_class, p_message]):
+        key = xdoc.createElement(key)
+        value = xdoc.createTextNode(value)
+        key.appendChild(value)
+        error.appendChild(key)
+
+    # /notice/error/backtrace/error/line
+    backtrace = xdoc.createElement('backtrace')
+    # i do this here because I'm afraid of circular reference..
+    reversed_backtrace = reversed(traceback.extract_tb(sys.exc_info()[2]))
+    for filename, lineno, funcname, text in reversed_backtrace:
+        line = xdoc.createElement('line')
+        line.setAttribute('file', str(filename))
+        line.setAttribute('number', str(lineno))
+        line.setAttribute('method', str(funcname))
+        backtrace.appendChild(line)
+    error.appendChild(backtrace)
+    notice.appendChild(error)
+
+    # /notice/request
+    xrequest = xdoc.createElement('request')
+
+    # /notice/request/url -- request.build_absolute_uri()
+    xurl = xdoc.createElement('url')
+    xurl_data = xdoc.createTextNode(request.build_absolute_uri())
+    xurl.appendChild(xurl_data)
+    xrequest.appendChild(xurl)
+
+    # /notice/request/component -- not sure..
+    comp = xdoc.createElement('component')
+    #comp_data = xdoc.createTextNode('')
+    xrequest.appendChild(comp)
+
+    # /notice/request/action -- action which error occured
+    # ... no fucking clue..
+    action = xdoc.createElement('action') # maybe GET/POST??
+    action_data = u"%s %s" % (request.method, request.META['PATH_INFO'])
+    action_data = xdoc.createTextNode(action_data)
+    action.appendChild(action_data)
+    xrequest.appendChild(action)
+
+    # /notice/request/params/var -- check request.GET/request.POST
+    params = xdoc.createElement('params')
+    for key, value in _parse_request(request).iteritems():
+        var = xdoc.createElement('var')
+        var.setAttribute('key', key)
+        value = xdoc.createTextNode(str(value))
+        var.appendChild(value)
+        params.appendChild(var)
+    xrequest.appendChild(params)
+
+    # /notice/request/session/var -- check if sessions is enabled..
+    sessions = xdoc.createElement('session')
+    for key, value in _parse_session(request.session).iteritems():
+        var = xdoc.createElement('var')
+        var.setAttribute('key', key)
+        value = xdoc.createTextNode(str(value))
+        var.appendChild(value)
+        sessions.appendChild(var)
+    xrequest.appendChild(params)
+
+    # /notice/request/cgi-data/var -- all meta data
+    cgidata = xdoc.createElement('cgi-data')
+    for key, value in _parse_environment(request).iteritems():
+        var = xdoc.createElement('var')
+        var.setAttribute('key', key)
+        value = xdoc.createTextNode(str(value))
+        var.appendChild(value)
+        cgidata.appendChild(var)
+    xrequest.appendChild(cgidata)
+    notice.appendChild(xrequest)
+
+    serverenv = xdoc.createElement('server-environment')
+    # /notice/server-environment/project-root -- default to sys.path[0] 
+    projectroot = xdoc.createElement('project-root')
+    projectroot.appendChild(xdoc.createTextNode(sys.path[0]))
+    serverenv.appendChild(projectroot)
+    # /notice/server-environment/environment-name -- environment name? wtf..
+    envname = xdoc.createElement('environment-name')
+    # no idea...
+    # envname.appendChild(xdoc.createTextNode())
+    serverenv.appendChild(envname)
+    notice.appendChild(serverenv)
+
+    return xdoc.toxml('utf-8')
+
+def _ride_the_toad(payload, timeout, use_ssl):
+    """Send a notification (an HTTP POST request) to Hoptoad.
+
+    Parameters:
+    payload -- the YAML payload for the request from _generate_payload()
+    timeout -- the maximum timeout, in seconds, or None to use the default
+
+    """
+    headers = { 'Content-Type': 'text/xml' }
+
+    # url calculation
+    url_template = '%s://hoptoadapp.com/notifier_api/v2/notices'
+    notification_url = url_template % ("https" if use_ssl else "http")
+    # allow the settings to override all urls
+    notification_url = get_hoptoad_settings().get('HOPTOAD_NOTIFICATION_URL',
+                                                   notification_url)
+
+    r = urllib2.Request(notification_url, payload, headers)
+    try:
+        if timeout:
+            # timeout is 2.6 addition!!
+            response = urllib2.urlopen(r, timeout=timeout)
+        else:
+            response = urllib2.urlopen(r)
+    except urllib2.URLError:
+        pass
+    else:
+        # getcode is 2.6 addition!!
+        status = response.getcode()
+
+        if status == 403:
+            # if we can not use SSL, re-invoke w/o using SSL
+            _ride_the_toad(payload, timeout, use_ssl=False)
+        if status == 422:
+            # couldn't send to hoptoad..
+            pass
+        if status == 500:
+            # hoptoad is down
+            pass
+
+def report(payload, timeout):
+    use_ssl = get_hoptoad_settings().get('HOPTOAD_USE_SSL', False)
+    return _ride_the_toad(payload, timeout, use_ssl)

hoptoad/handlers/__init__.py

+"""Implementations of different handlers that communicate with hoptoad in
+various different protocols.
+"""
+import logging
+
+from hoptoad import get_hoptoad_settings
+from hoptoad.handlers.threaded import ThreadedNotifier
+
+logger = logging.getLogger(__name__)
+
+
+def get_handler(*args, **kwargs):
+    """Returns an initialized handler object"""
+    hoptoad_settings = get_hoptoad_settings()
+    handler = hoptoad_settings.get("HOPTOAD_HANDLER", "threadpool")
+    if handler.lower() == 'threadpool':
+        threads = hoptoad_settings.get("HOPTOAD_THREAD_COUNT", 4)
+        return ThreadedNotifier(threads , *args, **kwargs)

hoptoad/handlers/threaded.py

+import os
+import threading
+import time
+import logging
+
+from hoptoad.api import htv2
+
+from hoptoad.handlers.utils.threadpool import WorkRequest, ThreadPool
+from hoptoad.handlers.utils.threadpool import NoResultsPending
+
+
+logger = logging.getLogger(__name__)
+
+
+def _exception_handler(request, exc_info):
+    """Rudimentary exception handler, simply log and moves on.
+
+    If there's no tuple, it means something went really wrong. Critically log
+    and exit.
+
+    """
+    if not isinstance(exc_info, tuple):
+        logger.critical(str(request))
+        logger.critical(str(exc_info))
+        sys.exit(1)
+    logger.warn(
+        "* Exception occured in request #%s: %s" % (request.requestID, exc_info)
+    )
+
+
+class ThreadedNotifier(threading.Thread):
+    """A daemon thread that spawns a threadpool of worker threads.
+
+    Waits for queue additions through the enqueue method.
+
+    """
+    def __init__(self, threadpool_threadcount, cb=None, exc_cb=None):
+        _threadname = "Hoptoad%s-%d" % (self.__class__.__name__, os.getpid())
+        threading.Thread.__init__(self, name=_threadname)
+        self.threads = threadpool_threadcount
+        self.daemon = True # daemon thread... important!
+        self.callback = cb
+        self.exc_callback = exc_cb or _exception_handler
+        self.pool = ThreadPool(self.threads)
+        # start the thread pool
+        self.start()
+
+    def enqueue(self, payload, timeout):
+        request = WorkRequest(
+            htv2.report,
+            args=(payload, timeout),
+            callback=self.callback,
+            exc_callback=self.exc_callback
+        )
+
+        # Put the request into the queue where the detached 'run' method will
+        # poll its queue every 0.5 seconds and start working.
+        self.pool.putRequest(request)
+
+    def run(self):
+        """Actively poll the queue for requests and process them."""
+        while True:
+            try:
+                time.sleep(0.5) # TODO: configure for tuning
+                self.pool.poll()
+            except KeyboardInterrupt:
+                logger.info("* Interrupted!")
+                break
+            except NoResultsPending:
+                pass
+
+

hoptoad/handlers/utils/__init__.py

Empty file added.

hoptoad/handlers/utils/threadpool.py

+# -*- coding: UTF-8 -*-
+"""Easy to use object-oriented thread pool framework.
+
+A thread pool is an object that maintains a pool of worker threads to perform
+time consuming operations in parallel. It assigns jobs to the threads
+by putting them in a work request queue, where they are picked up by the
+next available thread. This then performs the requested operation in the
+background and puts the results in another queue.
+
+The thread pool object can then collect the results from all threads from
+this queue as soon as they become available or after all threads have
+finished their work. It's also possible, to define callbacks to handle
+each result as it comes in.
+
+The basic concept and some code was taken from the book "Python in a Nutshell,
+2nd edition" by Alex Martelli, O'Reilly 2006, ISBN 0-596-10046-9, from section
+14.5 "Threaded Program Architecture". I wrapped the main program logic in the
+ThreadPool class, added the WorkRequest class and the callback system and
+tweaked the code here and there. Kudos also to Florent Aide for the exception
+handling mechanism.
+
+Basic usage::
+
+    >>> pool = ThreadPool(poolsize)
+    >>> requests = makeRequests(some_callable, list_of_args, callback)
+    >>> [pool.putRequest(req) for req in requests]
+    >>> pool.wait()
+
+See the end of the module code for a brief, annotated usage example.
+
+Website : http://chrisarndt.de/projects/threadpool/
+
+"""
+__docformat__ = "restructuredtext en"
+
+__all__ = [
+    'makeRequests',
+    'NoResultsPending',
+    'NoWorkersAvailable',
+    'ThreadPool',
+    'WorkRequest',
+    'WorkerThread'
+]
+
+__author__ = "Christopher Arndt"
+__version__ = '1.2.7'
+__revision__ = "$Revision: 416 $"
+__date__ = "$Date: 2009-10-07 05:41:27 +0200 (Wed, 07 Oct 2009) $"
+__license__ = "MIT license"
+
+
+# standard library modules
+import sys
+import threading
+import Queue
+import traceback
+
+
+# exceptions
+class NoResultsPending(Exception):
+    """All work requests have been processed."""
+    pass
+
+class NoWorkersAvailable(Exception):
+    """No worker threads available to process remaining requests."""
+    pass
+
+
+# internal module helper functions
+def _handle_thread_exception(request, exc_info):
+    """Default exception handler callback function.
+
+    This just prints the exception info via ``traceback.print_exception``.
+
+    """
+    traceback.print_exception(*exc_info)
+
+
+# utility functions
+def makeRequests(callable_, args_list, callback=None,
+        exc_callback=_handle_thread_exception):
+    """Create several work requests for same callable with different arguments.
+
+    Convenience function for creating several work requests for the same
+    callable where each invocation of the callable receives different values
+    for its arguments.
+
+    ``args_list`` contains the parameters for each invocation of callable.
+    Each item in ``args_list`` should be either a 2-item tuple of the list of
+    positional arguments and a dictionary of keyword arguments or a single,
+    non-tuple argument.
+
+    See docstring for ``WorkRequest`` for info on ``callback`` and
+    ``exc_callback``.
+
+    """
+    requests = []
+    for item in args_list:
+        if isinstance(item, tuple):
+            requests.append(
+                WorkRequest(callable_, item[0], item[1], callback=callback,
+                    exc_callback=exc_callback)
+            )
+        else:
+            requests.append(
+                WorkRequest(callable_, [item], None, callback=callback,
+                    exc_callback=exc_callback)
+            )
+    return requests
+
+
+# classes
+class WorkerThread(threading.Thread):
+    """Background thread connected to the requests/results queues.
+
+    A worker thread sits in the background and picks up work requests from
+    one queue and puts the results in another until it is dismissed.
+
+    """
+
+    def __init__(self, requests_queue, results_queue, poll_timeout=5, **kwds):
+        """Set up thread in daemonic mode and start it immediatedly.
+
+        ``requests_queue`` and ``results_queue`` are instances of
+        ``Queue.Queue`` passed by the ``ThreadPool`` class when it creates a new
+        worker thread.
+
+        """
+        threading.Thread.__init__(self, **kwds)
+        self.setDaemon(1)
+        self._requests_queue = requests_queue
+        self._results_queue = results_queue
+        self._poll_timeout = poll_timeout
+        self._dismissed = threading.Event()
+        self.start()
+
+    def run(self):
+        """Repeatedly process the job queue until told to exit."""
+        while True:
+            if self._dismissed.isSet():
+                # we are dismissed, break out of loop
+                break
+            # get next work request. If we don't get a new request from the
+            # queue after self._poll_timout seconds, we jump to the start of
+            # the while loop again, to give the thread a chance to exit.
+            try:
+                request = self._requests_queue.get(True, self._poll_timeout)
+            except Queue.Empty:
+                continue
+            else:
+                if self._dismissed.isSet():
+                    # we are dismissed, put back request in queue and exit loop
+                    self._requests_queue.put(request)
+                    break
+                try:
+                    result = request.callable(*request.args, **request.kwds)
+                    self._results_queue.put((request, result))
+                except:
+                    request.exception = True
+                    self._results_queue.put((request, sys.exc_info()))
+
+    def dismiss(self):
+        """Sets a flag to tell the thread to exit when done with current job."""
+        self._dismissed.set()
+
+
+class WorkRequest:
+    """A request to execute a callable for putting in the request queue later.
+
+    See the module function ``makeRequests`` for the common case
+    where you want to build several ``WorkRequest`` objects for the same
+    callable but with different arguments for each call.
+
+    """
+
+    def __init__(self, callable_, args=None, kwds=None, requestID=None,
+            callback=None, exc_callback=_handle_thread_exception):
+        """Create a work request for a callable and attach callbacks.
+
+        A work request consists of the a callable to be executed by a
+        worker thread, a list of positional arguments, a dictionary
+        of keyword arguments.
+
+        A ``callback`` function can be specified, that is called when the
+        results of the request are picked up from the result queue. It must
+        accept two anonymous arguments, the ``WorkRequest`` object and the
+        results of the callable, in that order. If you want to pass additional
+        information to the callback, just stick it on the request object.
+
+        You can also give custom callback for when an exception occurs with
+        the ``exc_callback`` keyword parameter. It should also accept two
+        anonymous arguments, the ``WorkRequest`` and a tuple with the exception
+        details as returned by ``sys.exc_info()``. The default implementation
+        of this callback just prints the exception info via
+        ``traceback.print_exception``. If you want no exception handler
+        callback, just pass in ``None``.
+
+        ``requestID``, if given, must be hashable since it is used by
+        ``ThreadPool`` object to store the results of that work request in a
+        dictionary. It defaults to the return value of ``id(self)``.
+
+        """
+        if requestID is None:
+            self.requestID = id(self)
+        else:
+            try:
+                self.requestID = hash(requestID)
+            except TypeError:
+                raise TypeError("requestID must be hashable.")
+        self.exception = False
+        self.callback = callback
+        self.exc_callback = exc_callback
+        self.callable = callable_
+        self.args = args or []
+        self.kwds = kwds or {}
+
+    def __str__(self):
+        return "<WorkRequest id=%s args=%r kwargs=%r exception=%s>" % \
+            (self.requestID, self.args, self.kwds, self.exception)
+
+class ThreadPool:
+    """A thread pool, distributing work requests and collecting results.
+
+    See the module docstring for more information.
+
+    """
+
+    def __init__(self, num_workers, q_size=0, resq_size=0, poll_timeout=5):
+        """Set up the thread pool and start num_workers worker threads.
+
+        ``num_workers`` is the number of worker threads to start initially.
+
+        If ``q_size > 0`` the size of the work *request queue* is limited and
+        the thread pool blocks when the queue is full and it tries to put
+        more work requests in it (see ``putRequest`` method), unless you also
+        use a positive ``timeout`` value for ``putRequest``.
+
+        If ``resq_size > 0`` the size of the *results queue* is limited and the
+        worker threads will block when the queue is full and they try to put
+        new results in it.
+
+        .. warning:
+            If you set both ``q_size`` and ``resq_size`` to ``!= 0`` there is
+            the possibilty of a deadlock, when the results queue is not pulled
+            regularly and too many jobs are put in the work requests queue.
+            To prevent this, always set ``timeout > 0`` when calling
+            ``ThreadPool.putRequest()`` and catch ``Queue.Full`` exceptions.
+
+        """
+        self._requests_queue = Queue.Queue(q_size)
+        self._results_queue = Queue.Queue(resq_size)
+        self.workers = []
+        self.dismissedWorkers = []
+        self.workRequests = {}
+        self.createWorkers(num_workers, poll_timeout)
+
+    def createWorkers(self, num_workers, poll_timeout=5):
+        """Add num_workers worker threads to the pool.
+
+        ``poll_timout`` sets the interval in seconds (int or float) for how
+        ofte threads should check whether they are dismissed, while waiting for
+        requests.
+
+        """
+        for i in range(num_workers):
+            self.workers.append(WorkerThread(self._requests_queue,
+                self._results_queue, poll_timeout=poll_timeout))
+
+    def dismissWorkers(self, num_workers, do_join=False):
+        """Tell num_workers worker threads to quit after their current task."""
+        dismiss_list = []
+        for i in range(min(num_workers, len(self.workers))):
+            worker = self.workers.pop()
+            worker.dismiss()
+            dismiss_list.append(worker)
+
+        if do_join:
+            for worker in dismiss_list:
+                worker.join()
+        else:
+            self.dismissedWorkers.extend(dismiss_list)
+
+    def joinAllDismissedWorkers(self):
+        """Perform Thread.join() on all worker threads that have been dismissed.
+        """
+        for worker in self.dismissedWorkers:
+            worker.join()
+        self.dismissedWorkers = []
+
+    def putRequest(self, request, block=True, timeout=None):
+        """Put work request into work queue and save its id for later."""
+        assert isinstance(request, WorkRequest)
+        # don't reuse old work requests
+        assert not getattr(request, 'exception', None)
+        self._requests_queue.put(request, block, timeout)
+        self.workRequests[request.requestID] = request
+
+    def poll(self, block=False):
+        """Process any new results in the queue."""
+        while True:
+            # still results pending?
+            if not self.workRequests:
+                raise NoResultsPending
+            # are there still workers to process remaining requests?
+            elif block and not self.workers:
+                raise NoWorkersAvailable
+            try:
+                # get back next results
+                request, result = self._results_queue.get(block=block)
+                # has an exception occured?
+                if request.exception and request.exc_callback:
+                    request.exc_callback(request, result)
+                # hand results to callback, if any
+                if request.callback and not \
+                       (request.exception and request.exc_callback):
+                    request.callback(request, result)
+                del self.workRequests[request.requestID]
+            except Queue.Empty:
+                break
+
+    def wait(self):
+        """Wait for results, blocking until all have arrived."""
+        while 1:
+            try:
+                self.poll(True)
+            except NoResultsPending:
+                break
+
+
+################
+# USAGE EXAMPLE
+################
+
+if __name__ == '__main__':
+    import random
+    import time
+
+    # the work the threads will have to do (rather trivial in our example)
+    def do_something(data):
+        time.sleep(random.randint(1,5))
+        result = round(random.random() * data, 5)
+        # just to show off, we throw an exception once in a while
+        if result > 5:
+            raise RuntimeError("Something extraordinary happened!")
+        return result
+
+    # this will be called each time a result is available
+    def print_result(request, result):
+        print "**** Result from request #%s: %r" % (request.requestID, result)
+
+    # this will be called when an exception occurs within a thread
+    # this example exception handler does little more than the default handler
+    def handle_exception(request, exc_info):
+        if not isinstance(exc_info, tuple):
+            # Something is seriously wrong...
+            print request
+            print exc_info
+            raise SystemExit
+        print "**** Exception occured in request #%s: %s" % \
+          (request.requestID, exc_info)
+
+    # assemble the arguments for each job to a list...
+    data = [random.randint(1,10) for i in range(20)]
+    # ... and build a WorkRequest object for each item in data
+    requests = makeRequests(do_something, data, print_result, handle_exception)
+    # to use the default exception handler, uncomment next line and comment out
+    # the preceding one.
+    #requests = makeRequests(do_something, data, print_result)
+
+    # or the other form of args_lists accepted by makeRequests: ((,), {})
+    data = [((random.randint(1,10),), {}) for i in range(20)]
+    requests.extend(
+        makeRequests(do_something, data, print_result, handle_exception)
+        #makeRequests(do_something, data, print_result)
+        # to use the default exception handler, uncomment next line and comment
+        # out the preceding one.
+    )
+
+    # we create a pool of 3 worker threads
+    print "Creating thread pool with 3 worker threads."
+    main = ThreadPool(3)
+
+    # then we put the work requests in the queue...
+    for req in requests:
+        main.putRequest(req)
+        print "Work request #%s added." % req.requestID
+    # or shorter:
+    # [main.putRequest(req) for req in requests]
+
+    # ...and wait for the results to arrive in the result queue
+    # by using ThreadPool.wait(). This would block until results for
+    # all work requests have arrived:
+    # main.wait()
+
+    # instead we can poll for results while doing something else:
+    i = 0
+    while True:
+        try:
+            time.sleep(0.5)
+            main.poll()
+            print "Main thread working...",
+            print "(active worker threads: %i)" % (threading.activeCount()-1, )
+            if i == 10:
+                print "**** Adding 3 more worker threads..."
+                main.createWorkers(3)
+            if i == 20:
+                print "**** Dismissing 2 worker threads..."
+                main.dismissWorkers(2)
+            i += 1
+        except KeyboardInterrupt:
+            print "**** Interrupted!"
+            break
+        except NoResultsPending:
+            print "**** No pending results."
+            break
+    if main.dismissedWorkers:
+        print "Joining all dismissed worker threads..."
+        main.joinAllDismissedWorkers()
+

hoptoad/middleware.py

-import sys
-import traceback
-import urllib2
-import yaml
 import re
-import os
-import threading
 import logging
-import time
-
-from threadpool import WorkRequest, ThreadPool
-from threadpool import NoResultsPending
+import itertools
 
 from django.core.exceptions import MiddlewareNotUsed
-from django.views.debug import get_safe_settings
 from django.conf import settings
 
+from hoptoad import get_hoptoad_settings
+from hoptoad.handlers import get_handler
+from hoptoad.api import htv2
+
 
 logger = logging.getLogger(__name__)
 
-def _parse_environment(request):
-    """Return an environment mapping for a notification from the given request."""
-    env = dict( (str(k), str(v)) for (k, v) in get_safe_settings().items() )
-    env.update( dict( (str(k), str(v)) for (k, v) in request.META.items() ) )
-    
-    env['REQUEST_URI'] = request.build_absolute_uri()
-    
-    return env
-
-def _parse_traceback(trace):
-    """Return the given traceback string formatted for a notification."""
-    p_traceback = [ "%s:%d:in `%s'" % (filename, lineno, funcname) 
-                    for filename, lineno, funcname, _
-                    in traceback.extract_tb(trace) ]
-    p_traceback.reverse()
-    
-    return p_traceback
-
-def _parse_message(exc):
-    """Return a message for a notification from the given exception."""
-    return '%s: %s' % (exc.__class__.__name__, str(exc))
-
-def _parse_request(request):
-    """Return a request mapping for a notification from the given request."""
-    request_get = dict( (str(k), str(v)) for (k, v) in request.GET.items() )
-    request_post = dict( (str(k), str(v)) for (k, v) in request.POST.items() )
-    return request_post if request_post else request_get
-
-def _parse_session(session):
-    """Return a request mapping for a notification from the given session."""
-    return dict( (str(k), str(v)) for (k, v) in session.items() )
-
-
-def _generate_payload(request, exc=None, trace=None, message=None, error_class=None):
-    """Generate a YAML payload for a Hoptoad notification.
-    
-    Parameters:
-    request -- A Django HTTPRequest.  This is required.
-    
-    Keyword parameters:
-    exc -- A Python Exception object.  If this is not given the 
-           mess parameter must be.
-    trace -- A Python Traceback object.  This is not required.
-    message -- A string representing the error message.  If this is not
-               given, the exc parameter must be.
-    error_class -- A string representing the error class.  If this is not
-                   given the excc parameter must be.
-    """
-    p_message = message if message else _parse_message(exc)
-    p_error_class = error_class if error_class else exc.__class__.__name__
-    p_traceback = _parse_traceback(trace) if trace else []
-    p_environment = _parse_environment(request)
-    p_request = _parse_request(request)
-    p_session = _parse_session(request.session)
-    
-    return yaml.dump({ 'notice': {
-        'api_key':       settings.HOPTOAD_API_KEY,
-        'error_class':   p_error_class,
-        'error_message': p_message,
-        'backtrace':     p_traceback,
-        'request':       { 'url': request.build_absolute_uri(),
-                           'params': p_request },
-        'session':       { 'key': '', 'data': p_session },
-        'environment':   p_environment,
-    }}, default_flow_style=False)
-
-def _ride_the_toad(payload, timeout):
-    """Send a notification (an HTTP POST request) to Hoptoad.
-    
-    Parameters:
-    payload -- the YAML payload for the request from _generate_payload()
-    timeout -- the maximum timeout, in seconds, or None to use the default
-    """
-    headers = { 'Content-Type': 'application/x-yaml', 
-                'Accept': 'text/xml, application/xml', }
-    r = urllib2.Request('http://hoptoadapp.com/notices', payload, headers)
-    try:
-        if timeout:
-            urllib2.urlopen(r, timeout=timeout)
-        else:
-            urllib2.urlopen(r)
-    except urllib2.URLError:
-        pass
-
-def _exception_handler(request, exc_info):
-    """Rudimentary exception handler, simply logs and moves on. If there's no
-    tuple, it means something went really wrong. Critically log and exit."""
-    if not isinstance(exc_info, tuple):
-        # Something is seriously wrong...
-        logger.critical(str(request))
-        logger.critical(str(exc_info))
-        sys.exit(1)
-    logger.warn("**** Exception occured in request #%s: %s" % \
-                (request.requestID, exc_info))
-
-
-class Runnable(threading.Thread):
-    """A daemon thread that spawns a threadpool of worker threads and waits
-    for queue additions through the enqueue method. 
-    
-    #TODO: Consider using asyncore instead of a threadpool
-    """
-    def __init__(self, threadpool_threadcount):
-        threading.Thread.__init__(self,
-                                  name="HoptoadThreadRunner-%d" % os.getpid()) 
-
-        self.threads = threadpool_threadcount
-        self.daemon = True #daemon thread..important!
-        self.pool = ThreadPool(self.threads)
-
-    def enqueue(self, 
-                payload, 
-                timeout, 
-                callback=None, 
-                exc_callback=_exception_handler):
-        #create the worker request
-        request = WorkRequest(_ride_the_toad, 
-                              args=(payload, timeout), 
-                              callback=callback,
-                              exc_callback=exc_callback)
-        #put the request into the queue where the detached 'run' method will
-        #poll its queue every 0.5 seconds and start working
-        self.pool.putRequest(request)
-
-    def run(self):
-        """Executed when .start() is invoked on Runnable. Actively polls the
-        queue for requests and processes them.
-        """
-        while True:
-            try:
-                time.sleep(0.5) #TODO: configure for tuning
-                self.pool.poll()
-            except KeyboardInterrupt:
-                logger.info("***** Interrupted!")
-                break
-            except NoResultsPending:
-                pass
 
 class HoptoadNotifierMiddleware(object):
     def __init__(self):
         """Initialize the middleware."""
-        all_settings = settings.get_all_members()
-        
-        if 'HOPTOAD_API_KEY' not in all_settings or not settings.HOPTOAD_API_KEY:
+
+        hoptoad_settings = get_hoptoad_settings()
+        self._init_middleware(hoptoad_settings)
+
+    def _init_middleware(self, hoptoad_settings):
+
+        if 'HOPTOAD_API_KEY' not in hoptoad_settings:
+            # no api key, abort!
             raise MiddlewareNotUsed
-        
-        if settings.DEBUG and \
-           (not 'HOPTOAD_NOTIFY_WHILE_DEBUG' in all_settings
-            or not settings.HOPTOAD_NOTIFY_WHILE_DEBUG ):
-            raise MiddlewareNotUsed
-        
-        self.timeout = ( settings.HOPTOAD_TIMEOUT 
-                         if 'HOPTOAD_TIMEOUT' in all_settings else None )
-        
-        self.notify_404 = ( settings.HOPTOAD_NOTIFY_404 
-                            if 'HOPTOAD_NOTIFY_404' in all_settings else False )
-        self.notify_403 = ( settings.HOPTOAD_NOTIFY_403 
-                            if 'HOPTOAD_NOTIFY_403' in all_settings else False )
-        self.ignore_agents = ( map(re.compile, settings.HOPTOAD_IGNORE_AGENTS)
-                            if 'HOPTOAD_IGNORE_AGENTS' in all_settings else [] )
-            
-        #creates a self.thread attribute and starts it
-        self.initialize_threadpool(all_settings)
-  
+
+        if settings.DEBUG:
+            if not hoptoad_settings.get('HOPTOAD_NOTIFY_WHILE_DEBUG', None):
+                # do not use hoptoad if you're in debug mode..
+                raise MiddlewareNotUsed
+
+        self.timeout = hoptoad_settings.get('HOPTOAD_TIMEOUT', None)
+        self.notify_404 = hoptoad_settings.get('HOPTOAD_NOTIFY_404', False)
+        self.notify_403 = hoptoad_settings.get('HOPTOAD_NOTIFY_403', False)
+
+        ignorable_agents = hoptoad_settings.get('HOPTOAD_IGNORE_AGENTS', [])
+        self.ignore_agents = map(re.compile, ignorable_agents)
+
+        self.handler = get_handler()
+
     def _ignore(self, request):
         """Return True if the given request should be ignored, False otherwise."""
         ua = request.META.get('HTTP_USER_AGENT', '')
         return any(i.search(ua) for i in self.ignore_agents)
-        
-    def initialize_threadpool(self, all_settings):
-        """Initialize an internal threadpool for allowing asynchronous POST
-        requests to hoptoad. Create a thread attribute and start the threadpool
-        """
-
-        if 'HOPTOAD_THREAD_COUNT' in all_settings:
-            threads = settings.HOPTOAD_THREAD_COUNT
-        else:
-            threads = 4
-
-        self.thread = Runnable(threads) 
-        self.thread.start()
 
     def process_response(self, request, response):
         """Process a reponse object.
-        
+
         Hoptoad will be notified of a 404 error if the response is a 404
         and 404 tracking is enabled in the settings.
-        
+
         Hoptoad will be notified of a 403 error if the response is a 403
         and 403 tracking is enabled in the settings.
-        
+
         Regardless of whether Hoptoad is notified, the reponse object will
         be returned unchanged.
+
         """
         if self._ignore(request):
             return response
-        
-        if self.notify_404 and response.status_code == 404:
-            error_class = 'Http404'
-            
-            message = 'Http404: Page not found at %s' % request.build_absolute_uri()
-            payload = _generate_payload(request, error_class=error_class, message=message)
-            
-            #_ride_the_toad(payload, self.timeout)
-            self.thread.enqueue(payload, self.timeout)
-        
-        if self.notify_403 and response.status_code == 403:
-            error_class = 'Http403'
-            
-            message = 'Http403: Forbidden at %s' % request.build_absolute_uri()
-            payload = _generate_payload(request, error_class=error_class, message=message)
-            
-            #_ride_the_toad(payload, self.timeout)
-            self.thread.enqueue(payload, self.timeout)
-        
+
+        sc = response.status_code
+        if sc in [404, 403] and getattr(self, "notify_%d" % sc):
+            self.handler.enqueue(htv2.generate_payload(request, response=sc),
+                                 self.timeout)
+
         return response
-    
+
     def process_exception(self, request, exc):
         """Process an exception.
-        
+
         Hoptoad will be notified of the exception and None will be
         returned so that Django's normal exception handling will then
         be used.
+
         """
         if self._ignore(request):
             return None
-        
-        excc, _, tb = sys.exc_info()
-        
-        payload = _generate_payload(request, exc, tb)
-        #_ride_the_toad(payload, self.timeout)
-        self.thread.enqueue(payload, self.timeout)
-        
+
+        self.handler.enqueue(htv2.generate_payload(request, exc=exc),
+                             self.timeout)
         return None
 
     
     def test_api_key_present(self):
         """Test to make sure an API key is present."""
-        self.assertTrue('HOPTOAD_API_KEY' in settings.get_all_members(),
+        self.assertTrue('HOPTOAD_API_KEY' in dir(settings),
             msg='The HOPTOAD_API_KEY setting is not present.')
         self.assertTrue(settings.HOPTOAD_API_KEY,
             msg='The HOPTOAD_API_KEY setting is blank.')
 
 setup(
     name='django-hoptoad',
-    version='0.1',
+    version='0.2',
     description='django-hoptoad is some simple Middleware for letting Django-driven websites report their errors to Hoptoad.',
-    long_description=open(os.path.join(os.path.abspath(os.path.dirname(__file__)), 'README')).read(),
+    long_description=open(os.path.join(os.path.abspath(os.path.dirname(__file__)), 'README.rst')).read(),
     author='Steve Losh',
     author_email='steve@stevelosh.com',
     url='http://stevelosh.com/projects/django-hoptoad/',
     packages=find_packages(),
-    requires='pyyaml',
+    install_requires=['pyyaml'],
     classifiers=[
         'Development Status :: 4 - Beta',
         'Environment :: Web Environment',