Anonymous avatar Anonymous committed e6d06a5

"Finished" articles from email. Time for a test release.

Comments (0)

Files changed (2)

 
 When that's done, you should be able to begin using ``django-articles``!
 
+Articles From Email
+===================
+
+.. versionadded:: 0.9.1
+   Articles from email
+
+I've been working on making it possible for ``django-articles`` to post articles that you email to a special mailbox.  This seems to be working on the most basic levels right now.  It's not been tested in very many scenarios, and I would appreciate it if you could post problems with it in the ticket tracker at http://bitbucket.org/codekoala/django-articles/ so we can make it work really well.
+
+Things to keep in mind:
+
+* Any **active** user who is a ``django.contrib.auth.models.User`` and has an email address associated with their user information is a valid sender for articles from email.  This is how the author of an article is determined.
+* Only the following fields are currently populated by the articles from email feature:
+
+    * author
+    * title
+    * slug (uniqueness is handled)
+    * content
+    * markup
+    * publish_date
+    * is_active
+
+  Any and all other attributes about an article must be configured later on using the standard mechanisms (aka the Django admin).
+* There is a new management command to handle all of the magic for this feature: ``check_for_articles_from_email``.  This command is intended to be called either manually or via external scheduling utilities (like ``cron``)
+* Email messages **are deleted** after they are turned into articles.  This means that you should probably have a *special mailbox dedicated to django-article and articles from email*.  However, only emails whose sender matches the email address of an active user are deleted (as described above).
+* Attachments are currently not bothered with.  Don't worry, they will be :D
+
+Configuration
+-------------
+
+There are several new variables that you can configure in your ``settings.py`` to enable articles from email:
+
+* ``ARTICLES_EMAIL_PROTOCOL`` - Either ``IMAP4`` or ``POP3``.  *Default*: ``IMAP4``
+* ``ARTICLES_EMAIL_HOST`` - The mail server. *Example*: mail.yourserver.com
+* ``ARTICLES_EMAIL_PORT`` - The port to use to connect to your mail server
+* ``ARTICLES_EMAIL_KEYFILE`` - The keyfile used to access your mail server *untested*
+* ``ARTICLES_EMAIL_CERTFILE`` - The certfile used to access your mail server *untested*
+* ``ARTICLES_EMAIL_USER`` - The username used to access your mailbox
+* ``ARTICLES_EMAIL_PASSWORD`` - The password associated with the user to access your mailbox
+* ``ARTICLES_EMAIL_SSL`` - Whether or not to connect to the mail server using SSL.  *Default*: ``False``
+* ``ARTICLES_EMAIL_AUTOPOST`` - Whether or not to automatically post articles that are created from email messages.  If this is ``False``, the articles will be marked as inactive and you must manually make them active. *Default*: ``False``
+* ``ARTICLES_EMAIL_MARKUP`` - The default markup language to use for articles from email.  Options include:
+
+    * ``h`` for HTML/plain text
+    * ``m`` for Markdown
+    * ``r`` for reStructuredText
+    * ``t`` for Textile
+
+  *Default*: ``h``
+
+* ``ARTICLES_EMAIL_ACK`` - Whether or not to email out an acknowledgment message when articles are created from email.  *Default*: ``False``
+
 Good luck!  Please contact me with any questions or concerns you have with the project!
 

articles/management/commands/check_for_articles_from_email.py

 from django.conf import settings
+from django.contrib.auth.models import User
+from django.contrib.sites.models import Site
 from django.core.management.base import BaseCommand
+from django.template.defaultfilters import slugify
+from django.utils.translation import ugettext_lazy as _
 
+from datetime import datetime
+from email.parser import FeedParser
+from email.utils import parseaddr, parsedate
 from optparse import make_option
+import socket
+import sys
+import time
+
+from django.db.models import connection
+from articles.models import Article, MARKUP_HTML, MARKUP_MARKDOWN, MARKUP_REST, MARKUP_TEXTILE
 
 MB_IMAP4 = 'IMAP4'
 MB_POP3 = 'POP3'
 
 class Command(BaseCommand):
     help = "Checks special e-mail inboxes for emails that should be posted as articles"
+
     option_list = BaseCommand.option_list + (
         make_option('--protocol', dest='protocol', default=MB_IMAP4, help='Protocol to use to check for email'),
         make_option('--host', dest='host', default=None),
         make_option('--ssl', action='store_true', dest='ssl', default=False),
     )
 
+    def log(self, message, level=2):
+        if self.verbosity >= level:
+            print message
+
     def handle(self, *args, **options):
-        protocol = options['protocol']
-        host = options['host']
-        port = options['port']
-        keyfile = options['keyfile']
-        certfile = options['certfile']
-        username = options['username']
-        password = options['password']
-        ssl = options['ssl']
+        """Main entry point for the command"""
 
-        params = (host, port, username, password, keyfile, certfile, ssl)
+        s = lambda k, d: getattr(settings, k, d)
+
+        # retrieve configuration options--give precedence to CLI parameters
+        protocol = options['protocol'] or s('ARTICLES_EMAIL_PROTOCOL', MB_IMAP4)
+        host = options['host'] or s('ARTICLES_EMAIL_HOST', 'mail.yourhost.com')
+        port = options['port'] or s('ARTICLES_EMAIL_PORT', None)
+        keyfile = options['keyfile'] or s('ARTICLES_EMAIL_KEYFILE', None)
+        certfile = options['certfile'] or s('ARTICLES_EMAIL_CERTFILE', None)
+        username = options['username'] or s('ARTICLES_EMAIL_USER', None)
+        password = options['password'] or s('ARTICLES_EMAIL_PASSWORD', None)
+        ssl = options['ssl'] or s('ARTICLES_EMAIL_SSL', False)
+
+        self.verbosity = int(options.get('verbosity', 1))
+
+        # try to guess if we don't have a port
+        if port is None:
+            if protocol == MB_IMAP4:
+                port = 993 if ssl else 143
+            elif protocol == MB_POP3:
+                port = 995 if ssl else 110
+
+        handle = None
+        try:
+            handle = self.get_mail_handle(protocol, host, port, username, password, keyfile, certfile, ssl)
+            messages = self.fetch_messages(protocol, handle)
+            created = self.create_articles(messages)
+            self.delete_messages(protocol, handle, created)
+        finally:
+            if handle:
+                if protocol == MB_IMAP4:
+                    # close the IMAP4 handle
+                    handle.close()
+                    handle.logout()
+
+                elif protocol == MB_POP3:
+                    # close the POP3 handle
+                    handle.quit()
+
+    def get_mail_handle(self, protocol, *args, **kwargs):
+        """
+        Returns a handle to either an IMAP4 or POP3 mailbox (or None if
+        something weird happens)
+        """
+
+        self.log('Creating handle to mail server')
+
         if protocol == MB_IMAP4:
-            messages = self.handle_imap4(*params)
-        elif protocol == MB_POP3:
-            messages = self.handle_pop3(*params)
+            return self.get_imap4_handle(*args, **kwargs)
+        else:
+            return self.get_pop3_handle(*args, **kwargs)
 
-        print messages
+        return None
 
-    def handle_imap4(self, host, port, username, password, keyfile, certfile, ssl):
+    def get_imap4_handle(self, host, port, username, password, keyfile, certfile, ssl):
+        """Connects to and authenticates with an IMAP4 mail server"""
+
         import imaplib
 
-        messages = []
-        if (keyfile and certfile) or ssl:
-            M = imaplib.IMAP4_SSL(host, port, keyfile, certfile)
+        try:
+            if (keyfile and certfile) or ssl:
+                self.log('Creating SSL IMAP4 handle')
+                M = imaplib.IMAP4_SSL(host, port, keyfile, certfile)
+            else:
+                self.log('Creating non-SSL IMAP4 handle')
+                M = imaplib.IMAP4(host, port)
+        except socket.error, err:
+            raise
         else:
-            M = imaplib.IMAP4(host, port)
+            self.log('Authenticating with mail server')
+            M.login(username, password)
+            M.select()
+            return M
 
-        M.login(username, password)
-        M.select()
+    def get_pop3_handle(self, host, port, username, password, keyfile, certfile, ssl):
+        """Connects to and authenticates with a POP3 mail server"""
 
-        typ, data = M.search(None, 'ALL')
+        import poplib
 
+        try:
+            if (keyfile and certfile) or ssl:
+                self.log('Creating SSL POP3 handle')
+                M = poplib.POP3_SSL(host, port, keyfile, certfile)
+            else:
+                self.log('Creating non-SSL POP3 handle')
+                M = poplib.POP3(host, port)
+        except socket.error, err:
+            raise
+        else:
+            self.log('Authenticating with mail server')
+            M.user(username)
+            M.pass_(password)
+            return M
+
+    def fetch_messages(self, protocol, handle):
+        """Fetches email messages from a server pased on the protocol"""
+
+        self.log('Fetching new messages')
+        try:
+            if protocol == MB_IMAP4:
+                messages = self.handle_imap4_fetch(handle)
+            elif protocol == MB_POP3:
+                messages = self.handle_pop3_fetch(handle)
+        except socket.error, err:
+            sys.exit('Error retrieving mail: %s' % (err,))
+        else:
+            parsed = {}
+
+            self.log('Parsing email messages')
+
+            # parse each email message
+            for num, mess in messages.iteritems():
+                fp = FeedParser()
+                fp.feed(mess)
+                parsed[num] = fp.close()
+
+            return parsed
+
+    def delete_messages(self, protocol, handle, created):
+        """Deletes the messages we just created articles for"""
+
+        self.log('Deleting consumed emails: %s' % (created,))
+        try:
+            if protocol == MB_IMAP4:
+                self.handle_imap4_delete(handle, created)
+            elif protocol == MB_POP3:
+                self.handle_pop3_delete(handle, created)
+        except socket.error, err:
+            sys.exit('Error deleting mail: %s' % (err,))
+
+    def handle_imap4_fetch(self, handle):
+        """Fetches messages from an IMAP server"""
+
+        messages = {}
+
+        self.log('Fetching from IMAP4')
+        typ, data = handle.search(None, 'ALL')
         for num in data[0].split():
-            typ, data = M.fetch(num, '(RFC822)')
-            messages.append((num, data))
-
-        M.close()
-        M.logout()
+            typ, data = handle.fetch(num, '(RFC822)')
+            messages[num] = data[0][1]
 
         return messages
 
-    def handle_pop3(self, host, port, username, password, keyfile, certfile, ssl):
-        import poplib
+    def handle_pop3_fetch(self, handle):
+        """Fetches messages from a POP3 server"""
 
-        messages = []
-        if (keyfile and certfile) or ssl:
-            M = poplib.POP3_SSL(host, port, keyfile, certfile)
-        else:
-            M = poplib.POP3(host, port)
+        messages = {}
 
-        M.user(username)
-        M.pass_(password)
-        num = len(M.list()[1])
+        self.log('Fetching from POP3')
+        num = len(handle.list()[1])
         for i in range(num):
-            for msg in M.retr(i + 1)[1]:
-                messages.append(msg)
-
-        M.quit()
+            messages[num] = '\n'.join([msg for msg in handle.retr(i + 1)[1]])
 
         return messages
 
+    def handle_imap4_delete(self, handle, created):
+        """Deletes the messages we just posted as articles"""
+
+        self.log('Deleting from IMAP4')
+        for num in created:
+            handle.store(num, '+FLAGS', '\\Deleted')
+
+        handle.expunge()
+
+    def handle_pop3_delete(self, handle, created):
+        """Deletes the messages we just posted as articles"""
+
+        self.log('Deleting from POP3')
+        map(handle.dele, created)
+
+    def get_email_content(self, email):
+        """Attempts to extract an email's content"""
+
+        if email.is_multipart():
+            self.log('Extracting email contents from multipart message')
+            for pl in email.get_payload():
+                if pl.get_content_type() in ('text/plain', 'text/html'):
+                    return pl.get_payload()
+        else:
+            return email.get_payload()
+
+        return None
+
+    def get_unique_slug(self, slug):
+        """Iterates until a unique slug is found"""
+
+        orig_slug = slug
+        year = datetime.now().year
+        counter = 1
+
+        while True:
+            not_unique = Article.objects.filter(publish_date__year=year, slug=slug)
+            if len(not_unique) == 0:
+                return slug
+
+            self.log('Found duplicate slug for year %s: %s. Trying again.' % (year, slug))
+            slug = '%s-%s' % (orig_slug, counter)
+            counter += 1
+
+    def create_articles(self, emails):
+        """Attempts to post new articles based on parsed email messages"""
+
+        self.log('Creating article objects')
+        created = []
+        site = Site.objects.get_current()
+
+        # make sure we have a valid default markup
+        ack = getattr(settings, 'ARTICLES_EMAIL_ACK', False)
+        markup = getattr(settings, 'ARTICLES_EMAIL_MARKUP', MARKUP_HTML)
+        if markup not in (MARKUP_HTML, MARKUP_MARKDOWN, MARKUP_REST, MARKUP_TEXTILE):
+            markup = MARKUP_HTML
+
+        for num, email in emails.iteritems():
+
+            name, sender = parseaddr(email['From'])
+
+            try:
+                author = User.objects.get(email=sender, is_active=True)
+            except User.DoesNotExist:
+                # unauthorized sender
+                self.log('Not processing message from unauthorized sender.', 0)
+                continue
+
+            # get the attributes for the article
+            title = email.get('Subject', '--- article from email ---')
+            slug = self.get_unique_slug(slugify(title))
+
+            content = self.get_email_content(email)
+            autopost = getattr(settings, 'ARTICLES_EMAIL_AUTOPOST', False)
+
+            try:
+                # try to grab the timestamp from the email message
+                publish_date = datetime.fromtimestamp(time.mktime(parsedate(email['Date'])))
+            except StandardError, err:
+                self.log("An error occured when I tried to convert the email's timestamp into a datetime object: %s" % (err,))
+                publish_date = datetime.now()
+
+            # post the article
+            article = Article(
+                author=author,
+                title=title,
+                slug=slug,
+                content=content,
+                markup=markup,
+                publish_date=publish_date,
+                is_active=autopost,
+            )
+
+            try:
+                article.save()
+            except StandardError, err:
+                # log it and move on to the next message
+                self.log('Error creating article: %s' % (err,), 0)
+                continue
+            else:
+                created.append(num)
+
+            if ack:
+                # notify the user when the article is posted
+                subject = u'%s: %s' % (_("Article Posted"), title)
+                message = _("""Your email (%(title)s) has been posted as an article on %(site_name)s.
+
+    http://%(domain)s%(article_url)s""") % {
+                    'title': title,
+                    'site_name': site.name,
+                    'domain': site.domain,
+                    'article_url': article.get_absolute_url(),
+                }
+
+                self.log('Sending acknowledgment email to %s' % (author.email,))
+                author.email_user(subject, message)
+
+        return created
+
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.