Add lingua plugin

#9 Merged at 639fa5a
  1. Wichert Akkerman

This pull requests adds a lingua plugin to Mako. In addition it adds a couple of high-level tests for the existing Babel plugin, and refactors that plugin a bit to allow reuse of the basic parsing code.

Please note that this is still a work in progress, so please don't merge it as-is. There are a couple of things I still want to do:

  • Run this on some real world mako templates. Since I've never used Mako myself any pointers to examples would be very welcome :)
  • Lingua itself needs a better API to access its Python parser. I want to have that in place before getting this out - the current use of _extract_python is a bit dirty.

Comments (18)

  1. Wichert Akkerman author

    I've just pushed out lingua 3.2 which cleans up the API to call into extractors and updated the Mako plugin to match. I also extended the tests a bit to run gettext.mako through the lingua plugin as well, which revealed an interesting bug.

    As far as I can see this is now ready for merging.

  2. Mike Bayer repo owner

    OK I'm just noticing "please dont merge as is", Im fixing up the tests a bit but then im going to merge it as is :) if you want to do any more cleanup for the release then let me know.

  3. Mike Bayer repo owner

    I've removed babel and lingua as test dependencies and added these back into the "skip" system (beaker is already not defaulted there either). babel in particular is causing problems on py3k and for a clean "python test" I'd prefer it not get into edgier cases, as that's what distros typically rely upon for vanilla package builds.

  4. Mike Bayer repo owner

    the refactoring has broken the babelplugin on python 3:

    test/ext/ [(1, '_', 'Page arg 1', []), (1, '_', 'Page arg 2', []), (10, 'gettext', 'Begin', []), (14, '_', 'Hi there!', ['TRANSLATOR: Hi there!']), (19, '_', 'Hello', []), (22, '_', 'Welcome', []), (25, '_', 'Yo', []), (36, '_', 'The', ['TRANSLATOR: Ensure so and', 'so, thanks']), (36, 'ungettext', ('bunny', 'bunnies', None), []), (41, '_', 'Goodbye', []), (44, '_', 'Babel', []), (45, 'ungettext', ('hella', 'hellas', None), []), (62, '_', 'The', ['TRANSLATOR: Ensure so and', 'so, thanks']), (62, 'ungettext', ('bunny', 'bunnies', None), []), (68, '_', 'Goodbye, really!', ['TRANSLATOR: HTML comment']), (71, '_', 'P.S. byebye', []), (77, '_', 'Top', []), (83, '_', 'foo', []), (83, '_', 'hoho', []), (85, '_', 'bar', []), (92, '_', 'Inside a p tag', []), (95, '_', 'Later in a p tag', []), (99, '_', 'No action at a distance.', [])]

    I'm trying to see what the origin of this is but so far there seems to be a lot changed here.

  5. Mike Bayer repo owner

    found it:

    diff --git a/mako/ext/ b/mako/ext/
    index 0d9c60c..758c3a9 100644
    --- a/mako/ext/
    +++ b/mako/ext/
    @@ -14,7 +14,7 @@ class MessageExtractor(object):
         def extract_nodes(self, nodes):
             translator_comments = []
             in_translator_comments = False
    -        comment_tags = filter(None, re.split(r'\s+', self.config['comment-tags']))
    +        comment_tags = list(filter(None, re.split(r'\s+', self.config['comment-tags'])))
             for node in nodes:
                 child_nodes = None
  6. Wichert Akkerman author

    I'm going to guess Python 3 changed filter to return a generator, and that confused the plugin somehow? I don't see how that would break anything considering the code just does a for loop over them.

  7. Wichert Akkerman author

    You are using an private (underscore-prefixed) method from Mako, so it isn't entirely surprising something broke for your. Is there a particular type of API you are looking for that Mako might be able to provide?

  8. Maxim Avanov

    It isn't underscored. My code still imports from mako.ext.babelplugin import extract and then renames it to _extract_mako for convenience within a module.

  9. Maxim Avanov

    Is there a particular type of API you are looking for that Mako might be able to provide?

    Yes, I am looking for a way to derive from the default messages extractor so the derived version can preprocess custom syntax and feed the resulting mako syntax into it in order to retrieve translation strings. Previously, I could do that by just wrapping around Mako's extract() method. How is it supposed to be done now? Thanks!

  10. Maxim Avanov

    Ok, I've got the idea. Actually, everything works fine, it's just that previously I had been able to call extract() with None as comment_tags in my tests, and apparently, this argument was used slightly differently within a function and such a case was never an issue. Now it is.

    Sorry for false alarm.