option to apply filters before the defaults ?

Issue #171 new
created an issue

Maybe I am wrong here, but shouldn't default filters such as unicode() be applied last? The fact that it's done the exactly opposite way makes it a pain, as we are limited to deal with the string/unicode representation of whatever variable is passed to the filter.

Hence, I propose the following change:

diff -r 29d9a098a70465450563e9c07ed613e227886936 mako/codegen.py
--- a/mako/codegen.py   Fri Aug 05 17:45:51 2011 -0400
+++ b/mako/codegen.py   Mon Aug 08 11:14:48 2011 +0200
@@ -669,7 +669,7 @@
                 if self.compiler.pagetag:
                     args = self.compiler.pagetag.filter_args.args + args
                 if self.compiler.default_filters:
-                    args = self.compiler.default_filters + args
+                    args = args + self.compiler.default_filters
         for e in args:
             # if filter given as a function, get just the identifier portion
             if e == 'n':

What do you think?



Comments (12)

  1. Michael Bayer repo owner

    Each of the `u`, `h`, and `x` filters expect a string. So if i said:

    ${x | h}

    expecting "x" to be HTML encoded, then "x" happens to be an integer or some other non-string that is coerced by ``unicode()``, kaboom.

    The patch above breaks the one test we have in this regard which checks this:

    1. !python def test_convert_str(self): """test that string conversion happens in expressions before sending to filters""" t = Template(""" ${x | trim} """) assert flatten_result(t.render(x=5)) == "5"

    now I am absolutely aware what a PITA this is for non-string oriented filters like formatters. All throughout our templates, we just have a lot of "n" going on.

    I have the impression that a better idea would be needed here. ${x | y} has "y" as a "filter" - it processes output. In practice, we use ${x | y} a lot for '''formatting'''. This is a different role. I'm seeing the "Filtering and Buffering" chapter being renamed "Filtering, Formatting, and Buffering" in the docs. Syntax change ?

    ${x | formatter1, formatter2, ... | filter1, filter2, ...}

    ${x & formatter1, formatter2, ... | filter1, filter2, ...}


    then again, the usual approach:

    ${x | formatter1, n}

    is actually more efficient than having the formatter + the unicode. The "formatters" we use already return a fully composed string, so the extra "unicode" call would be overhead (and all those unicode calls do add up). So, not sure I want to do anything here, perhaps update the docs to discuss the roles of "filter"/"formatting" regardless.

  2. guest reporter

    An alternative could be placing letting the conversion to string be done by those filters, no? If `h` and `trim` are expecting strings, they should make sure all the arguments are properly converted to strings beforehand.


  3. Michael Bayer repo owner

    Replying to [comment:2 guest]:

    An alternative could be placing letting the conversion to string be done by those filters, no? If `h` and `trim` are expecting strings, they should make sure all the arguments are properly converted to strings beforehand.

    but that is also wasteful and redundant, as well as an implicit behavior I don't like very much, for each filter to also embed string conversion within it. the filtering of unicode etc. is the primary performance bottleneck in mako.

  4. guest reporter

    Replying to [comment:3 zzzeek]:

    I see. Too bad then. Still, a solution that would be easier than changing the whole syntax could be finding a way of implying the `n` in some of the filters without having to specify it manually. Maybe something like this could work?

    1. !python def json(obj):
    2. ... json.exclusive = True

    I don't know about the performance penalty though.

  5. Michael Bayer repo owner

    having the formatter advertise as such might be nice, though again that's a runtime check since filters/formatters can be passed into the template.

    A special character, perhaps:

    ${mydata | >json}

  6. guest reporter

    (martin@geisler.net here — can I create a new account for Trac somehow so I can get followups by email?)

    I wanted to add a json filter and hoped to make it usable as ${x | json} without having to add any extra n filters. I only realized that the default_filters are applied before other filters after messing around with this for a day.

    Could the name be changed to initial_filters or something similar? That is, the name could be made clearer to indicate that the filters are not ''defaults'' (that can be overwritten later) but unconditionally applied filters.

    Adding "formatters" would be nice — anything that lets us work on the Python values before they're turned into strings.

    I'll now go and change my ${x | json} fragments into ${json(x)} in the templates. That '''does''' work since it is compiled into what I need: filters.html_escape(json(x)). My filter outputs a unicode subclass which defines __html__ to return self, which makes the html_escape a no-op.

  7. Michael Bayer repo owner

    my plan is entirely to move this trac to the bitbucket repo, but I don't have a completed script that can convert all the issues to their import format (I have one I was writing, but it's a big project).

    Feel free to just discuss on the mailing list, this trac is pretty out of the loop.

  8. Michael Bayer repo owner

    oh but name change, i dont really think we can change any names. mako is installed very broadly in projects that don't really want to look at their mako code very often so i dont think backwards incompatible anything is possible right now.

  9. Michael Bayer repo owner

    if "default_filters" is ambiguous, i dont see why we can't just make sure the documentation makes these clear. in my mind "default" means, "the filters that get applied in all cases no matter what".

    where would "formatters" go exactly, im not understanding that ? the "turning things into strings" is what default_filters does right now, if you put a function of your own in there, it replaces the "turning things into strings" with whatever you want. the proposal here is to somehow extend the "|" syntax to allow filters before or after the default chain.

  10. guest reporter

    (Martin here). I'm not a native English speaker, so my understanding of "default" may be wrong. I had just expected it to be the filters that would be applied ''unless'' an expression specified other filters. Like default arguments for Python: def foo(x=10) defines a function where x is 10 ''unless'' I specify otherwise when I call the function.

    It doesn't matter much, though, and I agree that updating the documentation would be the way to go here. You should definitely not make backwards incompatible changes ("renaming" the keyword argument could be done by adding a new one and only documenting that but still accepting the old one).

    With regard to formatters or filters: when I first read about Mako filters, I thought that ${x | foo} was equivalent to ${foo(x)}. This was further reinforced when I saw that you can define a function inside a template and use that as a filter. The crucial difference, though, is the order of application and that wasn't clear to me until I read create_filter_callable in codegen.py.

  11. Log in to comment