Make Postman's context processor lazy

[Meiyer]

In particular, it would be useful (and spare trips to the database) if the postman_unread_count variable could be made a SimpleLazyObject.

Motivation: we have different views that do not actually need the variable, and templates evaluated within other templates that cause duplicated trips to the database.

[Patrick Samson]

Fixed in repo.

[Meiyer]

I noticed already some time ago but did not have the time to investigate this further, that on each page load on our website there are 8 requests done to the DB to fetch the number of unread messages (SELECT COUNT(*) FROM "postman_message"). We have a template snippet that includes the postman_unread_count variable several times, which is the reason the value is queried more than once.

This started happening after the change of the context variable to resolve this issue #111. Upon further investigation that I finally managed to do, the culprit is the wrapping of the value in a lazy call. Per the documentation in Django’s code: “Results are not memoized; the function is evaluated on every access”, which causes the excessive database queries. Changing from lazy to a SimpleLazyObject fixed the issue: the value is evaluated only once and then remembered.

[Meiyer]

@Patrick Samson Are you planning to release a fix soon or should we implement a temporary workaround?

[Patrick Samson]

re-open ; 2019-03-02 first fix was SimpleLazyObject ; 2020-07-10 conversion to lazy() was supposed to be neutral, but it seems not. Under investigation.

[Patrick Samson]

There are pros and cons to choose between a cached value or a refreshed value for each reference to a context variable.

For example, csrf_token is SimpeLazyObject(), but sql_queries is lazy() to reflect the exact current state.

postman_unread_count should be stable during the rendering of a view, but on the other hand, it is not expected, in a basic view, to reference the variable multiple times.

Could you please check if your “template snippet” can be optimized with a cache:

{% with uc=postman_unread_count %}
... reference {{ uc }} as many times you want
{% endwith %}

‌

[Meiyer]

I tried the caching optimization, and the result is as follows:

‌

Not sure why the count cannot be referenced several times. For example, our template code (simplified) looks a bit like this:

{% if not postman_unread_count %}
    <!-- show “empty inbox” UI element -->
{% else %}
    {% blocktranslate count unread=postman_unread_count trimmed %}
        {{ count}} new message
    {% plural %}
        {{ count}} new messages
    {% endblocktranslate %}
    {% if postman_unread_count <= 99 %}
        <!-- show “half-full inbox” UI element -->
    {% else %}
        <!-- show “overflowing inbox” UI element -->
    {% endif %}
{% endif %}

‌

[Patrick Samson]

I still don’t see the usage of the with tag. But instead 3 evaluations of the variable in this template example.

[Meiyer]

The snippet above is an example. Earlier you said “it is not expected, in a basic view, to reference the variable multiple times” – thus I provided the above snippet as an example of legitimate code that needs the variable several times. I think it is not out of the ordinary to use it like that..?

On our actual website, I wrapped the code (which is similar to the above) in a {% with %} ... {% endwith %} , as you suggested; this did not reduce the number of queries. Each time I add 1 more reference (within the {% with %} block!) to the variable that received the postman_unread_count’s value, there is 1 more query to the database. This is what the picture with the 9 queries shows.

I just added a bunch of print statements into Django code, and got the following output. Apparently, even the {% with %} tag does not evaluate the lazy expression and keeps it lazy. As you can see, each reference to the postman_unread_count_value variable (assigned to in the {% with %} tag) results in a new evaluation.

WithNode __init__() ctx={'postman_unread_count_value': <django.template.base.FilterExpression object at 0x7fc3e446efd0>}
FilterExpression resolve() <Variable: 'postman_unread_count'>
Variable resolve() var_name=postman_unread_count in context
      <class 'function'> ==> <function inbox.<locals>.<lambda> at 0x7f2d1c4a0a70>
WithNode render() values = {'postman_unread_count_value': "<class 'django.utils.functional.lazy.<locals>.__proxy__'> 6"}
FilterExpression resolve() 'postman:inbox'
FilterExpression resolve() <Variable: 'postman_unread_count_value'>
Variable resolve() var_name=postman_unread_count_value in context
      <class 'django.utils.functional.lazy.<locals>.__proxy__'> ==> 6
FilterExpression resolve() <Variable: 'postman_unread_count_value'>
Variable resolve() var_name=postman_unread_count_value in context
      <class 'django.utils.functional.lazy.<locals>.__proxy__'> ==> 6
FilterExpression resolve() <Variable: 'postman_unread_count_value'>
Variable resolve() var_name=postman_unread_count_value in context
      <class 'django.utils.functional.lazy.<locals>.__proxy__'> ==> 6
/root/.venvs/mydev/lib/python3.7/site-packages/django/utils/translation/trans_real.py:250: DeprecationWarning: Plural value must be an integer, got __proxy__
  tmsg = self._catalog.plural(msgid1, n)
FilterExpression resolve() <Variable: 'postman_unread_count_value'>
Variable resolve() var_name=postman_unread_count_value in context
      <class 'django.utils.functional.lazy.<locals>.__proxy__'> ==> 6
FilterExpression resolve() <Variable: 'postman_unread_count_value'>
Variable resolve() var_name=postman_unread_count_value in context
      <class 'django.utils.functional.lazy.<locals>.__proxy__'> ==> 6

‌

[Patrick Samson]

You’re right. I’m trying to find if there is a solution (may be not). Meanwhile, a workaround is to force the evaluation to a string with a noop filter, such as:

{% with uc=postman_unread_count|lower %}

‌

[Patrick Samson]

Rollback to a SimpleLazyObject implementation in the 4.3.1 release.