Investigate task processing options - Python 3.7 not possible with Celery 3 on Windows

Issue #788 closed
Ed McDonagh created an issue

From @Luuk

Python 3.7 introduces the reserved word async (https://docs.python.org/3/whatsnew/3.7.html). This is a breaking change and will break celery 3 as the underlying kombu library uses the word “async” as a package. This is fixed in newer versions used by celery 4, but celery 4 is not supporting Windows anymore. So we need to rethink eventually: or to keep using celery or to support Windows (both at the same time will eventually become impossible). For now: stick to Python 3.6

My opinion: I don’t want to get stuck on Python 3.6! So I want to use this ticket to research alternative solutions

Comments (24)

  1. Ed McDonagh reporter

    I think that you can run RabbitMQ, Celery and everything else in a (Linux) Docker container on Windows. This would potentially make installation easier and upgrades smoother for Windows users. I think. But I don’t have much experience with Docker yet.

    I’d also have to change my development environment, as I run Ubuntu in Virtualbox on Windows 10 on my laptop, and Docker conflicts with VirtualBox.

    What do you Windows users think the response would be if from version 1 onwards, Docker was the only supported method for WIndows users?

  2. David Platten

    I am not familiar with Docker, so can’t comment on that at the moment.

    Regarding the workaround for Celery 4 on Windows in the earlier link, I’m pretty sure that I’ve tried both methods suggested without success. However, I’ll try again and report back here.

  3. Luuk

    I’m not familiar with Docker either. I know a colleague of mine really loves it, just for the reasons you mention. But my own experience is zero.

    I will also try if I can get celery4 working with the workarounds mentioned in the above link.

  4. David Platten

    Swap Celery for Dramatiq. Swapped basestring for str because basestring doesn't exist in python 3.x. References issue #788. Charts of mean or median not working at the moment; pie charts work OK.

    Also references issue #790

    → <<cset 5bc1d8170307>>

  5. David Platten

    A correction on my previous comment. Plots using mean values work; trying to use median values fails.

  6. David Platten

    Failure to calculate median chart values was not due to a problem with my database: an OpenREM 0.10 install on python 2.7 is able to calculate medians from exactly the same database that when used with a python 3.7-based Django 2.2 version of OpenREM results in failure. I have replaced the median calculation with a simpler solution from here: https://gist.github.com/rdmurphy/3f73c7b1826cacee34f6c2a855b12e2e. This does not require the special median migration in order to work, which is a bonus. References issue #788 and #789

    → <<cset 350e15417f8c>>

  7. Ed McDonagh reporter

    Using dramatic looks exciting! Has you seen any disadvantages yet?

    Thanks for opening a new issue/branch for medians. Can you do the same for the basestrings issue?

  8. David Platten

    I’ve not actually got around to doing anything with dramatiq yet due to the other issues I’ve encountered.

  9. David Platten

    Dramatiq has a page on the project’s motivation that includes a feature comparison with alternatives including celery (https://dramatiq.io/motivation.html). @Ed McDonagh do you think that this table flags any concerns with trying to use Dramatiq?

    There’s a default time limit of 10 minutes for a Dramatiq actor, but this can be configured on a per-actor basis (https://dramatiq.io/guide.html#message-time-limits), so if we know that a certain task may take a long time we can configure a longer time-out for it - may need this for some chart calculations.

    We use flower to monitor celery tasks. Dramatiq doesn’t seem to have a direct equivalent, although it does have an API that can be used to monitor what’s going on. One user has developed something to do this (https://www.reddit.com/r/dramatiq/comments/b1suh9/monitoring_dramatiq/).

  10. Ed McDonagh reporter

    I did look at that. The two things that I thought we’d need to look at further are flower, or equivalent function, and bring able to track and kill tasks. Which I guess is the same thing.

  11. David Platten

    Commiting these changes to show how to configure django-carrot. However, I don't think we can make use of this because it doesn't seem possible to cancel tasks. Also, I was unable to get the simple example running using Windows, python 3.7 and Django 2.2 - it returned an error. References issue #788

    → <<cset c6169d5ea108>>

  12. David Platten

    This is tricky. It looks to me like Dramatiq is the best bet, but it is not able to cancel a task.

    I re-tried the Celery 4.x Windows work-around using eventlet earlier (I was using it when I found the DX export problem). It does work, but seems very flaky. I’ll test again.

  13. David Platten

    Celery 4.2.2; Windows 10.

    Using:

    celery worker -A openremproject -P gevent

    celery -A openremproject flower

    Makes it possible to export studies. Using eventlet fails when trying to write the actual file for some reason; gevent works.

    However, it remains flaky.

  14. David Platten

    I don’t think we can use Celery 4.0 on Windows. The latest error:

    Fatal Python error: ffi.from_handle() detected that the address passed points to garbage. If it is really the result of ffi.new_handle(), then the Python object has already been garbage collected.

  15. David Platten

    So I think we are left with Docker. @Ed McDonagh do you know what an end-user would have to do to use a Docker-based OpenREM install?

    Perhaps we should set up a wiki page on here to put together our thoughts?

    I’ve installed Docker for Windows and will have a tinker with it.

  16. Tim de Wit

    Did anyone try windows subsystem for linux (WSL) yet? Might solve the whole windows problem altogether.
    Personally I’ve always seen docker as a great and flexible development tool, not really meant for running larger applications (including services) in production. Also requires some hacks to use systemd/systemctl (which does not run by default on docker). If you do consider using docker, perhaps docker compose would be best?

    Btw WSL can be enabled on Windows 10 home/professional and on Windows Server 2019 or later.

  17. Log in to comment