1. OpenREM Avatar OpenREM
  2. OpenREM
  3. OpenREM
  4. Issues

Investigate task processing options - Python 3.7 not possible with Celery 3 on Windows

Issue #788 resolved
Ed McDonagh created an issue

From @Luuk

Python 3.7 introduces the reserved word async (https://docs.python.org/3/whatsnew/3.7.html). This is a breaking change and will break celery 3 as the underlying kombu library uses the word “async” as a package. This is fixed in newer versions used by celery 4, but celery 4 is not supporting Windows anymore. So we need to rethink eventually: or to keep using celery or to support Windows (both at the same time will eventually become impossible). For now: stick to Python 3.6

My opinion: I don’t want to get stuck on Python 3.6! So I want to use this ticket to research alternative solutions

Comments (54)

  2. Ed McDonagh reporter

    I think that you can run RabbitMQ, Celery and everything else in a (Linux) Docker container on Windows. This would potentially make installation easier and upgrades smoother for Windows users. I think. But I don’t have much experience with Docker yet.

    I’d also have to change my development environment, as I run Ubuntu in Virtualbox on Windows 10 on my laptop, and Docker conflicts with VirtualBox.

    What do you Windows users think the response would be if from version 1 onwards, Docker was the only supported method for WIndows users?

  3. David Platten

    I am not familiar with Docker, so can’t comment on that at the moment.

    ‌

    Regarding the workaround for Celery 4 on Windows in the earlier link, I’m pretty sure that I’ve tried both methods suggested without success. However, I’ll try again and report back here.

    ‌

  4. Luuk

    I’m not familiar with Docker either. I know a colleague of mine really loves it, just for the reasons you mention. But my own experience is zero.

    I will also try if I can get celery4 working with the workarounds mentioned in the above link.

    ‌

  6. David Platten

    Swap Celery for Dramatiq. Swapped basestring for str because basestring doesn't exist in python 3.x. References issue #788. Charts of mean or median not working at the moment; pie charts work OK.

    Also references issue #790

    → <<cset 5bc1d8170307>>

  7. David Platten

    A correction on my previous comment. Plots using mean values work; trying to use median values fails.

  8. David Platten

    Failure to calculate median chart values was not due to a problem with my database: an OpenREM 0.10 install on python 2.7 is able to calculate medians from exactly the same database that when used with a python 3.7-based Django 2.2 version of OpenREM results in failure. I have replaced the median calculation with a simpler solution from here: https://gist.github.com/rdmurphy/3f73c7b1826cacee34f6c2a855b12e2e. This does not require the special median migration in order to work, which is a bonus. References issue #788 and #789

    → <<cset 350e15417f8c>>

  9. Ed McDonagh reporter

    Using dramatic looks exciting! Has you seen any disadvantages yet?

    Thanks for opening a new issue/branch for medians. Can you do the same for the basestrings issue?

  10. David Platten

    I’ve not actually got around to doing anything with dramatiq yet due to the other issues I’ve encountered.

  11. David Platten

    Dramatiq has a page on the project’s motivation that includes a feature comparison with alternatives including celery (https://dramatiq.io/motivation.html). @Ed McDonagh do you think that this table flags any concerns with trying to use Dramatiq?

    ‌

    There’s a default time limit of 10 minutes for a Dramatiq actor, but this can be configured on a per-actor basis (https://dramatiq.io/guide.html#message-time-limits), so if we know that a certain task may take a long time we can configure a longer time-out for it - may need this for some chart calculations.

    ‌

    We use flower to monitor celery tasks. Dramatiq doesn’t seem to have a direct equivalent, although it does have an API that can be used to monitor what’s going on. One user has developed something to do this (https://www.reddit.com/r/dramatiq/comments/b1suh9/monitoring_dramatiq/).

  12. Ed McDonagh reporter

    I did look at that. The two things that I thought we’d need to look at further are flower, or equivalent function, and bring able to track and kill tasks. Which I guess is the same thing.

  17. David Platten

    Commiting these changes to show how to configure django-carrot. However, I don't think we can make use of this because it doesn't seem possible to cancel tasks. Also, I was unable to get the simple example running using Windows, python 3.7 and Django 2.2 - it returned an error. References issue #788

    → <<cset c6169d5ea108>>

  18. David Platten

    This is tricky. It looks to me like Dramatiq is the best bet, but it is not able to cancel a task.

    I re-tried the Celery 4.x Windows work-around using eventlet earlier (I was using it when I found the DX export problem). It does work, but seems very flaky. I’ll test again.

  19. David Platten

    Celery 4.2.2; Windows 10.

    ‌

    Using:

    celery worker -A openremproject -P gevent

    celery -A openremproject flower

    ‌

    Makes it possible to export studies. Using eventlet fails when trying to write the actual file for some reason; gevent works.

    ‌

    However, it remains flaky.

  20. David Platten

    I don’t think we can use Celery 4.0 on Windows. The latest error:

    Fatal Python error: ffi.from_handle() detected that the address passed points to garbage. If it is really the result of ffi.new_handle(), then the Python object has already been garbage collected.

    ‌

  21. David Platten

    So I think we are left with Docker. @Ed McDonagh do you know what an end-user would have to do to use a Docker-based OpenREM install?

    Perhaps we should set up a wiki page on here to put together our thoughts?

    I’ve installed Docker for Windows and will have a tinker with it.

  22. Tim de Wit

    Did anyone try windows subsystem for linux (WSL) yet? Might solve the whole windows problem altogether.
    Personally I’ve always seen docker as a great and flexible development tool, not really meant for running larger applications (including services) in production. Also requires some hacks to use systemd/systemctl (which does not run by default on docker). If you do consider using docker, perhaps docker compose would be best?

    Btw WSL can be enabled on Windows 10 home/professional and on Windows Server 2019 or later.

  25. Ed McDonagh reporter

    @Jannis Widmer - take a look at this issue in relation to task managers. We can currently kill ongoing tasks - I didn’t see a way of doing that with django-backround-tasks.

    Also, I’m worried about the fact the original project is archived, the main fork as far as I can see is mothballed: "Maintainership: I'm looking for a new Captain and a new Quartermaster! " and has not been taken over; there is a fork for compatibility with Django 4.0 but that doesn’t have an issues function and doesn’t give me much confidence. There are lots of other forks, but who knows which one is reliable!

  26. Jannis Widmer

    I think you are actually right, sorry I did not look at the supported versions. Anyway given that I would actually like to simply use python’s multiprocessing module. That also has a few difficulties and will be a bit harder to implement than just using a django specific library, but it has full support in the newer versions of python, and we can replace all the functionality that celery provided. Under Linux I got this running already, but I still have to test if it works under Windows too.

  27. Ed McDonagh reporter

    That would be amazing @Jannis Widmer - I’d much rather put additional work in at this stage and be able to release for Windows Server if we possibly can.

  28. Ed McDonagh reporter
    • changed status to open

    Issue being revisited due to Docker not running on Windows Server and Jannis having brilliant ideas as to alternatives...

  31. Jannis Widmer

    Removed methods for direct run from all exporters. Updated all scripts and the docker import such that import happens as a (sequential) task refs #788

    → <<cset 76dc6530cb85>>

  35. Ed McDonagh reporter

    Adding removal of Flower and RabbitMQ to release notes. Attempt to add some of the system component diagram elements back in. Refs #788 [skip ci] docs only

    → <<cset 41cf71db552f>>

  37. Ed McDonagh reporter

    Extracts use Django app to update database, so direct line not needed. Removed reference to Flower in troubleshooting. Refs #788 [skip ci] docs only

    → <<cset 887911fb8e33>>

  55. Log in to comment
Assignee
Type
bug
Priority
critical
Status
resolved
Component
Tasks
Milestone
1.0.0
Votes
0
Watchers
1
Jira Software: the preferred issue tracker for Bitbucket. Join the team!