Investigate task processing options - Python 3.7 not possible with Celery 3 on Windows
From @Luuk
Python 3.7 introduces the reserved word async (https://docs.python.org/3/whatsnew/3.7.html). This is a breaking change and will break celery 3 as the underlying kombu library uses the word “async” as a package. This is fixed in newer versions used by celery 4, but celery 4 is not supporting Windows anymore. So we need to rethink eventually: or to keep using celery or to support Windows (both at the same time will eventually become impossible). For now: stick to Python 3.6
My opinion: I don’t want to get stuck on Python 3.6! So I want to use this ticket to research alternative solutions
Comments (54)
-
reporter -
reporter I think that you can run RabbitMQ, Celery and everything else in a (Linux) Docker container on Windows. This would potentially make installation easier and upgrades smoother for Windows users. I think. But I don’t have much experience with Docker yet.
I’d also have to change my development environment, as I run Ubuntu in Virtualbox on Windows 10 on my laptop, and Docker conflicts with VirtualBox.
What do you Windows users think the response would be if from version 1 onwards, Docker was the only supported method for WIndows users?
-
I am not familiar with Docker, so can’t comment on that at the moment.
Regarding the workaround for Celery 4 on Windows in the earlier link, I’m pretty sure that I’ve tried both methods suggested without success. However, I’ll try again and report back here.
-
I’m not familiar with Docker either. I know a colleague of mine really loves it, just for the reasons you mention. But my own experience is zero.
I will also try if I can get celery4 working with the workarounds mentioned in the above link.
-
Some other options:
-
Swap Celery for Dramatiq. Swapped basestring for str because basestring doesn't exist in python 3.x. References issue
#788. Charts of mean or median not working at the moment; pie charts work OK.Also references issue
#790→ <<cset 5bc1d8170307>>
-
A correction on my previous comment. Plots using mean values work; trying to use median values fails.
-
Failure to calculate median chart values was not due to a problem with my database: an OpenREM 0.10 install on python 2.7 is able to calculate medians from exactly the same database that when used with a python 3.7-based Django 2.2 version of OpenREM results in failure. I have replaced the median calculation with a simpler solution from here: https://gist.github.com/rdmurphy/3f73c7b1826cacee34f6c2a855b12e2e. This does not require the special median migration in order to work, which is a bonus. References issue
#788and#789→ <<cset 350e15417f8c>>
-
reporter Using dramatic looks exciting! Has you seen any disadvantages yet?
Thanks for opening a new issue/branch for medians. Can you do the same for the basestrings issue?
-
I’ve not actually got around to doing anything with dramatiq yet due to the other issues I’ve encountered.
-
Dramatiq has a page on the project’s motivation that includes a feature comparison with alternatives including celery (https://dramatiq.io/motivation.html). @Ed McDonagh do you think that this table flags any concerns with trying to use Dramatiq?
There’s a default time limit of 10 minutes for a Dramatiq actor, but this can be configured on a per-actor basis (https://dramatiq.io/guide.html#message-time-limits), so if we know that a certain task may take a long time we can configure a longer time-out for it - may need this for some chart calculations.
We use flower to monitor celery tasks. Dramatiq doesn’t seem to have a direct equivalent, although it does have an API that can be used to monitor what’s going on. One user has developed something to do this (https://www.reddit.com/r/dramatiq/comments/b1suh9/monitoring_dramatiq/).
-
reporter I did look at that. The two things that I thought we’d need to look at further are flower, or equivalent function, and bring able to track and kill tasks. Which I guess is the same thing.
-
I don’t think Dramatiq is able to kill tasks. See this link, amongst others: https://github.com/Bogdanp/django_dramatiq/issues/24
-
I’ll take a look at https://django-carrot.readthedocs.io/en/1.2.1/index.html
This has a built-in monitoring tool.
It’s unclear whether tasks can be cancelled.
-
reporter Can’t see a way of killing tasks. Also, doesn’t seem to be active, not sure if it is Python 3/Django 2.2 compatible or not: https://github.com/chris104957/django-carrot/issues/109
-
reporter Looks like it requires Python 3. Not sure about compatibility with Django >2.0 though: https://pypi.org/project/django-carrot/#data
-
Commiting these changes to show how to configure django-carrot. However, I don't think we can make use of this because it doesn't seem possible to cancel tasks. Also, I was unable to get the simple example running using Windows, python 3.7 and Django 2.2 - it returned an error. References issue
#788→ <<cset c6169d5ea108>>
-
This is tricky. It looks to me like Dramatiq is the best bet, but it is not able to cancel a task.
I re-tried the Celery 4.x Windows work-around using eventlet earlier (I was using it when I found the DX export problem). It does work, but seems very flaky. I’ll test again.
-
Celery 4.2.2; Windows 10.
Using:
celery worker -A openremproject -P gevent
celery -A openremproject flower
Makes it possible to export studies. Using eventlet fails when trying to write the actual file for some reason; gevent works.
However, it remains flaky.
-
I don’t think we can use Celery 4.0 on Windows. The latest error:
Fatal Python error: ffi.from_handle() detected that the address passed points to garbage. If it is really the result of ffi.new_handle(), then the Python object has already been garbage collected.
-
So I think we are left with Docker. @Ed McDonagh do you know what an end-user would have to do to use a Docker-based OpenREM install?
Perhaps we should set up a wiki page on here to put together our thoughts?
I’ve installed Docker for Windows and will have a tinker with it.
-
Did anyone try windows subsystem for linux (WSL) yet? Might solve the whole windows problem altogether.
Personally I’ve always seen docker as a great and flexible development tool, not really meant for running larger applications (including services) in production. Also requires some hacks to use systemd/systemctl (which does not run by default on docker). If you do consider using docker, perhaps docker compose would be best?Btw WSL can be enabled on Windows 10 home/professional and on Windows Server 2019 or later.
-
reporter Adding ref
#788to changes. Closes#788→ <<cset bbb271f1e1ab>>
-
reporter - changed status to closed
Adding ref
#788to changes. Closes#788→ <<cset bbb271f1e1ab>>
-
reporter @Jannis Widmer - take a look at this issue in relation to task managers. We can currently kill ongoing tasks - I didn’t see a way of doing that with django-backround-tasks.
Also, I’m worried about the fact the original project is archived, the main fork as far as I can see is mothballed: "Maintainership: I'm looking for a new Captain and a new Quartermaster! " and has not been taken over; there is a fork for compatibility with Django 4.0 but that doesn’t have an issues function and doesn’t give me much confidence. There are lots of other forks, but who knows which one is reliable!
-
I think you are actually right, sorry I did not look at the supported versions. Anyway given that I would actually like to simply use python’s multiprocessing module. That also has a few difficulties and will be a bit harder to implement than just using a django specific library, but it has full support in the newer versions of python, and we can replace all the functionality that celery provided. Under Linux I got this running already, but I still have to test if it works under Windows too.
-
reporter That would be amazing @Jannis Widmer - I’d much rather put additional work in at this stage and be able to release for Windows Server if we possibly can.
-
reporter - changed status to open
Issue being revisited due to Docker not running on Windows Server and Jannis having brilliant ideas as to alternatives...
-
reporter Adding reverse.js back in. Refs
#788→ <<cset 72a9ab70638e>>
-
reporter Removing flower from requirements. Refs
#788→ <<cset 37bd4334868e>>
-
Removed methods for direct run from all exporters. Updated all scripts and the docker import such that import happens as a (sequential) task refs
#788→ <<cset 76dc6530cb85>>
-
reporter Just correcting typos at the moment as I work through. Refs
#788→ <<cset a636a258a372>>
-
reporter Changed my mind regarding field name! Refs
#788[skip ci]→ <<cset 991dc7167728>>
-
reporter Reinserting missing title. Refs
#788[skip ci] docs only→ <<cset 8d38d66f5c87>>
-
reporter Adding removal of Flower and RabbitMQ to release notes. Attempt to add some of the system component diagram elements back in. Refs
#788[skip ci] docs only→ <<cset 41cf71db552f>>
-
reporter Removing PACS - OpenREM line, not sure what it represents. Complete mess currently, can tidy later! Refs
#788[skip ci] docs only→ <<cset cabb34fede65>>
-
reporter Extracts use Django app to update database, so direct line not needed. Removed reference to Flower in troubleshooting. Refs
#788[skip ci] docs only→ <<cset 887911fb8e33>>
-
reporter Removing Flower settings from settings. Refs
#788[skip ci] Not tested→ <<cset a2901df676aa>>
-
reporter Renamed model field pid to proc_id to avoid confusion with pid-patient ID that is used extensively. Refs
#788→ <<cset 3e6167de13a7>>
-
reporter Updating more of the pid->proc_id. Refs
#788Also removed test print statement (unrelated).→ <<cset cdc29d05c36c>>
-
reporter Removed span placeholder for service status that is no longer relevant. Refs
#788. [skip ci] not-tested web page→ <<cset 4e4d6d46a989>>
-
reporter Updating pid to proc_id in this branch to see if merge will work. Refs
#788.→ <<cset 3d32c4d8a672>>
-
reporter -
assigned issue to
-
assigned issue to
-
reporter Minor edits to tasks docs. Refs
#788→ <<cset 6b78bf205df1>>
-
reporter UK Spelling and grammar edits! Refs
#788[skip ci] comments only→ <<cset 9810e2b93c02>>
-
reporter Black. Refs
#788→ <<cset 039a5d510109>>
-
reporter Attempt to trim a few of the +88 issues. Refs
#94,#788,#934.→ <<cset 1d35a052b783>>
-
reporter Try nosec again. Refs
#788.→ <<cset ea3fda0d33d4>>
-
reporter Merged in issue94issue788issue934tidying (pull request #504)
→ <<cset 460d4b1bb85d>>
-
reporter Listing tasks newest at top. Reducing size of NM on top bar. Refs
#788.→ <<cset 0180b3d3bd30>>
-
reporter Sonarcloud fixes. Refs
#94,#788.→ <<cset ecc9b4d14694>>
-
reporter A few more sonarcloud fixes. Refs
#94,#788.→ <<cset ac7c3ce2e37a>>
-
reporter Updating changes with refs
#94,#788,#934, updating release notes. [skip ci] docs only→ <<cset 5aa12e5d8342>>
-
reporter - changed status to resolved
Merged in issue94issue788issue934Merge (pull request #500)
Issue94issue788issue934Merge
Fixes
#94,#788,#934, though there will be lots of docs revision to do.Approved-by: Ed McDonagh
→ <<cset ae758c3481e7>>
- Log in to comment
https://www.distributedpython.com/2018/08/21/celery-4-windows/