1. OpenREM Avatar OpenREM
  2. OpenREM
  3. OpenREM
  4. Issues

Add Flower to install and link to manage Celery tasks

Issue #705 resolved
Ed McDonagh created an issue

Refs #662 which sorted out RabbitMQ, but we need to monitor Celery too.

http://docs.celeryproject.org/en/latest/userguide/monitoring.html#flower-real-time-celery-web-monitor

Either just install and link to same host on port 5555, or make use of the API to display in OpenREM as we do with RabbitMQ: https://flower.readthedocs.io/en/latest/api.html

Need to add as systemd daemon, eg

[Unit]
Description=Flower Celery Service

[Service]
User=your_user
Group=www-data
WorkingDirectory=/var/www/project-working-directory
ExecStart=/home/user/miniconda3/envs/virtualenv/bin/flower --port=5555  --loglevel=info -A yourproject
Restart=on-failure
Type=simple

[Install]
WantedBy=multi-user.target

A command equivalent to celery -A openremproject flower --address=127.0.0.1 would mean that it could be used with the API from OpenREM without being accessible password-free from other computers.

Comments (85)

  4. Ed McDonagh reporter

    Also need to not suggest that RabbitMQ queues that are waiting for a result to be returned can be purged, because they can't!

  9. David Platten

    Can I suggest something like the following command to run flower? We can make it clear that the user can specify the port and log file name / location.

    celery -A openremproject flower --port=5555 --loglevel=info --log-file-prefix=MEDIA_ROOT\celery\flower.log
  10. David Platten

    The Windows batch file that I use to run / restart celery kills all processes with the name "celery". This will kill flower as well. I've re-worked the batch file so that it uses the pid of the celery worker process to kill the correct celery. See below. I've also created a flower batch file that can be used to run flower as a service on Windows (this isn't capable of restarting Flower - it can just be used to start it up).

    Batch file for starting / restarting celery:

    ECHO OFF

:: Create variables containing the path to OpenREM and the name and path of the
:: celery pid  and log files
SET openremPath=D:\Server_Apps\python27\Lib\site-packages\openrem
SET celeryPidFile=E:\media_root\celery\default.pid
SET celeryLogFile=E:\media_root\celery\default.log


:: Change to the drive on which OpenREM is installed and navigate to the
:: OpenREM folder
D:
CD %openremPath%


:: Attempt to shutdown celery gracefully
celery -A openremproject control shutdown --timeout=10

:: Pause this file for 10 s to ensure that the above has time to work (you may
:: need to check that the 'timeout' command is available on your Windows
:: system. Some systems may have 'sleep' instead, in which case replace the
:: line below with:
:: SLEEP 10
TIMEOUT /T 10


:: Kill any remaining celery tasks (ungraceful) and delete the pid file in case
:: the above graceful shutdown did not work

:: Read the celery PID in from from the default.pid file
IF EXIST %celeryPidFile% (
    :: Read the pid values in from the file
    SET /P CeleryPid=<%celeryPidFile%

    :: Kill the process with that pid value
    TASKKILL /F /PID %celeryPid%

    :: Force the deletion of the pid file
    DEL /F %celeryPidFile%
)


:: Restart a new instance of celery 
::celery worker -n default -P solo -Ofair -A openremproject -c 4 -Q default --pidfile=e:\media_root\celery\default.pid --logfile=e:\media_root\celery\default.log
:: Changing concurrency to 1 to see if it stops the kombu messages in default.log about "Couldn't ack" - https://stackoverflow.com/questions/35325207/celery-and-rabbitmq-timeouts-and-connection-resets
celery worker -n default -P solo -Ofair -A openremproject -c 1 -Q default --pidfile=%celeryPidFile% --logfile=%celeryLogFile%

    Batch file for starting flower:

    ECHO OFF

:: Create variables containing the path to OpenREM and the name and path of the
:: celery pid  and log files
SET openremPath=D:\Server_Apps\python27\Lib\site-packages\openrem
SET flowerLogFile=E:\media_root\celery\flower.log
SET flowerPort=5555


:: Change to the drive on which OpenREM is installed and navigate to the
:: OpenREM folder
D:
CD %openremPath%


:: Run flower
celery -A openremproject flower --port=%flowerPort% --loglevel=info --log-file-prefix=%flowerLogFile%
  11. David Platten

    I now have flower running on my live Windows installation. I'm going to see whether it works OK when run using the two batch files I've posted above.

  12. Ed McDonagh reporter

    Modified main revoke functions to modern Celery method, which has the advantage of removing red error warnings! Also some tidying up and a 'reverse' that was left in views.py after it had been removed from the imports. Refs #705

    → <<cset d7335e521f40>>

  14. Ed McDonagh reporter

    Changed RabbitMQ table around, added link to Celery, added rudimentary text to Celery and connection error template. All needs linking to appropriate docs. Refs #705

    → <<cset 4f564b0a298e>>

  15. Ed McDonagh reporter

    @dplatten / anyone else who fancies it, would you try this branch (issue705flower, Diff to develop )

    As long as you have flower installed and running, it should work. Needs fine tuning, tidying up and docs, but it would be good to see if it reliably shows and controls the queues and tasks on different setups.

  16. David Platten

    @edmcdonagh I'll give this a go, but it will be easiest for me to test properly if I install it on my live system. Is the beta behaving on your system?

  18. Ed McDonagh reporter

    And if the question was referring to beta 2 on my production system, apart from the DX exports, yes 😁

  19. David Platten

    Thanks Ed - a beta release that incorporates this branch would be easiest for me to use.

    Yes, I was referring to the beta 2 on your production system. Glad to hear it's working OK.

    I'm looking forward to having the fluoroscopy high dose alerts available on our live system.

  26. David Platten

    Beta 3 is running on my live system. All seems to be OK so far. It's good to see celery jobs in the OpenREM interface - I have entries for skin dose map items, DICOM queries, exports etc.

  35. Ed McDonagh reporter

    Updated celery-linux deamonising docs to reflect one-page Ubuntu docs for Celery and Flower. Linked from start services. Refs #705 [skip ci] docs @dplatten - can you update the Windows docs please (celery-windows.rst)

    → <<cset e230ecaf7fbd>>

  37. David Platten

    Updating Windows instructions for setting up Celery and Flower as a . Need to check that I have used the correct line numbers in the text describing the batch files. References issue #705

    → <<cset 5ec231d24996>>

  42. Ed McDonagh reporter

    Revising to feed current and complete tasks separately to page, before laptop died! New branch as this is not a working setup. Refs #705 [skip ci] docs

    → <<cset d71a295b68bd>>

  43. David Platten

    I've been having trouble with queries. I think this typo may have been the route cause, preventing the previous Celery from being killed. Corrected typo in windows Celery documentation [skip ci]. References issue #705

    → <<cset 42f0f697e25a>>

  44. David Platten

    My Windows batch file that shuts down Celery and then restarts it doesn't work. The pid stored in default.pid is the process code associated with python.exe that is running Celery. Killing this pid stops python.exe, but does not stop celery.exe.

  46. David Platten

    I may have to redesign the batch file to kill all processes with the name celery.exe. This will also kill Flower, so the file will have to restart Flower as well as Celery.

    What's slightly odd is that when I try the graceful shutdown from the command line it always seems to work.

  47. Ed McDonagh reporter

    That would be annoying as by default you'd lose all the jobs each time and flower would be empty.

    There is a persistent flag or similar though that would be worth ticking. Unless you only care about the right now jobs. Not sure.

    Flower might of course point out that with your aggressive restart regeme that tasks are being revoked? It maybe not

  48. David Platten

    I have an updated batch file running on my live system. I'll report back tomorrow morning on how it has got on. The system is scheduled to carry out several PACS queries between now and then.

  53. David Platten

    I have re-written the Windows batch files used to start up Celery and Flower, and restart them if required. The original ones further up this issue didn't work correctly because executing a celery command stops execution of the file until the celery command has finished... which it never should. Celery and Flower need to be run from the batch file using their own batch file. I have done this for my live system and need to put them on here and in the official documentation.

  58. David Platten

    Updated the example Windows batch files to correct an error in the TASKKILL command and also address a problem with my original Flower version where the batch file waited for each celery task to complete before moving to the next command. These pauses meant that if the batch file started Flower then it never got to the line to restart Celery (the erroneous version was never available in release documentation, so shouldn't affect any users except me). This new version starts a separate background batch file to run the celery or flower commands. These files are in-use on my live system and are working well [skip ci]. References issue #705

    → <<cset e602708e4022>>

  59. Ed McDonagh reporter

    Great. Do you mind if I merge my issu705flower-reorg branch back into this one, with the intention of merging this into develop for the next beta? It means losing some of the sorting javascript you worked on, hence my hesitation... See Pull Request #266

  61. David Platten

    Making it possible to sort those tables made my head hurt, but I think we should be able to re-instate the code for the new version.

  62. Ed McDonagh reporter

    I knew it had been troublesome, hence my hesitation! The code is mostly all there, but a lot of it commented out. It all went wrong when I changed from one list to three, which on the RabbitMQ page has gone back to one, which is still sorted. I don't think there is any sorting now on the Flower page.

  66. Ed McDonagh reporter

    Very happy for it to be restyled - bearing in mind the active tasks has an extra column.

    What do you think about the concept of active, recent and older?

    Any thoughts on the RabbitMQ layout?

  67. David Platten

    I really like the active, recent and older tables on the Celery page. I've not thought about the RabbitMQ page yet.

    I'll put the Celery active table in the same format as the others and fix the table sorting at some point over the weekend.

  74. Ed McDonagh reporter

    Do you mean the ev ones? On the RabbitMQ page?

    In which case no, I've suppressed them, as they don't tell you anything useful.

    Instead there is a call to Flower which feeds the status on the RabbitMQ page.

  75. David Platten

    It's unlikely I'm going to be able to add the sorting to the new celery tables this week. I've also managed to make the new pages throw some errors when testing. I'll try and reproduce reliably and post here at some point.

  76. David Platten

    Celery tables can now be sorted. The time-related columns do not sort correctly at the moment - need to add a custom sort key. References issue #705.

    → <<cset de1e828a0397>>

  78. David Platten

    Reformatted the RabbitMQ page to match the Celery page. The panels on the RabbitMQ page can't be closed and stay shut as the whole section of HTML, including the panel code, is replaced every two seconds. References issue #705

    → <<cset 033ddff10044>>

  86. Log in to comment
Assignee
Type
enhancement
Priority
major
Status
resolved
Component
Tasks
Milestone
0.9.0
Votes
0
Watchers
1
Jira Software: the preferred issue tracker for Bitbucket. Join the team!