Add web import function for height and weight data

Issue #21 resolved
Ed McDonagh created an issue

Comments (42)

  1. Ed McDonagh reporter

    Added new view and moved the csv opening and checking logic to it, plus copied and adapted the upload template. New size_process view currently just checks the csv and displays the first line and presents a form entry for height field. Refs #21.

    → <<cset e35fb4254410>>

  2. Ed McDonagh reporter

    Now feeds through the real headers. Zip puts two lists together as a list of tuples. This is then converted to a tuple of tuples. Refs #21.

    → <<cset b443418dfe0c>>

  3. Ed McDonagh reporter

    Put the table back into the template. Corrected error with ID type field in both form and template. Now need to look at what it returns. Refs #21.

    → <<cset 6e0756b56ad9>>

  4. Ed McDonagh reporter

    Changed 'self' to 'request' in sizeprocess and added responseredirects to each of the error conditions to return to the upload page with a message. Refs #21

    → <<cset a06537dc9921>>

  5. Ed McDonagh reporter

    Added in basic template for processing the form with data, but with nothing added yet. Changed template to have 'submit' on the button. Need to work out if the submit has to include the uploaded form ID, and if so to include it in the template. Currently fails on this, but have seen that all the required data is in the post message. Refs #21.

    → <<cset 3ed317545ea5>>

  6. Ed McDonagh reporter

    csvid now passed to template so it can be used in the form URL. Changed request.POST to data = request.POST, which has stopped the error messages, but form is still not valid. It might be that the values for the choice are not there when it is being validated as the form html tells me that Height isn't one of the available choices. Refs #21.

    → <<cset ad47b3f799d3>>

  7. Ed McDonagh reporter

    Attempt to get form validation working, not successful. After this commit, will abandon that task and instead validate the data in the POST message to extract the information we need. Refs #21.

    → <<cset e9ad34b09756>>

  8. Ed McDonagh reporter

    Taken out the attempt to validate the form, and instead make sure that there are no duplicates in the column headers chosen, and made a first attempt at the call to be made to the patient size import function. Currently doesn't render anything so runs through to the end and goes wrong. Refs #21.

    → <<cset 81fdfce24942>>

  9. Ed McDonagh reporter

    Added routine to be called from web function. Adapted existing code to allow for the extra functions and variables - needs testing. Verbose logging is pushed into the database. Needs to be converted to celery. Refs #21.

    → <<cset dc9430e7c065>>

  10. Ed McDonagh reporter

    Added url for doing the actual import, and an associated view. The view currently just launches the new import function, passing the id of the record in the database. Need to convert to a celery task launch, and hopefully an ajax view to display the log. Refs #21.

    → <<cset 63247bcf2d5a>>

  11. Ed McDonagh reporter

    Converted websizeimport to a celery task, and added a template and response. Works, but not a good solution as the import is repeated every time the page is refreshed. Refs #21.

    → <<cset 83d6d62fd586>>

  12. Ed McDonagh reporter

    Moved task launch to size_process and added auto refresh and styling to template. Works, but not elegant. Should write logs to file instead of db, add status updating messages to db and model current and previous imports on the exports interface. Links to download logs as text file should be made. Add time taken to db for this and exports. Add Admin menu. Add security. Refs #21.

    → <<cset 9f5aee214b9b>>

  13. Ed McDonagh reporter

    Added fields to Size_upload model to allow for Celery progress status updates like the exports. Added process time to both. Had difficulty migrating the Size_upload table, but this shouldn't be encountered in the wild because a/ the entire table will be new to everyone and b/ they shouldn't be using sqlite3 in production environments, and it seems to be tied to the functionalit of sqlite3. Refs #21.

    → <<cset 2d4bdf0f3b6f>>

  14. Ed McDonagh reporter

    Added import date to size upload. Changed field type for processtime to FloatField to store deltatime coverted to seconds. Need to change the export version likewise. Refs #21.

    → <<cset 0e7502a1c51c>>

  15. Ed McDonagh reporter

    Changed size import to copy the export version with current and complete and with no id being passed between the view that launches the task and the view that displays the processing. Refs #21.

    → <<cset fae6eb6f6bad>>

  16. Ed McDonagh reporter

    Open csv file initially to see how many rows it has and close it again. Now updates progress to database and template and reports total number of rows processed. Refs #21.

    → <<cset 888d094ae630>>

  17. Ed McDonagh reporter

    Added Admin navigation dropdown with two patient size related links to base. Each size template then highlights the Admin block. Changed name of sizeupload to sizeuploads to better reflect its final purpose. Some templates aren't showing the Export or Admin navs - need to fix. Refs #21.

    → <<cset 65db8a5e5e9c>>

  18. Ed McDonagh reporter

    Changed method of opening csv files to use the FieldFile methods. Now automatically deletes the uploaded file when it has been processed. Consider not using the date structure for file names and recording what the file name was in the log. Need to implement delete function for logs. Refs #21.

    → <<cset 2c3270e63b55>>

  19. Ed McDonagh reporter

    Minor prettifying of process form. Duplicate header selections now generates error message displayed on a reloaded process page. Refs #21.

    → <<cset 7f10d48e4ccf>>

  20. Ed McDonagh reporter

    Improved import log by not logging the entries that aren't found, added logging of studies that are found but height or weight already exist, improved formatting. Need to add security and handling of attempts to reprocess files that have already been deleted. Refs #21.

    → <<cset 963e2625700f>>

  21. Ed McDonagh reporter

    Retested ptsizeimport from the command line to check if all the changes made for the web version (ref #21) had caused any problems. All fine. Information included in the log was changed when it was implemented for the log version - further change here to add the spaces to the start of the action lines to improved readability.

    → <<cset a5f00c2414e1>>

  22. Ed McDonagh reporter

    Added new template tag to convert seconds into a natural time duration. Applied it to both the exports table and the patient size imports table. Refs #106, #21.

    → <<cset 90005314d6fa>>

  23. Log in to comment