Allow complex filtering - NOT, OR

Issue #17 new
Ed McDonagh created an issue

Implement for 1.0 release, with django-filter 2, Python 3, Django 2.2 etc.

Refs #437

Comments (30)

  1. Ed McDonagh reporter

    Thanks David. It hadn't occurred to me that the actual database lookups were going to be difficult. Looking at the Q docs, it appears that I hadn't really looked into it very far!

    My initial issue is how to have an interface that allows complex sets of queries in a usable form, both for the end user and the person maintaining the code.

  2. David Platten

    You could have a series of drop-down lists containing field names. Each drop-down list could be separated from the next by a pair of fixed-choice buttons setting "OR" or "AND" between the lists. Where it gets tricky is that you may want the user to be able to use brackets around some of their selections.

  3. Luuk

    Does someone know if their are "plugins" for Django to get a type of filtering on the models like the "advanced" filter options on the Bitbucket issues page or even "more advanced" like pubmed? This last one can also do the "bracket-option". I would think this is a quite common feature needed, but I can't find anything. It might be I'm searching with the wrong terms.

  4. Luuk

    Refs #17 allow complex filtering

    First attempt implementing complex filtering by a query builder form using Django Q objects: - For now only fields in CT models are selectable (DX/RF/Mammo-fields are not added, but could easily) - AND / OR / AND NOT / OR NOT can be chosen to connect constraints on fields - Brackets can be used (first and last textbox, this might be unclear for the user) - For now the search is completely modality independent, although results use CT template and only CT models are loaded - For now page is only reachable by manually entering http://server:port/openrem/advanced_search

    Things that need to be done (at least): - Check for validity of user input * only decimals in decimal fields, dates in date fields, etc.. * Correct placing of brackets - Results for queries without brackets are almost certainly correct, but with brackets need to be tested thoroughly - If query is sent, the results page should also contain the query as built by the user (textboxes should be filled). So the user can adapt the query easily - Think about available fields...There are a lot. Do we need to sort or limit them? - Verbose name is used for displaying field names, but we may need to add these to the model for a number of fields for clarity - Should it be possible to select "NOT" for the first field search? - Pressing <Enter> while "add line" is selected will add extra line, but selected "add line" icon remains visible and next line added is empty - Write test script

    → <<cset 638fb3e0f115>>

  5. Luuk

    I just committed a first attempt for making complex filtering possible (advanced search). It isn't finished by far, but I'm really curious what you (@edmcdonagh, @dplatten, @tcdewit) think about the principal and possibilities. See commit 638fb3e, for some additional remarks

  6. Luuk

    Refs #17 allow complex filtering

    First attempt implementing complex filtering by a query builder form using Django Q objects: - For now only fields in CT models are selectable (DX/RF/Mammo-fields are not added, but could easily) - AND / OR / AND NOT / OR NOT can be chosen to connect constraints on fields - Brackets can be used (first and last textbox, this might be unclear for the user) - For now the search is completely modality independent, although results use CT template and only CT models are loaded - For now page is only reachable by manually entering http://server:port/openrem/advanced_search

    Things that need to be done (at least): - Check for validity of user input * only decimals in decimal fields, dates in date fields, etc.. * Correct placing of brackets - Results for queries without brackets are almost certainly correct, but with brackets need to be tested thoroughly - If query is sent, the results page should also contain the query as built by the user (textboxes should be filled). So the user can adapt the query easily - Think about available fields...There are a lot. Do we need to sort or limit them? - Verbose name is used for displaying field names, but we may need to add these to the model for a number of fields for clarity - Should it be possible to select "NOT" for the first field search? - Pressing <Enter> while "add line" is selected will add extra line, but selected "add line" icon remains visible and next line added is empty - Write test script

    → <<cset faabb7ec3cdc>>

  7. Luuk

    Refs #17 Advanced Filtering

    • Added advanced filtering option to CT filtered page as example (can be quite easily implemented for other modalities)
    • Needed to rewrite ctfiltered template as "form" has to extend multiple blocks
    • Made it possible to swich from exam filter to advanced filter and vice versa
    • Improved advanced filtering with possibility having NOT in first line
    • Updated logic to never negate root node (but always add a NOT node)
    • Updated logic to respect logic for user specific "pid"
    • Updated logic to be able to insert filter on modality independent of user input (depending on modality viewed)
    • Resolved PEP8 issues in mod_filters

    → <<cset 406a6f51acae>>

  8. Luuk

    I just uploaded an advanced filtering option in the CT filtered page. I'm rather happy with it (of course there are some "issues'). It can be quite easily build in the DX and RF filtered pages also.

    Some "highlights":

    • The idea of the advanced filtering is based on the Pubmed advanced search
    • Default behaviour of the CT filtered is unchanged
    • It is possible to switch from the default "exam filter" to the advanced filtering within the CT filtered page and vice versa
    • Currently a lot of filter parameters can be selected (should they be sorted? reduced?).
    • You can use brackets and AND/OR/NOT operators in the search options (next to the comparison operators depending on the field type).

    Some Remarks:

    • Currently the plots don't respect the advanced filter
      • I don't understand how data flows to the plots, but it seems dependent on the specific "exam filter" fields
      • It would be great if you could pass a filter- or querset-object
    • I tested it with a limited number of datasets and it seems to do the right thing, but I'm not sure if it works in all cases
    • I think it is unclear that the textbox just before the filterfield and the last textbox in each line are meant to place brackets "(" or ")"
      • Any ideas how to improve this?
    • Checking that "number-fields" only contain a number in the filter value is not implemented (yet, same holds through for dates)
    • In the end it would be great if you would be able to save (and load) queries
      • Then it would also be nice if date-fields could be relative ("from on year ago till now")
      • For now you can just copy-paste URLs

    @edmcdonagh, @dplatten what do you think? Any suggestions to improve this filtering?

  9. David Platten

    This would be useful when wanting to filter to just “CT head” requests, but exclude “CT head with contrast” requests. I would like to be able to do this for the PHE CT dose export (#737); it is likely to be useful for the PHE radiographic and fluoroscopy survey too (#750)

  10. Ed McDonagh reporter

    Hmm. I don’t think we are going to be able to do this in the current beta/release. For version 1.0, I am intending to start again with django-filters (current release version), and make sure we learn from all the work that @LuukO has done here to implement AND/OR/NOT in the standard filter view (including charts). If you have any ideas about that, I’d be please to hear them.

    For your current situation - is there anything in the combination of study description, requested procedure, procedure etc that can narrow down what you need instead?

  11. David Platten

    I think that incorporating the logic in the standard filter view is the way to go.

    For my current situation I’m unable to filter to avoid a little “pollution” from “…with contrast” studies.

  12. Ed McDonagh reporter

    I’ve been trying to work out how to move this forward @Luuk , @David Platten , and I’ve not got very far!

    I’m assuming that after the changes for Python 3, Django 2.2 and most importantly django-filters 2.2, we can’t easily slot in Luuk’s code from before? I also can’t exactly remember how it was implemented in the user interface?

    One of the things I have been looking at is whether there was a way of using a library of some sort to parse the string so that we don’t have to maintain the code to find all the brackets and operators etc. The closest I have come to is Luqum which can parse the query string and manipulate it, primarily for feeding into Elasticsearch but it might be useful. For instance if we had a querystring of ((model:"*flash" AND study:"Thorax^TAP_IV") OR (model:"*edge” AND study:”Thorax^TAP_120kV”)) AND NOT station:”CT Two” we could use Luqum to find all the operators and groups and wildcards and phrases and words and child nodes etc, and we can swap the name model for generalequipmentmoduleattr__manufacturer_model_nameetc. But I don’t know how to then feed the result into django-filter or filtering using standard Django filters.

    I also don’t know how we should present this to the user. In my mind, I’d like to have some sort of query builder - either a single box to choose a filter, then one pops up to choose a filter type then a value one pops up, and you can add groups and further filters; or a text input box with buttons to add appropriate filter and operator text. What I’d like to move away from is a list of 30 or more boxes down the right hand side, which really restricts how many filters you can offer because it becomes unusable in how much scrolling you need to do!

    What do you think? Am I being too radical? I’m keen not to delay version 1 release for too long, so we could go conservative in this release where filtering is concerned, but it would be a real improvement if we could crack it!

  13. Luuk

    Hi @Ed McDonagh , @David Platten ,

    I had a more or less working version in which the default filtering (as we have it now) was shown first. In the filter header you could click on “complex filtering” and then the side bar filtering disappeared and above the modality rows a method to make “complex” filter rules appeared. Here also an option was available to return to the default filtering.

    Issues, in my opinion, were:

    • The number of possible parameters is large, how to maintain overview for the user?

      • Make subsets with parameters belonging to each other?
    • You need somehow to set round brackets, this was a bit rudimentary in my implementation, but I didn’t know how to do this nicely.

    I can have a look if I can make this version available. But it might take till after ECR (but maybe I find some time before). It was of course also made for Python 2.7/Django 1.8.

    I also looked (back then) to other libraries, but I couldn’t find anything suitable.

  14. Log in to comment