add current filter to leaflets endpoint - #431
Conversation
| is_current = RawSQL( | ||
| """ | ||
| EXISTS ( | ||
| SELECT 1 | ||
| FROM jsonb_array_elements(ballots) AS ballot | ||
| WHERE substring( | ||
| ballot->>'ballot_paper_id' from '\\d{4}-\\d{2}-\\d{2}$' | ||
| )::date >= %s | ||
| ) | ||
| """, | ||
| (current_from,), | ||
| output_field=BooleanField(), | ||
| ) |
There was a problem hiding this comment.
This is not the easiest to read query. However we don't store the poll date in the JSON as a standalone field so in order to do this efficiently we have to parse it out of the ballot ids with regex.
There was a problem hiding this comment.
This feels like it might end up using a lot of CPU. Can we investigate either:
- Also filtering on the upload date...anything that was uploaded more than....a year ago can't be in a current election, I assume, so we can eliminate the majority of the data in the database that's being parsed for JSON? (I assume the postgres planner will figure this out)
- Add a date field to the leaflet model: more work and maybe not an ideal direction of travel, but would make this a lot faster.
There was a problem hiding this comment.
I think I will have to come back to this one after the move, but..
-
I like the idea of filtering on upload date to reduce the initial search space more 👍
-
At the moment, a
Leafletobject has aJSONFieldcalledballots, which is an array of (ballot) objects with propertiesballot_paper_idand.. something else. The name or title of the ballot basically. This means that in principle a Leaflet can be related to ballots taking place on multiple different dates. I expect this would be irregular, but it is possible.
When you say "Add a date field to the leaflet model", do you mean storing a single date on the Leaflet object (in which case, we have to decide what to do in the case I just described) or do you mean change the shape of the JSON object so that each ballot in the array also has an explicit poll_date? The second one would still need a deep json object query, but should be more efficient than parsing the ballot id with regex.
There was a problem hiding this comment.
Happy to leave this until later. On dates: understood, but I think we can use ArrayField(models.DateField()) and still query faster than regexing JSON.
2 participants
Refs https://app.asana.com/1/1204880536137786/project/1204880927741389/task/1217118151808079?focus=true
LLM assisted
The definition of "current" in this context is:
I think that's about the best we can do.
This will also want a follow up PR to consume it on the WCIVF side.