Handeling end_date search

[magnarem]

When we search using date filter in Drupal, we will need to alter the solr-query such that we also return documents which do not have an end date, even if the user have an end date filter. This is needed so that the user also get the datasets with open end date that are in the selected time range.

The solution will be to rewrite the solr-query generated by drupal by moving the end date filter query to the query parameter and add extra query for open end dates.

As of now a query with start and end date will look somthing like this:

{
   q=full_text:osisaf
   fq=temporal_extent_start_date:[<start_date> TO *],
   fq=temporal_extent_end_date:[ * TO <end_date>]
}

this will have to be rewritten to

{
   q=(full_text:osisaf AND temporal_extent_end_date:[* TO <end_date>]) OR (full_text:osisaf NOT temporal_extent_end_date:[* TO *])
   fq=temporal_extent_start_date:[<start_date> TO *],
 }

[magnarem]

After testing a bit, I have found out the following.
It is possible to change the metsis_search view, so that the queries mentioned in this issue will be generated by the search_api view. This allow us to get all the datasets that have a start/end-date or open end date, that are covered by the search start/end date filter.
But whit this configuration, then products like sentinel with very short timespan (ie. start and end date close togheter), will not be in the search results, since returning back all products that togheter falls inside the search range, will require the solr query to be different for including those results.

I experimented with Solr DateRangeField, earlier, but could not find out how to add daterange field to the products missing an end date. Now i figured it is possible to add infinite end date to those products when indexing.
With the Solr DateRangeField it is possible to do more advanced query conjunctions.

The date range query for date range fields supports the following.
Intersects (default), Contains, Within. Example fq={!field f=dateRange op=Contains}[2013 TO 2018]

Incorperating this field, we can expose this operator to the users, so they can have more control over results returned using a date range search.

This means we have to index everything again. First I will add some records to the testcore to see if this helps us with our problem.

[steingod]

But also that we would have to modify the indexing or? How do you add that functionality?

[magnarem]

Yes. in the tosolr()- function the following lines have to be added. (tested locally using the metno/solr-indexer):

            if "temporal_extent_end_date" in mydict:
                logger.debug('Creating daterange with end date')
                mydict['temporal_extent_period_dr'] = '[' + str(mydict["temporal_extent_start_date"]) + ' TO ' + str(mydict["temporal_extent_end_date"]) + ']'
            else:
                logger.debug('Creating daterange with open end date')
                mydict['temporal_extent_period_dr'] = '[' + str(mydict["temporal_extent_start_date"]) + ' TO *]'
            logger.debug("Temporal extent date range: %s", str(mydict['temporal_extent_period_dr']))

the `temporal_extent_period_dr' field is already in the solr schema.

I will test how drupal search_api_solr works with the daterange, and how the exposed forms for date search will work with this field.

If this solution do what we want, then we can implement it, and do a re-index.

[magnarem]

search_api_solr works with the solr daterange field. However it as not implemented the search operators (contains,intersects,within), so this have to be added programatically.

In the mean time, I have implemented some other date-filtering using the start and end dates we have already indexed. It should work more like we want to. Now the date filter will be applied to the elements/children search, and the children count will be updated for the date filter. The changes has now been applied to metsis-staging.met.no and nbs-staging.met.no

[magnarem]

Implemented date_range queries for metsis-drupal. As of now this is only implemented in simple_search, but can be added to the main search also.

It is possible to test this on metsis-staging.met.no. Also possible to test on nbs-staging.met.no, but it will only work for the latest products added the few previous days.

For NBS i think it will be nice to also add support for date AND time, not just date for this filter.

[magnarem]

Simple search updated with date range search on staging sites

[steingod]

On Wed, 2024-03-27 at 05:17 -0700, magnarem wrote: Simple search updated with date range search on staging sites — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

This looks fine, but I have some questions. - In the first view there is no map? This first shows when I push search? Shouldn't it be in the first view as well? - I am not sure on the handling of parent/child. It seems all children are listed instead of the parent and the listing of children in the next step? The current approach is quite voluminous and will mask much of the relevant data. Aside from this buttons looks really nice. Concerning download I am wondering whether we should rephrase HTTP download to "Direct download" or similar, but this goes back to our MMD document as well. Same for OPeNDAP and OGC WMS, these are more access than download (but semantics). MEPS datasets masked my view so I never got to test WMS, but assume this is a TDS issue and not a catalogue issue... -- Dr. Øystein Godøy Norwegian Meteorological Institute  P.O.BOX 43, Blindern, N-0313 OSLO, Norway Ph: (+47) 9802 4433

[magnarem]

The main search is also now updated with better search using the start and end date. I recon this might be closed.