Create a "Provider Manifest" endpoint for operating dates/down time dates/service suspension

[tybaltspark]

Edit by @schnuerle - turning this issue into a 'Provider Manifest' issue which includes what you'd like to see here, similar to system_information in GBFS and complimentary to agency's Policy Requirements. Original issue title was: "Provide the operating date ranges within the Provider API so that we know when to query data"

---

Is your feature request related to a problem? Please describe.

We are struggling to know the collection of date ranges when a provider has been operating in the city.
It makes it difficult to query efficiently the Provider API. It makes it difficult for the city to have a good visibility on this key information.

Describe the solution you'd like

I wonder whether this info could be provided within the Provider API as part of an info endpoint. The operating zone could also be provided by the operator this way, as well as the data availability. Operators may indeed have no data historically for certain periods they were operating due to technical issues.

Is this a breaking change

No, it's not a breaking change. It's additive

Impacted Spec

• provider

Describe alternatives you've considered

Asking directly the city or the provider. Trial & guess day by day in the provider API ingestion.

Additional context

NA

[dirkdk]

As a provider I have this question. Are you talking about officially operating, since start of the permit program? Or are you talking about certain days where a provider by themselves decides not to operate? Like because of weather conditions, onset of winter, protests, other internal reasons?

[tybaltspark]

I'm talking about the official operating start date. This would be a good start.
Other range dates of operations would be great too, since otherwise we actually don't know whether this is an API issue or a range of dates where the operator withdrew its fleet

[schnuerle]

Hi @tybaltspark we could talk about this on tomorrow's WG call.

[schnuerle]

Hello @tybaltspark we didn't get to this last week as you know, but we will talk about it on today's WG call.

[jiffyclub]

For me at Populus I'm most interested in the range of dates that an API will return 200 responses. For snow days or seasonal hiatuses I think the APIs can return 200 responses with no trips/status changes to indicate there was nothing to report then. But having some official calendar of dates when the API will definitely not return 200 responses saves us from querying for those.

On the other hand, I can imagine a city observer also wanting an official record of seasonal hiatuses so they aren't wondering where all the scooters went.

[quicklywilliam]

@jiffyclub I think that's right – there really are two use cases here: exposing richer data to cities, and making the API easier to query. For both use cases, I'd love to see us work closely with GBFS so that city-facing and user-facing data are in lock-step.

For the former use case, we discussed using GBFS's system_calendar, making enhancements as mentioned by @mplsmitch on the call.

For the latter use case, we discussed potentially adding some kind of top-level system_suspensions object to the trips and status endpoints.

[schnuerle]

Per the call today, we came up with a few conclusions and follow-ups.

1. Utility - it did seem useful to everyone to know in the feed when operations started or when there are current, past, or future gaps in operation. For cities, explaining gaps in data is useful. For cities and third parties, knowing the start data of operations is good to reduce trial and error discovery and over-querying the data. For providers it could reduce the amount questions they get asking where the data is or saying the feed is broken.
2. GBFS offers some of this already with system_information (start date) and system_calendar and system_hours. It seems better if cities/third parties could use this existing data feed than ask providers to duplicate the info or create a similar feed in MDS.
3. The system_calendar however does not yet support multiple date ranges and other flexibility. There is a new proposal 289 opened last month with an accompanying document that would expand this. But it's not clear to me if this proposal meets the needs here since it seems to talk about hours for stations (not the whole operator system) and removed the system_hours. Maybe @mplsmitch could let us know more.
4. Another useful suggestion was to add some info at the head of an existing endpoint like /trips that has 1) a text and a link to the correct GBFS feed with more information and 2) possibly the start date of operations.
5. The start date could alternatively be included in the proposed Requirements API, at an operator level for the city, or even more granular per operator, per MDS version, and per API endpoint if that is important.

[mplsmitch]

GBFS currently addresses a number of these issues. h/t to @quicklywilliam for pointing this out.

• The date that the system began operations is contained in start_date in the system_information endpoint.
• Dates of operation are currently covered by system_calendar.json which actually does support multiple date ranges, each contained in a calendar object. We have a proposal to change the way that dates and hours of operation are handled. This would provide some additional capabilities with regard to dates. Per GBFS governance we need support from both data producers and consumers for the proposal to become part of the spec. Please weigh in on it if this interests you.
• Regarding when an API returns a 200 response, GBFS views seasonal shutdowns or disruptions of service like severe weather events, pandemics or insurrections as temporary so APIs should continue to return 200 responses so long as the operator plans to return to service in the future. More on how that should be handled here.
• Disruptions of service where vehicles are not available for rent are reflected in station based systems via is_renting in station_status.json and/or in dockless by the presence of available vehicles listed in free_bike_status.json.

One other thing regarding querying and dates to keep in mind is that generally GBFS is forward looking so if the hope is to use this to explain gaps in MDS data by pointing to times and dates in the past, that would require operators to continue to publish those past dates in their feeds, or for consumers to have captured and stored those data in the moment.

[robinef]

Agree with @mplsmitch,

Actually we need 2 separated things :

• (A) the official operating dates (when a provider has officially and physically run a service in a particular city or context)
• (B) a way to quickly understand what are the time range of data availability on the provider API (digitally available)
  Usually both informations can be the same, but it happens that for some technical reason, the provider was not able to record or process trips and status_changes informations.
  Currently, with MDS 0.4 and higher it's hard to detect for which dates we can make queries. There is no guidance on which dates will likely return 400, and which ones will return 200 with valid data, so we need to guess and check for arbitrary time ranges. If we instead had an explicit set of start/end date-ranges over which the /trips and /status_changes endpoints will return 200 responses, provided in the MDS provider spec, then the process of fetching historical data will be greatly simplified.

Thus, the two different set of date-ranges (A) and (B) would then be used in tandem in the following way:

1. I need to know if the operator was physically operating for this day (A)
2. I need to know if I can query data for this day (B)
3. If the previous steps 1(A) and 2(B) were OK, and I got 0 trips, I can thus infer that there was an external factor (ex: snow storm)

[robinef]

@schnuerle, any plan to further discuss this issue ?

[schnuerle]

@robinef This definitely warrants more discussion, which we can continue to do here.

More details about a specific provider's operating dates, down time dates, and services suspension dates for a specific mode or program could be provided in the discussed future Provider Manifest endpoint, a complimentary system to the agency policy program Requirements #608 endpoint. This endpoint could also contain other information about the Provider's operations, like operating area geography #639, etc.

For now this seems to be solved as good as it can be with the current and future system_information file from GBFS. And since with 1.2.0 agencies and require providers to provide specific GBFS endpoints and optional fields, which can be specified in the Requirements endpoint.

If were to add start and end dates to /trips or /status_changes, that would 1) be duplicative of what could be done in GBFS 2) be a breaking change if required and have to wait for 2.0 (or it could be optional in 1.2 but I don't think you'd see much adoption) and 3) not solve for the more complex situations where you'd like multiple date ranges for disruptions and operation holds to be communicated.

What does everyone else think about this?

[jiffyclub]

I like that "provider manifest" idea, I don't think this kind of system metadata belongs in the trips and status changes endpoints.

[schnuerle]

From #466:

As an MDS-consuming agency, it would be valuable to have the ability to query an endpoint that provides information about system problems with a provider's MDS feed or system in general.
Example problems could be:

• IOT/comms issues with overall system.
• IOT/comms issues with vehicle(s)
• Other problems that aren't vehicle-specific status changes, but overall system problems that are not easily conveyed in current feed.

Possibly, the GBFS endpoint 'system_alerts.json' could be used, but seems a bit hacky and not intended for communicating problems provider <---> agency, but rather to the public.

[schnuerle]

We can revisit this for MDS 2.1 or 3.0 if people want to chime in about their need for it and suggest details on how it could work. I think it would work differently that the new GBFS manifest.json, and more like Policy Requirements, but coming from and including details from operators to agencies.