Clarify no pagination on static provider endpoints /trips and /status_changes

[hyperknot]

Explain pull request

It clarifies that the static provider endpoints (/trips and /status_changes) should not have server-side pagination.

Is this a breaking change

A breaking change would require consumers or implementors of the API to modify their code for it to continue to function (ex: renaming of a required field or the change in data type of an existing field). A non-breaking change would allow existing code to continue to function (ex: addition of an optional field or the creation of a new optional endpoint).

• No, not breaking

Impacted Spec

Which spec(s) will this pull request impact?

• provider

Additional context

As one of the contributors who proposed the original static-file backed provider endpoint idea for /trips and /status_changes endpoints, I believe it's important that there should be no server-side pagination on the short time intervals.

Longer explanation is copied from my original post here:

---

One more important point I'd like to raise is the removal of pagination from static-backed endpoints.

1. Pagination currently is the most "broken" part of the MDS specs. I work on MDS acquisition at Populus, we work with 10+ operators' MDS API, and pagination code is by far the most complex part of our MDS acquisition infrastructure. Most operators have slightly different behavior, and errors are not uncommon.

Some operators do not support pagination at all but require us to query data in hourly blocks (exactly like the new static endpoints), some do not add links to the last page, some require custom query parameters and the list goes on.

2. Since on the static endpoints, the time-period is fixed, I went ahead and analyzed all our historical data.

=> The average size for an hourly block of trips data, gzipped is 100 kB.
=> The maximum, extreme value for an hourly block of trips data is 7 MB, but this is very rare.

Since we are talking about servers in datacenters communicating with each other, even sending 7 MB in a single HTTPS request should be a non-issue today. Status changes are tiny in comparison, there is no point in analyzing them.

=> As a conclusion, I recommend removing pagination from the static-backed endpoints (provider /trips and /status_changes).

[claassistantio]

All committers have signed the CLA.

[jiffyclub]

Looking at the provider spec I don't see anything that requires the trips and status changes endpoints to be implemented via static files, they could choose to implement it as an API endpoint that queries a database.

In fact I'm little unclear on how we imagine operators will implement this with static files. Do these endpoints support redirect responses?

All that said, we could still disallow pagination in the trips and status changes endpoints, but I'd phrase it as:

The /trips and /status_changes APIs should not use pagination. If providers choose to use pagination for the /events endpoint the pagination must comply with the JSON API specification.

[jiffyclub]

I'll update this PR with my language from above.

[hyperknot]

Thanks, it is much cleaner this way!

[thekaveman]

The language change looks great! As we've requested in other PRs, please do not include additional cleanup type fixes elsewhere in the document that aren't related to the specific change you are proposing. It makes tracing history and review much more complicated (in general, probably not in this specific case). We are more than happy to accept simple self-contained cleanup PRs for things like spacing etc.

[hyperknot]

@thekaveman you mean the end-of-line empty space removal in @jiffyclub's commit? It was probably done by his text editor I guess.

[thekaveman]

@thekaveman you mean the end-of-line empty space removal in @jiffyclub's commit?

In this case, yes that is what I am referring to.

[hyperknot]

PR updated with removed formatting.

[thekaveman]

May I request one final change:

-- The `/trips` and `/status_changes` APIs should not use pagination.
++ The `/trips` and `/status_changes` APIs must not use pagination.

This aligns with our usage of "must" to imply "required for valid implementation".

[hyperknot]

May I request one final change

Updated

[jiffyclub]

Yes, sorry, my editor automatically trims trailing white space.

[thekaveman]

Thank you!