Method to Exclude some Provider Fields from Response

[schnuerle]

Is your feature request related to a problem? Please describe.

It is possible that cities or third parties may not want or need certain fields returned.

Describe the solution you'd like

Add an option through the API request that allows some fields to be removed from the response.

An example solution could be the inclusion of a new parameter, e.g. exclude, that could have a list of fields the city does not want to have returned in the response. If a city didn’t want trips, then they can add /trips?exclude=route to the API call. If they don’t want trip or fee data, they could add /trips?exclude=route,standard_cost,actual_cost. This list of possible options can be defined in the spec.

Not sure if the resulting fields would be completely removed, or be returned empty, or with a special value. I prefer the first option.

This also has a benefit of getting only the data you need for the task at hand, reducing the returned data file size, and reducing processing on the provider’s side.

Is this a breaking change

A breaking change would require consumers or implementors of the API to modify their code for it to continue to function (ex: renaming of a required field or the change in data type of an existing field). A non-breaking change would allow existing code to continue to function (ex: addition of an optional field or the creation of a new optional endpoint).

• Yes, breaking

Impacted Spec

For which spec is this feature being requested?

• provider - for now let's just focus on Provider

Describe alternatives you've considered

This is a proposed solution and discussion point to PR #504.

Additional context

I have seen this in other APIs, so there is some precedent here.

[vperron]

@schnuerle I think you meant routes ? trips is a whole endpoint, I'm not sure it can exclude itself.

I support this idea, even though I'm a little worried this would only land in a quite remote MDS version instead of a first version clarifying which fields are "optional" and what it means, which could get us going with 1.0 until this is settled.

[schnuerle]

@vperron Yes that's what I meant. Updated the original to clarify, more like this: /trips?exclude=route,standard_cost,actual_cost

[Retzoh]

@schnuerle if I understand correctly, this means there would be two factors determining the fields included in the responses:

• providers may choose to include or exclude optional fields depending on data availability or other reasons
• cities may ask for some fields, optional or required, to be excluded from the responses

Is that correct?

The way I understood #504 is to make it possible for providers to implement the Provider API without having to transmit routes at all if the data is judged too sensitive. The approach here is a bit different since an agency may or may not ask for routes. But if it does, the routes would still have to be transmitted.

[schnuerle]

Yes @Retzoh that is right.

'Optional' fields are up to the city/provider to decide if they are going to receive/send them or support them, likely through some conversations. Required fields would come through by default, unless exclude is used.

'Exclude' fields would be handled programmatically through the logic of the API on the provider's side. It would be up to the cities/third parties to exclude certain fields when making a call, but the resulting removal would be automatic.

Side note any field could also be labeled as "beta" which means it's meant for testing and is not fully supported.

[vperron]

I'm wondering how mandatory the ability to mark "removable" or "beta" fields should be made, in order to maybe ease the implementation effort it would put on providers.

[thekaveman]

This almost feels like a GraphQL type interface for the API. If we're going to go down this route, perhaps we should look into adopting that style?

I will also add that this type of approach (selective include/exclude of fields via query) lessens the impact and relevance of the JSON schemas and may make endpoint validation more difficult.

From the privacy perspective, I can understand the position of cities/agencies not wanting to receive potentially sensitive information. If said city/agency was going to make the determination in an API query, couldn't they also make the same determination in what to do with the response? i.e. just don't further process or store the potentially sensitive information, let it die in-memory.

[dirkdk]

Making responses dynamic based on request parameters makes it virtually impossible to precompute the responses. With going to hourly parameters like end_time for trips we went into the directions of bulk ETL by consumers which would allow providers to optimize their apis by serving from static files. Having lots of possible responses would do away with that work.

One option I would see is doing the same for trips as we did for status_changes and events: historical data for a year by hourly parameters, two week data by a multitude of parameters including min_end_time and max_end_time.

[quicklywilliam]

It came up in today's call that we could define a new Policy rule that would allow agencies to express data sharing requirements. The advantages of this approach might be:

• Allow certain fields or endpoints to become "optional" in the sense of not being required by the spec, but still allow them to be "required" in the sense of the city explicitly requiring them of providers
• Allow for some ability for the city to change their data sharing Policy while still providing consistency at request-time. This would allow providers to pre-compute requests.
• Allow for the development of a robust and explicit future best practice that addresses how to enact policies changes around access to data
• Allow operators to allocate access properly so that a city that does not want/need a given field/endpoint does not in fact have access to it

We're happy to put together a proposal for that if desired.

[marie-x]

I am not excited about this idea broadly, for the same reason I don't like using Policy for Jurisdictions. Feels like we're overloading Policy. If anything, I would consider expanding Jurisdictions to a more generalized discovery mechanism. However, happy to engage on a specific proposal.

[johnclary]

I would love to see functionality in MDS. Currently any /provider consumer is forced to handle telemetry data, even if they don't want it.

Event better, I'd vote for flipping this from "ability to exclude" to "ability to include", and force an explicit request for optional fields. And let's make route optional as proposed in #504.

[schnuerle]

This is being considered in MDS 1.2.0 in the #608 Issue and #646 PR with agencies being able to specific what endpoints and fields the require, and which they don't want to have at all

[schnuerle]

Making responses dynamic based on request parameters makes it virtually impossible to precompute the responses. With going to hourly parameters like end_time for trips we went into the directions of bulk ETL by consumers which would allow providers to optimize their apis by serving from static files. Having lots of possible responses would do away with that work.

@dirkdk : getting back to your thoughts here on the burden of this for providers. It seems that our proposed solution now with #646 will still allow providers to precompute responses - they would just be on a per-agency basis (maybe like how they are now for some agencies requesting certain optional data).

[schnuerle]

Completed with #646! Thanks to everyone who worked on this solution to address including optional fields and excluding required fields per a city's use cases and needs.