Swagger for Server Sent Events

[bmeck]

It would be nice if Swagger supported Server Sent Events. Which is a Uni-directional alternative for streaming content. This still would fall within REST to my knowledge (in particular due to last seen ID). A more concrete example can be seen at HTML5Rocks

semi-related : #55

[buremba]

+1

[wiltshiretom]

+1!

[casualjim]

I think you can kind of represent it (without tooling support though) like this

paths:
  /:
    get:
      produces: ["text/event-stream"]
      responses:
        default:
          description: example of defining events for sse 
          schema:
            type: object
            x-events:
              added:
                type: object
                properties:
                  message:
                    type: string
                  createdAt:
                    type: string
                    format: date-time
              updated:
                type: object
                  properties:
                    previous:
                      type: string
                    newValue:
                      type: string
                    updatedAt:
                      type: string
                      format: date-time

[elbrujohalcon]

+1

[artem-korolev]

Well. But why to represent it in swagger without tooling support? Does it have any use cases?

[casualjim]

step 1, make the proposal to start the discussion

[ikitommi]

👍

[mrhornsby]

👍

[casualjim]

parent: #586

[jroper]

I don't think this should be limited to uni directional streaming. While of course, web browsers and many existing HTTP implementations require a request to be fully consumed/produced before sending/reading the response, HTTP itself does not impose this limitation, nor does anything about the SSE protocol, and so there's no reason why two services using an HTTP client/server implementation that supports it can't speak SSE both up and down at the same time.

Another reason why it shouldn't be limited to uni directional streaming is that this and a possible WebSockets streaming extension could share a lot in common, it would be a shame to have two completely different ways of specifying an SSE stream vs a WebSocket event stream.

[dolmen]

@jroper:

• With HTTP you don't have a bidirectional link over a single TCP connection: the client speaks first and the server answers.
• A link over WebSockets is not HTTP anymore, so I think it is out of scope for Swagger/OpenAPI.

Personally, for a text oriented protocol over a bidirectionnal link (TCP or WebSocket) I would use JSON RPC 2.0 for fully asynchronous dialog, not Event Source.

[jroper]

With HTTP you don't have a bidirectional link over a single TCP connection: the client speaks first and the server answers.

That's not actually true. There is nothing in the HTTP spec that says that a server has to fully consume a client request before it starts sending the response, in fact, the spec actually has provisions that allow for this to be done, from section 8.2.3:

If an origin server receives a request that does not include an Expect request-header field with the "100-continue" expectation, the request includes a request body, and the server responds with a final status code before reading the entire request body from the transport connection, then the server SHOULD NOT close the transport connection until it has read the entire request, or until the client closes the connection.

This implies that the server is allowed to send a response before the entire request body is read. So bidirectional communication over a single TCP connection is definitely allowed. A number of asynchronous HTTP servers (eg akka-http, Netty, and Play framework, which I'm a maintainer of) do explicitly support this too.

[dolmen]

Please stop those polluting +1 and use instead built-in reactions on the original submission.
Do you know that you can even delete your own useless comments?

[jroper]

Getting everyone in your company to +1 an issue is the dumbest strategy ever, as a maintainer of a number of open source projects, when I see lots of noise on an issue, and then I see most of that noise is from one company, it leads me to believe that that company is the only company that wants the feature, which leads me to deprioritise the issue. Please streamdata.io, there are serious people here who are serious about wanting to see this become part of the spec and want to have serious discussion about it, the only thing you're doing is hurting our efforts.

[darrelmiller]

Now that V3 supports OneOf in JSON Schema, I'm not exactly sure what it is that people want more from OpenAPI in order to describe SSE payloads. Perhaps if someone could describe the challenges currently faced then we would have a better idea as to what needs enhancing.

[allandenis38]

Sorry @jroper, our intent was not to hurt your efforts at all. In order to show that you are also serious people ;-) we will remove those useless +1 and use reactions instead.

[jroper]

@darrelmiller SSE events have a few properties, and these are not JSON properties, they are part of the SSE protocol. Most interesting to OpenAPI is the event and data properties. The data property is the payload, this would typically contain the event data as JSON, or maybe as a plain text string. The event field however contains the events type. So, to support idiomatic SSE, OpenAPI would have to allow specifying multiple event types, most likely with a schema for each.

A non idiomatic approach could be taken where the OneOf JSON schema support is used with no event type, but even then there are a few questions that should probably be answered by the spec to allow tooling to generate interoperable implementations. For example, how should an OpenAPI spec indicate that SSE is being used? By listing text/event-stream in the produces? What if the spec lists both text/event-stream and application/json in the produces? How should a tool interpret that? Because in one case, a client code generator will need to generate a method that returns the object directly, and in the other it would need to return a stream. And if text/event-stream is listed in produces, how does the client then know that the contents of text/event-stream of the data payload is application/json, or text/plain?

[dolmen]

In my use case I want to return text/event-stream for status 200, but application/json for other statuses (structured error reporting for bad usage of the API or internal errors).
For now I'm specifying this by cheating with the headers property of a Response object. But I do not expect Swagger clients to properly handle that.

      produces: ["text/event-stream","application/json"]
      responses:
        '200':
          description: Stream is starting
          x-produces: ["text/event-stream"]
          headers:
            'Content-Type':
              type: string
              enum: ["text/event-stream"]
        default:
          $ref: '#/responses/DefaultError'

[whitlockjc]

What is really being discussed (in my opinion) is how one would use OAS to describe a streaming API, regardless of which technology (SSE, WebSockets, ...) you're using. Since all of these technologies all use HTTP, the OAS already has the primitives in place so we just need to decide is at what level we want to add support to the OAS, and what that looks like.

In a recent discussion with @darrelmiller, the main question I needed answered was to the following: "When describing a streaming API via OAS, is the response schema suppose to describe each "chunk" of the response or for the response as a whole?" This came up when looking at the OAS for the Kubernetes API. For their OAS usage, the response schema was written to describe each chunk in the stream as a JSON Object. To me, this is intuitive but from a tooling perspective, there is no way for the tooling to know that this is a streaming API and would potentially have an issue where it could fail to validate appropriately in all situations.

My hope in answering this is for consistency, so OAS users know that there is a well-defined approach for documenting streaming APIs.

[dolmen]

@whitlockjc For SSE we want to be able to describe multiple types of chunks (called events), each being identified by an event name. So if events are sent are JSON, the spec format should allow to describe the schema of the message for each event name.

[darrelmiller]

@dolmen Would it be sufficient to be able to indicate that the payload schema only applies to a "chunk" and then use oneOf in JSON Schema to define all of the events?
The other option might be to have a chunkSchema property that is an array of schema objects. The end result is effectively the same though. At least by using oneOf a JSON Schema validator could validate all the chunks using a single schema.

[allandenis38]

@darrelmiller In case of SSE, an event contains non-JSON properties and this is true for every event/chunk. So applying a JSON validation to the entire event would not make sense but rather the value of an SSE property.

[ahopkins]

Why closed?

[darrelmiller]

There have a been a couple of suggestions of how to model SSE in OpenAPI. SSE is just a long polling HTTP GET with a payload that supports sending discrete elements of data.

[twhittock]

#396 (comment) is this the canonical way to represent SSE, then? And we bypass any tooling to write the handler manually?

[philsturgeon]

After years of inactivity I invited the community interested in SSE to get involved in May 2021. Nobody did.

Then in August 2021 @darrelmiller invited the community interested in SSE to participate. Nobody did.

Be the change you want to see. 🙌🏻

https://github.com/OAI/OpenAPI-Specification#participation

[twhittock]

I'm not sure what it means for me to be the change I want to see @philsturgeon - is there a specific issue with considering the proposal in #396 (comment) canonical for SSE, as I suggested?

The participation section you link to states that we should "Check the issues and pull requests to see if someone has already documented your idea or feedback on the specification", which surely is what is happening when people contribute to this issue, but it keeps getting closed by members of the project.

Perhaps this falls under "Not all feedback can be accommodated and there may be solid arguments for or against a change being appropriate for the specification" - if so, why not explain that Open API will not support SSE and give those solid arguments against it?

I genuinely don't know what you expect to see.

[philsturgeon]

I was pointing you to the weekly calls:

The TSC holds weekly web conferences to review open pull requests and discuss open issues related to the evolving OpenAPI Specification. Participation in weekly calls and scheduled working sessions is open to the community. You can view the entire OpenAPI technical meeting calendar online.

Swing by, talk about what you want to do, and disucss a potential proposa.

There are examples above of how it could be done to start from, so perhaps that can be the basis of a x-foo draft proposal. We can talk that through on the weekly call.

[giowe]

any update?

[twhittock]

No, I attended one of the calls which was cut off early due to key people not attending and no proposals were entertained. I get the feeling it's really out of scope for this project, so I just gave up and rolled my own.

[giowe]

@twhittock do you have something I can see to share ?

[shalberd]

@philsturgeon this is very disappointing and probably more of a communication- and approach problem. We, too, are using Server Sent Events. It is a clear part of http and API frameworks. The facts that it cannot be described in OpenAPI is deplorable. Then again, I just saw that something as basic as mTLS authentication mentioning (has been around forever) was only recently added as a field to SecuritySpec, so maybe I am expecting too much of OpenAPI spec.
Will explore asyncAPI for Server Sent Events. It's a pity two different frameworks are necessary for in our case one API set.

[jroper]

@shalberd Using words like "deplorable" and "disappointing" is not the way to engage in an open source community. Please adjust your tone. @philsturgeon has invited anyone that wants to take the lead on putting this into the spec to join the weekly calls and do so. That includes you. Open source projects only work when community members that need features step up and engage to help provide those features. When you tell an open source project they are disappointing and deplorable for not adding a feature that they are happy to accept from a contributor, but don't volunteer to contribute yourself, you're actually pointing the finger at yourself, calling yourself disappointing and deplorable, since you're unwilling to do the work yourself.

[hasufell]

I think the signals from the maintainers of this project are not too clear. First off: this ticket has been closed as "completed". Why? Where's the "call for help" exactly?

[jroper]

It was not closed as "completed". The GitHub close reason feature went live in May 2022, with the preview starting for a few projects in March 2022. This was closed in February 2022. Completed is just the default reason. All issues that were closed prior to when the close reasons feature was added to GitHub have the reason of completed. For example, this issue from 2014 was also closed as "completed", 8 years prior to the ability to specify a close reason in GitHub.

As for the calls for help, just read through the comment history. Here's one, here's another and here's yet another.

I'm not part of the OpenAPI project, but I have led the development of multiple open source projects, and been very involved in several specification style projects like this. Looking over this issue, honestly, there's not a lot of serious interest in it. It takes you 2 minutes to write a comment saying you want this feature. That's not serious interest. Serious interest means submitting serious proposals, attending meetings where you sell those proposals, talking to multiple stakeholders in the project to understand what the different concerns are and addressing that. No one's doing that.

The OpenAPI spec we have today has been built off the backs of that kind of engagement for a decade. It's hard work, working with many different stakeholders all having their different opinions, going back and forth, coming up with solutions, reworking, reworking, reworking, incorporating lots of different feedback. And the result is that we have a quality spec that we all benefit from. People commenting on an issue saying this feature is important, that's just noise. People just turning up to one call and stating their pet feature, that's just noise too. This kind of half-hearted engagement happens a lot, but helps nothing. That's probably why the maintainers have closed this issue. Unless there's someone out there that is willing to do the many hours of work of leading this feature and getting it into the spec, no one can claim that it's important enough to be in the spec.

[hasufell]

@jroper I'm maintaining popular open source projects as well. I know how things work. There was no call for help here.

A call for help is when maintainers say "yeah, great idea, we need your help" or "important topic, let's discuss this asap".

As a maintainer, it's easy to get burned out and dismissive, because people are mostly just complaining. But you have to understand that you're hoping for someone else to potentially dedicate their free time to your project.

Small things can make a difference here. E.g. the issue has not been reopened. That in itself is a signal to potential contributors, whether that was intended or not.

[jroper]

Disagree. The maintainers have yet to be convinced of the importance of this issue, that's very clear from the comments they've made. They're not hoping for anyone to dedicate their free time to do this because they don't think it's important enough. But they have invited us to convince them otherwise. The fact that no one has done the work to show that has given them their answer. Until someone does, I don't see why they would reopen the issue.

[hasufell]

@jroper then don't be surprised when people express their frustration due to the lack of interest and engagement of the maintainers.

I'm maintaining a spec myself. And I encourage contributors to write even radical proposals, while being very upfront with the expectations: most of them will likely not be accepted. But it's good to have them anyway.

At any rate, I think most of us here lost interest in contributing.

[philsturgeon]

As I’ve said several times in this thread (but it’s been buried under lots of long replies), if you’re interested in this topic please join a weekly call and take the lead on making it happen.

I’m not really involved right now, busy running a climate charity unpaid, but as far as I know there’s nobody on the OpenAPI team who’s being paid to work on things you want. You can work on things you want, if you want.

Alternatively I am available for hire. If you’d like to put me on a retainer I can pioneer efforts to get this change discussed. My rate is $200/hr.

Otherwise it’s on you. Good luck!

[olawalejuwonm]

Is there any resolution as regards this? Will AsyncAPI be combined to OpenAPI so we can have server side event documented with it?

[nikojpapa]

Here is an example of a way I did it back in 2020. This was based off of v4.3.1. I don't remember exactly what I did, so I just remade, in a new repo, the relevant changes to the files that I think were pertinent. I think there is a similar structure in the latest version, so this can likely be adapted as needed.

This is using the new SSE in the typescript-fetch language, but you can change the mustache file that is relevant to you.

https://github.com/nikojpapa/openapi-generator/compare/003165c2c2..03081c9

You would then set the response in the API definition as something like the following:

  responses:
    "200":
      description: success
      content:
        text/event-stream:
          schema:
            $ref: "..."

[ihakh]

I used it to update the live chart
so in my case the data has a fix structure and api can promise to have a fix structure