Airbyte API: Request for Input on Configuration API deprecation

[nataliekwong]

We want to better understand how our users rely on our APIs to operate Airbyte and ensure we provide a great experience in using Airbyte, and are looking for your feedback to improve the experience.

A brief history:

• We released the Airbyte API in April 2023. The Airbyte API is a reliable, easy-to-use, and versioned interface for programmatically controlling the Airbyte platform with full support from the Airbyte team. The SDKs and Terraform provider all support the Airbyte API, and will not support the config API.
• We released the Config API in the early days of Airbyte and stopped supporting it in April 2023 in favor of the Airbyte API.

While the config API is a convenient way to interact with Airbyte, it is not designed to be a public-facing API and will be formally deprecated this year. The correct API to use with Airbyte is the Airbyte API. Our goal is to ensure that you have the ability to manage and run core Airbyte functionality without resorting to functionality only available in the config API.

Our ask to you:

What endpoints or data are you missing from the Airbyte API? Share with us (using the template below) what you are currently using the config API for today that you think we should support so that we can better understand your use case and build a roadmap for the API.

Examples

Please follow this template in your response:

Config API endpoint: /v1/source_definition/list
Endpoint exist in the Airbyte API? No
Use Case: I need to list all available source definitions because my app will display the connector to the final user

Config API endpoint: /v1/sources/list
Endpoint exist in the Airbyte API? Yes
Use Case: The current Airbyte API doesn't bring me all the information I need to ….

For now, we are only looking for input on usage of the Configuration API and will keep this open until July 2024. In the future, we’ll open a separate discussion on expanding API endpoints. We look forward to hearing from you!

[rensoostenbachBL]

Config API endpoint: v1/connector_builder_projects/list
Endpoint exist in the Airbyte API? No
Use Case: I need to list all connector builder projects because I am using the Configuration API to deploy custom connectors.

[rensoostenbachBL]

Config API endpoint: v1/connector_builder_projects/create
Endpoint exist in the Airbyte API? No
Use Case: I need to create a connector builder project because I am using the Configuration API to deploy custom connectors.

[rensoostenbachBL]

Config API endpoint: v1/connector_builder_projects/publish
Endpoint exist in the Airbyte API? No
Use Case: I need to publish a connector builder project because I am using the Configuration API to deploy custom connectors.

[rensoostenbachBL]

Config API endpoint: v1/declarative_source_definitions/list_manifests
Endpoint exist in the Airbyte API? No
Use Case: I need to list all declaraative source definitions because I am using the Configuration API to deploy custom connectors.

[rensoostenbachBL]

Config API endpoint: v1/declarative_source_definitions/update_active_manifest
Endpoint exist in the Airbyte API? No
Use Case: I need to update an active manifest because I am using the Configuration API to deploy custom connectors.

[rensoostenbachBL]

Config API endpoint: v1/declarative_source_definitions/create_manifest
Endpoint exist in the Airbyte API? No
Use Case: I need to create a manifest because I am using the Configuration API to deploy custom connectors.

[rensoostenbachBL]

Config API endpoint: /v1/sources/discover_schema
Endpoint exist in the Airbyte API? No
Use Case: I need to create connections that require me to discover the schema.

[tomlynchRNA]

Config API endpoint: /v1/sources/check_connection
Endpoint exist in the Airbyte API? No
Use Case: We want to test the connection to a source before we try creating a connection with it.

[jlia0]

You can make a POST request to the Airbyte cloud API instead, with the same bearer token from the developer portal.

[tomlynchRNA]

Is that available in open source?

[jlia0]

I believe so

[superdosh]

Config API endpoint: /v1/connections/get
Endpoint exist in the Airbyte API? No
Use Case: I need the stream names and headers in order to show to our users. The Airbyte API has a get connection, but it doesn't include sufficient information to get the stream names and the headers for each stream. The config APIs call has the syncCatalog which contains that information.

[superdosh]

Config API endpoint: /v1/sources/check_connection
Endpoint exist in the Airbyte API? No
Use Case: I need to make sure the source is still valid before doing anything with a source. If it's not valid, I need to have our users go through a flow again to create a source. This is not available at all in the Airbyte API.

[superdosh]

Config API endpoint: /v1/jobs/get_debug_info
Endpoint exist in the Airbyte API? No
Use Case: I need to provide progress updates on a user's sync job, but the best available are the logs which come back from this debug endpoint. I can parse these logs from the config api debug endpoint to provide that. Would love if there was a cleaner way to do this.

[lraivio]

Config API endpoint: /v1/scheduler/destinations/check_connection
Endpoint exist in the Airbyte API? No
Use Case: I need to check if the configuration for a destination is correct and we can connect to the destination database before proceeding with creating the destination.

[tomlynchRNA]

Config API endpoint: /v1/source_definitions/list_latest
Endpoint exist in the Airbyte API? No
Use Case: I need to get a list of available source definitions with ID from airbyte, to display a list of available source connectors to the end user of my app

Config API endpoint: /v1/source_definition_specifications/get
Endpoint exist in the Airbyte API? No
Use Case: I need to fetch a specific source definition to get the json schema spec, so I can show a form based on the spec the user chose from the list

[evehmane]

Config API endpoint: /v1/source_definitions/list_for_workspace
Endpoint exist in the Airbyte API? No
Use Case: I need to list all available source definitions for workspace, to display a list of available source connectors to the end user of my app

[evehmane]

Config API endpoint: /v1/source_definitions/create_custom
Endpoint exist in the Airbyte API? No
Use Case: I need to create a custom source definition for the given workspace, to make custom source connectors available to the end user

[evehmane]

Config API endpoint: /v1/source_definitions/update
Endpoint exist in the Airbyte API? No
Use Case: I need to update a custom source definition to upgrade (or downgrade) the docker image version, so that the end user has the best source connector version available

[evehmane]

Config API endpoint: /v1/sources/list
Endpoint exist in the Airbyte API? Yes
Use Case: The current Airbyte API doesn't bring me source definition id for custom sources. The sourceType is returned for non-custom sources but the equivalent info for custom sources is not returned. I need this to know the type (source definition id) of all sources.