Add ADaaS documentation. (#64)

* Add ADaaS documentation. * Update fern/versions/public.yml Move the documentation one level up. Co-authored-by: Ben Colborn <[email protected]> * Update fern/versions/public.yml Use hyphens instead of underscores in directory names. Co-authored-by: Ben Colborn <[email protected]> * Update fern/docs/pages/references/adaas/extracting_metadata.mdx Co-authored-by: Ben Colborn <[email protected]> * Update fern/docs/pages/references/adaas/adaas_overview.mdx Co-authored-by: Ben Colborn <[email protected]> * Update fern/docs/pages/references/adaas/extracting_attachments.mdx Co-authored-by: Ben Colborn <[email protected]> * Update fern/docs/pages/references/adaas/adaas_overview.mdx Co-authored-by: Ben Colborn <[email protected]> * Move adaas/ up one level. Use hyphens in file names. Remove duplicate page in index. * Modify titles to sentence case. * Add Keyrings and Import sections to Concepts page. * Add Keyrings and Import sections to Concepts page. * Migrate entities definitions to separate pages. * Lowercase snap-ins. Remove terminology. * Add Extract metadata update. Fix of capitalized terminology. * Add delete phases. Replace Worker with snap-in. * Update fern/docs/pages/adaas/overview.mdx Co-authored-by: radovan-jorgic <[email protected]> * Code review fixes. * Update fern/docs/pages/adaas/overview.mdx Co-authored-by: Ben Colborn <[email protected]> * Update fern/docs/pages/adaas/overview.mdx Co-authored-by: Ben Colborn <[email protected]> * Update fern/docs/pages/adaas/extracting-attachments.mdx Co-authored-by: Ben Colborn <[email protected]> * Update fern/docs/pages/adaas/extracting-attachments.mdx Co-authored-by: Ben Colborn <[email protected]> * Update fern/docs/pages/adaas/delete-phases.mdx Co-authored-by: Ben Colborn <[email protected]> * Update fern/versions/public.yml Co-authored-by: Ben Colborn <[email protected]> * Remove latin abbrevations. * Remove cluttering with EVENT_TYPE. * Fix sentence cases. * Update fern/docs/pages/adaas/getting-started.mdx Co-authored-by: Ben Colborn <[email protected]> * Update fern/docs/pages/adaas/getting-started.mdx Co-authored-by: Ben Colborn <[email protected]> * Update fern/docs/pages/adaas/getting-started.mdx Co-authored-by: Ben Colborn <[email protected]> * First term use in italics, pt. 1. * Modify the order of Getting Started. * Numbered list on extraction phases. * Correct the Extract metadata. Replace with . * Replace Artifacts with artifacts. * Fix Data extraction section. * Fix ATtachments extraction section. * Remove commas from JSON snippet. * Fix a typo. * Image edit. * First uses of terms in italics, pt. 2. * Apply suggestions from code review Co-authored-by: Erazem Kokot <[email protected]> * Code review fixes. * Add chef-cli public repo. --------- Co-authored-by: Ben Colborn <[email protected]> Co-authored-by: radovan-jorgic <[email protected]> Co-authored-by: Erazem Kokot <[email protected]>
devrev · Jul 24, 2024 · 7a14a70 · 7a14a70
1 parent 4044474
commit 7a14a70
Show file tree

Hide file tree

Showing 16 changed files with 710 additions and 4 deletions.
diff --git a/fern/docs/img/adaas/adaas-architecture.mmd b/fern/docs/img/adaas/adaas-architecture.mmd
@@ -0,0 +1,21 @@
+---
+title: ADaaS architecture high-level overview
+---
+graph TB
+externalSystem[External system]
+worker([Worker])
+s3interact[s3interact]
+airdrop[Airdrop]
+s3[(AWS S3)]
+
+	externalSystem <-- Get/create users, issues, ... --> worker
+
+	worker <-- Exchange artifact upload/download URLs --> s3interact
+	worker <-- Upload/download artifacts --> s3
+	worker <-- ADaaS messages and REST API --> airdrop
+
+
+	subgraph DevRev
+	s3interact <-- Prepare upload/download URL --> s3
+	airdrop <-- Download/upload artifacts --> s3
+	end
diff --git a/fern/docs/img/adaas/adaas-architecture.svg b/fern/docs/img/adaas/adaas-architecture.svg
diff --git a/fern/docs/img/adaas/adaas-extraction-phases.mmd b/fern/docs/img/adaas/adaas-extraction-phases.mmd
@@ -0,0 +1,53 @@
+---
+title: Extraction Phases
+---
+sequenceDiagram
+  actor user as UI
+  participant gw as REST API Gateway
+  participant ad as Airdrop components
+  participant ee as External extractor
+
+	note over gw,ad: Extract External Sync Units
+	user ->> gw: Select import connection
+	gw ->> ad: Start External Sync Unit extraction
+	ad ->> ee: Start External Sync Unit extraction
+	ee -->> gw: List of External Sync Units
+	gw ->> ad: List of External Sync Units
+	ad ->> gw: List of External Sync Units
+	gw -->> user: Show available External Sync Units
+
+
+	note over gw,ad: Extract Metadata
+	ad ->> ee: Start Metadata extraction
+	ee -->> gw: List of extracted metadata artifacts
+	gw ->> ad: List of extracted metadata artifacts
+
+	note over gw,ad: Extract data
+	ad ->> ee: Start data extraction
+	ee -->> gw: List of extracted data artifacts
+	gw ->> ad: List of extracted data artifacts
+
+	note over ad: Transform and import data into DevRev
+
+	note over gw,ad: Extract attachments
+	ad ->> ee: Start attachment extraction
+	ee -->> gw: List of extracted attachments
+	gw ->> ad: List of extracted attachments
+
+	ad -->> gw: Finished import
+	gw -->> user: Show finished import and report
+
+	note over gw,ad: Delete data
+	user ->> gw: Delete import
+	gw ->> ad: Delete import
+	ad ->> ee: Delete data
+	ee -->> gw: Finished deleting data
+	gw ->> ad: Finished deleting data
+
+	note over gw,ad: Delete attachments
+	ad ->> ee: Delete attachments
+	ee -->> gw: Finished deleting attachments
+	gw ->> ad: Finished deleting attachments
+	note over ad: Deletes internal data
+	ad -->> gw: Finished deleting import
+	gw -->> user: Import deleted
diff --git a/fern/docs/img/adaas/adaas-extraction-phases.svg b/fern/docs/img/adaas/adaas-extraction-phases.svg
diff --git a/fern/docs/pages/adaas/attachments-extraction.mdx b/fern/docs/pages/adaas/attachments-extraction.mdx
@@ -0,0 +1,64 @@
+For the attachment extraction phase of the import process, the extractor has to upload each attachment to DevRev’s S3
+using the `S3Interact` API.
+
+## Triggering event
+
+Airdrop initiates the attachment extraction by starting the snap-in with a message with an event of type `EXTRACTION_ATTACHMENTS_START`.
+This is done after the data extraction, transformation and loading into DevRev are completed.
+
+During the attachment extraction phase,
+the snap-in extracts attachments from the external system and uploads them as artifacts to DevRev.
+
+The snap-in must respond to Airdrop with a message with an event of type `EXTRACTION_ATTACHMENTS_PROGRESS` together with an optional progress estimate and relevant artifacts
+when it extracts some data and the maximum snap-in run time (12 minutes) has been reached.
+
+The snap-in must respond to Airdrop with a message with an event of type `EXTRACTION_ATTACHMENTS_DELAY` and specify a back-off time
+when the extraction has been rate-limited by the external system and back-off is required.
+
+In both cases, Airdrop starts the snap-in with a message with an event of type `EXTRACTION_ATTACHMENTS_CONTINUE`.
+The restart is immediate in case of `EXTRACTION_ATTACHMENTS_PROGRESS`, or delayed
+in case of `EXTRACTION_ATTACHMENTS_DELAY`.
+
+Once the attachment extraction phase is done, the snap-in must respond to Airdrop with a message with an event of type `EXTRACTION_ATTACHMENTS_DONE`.
+
+If attachment extraction fails the snap-in must respond to Airdrop with a message with an event of type `EXTRACTION_ATTACHMENTS_ERROR`.
+
+## Snap-in response
+
+After uploading an attachment or a batch of attachments, the extractor also has to prepare and upload a file specifying
+the extracted and uploaded attachments.
+
+It should contain the DevRev IDs of the extracted and uploaded attachments, along with the parent domain object ID
+from the external system and the actor ID from the external system.
+
+The uploaded artifact is structured like a normal artifact containing extracted data in JSON Lines (JSONL) format and requires specifying `ssor_attachment` as the item type.
+
+## Examples
+
+Here is an example of an SSOR attachment file:
+```json
+[
+	{
+		"id": {
+			"external": "don:core:dvrv-us-1:devo/1:artifact/1" // The DON of the uploaded artifact
+		},
+		"parent_id": {
+			"external": "12345" // ID of the parent in the source system
+		},
+		"actor_id": {
+			"external": "123456" // ID of the uploader in the source system
+		}
+	},
+	{
+		"id": {
+			"external": "don:core:dvrv-us-1:devo/1:artifact/2" // The DON of the uploaded artifact
+		},
+		"parent_id": {
+			"external": "12344" // ID of the parent in the source system
+		},
+		"actor_id": {
+			"external": "123457" // ID of the uploader in the source system
+		}
+	}
+]
+```
diff --git a/fern/docs/pages/adaas/data-attachments-deletion.mdx b/fern/docs/pages/adaas/data-attachments-deletion.mdx
@@ -0,0 +1,28 @@
+During the deletion phases, the snap-in may clean up its state or other side effects in the third party systems.
+In the most common extraction use cases, there is nothing to do and snap-ins may reply with the completion message.
+
+## Data deletion
+
+### Triggering event
+
+Airdrop initiates the data deletion phase when the import is deleted in the DevRev app.
+It is started by sending the worker a message with an event of type `EXTRACTION_DATA_DELETE`.
+
+### Snap-in response
+
+The worker must respond to Airdrop with a message with an event of type `EXTRACTION_DATA_DELETE_DONE` when done or
+`EXTRACTION_DATA_DELETE_ERROR` in case of an error.
+
+## Attachments deletion
+
+
+### Triggering event
+
+Airdrop initiates the attachments deletion phase when an import is deleted from the DevRev app, after the data deletion has completed.
+It is started by sending the snap-in a message with an event of type `EXTRACTION_ATTACHMENTS_DELETE`.
+
+### Snap-in response
+
+The snap-in must respond to Airdrop with a message with an event of type `EXTRACTION_ATTACHMENTS_DELETE_DONE` when done,
+or `EXTRACTION_ATTACHMENTS_DELETE_ERROR` in case of an error.
+