Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ADaaS documentation. #64

Merged
merged 43 commits into from
Jul 24, 2024
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
abdb35b
Add ADaaS documentation.
samod Jul 9, 2024
b420bd6
Update fern/versions/public.yml
samod Jul 10, 2024
40aeee6
Update fern/versions/public.yml
samod Jul 10, 2024
0cb1eeb
Update fern/docs/pages/references/adaas/extracting_metadata.mdx
samod Jul 10, 2024
626a694
Update fern/docs/pages/references/adaas/adaas_overview.mdx
samod Jul 10, 2024
53f4ee5
Update fern/docs/pages/references/adaas/extracting_attachments.mdx
samod Jul 10, 2024
698f054
Update fern/docs/pages/references/adaas/adaas_overview.mdx
samod Jul 10, 2024
9c9069e
Move adaas/ up one level. Use hyphens in file names. Remove duplicate…
samod Jul 10, 2024
ac67db1
Modify titles to sentence case.
samod Jul 10, 2024
123a77b
Add Keyrings and Import sections to Concepts page.
samod Jul 15, 2024
21c2dea
Add Keyrings and Import sections to Concepts page.
samod Jul 15, 2024
902c913
Migrate entities definitions to separate pages.
samod Jul 15, 2024
5073650
Lowercase snap-ins. Remove terminology.
samod Jul 18, 2024
9db8cd8
Add Extract metadata update. Fix of capitalized terminology.
samod Jul 18, 2024
e89cbfb
Add delete phases. Replace Worker with snap-in.
samod Jul 18, 2024
ca048a8
Update fern/docs/pages/adaas/overview.mdx
samod Jul 18, 2024
940e2b9
Code review fixes.
samod Jul 18, 2024
7fde58b
Update fern/docs/pages/adaas/overview.mdx
samod Jul 22, 2024
71e116f
Update fern/docs/pages/adaas/overview.mdx
samod Jul 22, 2024
257d901
Update fern/docs/pages/adaas/extracting-attachments.mdx
samod Jul 22, 2024
a42552f
Update fern/docs/pages/adaas/extracting-attachments.mdx
samod Jul 22, 2024
8a92b12
Update fern/docs/pages/adaas/delete-phases.mdx
samod Jul 22, 2024
a0c203b
Update fern/versions/public.yml
samod Jul 22, 2024
f0e56e4
Remove latin abbrevations.
samod Jul 22, 2024
79f51de
Remove cluttering with EVENT_TYPE.
samod Jul 22, 2024
87c9371
Fix sentence cases.
samod Jul 22, 2024
4f86538
Update fern/docs/pages/adaas/getting-started.mdx
samod Jul 22, 2024
a1728f8
Update fern/docs/pages/adaas/getting-started.mdx
samod Jul 22, 2024
15f1de2
Update fern/docs/pages/adaas/getting-started.mdx
samod Jul 22, 2024
d6e2cde
First term use in italics, pt. 1.
samod Jul 22, 2024
10040b6
Modify the order of Getting Started.
samod Jul 22, 2024
55da773
Numbered list on extraction phases.
samod Jul 22, 2024
686b317
Correct the Extract metadata. Replace with .
samod Jul 22, 2024
8becb0a
Replace Artifacts with artifacts.
samod Jul 22, 2024
5473d59
Fix Data extraction section.
samod Jul 22, 2024
ad09872
Fix ATtachments extraction section.
samod Jul 22, 2024
f5959ac
Remove commas from JSON snippet.
samod Jul 22, 2024
61007ef
Fix a typo.
samod Jul 22, 2024
df9a5eb
Image edit.
samod Jul 22, 2024
89b1e33
First uses of terms in italics, pt. 2.
samod Jul 22, 2024
6c73f00
Apply suggestions from code review
samod Jul 23, 2024
91bdf92
Code review fixes.
samod Jul 23, 2024
b67952f
Add chef-cli public repo.
samod Jul 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions fern/docs/img/adaas/adaas-architecture.mmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
---
title: ADaaS architecture high-level overview
---
graph TB
externalSystem[External system]
worker([Worker])
s3interact[s3interact]
airdrop[Airdrop]
s3[(AWS S3)]

externalSystem <-- Get/create users, issues, ... --> worker

worker <-- Exchange artifact upload/download URLs --> s3interact
worker <-- Upload/download artifacts --> s3
worker <-- ADaaS messages and REST API --> airdrop


subgraph DevRev
s3interact <-- Prepare upload/download URL --> s3
airdrop <-- Download/upload artifacts --> s3
end
1 change: 1 addition & 0 deletions fern/docs/img/adaas/adaas-architecture.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
53 changes: 53 additions & 0 deletions fern/docs/img/adaas/adaas-extraction-phases.mmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
---
title: Extraction Phases
---
sequenceDiagram
actor user as UI
participant gw as Gateway
participant ad as Airdrop components
participant ee as External extractor

note over gw,ad: Extract External Sync Units
user ->> gw: Select import connection
gw ->> ad: Start External Sync Unit extraction
ad ->> ee: Start External Sync Unit extraction
ee -->> gw: List of External Sync Units
gw ->> ad: List of External Sync Units
ad ->> gw: List of External Sync Units
gw -->> user: Show available External Sync Units


note over gw,ad: Extract Metadata
ad ->> ee: Start Metadata extraction
ee -->> gw: List of extracted metadata artifacts
gw ->> ad: List of extracted metadata artifacts

note over gw,ad: Extract data
ad ->> ee: Start data extraction
ee -->> gw: List of extracted data artifacts
gw ->> ad: List of extracted data artifacts

note over ad: Transform and import data into DevRev

note over gw,ad: Extract attachments
ad ->> ee: Start attachment extraction
ee -->> gw: List of extracted attachments
gw ->> ad: List of extracted attachments

ad -->> gw: Finished import
gw -->> user: Show finished import and report

note over gw,ad: Delete data
user ->> gw: Delete import
gw ->> ad: Delete import
ad ->> ee: Delete data
ee -->> gw: Finished deleting data
gw ->> ad: Finished deleting data

note over gw,ad: Delete attachments
ad ->> ee: Delete attachments
ee -->> gw: Finished deleting attachments
gw ->> ad: Finished deleting attachments
note over ad: Deletes internal data
ad -->> gw: Finished deleting import
gw -->> user: Import deleted
1 change: 1 addition & 0 deletions fern/docs/img/adaas/adaas-extraction-phases.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
28 changes: 28 additions & 0 deletions fern/docs/pages/adaas/delete-phases.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
During the delete phases, the snap-in may clean up its state or other side effects in the third party systems.
In the most common extraction use cases, there is nothing to do and snap-ins may reply with the completion message.

## Data deletion

### Triggering event

Airdrop initiates the data delete phase when the sync is deleted from DevRev.
It is started by sending the worker a message with eventType: `EXTRACTION_DATA_DELETE`.

### Snap-in response

The worker must respond to Airdrop with a message with eventType `EXTRACTION_DATA_DELETE_DONE` when done or
`EXTRACTION_DATA_DELETE_ERROR` in case of an error.

## Attachments deletion


### Triggering event

Airdrop initiates the attachments delete phase when the Sync is deleted from DevRev.
Copy link
Contributor

@radovan-jorgic radovan-jorgic Jul 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't know this. Attachments deletion starts when import is deleted (and not visible through the UI)? Or should we say when data (and metadata) is deleted instead of when Sync is deleted.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FIxed.

It is started by sending the span-in a message with eventType: `EXTRACTION_ATTACHMENTS_DELETE`.

### Snap-in response

The snap-in must respond to Airdrop with a message with eventType `EXTRACTION_ATTACHMENTS_DELETE_DONE` when done,
or `EXTRACTION_ATTACHMENTS_DELETE_ERROR` in case of an error.

64 changes: 64 additions & 0 deletions fern/docs/pages/adaas/extracting-attachments.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
For the attachment extraction phase of the import process, the extractor has to upload each attachment to DevRev’s S3 using the `S3Interact` API.

## Triggering event

Airdrop initiates the Attachment Extraction by starting the snap-in with a message with eventType `EXTRACTION_ATTACHMENTS_START`
when transitioning to the Data Extraction phase.
radovan-jorgic marked this conversation as resolved.
Show resolved Hide resolved

During the Attachment Extraction phase,
the snap-in extracts attachments from the External System and uploads them as Artifacts to DevRev.

The snap-in must respond to Airdrop with a message with eventType `EXTRACTION_ATTACHMENTS_PROGRESS` together with an optional progress estimate and relevant Artifacts
when it extracts some data and the maximum snap-in runtime (12 minutes) has been reached.

The snap-in must respond to Airdrop with a message with eventType `EXTRACTION_ATTACHMENTS_DELAY` and specify a back-off time
when the extraction has been rate-limited by the External System and back-off is required.

In both cases, Airdrop starts the snap-in with a message with eventType `EXTRACTION_ATTACHMENTS_CONTINUE`.
The restart is immediate (in case of `EXTRACTION_ATTACHMENTS_PROGRESS`) or delayed
(in case of `EXTRACTION_ATTACHMENTS_DELAY`).

Once the Attachment Extraction phase is done, the snap-in must respond to Airdrop with a message with eventType `EXTRACTION_ATTACHMENTS_DONE`.

If Attachment Extraction failed, the snap-in must respond to Airdrop with a message with eventType `EXTRACTION_ATTACHMENTS_ERROR`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious if there is agreement regarding usage of word "eventType". Should we use "human language"? Something like "with event of type `EVENT_TYPE``?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant "event of type EXTRACTION_ATTACHMENTS_DONE" and not "event of type EVENT_TYPE EXTRACTION_ATTACHMENTS_DONE".

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So replace all "event of type EVENT_TYPE" with just "event of type" now.


## Snap-in response

After uploading an attachment or a batch of attachments, the extractor also has to prepare and upload a file specifying
the extracted and uploaded attachments.

It should contain the DevRev IDs of the extracted and uploaded attachments, along with the parent domain object ID
from the external system and the actor ID from the external system.

The uploaded artifact is structured like a normal artifact containing extracted data in JSON Lines format with specifying the
`ssor_attachment` as an item type.

## Examples

Here is an example of an SSOR attachment file:
```json
[
{
"id": {
"external": "don:core:dvrv-us-1:devo/1:artifact/1" // The DON of the uploaded artifact
},
"parent_id": {
"external": "12345", // ID of the parent in the source system
},
"actor_id": {
"external": "123456", // ID of the uploader in the source system
}
},
{
"id": {
"external": "don:core:dvrv-us-1:devo/1:artifact/2" // The DON of the uploaded artifact
},
"parent_id": {
"external": "12344", // ID of the parent in the source system
},
"actor_id": {
"external": "123457", // ID of the uploader in the source system
}
}
]
```
Loading
Loading