-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add ADaaS documentation. #64
Changes from 15 commits
abdb35b
b420bd6
40aeee6
0cb1eeb
626a694
53f4ee5
698f054
9c9069e
ac67db1
123a77b
21c2dea
902c913
5073650
9db8cd8
e89cbfb
ca048a8
940e2b9
7fde58b
71e116f
257d901
a42552f
8a92b12
a0c203b
f0e56e4
79f51de
87c9371
4f86538
a1728f8
15f1de2
d6e2cde
10040b6
55da773
686b317
8becb0a
5473d59
ad09872
f5959ac
61007ef
df9a5eb
89b1e33
6c73f00
91bdf92
b67952f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
--- | ||
title: ADaaS architecture high-level overview | ||
--- | ||
graph TB | ||
externalSystem[External system] | ||
worker([Worker]) | ||
s3interact[s3interact] | ||
airdrop[Airdrop] | ||
s3[(AWS S3)] | ||
|
||
externalSystem <-- Get/create users, issues, ... --> worker | ||
|
||
worker <-- Exchange artifact upload/download URLs --> s3interact | ||
worker <-- Upload/download artifacts --> s3 | ||
worker <-- ADaaS messages and REST API --> airdrop | ||
|
||
|
||
subgraph DevRev | ||
s3interact <-- Prepare upload/download URL --> s3 | ||
airdrop <-- Download/upload artifacts --> s3 | ||
end |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,53 @@ | ||
--- | ||
title: Extraction Phases | ||
--- | ||
sequenceDiagram | ||
actor user as UI | ||
participant gw as Gateway | ||
participant ad as Airdrop components | ||
participant ee as External extractor | ||
|
||
note over gw,ad: Extract External Sync Units | ||
user ->> gw: Select import connection | ||
gw ->> ad: Start External Sync Unit extraction | ||
ad ->> ee: Start External Sync Unit extraction | ||
ee -->> gw: List of External Sync Units | ||
gw ->> ad: List of External Sync Units | ||
ad ->> gw: List of External Sync Units | ||
gw -->> user: Show available External Sync Units | ||
|
||
|
||
note over gw,ad: Extract Metadata | ||
ad ->> ee: Start Metadata extraction | ||
ee -->> gw: List of extracted metadata artifacts | ||
gw ->> ad: List of extracted metadata artifacts | ||
|
||
note over gw,ad: Extract data | ||
ad ->> ee: Start data extraction | ||
ee -->> gw: List of extracted data artifacts | ||
gw ->> ad: List of extracted data artifacts | ||
|
||
note over ad: Transform and import data into DevRev | ||
|
||
note over gw,ad: Extract attachments | ||
ad ->> ee: Start attachment extraction | ||
ee -->> gw: List of extracted attachments | ||
gw ->> ad: List of extracted attachments | ||
|
||
ad -->> gw: Finished import | ||
gw -->> user: Show finished import and report | ||
|
||
note over gw,ad: Delete data | ||
user ->> gw: Delete import | ||
gw ->> ad: Delete import | ||
ad ->> ee: Delete data | ||
ee -->> gw: Finished deleting data | ||
gw ->> ad: Finished deleting data | ||
|
||
note over gw,ad: Delete attachments | ||
ad ->> ee: Delete attachments | ||
ee -->> gw: Finished deleting attachments | ||
gw ->> ad: Finished deleting attachments | ||
note over ad: Deletes internal data | ||
ad -->> gw: Finished deleting import | ||
gw -->> user: Import deleted |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
During the delete phases, the snap-in may clean up its state or other side effects in the third party systems. | ||
In the most common extraction use cases, there is nothing to do and snap-ins may reply with the completion message. | ||
|
||
## Data deletion | ||
|
||
### Triggering event | ||
|
||
Airdrop initiates the data delete phase when the sync is deleted from DevRev. | ||
It is started by sending the worker a message with eventType: `EXTRACTION_DATA_DELETE`. | ||
|
||
### Snap-in response | ||
|
||
The worker must respond to Airdrop with a message with eventType `EXTRACTION_DATA_DELETE_DONE` when done or | ||
`EXTRACTION_DATA_DELETE_ERROR` in case of an error. | ||
|
||
## Attachments deletion | ||
|
||
|
||
### Triggering event | ||
|
||
Airdrop initiates the attachments delete phase when the Sync is deleted from DevRev. | ||
It is started by sending the span-in a message with eventType: `EXTRACTION_ATTACHMENTS_DELETE`. | ||
|
||
### Snap-in response | ||
|
||
The snap-in must respond to Airdrop with a message with eventType `EXTRACTION_ATTACHMENTS_DELETE_DONE` when done, | ||
or `EXTRACTION_ATTACHMENTS_DELETE_ERROR` in case of an error. | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,64 @@ | ||
For the attachment extraction phase of the import process, the extractor has to upload each attachment to DevRev’s S3 using the `S3Interact` API. | ||
|
||
## Triggering event | ||
|
||
Airdrop initiates the Attachment Extraction by starting the snap-in with a message with eventType `EXTRACTION_ATTACHMENTS_START` | ||
when transitioning to the Data Extraction phase. | ||
radovan-jorgic marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
During the Attachment Extraction phase, | ||
the snap-in extracts attachments from the External System and uploads them as Artifacts to DevRev. | ||
|
||
The snap-in must respond to Airdrop with a message with eventType `EXTRACTION_ATTACHMENTS_PROGRESS` together with an optional progress estimate and relevant Artifacts | ||
when it extracts some data and the maximum snap-in runtime (12 minutes) has been reached. | ||
|
||
The snap-in must respond to Airdrop with a message with eventType `EXTRACTION_ATTACHMENTS_DELAY` and specify a back-off time | ||
when the extraction has been rate-limited by the External System and back-off is required. | ||
|
||
In both cases, Airdrop starts the snap-in with a message with eventType `EXTRACTION_ATTACHMENTS_CONTINUE`. | ||
The restart is immediate (in case of `EXTRACTION_ATTACHMENTS_PROGRESS`) or delayed | ||
(in case of `EXTRACTION_ATTACHMENTS_DELAY`). | ||
|
||
Once the Attachment Extraction phase is done, the snap-in must respond to Airdrop with a message with eventType `EXTRACTION_ATTACHMENTS_DONE`. | ||
|
||
If Attachment Extraction failed, the snap-in must respond to Airdrop with a message with eventType `EXTRACTION_ATTACHMENTS_ERROR`. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Just curious if there is agreement regarding usage of word "eventType". Should we use "human language"? Something like "with event of type `EVENT_TYPE``? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Perfect. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I meant "event of type There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So replace all "event of type |
||
|
||
## Snap-in response | ||
|
||
After uploading an attachment or a batch of attachments, the extractor also has to prepare and upload a file specifying | ||
the extracted and uploaded attachments. | ||
|
||
It should contain the DevRev IDs of the extracted and uploaded attachments, along with the parent domain object ID | ||
from the external system and the actor ID from the external system. | ||
|
||
The uploaded artifact is structured like a normal artifact containing extracted data in JSON Lines format with specifying the | ||
`ssor_attachment` as an item type. | ||
|
||
## Examples | ||
|
||
Here is an example of an SSOR attachment file: | ||
```json | ||
[ | ||
{ | ||
"id": { | ||
"external": "don:core:dvrv-us-1:devo/1:artifact/1" // The DON of the uploaded artifact | ||
}, | ||
"parent_id": { | ||
"external": "12345", // ID of the parent in the source system | ||
}, | ||
"actor_id": { | ||
"external": "123456", // ID of the uploader in the source system | ||
} | ||
}, | ||
{ | ||
"id": { | ||
"external": "don:core:dvrv-us-1:devo/1:artifact/2" // The DON of the uploaded artifact | ||
}, | ||
"parent_id": { | ||
"external": "12344", // ID of the parent in the source system | ||
}, | ||
"actor_id": { | ||
"external": "123457", // ID of the uploader in the source system | ||
} | ||
} | ||
] | ||
``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Didn't know this. Attachments deletion starts when import is deleted (and not visible through the UI)? Or should we say when data (and metadata) is deleted instead of when Sync is deleted.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FIxed.