[indexer] File store backfiller #14058

larry-aptos · 2024-07-19T20:21:56Z

Description

Create a file store backfiller. This consumes data from an archive fullnode.

Type of Change

Which Components or Systems Does This Change Impact?

How Has This Been Tested?

Tested in testnet and mainnet; looks good.

Key Areas to Review

Checklist

I have read and followed the CONTRIBUTING doc
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I identified and added all stakeholders and component owners affected by this change as reviewers
I tested both happy and unhappy path of the functionality
I have made corresponding changes to the documentation

trunk-io · 2024-07-19T20:22:00Z

⏱️ 4h 1m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
test-target-determinator	39m	🟩 ⬜ 🟩 ⬜ 🟩 (+5 more)
execution-performance / single-node-performance	22m	🟩
general-lints	21m	🟩 ⬜ 🟩 ⬜ 🟩 (+8 more)
rust-cargo-deny	20m	🟩 ⬜ 🟩 ⬜ 🟩 (+8 more)
rust-move-tests	15m	🟩
rust-move-tests	14m	🟩
rust-move-tests	14m	🟩
check-dynamic-deps	13m	🟩 🟩 🟩 🟩 🟩 (+9 more)
rust-move-tests	10m	🟩
rust-move-tests	10m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	⬜
rust-move-tests	9m	🟩
rust-move-tests	6m	⬜
semgrep/ci	6m	🟩 🟩 🟩 🟩 🟩 (+9 more)
rust-move-tests	4m	⬜
file_change_determinator	3m	🟩 🟩 🟩 🟩 🟩 (+8 more)
file_change_determinator	3m	🟩 🟩 🟩 🟩 🟩 (+8 more)
file_change_determinator	2m	🟩 🟩 🟩 🟩 🟩 (+5 more)
permission-check	53s	🟩 🟩 🟩 🟩 🟩 (+8 more)
rust-move-tests	52s	⬜
permission-check	40s	🟩 🟩 🟩 🟩 🟩 (+9 more)
permission-check	38s	🟩 🟩 🟩 🟩 🟩 (+9 more)
permission-check	38s	🟩 🟩 🟩 🟩 🟩 (+8 more)
permission-check	30s	🟩 🟩 🟩 🟩 🟩 (+5 more)
determine-docker-build-metadata	22s	🟩 🟩 🟩 🟩 🟩 (+5 more)
Backport PR	8s	🟥 🟥
rust-move-tests	7s	⬜
permission-check	6s	🟩 🟩

🚨 1 job on the last run was significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
test-target-determinator	4m	5m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

yuunlimm · 2024-07-23T23:08:05Z

has this already finished backfilling in testnet ?

larry-aptos · 2024-07-23T23:09:06Z

nope; updated the description with current status.

ecosystem/indexer-grpc/indexer-grpc-file-store-backfiller/src/lib.rs

ecosystem/indexer-grpc/indexer-grpc-file-store-backfiller/src/metrics.rs

ecosystem/indexer-grpc/indexer-grpc-file-store-backfiller/src/processor.rs

larry-aptos · 2024-07-31T21:55:33Z

docker/builder/build-indexer.sh

@@ -16,7 +16,7 @@ cargo build --locked --profile=$PROFILE \
    -p aptos-indexer-grpc-file-store \
    -p aptos-indexer-grpc-data-service \
    -p aptos-nft-metadata-crawler-parser \
-    -p aptos-indexer-grpc-in-memory-cache-benchmark \


this is replaced by @just-in-chang's work.

github-actions · 2024-07-31T22:06:21Z

✅ Forge suite `framework_upgrade` success on `1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5` ==> `58e3e3252a80edcb4aa0d2532111465e613d5679`

Compatibility test results for 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 58e3e3252a80edcb4aa0d2532111465e613d5679 (PR)
Upgrade the nodes to version: 58e3e3252a80edcb4aa0d2532111465e613d5679
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1014.4425141849337 txn/s, submitted: 1016.4992798140538 txn/s, failed submission: 2.056765629120163 txn/s, expired: 2.056765629120163 txn/s, latency: 3032.905395359315 ms, (p50: 2200 ms, p90: 6000 ms, p99: 11400 ms), latency samples: 88780
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1129.4311199142942 txn/s, submitted: 1132.2464862289955 txn/s, failed submission: 2.8153663147011905 txn/s, expired: 2.8153663147011905 txn/s, latency: 2798.5703053593684 ms, (p50: 2100 ms, p90: 5100 ms, p99: 9200 ms), latency samples: 96280
5. check swarm health
Compatibility test for 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 58e3e3252a80edcb4aa0d2532111465e613d5679 passed
Upgrade the remaining nodes to version: 58e3e3252a80edcb4aa0d2532111465e613d5679
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1234.7654671315581 txn/s, submitted: 1236.4675857353234 txn/s, failed submission: 1.7021186037654403 txn/s, expired: 1.7021186037654403 txn/s, latency: 2838.9984935013786 ms, (p50: 2400 ms, p90: 4800 ms, p99: 7500 ms), latency samples: 101560
Test Ok

github-actions · 2024-07-31T22:06:56Z

✅ Forge suite `compat` success on `1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5` ==> `58e3e3252a80edcb4aa0d2532111465e613d5679`

Compatibility test results for 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 58e3e3252a80edcb4aa0d2532111465e613d5679 (PR)
1. Check liveness of validators at old version: 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5
compatibility::simple-validator-upgrade::liveness-check : committed: 7111.05806563108 txn/s, latency: 3840.072512139851 ms, (p50: 2900 ms, p90: 6900 ms, p99: 14800 ms), latency samples: 308900
2. Upgrading first Validator to new version: 58e3e3252a80edcb4aa0d2532111465e613d5679
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 6493.644415700335 txn/s, latency: 4303.446478755436 ms, (p50: 4900 ms, p90: 5300 ms, p99: 5500 ms), latency samples: 119560
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6333.589991856439 txn/s, latency: 5052.18254832051 ms, (p50: 5500 ms, p90: 6300 ms, p99: 6700 ms), latency samples: 213160
3. Upgrading rest of first batch to new version: 58e3e3252a80edcb4aa0d2532111465e613d5679
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7205.539650828151 txn/s, latency: 3781.7678009701604 ms, (p50: 4200 ms, p90: 4600 ms, p99: 4700 ms), latency samples: 136060
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 6694.803016338517 txn/s, latency: 4234.097260644463 ms, (p50: 4200 ms, p90: 4600 ms, p99: 6300 ms), latency samples: 258820
4. upgrading second batch to new version: 58e3e3252a80edcb4aa0d2532111465e613d5679
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 11151.67162582158 txn/s, latency: 2531.4380228136883 ms, (p50: 2500 ms, p90: 3400 ms, p99: 3900 ms), latency samples: 205140
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 9976.529380505908 txn/s, latency: 3166.95516355692 ms, (p50: 2700 ms, p90: 5000 ms, p99: 5800 ms), latency samples: 338720
5. check swarm health
Compatibility test for 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 58e3e3252a80edcb4aa0d2532111465e613d5679 passed
Test Ok

github-actions · 2024-07-31T22:29:04Z

✅ Forge suite `realistic_env_max_load` success on `58e3e3252a80edcb4aa0d2532111465e613d5679`

two traffics test: inner traffic : committed: 11994.194796883823 txn/s, latency: 3347.9232928406973 ms, (p50: 3300 ms, p90: 3600 ms, p99: 11400 ms), latency samples: 4560500
two traffics test : committed: 99.99145545444134 txn/s, latency: 3068.89156626506 ms, (p50: 2700 ms, p90: 3300 ms, p99: 10300 ms), latency samples: 1660
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.241, avg: 0.228", "QsPosToProposal: max: 0.368, avg: 0.290", "ConsensusProposalToOrdered: max: 0.340, avg: 0.328", "ConsensusOrderedToCommit: max: 0.847, avg: 0.753", "ConsensusProposalToCommit: max: 1.176, avg: 1.081"]
Max round gap was 1 [limit 4] at version 1264046. Max no progress secs was 8.102447 [limit 15] at version 1264046.
Test Ok

larry-aptos requested a review from a team as a code owner July 19, 2024 22:07

larry-aptos added CICD:build-images when this label is present github actions will start build+push rust images from the PR. CICD:build-indexer-images labels Jul 19, 2024

larry-aptos requested review from rtso and a team and removed request for rtso July 23, 2024 23:03

rtso requested changes Jul 24, 2024

View reviewed changes

larry-aptos force-pushed the create-a-file-store-backfiller branch from 42e1a3b to 9167c72 Compare July 24, 2024 22:56

larry-aptos requested a review from rtso July 25, 2024 00:03

larry-aptos enabled auto-merge (squash) July 31, 2024 21:05

add file store backfiller.

0080112

larry-aptos force-pushed the create-a-file-store-backfiller branch from 9167c72 to 0080112 Compare July 31, 2024 21:10

yuunlimm approved these changes Jul 31, 2024

View reviewed changes

ecosystem/indexer-grpc/indexer-grpc-file-store-backfiller/src/processor.rs Outdated Show resolved Hide resolved

larry-aptos added 2 commits July 31, 2024 14:22

fix logging.

5665f20

udpate the readme.

58e3e32

larry-aptos requested a review from ying-w July 31, 2024 21:50

This comment has been minimized.

Sign in to view

ying-w approved these changes Jul 31, 2024

View reviewed changes

larry-aptos commented Jul 31, 2024

View reviewed changes

larry-aptos requested review from rtso and removed request for rtso July 31, 2024 21:58

rtso approved these changes Jul 31, 2024

View reviewed changes

This comment has been minimized.

Sign in to view

larry-aptos merged commit 81984fa into main Jul 31, 2024
53 checks passed

larry-aptos deleted the create-a-file-store-backfiller branch July 31, 2024 22:52

larry-aptos added a commit that referenced this pull request Jul 31, 2024

add file store backfiller. (#14058)

03e1d26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[indexer] File store backfiller #14058

[indexer] File store backfiller #14058

larry-aptos commented Jul 19, 2024 •

edited

Loading

trunk-io bot commented Jul 19, 2024 •

edited

Loading

yuunlimm commented Jul 23, 2024

larry-aptos commented Jul 23, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

larry-aptos Jul 31, 2024

This comment has been minimized.

github-actions bot commented Jul 31, 2024

github-actions bot commented Jul 31, 2024

This comment has been minimized.

github-actions bot commented Jul 31, 2024

[indexer] File store backfiller #14058

[indexer] File store backfiller #14058

Conversation

larry-aptos commented Jul 19, 2024 • edited Loading

Description

Type of Change

Which Components or Systems Does This Change Impact?

How Has This Been Tested?

Key Areas to Review

Checklist

trunk-io bot commented Jul 19, 2024 • edited Loading

yuunlimm commented Jul 23, 2024

larry-aptos commented Jul 23, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

larry-aptos Jul 31, 2024

Choose a reason for hiding this comment

This comment has been minimized.

github-actions bot commented Jul 31, 2024

✅ Forge suite framework_upgrade success on 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 58e3e3252a80edcb4aa0d2532111465e613d5679

github-actions bot commented Jul 31, 2024

✅ Forge suite compat success on 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 58e3e3252a80edcb4aa0d2532111465e613d5679

This comment has been minimized.

github-actions bot commented Jul 31, 2024

✅ Forge suite realistic_env_max_load success on 58e3e3252a80edcb4aa0d2532111465e613d5679

larry-aptos commented Jul 19, 2024 •

edited

Loading

trunk-io bot commented Jul 19, 2024 •

edited

Loading

✅ Forge suite `framework_upgrade` success on `1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5` ==> `58e3e3252a80edcb4aa0d2532111465e613d5679`

✅ Forge suite `compat` success on `1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5` ==> `58e3e3252a80edcb4aa0d2532111465e613d5679`

✅ Forge suite `realistic_env_max_load` success on `58e3e3252a80edcb4aa0d2532111465e613d5679`