Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LIVE-2174 Dynamic ERC20 CAL #1288

Merged
merged 44 commits into from
Oct 26, 2022
Merged

LIVE-2174 Dynamic ERC20 CAL #1288

merged 44 commits into from
Oct 26, 2022

Conversation

henri-ly
Copy link
Contributor

@henri-ly henri-ly commented Sep 15, 2022

📝 Description

We sometimes need to add erc20 tokens out of a release cycle.

There was some issues since we are using electron with two thread, the renderer and the internal doesn't share the same tokens list so we need to initialize / preload the data on both side.

We also work on a automatic PR to be able to push update from the crypto-assets repository toward an S3 (still WIP waiting for the bucket)
https://github.com/LedgerHQ/ledger-live/actions/runs/3061492352

We also need to work on mobile to be sure everything is working

❓ Context

✅ Checklist

  • Test coverage
  • Atomic delivery
  • No breaking changes

📸 Demo

🚀 Expectations to reach

Please make sure you follow these Important Steps.

Pull Requests must pass the CI and be internally validated in order to be merged.

@vercel
Copy link

vercel bot commented Sep 15, 2022

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Updated
live-common-tools ❌ Failed (Inspect) Oct 26, 2022 at 11:20AM (UTC)
3 Ignored Deployments
Name Status Preview Updated
ledger-live-github-bot ⬜️ Ignored (Inspect) Oct 26, 2022 at 11:20AM (UTC)
native-ui-storybook ⬜️ Ignored (Inspect) Oct 26, 2022 at 11:20AM (UTC)
react-ui-storybook ⬜️ Ignored (Inspect) Oct 26, 2022 at 11:20AM (UTC)

@changeset-bot
Copy link

changeset-bot bot commented Sep 15, 2022

🦋 Changeset detected

Latest commit: 850aa79

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 6 packages
Name Type
@ledgerhq/hw-app-eth Minor
live-mobile Minor
ledger-live-desktop Minor
@ledgerhq/live-common Patch
@ledgerhq/live-cli Patch
live-common-tools Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions bot added automation CI/CD stuff common Has changes in live-common desktop Has changes in LLD ledgerjs Has changes in the ledgerjs open source libs labels Sep 15, 2022
@github-actions
Copy link

github-actions bot commented Sep 15, 2022

@henrily-ledger

Screenshots: ❌

It seems this PR contains screenshots that are different from the base branch.
If you are sure all those changes are correct, you can comment on this PR with /generate-screenshots to update those screenshots.

Make sure all the changes are correct before running the command, as it will commit and push the new result to the PR.

linux

Actual Diff Expected
v3-onboarding-complete-actual v3-onboarding-complete-diff v3-onboarding-complete-expected
v3-onboarding-complete-actual v3-onboarding-complete-diff v3-onboarding-complete-expected

@codecov
Copy link

codecov bot commented Sep 19, 2022

Codecov Report

Base: 47.87% // Head: 45.10% // Decreases project coverage by -2.76% ⚠️

Coverage data is based on head (850aa79) compared to base (d51a138).
Patch coverage: 79.34% of modified lines in pull request are covered.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1288      +/-   ##
===========================================
- Coverage    47.87%   45.10%   -2.77%     
===========================================
  Files          691      635      -56     
  Lines        30376    26757    -3619     
  Branches      7968     7273     -695     
===========================================
- Hits         14541    12069    -2472     
+ Misses       14622    13549    -1073     
+ Partials      1213     1139      -74     
Flag Coverage Δ
test 45.10% <79.34%> (-2.77%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
libs/ledger-live-common/src/env.ts 74.50% <ø> (ø)
...kages/hw-app-eth/src/services/ledger/loadConfig.ts 100.00% <ø> (ø)
...live-common/src/families/ethereum/signOperation.ts 23.80% <25.00%> (+0.63%) ⬆️
...live-common/src/families/ethereum/modules/erc20.ts 32.22% <58.62%> (+10.00%) ⬆️
...s/packages/hw-app-eth/src/services/ledger/erc20.ts 93.44% <92.15%> (-4.00%) ⬇️
libs/ledger-live-common/src/currencies/index.ts 94.54% <100.00%> (+0.10%) ⬆️
libs/ledgerjs/packages/cryptoassets/src/tokens.ts 75.40% <100.00%> (+0.40%) ⬆️
...s/packages/hw-app-eth/src/services/ledger/index.ts 83.87% <100.00%> (+1.90%) ⬆️
libs/ledgerjs/packages/hw-app-btc/src/BtcNew.ts
...dgerjs/packages/hw-app-btc/src/getAppAndVersion.ts
... and 56 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@henri-ly
Copy link
Contributor Author

It's seems to work fine now :
https://github.com/LedgerHQ/ledger-live/actions/runs/3136154840/jobs/5092709009

So it's now accessible by this CDN :
https://cdn.live.ledger-stg.com/cryptoassets/erc20.json
thanks to infra team !

gre
gre previously requested changes Sep 29, 2022
Copy link
Contributor

@gre gre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • live-common: it seems like we do redundant work at loading erc20? (both in ethereum module erc20 and in the global currencies.ts) => in any case we need to kill this idea of a global "preloadTokens" I think => instead we put this in the preload scope of each bridge that needs to load assets. (and the mock run won't be impacted by this , yet we can implement a test scenario in the mock bridges independently that don't depends on network)
  • ledgerjs: some fundamental bug on the previously "get" cache, either we drop the cache or we need a "cache map" as you introduced an extra parameter

.github/workflows/cal-importer.yml Show resolved Hide resolved
apps/ledger-live-desktop/src/renderer/init.jsx Outdated Show resolved Hide resolved
apps/ledger-live-mobile/src/live-common-setup.ts Outdated Show resolved Hide resolved
libs/ledger-live-common/src/currencies/tokens.unit.test.ts Outdated Show resolved Hide resolved
libs/ledger-live-common/src/currencies/tokens.ts Outdated Show resolved Hide resolved
libs/ledger-live-common/src/env.ts Outdated Show resolved Hide resolved
@@ -35,11 +60,11 @@ const asContractAddress = (addr: string) => {
};

// this internal get() will lazy load and cache the data from the erc20 data blob
const get: () => API = (() => {
const get: (erc20SignaturesBlob?: string) => API = (() => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the get() was a parameter-less function that literally acts as a lazy evaluated value.

there is now a fundamental problem because introducing a parameter here breaks the internal cache that was inside the self called function. (there is a "global" cache variable that bypass the returned value and independently on if you pass a arg or not so we will have a bug.)
so you may need to have a "cache map" approach. You can use a WeakMap for memory.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have try something, but I think i'll need your insight can you review this part pls ?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @gre

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

after reviewing again, something "feels" wrong on this function signature to be honest. this is no longer a get. a get don't take the parameter in parameter that it's going to return 🤔

Copy link
Contributor

@gre gre Oct 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a possibility to rename it parse(blob: string) and make this parameter non optional?
or is get() still used somewhere? In that case, get can be implemented by get = () => parse(blob) (where blob is the static import one)

it would be good to split the 2 functions and document each usecase, because the semantic isn't easy here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So when we fetch data from the s3 bucket it will only use the parse command, otherwise it will use the usual get() and cache it

@gre gre marked this pull request as draft September 29, 2022 09:04
Copy link

@ghost ghost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gre if you want to review again. 🙏
This is ready to review, still in draft only because of test config that needs to be reverted before merging.

@henrily-ledger Need to remove that config when you think this is ready.

.changeset/friendly-rocks-yawn.md Outdated Show resolved Hide resolved
libs/ledger-live-common/src/env.ts Outdated Show resolved Hide resolved
@@ -35,11 +60,11 @@ const asContractAddress = (addr: string) => {
};

// this internal get() will lazy load and cache the data from the erc20 data blob
const get: () => API = (() => {
const get: (erc20SignaturesBlob?: string) => API = (() => {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @gre

Copy link

@ghost ghost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, would need another QA check since there were a few changes since last time 🙏

Copy link

@ghost ghost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👌

@ghost ghost dismissed gre’s stale review October 26, 2022 10:41

Requested changes have been adressed

@henri-ly henri-ly merged commit f5f4db4 into develop Oct 26, 2022
@henri-ly henri-ly deleted the feat/LIVE-2174-dynamic-cal branch October 26, 2022 12:06
adrienlacombe pushed a commit that referenced this pull request Oct 26, 2022
* Dynamic cal

* dynamic call script edit

* readd spl token

* mobile dyn cal

* remove spl tokens

* cal importer with s3 staging

* move to ubuntu latest for aws

* cal importer in folder

* staging dyn cal

* fix caching on staging env dyn cal

* dyncal erc20-signatures

* Feedback from github

* fix lint

* staging change role

* try-selfhosted

* selfrunner for aws

* change yalm runner

* change yalm runner

* reomve runner

* update doc

* Test + bugfix on integration

Co-authored-by: Kévin Lambert <[email protected]>

* update lock + add schedule to importer

* cal importer with prod env

* change find to getCryptoCurrencyById

* prod cal importer

* prd cal importer

* redoing cache

* fix: different ref

* typing

* lint fix

* add tojson method

* feedback fix

* add cache for findERC20Signature

* better with save file

* cache fix lint

* remove cache from dyn cal et add it to static

* add return

* revert pnpm-lock

Co-authored-by: Kévin Lambert <[email protected]>
adrienlacombe pushed a commit that referenced this pull request Oct 26, 2022
* Dynamic cal

* dynamic call script edit

* readd spl token

* mobile dyn cal

* remove spl tokens

* cal importer with s3 staging

* move to ubuntu latest for aws

* cal importer in folder

* staging dyn cal

* fix caching on staging env dyn cal

* dyncal erc20-signatures

* Feedback from github

* fix lint

* staging change role

* try-selfhosted

* selfrunner for aws

* change yalm runner

* change yalm runner

* reomve runner

* update doc

* Test + bugfix on integration

Co-authored-by: Kévin Lambert <[email protected]>

* update lock + add schedule to importer

* cal importer with prod env

* change find to getCryptoCurrencyById

* prod cal importer

* prd cal importer

* redoing cache

* fix: different ref

* typing

* lint fix

* add tojson method

* feedback fix

* add cache for findERC20Signature

* better with save file

* cache fix lint

* remove cache from dyn cal et add it to static

* add return

* revert pnpm-lock

Co-authored-by: Kévin Lambert <[email protected]>
cgrellard-ledger pushed a commit that referenced this pull request Oct 27, 2022
* Dynamic cal

* dynamic call script edit

* readd spl token

* mobile dyn cal

* remove spl tokens

* cal importer with s3 staging

* move to ubuntu latest for aws

* cal importer in folder

* staging dyn cal

* fix caching on staging env dyn cal

* dyncal erc20-signatures

* Feedback from github

* fix lint

* staging change role

* try-selfhosted

* selfrunner for aws

* change yalm runner

* change yalm runner

* reomve runner

* update doc

* Test + bugfix on integration

Co-authored-by: Kévin Lambert <[email protected]>

* update lock + add schedule to importer

* cal importer with prod env

* change find to getCryptoCurrencyById

* prod cal importer

* prd cal importer

* redoing cache

* fix: different ref

* typing

* lint fix

* add tojson method

* feedback fix

* add cache for findERC20Signature

* better with save file

* cache fix lint

* remove cache from dyn cal et add it to static

* add return

* revert pnpm-lock

Co-authored-by: Kévin Lambert <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
automation CI/CD stuff common Has changes in live-common desktop Has changes in LLD ledgerjs Has changes in the ledgerjs open source libs mobile Has changes in LLM
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants