Streams the following Ethereum entities to Pub/Sub using ethereum-etl stream:
- blocks
- transactions
- logs
- token_transfers
- traces
- contracts
- tokens
Streams blocks and transactions to Pub/Sub using bitcoin-etl stream. Supported chains:
- bitcoin
- bitcoin_cash
- dogecoin
- litecoin
- dash
- zcash
- Create a cluster:
gcloud container clusters create ethereum-etl-streaming \
--zone us-central1-a \
--num-nodes 1 \
--disk-size 10GB \
--machine-type custom-2-4096 \
--network default \
--subnetwork default \
--scopes pubsub,storage-rw,logging-write,monitoring-write,service-management,service-control,trace
- Get
kubectl
credentials:
gcloud container clusters get-credentials ethereum-etl-streaming \
--zone us-central1-a
- Create Pub/Sub topics (use
create_pubsub_topics_ethereum.sh
)
- "crypto_ethereum.blocks"
- "crypto_ethereum.transactions"
- "crypto_ethereum.token_transfers"
- "crypto_ethereum.logs"
- "crypto_ethereum.traces"
- "crypto_ethereum.contracts"
- "crypto_ethereum.tokens"
Put the prefix to ethereum_base/configMap.yaml
, PUB_SUB_TOPIC_PREFIX
property.
-
Create GCS bucket. Upload a text file with block number you want to start streaming from to
gs:/<YOUR_BUCKET_HERE>/ethereum-etl/streaming/last_synced_block.txt
. Put your GCS path tooverlays/ethereum/block_data/configMap.yaml
,GCS_PREFIX
property, e.g.gs:/<YOUR_BUCKET_HERE>/ethereum-etl/streaming
. -
Update
ethereum_base/configMap.yaml
,PROVIDER_URI
property to point to your Ethereum node. -
Create "ethereum-etl-app" service account with roles:
- Pub/Sub Editor
- Storage Object Admin
Download the key. Create a Kubernetes secret:
kubectl create secret generic streaming-app-key --namespace eth --from-file=key.json=$HOME/Downloads/key.json
- Install [helm] (https://github.com/helm/helm#install)
brew install helm
helm init
bash patch-tiller.sh
- Copy example values directory to
values
dir and adjust all the files at least with your bucket and project ID. - Install ETL apps via helm using chart from this repo and values we adjust on previous step, for example:
helm install --name btc --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/bitcoin/values.yaml
helm install --name bch --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/bitcoin_cash/values.yaml
helm install --name dash --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/dash/values.yaml
helm install --name dogecoin --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/dogecoin/values.yaml
helm install --name litecoin --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/litecoin/values.yaml
helm install --name zcash --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/zcash/values.yaml
helm install --name eth-blocks --namespace eth charts/blockchain-etl-streaming \
--values values/ethereum/values.yaml --values values/ethereum/block_data/values.yaml
helm install --name eth-traces --namespace eth charts/blockchain-etl-streaming \
--values values/ethereum/values.yaml --values values/ethereum/trace_data/values.yaml
helm install --name eos-blocks --namespace eos charts/blockchain-etl-streaming --values values/eos/block_data/values.yaml
Ethereum block and trace data streaming are decoupled for higher reliability.
- Use
describe
command to troubleshoot, f.e. :
kubectl describe pods -n btc
kubectl describe node [NODE_NAME]
Refer to blockchain-etl-dataflow for connecting Pub/Sub to BigQuery.