[Feature Request] Modification of Mining Module

[gzelda]

Is your feature request related to a problem? Please describe.

Right now the official code for a Stacks 2.0 node does not readily expose information/data/API that is both necessary and desired by STX Miners. This extensively limits the operations an individual miner can easily execute, as well as the ability for third parties to provide STX Mining solutions that make it easy for anyone to participate. The two biggest issues are described in detail below:

The operating mechanism of existing nodes is to read configuration information from the configuration file of .toml, such as whether to mine, miner address, etc. This method of statically reading files limits the operations related to mining. For example, starting and stopping mining are currently controlled by opening and closing the node program, and the node controller cannot change the behavior during the node operation. (For example, only normal node status at the beginning of the node, and then perform mining operation later).

Specifically, the two missing features that are of highest priority are:

• 1. A Stacks node can start or stop mining at any time while it is running. When mining is stopped, the node is just running as a normal node, synchronizing network progress.
• 2. The node can change the mining-related parameters (e.g. amount of BTC) at any time while it is running.

Describe the solution you'd like

Daemon Technologies has, up until now, been modifying the official Stacks Blockchain code in order to facilitate the above. These changes, however, are unofficial/not part of master, and therefore need to be redone every time Blockstack PBC changes the official code. Given that the changes we propose will be necessary, and large value add for any and all STX Miners, in addition to data that will be important for other products such as the Explorer, Daemon Technologies would like to propose working with Blockstack PBC to merge a solution to the issues described above to the master/official Stacks Blockchain code.

The initial solution Daemon Technologies is proposing to the issues above is as follows:
Since the pre-run check and runtime state changes of existing nodes are both in run_loop, the following points need to be adjusted at this stage:

• 1. For the first issue, we are looking at Ethereum as a good example of how running a node and mining can interact. Refer to Geth, firstly initialize the miner-related component instance in run_loop module (for example, develop a package called miner responsible for mining-related processing), and then wait for the call of the miner instance related method (Here I want to design Miner-related RPC to achieve), then miner.start() miner.stop() and the method of configuring miner related parameters (miner btc private key) can be realized through RPC method.
• 2. Regarding the issue of dynamically modifying the mining-related parameters (the amount of BTC), we cannot refer to other frameworks, because this is a unique mechanism of Blockstack, the time interval for stacks-blockchain to select a new leader (the same amount as the Bitcoin network block generation Level) is long enough for users to operate, so in this process we want to modify some member variables in the miner class, mentioned above, through the RPC method (the number of bitcoins burned at this stage is a fixed value, written in a variable called burn_fee_cap , Please see link for details, so that you can modify the number of burned BTC to participate in the next round of elections.

Describe alternatives you've considered

If Blockstack PBC is not willing to work with Daemon Technologies to merge these features to master at this stage(maybe because of time/energy/priority), Daemon plans to continue it's development, but with a delayed timeline. If these features are not offered in the official Stacks chain code, Daemon will need to take more time to develop and release the STX Mining bot. If these changes are merged to master and officially offered, we believe we can release a fully usable version of the mining bot by mainnet launch. If not, we anticipate the official release of the mining bot occuring 1-2 months post mainnet launch.

Additional context

[xanbots]

Thanks Gavin! Just to echo the above, we think some of the changes we're making a lot of sense to be offered officially by Blockstack PBC. The first we're in the process of working on, and the second we've created one possible solution for.

Appreciate the consideration @muneeb-ali @diwakergupta @kantai @jcnelson

[diwakergupta]

Thanks @tyGavinZJU @xanbots -- please do send PRs! Some relevant issues that add context on prior discussions about what functionality belongs in stacks-blockchain proper vs. mining software: #1451 , #1441 , #1420

[jcnelson]

Hey @tyGavinZJU @xanbots, appreciate your input. This is something that @kantai, @lgalabru, and myself have discussed in the past (as early as April), and we have a pretty good idea about what mining is going to look like in the future. If you want to help us with this, there's a specific architecture we had in mind that we'd like you to use.

The biggest difference between how the node behaves right now and how the node will be have in the future is that most mining activity will occur in a separate program (#1441). The mining component was never intended to be part of the Stacks node. It's only there now because it was the fastest way to enable mining on the testnet.

What we intend to do -- and would accept PRs for -- is to split the mining component out of the Stacks node to create a separate mining program, and expand the RESTful API to make it possible for the mining program to generate and broadcast Stacks blocks and Bitcoin transactions.

A Stacks mining program would need to do the following:

• Manage the user's mining keychains -- one for sending Bitcoin transactions, one for generating VRF proofs, and one for signing Stacks microblocks.
• Generate and broadcast the LeaderKeyRegister and LeaderBlockCommit Bitcoin transactions to both register a VRF key for the miner and commit blocks to the chain. This would include fetching and signing UTXOs from a Bitcoin node in order to form these transactions.
• Fetch transactions from the Stacks node's mempool to package into an anchored Stacks block (whose hash will be written to the Bitcoin chain in a LeaderBlockCommit transaction).
• Fetch transactions from the Stacks node's mempool to package into a microblock stream.
• Upload Stacks blocks and microblocks to the Stacks node for broadcasting.

All of this currently happens within the Stacks testnet software, and could be split out into a separate Rust binary. However, the Stacks node still needs the following RESTful API endpoints at a minimum to make this happen:

• An API endpoint for generating a block template (Add mine_block RPC endpoint #1420). This currently happens in the mining component as part of building a Stacks block, but would need to be exposed as a (privileged) API endpoint. The API endpoint would return a Stacks block without the VRF proof or microblock public key hash set. The miner program would need to fill these fields in as part of generating a LeaderBlockCommit.
• An API endpoint for generating a microblock template. This is similar to the above -- it just returns an unsigned microblock.
• An API endpoint for uploading a signed Stacks block (Add RPC endpoint for uploading a mined block #1451). The API endpoint would need to ensure the block gets scheduled to be broadcasted and processed.
• An API endpoint for uploading a signed Stacks microblock. The API endpoint would need to ensure that the microblock gets scheduled to be broadcasted and processed.

[gzelda]

Hey @tyGavinZJU @xanbots, appreciate your input. This is something that @kantai, @lgalabru, and myself have discussed in the past (as early as April), and we have a pretty good idea about what mining is going to look like in the future. If you want to help us with this, there's a specific architecture we had in mind that we'd like you to use.

The biggest difference between how the node behaves right now and how the node will be have in the future is that most mining activity will occur in a separate program (#1441). The mining component was never intended to be part of the Stacks node. It's only there now because it was the fastest way to enable mining on the testnet.

What we intend to do -- and would accept PRs for -- is to split the mining component out of the Stacks node to create a separate mining program, and expand the RESTful API to make it possible for the mining program to generate and broadcast Stacks blocks and Bitcoin transactions.

A Stacks mining program would need to do the following:

• Manage the user's mining keychains -- one for sending Bitcoin transactions, one for generating VRF proofs, and one for signing Stacks microblocks.
• Generate and broadcast the LeaderKeyRegister and LeaderBlockCommit Bitcoin transactions to both register a VRF key for the miner and commit blocks to the chain. This would include fetching and signing UTXOs from a Bitcoin node in order to form these transactions.
• Fetch transactions from the Stacks node's mempool to package into an anchored Stacks block (whose hash will be written to the Bitcoin chain in a LeaderBlockCommit transaction).
• Fetch transactions from the Stacks node's mempool to package into a microblock stream.
• Upload Stacks blocks and microblocks to the Stacks node for broadcasting.

All of this currently happens within the Stacks testnet software, and could be split out into a separate Rust binary. However, the Stacks node still needs the following RESTful API endpoints at a minimum to make this happen:

• An API endpoint for generating a block template (Add mine_block RPC endpoint #1420). This currently happens in the mining component as part of building a Stacks block, but would need to be exposed as a (privileged) API endpoint. The API endpoint would return a Stacks block without the VRF proof or microblock public key hash set. The miner program would need to fill these fields in as part of generating a LeaderBlockCommit.
• An API endpoint for generating a microblock template. This is similar to the above -- it just returns an unsigned microblock.
• An API endpoint for uploading a signed Stacks block (Add RPC endpoint for uploading a mined block #1451). The API endpoint would need to ensure the block gets scheduled to be broadcasted and processed.
• An API endpoint for uploading a signed Stacks microblock. The API endpoint would need to ensure that the microblock gets scheduled to be broadcasted and processed.

A quick question, does the mining-program contain a stacks-node for sync blocks and broadcast transactions?

I guess the answer is NO, because from your description that the mining-program is just like a signing machine which process certain transaction of Stacks and Bitcoin. But in reality, each miner should has his own node to broadcast new transactions like LeaderKeyRegister and LeaderBlockCommit. Other nodes could refuse the outer transaction for some reason(maybe evil).

[psq]

Synchronizing your own regtest bitcoind locally with the one krypton uses is easy:

bitcoin.conf

server=1
regtest=1
rpcallowip=0.0.0.0/0
rpcallowip=::/0
rpcuser=blockstack
rpcpassword=blockstacksystem
txindex=1
listen=1
debug=1
rpcserialversion=0
datadir=/absolute/path/to/data/bitcoin-psq2

[regtest]
bind=0.0.0.0:19444
rpcbind=0.0.0.0:19443
addnode=bitcoind.krypton.blockstack.org:18444

Start bitcoind

bitcoind -conf=/absolute/path/to/bitcoin.conf &

and to query with bitcoin-cli:

bitcoin-cli -regtest -conf=/absolute/path/to/bitcoin.conf -rpcport=19443 getblockchaininfo

not sure why it can't pick up the rpcport from the conf file, but whatever network operations are whitelisted are sufficient for the node to synchronize, and now you have access to all rpc calls.

[psq]

Here are are some thoughts on how I would approach mining, and incrementally splitting some of the mining responsibilities.

First of all, the mining process needs to be driven by the node, which reacts to other nodes when it receives new btc blocks, new stacks blocks and new transactions. So when considering an RPC API, the miner is the server, and the node a client to the miner, in the same fashion the stacks-blockchain-api node is a server and the stacks-blockchain node a client.

Secondly, the more granular you make the api, the more serialization/deserialization cost you'll be incurring, slowing down the process, and making it more complex (and also more brittle), so you need to keep the exchanges to a minimum (when speed may matter).

I started writing this thinking you'd split the current mining code into smaller chunks, the node sending data needed for each basic operation (register key, generate proof, build block, sign and commit, new block received, new transaction received, ...), but it turns out building a block outside of the node is not really feasible without pulling most of the node code (Clarity VM, including its state for example), so here's a more nimble proposal.

The node keeps managing the Leader Key, and assembles the block, and once the block is assembled, calls the miner to let it decide whether it wants to mine the block (based on the current chain tip, the tx count, the tx fees, and a few other parameters) and how much it wants to transfer to stackers, and what fee to pay to the btc network, or the miner can decide to skip this block. The node will call the miner each time it detects a new chain tip, which can be more than once per epoch (btc block), but gives the miner the best chance of mining a block on the best known fork for the current epoch as the node learns about it.

This should be fairly simple and only a few modifications to the current node code, and a few extra configuration parameters (miner's address and port may be sufficient), and can be modeled on the event dispatcher (or even be an extension of it with a few more events).

A second version could let the miner pick which transactions to include, and in what order (a first call the node would make to the miner), then let the node assemble the block, and again, the node would call the miner to let it decide once the block has been assembled, whether to commit to it or not. The miner could already reuse the existing events used to get new transactions and blocks so it could manage its own mempool and does not need to get a copy of the mempool for each block).

This does not take microblocks into accounts, but once this has been enabled, the miner would also need to decide which transaction to mine in a microblock, and when to stop mining microblocks for the current epoch, some fairly simple addition to the protocol.

[diwakergupta]

This should be fairly simple and only a few modifications to the current node code, and a few extra configuration parameters (miner's address and port may be sufficient), and can be modeled on the event dispatcher (or even be an extension of it with a few more events).

Interesting idea @psq ! There's some elegance in re-using the event dispatcher pattern to "push" updates to the miner software. In this model, would the miner program would still make API calls into the node to upload the block etc (basically the set of endpoints in Jude's earlier comment)? I think a quick diagram (whiteboard picture would also work) might help clarify what you have in mind.

@blockstack/blockchain-team please chime in with your thoughts! 🙏🏽

[jcnelson]

All you need to do to get the node to produce a block template is add a RPC endpoint that takes a VRF proof, a microblock public key hash, and a list of txids to mine as input, and spits out an unsigned Stacks block that includes those transactions in the order given by the txids (or, returns an error of the node is missing one or more of those transactions or if the transactions cannot be applied in the given order). Internally, the node would use a StacksChainState instance to create an anchored block (i.e. by instantiating a StacksBlockBuilder, loading the referenced transactions out of the mempool, and calling make_anchored_block_from_txs()). This is what #1420 is.

There's absolutely no need at all to involve the event-dispatcher. It's the mining program's job to identify transactions for inclusion, not the node's. Moreover, the node shouldn't be producing blocks that the mining program didn't ask for, and it shouldn't be polluting the event stream (to which the side-car is attached) with data that it didn't want in the first place. Making a block template is expensive and time-consuming, and should only be done at the request of the miner (implying that this RPC endpoint should be privileged, and/or accessible only from localhost). The mining program, not the node, should be driving block template generation. This is how most mining software for existing blockchains work -- the node makes the block template at the miner's request, and the miner fills in the template with any missing information to finish mining the block (in our case, the VRF proof and the block-commit transaction).

Let's not over-complicate the design.

[psq]

@jcnelson the miner is most likely going to need specific events that the explorer would not find interesting (btc blocks information for example, and yes, it could connect directly to bitcoind, but duplicating work is not ideal), so the explorer will eventually need to subscribe to a subset of the events available (that's already something that is available), so it is not much of a stretch, if push is a model that would work better, to extend the event model, and I don't see how that would be "polluting" per se.

And it seems anyway we are saying very similar things, and mostly disagree on whether the node is making the http requests or the miner.

@diwakergupta in both proposals, the block is still assembled by the node (which has the state and the Clarity VM), but the miner decides which transactions to include and in what order (by providing an ordered tx list). Except my first proposed step that would skip that choice to get to a working solution faster, and the only decision the miner would make is whether to mine a block or not for a given epoch, and how much to pay for it.

With all that said, now that I understand more precisely what @jcnelson was suggesting, I think I could create a crude prototype for a miner in a few days by adding one RPC call, and maybe a few extra events that can then be extended by @xanbots and @tyGavinZJU. That prototype miner would pick the transactions to mine, set the btc spend and decide when and whether to mine a new block.

I was planning to work on #1981 tomorrow, but I'd be happy to get started on the above right away. Let me know.

[gzelda]

I think I could create a crude prototype for a miner in a few days by adding one RPC call, and maybe a few extra events that can then be extended by @xanbots and @tyGavinZJU. That prototype miner would pick the transactions to mine, set the btc spend and decide when and whether to mine a new block.

Awesome discussion and summary @psq.

Several quick questions @psq:

• Do you want to modify the source code in stacks-node? Or create a new Repo like Mining-Program?
  Now I am working on adding RPC call in stacks-node for BTC burn amount and whether to mine a new block two APIs above. Please tell me what do you want to create for features above.
• I am a little confused the crude prototype you mentioned above is the parameter format of API or maybe something else?
• About the new feature you mentioned "pick the transactions to mine", I think I need your more detailed explanation for API format and implementation.

Thank you!

[jcnelson]

Here is a protocol diagram of how mining is going to work.

To make this work, the Stacks node only needs to provide two new API calls: one to produce a block template (as I just described), and one to receive an uploaded block and broadcast it to the peer network (which will work largely the same way as /v2/transactions). The miner program is expected to know how to talk to the Stacks node and the Bitcoin node to get the block template and to get UTXOs. The miner program is expected to know how to sign and broadcast a block-commit transaction to the Bitcoin network. The miner program is also expected to know when to try (or re-try) to send the Stacks block (omitted from this diagram because this is trivial).

Like I said earlier, let's not over-complicate this. I don't want to see any PRs that touch the event notification system, because that will almost certainly lead to unintentional breakage in the side-car.

EDIT: to be clear, the @blockstack/blockchain-team has already thought long and hard about how mining is going to work. We're pretty much set on this design.

[gzelda]

Here is a protocol diagram of how mining is going to work.

To make this work, the Stacks node only needs to provide two new API calls: one to produce a block template (as I just described), and one to receive an uploaded block and broadcast it to the peer network (which will work largely the same way as /v2/transactions). The miner program is expected to know how to talk to the Stacks node and the Bitcoin node to get the block template and to get UTXOs. The miner program is expected to know how to sign and broadcast a block-commit transaction to the Bitcoin network. The miner program is also expected to know when to try (or re-try) to send the Stacks block (omitted from this diagram because this is trivial).

Like I said earlier, let's not over-complicate this. I don't want to see any PRs that touch the event notification system, because that will almost certainly lead to unintentional breakage in the side-car.

EDIT: to be clear, the @blockstack/blockchain-team has already thought long and hard about how mining is going to work. We're pretty much set on this design.

@jcnelson Very clear diagram! Thanks. As we discussed before in zoom, I am working on adding 2 API calls in stacks-node:

• change burn_fee in next_cycle.
• start/stop mining function.

Developing two APIs above will give great convenience to the community to participate in mining and it's not that hard to implement compared to mining-program. I want to make sure if there is a big conflict between @blockstack/blockchain-team's proposal and what I am working on. If there is, I will stop and discuss in-depth. @jcnelson

[psq]

@tyGavinZJU

Do you want to modify the source code in stacks-node? Or create a new Repo like Mining-Program?

If you create a new repo, a fork essentially, you'll forgo the improvements or have to keep merging the changes from the upstream repo, so changes should go in stacks-node.

Now I am working on adding RPC call in stacks-node for BTC burn amount and whether to mine a new block two APIs above. Please tell me what do you want to create for features above.

Just adding these 2 api calls would most likely result in throwaway code, and would not allow implementing more than a basic miner, and if that's what you intend to do short term, then this might be best done in a separate repo, despite what I said above.

I am a little confused the crude prototype you mentioned above is the parameter format of API or maybe something else?
About the new feature you mentioned "pick the transactions to mine", I think I need your more detailed explanation for API format and implementation.

The crude part would be on the mining logic, to let you implement what you mentioned (start/stop mining, and set the burn fee. Picking the transactions, and I'm simplifying, means providing the equivalent of what iterate_candidates does today (https://github.com/blockstack/stacks-blockchain/blob/master/src/core/mempool.rs#L496) for build_anchored_block (https://github.com/blockstack/stacks-blockchain/blob/master/src/chainstate/stacks/miner.rs#L936).

[xanbots]

Thanks everyone for the discussion. To summarize and ask for a point of clarification:

• PBC will not be shipping a separate mining program/module for the launch of Stacks 2.0. That will be developed and released separately, at some point following mainnet launch.
• PBC will offer a way to mine STX at the time of Stacks 2.0 launch that is similar to what is available today on testnet.

What Daemon wants to do is first release a mining bot that makes it easy for anyone to mine in time for mainnet launch, as we've planned. We will then work on a full release of the mining bot that can use the separate mining program/module that will be developed later.

@diwakergupta @jcnelson @psq What do you think might be the difference in experience STX Mining at mainnet launch compared to the experience mining today on testnet?

I appreciate it may be a tough question to answer at the moment, but any information will give me a clearer picture of what Daemon can try to build for mainnet launch. Right now we're still not clear enough on this to start any development. Thank you.

[diwakergupta]

Thanks everyone for the discussion. To summarize and ask for a point of clarification:

• PBC will not be shipping a separate mining program/module for the launch of Stacks 2.0. That will be developed and released separately, at some point following mainnet launch.
• PBC will offer a way to mine STX at the time of Stacks 2.0 launch that is similar to what is available today on testnet.

I think it's best to keep the issue focused on what sort of "miner interface" will exist as part of stacks-blockchain proper in time for mainnet: it's not germane whether that's developed by PBC or others.

The expectation is that there will be set of stable API endpoints in stacks-blckchain in time for mainnet that would allow anyone interested to create their own miners with relatively modest effort.

@diwakergupta @jcnelson @psq What do you think might be the difference in experience STX Mining at mainnet launch compared to the experience mining today on testnet?

There's alignment on the architecture described earlier in this issue, and I believe @psq is actively working on implementing that.

@xanbots does that clarify?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[friedger]

This is important to me.
Related to #2629

[gzelda]

Can you take a look at the diagram @jcnelson.
Three questions:
(1) subscribe mode is not good for multi-client, maybe the restful API is more decentralized
(2) RPC API for winner_txid, because normal client cannot get snapshot table
(3) /v2/block/upload/{consensus} response

[gzelda]

@jcnelson
Further questions is:
(4) Diagram above doesn't include microblock. what else should we do for microblock?

[jcnelson]

Hi @tyGavinZJU,

(1) I'm not sure I understand the ask? If operated with an event observer, the node will synchronously POST all events to the observer as it processes blocks. It's up to the observer to do useful things with this data, such as store it in a database and serve it later via a Web server. The node does not (and cannot) know what the downstream sinks need.

(2) That can be added. Can you an open an issue for it? I'll try and get it in for the next release.

(3) The success response is HTTP 200, with a body that is a JSON blob with the following schema:

{
   "stacks_block_id": "^[0-9a-f]{64}$",
   "accepted": "bool"
}

The stacks_block_id field is the index block hash of the Stacks block. This is not the block hash; this is the SHA512/256 of the Bitcoin block's consensus hash concatenated with the Stacks block's hash. I use this script to calculate the index block hash from a Stacks block hash and a consensus hash:

$ cat ~/bin/stacks-block-id.sh 
#!/bin/sh

CONSENSUS_HASH="$1"
BLOCK_HASH="$2"

if [ -z "$BLOCK_HASH" ] || [ -z "$CONSENSUS_HASH" ]; then
   echo >&2 "Usage: $0 CONSENSUS_HASH BLOCK_HASH"
   exit 1
fi

echo -n "${BLOCK_HASH}${CONSENSUS_HASH}" | xxd -r -p | openssl dgst -sha512-256 - | cut -f 2 -d ' '

The accepted field will be true if the block was accepted (and relayed); false otherwise.

(4) Right now, the mock miner does not attempt to mine microblocks because its block never wins. We'd have to add that functionality.

[gzelda]

@jcnelson, it is the current diagram above. Several questions below:
(1) I met some challenges when creating mempool-observer
may need your help to directly modify stacks-node to support posting mempooleventst to observer. Do I need to create a related issue to deal with it? Or is there an existing observer program or demo I can try?.
(2) Getting winner-txid, I wanna record stacks-block height and fetch data from /extended/v1/block and the using the miner_txid as result to compare. There is an extreme situation, for example, the mining software is generating block N, but after I fetch data from /extended/v1/block the N+1 block already exists. If it means I miss the public block sortition although I have some probability to win block N.
(3)Is there response for mining software to check if the block upload is successful?

[jcnelson]

(1) The observer should be posting mempool events to http://your-observer-address/new_mempool_tx and http://your-observer-address/drop_mempool_tx. Are you seeing these when the node is running?

(2) I think there needs to be a new API endpoint for reading snapshot state, given either a burnchain hash or a burnchain height. Can you open an issue for it?

(3) You will get an HTTP 200 response on success, with a JSON body that contains the index block hash of the block and a boolean flag indicating whether or not the block was accepted into the database (you will get false if the node already has the block, or if the block does not correspond to a known sortition).

[gzelda]

(1) The observer should be posting mempool events to http://your-observer-address/new_mempool_tx and http://your-observer-address/drop_mempool_tx. Are you seeing these when the node is running?

Is there an observer option in the .toml configuration?

[jcnelson]

Yeah -- it looks like this:

[events_observer]
endpoint = "http://your-observer-address-endpoint-url"
event_keys = "comma-separated-list-of-event-types"

For example, you'd want event_keys = "memtx" to get mempool events.

Your config file can have multiple [events_observer] entries if you have multiple event observers.

[jcnelson]

Closing due to inactivity. This is being taken up by https://github.com/stacksgov/Stacks-Grant-Launchpad/issues/790

[BowTiedDeployer]

Hello @jcnelson !

We will build on top of the miner module so that it will read from the smart contract, know how much the miner should spend on each transaction, and the miners will communicate with each other.

But we are not restructuring the miner module code itself. We are not developing the functionality of dynamically running/stopping the miners for each block, that would also be counterproductive for the mining pool flow as they should leave the smart contract pool before stopping the BTC contributions sent in each block. The amount sent by each miner is picked from the smart contract as the total contribution amount of the pool is split evenly between the miners, so the amount will not be dynamic in that sense.

[friedger]

@BowTiedDeployer the smart contract will be queried for each tx? If I provide an api similar to the smart contract, then I could change the bitcoin commit for each tx?