Facilitator

[deepesh-kn]

Facilitator (Executable):

• The executable must be able to support multiple chains simultaneously.

• Support multiple workers (acceptStakeRequest & progressStake) simultaneously. (Q: private keys are held on node, or on client side? A: on the client side, local machine)

• Subscribe / Scan StakeRequested event.

• Workers should avoid race on acceptStakeRequest; eg round-robin

• Track the gas available for each worker, if the worker gas too low to perform any transaction then it must not do facilitation.

• Calculate if the gasPrice and gasLimit should be acceptable to do facilitation. (What's the threshold value? for now, we can use a constant value)

• It must act as a hunter for any failed facilitation. (Out of scope for now.)

• Handle SIGTERM gracefully.

• Should be able to resume facilitation if it was not complete.

• It should track the states of each step of facilitation in DB. Update the DB with the latest state e.g REQUESTED, ACCEPTED, PROGRESSED, etc.

• Manage hash lock and unlock secret.

---

config.json :

{
    "db_path": "<db_path>",
    "chains": {
        <chain_id_1>: {
            "rpc": "http://localhost:3000",
            "ws": "ws://localhost:3001",
            "workers": [0xaddress1, 0xaddress2],
            "composer_address": "0xcomposerContractAddress",
            "supported_gateway_address": ["0xgatewayContractAddress_1", "0xgatewayContractAddress_2"]
		},
		<chain_id_2>: {
            "rpc": "http://localhost:4000",
            "ws": "ws://localhost:4001",
            "workers": [0xaddress3, 0xaddress4]
		}	
	}
    "encrypted_accounts": {
        "0xaddress1": {
        
        },
        "0xaddress2": {
        
        },
        "0xaddress3": {
        
        },
        ,
        "0xaddress4": {
        
        }
    }
}

• db_path: Database file path
• chains: Contains details of all the chains.
• chain_id: Actual chain id number.
• rpc: RPC end point.
• ws: Websocket end point.
• workers: Account address that can perform facilitation for the given chain
• composer_address: Composer contract address for the given chain
• supported_gateway_address: Supported gateway contract address. If StakeRequested event will have the gateway address other than specified here, that will be not taken for facilitation.
• encrypted_accounts: This are the encrypted account that is generated by web3.eth.accounts.wallet.add(account); Here is a reference https://web3js.readthedocs.io/en/1.0/web3-eth-accounts.html#wallet-add

Please note:

• If RPC and WS endpoints are not provided in the config file, then try to use localhost + default port
• If composer_address and supported_gateway_address both are provided for the origin chain only. This will be checked to start the event scanner.

---

Proposal:

Commands :

1. Start facilitator :

./facilitator start <config_file_path> --chains <specific chain ids>

• config_file_path: This is the path for the config file. If this is not provided then the facilitator will look at the default location.
• --chains: This option is used to specify the chain ids that facilitator should facilitate. This will be origin chain ids

2. help :

./facilitator --help

• Provides the help on how to use the command.

Executable :

---

When facilitator starts.

1. Update the config file.
2. Execute command ./facilitator start
3. The facilitator will validate the config files and the chain ids if it is supported.
4. The facilitator will start the event scanner.
5. The facilitator will check the DB for the pending facilitation.
6. The facilitator will fork worker processes for the pending tasks. (Updates the DB entry with PID). PID is the process id of the worker process. This PID indicates that the facilitator is in progress.
7. The worker will query the latest state of the transactions from the chain and performs the facilitation accordingly.

When facilitator gets the StakeRequested events.

1. The facilitator will validate the Event data.

2. The facilitator will check if the gateway is supported.

3. The facilitator will check the addresses table to get the origin and auxiliary chain pair and addresses related to that.

4. If the address table does not have any data for the given chain id and gateway pair, then query the contract and get all the addresses and update the table.

5. Facilitator gets the worker addresses for the origin and auxiliary chain.

6. Facilitator forks a new worker child process and gets the PID.

7. Facilitator inserts a new entry in thestake_requests table. This includes the stateRequestHash, chain id, gateway address, stake params, worker address of both chains, pid, state .. etc.

8. The worker creates the hash lock and unlock secret. Unlocks the accounts. Performs accept stake request. Updates the stake_requests table with the latest state.

9. Worker checks if the latest state root is committed in the auxiliary chain. If it is committed then it proceeds with the facilitator. If it is not committed then it will give a call back to Facilitator and stop the process.

   The facilitator can try to resume facilitation in time intervals, or there can be a new child process that can observe the state root commit event from the anchor, and give a call back to the facilitator. The facilitator can resume the facilitation.

On Handle SIGTERM:

1. All the process should handle the SIGTERM
2. If the child process gets the SIGTERM, complete the existing task and inform the parent process (facilitator).
3. If parent process (facilitator) get he sigterm then inform the child process to terminate and wait till the child process are stopped.

Example

const { fork } = require('child_process');


const scanner = fork('scan_events.ts', args); 
// TODO: Check how we can pass chain Id as and argument in the above statement.

// If we cannot pass the chain Id then we can use send command to
scanner.send({ chain_is: 3 });
// Listen call back.
scanner.on('message', (event) => {
  	console.log('Event received', event);
	// Update DB
	// Do things here
	facilitate(event);
});


function facilitate(event) {
	// Check if gateway address is known
	// Check if gasPrice and gasLimit is sufficient to perform facilitation.
	// Get worker address (round robin .. etc.)
	// Validate worker address can perform transactions.
	// Fork a new child process to do facilitation.

	const facilitator = fork('facilitate.js'); 
	facilitator.send(< Object with all params that are needed for facilitation>);

	// Subscribe to all the events to handle the callbacks
    }
    

Facilitator as a hunter:

There can be one more child process that can act as Hunter which keeps looking for stake and mint events. Not in the scope for now.

DB proposal:

Please note: This will give a rough idea, the schema can be improved by the implementer.

This executable should resume any pending or failed task when started. To enable this the status of each StakeRequest must be persisted. We can use a database to achieve this (e.g: sqlite3).

All the child process have the access to read, insert and update contents in the DB.

stake_requests

name	type	references	description
"stake_request_hash"	VARCHAR		The unique identifier for the table type.
"message_hash"	VARCHAR
"origin_chain_id"	INTEGER
"gateway_address"	VARCHAR
"origin_account_address"	VARCHAR
"auxiliary_account_address"	VARCHAR
"hash_lock"	VARCHAR
"unlock_secret"	VARCHAR
"transactions"	VARCHAR		This will contain transaction hashed of the events(json string)
"pid"	INTEGER		This is process id, this is assigned when the facilitation is in progress by a worker process.
"state"	INTEGER		Enum : REQUESTED ACCEPTED ACCOUNT_PROVEN STAKE_INTENT_COMFIRMED PROGRESS_STAKE PROGRESS_MINT COMPLETE

addresses

name	type	references	description
"orign_chain_id"	INTEGER
"gateway_address"	VARCHAR
"auxiliary_chain_id"	VARCHAR
"composer_address"	VARCHAR
"cogateway_address"	VARCHAR
"anchor_address"	VARCHAR
"co_anchor_address"	VARCHAR

facilitator.ts

• This is the main file.
• This forks a new child-process (scanner, worker, etc).
• Listen to the events from all the child process and perform actions.
• Handles SIGTERM.
• At startup it checks the DB for pending, incomplete and failed facilitation tasks and forks new worker process accordingly.

scanner.ts

• This file scans the blocks, gets the transactions, gets the event of the transactions in batches.
• Keeps track of last scanned block number in memory.
• It always scans from the latest block when started.
• Emits event when StakeRequested event is found.

worker.ts

• This file basically does the facilitation.
• This will have an ethereum address associated with it in the variable (this is provided by the facilitator.ts while forking the new child-process)
• The password for this address should be in the ENV variable password_0xethereumAccountAddress, that can be used for unlocking.
• This always checks the latest state from the chain while progressing the task.
• This will generate the hashLock and unlockSecret. Update DB with these values to keep track in case this process dies.
• On every step of stake and mint, update the Status in DB.
• Emit event when stake and mint are successful or failed.

---

Tickets:

1. Create a new repository. (Done)

2. Finalize config.json (Review and suggest)

3. Finalize DB schema. (Review and suggest)

4. Finalize the enum value for states column instake_request table.

5. Implement DB interaction layer

   • create tables if does not exists.
   • function to insert new entry for StakeRequested event in stake_requests table.
   • function to update the state in stake_requests.
   • function to insert proof data in stake_requests.
   • function to insert and update worker address in stake_requests.
   • function to insert and update PID value in stake_requests.
   • function to insert new entry for addresses table.

6. Implement block scanner.

   • Scan from the latest block when the process starts.
   • Keep track of last block number that was scanned.
   • function to validate the chain configs
   • function to scan the blocks in batches
   • function to get transactions
   • function to get event from the transactions
   • function to emit an event when StakeRequested event is found.
   • Handle SIGTERM to inform parent process.

7. Add commander: ./facilitator start and ./facilitator --help

   following the proposed command

   ./facilitator start <config_file_path> --chains <specific_chain_ids>
   

   • The facilitator can be started by ./facilitator start.
   • config_file_path is the path for the config file. If this is not provided then the default path is used.
   • --chains <specific_chain_ids> this options can be used to specify the chains. The config file may have many chains specified, if this option is provided then the facilitator will start the process for this specific chains only. By default, it will start the process for all origin chains.

8. Implement worker. (can mosaic.js facilitator be used?)

   • This process will do facilitation.
   • Updates the DB with the latest state of facilitation.
   • function to unlock worker address.
   • function to create hash lock and unlock secret.
   • function to approve bounty amount.
   • function to accept stake request.
   • function to generate a proof.
   • function to confirm stake intent.
   • function to progress stake.
   • function to progress mint.
   • Handle SIGTERM to inform parent process.

9. Implement Facilitator

   • validates the config files.
   • function to validate gateways.
   • function to start the event scanner.
   • function to resume facilitation.
   • function to start the worker process.
   • listen to call backs of all the child processes.
   • function to select proper worker address for chains.
   • function to determine if gasPrice and gasLimit is sufficient to perform facilitations.
   • Handle SIGTERM to inform parent process.

10. Update readme

[schemar]

Nice! 🦖

My comments:

Facilitator (Executable):

• The executable must be able to support multiple chains simultaneously.

I don't agree. I think it is much easier to implement and maintain, if one facilitator process tracks only a single gateway pair. If you want to track multiple pairs, run multiple instances. Should make configuration, account tracking, etc., etc. much easier. At the same time, I don't see it as extra burden to run multiple processes for multiple chains. In a way, it is even more robust.

• Support multiple workers (acceptStakeRequest & progressStake) simultaneously. (Q: private keys are held on node, or on client side? A: on the client side, local machine)

Agreed. These could even run in separate, independent sub-processes.

• Subscribe / Scan StakeRequested event.
• Workers should avoid race on acceptStakeRequest; eg round-robin

How do different facilitators coordinate? I think this is a huge topic. Maybe out of the scope of the initial implementation?

• Track the gas available for each worker, if the worker gas too low to perform any transaction then it must not do facilitation.

I am not sure I am following you entirely. It would be helpful if you would define what exactly you mean by "facilitator" and "worker" in the beginning of the ticket.

• Calculate if the gasPrice and gasLimit should be acceptable to do facilitation. (What's the threshold value? for now, we can use a constant value)

It could also check things like: is the gas limit even realistic? If too low: pick up anyway?
Gas limit and price should be treated independently, not only in combination. E.g. the gas price must itself be acceptable, as the gas used could be way below the provided gas limit.

• It must act as a hunter for any failed facilitation. (Out of scope for now.)
• Handle SIGTERM gracefully.

Agreed, however a lot of thought should be put into what that actually means.

• Should be able to resume facilitation if it was not complete.

Agreed. This point and the previous point have a strong relation I think 👍

• It should track the states of each step of facilitation in DB. Update the DB with the latest state e.g REQUESTED, ACCEPTED, PROGRESSED, etc.

Yes, using a DB with a popular ORM is probably the easiest, most reliable way here. SQLite?

• Manage hash lock and unlock secret.

What does that mean? Generate and store?

config.json :

{
    "db_path": "<db_path>",
    "chains": {
        "<chain_id_1>": {
            "rpc": "http://localhost:3000",
            "ws": "ws://localhost:3001",
            "workers": ["0xaddress1", "0xaddress2"],
            "composer_address": "0xcomposerContractAddress",
            "supported_gateway_address": ["0xgatewayContractAddress_1", "0xgatewayContractAddress_2"]
        }, "<chain_id_2>": {
            "rpc": "http://localhost:4000",
            "ws": "ws://localhost:4001",
            "workers": ["0xaddress3", "0xaddress4"]
        }	
    },
    "encrypted_accounts": {
        "0xaddress1": {
        }, "0xaddress2": {
        }, "0xaddress3": {
        }, "0xaddress4": {
        }
    }
}

• db_path: Database file path
• chains: Contains details of all the chains.

As stated before, would opt to limit to one pair per facilitator; name them origin and auxiliary.

• chain_id: Actual chain id number.

How does the facilitator identify "pairs"? I.e. how does the facilitator know which origin/auxiliary belong together? Isn't one address/account per chain sufficient?

• rpc: RPC end point.
• ws: Websocket end point.

Do we really need both? It seems HTTP is the only JSON RPC supported by all clients: https://github.com/ethereum/wiki/wiki/JSON-RPC#json-rpc-support

• workers: Account address that can perform facilitation for the given chain
• composer_address: Composer contract address for the given chain

Will a composer be mandatory?

• supported_gateway_address: Supported gateway contract address. If StakeRequested event will have the gateway address other than specified here, that will be not taken for facilitation.
• encrypted_accounts: This are the encrypted account that is generated by web3.eth.accounts.wallet.add(account); Here is a reference https://web3js.readthedocs.io/en/1.0/web3-eth-accounts.html#wallet-add

Please note:

• If RPC and WS endpoints are not provided in the config file, then try to use localhost + default port

Are you sure? Couldn't that lead to problems if it happens accidentally and it somehow connects to the wrong of multiple nodes?

• If composer_address and supported_gateway_address both are provided for the origin chain only. This will be checked to start the event scanner.

Not sure what this means. Please elaborate.

Proposal:

Commands :

1. Start facilitator :

./facilitator start <config_file_path> --chains <specific chain ids>

• config_file_path: This is the path for the config file. If this is not provided then the facilitator will look at the default location.
• --chains: This option is used to specify the chain ids that facilitator should facilitate. This will be origin chain ids

I like the proposal 👍 But I want to note that it could come out differently depending on the findings during implementation. This shouldn't be set in stone.

2. help :

./facilitator --help

• Provides the help on how to use the command.

Executable :

When facilitator starts.

1. Update the config file.

What does that mean?

2. Execute command ./facilitator start
3. The facilitator will validate the config files and the chain ids if it is supported.
4. The facilitator will start the event scanner.
5. The facilitator will check the DB for the pending facilitation.
6. The facilitator will fork worker processes for the pending tasks. (Updates the DB entry with PID). PID is the process id of the worker process. This PID indicates that the facilitator is in progress.

Woludn't it be sufficient to keep all sub-processes in memory? Wouldn't they die together with the parent process anyways? I wouldn't detach them ...

7. The worker will query the latest state of the transactions from the chain and performs the facilitation accordingly.

When facilitator gets the StakeRequested events.

1. The facilitator will validate the Event data.
2. The facilitator will check if the gateway is supported.
3. The facilitator will check the addresses table to get the origin and auxiliary chain pair and addresses related to that.
4. If the address table does not have any data for the given chain id and gateway pair, then query the contract and get all the addresses and update the table.

Which contract? Get that from where? If it is not recorded already, how high are the chances that the facilitator can actually facilitate the request? Again, I think single pair->single facilitator would make things a lot easier.

5. Facilitator gets the worker addresses for the origin and auxiliary chain.
6. Facilitator forks a new worker child process and gets the PID.
7. Facilitator inserts a new entry in thestake_requests table. This includes the stateRequestHash, chain id, gateway address, stake params, worker address of both chains, pid, state .. etc.

While this is probably right, I wouldn't define these things ahead of the implementation. Who knows what we are forgetting about here which will become apparent when implementing it.

8. The worker creates the hash lock and unlock secret. Unlocks the accounts. Performs accept stake request. Updates the stake_requests table with the latest state.
9. Worker checks if the latest state root is committed in the auxiliary chain. If it is committed then it proceeds with the facilitator. If it is not committed then it will give a call back to Facilitator and stop the process.

I.e. any state root in a block later than the block that contains the stake (which was done by the facilitator).

   The facilitator can try to resume facilitation in time intervals, or there can be a new child process that can observe the state root commit event from the anchor, and give a call back to the facilitator. The facilitator can resume the facilitation.

On Handle SIGTERM:

1. All the process should handle the SIGTERM
2. If the child process gets the SIGTERM, complete the existing task and inform the parent process (facilitator).
3. If parent process (facilitator) get he sigterm then inform the child process to terminate and wait till the child process are stopped.

Example

const { fork } = require('child_process');


const scanner = fork('scan_events.ts', args); 
// TODO: Check how we can pass chain Id as and argument in the above statement.

// If we cannot pass the chain Id then we can use send command to
scanner.send({ chain_is: 3 });
// Listen call back.
scanner.on('message', (event) => {
  	console.log('Event received', event);
	// Update DB
	// Do things here
	facilitate(event);
});


function facilitate(event) {
	// Check if gateway address is known
	// Check if gasPrice and gasLimit is sufficient to perform facilitation.
	// Get worker address (round robin .. etc.)
	// Validate worker address can perform transactions.
	// Fork a new child process to do facilitation.

	const facilitator = fork('facilitate.js'); 
	facilitator.send(< Object with all params that are needed for facilitation>);

	// Subscribe to all the events to handle the callbacks
    }
    

Thank you for the extra effort, however I think these details should be left to the implementer. What do you think?

Facilitator as a hunter:

There can be one more child process that can act as Hunter which keeps looking for stake and mint events. Not in the scope for now.

DB proposal:

Please note: This will give a rough idea, the schema can be improved by the implementer.

This executable should resume any pending or failed task when started. To enable this the status of each StakeRequest must be persisted. We can use a database to achieve this (e.g: sqlite3).

All the child process have the access to read, insert and update contents in the DB.

stake_requests

name	type	references	description
"stake_request_hash"	VARCHAR		The unique identifier for the table type.
"message_hash"	VARCHAR
"origin_chain_id"	INTEGER
"gateway_address"	VARCHAR
"origin_account_address"	VARCHAR
"auxiliary_account_address"	VARCHAR
"hash_lock"	VARCHAR
"unlock_secret"	VARCHAR
"transactions"	VARCHAR		This will contain transaction hashed of the events(json string)
"pid"	INTEGER		This is process id, this is assigned when the facilitation is in progress by a worker process.
"state"	INTEGER		Enum: REQUESTED ACCEPTED ACCOUNT_PROVEN STAKE_INTENT_COMFIRMED PROGRESS_STAKE PROGRESS_MINT COMPLETE

addresses

name	type	references	description
"orign_chain_id"	INTEGER
"gateway_address"	VARCHAR
"auxiliary_chain_id"	VARCHAR
"composer_address"	VARCHAR
"cogateway_address"	VARCHAR
"anchor_address"	VARCHAR
"co_anchor_address"	VARCHAR

facilitator.ts

• This is the main file.
• This forks a new child-process (scanner, worker, etc).
• Listen to the events from all the child process and perform actions.
• Handles SIGTERM.
• At startup it checks the DB for pending, incomplete and failed facilitation tasks and forks new worker process accordingly.

scanner.ts

• This file scans the blocks, gets the transactions, gets the event of the transactions in batches.
• Keeps track of last scanned block number in memory.
• It always scans from the latest block when started.
• Emits event when StakeRequested event is found.

worker.ts

• This file basically does the facilitation.
• This will have an ethereum address associated with it in the variable (this is provided by the facilitator.ts while forking the new child-process)
• The password for this address should be in the ENV variable password_0xethereumAccountAddress, that can be used for unlocking.
• This always checks the latest state from the chain while progressing the task.
• This will generate the hashLock and unlockSecret. Update DB with these values to keep track in case this process dies.
• On every step of stake and mint, update the Status in DB.
• Emit event when stake and mint are successful or failed.

Tickets:

1. Create a new repository. (Done)
2. Finalize config.json (Review and suggest)
3. Finalize DB schema. (Review and suggest)
4. Finalize the enum value for states column instake_request table.
5. Implement DB interaction layer
   • create tables if does not exists.
   • function to insert new entry for StakeRequested event in stake_requests table.
   • function to update the state in stake_requests.
   • function to insert proof data in stake_requests.
   • function to insert and update worker address in stake_requests.
   • function to insert and update PID value in stake_requests.
   • function to insert new entry for addresses table.
6. Implement block scanner.
   • Scan from the latest block when the process starts.
   • Keep track of last block number that was scanned.
   • function to validate the chain configs
   • function to scan the blocks in batches
   • function to get transactions
   • function to get event from the transactions
   • function to emit an event when StakeRequested event is found.
   • Handle SIGTERM to inform parent process.
7. Add commander: ./facilitator start and ./facilitator --help
   following the proposed command

   ./facilitator start <config_file_path> --chains <specific_chain_ids>
   

   • The facilitator can be started by ./facilitator start.
   • config_file_path is the path for the config file. If this is not provided then the default path is used.
   • --chains <specific_chain_ids> this options can be used to specify the chains. The config file may have many chains specified, if this option is provided then the facilitator will start the process for this specific chains only. By default, it will start the process for all origin chains.
8. Implement worker. (can mosaic.js facilitator be used?)
   • This process will do facilitation.
   • Updates the DB with the latest state of facilitation.
   • function to unlock worker address.
   • function to create hash lock and unlock secret.
   • function to approve bounty amount.
   • function to accept stake request.
   • function to generate a proof.
   • function to confirm stake intent.
   • function to progress stake.
   • function to progress mint.
   • Handle SIGTERM to inform parent process.
9. Implement Facilitator
   • validates the config files.
   • function to validate gateways.
   • function to start the event scanner.
   • function to resume facilitation.
   • function to start the worker process.
   • listen to call backs of all the child processes.
   • function to select proper worker address for chains.
   • function to determine if gasPrice and gasLimit is sufficient to perform facilitations.
   • Handle SIGTERM to inform parent process.
10. Update readme

Thanks for all the effort @deepesh-kn! This is a long read 😅 I think it would be easier to digest if it was split into tickets right away and each ticket would only contain a contained scope of the overall scope.

[abhayks1]

Nice detailed epic 👍

I think it is much easier to implement and maintain, if one facilitator process tracks only a single
gateway pair. If you want to track multiple pairs, run multiple instances. Should make configuration,
account tracking, etc., etc. much easier. At the same time, I don't see it as extra burden to run
multiple processes for multiple chains. In a way, it is even more robust.```

I agree with @schemar that we should support multiple facilitator processes as it gives lot of flexibility and robustness.

I would also like to add below:

• It's very practical that a chain will receive lot of requests(hot chain) and some chains will receive relatively less number of requests(cold chain). Knowing this paradigm we can divide the chains into hot chains and cold chains.

• While starting facilitators we should be able to provide multiple chain ids not only single. This way a single Facilitator can process multiple cold chains.

• Hot chains can be processed by a single dedicated Facilitator.

Above architecture 🏗 🏛 is very common paradigm for better scaling and resource ♻️ management.

[abhayks1]

To enable this the status of each StakeRequest must be persisted. We can use a database to achieve > this (e.g: sqlite3).
It should track the states of each step of facilitation in DB. Update the DB with the latest state e.g
REQUESTED, ACCEPTED, PROGRESSED, etc.

I propose mysql here. It's popular, open source and uses InnoDB engine. Specially mysql handles the concurrency aspects better than sqlite3. It also has great community. Please see the comparison here:
https://stackoverflow.com/questions/4813890/sqlite-or-mysql-how-to-decide

[abhayks1]

./facilitator start

We should also support below commands to gracefully stop and restart the facilitator process:

• ./facilitator stop

• ./facilitator restart

[abhayks1]

Workers should avoid race on acceptStakeRequest; eg round-robin
How do different facilitators coordinate? I think this is a huge topic.

Below solution can be followed:

• Add below columns in stake_requests table:
  processed_by_worker
  processing_start_time

• Worker before starting to process, checks if there are available stake requests which are not processed by any worker.

• Worker can take lock on the stake request by updating column processed_by_worker with it's address. Also it should update processing_start_time to track when the process started.

With the above locking approach a single stake request will be processed by a single worker.

[abhayks1]

It must act as a hunter for any failed facilitation. (Out of scope for now.)
Handle SIGTERM gracefully.

Agreed, however a lot of thought should be put into what that actually means.

I agree with @schemar Martin 👍 . I feel there should be separate ticket 🎫 🎟 to handle Facilitation failures or for SIGTERM 🔴. The scope of the work will depend on happy case implementation outcome, as there could be rollback ⛔️ of multiple entities will be needed.

[gulshanvasnani]

• It should track the states of each step of facilitation in DB. Update the DB with the latest state e.g REQUESTED, ACCEPTED, PROGRESSED, etc.

I think state like StakerRevoked must be there because stake request can be revoked by staker. itself,

[gulshanvasnani]

• Worker can take lock on the stake request by updating column processed_by_worker with it's address. Also it should update processing_start_time to track when the process started.

With the above locking approach a single stake request will be processed by a single worker.

Worker after locking the stake request can die. So,I think there should be time devoted to process each request. After this time, the staker request will be unlocked and it can be picked by other workers.