How to perform automated mining in regtest mode?

Question

I'm trying to setup a private network of bitcoin nodes for testing and experimentation. From what i understand, i have two choices while setting up a network:

1. I can run bitcoind -testnet and connect to the public test network. The testnet has nodes that will facilitate the mining of my transactions as long as i just broadcast them.
2. I can configure the node with regtest=1 and manually run the generate command whenever i want to create blocks.

I've also managed to find out that the mining is no longer a part of the core codebase and is usually in an external component.

What i don't understand is how any of these mining softwares interact with the bitcoin node. Does it periodically poll through the JSON RPC to check if the transaction pool has reached a certain threshold (in numbers or size)? If not, what triggers the beginning of a mining operation? Can the bitcoin node send out events when the transaction pool reaches a threshold?

I want to setup a similar automated process for creating blocks in my regtest network, but using the generate command instead of any mining software. I don't need a robust PoW setup in my local network. My aim is to just build an app which interacts with the nodes and makes transactions. Is writing an RPC client which periodically queries the pool size and fires a generate command on reaching a threshold the right way to proceed? Or does the bitcoin node itself provide some way to setup periodic generation of blocks?

Thanks in advance.

P.S. I'm very new to bitcoin development, so apologies if this is an obvious question.

Answer 1

The miner can generate a block candidate with the getblocktemplate call, which uses certain heuristics to construct an approximation of an fee-optimised block given the current local mempool state.

1) The question in regards to when to begin mining

Generally, the miner will want to begin mining the next block as soon as the strong chain is extended, maximising the time available to search for the solution.

However, consider building an optimal block template from a growing number of unconfirmed transactions being accepted into the mempool. This means a transaction selection so that total block fee reward is optimised, whilst ensuring no double-spend. A growing mempool provides a greater optimisation space, but takes time. Solving for the exact optimal block from an increasingly large pool of valid transactions is NP-Hard (knapsack/max independent sets) and expensive.

At some point, the gains in fee rewards of solving this optimisation problem will be offset by the opportunity cost of the miner (during optimisation, the miner hashing equipment will necessarily be idle). Simpler heuristics may optimise for a less than optimal block template (as in getblocktemplate), but enables the miner to begin mining immediately.

If you wish to apply your own block template solver, you can get a mempool dump via getrawmempool (not very efficient to acquire over RPC).

2) Regarding regtest in a local network

Is it possible to create bitcoind regtest network on Ubuntu (which itself is on Virtualbox)?