SRI Template Provider using ZMQ and RPC

[Sjors]

An alternative approach for implementing the Template Provider was suggested during review.

Rather than having Bitcoin Core accept incoming network connections, deal with Noise encryption and stratum messages, an external tool would do this. This tool would get its information from the node via ZMQ and RPC:

1. a new zmq publisher, e.g. -zmqpubtemplate, which publishes block templates as soon as they become available.
2. a new RPC updatetemplatepub to configure the template publisher, e.g. for setting the coinbase output data size
3. a new RPC submitsolution

The external tool should probably be part of SRI, e.g. by adding roles/template_provider.

However, in this scenario it may not make sense to have that role in the first place. It just needs to be code shared by the The Job Declarator Client and the Pool roles.

It has similar implications as the idea of making bitcoind a Job Declarator Client, discussed in stratum-mining/sv2-spec#85. Except in that scenario we had to keep the Template Provider role around because the pool needs it, whereas here that role could disappear entirely.

I could look into adding those two things in a new Bitcoin Core branch. But someone else would have to take on the SRI side.

There are a few downsides to this approach:

1. It limits the number of connected stratum clients to 1, since there's only one ZMQ template feed. That's probably fine for the standard suggested workflows.
2. It means users have to install two pieces of software. That's already the case, but it removes any prospect of avoiding that such as `bitcoind` as a JDC (without making it into a proxy) sv2-spec#85.
3. It precludes the ability to make the template provider public facing, even with an external tool (though from the Bitcoin Core perspective, this is already very challenging to do safely)

The advantage is that it (significantly) decreases the complexity added to Bitcoin Core.

Not sure if we actually want to go this route, but we can the discuss the pros and cons here.

[Fi3]

Having the additional role make sense also in this scenario, for example to allow public facing TPs.

[Fi3]

Going this route was my original plan and then we/I ruled it out (I'm trying hard to remember why but I don't). Honestly to me seems that the advantages exceed the disadvantages. We don't even need to refactor the spec, and we will be much more flexible for example adding an extension that now need to be implemented in core will most likley be done in the TP (not core) role much faster

[Fi3]

@TheBlueMatt what do you think?

[Fi3]

Another advantage of separeted TP is that the JDS don't need anymore to talk bitcoind rpc

[Sjors]

for example to allow public facing TPs

Having a public facing TP won't work in this design, or at least you'd have to pick one additional_coinbase_output_data_size number (like 4000) for all clients (probably not too bad).

I'm trying hard to remember why but I don't

Hopefully someone remembers...

[Sjors]

JDS don't need anymore to talk bitcoind rpc

I think it does. The protocol between the JDC and JDS wouldn't change, the latter still needs to check against its mempool.

JDC, JDS and the pool (unless we keep the TP role) will need RPC access.

[plebhash]

instead of writing a new comment quoting a previous comment, this could have been just a in-thread reply to the original comment

the ability to do in-thread discussions is one of the features that makes Github Discussions better suited for these kind of interactions when compared to issues, and it's one of the reasons I started encouraging using this feature more often on SRI

but also I'm being kinda pedantic here, so no need to take this too seriously!

[Sjors]

I think this broke while moving from issues to discussions.

[Sjors]

Is there (good) Rust support for ZMQ?

[Fi3]

JDS don't need anymore to talk bitcoind rpc

I think it does. The protocol between the JDC and JDS wouldn't change, the latter still needs to check against its mempool.

JDC, JDS and the pool (unless we keep the TP role) will need RPC access.

yep I mean with TP role. It will be easier to implement a JDS (I expect pools to implement custom JDS not using the SRI one). We can just add an extension that do what now we are doing with RPC in the JDS.

[Fi3]

Is there (good) Rust support for ZMQ?

It seems so https://crates.io/search?q=zeromq&sort=downloads

[plebhash]

cross-referencing from braidpool:

Here's a summary of the tradeoffs for each option, ranked according to my perceived fitness the problem at hand:

• zmq: the most popular crate for ZMQ, and the most likely to recieve long-term maintenance. Doesn't provide async out-of-the-box.
• async_zmq: an async port of the crate mentioned above, which allows for simpler code. The project is relatively less popular than its sync counterpart, so I wonder if it could be abandoned in the future.
• bitcoincore-zmq: goes straight to the point (Bitcoin+ZMQ), but the project is very new and seems WIP (lots of TODOs on the README).
• bitcoin-net-zmq: also has the advantage of being a Bitcoin+ZMQ implementation. It is actually part of the larger effort bitcoin-rs, which is a WIP translation of C++ into Rust. There's no documentation.
• futures-zmq: pretty much dead.
• tokio-zmq: pretty much dead.
• bitcoin-zmq: pretty much dead.

While searching for zmq on crates.io there's actually many other options. But most of them follow the patterns described above (low adoption, not actively maintained).
...
Since we have to decode all ZMQ messages, I reconsidered bitcoincore-zmq (Option 3). Even though the project is still WIP, it already implements everything we need for now (logic for decoding hashtx, hashblock, rawblock, rawtx, and sequence topics, including integration tests with bitcoind).

[TheBlueMatt]

It limits the number of connected stratum clients to 1, since there's only one ZMQ template feed. That's probably fine for the standard suggested workflows.

In principle its okay because you can always just run a translator from ZMQ to a multi-TP listener, but...

It means users have to install two pieces of software.

making people run two pieces of software really sucks. Its more crap to debug when it goes wrong, and in general getting people to run one piece of software is hard, two is nearly impossible. Historically any time we've thought about some contrib script in Bitcoin Core or written something for people to run its been run exactly never. If its more than a config flag on software people are already running its generally hard to get people to do it.

That's already the case, but it removes any prospect of avoiding that such as stratum-mining/sv2-spec#85.

IMO this is a major issue. Its a travesty that its not trivial to take an ASIC and point it at a Bitcoin Core node. Sure, Sv2 support on actual miners is a ways off, but I've heard from many people over the years that they have some home miner for heating or whatever and just want to solo mine, but they've tried two or three pieces of software and didn't get it to work. Instead, people end up paying a few-% fee to some "solo pool" which is ab absolutely absurd state - if people want to improve the decentralization in the Bitcoin network, why the hell don't we have software that lets them.

It precludes the ability to make the template provider public facing, even with an external tool

Not quite sure I understand why this is precluded? Or do you mean because it could be built into the proxy, which would preclude it? I think this is an issue and we should at least build a ZMQ -> template provider thing to support this.

(though from the Bitcoin Core perspective, this is already very challenging to do safely)

Why? Bitcoin Core is designed to accept connections from untrusted parties and interact with them. The TP protocol is almost entirely send, so supporting untrusted connections to it safely should be rather trivial?

[Sjors]

making people run two pieces of software really sucks.

I tend to agree. It's made worse by the fact that external software doesn't have first class access to the node. RPC, P2P and ZMQ are all not great in different ways. And both the node and this external piece of software have to keep running 24/7 unattended, or the ASIC(s) will just push their shares into the void without complaining.

Why? Bitcoin Core is designed to accept connections from untrusted parties and interact with them.

This referred back to @dergoegge's statement bitcoin/bitcoin#29432 (review):

I would strongly discourage us from expanding on public facing interfaces that process untrusted inputs. We are busy enough testing the existing interfaces.

I take that to be a generic statement against expanding the attack surface, not something specific to sv2.

Not quite sure I understand why this is precluded?

So from the outside in:

External node 1 --- stratum ---> TP ---- RPC --> Node
                                 |   ^-- ZMQ --- 
External node 2 --- stratum ---> ^                    
External node 3 --- stratum ---> ^                                  

Each external node can communicate its preferred CoinbaseOutputSize to the TP. The TP then needs to make RPC calls to set the values for each node. The node would then for each new block and increased fee generate templates for each CoinbaseOutputSize send those in sequence over ZMQ to the TP, which then needs to split them and return. It also has track disconnections. So I guess it's not precluded but it sounds rather painful to build.

I think IPC would allow for a more sane architecture.

[Sjors]

At this point I think the most likely alternative approach to bitcoin/bitcoin#29432 will use IPC, not ZMQ and RPC.

I have a working prototype in Sjors/bitcoin#48.

Closing this for now. Though if someone ends up implementing the approach here, please reopen.