In November 2021, the Taproot soft-fork activated on the Bitcoin network. I streamed my activation monitoring and helped pools not mining P2TR spends to fix their issues.

Soft-fork activations can keep Bitcoin developers awake at night. After all, up-to-date nodes reduce what is allowed under the consensus rules to a subset. If, for example, a block not adhering to a newly introduced rule is broadcast shortly after soft-fork activation, the updated nodes will reject it while the old nodes will still accept it. This could cause a chain split between up-to-date and old nodes - a situation we want to avoid.

Taproot getting activated meant that certain SegWit version 1 outputs now could only be spent according to the new consensus rules defined in BIP-341. Before activation, there were no consensus rules on how to spend these outputs. Bitcoin Core nodes wouldn’t accept and relay transactions spending P2TR (Pay-to-Taproot) outputs to their peers. You could, however, spend them directly asking a miner to include them, as I’ve demonstrated here.

Taproot Activation Livestream

I’ve set out to monitor the activation of the Taproot soft-fork scheduled for block 709632. I set up a machine that runs a Bitcoin Core node, connected it to a script I’ve written, and streamed the console output it generated. The left side of the screen showed blocks being mined with, for example, the pool name, information about the coinbase, and the number of transactions in the block. The right side showed new taproot transactions (transactions with P2TR inputs or P2TR outputs) entering my mempool. It showed the transaction size, feerate, inputs and outputs, and eventual OP_RETURN messages the transactions contained.

I expected users to create P2TR outputs in the hours before activation and broadcast P2TR spending transactions right after block 709631, where the policy rules changed to allow P2TR spends to be relayed. Then, the next block should include the first P2TR spends.

As expected, we saw P2TR output spends being broadcast right after block 709631. Some of them included an OP_RETURN message celebrating the activation.

After 18 minutes and 17 seconds, my node received block 709632 mined by F2Pool. Taproot is active, and this block could have contained P2TR spends. However, it didn’t. Why? Was there a bug? There were P2TR spends in my mempool, which should have been included. The YouTube live stream chat¹ started discussing what could have happened.

The next block was mined by AntPool about 10 minutes later. It contained five transactions creating P2TR outputs, but still, no P2TR spends. After 6 minutes and 30 seconds, AntPool found the next block. Again, this one did not include any P2TR spends. At this point, there were plenty of P2TR spending transactions in my mempool, which should have been included. Both pools previously signaled that they were ready for Taproot and upgraded their nodes.

Foundry USA mined block 709635 8 minutes and 20 seconds later. It contained all 16 P2TR spending transactions currently in my mempool, which made up 1.69% of all 945 transactions in this block. Foundry including P2TR spends in their block was a relief for everyone watching. This means miners were able to mine P2TR spends. However, it was unclear why F2Pool and AntPool hadn’t mined them.

Over the following blocks, it became clear that Poolin, SlushPool, BTC.com, and Luxor also included P2TR spends, reassuring that the soft-fork activated as planned. You can find the full live stream recording here:

Pools not mining P2TR spends

Over the next day, it became clear that F2Pool and AntPool still weren’t mining P2TR spends. It was unclear if this was a technical problem or if they didn’t like Taproot and were purposefully ignoring these transactions. When asked why F2Pool didn’t mine P2TR spends and hasn’t upgraded their infrastructure, the owner of F2Pool replied that they “Will upgrade soon”. I suspect they had upgraded but didn’t know why they weren’t mining P2TR spending transactions.

Looking at the pools that mined more than 3 blocks since taproot activation or included a P2TR spend, it's clear that F2Pool and AntPool are, very likely, NOT including P2TR spends. F2Pool already mentioned that they will upgrade their infrastructure soon. pic.twitter.com/qVOwZvWO4q

— b10c (@0xB10C) November 14, 2021

Shortly after, an F2Pool employee contacted me to help troubleshoot why they weren’t mining P2TR spends. He told me that they had upgraded their node a while ago. They asked if I could craft a P2TR spending transaction to test with testmempoolaccept on their node. The transaction was accepted by their node and showed that they indeed upgraded their node. I asked them if they could send me the output of bitcoin-cli getpeerinfo | jq .[].subver and if they are configuring their peers manually. It turned out that they were only connected to peers version /Satoshi:0.19.1/ or lower. This meant they were ready to mine P2TR spends but didn’t have any peers relaying P2TR spends to them. Only Bitcoin Core nodes version v0.21.1 or newer relay P2TR spends after activation. Other versions don’t know about the soft-fork and reject these.

It turned out that F2Pool was running up-to-date Bitcoin Core nodes but had applied an old, custom patch. This patch disconnected nodes which version string didn’t start with /Satoshi:0.1. Bitcoin Core nodes with version /Satoshi:0.21.*/ and /Satoshi:22.0/ were disconnected. It seems like they didn’t signal readiness in bad faith. They just didn’t know they weren’t ready for Taproot activation due to a legacy patch they had forgotten about. After fixing this, they mined their first P2TR spend on November 18th, four days after activation.

Update: F2Pool included their first P2TR spend (https://t.co/Kr9Z0HXdzn) today in block 710269.

F2Pool reached out, and we debugged the issue: Nodes are up-to-date but didn't have any >=v0.21.1 peers (due to custom patches and addnode) that relayed P2TR spends. That's fixed now. https://t.co/Tg9jmIUmFq pic.twitter.com/2y1WtYjt40

— b10c (@0xB10C) November 18, 2021

AntPool was now the only pool with a significant number of blocks (107 blocks mined since activation) that hadn’t included a P2TR spending transaction on the 18th. The communication with them turned out to be not as easy as with F2Pool. Alejandro De La Torre communicated with AntPool. They told him that only having connections to old peers “may be the issue”. They were running v0.21.1 and would soon upgrade to v22.0. It seems like they also weren’t signaling readiness in bad faith.

AntPool mined their P2TR spend in block 710494 on the November 20th, 2021. All pools that mined ten or more blocks since taproot activation had now included at least one P2TR spend. Alejandro and I summarized this in a bitcoin-dev mailing list post too.

Update 2: AntPool included their first P2TR spend (https://t.co/H7FcJnLxRb) in block 710494.

All pools that mined ten or more blocks since taproot activation have included at least one P2TR spend. https://t.co/oKqj8BHb6H pic.twitter.com/jHqCbPRlUS

— b10c (@0xB10C) November 20, 2021

Learnings

We’ve learned that it’s important to communicate that miners do not only need to upgrade their node but also should make sure you are at least connected to a few upgraded peers. Direct communication channels between mining pools and developers are sometimes also helpful.

I’ve received a lot of positive comments regarding the activation stream and the monitoring I did. I think it’s essential to have some monitoring in place, even if it’s only to see that everything went as smoothly as expected. Or, in case it doesn’t, to be able to react to it as soon as possible. And more generally, documenting soft-fork activations for future Bitcoin developers is a good habit. This ensures they know what to expect and might not repeat the same mistakes.