Notes on the invalid Bitcoin mainnet block at height 809478 mined by experimental, in-house MARAPool mining pool software on September 27, 2023.
On September 26, 2023, I noticed that a testnet node attached to a
fork-observer instance reported frequent invalid blocks. The nodes
showed multiple messages similar to the following:
ERROR: ConnectBlock: Consensus::CheckTxInputs: aca785e8.., bad-txns-inputs-missingorspent, CheckTxInputs: inputs missing/spent InvalidChainFound: invalid block=00000000.. height=2505274 log2_work=75.527022 date=2023-09-26T15:20:05Z InvalidChainFound: current best=00000000.. height=2505273 log2_work=75.527015 date=2023-09-26T15:13:45Z ERROR: ConnectTip: ConnectBlock 00000000.. failed, bad-txns-inputs-missingorspent, CheckTxInputs: inputs missing/spent InvalidChainFound: invalid block=00000000.. height=2505274 log2_work=75.527022 date=2023-09-26T15:20:05Z InvalidChainFound: current best=00000000.. height=2505273 log2_work=75.527015 date=2023-09-26T15:13:45Z ERROR: AcceptBlockHeader: block 00000000.. is marked invalid
This isn’t concerning by itself, as the Bitcoin testnet can be weird at times due to its low mining difficulty. No one was losing money, as testnet coins do not have any value. Someone could be testing a new block template algorithm and generating invalid blocks. Since this stopped after a few blocks, I didn’t investigate deeper after posting about it on Twitter.
On September 27, fork-observer notified me about the invalid Bitcoin mainnet
height 809478. Invalid blocks on mainnet are rather uncommon. Someone is losing
money when they use time and energy to grind the block header of an invalid
block. I checked a few monitoring nodes, and all of them saw the block and
considered it invalid. BitMex Research’s forkmonitor also lists the block as
invalid and attributes the block to MARAPool.
While validating this block, Bitcoin Core reported:
ERROR: ConnectBlock: Consensus::CheckTxInputs: 66dfefcdc3eeec2745c11f511f6068d62f34c34c767ba0feed47f63f01ccc2d8, bad-txns-inputs-missingorspent, CheckTxInputs: inputs missing/spent
This means, there was a problem validating the transaction
Specificially, a previous output refferenced in an input wasn’t found in the
UTXO-set during block verification. This is usually either caused by a
transaction ordering problem (output is being spent before it’s created) or a
corrupt UTXO set. Problems like this always raise the question where the problem
originates from and who else might be affected. Is this a bug only on the mining
pool side? Or is this a bug in some version of Bitcoin Core possibly affecting
the whole network?
Sjors Provoost pointed out that
66dfefcd[..], which is
simple a one-input and one-output transaction, did end up being mined by Foundry
USA in the competing, valid block 809484.
The input of
66dfefcd[..] spends the first output of
7d18f0ee[..]2. Looking at
the invalid block with
bitcoin-cli getblock 000000000000000000006840568a01091022093a176d12a1e8e5e261e4f11853
66dfefcd[..] is the 6th transaction while
7d18f0ee[..] is the 1454th.
In a valid block,
7d18f0ee[..] should have been included before
This shows a transaction ordering problem. Bitcoin Core checks created block
templates for validity before responding to a block template request. This check
would have caught transaction ordering problems before the template was sent to
Upon closer inspection, it’s noticeable that the coinbase transaction of the
invalid block includes the string
/MARA Pool (v092623)/. The
looks like a version number and could indicate software built on September 26,
2023 (MM-DD-YY), the day the invalid blocks on testnet first appeared. Checking
the invalid blocks on testnet shows a similar but differently formatted string:
/MARA Pool (v230812AUG)/. This could be August 12, 2023, when read as
YY-MM-DD. While the date formats don’t match, the testnet and mainnet activity
@mononaut further inspected the invalid block and found 145 transactions that were included before their parent. They, too, came to the conclusion that this likely isn’t related to Bitcoin Core’s block template algorithm. Moreover, the transactions in the block were sorted by fee before mining, which breaks Bitcoin’s consensus rules if there are parent-child relationships in the transactions.
This is what MARA's invalid block at 809478 looks like:— mononaut (@mononautical) September 27, 2023
- pink transactions no longer exist in the main chain
- blue transactions are invalid due to ordering (they spend an output from a transaction included later in the block) https://t.co/SJI1azOB5Z pic.twitter.com/5gY9TRA2eG
Marathon Digital Holdings later confirmed that they mined an invalid block and clarified that they are using a small portion of their hash rate to experiment with a new, internal mining pool software in a development environment. This software included a bug that caused it to produce an invalid block. They also made sure to clarify that this isn’t a problem with Bitcoin Core in any way. No other mining pool should be affected.
We can confirm that Marathon did mine an invalid block. We utilize a small portion of our hash rate to experiment with our development pool and research potential methods to optimize our operations. The error was the result of an unanticipated bug that came from one of our experiments. In no way was this experiment an attempt to alter Bitcoin Core in any way. Our team noticed the invalid block around the same time as the rest of the world, and we immediately corrected the error. This incident, while unintended, underscores the robust security of the Bitcoin network, which rejected and rectified the anomaly.
They mention that their team noticed the invalid block at the same time as everyone else. It’s, however, unclear to me why they didn’t notice the six invalid blocks on testnet a day before switching to mining on mainnet.
If a mining pool (or any other company) is interested, I’d be happy to help set up an internal and private fork-observer (FOSS) instance that can be used to monitor what your nodes consider to be the valid chain tip and alert you on invalid blocks and reorgs.
To clarify, this bug emanated from Marathon's own internal development environment. It was not related to Marathon's production pool. It was also not related to Bitcoin Core. Bitcoin functioned exactly as designed by excluding the invalid block.
Marathon ends their tweet with “Bitcoin functioned exactly as designed by excluding the invalid block”. I agree - this was just a 6.43701991 BTC (USD $170k at the time of writing) bug that could probably have been avoided.