Notes on the invalid Bitcoin mainnet block at height 809478 mined by experimental, in-house MARAPool mining pool software on September 27, 2023.

On September 26, 2023, I noticed that a testnet node attached to a fork-observer instance reported frequent invalid blocks. The nodes debug.log showed multiple messages similar to the following:

ERROR: ConnectBlock: Consensus::CheckTxInputs: aca785e8.., bad-txns-inputs-missingorspent, CheckTxInputs: inputs missing/spent
InvalidChainFound: invalid block=00000000..  height=2505274  log2_work=75.527022  date=2023-09-26T15:20:05Z
InvalidChainFound: current best=00000000..  height=2505273  log2_work=75.527015  date=2023-09-26T15:13:45Z
ERROR: ConnectTip: ConnectBlock 00000000.. failed, bad-txns-inputs-missingorspent, CheckTxInputs: inputs missing/spent
InvalidChainFound: invalid block=00000000..  height=2505274  log2_work=75.527022  date=2023-09-26T15:20:05Z
InvalidChainFound: current best=00000000..  height=2505273  log2_work=75.527015  date=2023-09-26T15:13:45Z
ERROR: AcceptBlockHeader: block 00000000.. is marked invalid

This isn’t concerning by itself, as the Bitcoin testnet can be weird at times due to its low mining difficulty. No one was losing money, as testnet coins do not have any value. Someone could be testing a new block template algorithm and generating invalid blocks. Since this stopped after a few blocks, I didn’t investigate deeper after posting about it on Twitter.

On September 27, fork-observer notified me about the invalid Bitcoin mainnet block 000000000000000000006840568a01091022093a176d12a1e8e5e261e4f11853 at height 809478. Invalid blocks on mainnet are rather uncommon. Someone is losing money when they use time and energy to grind the block header of an invalid block. I checked a few monitoring nodes, and all of them saw the block and considered it invalid. BitMex Research’s forkmonitor also lists the block as invalid and attributes the block to MARAPool.

While validating this block, Bitcoin Core reported:

ERROR: ConnectBlock: Consensus::CheckTxInputs: 66dfefcdc3eeec2745c11f511f6068d62f34c34c767ba0feed47f63f01ccc2d8,
  bad-txns-inputs-missingorspent, CheckTxInputs: inputs missing/spent

This means, there was a problem validating the transaction 66dfefcd[..]¹. Specificially, a previous output refferenced in an input wasn’t found in the UTXO-set during block verification. This is usually either caused by a transaction ordering problem (output is being spent before it’s created) or a corrupt UTXO set. Problems like this always raise the question where the problem originates from and who else might be affected. Is this a bug only on the mining pool side? Or is this a bug in some version of Bitcoin Core possibly affecting the whole network?

Sjors Provoost pointed out that 66dfefcd[..], which is simple a one-input and one-output transaction, did end up being mined by Foundry USA in the competing, valid block 809484. The input of 66dfefcd[..] spends the first output of 7d18f0ee[..]². Looking at the invalid block with

bitcoin-cli getblock 000000000000000000006840568a01091022093a176d12a1e8e5e261e4f11853

(output) shows that 66dfefcd[..] is the 6th transaction while 7d18f0ee[..] is the 1454th. In a valid block, 7d18f0ee[..] should have been included before 66dfefcd[..]. This shows a transaction ordering problem. Bitcoin Core checks created block templates for validity before responding to a block template request. This check would have caught transaction ordering problems before the template was sent to miners.

Upon closer inspection, it’s noticeable that the coinbase transaction of the invalid block includes the string /MARA Pool (v092623)/. The v092623 part looks like a version number and could indicate software built on September 26, 2023 (MM-DD-YY), the day the invalid blocks on testnet first appeared. Checking the invalid blocks on testnet shows a similar but differently formatted string: /MARA Pool (v230812AUG)/. This could be August 12, 2023, when read as YY-MM-DD. While the date formats don’t match, the testnet and mainnet activity seem related.

@mononaut further inspected the invalid block and found 145 transactions that were included before their parent. They, too, came to the conclusion that this likely isn’t related to Bitcoin Core’s block template algorithm. Moreover, the transactions in the block were sorted by fee before mining, which breaks Bitcoin’s consensus rules if there are parent-child relationships in the transactions.

This is what MARA's invalid block at 809478 looks like:

- pink transactions no longer exist in the main chain
- blue transactions are invalid due to ordering (they spend an output from a transaction included later in the block) https://t.co/SJI1azOB5Z pic.twitter.com/5gY9TRA2eG

— mononaut (tx/acc) (@mononautical) September 27, 2023

Marathon Digital Holdings later confirmed that they mined an invalid block and clarified that they are using a small portion of their hash rate to experiment with a new, internal mining pool software in a development environment. This software included a bug that caused it to produce an invalid block. They also made sure to clarify that this isn’t a problem with Bitcoin Core in any way. No other mining pool should be affected.

We can confirm that Marathon did mine an invalid block. We utilize a small portion of our hash rate to experiment with our development pool and research potential methods to optimize our operations. The error was the result of an unanticipated bug that came from one of our experiments. In no way was this experiment an attempt to alter Bitcoin Core in any way. Our team noticed the invalid block around the same time as the rest of the world, and we immediately corrected the error. This incident, while unintended, underscores the robust security of the Bitcoin network, which rejected and rectified the anomaly.
― @MarathonDH

They mention that their team noticed the invalid block at the same time as everyone else. It’s, however, unclear to me why they didn’t notice the six invalid blocks on testnet a day before switching to mining on mainnet.

If a mining pool (or any other company) is interested, I’d be happy to help set up an internal and private fork-observer (FOSS) instance that can be used to monitor what your nodes consider to be the valid chain tip and alert you on invalid blocks and reorgs.

To clarify, this bug emanated from Marathon's own internal development environment. It was not related to Marathon's production pool. It was also not related to Bitcoin Core. Bitcoin functioned exactly as designed by excluding the invalid block.
― @MarathonDH

Marathon ends their tweet with “Bitcoin functioned exactly as designed by excluding the invalid block”. I agree - this was just a 6.43701991 BTC (USD $170k at the time of writing) bug that could probably have been avoided.