The 300MB default maxmempool Problem

Unconfirmed transaction are quite a hassle for bitcoin users. I recently came across an interesting problem which is not the usual “my transaction is stuck” problem.

In the last weeks of 2017 the number of unconfirmed transactions has again reached all time highs. Periods with over 100k transactions in my mempool where not unusal. Bitcoin Core limits the system memory allocated for storing unconfirmed transactions to 300MB by default. This serves as an anti-DoS feature. Fully used 300MB of RAM equal about 110MB of actual raw transaction data. This default value can be changed with the -maxmempool <n> option where n is the amount of megabytes allocated to store unconfirmed transactions.

$ bitcoin-cli getmempoolinfo
{
  "size": 123803,               # amount of tx
  "bytes": 107840125,           # raw tx bytes
  "usage": 298526272,           # tx in ram bytes
  "maxmempool": 300000000,      # maxmempool in bytes
  "mempoolminfee": 0.00016486   # min tx-fee to get accepted
}

Once 300MB of system memory are reached, Bitcoin Core starts dropping low-feerate transactions from the mempool for high-feerate transactions. Additionally a mempoolminfee-threshold is set to prevent new low fee transactions from entering the mempool.

chained mempool transactions Transactions dropping from mempool - https://mempool.observer

A node operator can increase or decrease the maxmempool option to better fit their system. I assume that for example some blockexplorers have increased their maxmempool. I know that Jochen Hoenicke for example has an node running with a maxmempool of 2GB.

The Problem

Small differences in the mempool are common. However with an increased maxmempool a nodes mempool can differ vastly from a majority of other nodes on the network. These nodes, especially when they power a blockexplorer, still many keep unconfirmed transactions the majority of the network has already dropped. This can be irritating for users.

I came a cross a particular form of this problem, when a friend asked me, why he can see a certain low-fee transaction on bitaps.com, but not on blockchain.info. My local node, with a default 300MB maxmempool, responded with an error on calling the getrawtransaction RPC with the txid.

$ bitcoin-cli getrawtransaction <txid>
error code: -5
error message:
No such mempool transaction.

However using the bitaps.com API to query the raw transaction worked fine. I tried rebroadcasting the raw transaction with sendrawtransaction. This failed.

$ bitcoin-cli sendrawtransaction <rawtx>
error code: -25
error message:
Missing inputs

Looking at the missing input I noticed that it references a unconfirmed change output from an transaction with equally low fees. However I again wasn’t able to find the parent transaction for the input in my local mempool or on blockchain.info. Only bitaps.com had it. And this transaction again referenced an unconfirmed low-fee output as an input.

With the context of the maxmempool limit this started to make sense.

My guess on what happened

My guess on what happened is, that bitaps.com has an higher maxmempool-limit than e.g. blockchain.info and therefore does not drop the transactions. A user checking bitaps.com sees the transaction, a user checking blockchain.info doesn’t.

chained mempool transactions

The real issue appears when the original sender is not aware of this and has a peer with a larger-than-default mempool. He unknowingly continues to chain transactions to a unconfirmed parent transaction that the vast majority of the network rejects. The broadcasted transaction spends an output which references a dropped transaction and therefore is invalid.

A Solution

A Solution might simply be:

Pay more fees if you want to make sure your transactions don’t get dropped.

And maybe consider doing some sort of checking if you create long chains of unconfirmed low-fee transactions. To temporarily fix this one could rebroadcast all dropped transactions in the correct order hoping that they don’t get dropped again.

Addendum 2018/02/10:

As David Harding points out the sender could use BIP125 ReplaceByFee to update a unconfirmed transaction with new outputs.

An alternative solution for the original sender is to use BIP125 RBF to update his first unconfirmed transaction with new outputs rather than creating additional new transactions. This is more efficient and so requires less overall fee, and it avoids the problem you identified.

– David A. Harding on 22. Jan. 2018

Written on December 18, 2017