b10c's blog: Bitcoin Developer on b10c's blog

localprobe: detect Bitcoin nodes running on the same machine as your (Firefox) browser

Sat, 28 Feb 2026 00:00:00 +0000

You’re running a Bitcoin node on the same machine as your (Firefox) web browser? Yeah, I and everybody else can tell…

localprobe is a small JavaScript snippet I built at the btc++ Floripa 2026 exploits hackathon, where it won 2nd place. It detects whether you are running a Bitcoin node on the same machine as your Firefox browser and alerts you if so.

The project has a dedicated page at 0xb10c.github.io/localprobe.

Firefox, unlike Chromium-based browsers (Chrome, Brave, Edge), allows web pages to make cross-origin requests to localhost. This means any website you visit in Firefox can probe ports on your machine. Chromium-based browsers block this via the Private Network Access spec. Firefox is actively implementing Local Network Access, with a gradual rollout starting in Firefox 147 for users with Enhanced Tracking Protection set to Strict. Tor Browser is also not affected, though not due to LNA — it blocks access to localhost by default for privacy reasons.

localprobe probes the default Bitcoin Core RPC and P2P ports for mainnet, testnet3, testnet4, signet, and regtest, as well as Tor control and proxy ports. If any of these ports respond, it shows a privacy warning alerting you that any website you visit in Firefox can detect that you’re running a node.

A privacy alert from localprobe.

You can test it by running bitcoind -regtest and visiting 0xb10c.github.io/localprobe.

Recap: A network wide DoS bug in Bitcoin Core

Thu, 26 Feb 2026 00:00:00 +0000

I did a recap on the a network wide DoS bug and highlighted a metrics that we are monitoring now to catch similar bugs in the future.

Slides (Google Slides)

Monitoring Workshop at Vinteum BDL 2026

Sat, 21 Feb 2026 00:00:00 +0000

I held a four hour long monitoring workshop for the current Vinteum Bitcoin Developer Launchpad cohort. I talked about my experience working on open-source Bitcoin projects over the last few years and I went over my Map of Bitcoin Monitoring I created as an overview for this workshop.

OpenSats Report 8

Sat, 31 Jan 2026 00:00:00 +0000

This is a slighly edited copy of the 8th report I sent to OpenSats for my LTS grant. Note that I may have redacted some information that is not or not yet meant to be published.

What did you work on?

In November and December 2025, and January of 2026, I primarily focused on building out peer-observer tooling and infrastructure, but also worked on mainnet-observer, posted to the Bitcoin Network Operations Collective, contributed to Bitcoin Core, helped in the rust-bitcoin corepc and bip324 libraries, and more.

Publications / Podcasts / Talks

Attended the Chaincode Labs Bitcoin Research Week 2025 and led an participated in sessions there.
Presented on the Chaincode Labs Bitcoin Research Day 2025 on “Monitoring Bitcoin’s P2P network: Insights from the peer-observer Project”
Spoke with Stephan Livera about “Monitoring the Bitcoin Network” in SLP707: https://stephanlivera.com/episode/707/

BNOC

In the Bitcoin Network Operations Collective discourse forum I posted about:

peer-observer

tooling (peer-observer/peer-observer)

Added metrics for connections to and from bitprojects.io: peer-observer/peer-observer#300. These showed that there is indeed a problem as Bitcoin Core can make multiple outbound connections to the bitproject.io IPs operated by the same entity. See discussion in https://bnoc.xyz/t/many-connections-to-bitproject-io-nodes/40/6
Add support for inv_to_send and cpu_load fields from getpeerinfo to the peer-observer rpc-extractor in peer-observer/peer-observer#299. inv_to_send metrics allow tracking if a bug from the past is really fixed (see e.g. https://bnoc.xyz/t/increased-b-msghand-thread-utilization-due-to-many-352-byte-runestone-inscriptions-on-2025-11-17/58/6). cpu_load is proposed in bitcoin/bitcoin#31672 and collecting data on it allows providing feedback on the usefulness of that metric.
Added support for inv and feefilter messages for the p2p-extractor peer-observer/peer-observer#297. These allow us to check when and how often feefilter and inv messages are send from our node and what contents are included. See e.g. WTx inv rate chart (and below) in https://bnoc.xyz/t/increased-b-msghand-thread-utilization-due-to-many-352-byte-runestone-inscriptions-on-2025-11-17/58/6
Fixed addr service bit metrics to be 64 bit instead of 32 bit in peer-observer/peer-observer#294 and peer-observer/peer-observer#286
Fixed peer-observer extractors and tools not exiting if a RuntimeError occurred. They were still waiting for a Ctrl+C: peer-observer/peer-observer#292.
Added an event for received addrv2 messages to the p2p-extractor peer-observer/peer-observer#296. These are useful for showing metrics for, e.g., the size of the last address announcement the node sent, total number of addresses announced per announcement, rate of the announcements. Metrics were added in peer-observer/peer-observer#306.
Started a brainstorming issue for a better NATS subject for events from extractors to tools. This should allow using one NATS server for multiple nodes: peer-observer/peer-observer#313
Improved peer-observer CI by adding a lint job (peer-observer/peer-observer#318, peer-observer/peer-observer#326) and splitting up the jobs to run in parallel peer-observer/peer-observer#325. Also updated the NixOS version in peer-observer/peer-observer#322
Switched to using addconnection in the peer-observer integration tests, as this 1) made the tests faster (-addnode is only re-executed every minute) and 2) makes the test more stable on slower test runners (such as GHA). peer-observer/peer-observer#324
Added Grafana dashboards for some recently added metrics: peer-observer/peer-observer#317
Added documentation for the required websocket.json file describing the available websocket endpoints of the websocket tool. peer-observer/peer-observer#311
Added a overview HTML page for the websocket tools peer-observer/peer-observer#310
Cleaned up the protobuf definitions in peer-observer/peer-observer#305 which is required for e.g. the archiver tool which requires a stable protobuf API.
Started a discussion on how to best parse Bitcoin Core debug.logs for a the log-extractor in peer-observer/peer-observer#336. Discussing ML methods and literature, but also code based ways of extracting log statements and their arguments.
I added support for authenticated NATS to peer-observer in peer-observer/peer-observer#337 and to my NixOS module in 0xB10C/nix#282. This allows me to open a NATS server to the outside world without authentication for subscribers, but authentication required for publishers. This enables sharing a data stream of NATS events to external contributors, e.g. as part of the Vinteum residency, BOSS programm, B4OS, or other developer education programs.
Added support for getorphantxs for the rpc-extractor in peer-observer/peer-observer#340 to allow having metrics for the transaction orphanage.
Added support the getrawaddrman to the rpc-extractor in peer-observer/peer-observer#345. This allows tracking e.g. addrman changes in metrics.
Updated the LinkingLion IPs to the new 2026 IPs they seem to use now (https://bnoc.xyz/t/linkinglion-an-entity-linking-bitcoin-transactions-to-ips/12/11) in peer-observer/peer-observer#335.

infrastructure library (peer-observer/infra-library)

Upgraded to NixOS 25.11: peer-observer/infra-library#36
Upgraded the default Bitcoin Core master commit used: peer-observer/infra-library#32 and peer-observer/infra-library#67
Allow custom Bitcoin Core P2P ports: peer-observer/infra-library#29
Fixed intermittent integration test failures: peer-observer/infra-library#26
Added a CI job that opens an auto-update PR for my nix flake (containing the peer-observer package and module) each day, if there’s an update available. peer-observer/infra-library#23 and TODO
Added support for the new log-extractor in peer-observer/infra-library#40 and 0xB10C/nix#247

Bitcoin Core

bitcoin/bitcoin

Reviewed the tracepoint changes in bitcoin/bitcoin#33680
Posted a bit of data for the cpu_load PR: bitcoin/bitcoin#31672
Opened an issue for (too) frequent PCP warnings: bitcoin/bitcoin#34114
Fixed a minor log rate-limiting bug of a log message in bitcoin/bitcoin#34008
Added a functional test for the IP address self-announcement feature of Bitcoin Core, which previously didn’t have any test coverage, while playing an essential part in nodes on the network being able find others: bitcoin/bitcoin#34039. The test had a bug which caused it to intermittently fail, which only appeared on the CI runners. I fixed this in bitcoin/bitcoin#34204.
Made sure that IP address self-announcements are sent in a separate message first: bitcoin/bitcoin#34146

bitcoin-core/guix.sigs

Build and uploaded signed GUIX signatures for the following releases and release candidates:

v29.3rc1 bitcoin-core/guix.sigs#2164
v29.3rc2 bitcoin-core/guix.sigs#2180
v30.1rc1 bitcoin-core/guix.sigs#2073
v30.1 release bitcoin-core/guix.sigs#2104, bitcoin-core/guix.sigs#2087
v30.2rc1 bitcoin-core/guix.sigs#2115
v30.2 release bitcoin-core/guix.sigs#2149

0xb10c/mainnet-observer

added a chart for signature counts of Schnorr and ECDSA: 0xB10C/mainnet-observer#97
fixed a the extension of the downloaded png files to be .png and not .png.png: 0xB10C/mainnet-observer#99
reviewed PR adding support for tracking ephemeral dust: 0xB10C/mainnet-observer#103 and added charts for it in 0xB10C/mainnet-observer#119
added a chart for BIP-54 Consensus Cleanup locktimes set by mining pools 0xB10C/mainnet-observer#112. Since all pools need to be forward-compatible with BIP-54, tracking the share of hashrate being forward-compatible is interesting. Sadly, no pools are forward compatible yet (as of writing). See https://mainnet.observer/charts/transactions-coinbase-locktime-bip54/. At the same time, it’s interesting to track what percentage of hashrate sets a non-BIP-54 locktime on in their coinbase transaction: https://mainnet.observer/charts/transactions-coinbase-locktime/
Added a Cumulative OP_RETURN bytes chart in 0xB10C/mainnet-observer#111. See https://mainnet.observer/charts/outputs-cumulative-opreturn-data-bytes/.
Added a stats_version to improve 1) re-scanning of blocks when we add or change a stat and 2) make sure we have all blocks in the database, by improving the way we determine which blocks to insert: 0xB10C/mainnet-observer#113. This uncovered a bug in the database insertion, which I fixed in 0xB10C/mainnet-observer#120.
Beefed up the CI a bit in 0xB10C/mainnet-observer#116 and added linting support. As part of this, addressed all linter warnings first 0xB10C/mainnet-observer#115.
Added a chart and a table for ephemeral dust usage in 0xB10C/mainnet-observer#119 and for P2A in 0xB10C/mainnet-observer#123. See https://mainnet.observer/charts/mining-pools-mining-p2a/ and https://mainnet.observer/charts/mining-pools-mining-ephemeral-dust/. These were requested by people working on LN implementations using P2A and ephemeral anchors. This was also useful for https://bnoc.xyz/t/bitcoin-core-versions-run-by-mining-pools/57/5.
Reviewed PRs by contributors (thank you!):
- Coinbase output type and count stats: 0xB10C/mainnet-observer#118
- Fix: add bzip2.dev and clippy to shell.nix: 0xB10C/mainnet-observer#108
- Total number of inputs and outputs by script type over time: 0xB10C/mainnet-observer#107
- Add OP_RETURN bytes chart: 0xB10C/mainnet-observer#106
- Update type name order of segwit inputs: 0xB10C/mainnet-observer#105
- add: track ephemeral dust 0xB10C/mainnet-observer#103

misc

rust-bitcoin/bip324

I plan to use this library in an upcoming project. Playing around with it to test it, fixing some low-hanging bugs, and getting more familiar with it made sense.

I noticed a bug where reading on a closed bip324 connection (e.g. the other side went offline), didn’t produce an error. This was a bug in the implementation and I fixed it in: rust-bitcoin/bip324#160. I also added a integration test for coverage of this bug in the async code.
I ported that test to the sync code (from async) and added it in rust-bitcoin/bip324#163.
I went ahead and picked up an issue to update the bip324 test vectors from the BIP in the bip324 library in rust-bitcoin/bip324#162.

nixos/nixpkgs

I updated Bitcoin Core from v30.1 to v30.2 shortly after release NixOS/nixpkgs#478769. Bitcoin Core v30.0 and v30.1 had a wallet bug, so getting the update in quickly was a priority.

0xb10c/nix

Next to keeping my packages up-to-date and maintaining existing packages and modules, I also:

added a new package and module for @jamesob’s discourse-archive in 0xB10C/nix#283 to be able to automatically backup delvingbitcoin.org and bnoc.xyz discourse instances.

rust-bitcoin/corepc

I use rust-bitcoin/corepc in some of my monitoring tools, so it makes sense to spent some of my time adding features & tests, reporting bugs, and implementing fixes.

Added support for the hidden getorphantxs RPC to corepc along with integration test rust-bitcoin/corepc#435. This RPC is needed for peer-observer: peer-observer/peer-observer#340.
Added support for Bitcoin Core v30.2 and use it in integration tests. v30.x wasn’t covered by integration tests before rust-bitcoin/corepc#469.
Reported a bug that the GetMempoolInfo model FeeRate conversion is broken with a clear reproducer: rust-bitcoin/corepc#482

0xb10c/github-metadata-backup

#11: Reviewed and tested a contribution that added support for saving incremental progress on failure.
#13: Reviewed and tested a contribution that mitigated some backup failures when an issue or PR has not-yet-handled events.
#14: Upgraded the octocrab dependency to support not-yet-handled events.

bitcoin-data/block-arrival-times

Added block arrival timestamps for about 100k blocks from 6 nodes to the block-arrival-times dataset after a request from a researcher: bitcoin-data/block-arrival-times#25

bitcoin-data/stale-blocks

changed the get-data.py script to always fetch the full block, if possible: bitcoin-data/stale-blocks#54
added stale blocks at height:
- 922047 bitcoin-data/stale-blocks#53
- 923067 bitcoin-data/stale-blocks#56
- 925051 bitcoin-data/stale-blocks#57
- 926056 & 925605 bitcoin-data/stale-blocks#58
- 926981 bitcoin-data/stale-blocks#59
- 928484 & 928222 bitcoin-data/stale-blocks#60
- 929868 & 929872 bitcoin-data/stale-blocks#61

OpenSats Work-Log 8

Sat, 31 Jan 2026 00:00:00 +0000

This is a copy of the 8th work-log I sent to OpenSats for my LTS grant.

Disclaimer: Some information that is not (or not yet) meant to be published may have been redacted.

What did you work on?

Publications / Podcasts / Talks

Attended the Chaincode Labs Bitcoin Research Week 2025 and led an participated in sessions there.
Presented on the Chaincode Labs Bitcoin Research Day 2025 on “Monitoring Bitcoin’s P2P network: Insights from the peer-observer Project”
Spoke with Stephan Livera about “Monitoring the Bitcoin Network” in SLP707: https://stephanlivera.com/episode/707/

Bitcoin Network Operations Collective

In the Bitcoin Network Operations Collective discourse forum I posted about:

peer-observer

tooling (peer-observer/peer-observer)

Added metrics for connections to and from bitprojects.io: peer-observer/peer-observer#300. These showed that there is indeed a problem as Bitcoin Core can make multiple outbound connections to the bitproject.io IPs operated by the same entity. See discussion in https://bnoc.xyz/t/many-connections-to-bitproject-io-nodes/40/6
Add support for inv_to_send and cpu_load fields from getpeerinfo to the peer-observer rpc-extractor in peer-observer/peer-observer#299. inv_to_send metrics allow tracking if a bug from the past is really fixed (see e.g. https://bnoc.xyz/t/increased-b-msghand-thread-utilization-due-to-many-352-byte-runestone-inscriptions-on-2025-11-17/58/6). cpu_load is proposed in bitcoin/bitcoin#31672 and collecting data on it allows providing feedback on the usefulness of that metric.
Added support for inv and feefilter messages for the p2p-extractor peer-observer/peer-observer#297. These allow us to check when and how often feefilter and inv messages are send from our node and what contents are included. See e.g. WTx inv rate chart (and below) in https://bnoc.xyz/t/increased-b-msghand-thread-utilization-due-to-many-352-byte-runestone-inscriptions-on-2025-11-17/58/6
Fixed addr service bit metrics to be 64 bit instead of 32 bit in peer-observer/peer-observer#294 and peer-observer/peer-observer#286
Fixed peer-observer extractors and tools not exiting if a RuntimeError occurred. They were still waiting for a Ctrl+C: peer-observer/peer-observer#292.
Added an event for received addrv2 messages to the p2p-extractor peer-observer/peer-observer#296. These are useful for showing metrics for, e.g., the size of the last address announcement the node sent, total number of addresses announced per announcement, rate of the announcements. Metrics were added in peer-observer/peer-observer#306.
Started a brainstorming issue for a better NATS subject for events from extractors to tools. This should allow using one NATS server for multiple nodes: peer-observer/peer-observer#313
Improved peer-observer CI by adding a lint job (peer-observer/peer-observer#318, peer-observer/peer-observer#326) and splitting up the jobs to run in parallel peer-observer/peer-observer#325. Also updated the NixOS version in peer-observer/peer-observer#322
Switched to using addconnection in the peer-observer integration tests, as this 1) made the tests faster (-addnode is only re-executed every minute) and 2) makes the test more stable on slower test runners (such as GHA). peer-observer/peer-observer#324
Added Grafana dashboards for some recently added metrics: peer-observer/peer-observer#317
Added documentation for the required websocket.json file describing the available websocket endpoints of the websocket tool. peer-observer/peer-observer#311
Added a overview HTML page for the websocket tools peer-observer/peer-observer#310
Cleaned up the protobuf definitions in peer-observer/peer-observer#305 which is required for e.g. the archiver tool which requires a stable protobuf API.
Started a discussion on how to best parse Bitcoin Core debug.logs for a the log-extractor in peer-observer/peer-observer#336. Discussing ML methods and literature, but also code based ways of extracting log statements and their arguments.
I added support for authenticated NATS to peer-observer in peer-observer/peer-observer#337 and to my NixOS module in 0xB10C/nix#282. This allows me to open a NATS server to the outside world without authentication for subscribers, but authentication required for publishers. This enables sharing a data stream of NATS events to external contributors, e.g. as part of the Vinteum residency, BOSS programm, B4OS, or other developer education programs.
Added support for getorphantxs for the rpc-extractor in peer-observer/peer-observer#340 to allow having metrics for the transaction orphanage.
Added support the getrawaddrman to the rpc-extractor in peer-observer/peer-observer#345. This allows tracking e.g. addrman changes in metrics.
Updated the LinkingLion IPs to the new 2026 IPs they seem to use now (https://bnoc.xyz/t/linkinglion-an-entity-linking-bitcoin-transactions-to-ips/12/11) in peer-observer/peer-observer#335.

infrastructure library (peer-observer/infra-library)

Upgraded to NixOS 25.11: peer-observer/infra-library#36
Upgraded the default Bitcoin Core master commit used: peer-observer/infra-library#32 and peer-observer/infra-library#67
Allow custom Bitcoin Core P2P ports: peer-observer/infra-library#29
Fixed intermittent integration test failures: peer-observer/infra-library#26
Added a CI job that opens an auto-update PR for my nix flake (containing the peer-observer package and module) each day, if there’s an update available. peer-observer/infra-library#23 and TODO
Added support for the new log-extractor in peer-observer/infra-library#40 and 0xB10C/nix#247

Bitcoin Core

bitcoin/bitcoin

Reviewed the tracepoint changes in bitcoin/bitcoin#33680
Posted a bit of data for the cpu_load PR: bitcoin/bitcoin#31672
Opened an issue for (too) frequent PCP warnings: bitcoin/bitcoin#34114
Fixed a minor log rate-limiting bug of a log message in bitcoin/bitcoin#34008
Added a functional test for the IP address self-announcement feature of Bitcoin Core, which previously didn’t have any test coverage, while playing an essential part in nodes on the network being able find others: bitcoin/bitcoin#34039. The test had a bug which caused it to intermittently fail, which only appeared on the CI runners. I fixed this in bitcoin/bitcoin#34204.
Made sure that IP address self-announcements are sent in a separate message first: bitcoin/bitcoin#34146

bitcoin-core/guix.sigs

Build and uploaded signed GUIX signatures for the following releases and release candidates:

v29.3rc1 bitcoin-core/guix.sigs#2164
v29.3rc2 bitcoin-core/guix.sigs#2180
v30.1rc1 bitcoin-core/guix.sigs#2073
v30.1 release bitcoin-core/guix.sigs#2104, bitcoin-core/guix.sigs#2087
v30.2rc1 bitcoin-core/guix.sigs#2115
v30.2 release bitcoin-core/guix.sigs#2149

0xb10c/mainnet-observer

added a chart for signature counts of Schnorr and ECDSA: 0xB10C/mainnet-observer#97
fixed a the extension of the downloaded png files to be .png and not .png.png: 0xB10C/mainnet-observer#99
reviewed PR adding support for tracking ephemeral dust: 0xB10C/mainnet-observer#103 and added charts for it in 0xB10C/mainnet-observer#119
added a chart for BIP-54 Consensus Cleanup locktimes set by mining pools 0xB10C/mainnet-observer#112. Since all pools need to be forward-compatible with BIP-54, tracking the share of hashrate being forward-compatible is interesting. Sadly, no pools are forward compatible yet (as of writing). See https://mainnet.observer/charts/transactions-coinbase-locktime-bip54/. At the same time, it’s interesting to track what percentage of hashrate sets a non-BIP-54 locktime on in their coinbase transaction: https://mainnet.observer/charts/transactions-coinbase-locktime/
Added a Cumulative OP_RETURN bytes chart in 0xB10C/mainnet-observer#111. See https://mainnet.observer/charts/outputs-cumulative-opreturn-data-bytes/.
Added a stats_version to improve 1) re-scanning of blocks when we add or change a stat and 2) make sure we have all blocks in the database, by improving the way we determine which blocks to insert: 0xB10C/mainnet-observer#113. This uncovered a bug in the database insertion, which I fixed in 0xB10C/mainnet-observer#120.
Beefed up the CI a bit in 0xB10C/mainnet-observer#116 and added linting support. As part of this, addressed all linter warnings first 0xB10C/mainnet-observer#115.
Added a chart and a table for ephemeral dust usage in 0xB10C/mainnet-observer#119 and for P2A in 0xB10C/mainnet-observer#123. See https://mainnet.observer/charts/mining-pools-mining-p2a/ and https://mainnet.observer/charts/mining-pools-mining-ephemeral-dust/. These were requested by people working on LN implementations using P2A and ephemeral anchors. This was also useful for https://bnoc.xyz/t/bitcoin-core-versions-run-by-mining-pools/57/5.
Reviewed PRs by contributors (thank you!):
- Coinbase output type and count stats: 0xB10C/mainnet-observer#118
- Fix: add bzip2.dev and clippy to shell.nix: 0xB10C/mainnet-observer#108
- Total number of inputs and outputs by script type over time: 0xB10C/mainnet-observer#107
- Add OP_RETURN bytes chart: 0xB10C/mainnet-observer#106
- Update type name order of segwit inputs: 0xB10C/mainnet-observer#105
- add: track ephemeral dust 0xB10C/mainnet-observer#103

misc

rust-bitcoin/bip324

I plan to use this library in an upcoming project. Playing around with it to test it, fixing some low-hanging bugs, and getting more familiar with it made sense.

I noticed a bug where reading on a closed bip324 connection (e.g. the other side went offline), didn’t produce an error. This was a bug in the implementation and I fixed it in: rust-bitcoin/bip324#160. I also added a integration test for coverage of this bug in the async code.
I ported that test to the sync code (from async) and added it in rust-bitcoin/bip324#163.
I went ahead and picked up an issue to update the bip324 test vectors from the BIP in the bip324 library in rust-bitcoin/bip324#162.

nixos/nixpkgs

I updated Bitcoin Core from v30.1 to v30.2 shortly after release NixOS/nixpkgs#478769. Bitcoin Core v30.0 and v30.1 had a wallet bug, so getting the update in quickly was a priority.

0xb10c/nix

Next to keeping my packages up-to-date and maintaining existing packages and modules, I also:

added a new package and module for @jamesob’s discourse-archive in 0xB10C/nix#283 to be able to automatically backup delvingbitcoin.org and bnoc.xyz discourse instances.

rust-bitcoin/corepc

I use rust-bitcoin/corepc in some of my monitoring tools, so it makes sense to spent some of my time adding features & tests, reporting bugs, and implementing fixes.

Added support for the hidden getorphantxs RPC to corepc along with integration test rust-bitcoin/corepc#435. This RPC is needed for peer-observer: peer-observer/peer-observer#340.
Added support for Bitcoin Core v30.2 and use it in integration tests. v30.x wasn’t covered by integration tests before rust-bitcoin/corepc#469.
Reported a bug that the GetMempoolInfo model FeeRate conversion is broken with a clear reproducer: rust-bitcoin/corepc#482

0xb10c/github-metadata-backup

0xB10C/github-metadata-backup#11: Reviewed and tested a contribution that added support for saving incremental progress on failure.
0xB10C/github-metadata-backup#13: Reviewed and tested a contribution that mitigated some backup failures when an issue or PR has not-yet-handled events.
0xB10C/github-metadata-backup#14: Upgraded the octocrab dependency to support not-yet-handled events.

bitcoin-data/block-arrival-times

Added block arrival timestamps for about 100k blocks from 6 nodes to the block-arrival-times dataset after a request from a researcher: bitcoin-data/block-arrival-times#25

bitcoin-data/stale-blocks

changed the get-data.py script to always fetch the full block, if possible: bitcoin-data/stale-blocks#54
added stale blocks at height:
- 922047 bitcoin-data/stale-blocks#53
- 923067 bitcoin-data/stale-blocks#56
- 925051 bitcoin-data/stale-blocks#57
- 926056 & 925605 bitcoin-data/stale-blocks#58
- 926981 bitcoin-data/stale-blocks#59
- 928484 & 928222 bitcoin-data/stale-blocks#60
- 929868 & 929872 bitcoin-data/stale-blocks#61

Bitcoin Network Monitoring with b10c: SLP707

Fri, 19 Dec 2025 00:00:00 +0000

In this episode, Stephan Livera and I discusses my work in the Bitcoin ecosystem, focusing on the importance of censorship resistance, the role of mining pools, and the implications of OFAC sanctions on Bitcoin transactions. I introduce my peer-observer project aimed at monitoring the Bitcoin P2P network for anomalies and attacks, and highlight the need for a collaborative approach to Bitcoin network operations through the Bitcoin Network Operations Collective.

stephanlivera.com/episode/707

Monitoring Bitcoin’s P2P network: Insights from the peer-observer Project

Thu, 06 Nov 2025 00:00:00 +0000

I presented about “Monitoring Bitcoin’s P2P network: Insights from the peer-observer Project” at the Chaincode Labs Bitcoin Research Day 2025.

Slides (Google Docs)

OpenSats Work-Log 7

Fri, 31 Oct 2025 00:00:00 +0000

This is a copy of the 7th work-log I sent to OpenSats for my LTS grant.

Disclaimer: Some information that is not (or not yet) meant to be published may have been redacted.

How did you spend your time?

Publications & Talks

Continued my “Stats on compact block reconstructions”: thread by posting an update and some of the current work: https://delvingbitcoin.org/t/stats-on-compact-block-reconstructions/1052/24
Created a topic for peer-observer on delving for people who want to stay up-to-date with the project: https://delvingbitcoin.org/t/peer-observer-a-tool-and-infrastructure-for-monitoring-the-bitcoin-p2p-network-for-attacks-and-anomalies/1988
At CoreDev, did a Spark session on “When do Bitcoin node operators upgrade?”. Plan to publish a blog post on this at some point.

Bitcoin Core

planned and attended CoreDev meeting: https://coredev.tech/2025_frankfurt.html
opened PR “p2p: reduce false-positives in addr rate-limiting” (https://github.com/bitcoin/bitcoin/pull/33699) based on the observation documented here https://bnoc.xyz/t/bitcoin-core-addr-gossip-is-rate-limited-right-after-connection-open/47
opened PR https://github.com/bitcoin/bitcoin/pull/33133 to fix the RPC docs of getpeerinfo ping field
opened issue “signet: disk-space-DoS due to low mining difficulty” (https://github.com/bitcoin/bitcoin/issues/33266) after previously disclosing the signet vulnerability and it being fixed
Bitcoin Core GUIX sigs for:

peer-observer

A tool and infrastrucuture to monitor for attacks and anomalies in the Bitcoin network with passive monitoring.

I worked on the tooling, the infrastructure, and a demo instance over the past few months.

Tool

https://github.com/0xB10C/peer-observer

Since August, there have been 180+ commits in that repo across 57 PRs by 4 authors. I’m only listing the most interesting ones below:

https://github.com/0xB10C/peer-observer/pull/191: added a new extractor that fetches from the getpeerinfo RPC
https://github.com/0xB10C/peer-observer/pull/256: added a new extractor that measurements against the nodes P2P interface (e.g. bitcoin-ping duration)
https://github.com/0xB10C/peer-observer/pull/272: added a new extractor that parses the debug.log file and emits events (contributed by @m4ycon; I only reviewed it)
https://github.com/0xB10C/peer-observer/pull/213: added a bunch of Grafana dashboards I stored elsewhere to the repo
https://github.com/0xB10C/peer-observer/pull/215: added a metric for LinkingLion traffic based on a discussion with gmax
https://github.com/0xB10C/peer-observer/pull/241: reduced the resource usage (CPU, bandwidth, RAM) of metrics tool by removing high-cardinallity metrics
https://github.com/0xB10C/peer-observer/pull/249: metric for peers relaying sub-1sat/vbyte txns to track adoption
https://github.com/0xB10C/peer-observer/pull/269: added new events for RPC-extractor based on getmempoolinfo events
As part of https://github.com/0xB10C/peer-observer/issues/26 (Integration test the tools and extractors):
- https://github.com/0xB10C/peer-observer/pull/222: integration tests for the metrics tool
- https://github.com/0xB10C/peer-observer/pull/229: integration tests for the websocket tool
- https://github.com/0xB10C/peer-observer/pull/268: integration tests for the logger tool
- https://github.com/0xB10C/peer-observer/pull/231: integration tests for the rpc-extractor

Infrastructure

Next to the tooling component of peer-observer, there is also a infrastructure component for running nodes, connecting the extractors to the nodes, collecting system metrics, and more. Since this is infrastructure is defined as code for NixOS systems, it’s possible to track in git and share. Previously, the infrastructure and individual honey pot host definitions were mixed and sharing them would have revealed too much information about the honey pots. However, I spent a bit of time extracting an opinionated library for defining peer-observer hosts. This has multiple benefits:

the host definitions can be tested in CI with full integration tests
the library can be published as FOSS and used by others too, if they want
the “production” implementation (i.e. “honey pot host definitions”) can remain private
the library can be used to set up a fully public demo instance

The library can be found in https://github.com/0xB10C/peer-observer-infra-library but as of writing, is still missing a bunch of documentation.

A fully public demo instance (sponsored by Localhost Research) is available on https://demo.peer.observer and the infrastructure definitions (using the above library) can be found in https://github.com/0xB10C/peer-observer-infra-demo.

Bitcoin Network Operations Collective (bnoc.xyz)

I hinted at something similar in https://b10c.me/projects/024-peer-observer/#a-bitcoin-network-operations-collective. The https://bnoc.xyz is a first step in bringing such a collective forward. While I haven’t announced it publicly yet, the forum is meant for people to post their raw network observations and allow for discussions on network events.

So far, I’ve been mainly been posting old notes there and a few others have started to post their observations too. I plan to annouce it in the next few months.

Misc

updates the NixOS bitcoind package:
- v29.0 -> v29.1: https://github.com/NixOS/nixpkgs/pull/440372
- v29.1 -> v30.0: https://github.com/NixOS/nixpkgs/pull/451606
https://github.com/jamesob/delving-bitcoin-archive/pull/1: looked into delving archive
Kept bitcoin-data/stale-blocks up-to-date with recent stale blocks:
Organized and held Socratic Seminar #4 for BTCFFM: https://github.com/btcffm/website-btcffm/pull/28

What do you plan to work on next quarter?

peer-observer-infra-library:
- work on set up documentation and better README
- plenty of other open issues
peer-observer tooling:
- continue working on open issues. The main ones are:
  - https://github.com/0xB10C/peer-observer/issues/263: more P2P-extractor events
  - https://github.com/0xB10C/peer-observer/issues/199: more RPC-extractor events
  - https://github.com/0xB10C/peer-observer/issues/185: alert on predefined heuristics
  - https://github.com/0xB10C/peer-observer/issues/121: figure out how to archive events
  - https://github.com/0xB10C/peer-observer/issues/13: finally get to anomaly detection?
bnoc.xyz
- annouce it publicly and continue filling it with content
- set up a backup and a simple HTML mirroring for bnoc
- while at it, set up a backup for delvingbitcoin too and mirror posts
Based on recent discussions, we noticed that we don’t have good data on block propagation. Possibly add some monitoring there.
blog posts:
- I have data on Bitcoin nodes upgrading, and plan to publish a blog post or similar on this at some point.
- I’m expecting to get data on historical block propagation soon. This allows to look into how e.g. non-standard transaction really affect block proagation.
The KIT DSN has been running P2P network monitoring and block propagation for close to 10 years now, however they might shutdown some of their services next year. It would be good to set up some block propatagion monitoring for our self.

Support from LocalhostResearch for peer-observer

Wed, 01 Oct 2025 00:00:00 +0000

Localhost Research supported my peer-observer project by sponsoring three servers for a demo instance, which can be found on demo.peer.observer. Compared to the public.peer.observer instance, this allows everyone to explore the metrics and data extracted from two Bitcoin Core nodes, while no information about the “production” honey pot nodes is leaked.

The full NixOS infrastructure is published on https://github.com/peer-observer/infra-demo.

OpenSats Work-Log 6

Thu, 31 Jul 2025 00:00:00 +0000

This is a copy of the 6th work-log I sent to OpenSats for my LTS grant.

Disclaimer: Some information that is not (or not yet) meant to be published may have been redacted.

How did you spend your time?

In May, June, and July of 2025 I finished mainnet-observer up and launched it, gave a talk on my peer-observer project at the Prague dev/hack/day 2025 and published a blog post about it, and more. I also took some time off to touch some grass, recharge, and enjoy summer a bit.

Publications:

I’ve written down my notes on the discovery of the Bitcoin Core DoS: “DoS due to inv-to-send sets growing too large” from May 2023. I think that having these documented somewhere is helpful and allows me to link to it. https://b10c.me/observations/15-inv-to-send-queue/
At the Prague dev/hack/day 2025 I gave a talk about my peer-observer project with the goal of monitoring the Bitcoin P2P network for attacks and anomalies. https://b10c.me/talks/#peer-observer-a-tool-and-infrastructure-for-monitoring-the-bitcoin-p2p-network-for-attacks-and-anomalies
As the talk wasn’t recorded, I also published a blog post on peer-observer to reach a wider audience and to have the project idea written down somewhere. https://b10c.me/projects/024-peer-observer/. I also have a section on a potential Bitcoin Network Operations Collective, which is something I want to further explore in the coming months.

mainnet-observer

The mainnet-observer project is an Open-Source rewrite of my 2017 transactionfee.info project (semi-closed source). It shows blockchain statistics interesting for developers and power users and is useful to make data-based decisions for protocol development. Over the course of April and in early May, I finalized an initial version ready for publishing.

In May, this included:

a new chart for UTXO set composition over time which partly addressed Murch’s issue #14 New chart request: UTXO set composition over time
A pie chart for pool hashrate distribution including AntPool & friends in #76 to have an updating version of the chart published at the bottom of Bitcoin Mining Centralization in 2025
A block-weight chart in #80
Charts for some OP_RETURN meta-protocol usage in #79
some minor and last-minute visual fixes like #71, #73, #75, #81, #82

peer-observer

Next to speaking about peer-observer and announcing the project in my blog post (see publications at the top), I also worked on a few things that had been on my list for a while.

#173 adds documentation and usage instructions on how to use the peer-observer tools - this helps with new contributors (coming from the talk and blog post) trying the tools
Initially, peer-observer was only using the eBPF/USDT interface of Bitcoin Core. The RPC interface can provide useful information too, especially stateful information. To use this data, I implemented an RPC-extractor with initially data from getpeerinfo in #191. More RPCs to implement are tracked in #199. This also caused of followup issues with other ideas to implement linked from the PR. As part of the RPC-exporter I also fixed the getpeerinfo RPC implementation of rust-bitcoin/corepc https://github.com/rust-bitcoin/corepc/pull/310
Some general maintenance in #171 and #172
Clube Bitcoin Universidade de Brasília has been starting to setup their own peer-observer infrastructure with my help. So I made some small contributions on their infra structure: https://github.com/ClubeBitcoinUnB/peer-observer-docker/pull/2

fork-observer

With “1sat/vbyte summer”, monitoring for stale blocks has become more important again as stale-blocks are an indication for poor network block propagation. By connecting to public electrum servers, we have a much better view into the network by having much more data sources. So I implemented Electrum backend support for fork-observer and set my instance up to connect to a bunch of public electrum servers on mainnet, testnet, testnet4, and signet.

misc

I started thinking about an IPC-based tracing interface for tracing Bitcoin Core (as opposed to the current USDT/eBPF interface), and asked ryanofsky, who has been spearheading the Bitcoin Core IPC and multiprocess work, for his thoughts on this in https://github.com/bitcoin-core/libmultiprocess/issues/185. He was so kind to write a quick demo of how this would look like. I haven’t found the time to dive into it, but I see this as worthwhile project to explore.
continued to maintain the website of the Frankfurt Bitcoin Meetup June meetup, July meetup
maintained github-metadata-mirror project with #6 and #8
Helping to organize an upcoming CoreDev event
Bitcoin Core GUIX sigs for v28.2 and its release candidates: https://github.com/bitcoin-core/guix.sigs/pulls?q=is%3Apr+author%3A0xB10C+is%3Aclosed

peer-observer: A tool and infrastructure for monitoring the Bitcoin P2P network for attacks and anomalies

Tue, 29 Jul 2025 00:00:00 +0000

Over the past few years, I’ve been working on monitoring tools for the Bitcoin network. One of these projects is peer-observer: A tool and infrastructure for monitoring the Bitcoin P2P network for attacks and anomalies. This post describes the motivation for starting yet another Bitcoin network observer. It details how the tool works, what my honeypot infrastructure looks like, and finishes with an idea for a decentralized Bitcoin Network Operations Collective and incident response team.

Motivation

At some point in late 2021, I stumbled across reports of an addr message flooding having happened a few months earlier on the Bitcoin network. It was first reported by Piotr Narewski, the maintainer of the Gocoin Bitcoin node implementation, in a thread called Loads of fake peers advertised on bitcoin network. Piotr details that his node was receiving “hundreds of thousands of non-working [IP] addresses” via the addr P2P message in July 2021. Since his node implementation stores all addresses¹, he is experiencing high resource usage and needs more connection attempts until a working peer is found.

Matthias Grundmann and Max Baumstark from the Karlsruhe Institute of Technology noticed this attack on their DSN Bitcoin Monitoring infrastructure, too. In a preprint for a paper, they write: “Some peers in the Bitcoin P2P network distributed a huge amount of spam IP addresses during July 2021. These spam IP addresses did not belong to actual Bitcoin peers.”.

At the same time, the Bitcoin Core PR #22387: Rate limit the processing of rumoured addresses, is being tested and reviewed. This PR implements rate-limiting for the number of addresses a peer can send to a node. Previously, Bitcoin Core would “happily accept and process an effectively unbounded rate from attackers”. During testing of this PR, reviewers note that the rate-limiting is being triggered by peers on mainnet. This is likely related to the same addr-flodding attack observed by Piotr, Matthias, and Max.

Three years later, the Bitcoin Core project discloses CVE-2024-52919 - Remote crash due to addr message spam discovered by Eugene Siegel. An attacker could remotely crash Bitcoin Core nodes by spamming addr messages that would then be inserted into the addrman. The node would crash due to a 32-bit identifier overflowing. This was fixed by the rate limiting implemented in PR #22387 and then later changing the 32-bit identifier to a 64-bit identifier.²

Learning about this attack a few months after it happened, I started to wonder what other attacks are happening on the Bitcoin P2P network. While the addr-flodding attack was detected by a few different parties, I had the feeling that it would be good to not only learn about these attacks by coincidence. The idea for another Bitcoin network monitoring tool was born.

The peer-observer tool

Early on, I decided that my monitoring should be passive and as minimally invasive to the Bitcoin network as possible. This means, for example, not to use up inbound connection slots by connecting to all possible Bitcoin nodes on the network. I decided to run multiple honeypot nodes that are normal, honest nodes that behave well and participate in block and transaction relay. They just have a lot of monitoring tools attached to them.

Since Bitcoin Core is currently the most widely used node software on the Bitcoin network, I choose to focus on it for now³. This allows for testing out various Bitcoin Core node features and configuration options that could be susceptible to attacks or anomalies (i.e., bugs).

Interfaces

To extract data and events about the P2P network and the peers connected to a Bitcoin Core node, interfaces are needed. To learn about events in real-time via machine-to-machine interface, I started implementing peer-observer primarily with the Bitcoin Core tracepoint interface I’ve been working on over the past few years. The tracepoint interface provides all data required for an MVP. As alternatives, I also considered parsing the Bitcoin Core debug.log similar to James O’Beirne’s bmon tool and fetching data from the RPC interface. The debug log is primarily an interface for humans and not machines. Parsing log messages that might change without warning over time didn’t seem optimal to me. Polling on the RPC interface doesn’t give me the required resolution. For example, between two calls to getpeerinfo, multiple peers might have connected and already disconnected.

Nonetheless, I’ve started exploring adding support for extracting data from more interfaces. The RPC interface, and particularly, getpeerinfo, can be useful to get more stateful information about peers. Since the tracepoint interface has a few pain points, I’ve also started thinking about an IPC-based alternative. In the future, it might become worthwhile to supplement the data with parsed debug.log output, while keeping in mind that the log statements might change over time.

For peer-observer, I primarily use the net tracepoints to learn about in- and outbound P2P messages and opened, closed, evicted, or misbehaving connections. The validation:block_connected tracepoint, along with mempool tracepoints, are also interesting to get insights into block processing and the node’s mempool. For a while, I’ve also maintained custom tracepoints in a patch, for example, addrman tracepoints, to see how much effect peers can have on our addrman.

Extractors and Tools

To be able to process data from multiple Bitcoin Core interfaces in multiple tools, I choose a message queue that allows for multiple publishers and consumers. Extractors, like the ebpf-extractor, hooking into the tracepoint interface, publish events into the message queue. On the other end, subscribers or Tools consume these events and further process them. The most basic tool, called logger, just logs all received events. As a message queue peer-observer is using a NATS.io server. Messages are serialized using protobuf.


┌──────────────────────┐
NATS.io │ Tools │
PUB-SUB │ │
┌──────┼──► logger │
Tracepoints │ │ │
┌───────────┐ via libbpf ├──────┼──► metrics │
│ Bitcoin │ ┌───────────────┐ │ │ │
│ Core Node ├───────► ebpf-extractor├────┼──────┼──► websocket │
└───────────┘ └───────────────┘ │ │ │
├──────┼──► addr-connectivty │
│ │ │
└──────┼──►... │
protobuf │ │
messages └──────────────────────┘

`logger` tool

The logger tool logs events to stdout and supports basic topic filtering (thanks to Nasser for PR #138). I mainly use it to show how much communication is happening between a Bitcoin Core node and its peers.

The output of the tool looks similar to the following snippet. Here, <- and -> indicate an in- and outbound P2P message to or from our node. Connection events are marked with # CONN.

--> to id=11937171 (conn_type=1): wtxidrelay
--> to id=11937171 (conn_type=1): sendaddrv2
--> to id=11937171 (conn_type=1): verack
<-- from id=10237648 (conn_type=1): Inv([WTx(205bfe2dfbeb46c7d91963a13097ef49511ad2d71c3018fdbdebbff83d8caa2f), WTx(0cd27eb1f63d95c0ec82adf0090756aef0eb1b1e840634ec7a4f440919ab991c), WTx(98bb7eb29ab06dcfdd30aa4875ebafcedd14da2738d63d0cc8d6dcc0f3a12e8b), WTx(a361125873bffb5d70636e50bac18bd71963821d05ba07d9d70c91e660779632)])
<-- from id=10752006 (conn_type=1): AddrV2([Address(timestamp=1750941674, address=IPv4(XXX), port=8333, services=3077), Address(timestamp=1750941953, address=IPv4(XXX), port=8333, services=3081), Address(timestamp=1750941813, address=IPv6(XXX), port=8333, services=1032)])
<-- from id=10162242 (conn_type=1): Inv([WTx(5a7a949a920cf57eacd8ad38906a56ba6882188dda4ff9ea5660aad35adf1ef4), WTx(1acc1f2f3ec70c4ffd2181783bb2407e204be39b1017b5ae13d45b9b54a19e43), WTx(5a68c9197c31b5629c146be6d789a44bbb03e2c43633216ec0ca8cd73bd737f2), WTx(78ad4525de5b0e03db2b552b0091275caf781d57b04affc468d960ba645c1370)])
# CONN EvictedInboundConnection(conn=Connection(id=11937171, addr=<linkinglion>, conn_type=1, network=2), time_established=1750942328)
# CONN InboundConnection(conn=Connection(id=11937172, addr=<linkinglion>, conn_type=1, network=1), existing_connections=115)
# CONN ClosedConnection(conn=Connection(id=11937171, addr=<linkinglion>, conn_type=1, network=2), time_established=1750942328)
<-- from id=11554724 (conn_type=1): Inv([WTx(12eff03d987ef34ec759abe864bd88c2ecb4c994bd23ac18680fed251440020a), WTx(1e34d5e8d3e34ee7257363a34c877ea6031f0657574c78d6d1379485e1a8b533), WTx(68ae6c3279f1797f08f90948d7599ec60a476f896013f164a49b38eae10c6cf9), WTx(d1d6a9a50d1059db07a8367e52a6569d989f2dcdde24e56369aae2aaab4cf0aa), WTx(8c3c1033296b4c593560f40fd22929cbcc6f63c3ec20287829e9b52dea9a4ea2), WTx(77c7e46e402f94c4a03445479e727b67008e105f2c97d3120e8c2d2008b6c6c3)])
<-- from id=11875990 (conn_type=1): Pong(16368378148765531861)
<-- from id=8950578 (conn_type=1): Inv([WTx(12eff03d987ef34ec759abe864bd88c2ecb4c994bd23ac18680fed251440020a), WTx(1e34d5e8d3e34ee7257363a34c877ea6031f0657574c78d6d1379485e1a8b533), WTx(68ae6c3279f1797f08f90948d7599ec60a476f896013f164a49b38eae10c6cf9), WTx(8c3c1033296b4c593560f40fd22929cbcc6f63c3ec20287829e9b52dea9a4ea2), WTx(77c7e46e402f94c4a03445479e727b67008e105f2c97d3120e8c2d2008b6c6c3)])
<-- from id=11833825 (conn_type=1): Inv([WTx(5a7a949a920cf57eacd8ad38906a56ba6882188dda4ff9ea5660aad35adf1ef4), WTx(1acc1f2f3ec70c4ffd2181783bb2407e204be39b1017b5ae13d45b9b54a19e43), WTx(205bfe2dfbeb46c7d91963a13097ef49511ad2d71c3018fdbdebbff83d8caa2f), WTx(0cd27eb1f63d95c0ec82adf0090756aef0eb1b1e840634ec7a4f440919ab991c), WTx(a361125873bffb5d70636e50bac18bd71963821d05ba07d9d70c91e660779632), WTx(5de0156daa756bdcad84a93699972a6ecb451841f2404ad181bd540c87006756), WTx(f2c8df33b2ef2e15c6d239e05f651927b4108758d99bafb478e76b9f7827e19d)])
--> to id=8444252 (conn_type=2): Inv([WTx(1e34d5e8d3e34ee7257363a34c877ea6031f0657574c78d6d1379485e1a8b533), WTx(68ae6c3279f1797f08f90948d7599ec60a476f896013f164a49b38eae10c6cf9), WTx(8c3c1033296b4c593560f40fd22929cbcc6f63c3ec20287829e9b52dea9a4ea2), WTx(77c7e46e402f94c4a03445479e727b67008e105f2c97d3120e8c2d2008b6c6c3)])
<-- from id=11937172 (conn_type=1): Version(version=70016, services=3081, timestamp=1750942328, receiver=Address(timestamp=0, address=IPv4(XXX), port=8333, services=0), sender=Address(timestamp=0, address=IPv4(XXX), port=8333, services=0), nonce=0, user_agent=/bitcoinj:0.14.5/Bitcoin Wallet:5.40/, start_height=902650, relay=true)
--> to id=11937172 (conn_type=1): Version(version=70016, services=3080, timestamp=1750942328, receiver=Address(timestamp=0, address=IPv4(XXX), port=62311, services=0), sender=Address(timestamp=0, address=IPv4(0.0.0.0), port=0, services=3080), nonce=redacted, user_agent=/Satoshi:28.00.0/, start_height=902811, relay=true)
--> to id=11937172 (conn_type=1): wtxidrelay
--> to id=11937172 (conn_type=1): sendaddrv2
--> to id=11937172 (conn_type=1): verack
<-- from id=11937172 (conn_type=1): verack
--> to id=11937172 (conn_type=1): SendCompact(send_compact=false, version=2)
--> to id=11937172 (conn_type=1): Ping(2927426282439637971)

`metrics` tool

The metrics tool transforms individual events into aggregated statistics and serves them as Prometheus metrics. These metrics can then be displayed in Grafana dashboards. This allows for visual exploration and dashboard playlists that can help to detect attacks and anomalies visually. While there are some Grafana alerts for restarted nodes and inbound connections dropping, more work can be done on automatic anomaly detection. For this, the Prometheus recording rules mentioned in #13 (comment) could be useful to explore.

A Grafana dashboard showing the time it takes to connect blocks per node. Some nodes are faster than others due to hardware and configuration differences. For example, node frank is usually slower as it doesn’t have a mempool and needs to validate all transactions. The other nodes already have validated the transactions. An interactive version of this dashboard can be found on snapshots.raintank.io.

`websocket` tool

The websocket tool publishes the events from NATS into a websocket as JSON objects. This allows us to work with the events in a browser and enables building web tools and visualizations. An example is the p2p-circle.html page, which displays the node connected to peer-observer in the middle and arranges the node’s peers in a circle around it. Exchanged messages and opened and closed connections are shown.

Your browser does not support the video tag.

The video shows the p2p-circle.html page with the peer-observer node in the middle and its peers arranged in a circle around it. The peers are labeled with their peer-id and colored by connection type: blue peers are inbound connections, red ones are full-relay-outbound, and yellow peers are block-only-outbound peers. Additionally, the P2P message exchange between the node and its peers, and new inbound connections being accepted and others being closed, can be seen.

The peer-observer infrastructure

As of writing, I host 12 honeypot nodes with different configurations across the globe as part of the peer-observer infrastructure. Running nodes with different configurations means having a bigger attack surface, and at the same time, being able to detect anomalies for more features. Some nodes run with privacy networks like Tor and I2P enabled. An attacker might perfer to attack via these to avoid leaving an IP address trail. Some nodes have bloom filters (known to be DoS-able, and observed in bitcoinj/bitcoinj #3404) and compact block filters enabled. Others run with a ban list to block, for example, connections from LinkingLion, and some are testing an ASMap file. Some nodes are pruned, others are not. Some accept an increased number of inbound connections, some don’t have a mempool and only process blocks, while others run behind a NAT and can only accept inbound connections from Tor, I2P, and CJDNS. A few nodes run binaries compiled with LLVM sanitizers (particularly ASan and UBSan) enabled. Most nodes run on x86_64, but I also have a few on aarch64. Generally, the nodes can run with different PRs, master versions, release candidates, or releases of Bitcoin Core. All nodes are configured with detailed debug.log logging enabled. As of writing, I have collected more than 35 TB⁴ of debug logs, which can be used in future research projects.

The MIT DCI generously sponsored six nodes between June 2023 and April 2025. I’m grateful for their trust and support in building out this project. Also, thanks to Sam, who has been very pleasant to work with. In April 2025, Brink took over sponsorship of these six nodes. The other six nodes are currently paid by me, as is the archival storage, web, and database servers. A public list of nodes can be found on public.peer.observer. Since the nodes are supposed to be honey pot nodes, I can’t share node IP addresses, hosting details, and data publicly. An attacker would know which nodes to ignore when attacking. I’ve been thinking about setting up a demo node with a public frontend and public IP address for people to explore and experiment with. However, I have had more urgent items in my backlog for now.

To manage and deploy nodes to different cloud providers and hardware with different node configurations and versions, NixOS has proven to be a useful tool. It allows me to write the infrastructure as code, track it in git, and have reproducible deployments across hosts. Without it, I don’t think maintaining the peer-observer infrastructure would have been possible as a one-man job. The Nix package and NixOS module for peer-observer are published in 0xb10c/nix, and I think I can publish a NixOS Flake for the node configuration at some point while maintaining my infrastructure in a separate, private Flake.

Next to peer-observer and Bitcoin Core nodes, the infrastructure also includes a fork-observer instance connected to the nodes (this is publicly accessible) and an installation of addrman-observer, which allows viewing the addrman of the nodes. Next to the metrics tool, each host runs a node_exporter that allows Prometheus to fetch metrics on CPU, RAM, disk, network, and more. Additionally, a process_exporter exports metrics on CPU time spent in each Bitcoin Core thread. Each host also runs a service that uploads the Bitcoin Core logs to a remote data share.

Some peer-observer findings

While building out the tooling and infrastructure, I already made a few observations of attacks and anomalies, and had the chance to use the node data for research into compact block relay efficiency. I’m linking to some write-ups below.

Early on, I discovered an entity I call LinkingLion, which opens multiple connections to Bitcoin nodes and listens to transaction announcements. The entity has been active since at least 2018 and is connecting to nodes on the Monero network, too. I assume the entity is a blockchain analysis company collecting data to improve its products. This is a privacy attack on Bitcoin users. Having access to data from multiple nodes makes it possible to detect attacks like these.

In May 2023, I noticed an anomaly in the number of inbound connections on one of my peer-observer nodes. Its inbound connections dropped, and the node had 100% CPU utilization. Looking into it, it turned out that an edge case in the Bitcoin Core transaction relay implementation had been triggered, and the node could not keep up with normal operation. Since many nodes in the network were affected by this, it had an effect on the whole network to the point where block and transaction relay were impacted. I’ve written down my notes from back then in this post. Having more monitoring and nodes back then would have helped to pinpoint and react to this anomaly faster.

One of my goals has always been to extract insights from data and feed them back into Bitcoin development. With detailed, historical logs from multiple nodes available, I published Stats on compact block reconstructions. This led to a renewed discussion on prefilling compact blocks to improve block relay. I hope to get back to finishing the implementation of this at some point.

There is a lot more data to process and monitoring to build out. I can’t monitor the Bitcoin network, analyze data, and build out tools alone. This could easily be a full-time effort for a small team of developers and data scientists. I’d be very happy to share data and help with processing, analyzing, and publishing findings. I’m also certain that finding funding for this work, given some prior Proof-of-Work, isn’t too hard at the moment.

A Bitcoin Network Operations Collective

As mentioned above, I noticed that I could use a few helping hands that are interested in monitoring the Bitcoin network health and analyzing data to provide data-based feedback for development. As of writing, Bitcoin has a market cap well over $2T USD. That’s more than Meta and Google, and close to Amazon. What monitoring infrastructure, Network Operation Centers, and incident response teams do these companies have to protect against attacks and anomalies? And what does Bitcoin have?

Current state of Bitcoin monitoring compared to companies with a similar market cap.

There are a few developers I know running nodes and looking at logs, the KIT DSN is running their Bitcoin Network Monitoring infrastructure, a Brazilian University is building out a Bitcoin Monitoring Lab, I heard, and Lopp’s statoshi.info is still running. And I have a spare Raspberry Pi on my desk that cycles through a few Grafana dashboards and might get a ping to my phone if on some node the connections drop faster than expected. What happens when I’m asleep, on vacation, or just don’t have the time to look into it?

With initial tooling and infrastructure in place, I’ve been thinking about the next step to improve the situation. If Bitcoin were a company, a Network Operations Center could be formed, and people could be hired for an incident response team. In Bitcoin, this works differently, and some might even reject a Network Operations Center as too centralizing. Similarly, people can’t be hired for a job like this. I think they need to be self-driven, curious about the behavior of network participants, and motivated to ensure the longevity of Bitcoin and its network.

What I’ve been thinking about might be better described as a Network Operations Collective. A loose, decentralized group of people who share the interest of monitoring the Bitcoin Network. A collective to enable sharing of ideas, discussion, data, tools, insights, and more. Maybe with a chat channel, a forum, a shared data store, and access to, for example, monitoring tools like my peer-observer. A place where a Bitcoin network incident could be analyzed, discussed, and ideally resolved, even if some members aren’t online. A collective with good relationships to developers, companies, and the community, to be able to reach out and be reachable if required.

I’m not sure if the time is right for this idea yet, and I’ll likely think about it for a bit longer. If you happen to have any input on this idea, want to support it, or want to get more into monitoring the Bitcoin network in some capacity, please reach out. If you don’t have an open communication channel with me yet, feel free to write an email to bitcoin-noc@(domain of my blog).

To avoid this problem, Bitcoin Core’s IP address manager (addrman) does not store all IP addresses it receives. It has a table with a fixed size and a DoS-resistant insertion and eviction policy. ↩︎
I don’t think the attacker tried to exploit CVE-2024-52919, but it remains unclear who the addr-flooding attacker and what motivation for this attack was. ↩︎
With the recent popularity of Bitcoin Knots, I’m considering adding a Bitcoin Knots node or two into my setup. However, since Knots and Core share most of the codebase, I don’t think there should be a lot of new insights to gain from observing a Knots node. Additionally, the main promise of the Knots patch set is the limited mempool policy, which ideally should make it less susceptible to e.g., mempool-based Denial-of-Service attacks. If you’re nonetheless interested in sponsoring a Knots node for my monitoring infrastructure, please feel free to reach out. ↩︎
Luckily, Bitcoin Core debug.logs compress fairly well, and I’ve been starting to combine and recompress them to save a bit of disk space. ↩︎

Notes on 'DoS due to inv-to-send sets growing too large' from May 2023

Tue, 24 Jun 2025 00:00:00 +0000

In October 2024, the Bitcoin Core project disclosed a Denial-of-Service due to inv-to-send sets growing too large, which I authored, for Bitcoin Core versions before v25.0. I have a few notes and screenshots from my investigation back then that I want to persist here. In early May 2023, my monitoring infrastructure noticed this bug affecting mainnet nodes, which allowed me to pinpoint where the problem came from. Credit for working on a fix goes to Anthony Towns.

Observation

On May 2nd, 2023, I noticed that on one of my monitoring nodes, the inbound connections had dropped from about 190¹ to only 35 over about two days. Normally, a node keeps its filled inbound slots until it restarts or loses network connectivity.

A screenshot of my Grafana dashboard for monitoring the number of in and outbound connections of my Bitcoin Core node. The yellow bars show the number of inbound connections. These drop off at the end of the graph.

Checking in with other contributors, I noticed the node had 100% CPU utilization. This affected the node to a point where it could not to keep communicating with its peers, which resulted in inbound connections timing out and dropping. Using perf top on the node process, I could see that a lot of CPU time was being spent on CTxMemPool::CompareDepthAndScore() in the b-msghand thread. I recorded the following flamegraph, which shows that make_heap(), which calls CompareDepthAndScore(), used over 45% of the CPU time of the process.

A flame graph showing where the CPU time of the Bitcoin Core process is being spent. Open in a new tab to interact with this flame graph.

At the same time, there was an open, unrelated 100% CPU usage issue with debug mode builds of Bitcoin Core. This confused some contributors and users who weren’t running debug mode builds but noticed the high CPU usage on their nodes. While the debug mode issue likely only affected some developers, the other high CPU usage issue affected the entire network. This included, for example, mining pools such as AntPool and others, who reported problems with their mining operations because to their nodes failing to process received blocks in a timely manner.

Effect

Observing ping timings across the network reveals the effect of this Denial-of-Service. Since Bitcoin Core’s message processing is single-threaded, only one message can be created or processed at a time, meaning that all other peers have to wait. Longer wait times impact the response time for a ping. The KIT DSN Bitcoin monitoring has data on ICMP and Bitcoin protocol pings. Comparing these allows us to determine when node software has problems keeping up with message processing. The data shows the ICMP ping to the host remained unaffected, however, the median ping to the Bitcoin node software nearly doubled from about 25ms to more than 50ms between the end of April and early May. The median Bitcoin ping spiked to 200ms on May 8th, while the ICMP ping remained unaffected.

Median Bitcoin protocol ping and ICMP ping. Based on data from DSN KIT (https://www.dsn.kastel.kit.edu/bitcoin/)

The effect can also be seen by looking at the block propagation delay data collected by the KIT DSN Bitcoin monitoring. Around May 8th, 2023, a spike in the block propagation delay is visible. The time it took 50% of the reachable nodes to announce the block to their monitoring nodes increased from less than a second to more than five seconds. Similarly, the 90% measurement spiked from about two seconds to more than 20 seconds.

Block propagation delay: average time until 50% and 90% of the network announced a block. Based on data by DSN KIT (https://www.dsn.kastel.kit.edu/bitcoin/)

Bad block propagation also causes more stale blocks as mining pools mine on their outdated blocks for longer, while a new block they haven’t seen yet already exists in the network. Based on the data from my stale-blocks dataset, ten stale blocks were observed during the week between May 3rd (starting with stale block 788016) and May 10th (and ending with block 789147). That’s a rate of about 8.84 stale blocks per 1000 blocks. In comparison, between blocks 800000 and 900000 (about two years), 73 stale blocks were observed. This is a rate of 0.73 stale blocks per 1000 blocks. This 10-fold increase in the stale-block rate was likely caused by block propagation being significantly affected.

Cause

Why did the function CTxMemPool::CompareDepthAndScore() slow down the node to a point where it had trouble processing P2P messages? In Bitcoin Core, the b-msghand thread processes P2P messages. For example, passing newly received blocks to validation, responding to pings, announcing transactions to other peers, and a lot more.

The function CTxMemPool::CompareDepthAndScore() is used when deciding which transactions to announce to a peer next. In the Bitcoin P2P protocol, transactions are announced via inv (inventory) messages. A Bitcoin Core transaction announcement to a peer usually contains up to 35 wtxid entries. To keep track of which transactions to announce to a peer next, there is a per-peer m_tx_inventory_to_send set. It contains the transactions the node thinks the peer hasn’t seen yet. When constructing an inventory message for a peer, the set is sorted by transaction dependencies and feerate to prioritize high-feerate transactions and to avoid leaking the order the node learned about the transactions. For this, the CTxMemPool::CompareDepthAndScore() comparison function is used.

In early May 2023, a huge amount of transactions related to BRC-20 tokens were broadcast. This meant that the m_tx_inventory_to_send sets grew faster than usual and larger than usual. As a result, sorting the sets took more time. On evening of May 7th (UTC), the mint of the VMPX BRC-20 token started, which resulted in more than 300k transactions being broadcast in 6h next to the other ongoing BRC-20 token mints. This caused the spikes in median ping and block propagation times observed on May 8th.

The effect is amplified by so-called spy nodes which only listen to inv messages and never announce transactions on their own. When a peer announces a transaction to a node, the node can remove it from their m_tx_inventory_to_send set as it’s known by the peer and does not need to be announced anymore. This meant that the sets for spy nodes were even larger and took even more time to sort as they were drained more slowly. Spy nodes, for example, LinkingLion and others, are common and often have multiple connections open to a node in parallel. At times, I count more assumed spy nodes than non-spy node connections to my nodes.

The huge amount of transactions being broadcast, combined with amplification by spy nodes, and non-optimal sorting of the large m_tx_inventory_to_send sets by CTxMemPool::CompareDepthAndScore() caused nodes to spend a lot of time creating new inventory messages for transaction relay. Since message handling is single-threaded, communication with other peers was significantly slowed down. This reached a point where blocks weren’t processed in a timely manner and some connections timed out.

Fix

The fix is twofold. First, all to-be-announced transactions that were already mined or for some other reason not in the mempool anymore, were removed before the m_tx_inventory_to_send set was sorted. Previously, these transactions were removed only after the set was sorted. This avoids spending time on sorting transaction entries that will never be announced anyway and reduces the size of the to-be-sorted set. Secondly, when the m_tx_inventory_to_send sets are large, the number of entries to drain from the set is dynamically increased based on the set size. This means that when many transactions are broadcast, a node will announce more transactions to its peers until the sets are smaller again. The fix was backported in time for the v25.0 release at the end of May 2023.

Reflection

While a set of regular contributors knew that this was going on, this issue was not openly communicated to the public. The 100% CPU usage issue with debug mode being discussed at the same time caused confusion, even among regular Bitcoin Core contributors. At the time, I had the feeling that this could and maybe should be fixed quietly and doesn’t need a lot of publicity for the time being. In hindsight, maybe being more public and transparent with the issue could have worked too. The high number of BRC-20 broadcasts only lasted for about a week (but this wasn’t known beforehand) and restarting the node would have helped for a while. To mitigate the issue for, for example, mining pools that can’t upgrade to a version with the fix immediately (due to running with custom patches), a ban list of spy nodes was prepared, but I don’t know if it was ever used.

While there was no dedicated communication channel for this event, a non-listed IRC channel with P2P contributors was used and interested contributors were invited or informed about the events via direct messages. As far as I’m aware, there was no incident response channel and I don’t know if one would be helpful given the ad hoc and decentralized nature of Bitcoin development. No contributor is responsible for incident response, but everyone can help.

Personally, I’m happy that my monitoring proved to be useful for this. While I didn’t have alerting for dropped connections set up at the time and only noticed it by looking at the dashboard, it was helpful to have it. To pinpoint the issue, having a few nodes to play around with and run, for example, perf top on was helpful. Future monitoring should include ping times and alerting on dropped connections.

I had increased the connection count from the default 125 connection to 200. ↩︎

peer-observer: A tool and infrastructure for monitoring the Bitcoin P2P network for attacks and anomalies

Wed, 18 Jun 2025 00:00:00 +0000

I presented about my peer-observer project during the BTC Prague dev/hack/day 2025.

Slides (Google Docs)

OpenSats Work-Log 5

Wed, 30 Apr 2025 00:00:00 +0000

This is a copy of the 5th work-log I sent to OpenSats for my LTS grant.

Disclaimer: Some information that is not (or not yet) meant to be published may have been redacted.

How did you spend your time?

Publications & Talks

Continued my “Stats on compact block reconstructions”: thread by posting an update and some of the current work: https://delvingbitcoin.org/t/stats-on-compact-block-reconstructions/1052/24
I’ve took a closer look at the invalid jobs by AntPool and friends during forks: https://b10c.me/observations/14-antpool-and-friends-invalid-mining-jobs/
I wrote about mining centralization and how we currently have only 6 blocks producing >95% of the templates. Also introduced a mining centralization index: https://b10c.me/blog/015-bitcoin-mining-centralization/
Did a talk for Chaincode’s BOSS program on Bitcoin monitoring
Held the third FFM Socratic Seminar with fjahr: https://btcffm.org/#2025-04-16

mainnet-observer

https://github.com/0xB10C/mainnet-observer

An open-source refresh of my transactionfee.info (closed source) project showing protocol level bitcoin statistics.

Over the last months, I’ve spent a bit of time to get this project closer to a point where it’s ready to be used. The original transactionfee.info project was started in 2017 and I did a refresh of it in early 2020. I think it’s important to know how the network and blockspace is used to reason about protocol changes, make data-based development decisions, and to generally be aware of how the Bitcoin network is being used. Since the old Go backend was hard to maintain and the handrolled D3.js frontend was brittle, I choose to rewrite the back- and frontend to be more maintainable and easy to work with. This also allowed me to open-source the code. READCTED

While I will host an instance on https://mainnet.observer, the site is complelty self-hostable. As of writing, the project features close to 100 charts and I plan to add more over time.

Bitcoin Core

attended CoreDev meeting: Next to the usual CoreDev program, I held a session on current data and monitoring efforts and two sessions about issues with Bitcoin Core self-hosting their CI
self-hosted CI: I mentioned the current self-hosted CI problems in my last progress report. At CoreDev, I wanted to figure out if it’s worthwhile for me to spend my time on it. After presenting and brainstorming on this at CoreDev, I came to the conclusion that I want to cut back my time spent on CI stuff for now. Other contributors agreed that the current self-hosted CI situation is not optimal and we started looking into alternatives. As of writing these efforts have pretty much died down for now. See this issue for some of the discussion https://github.com/bitcoin/bitcoin/issues/31965
Code wise, I only ended up PRing test, tracing: don't use problematic bpf_usdt_readarg_p() #31848 during the last three months.
As mentioned in https://delvingbitcoin.org/t/stats-on-compact-block-reconstructions/1052/24, I’ve been working on a PR for predicitvly prefilling compact blocks with transactions a node would likely have to request otherwise. I noticed problems in the fuzztest that covers this part of the code and have been speaking with the author of the fuzz test on how to best resolve them. Hope to pick up work on this again soon!
As usual, contributed my GUIX sigs for the v29.0 release (and release canidates): https://github.com/bitcoin-core/guix.sigs/pull/1626
REDACTED
REDACTED

peer-observer

https://github.com/0xB10C/peer-observer

A tool used to monitor for attacks and anomalies by hooking into the Bitcoin Core tracepoints.

Until May 2025, MIT DCI was sponsoring six peer-observer monitoring nodes. REDACTED. Subsequently I moved the nodes to different servers.
Add python tool to record getblocktxn msgs: I’ve used this tool for my research on compact block reconstruction. I posted preliminary results of it in the devling post linked above.
I’ve also been mentoring someone interested in this project and helping him make his first contributions there

Misc

I opened PR bitcoind: 28.1 -> 29.0 #398586 to upgrade the Nix Bitcoin Core package to v29.0. This was more work than usual as the build system changed with the v29 release.
For the bitcoin-data/stale-blocks dataset I maintain, we decided to add the full stale blocks there too as they might be interesting for future analysis. See #7, #8, and #11
In https://github.com/0xB10C/nix/pull/98, I added systemd hardening measures for the NixOS modules of my tools. This should make them somewhat more sandboxed and isolated for everyone wanting to run them.

What do you plan to work on next quarter?

continue the work on my predictivly prefilling compact blocks Bitcoin Core PR
continue working on peer-observer: This includes a presentation and possible an announcement blog post, but also implementing more data extractors like an RPC extractor (https://github.com/0xB10C/peer-observer/issues/141)
getting mainnet-observer ready and an initial version out. And add more metrics and charts as needed.
maintain and continue working on all the other small side projects I have

Next to work:

take some time off: I’ve been feeling a bit over-worked for the past half year and I think it’s healthy to take a few days off to enjoy summer and touch some grass. I’ve been very happy to see people like e.g. @bboerst looking into stratum stuff, which allows me to focus more on my other projects. I really hope other people find fun in doing some network monitoring too - sometimes I’ve been feeling overwhelmed by the all the projects that could and should be done, but aren’t.

Bitcoin Mining Centralization in 2025

Tue, 15 Apr 2025 09:00:00 +0000

This post explores Bitcoin Mining Centralization in 2025 by looking at the hashrate share of the current five biggest mining pools. It presents a Mining Centralization Index and updates it with the assumed proxy pooling by AntPool & friends. It shows that Bitcoin mining is highly centralized today, with only six pools mining more than 95% of the blocks.

In the current Bitcoin mining landscape, nearly the complete mining hashrate is controlled by a few large mining pools, which produce the templates for the blocks. These pools control which transactions they include in or exclude from their blocks. This doesn’t hurt Bitcoin’s censorship resistance as long as these mining pools don’t collude and decide to censor transactions. However, it raises the question of how many distinct block template producers exist. The 51%-attack, where a single miner or pool controls more than half the hashrate and can out-compete all other miners by building on its own chain, is well known. However, even with only 40% of the hashrate a pool has a ~50% chance of out-competing all other pools for six blocks ¹. Are there pools with 40% of the hashrate today? And in general, how centralized is Bitcoin mining today? Especially with the assumed proxy pooling where smaller pools proxy the mining jobs of larger pools while putting their own name into the coinbase transaction.

This post explores Bitcoin mining centralization in two parts. The first part looks at the mining pool information included in the coinbase transaction. The second part takes the assumed proxy pooling into account. Both parts show the current biggest mining pools and a mining centralization index.

Measuring Centralization by counting Coinbase tags

Mining pools usually leave an ASCII tag in their coinbase transactions. For example, the mining pool F2Pool includes the tag /F2Pool/. Additionally, mining pools frequently reuse their coinbase output addresses. A dataset of these identifiers can be used to find out which pool mined a block. For this blog post, I’ve used the bitcoin-data/mining-pools dataset. To measure the network share, I’ve looked at the blocks per day per mining pool divided by the total number of blocks per day. The data shown is averaged with a 31-day moving average.

Currently, the five largest mining pools are Foundry (30%), AntPool (19%), ViaBTC (14.5%), F2Pool (10%), and MARA Pool (5%). Out of these, MARA Pool is the youngest pool, having started only in May 2021. Additionally, they are the only private pool in the top five. At the end of 2020, Foundry started mining its first blocks. After one year of operation, Foundry surpassed the other mining pools and became the largest pool with about 17% of the network hashrate. A bit more than another year later, in January 2023, Foundry reached 30% of the hashrate and has remained at this level since. AntPool, the mining pool owned by the ASIC manufacturer Bitmain, increased its hashrate from about 10% in 2020 to more than 25% in 2024 before dropping below 20% again at the end of 2024. Last year, ViaBTC overtook F2Pool in the race for the third place.

With the largest two pools currently having over 50% of the hashrate, followed by places three and four having another 25% together, 75% of the hashrate is controlled by just four pools. To put this into perspective, the following Mining Centralization Index can help.

To measure the Bitcoin mining centralization over time, the Mining Centralization Index shows the hashrate sum of the largest 2, 3, 4, 5, and 6 pools at each point in time. Higher values mean more centralization.

For example, in May 2017, the top 2 pools together had a hashrate of less than 30% and the top 6 pools had less than 65% together. Then, Bitcoin mining was the most decentralized it has ever been in its pooled mining history. December 2023 is a strong contrast to this when more than 55% of the hashrate was controlled by the top 2 pools and 90% was controlled by the top 6 pools. Compared to the period between 2019 to 2022, where the top 2 pools had around 35% of the network hashrate and the top 6 pools had about 75%, Bitcoin mining is currently a lot more centralized.

Measuring Centralization in times of Proxy Pools

It has been observed that AntPool and multiple smaller mining pools send out very similar block templates. It’s assumed that these smaller pools are proxy pools for AntPool meaning they relay AntPools mining jobs but change the coinbase tags and addresses to their own. This causes hashrate estimates based on coinbase tags and addresses to become inaccurate. AntPool’s hashrate share is underreported as the blocks are attributed to the smaller miners. However, AntPool and these smaller miners, collectively referred to as “AntPool & friends” can be counted together as one big pool consisting of multiple pools. It can only be assumed which pools belong to the “AntPool & friends” group², and it’s not clear when they joined that group.

This chart assumes all pools considered a part of “AntPool & friends” were a part of it starting in 2023. Since this is not proven to be the case, assume a slight over-reporting of the hashrate share throughout 2023. However, the data should be more accurate for 2024, where “AntPool & friends” made up about 40% of the network hashrate. While AntPool & friends had a higher hashrate share than Foundry for the last two years, Foundry seems to have gained 5% while AntPool & friends lost the same share during early 2025. This trend seems to be reversing with AntPool & friends growing again.

Given that AntPool & friends have had a 10% to 15% higher hashrate share than assumed in the previous Mining Centralization Index chart, revisiting the index is worthwhile.

The Mining Centralization Index including AntPool & friends shows that over the last two years, AntPool & friends and Foundry together controlled 60% to 70% of the hashrate. However, even worse, 96% to 99% of the blocks were mined by only six mining pools. These numbers indicate that Bitcoin mining is heavily centralized around a few template-producing pools.

Bitcoin needs smaller pools like MARA Pool with 5% hashrate. Some large US mining companies could probably leave Foundry and start solo mining given their hashrate. Additionally, shifting hashrate to pools like Ocean (<1%) or DEMAND (0%), where miners build their own template, helps make Bitcoin mining more decentralized. More individuals home-mining with small miners help too, however, the home-mining hashrate is currently still negligible compared to the industrial hashrate.

To answer the questions from the introduction:

How many distinct block template producers exist?

We can’t know, but we know that six block template producers mine more than 95% of the blocks.

Are there pools with 40% of the hashrate today?

No, but it’s assumed that AntPool & friends controlled about 40% of the network hashrate in 2023 and throughout the first half of 2024. Foundry had about 35% in early 2025. Currently, AntPool & Foundry each have more than 30% of the hashrate.

How centralized is Bitcoin mining today?

According to the Mining Centralization Index, Bitcoin mining was most decentralized for a short period in May 2017. 2019 to 2022 was also a good period. Starting in 2023, Bitcoin mining has become more and more centralized, especially with large pools like Foundry and proxy pooling a la AntPool & friends.

https://bitcoinops.org/en/tools/reorg-calculator/ ↩︎
As of writing, the following pools are assumed to be part of the AntPool & friends group: AntPool, Poolin (template similarities, invalid jobs, invalid jobs 2), CloverPool (formerly BTC.com, template similarities, invalid jobs 2), Braiins (template similarities, invalid jobs), Ultimus Pool (template similarities, invalid jobs, invalid jobs 2), Binance Pool (template similarities, invalid jobs), SecPool (template similarities), SigmaPool (template similarities), Rawpool (invalid jobs 2), Luxor (template similarities), Mining Squared (mining-squared). SpiderPool is currently not considered an AntPool proxy due to not having enough evidence for it yet. ↩︎

Invalid mining jobs by AntPool & friends during forks

Wed, 12 Mar 2025 00:00:00 +0000

Looking deeper into @boerst’s recent observation about invalid mining jobs by AntPool & friends to discuss his hypothesis about “selfish mining” and “glitchy template code”. I conclude that it’s probably a bug in AntPool’s coinbase creation code and agree with @boerst’s conclusion that this is another good data point for proxy pooling of “AntPool & friends”.

@boerst, the developer behind the stratum.work mining pool job monitoring tool recently observed that AntPool, CloverPool, Ultimus, Rawpool, and Poolin published mining jobs for invalid blocks. The pools publish empty mining jobs only including the coinbase transaction but have a coinbase output value higher than the subsidy. These jobs are invalid as they attempt to create more new coins than allowed. @boerst saw these jobs being published on March 1st, 2025 and noted that the previous block hash ..acdeb985ea¹ in these jobs isn’t known to the network. He speculates that these bad jobs could be caused by a “botched selfish mining attempt” or “glitchy template code”. Additionally, he notes that this makes it obvious that these pools are related.

I added historical data to https://t.co/GCd3WfVtZW over the weekend.

Looking at some really interesting block templates sent out for height 885797 by Antpool, CloverPool, Ultimus, Rawpool, and Poolin.https://t.co/0oNZepy6HA

These are interesting because:

- Empty merkle… pic.twitter.com/QOw7WqZRGW
— boerst (@boerst) March 10, 2025

Observations

Looking at historical data I collected with an instance of my stratum-observer tool, I noticed three similar cases in December 2024 next to the occurrence on March 1st, 2025. While I don’t collect jobs from CloverPool and Rawpool, I saw jobs from Braiins and Binance Pool that have the same characteristics. I also noticed that these only occur when AntPool and friends are involved in a block-race during a fork.

While building block 873559

On December 6th, 2024, AntPool, Braiins, Poolin, Binance Pool, and Ultimus published invalid jobs building block 873559 on the previous block hash ..2ba3bd04af². This previous block mined by AntPool ended up losing in a block-race against a ViaBTC block. While a coinbase output value of 3.125 BTC would have been allowed, Braiins and Ultimus set a coinbase output value of 3.28466550 BTC (0.1596655 BTC more than allowed) while AntPool, Binance Pool, and Poolin had a 5606 sat higher coinbase output value of 3.28460944 BTC (+0.1596094 BTC) in their mining jobs.

I received the first jobs building block 873559 from Poolin, Binance Pool, and F2Pool at nearly the same time. While Poolin and Binance Pool were building on AntPool’s block, F2Pool was building on ViaBTC’s block. The initial jobs by Binance Pool and Poolin had an empty template (no Merkle branches) and a correct coinbase output value of 3.125 BTC. Similar, valid mining jobs arrived for AntPool, Braiins, and Ultimus. However, 400ms after receiving the initial jobs by Binance Pool and Poolin, AntPool, and the other pools sent new, invalid jobs with a too-high coinbase output value. The incorrect coinbase output values excatly match the coinbase output values of the last jobs mining on the previous block indicating the output value was cached from the previous block. A detailed look at the AntPool coinbase transaction reveals that the invalid coinbase and the coinbase from the previous block job only differ in the merge-mining commitments (OP_RETURN outputs and the Namecoin commitment in the coinbase script) and the BIP 30 block height. After 17 seconds AntPool sent a non-empty job with a correct coinbase output value.

The relevant jobs are listed in this table.

time	pool	Merkle branches	value in BTC	building block	prev hash	note
2024-12-06 22:34:09.49	AntPool	13	3.28460944	873558	..8be5b7a176	last job for prev block
…
2024-12-06 22:34:19.90	Poolin	0	3.12500000	873559	..2ba3bd04af	first new jobs for 873559
2024-12-06 22:34:19.90	Binance Pool	0	3.12500000	873559	..2ba3bd04af	mining on AntPool’s block
2024-12-06 22:34:19.90	F2Pool	0	3.12500000	873559	..ea5ba4af6a	mining on ViaBTC’s block
…
2024-12-06 22:34:20.01	AntPool	0	3.12500000	873559	..2ba3bd04af	valid, empty AntPool job
…
2024-12-06 22:34:20.31	AntPool	0	3.28460944⚠️	873559	..2ba3bd04af	invalid AntPool job
…
2024-12-06 22:34:37.43	AntPool	13	3.24736182	873559	..2ba3bd04af	valid, non-empty AntPool job after 17s

While building block 874037

On December 10th, 2024, only AntPool, Binance Pool, and Poolin published an invalid job for block 874037 on AntPool’s block ..e581c9c12f³ during a block-race with F2Pool’s ..07a3cdc451⁴. AntPool’s block ended up winning while F2Pools block became stale. Braiins and Ultimus Pool didn’t publish invalid blocks. The invalid jobs had a coinbase output value of 3.17199711 BTC which again exactly matches the coinbase output value of the last job of the previous block. The coinbase transactions again only differ in the merge-mining commitments and the BIP30 block height. It took AntPool about 16 seconds before they sent a valid job after their invalid job.

While building block 875590

On December 20th, 2024, AntPool, Braiins, Poolin, Binance Pool, and Ultimus published a job building block 875590 on the AntPool’s block ..b905ac264e⁵ during a block-race with ViaBTC’s ..b911cbd12c⁶ which ended up winning. AntPool, Binance Pool, and Poolin had a coinbase transaction with a value of 3.25665133 BTC (0.13165133 BTC too much), and the coinbase transaction of Braiins and Ultimus had a value of 3.25664121 BTC (0.13164121 BTC too much). The coinbase output values again exactly match the values of the last job of the previous block. After 22 seconds, AntPool sent out a valid mining job.

While building block 885797

On March 1st, 2025, AntPool, Braiins, Poolin, Binance Pool, and Ultimus published invalid jobs building block 885797 on the unknown block ..acdeb985ea¹ which was in a block-race with a block mined by Foundry. The jobs by Braiins and Ultimus had an invalid coinbase output value of 3.16436527 BTC while AntPool, Poolin, and Ultimus had a value of 3.14557552 BTC. This time, the coinbase output value of the last AntPool job for the previous block was 3.16295187 BTC which doesn’t match with either of the invalid jobs. For this block, AntPool didn’t send out a valid job and even refreshed the invalid job multiple times until Foundry found block 885797 about 35 seconds after 885796 had been found. This means AntPool and the other pools tried to mine an invalid block for about 35 seconds.

The unknown block ..acdeb985ea¹ may be a block by AntPool, Braiins, Poolin, Binance Pool, or Ultimus that didn’t propagate well during a block-race with the Foundry block.

Discussion

Based on the observations, the behavior of AntPool & friends can be summarized as follows: On a new block, AntPool and friends send a job for an empty block. If they detect that one of AntPool’s blocks is in a block-race with another block, they send an empty mining job with a coinbase output value of or similar to their last job for the previous block. At some point, they will issue a new, valid job with a correct coinbase output value. Based on the four observations, this usually takes more than 15 seconds.

This behavior doesn’t seem like this is a selfish-mining attempt. The three blocks in December propagated through the network and in 874037 even arrived before F2Pool’s block. The block in March might not have propagated well due to being found slightly after the Foundry block 885796.

@boerst speculates that the invalid jobs could be related to “glitchy template code”. Based on the observations, I assume it’s related to the coinbase building code as opposed to the template building code. In three cases, the coinbase output value of the last job of the previous block is reused. This leads to the assumption that it’s cached somewhere and in some way not updated or reset in the block-race scenario. On March 1st, this assumption doesn’t hold as the AntPool coinbase value of the last job of the previous block doesn’t match the value in the invalid job. However, an explanation could be that the coinbase value for the previous block had been updated internally, but the corresponding mining job hadn’t been published yet.

While I don’t track the the same pools as boerst, I agree with his assumption that this weird and incorrect behavior once more confirms that these pools are operated by the same entity. It might be time to rename AntPool and its proxy pools Braiins, Poolin, Binance Pool, and Ultimus (and probably more) to “AntPool & friends”.

A list of all invalid jobs I received can be found below:

stratumobserver=> select pool, timestamp, coinbase_value, coinbase_height, cardinality(merkle_branches) as merkle_branches_count, header_prev_hash from job_updates where coinbase_value > 312500000 and cardinality(merkle_branches) = 0 order by timestamp desc;
pool | timestamp | coinbase_value | coinbase_height | merkle_branches_count | header_prev_hash
--------------+----------------------------+----------------+-----------------+-----------------------+------------------------------------------------------------------
Braiins | 2025-03-01 02:32:48.256093 | 316436527 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
AntPool | 2025-03-01 02:32:47.95568 | 314557552 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
Ultimus | 2025-03-01 02:32:47.934396 | 316436527 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
Binance Pool | 2025-03-01 02:32:47.834271 | 314557552 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
Poolin | 2025-03-01 02:32:47.76899 | 314557552 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
Braiins | 2025-03-01 02:32:38.442376 | 316436527 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
AntPool | 2025-03-01 02:32:38.342103 | 314557552 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
Ultimus | 2025-03-01 02:32:38.220214 | 316436527 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
Binance Pool | 2025-03-01 02:32:38.120078 | 314557552 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
Poolin | 2025-03-01 02:32:38.054957 | 314557552 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
Braiins | 2025-03-01 02:32:29.622593 | 316436527 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
AntPool | 2025-03-01 02:32:29.322023 | 314557552 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
Ultimus | 2025-03-01 02:32:29.19731 | 316436527 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
Binance Pool | 2025-03-01 02:32:29.097238 | 314557552 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
Poolin | 2025-03-01 02:32:29.036254 | 314557552 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
AntPool | 2025-03-01 02:32:28.020701 | 314557552 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
Binance Pool | 2025-03-01 02:32:27.69472 | 314557552 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
Poolin | 2025-03-01 02:32:27.634525 | 314557552 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
Braiins | 2025-03-01 02:32:26.719377 | 316436527 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
Braiins | 2025-03-01 02:32:26.218713 | 316436527 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
Ultimus | 2025-03-01 02:32:26.19205 | 316436527 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
AntPool | 2025-03-01 02:32:25.918188 | 314557552 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
Ultimus | 2025-03-01 02:32:25.891502 | 316436527 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
Binance Pool | 2025-03-01 02:32:25.590893 | 314557552 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
Poolin | 2025-03-01 02:32:25.532066 | 314557552 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
Poolin | 2025-03-01 02:32:25.431804 | 314557552 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
Binance Pool | 2025-03-01 02:32:25.390627 | 314557552 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
Poolin | 2025-03-01 02:32:25.331699 | 314557552 | 885797 | 0 | 00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea
AntPool | 2024-12-20 12:45:27.914581 | 325665133 | 875590 | 0 | 0000000000000000000005eee6163885d621cd0b4e999630d00433b905ac264e
AntPool | 2024-12-20 12:45:09.963704 | 325665133 | 875590 | 0 | 0000000000000000000005eee6163885d621cd0b4e999630d00433b905ac264e
Binance Pool | 2024-12-20 12:45:09.865582 | 325665133 | 875590 | 0 | 0000000000000000000005eee6163885d621cd0b4e999630d00433b905ac264e
Poolin | 2024-12-20 12:45:09.865545 | 325665133 | 875590 | 0 | 0000000000000000000005eee6163885d621cd0b4e999630d00433b905ac264e
Ultimus | 2024-12-20 12:45:09.86551 | 325664121 | 875590 | 0 | 0000000000000000000005eee6163885d621cd0b4e999630d00433b905ac264e
Braiins | 2024-12-20 12:45:09.817793 | 325664121 | 875590 | 0 | 0000000000000000000005eee6163885d621cd0b4e999630d00433b905ac264e
Binance Pool | 2024-12-20 12:45:09.36458 | 325665133 | 875590 | 0 | 0000000000000000000005eee6163885d621cd0b4e999630d00433b905ac264e
AntPool | 2024-12-20 12:45:09.362357 | 325665133 | 875590 | 0 | 0000000000000000000005eee6163885d621cd0b4e999630d00433b905ac264e
Braiins | 2024-12-20 12:45:09.317215 | 325664121 | 875590 | 0 | 0000000000000000000005eee6163885d621cd0b4e999630d00433b905ac264e
Poolin | 2024-12-20 12:45:09.264466 | 325665133 | 875590 | 0 | 0000000000000000000005eee6163885d621cd0b4e999630d00433b905ac264e
Ultimus | 2024-12-20 12:45:09.16438 | 325664121 | 875590 | 0 | 0000000000000000000005eee6163885d621cd0b4e999630d00433b905ac264e
AntPool | 2024-12-10 01:33:10.167938 | 317199711 | 874037 | 0 | 0000000000000000000026da3d0aabc84cc20868a847f629b722a1e581c9c12f
Binance Pool | 2024-12-10 01:33:10.008015 | 317199711 | 874037 | 0 | 0000000000000000000026da3d0aabc84cc20868a847f629b722a1e581c9c12f
Poolin | 2024-12-10 01:33:09.915384 | 317199711 | 874037 | 0 | 0000000000000000000026da3d0aabc84cc20868a847f629b722a1e581c9c12f
AntPool | 2024-12-10 01:32:57.246391 | 317199711 | 874037 | 0 | 0000000000000000000026da3d0aabc84cc20868a847f629b722a1e581c9c12f
Binance Pool | 2024-12-10 01:32:57.088351 | 317199711 | 874037 | 0 | 0000000000000000000026da3d0aabc84cc20868a847f629b722a1e581c9c12f
Poolin | 2024-12-10 01:32:56.998516 | 317199711 | 874037 | 0 | 0000000000000000000026da3d0aabc84cc20868a847f629b722a1e581c9c12f
AntPool | 2024-12-10 01:32:45.129636 | 317199711 | 874037 | 0 | 0000000000000000000026da3d0aabc84cc20868a847f629b722a1e581c9c12f
Binance Pool | 2024-12-10 01:32:44.967986 | 317199711 | 874037 | 0 | 0000000000000000000026da3d0aabc84cc20868a847f629b722a1e581c9c12f
Poolin | 2024-12-10 01:32:44.880904 | 317199711 | 874037 | 0 | 0000000000000000000026da3d0aabc84cc20868a847f629b722a1e581c9c12f
Binance Pool | 2024-12-10 01:32:44.667515 | 317199711 | 874037 | 0 | 0000000000000000000026da3d0aabc84cc20868a847f629b722a1e581c9c12f
Poolin | 2024-12-10 01:32:44.580475 | 317199711 | 874037 | 0 | 0000000000000000000026da3d0aabc84cc20868a847f629b722a1e581c9c12f
AntPool | 2024-12-10 01:32:37.120691 | 317199711 | 874037 | 0 | 0000000000000000000026da3d0aabc84cc20868a847f629b722a1e581c9c12f
Poolin | 2024-12-10 01:32:36.969376 | 317199711 | 874037 | 0 | 0000000000000000000026da3d0aabc84cc20868a847f629b722a1e581c9c12f
Binance Pool | 2024-12-10 01:32:36.955785 | 317199711 | 874037 | 0 | 0000000000000000000026da3d0aabc84cc20868a847f629b722a1e581c9c12f
AntPool | 2024-12-10 01:32:31.613235 | 317199711 | 874037 | 0 | 0000000000000000000026da3d0aabc84cc20868a847f629b722a1e581c9c12f
Binance Pool | 2024-12-10 01:32:31.447633 | 317199711 | 874037 | 0 | 0000000000000000000026da3d0aabc84cc20868a847f629b722a1e581c9c12f
Poolin | 2024-12-10 01:32:31.360356 | 317199711 | 874037 | 0 | 0000000000000000000026da3d0aabc84cc20868a847f629b722a1e581c9c12f
AntPool | 2024-12-10 01:32:31.11264 | 317199711 | 874037 | 0 | 0000000000000000000026da3d0aabc84cc20868a847f629b722a1e581c9c12f
Binance Pool | 2024-12-10 01:32:30.946879 | 317199711 | 874037 | 0 | 0000000000000000000026da3d0aabc84cc20868a847f629b722a1e581c9c12f
Poolin | 2024-12-10 01:32:30.859647 | 317199711 | 874037 | 0 | 0000000000000000000026da3d0aabc84cc20868a847f629b722a1e581c9c12f
Braiins | 2024-12-06 22:34:21.411346 | 328466550 | 873559 | 0 | 00000000000000000000ba2ab3a02b9de883cd7368bf57da3e04cd2ba3bd04af
Ultimus | 2024-12-06 22:34:20.910805 | 328466550 | 873559 | 0 | 00000000000000000000ba2ab3a02b9de883cd7368bf57da3e04cd2ba3bd04af
AntPool | 2024-12-06 22:34:20.810713 | 328460944 | 873559 | 0 | 00000000000000000000ba2ab3a02b9de883cd7368bf57da3e04cd2ba3bd04af
Poolin | 2024-12-06 22:34:20.710726 | 328460944 | 873559 | 0 | 00000000000000000000ba2ab3a02b9de883cd7368bf57da3e04cd2ba3bd04af
Braiins | 2024-12-06 22:34:20.710714 | 328466550 | 873559 | 0 | 00000000000000000000ba2ab3a02b9de883cd7368bf57da3e04cd2ba3bd04af
Binance Pool | 2024-12-06 22:34:20.710682 | 328460944 | 873559 | 0 | 00000000000000000000ba2ab3a02b9de883cd7368bf57da3e04cd2ba3bd04af
AntPool | 2024-12-06 22:34:20.710616 | 328460944 | 873559 | 0 | 00000000000000000000ba2ab3a02b9de883cd7368bf57da3e04cd2ba3bd04af
Binance Pool | 2024-12-06 22:34:20.610491 | 328460944 | 873559 | 0 | 00000000000000000000ba2ab3a02b9de883cd7368bf57da3e04cd2ba3bd04af
Poolin | 2024-12-06 22:34:20.510439 | 328460944 | 873559 | 0 | 00000000000000000000ba2ab3a02b9de883cd7368bf57da3e04cd2ba3bd04af
Ultimus | 2024-12-06 22:34:20.410317 | 328466550 | 873559 | 0 | 00000000000000000000ba2ab3a02b9de883cd7368bf57da3e04cd2ba3bd04af
AntPool | 2024-12-06 22:34:20.310274 | 328460944 | 873559 | 0 | 00000000000000000000ba2ab3a02b9de883cd7368bf57da3e04cd2ba3bd04af
Binance Pool | 2024-12-06 22:34:20.210146 | 328460944 | 873559 | 0 | 00000000000000000000ba2ab3a02b9de883cd7368bf57da3e04cd2ba3bd04af
Poolin | 2024-12-06 22:34:20.110058 | 328460944 | 873559 | 0 | 00000000000000000000ba2ab3a02b9de883cd7368bf57da3e04cd2ba3bd04af
Binance Pool | 2024-12-06 22:34:20.009977 | 328460944 | 873559 | 0 | 00000000000000000000ba2ab3a02b9de883cd7368bf57da3e04cd2ba3bd04af
Poolin | 2024-12-06 22:34:20.009938 | 328460944 | 873559 | 0 | 00000000000000000000ba2ab3a02b9de883cd7368bf57da3e04cd2ba3bd04af
(74 rows)

00000000000000000001509b28e96d4eba8508a8885e78dc83b60cacdeb985ea ↩︎ ↩︎ ↩︎
00000000000000000000ba2ab3a02b9de883cd7368bf57da3e04cd2ba3bd04af ↩︎
0000000000000000000026da3d0aabc84cc20868a847f629b722a1e581c9c12f ↩︎
00000000000000000002829ecf6967a9729412446dd571de06e02007a3cdc451 ↩︎
0000000000000000000005eee6163885d621cd0b4e999630d00433b905ac264e ↩︎
00000000000000000000d9940eb168eb3de849ee1d55da2b15f79db911cbd12c ↩︎

OpenSats Work-Log 4

Fri, 31 Jan 2025 00:00:00 +0000

This is a copy of the 4th work-log I sent to OpenSats for my LTS grant.

Disclaimer: Some information that is not (or not yet) meant to be published may have been redacted.

What did you work on?

Publications

My mininpool-observer project detected 15 OFAC sanctioned transactions missing from blocks. After analyzing them to rule out false-positives, I concluded that F2Pool is filtering transactions with OFAC sanctioned addresses again: https://b10c.me/observations/13-missing-sanctioned-transactions-2024-12/
I was asked by a fellow developer if I happen to have any insights into Bitcoin Core orphanage overflows. To make them available to everyone, I posted them to delving: Stats on orphanage overflows
I updated my stats on compact block reconstructions on delving with the newest data. This sparked new discussion and interest from other developers: https://delvingbitcoin.org/t/stats-on-compact-block-reconstructions/1052/5

Bitcoin Core self-hosted CI

Inspired by discussions at the last Bitcoin CoreDev meeting in fall 2024, I started looking into a NixOS based configuration for the Bitcoin Core self-hosted CI runners. At the time, I discovered that the self-hosted runners were running under a privileged user, which could easily stop/vandalize/otherwise negatively affect the underlying machine. CI tasks weren’t isolated from the machine and from other tasks. Note that a CI is basically one of your servers where you give someone from the internet Remote-Code-Execution access. Additionally, the CI token could easily be leaked, which allowed anyone to spawn new, and potentially malicious CI runners for the Bitcoin Core project.

This made is it worthwhile to spend a bit of time looking into a potential CI runner setup that is more secure, properly isolates CI tasks, and doesn’t leak the CI token. By choosing NixOS, the CI runners can be configured once as infrastructure-as-code, and then easily deterministically replicated across multiple hosts.

To isolate individual CI jobs from each other, I choose to run a ephemeral QEMU VM for each job. For this, the cirrus-ci runner used by Bitcoin Core needs to stop after it completed a single job (ephemeral mode). I opened a PR for this, but sadly, there hasn’t been much review activity by the Cirrus-CI folks: https://github.com/cirruslabs/cirrus-cli/pull/813.

The hardest part is to do caching of CI build inputs in ephemeral runners. The current Bitcoin Core CI runners aren’t ephemeral because caching dependency sources and built dependency artifacts, docker base images and task specific docker, previous releases, and ccache artifacts is important for both CI performance and resilience against e.g. network problems or rate-limiting. Managing these across CI jobs can be challenging, especially when a single CI job shouldn’t be able to clear the cache. I’ve written down some notes about this in https://github.com/bitcoin/bitcoin/issues/30852#issuecomment-2558198430.

The project is about 85% done, but still requires some work to get to a production ready-level. Based on discussions with other Bitcoin Core developers, the current CI situation might have been improved since last fall: CI jobs don’t run as privileged user anymore and other safeguards have been put into place. The plan is to present my CI setup at the upcoming CoreDev and evaluate if it makes sense to invest more time and energy into this project in the short-term.

I’ve published the CI-runner setup in https://github.com/0xB10C/bitcoin-core-cirrus-runner and my infrastructure in https://github.com/0xB10C/bitcoin-core-cirrus-runner-infra.

During development, I noticed that it is time consuming to manually parse the Bitcoin Core CI logs. To help during my development and to provide stats for the Bitcoin Core project, I’ve build a tool and website that parses and gives an overview over the recent tasks. The repository can be found at https://github.com/0xB10C/bitcoin-core-ci-stats and the website here https://0xb10c.github.io/bitcoin-core-ci-stats/.

On the Bitcoin Core side, I initially proposed https://github.com/bitcoin/bitcoin/pull/31377, which was then superseded by https://github.com/bitcoin/bitcoin/pull/31545. This makes the caching of the Bitcoin Core CI docker images possible.

Bitcoin Core

Opened a “Tracepoint Interface Tracking Issue” to track open tasks that could be done to improve the tracepoint interface https://github.com/bitcoin/bitcoin/issues/31274
Got two of my long-open Bitcoin Core tracepoint PRs merged: https://github.com/bitcoin/bitcoin/pull/26593 and https://github.com/bitcoin/bitcoin/pull/25832
Did some testing on “ci: detect outbound internet traffic generated while running tests #31349” and found that on some systems the tests still make DNS requests, which can be used to identify developers running the tests https://github.com/bitcoin/bitcoin/pull/31349#issuecomment-2499335672
Helped Suhas with tracing related changes in https://github.com/bitcoin/bitcoin/pull/31122#discussion_r1820876152
Quick fixup https://github.com/bitcoin/bitcoin/pull/31419

peer-observer

A tool used to monitor for attacks and anomalies by hooking into the Bitcoin Core tracepoints.

With Bitcoin Core # tracing: Only prepare tracepoint arguments when actually tracing #26593 merged, I opened Support PID / PID file to attach tracepoints to bitcoind #58 and addressed it in add: cli arg to specify a bitcoind PID #61 and add: use bitcoind PID in peer-observer #75. This made sure peer-observer stayed compatible with the changes to Bitcoin Core.
Added a feature to warn and exit if the peer-observer extractor didn’t receive any events for a while. Implemented in https://github.com/0xB10C/peer-observer/pull/63
Refactored peer-observer to use a NATS message server instead of an NNG (nanomessage) PUB-SUB. This allows to have multiple publishers instead of a single one. I’ve been planning to work on a RPC publisher in combination with the current tracepoint publisher. https://github.com/0xB10C/peer-observer/pull/76
A python tool to record getblocktxn msgs used for https://delvingbitcoin.org/t/stats-on-compact-block-reconstructions/1052/21?u=0xb10c: https://github.com/0xB10C/peer-observer/pull/106
websocket tool maintenance: https://github.com/0xB10C/peer-observer/pull/100
nix-shell maintenance: https://github.com/0xB10C/peer-observer/pull/96

fork-observer

Tool to visualize forks and reorgs on various Bitcoin networks.

testnet4 reorg mining improvements for sjors: show header difficulty and highlight difficulty 1 blocks #50
esplora backend (blockstream.info, mempool.space) support https://github.com/0xB10C/fork-observer/issues/52 and https://github.com/0xB10C/fork-observer/pull/55
added a legend explaining the tip status colors: https://github.com/0xB10C/fork-observer/issues/53 and https://github.com/0xB10C/fork-observer/pull/54
to display changes quicker to reduce desync, the backend now notifies the frontend about all changes: https://github.com/0xB10C/fork-observer/pull/57
btcd would send chain tips in random order, which confused fork-observer into thinking it the btcd tips changed. Fixed by sorting the tips before comparing them: https://github.com/0xB10C/fork-observer/pull/58

miningpool-observer

Transparency for Mining Pool Transaction Selection

general maintenance update rawtx-rs and bitcoin-pool-identification dependecies #86
add missing-sanctioned page https://github.com/0xB10C/miningpool-observer/pull/89
Use unique but non-sequential block id in database for easier importing of backups: https://github.com/0xB10C/miningpool-observer/issues/91 and https://github.com/0xB10C/miningpool-observer/pull/92
remove dead block explorer links https://github.com/0xB10C/miningpool-observer/issues/88

my nix packages

Collection Nix packages and NixOS modules of software I’ve written or software I use. Allows others to easily run my tools too.

integration test for asmap.dat files/package https://github.com/0xB10C/nix/pull/71
added a features that checks for new versions of packages in a nightly CI run and automatically opens a PR for the new version with me as a reviewer in https://github.com/0xB10C/nix/pull/63
other maintenance and package updating: https://github.com/0xB10C/nix/commits?author=0xB10C&since=2024-11-01&until=2025-01-31

Misc

maintained my ofac-sanctioned-digital-currency-addresses repo with, for example, https://github.com/0xB10C/ofac-sanctioned-digital-currency-addresses/pull/18
further maintenance of btcffm.org (https://github.com/btcffm/website-btcffm/pull/16, https://github.com/btcffm/website-btcffm/pull/17, https://github.com/btcffm/website-btcffm/pull/15)
contributed my GUIX signatures for two Bitcoin Core release candidates: https://github.com/bitcoin-core/guix.sigs/pull/1484, https://github.com/bitcoin-core/guix.sigs/pull/1455

What do you plan to work on next quarter?

Evaluate (e.g. at CoreDev) if makes sense to continue working on the self-hosted Bitcoin Core CI runners or if other projects might have a higher priority for me for now.
Continue working on open issues for my current projects like fork-observer, miningpool-observer, peer-observer and others
Continue working on the Bitcoin Core tracing interface. See e.g. https://github.com/bitcoin/bitcoin/issues/31274
Further analyzing data and research posts on delving (e.g. compact block reconstruction and more)

Fifteen OFAC-sanctioned transactions missing from blocks

Thu, 16 Jan 2025 00:00:00 +0000

My miningpool-observer project aims to detect when Bitcoin mining pools are not mining transactions they could have been mining. Over the past few weeks, it detected fifteen missing transactions spending from OFAC-sanctioned addresses. This post examines whether these transactions were filtered intentionally or if there are other possible explanations for these transactions to be missing from blocks. I conclude that F2Pool might have started to filter OFAC-sanctioned transactions again. However, as F2Pool is currently the only pool that’s possibly filtering, it does not affect Bitcoin’s censorship resistance: The sanctioned transactions confirmed in the following blocks.

An exchange used the recent low-fee environment to consolidate coins from deposit addresses to their hot wallet. These consolidations included more than a hundred transactions each spending at least one UTXO belonging to an OFAC-sanctioned Bitcoin address. These transactions were good candiates for re-evaluating if mining pools filter OFAC-sanctioned transactions.

OFAC-sanctioned transactions since 2023. The OFAC-sanctioned transactions at the end of 2024 were good candiates for re-evaluating if mining pools filter them.

The miningpool-observer tool provides insights into which transactions mining pools include and which transactions they exclude. This works by regularly building block templates with my node and comparing them to newly mined blocks. The methodology is explained in detail here. While all of the OFAC-sanctioned transactions were mined eventually, fifteen were not mined in the first block they could have been mined in. For these, a closer analysis can help to determine if a mining pool is purposefully excluding these or if they didn’t make it into the block for other reasons: For example, a well-connected mining pool may know high-feerate transactions my node does not yet know about. These higher-feerate can displace lower-feerate transactions my node considered for my block template. On the other hand, it can happen that my node already knows about a transaction, but the pool hasn’t yet sent out a mining job to its miners that includes the transaction. These “young” transactions can’t be used as evidence for transaction filtering.

Blocks with missing sanctioned transactions

In the following, I list all reports my miningpool-observer instance issued via its RSS feed. The open-source tool can be self-hosted to check mining pool blocks against your own block templates.

F2Pool Block 875575 (filtered?)

My miningpool-obsever instance reports the transaction 16f0ada6.. as missing from F2Pool’s block 875575. The transaction spends a TXO belonging to an address sanctioned by OFAC as part of sanctions against Chinese illicit drug producers and traders. The transaction had been in my node’s mempool for more than 3 hours at the time the block was found, which is more than enough time to propagate to other nodes and mining pools. The transaction has 300 2-of-3 multisig inputs and a single output resulting in a size of 86.4 kvByte and 345.88 kWU. It pays a feerate of 3.97 sat/vByte, which is only minimally higher than the 3.92 sat/vByte feerate of the last transaction package in the block. In mempool.space’s block audit, the transaction is marked as excluded: “marginal fee”. This indicates that “the transaction may have been displaced by an added transaction, or it may have been displaced by another transaction from the mempool that was also at the low end of the expected feerate range for the block”.

Sanctioned transaction position if it would have been included in the block based on its feerate and weight.

To determine how close the sanctioned transaction was to not being included, we can place it into its potential position in the block based on its feerate. The package feerate distribution plot above shows that the sanctioned transaction with a weight of 345.88 kWU would have started at roughly the 3.25 MWU mark based on the feerate it pays. The end of the transaction is more than 400 kWU away from the maximum block weight limit of 4 MWU. While the feerate difference between the sanctioned transaction to the last package feerate might be marginal, the transaction should have been included in the block. This indicates that F2Pool could have filtered the transaction.

F2Pool Block 875840 (too young?)

For F2Pool’s block 875840, miningpool-observer reports that the transaction 26a3693d.. is missing. The transaction spends from the same sanctioned address as in block 875575 and two other sanctioned addresses related to the same Chinese illicit drug producers and traders mentioned earlier.

The transaction is marked as “young” by miningpool-observer as it was only seen 58 seconds before the block was first seen. My node first saw the transaction at 06:17:10 UTC, while the block header timestamp of F2Pool’s block is 06:17:21 UTC (+11s). The miningpool-observer tool processed the block at 06:18:08 UTC (another +47s later). It’s unclear how accurate the header timestamp is as F2Pool’s and my clock might not be in sync and miners might engage in timestamp rolling. We can’t know if the transaction was filtered by F2Pool, or if it is a false-positive report. It could be that the transaction didn’t make it into F2Pool’s mining job for timing reasons related to transaction propagation or mining job publication. This missing transaction can’t be seen as evidence that F2Pool filters OFAC-sanctioned transactions.

F2Pool Block 875933 (filtered?)

The transaction 5c186378.. is reported to be missing from F2Pool’s block 875933. One of the outputs being spent belongs to a sanctioned address related to the same Chinese illicit drug producers and traders mentioned before.

The transaction had been in my node’s mempool for more than 5 minutes before the block was found. This is generally enough time for the transaction to propagate to the pool and to make it into a block template and mining job. With 300 inputs and one output, the transaction has a size of 85'713 vByte. It pays a fee of 339'876 sat, which results in a feerate of 3.965'279 sat/vByte. For this block, the miningpool-observer tool also reports that F2Pool included the “extra” transaction a64c4d64.. (among others) which wasn’t present in my node’s block template. This transaction is a similar 300-input and one-output consolidation with a size of 85'917 vByte (204 vByte larger than the missing transaction). It pays a fee of 340'632 sat (756 sat more than the missing transaction), which results in a feerate of 3.964'664 sat/vByte which is minimally lower than the feerate of the sanctioned transaction. Both transactions didn’t have any unconfirmed parents or children at the time the block was found. A fee-maximizing miner should have included the sanctioned transaction 5c186378.. in favor of a64c4d64...

Sanctioned transaction position if it would have been included in the block based on its feerate and weight.

While the mempool-space block audit marks the transaction with “marginal fee”, the package feerate distribution chart shows there would have been more than 500 kWU of space in the block after the transaction. This indicates that F2Pool could have filtered the transaction.

F2Pool Block 876028 (filtered?)

For F2Pool’s block 876028, two sanctioned transactions are reported missing. Both 4b073053.. and 3d9f2824.. spent from a sanctioned address related to the Chinese illicit drug producers and traders mentioned before.

Transaction 4b073053.. had been in my node’s mempool for more than 10 minutes and transaction 3d9f2824.. for more than 6 minutes. F2Pool ended up including the 300-input and one-output consolation transaction 906da5b5.. which my node didn’t consider for the block as it has a minimally lower feerate than the two sanctioned transactions:

The sanctioned transaction 4b073053.. has a size of 84'731 vByte while paying 336'096 sat in fees resulting in a feerate of 3.966'624 sat/vByte. With 86'079 vByte and a fee of 341'388 sat, the sanctioned transaction 3d9f2824.. pays a feerate of 3.965'985 sat/vByte. The extra transaction 906da5b5.. pays 336'096 sat at a size of 84'749 vByte, which results in a minimally lower feerate of 3.965'781 sat/vByte. A fee-maximizing miner should have included the extra transaction 906da5b5.. only after the two sanctioned transactions.

Position of the two sanctioned transactions if they would have been included in the block based on their feerate and weight. The additional weight added by the first transaction is accounted for in the position of the second transaction.

While mempool.space tags both transactions as “marginal fee”, both sanctioned transactions could have made it into the block based on their feerate and weight. This indicates that F2Pool could have filtered the two transactions.

F2Pool Block 876255 (filtered?)

The transaction 2d56f2f2.. is reported to be missing from F2Pool’s block 876255. It spends from a sanctioned address related to the Chinese illicit drug producers and traders. The transaction was in my node’s mempool for over an hour. The mempool.space block-audit marks this transaction as “removed” from the expected block too. This indicates that F2Pool could have filtered the transaction.

A screenshot of mempool-space block audit of this block shows the sanctioned transaction as removed.

SBI Crypto Block 876590 (displaced?)

For SBI Crypto’s block 876590, the transaction a8a2e533.. is reported to be missing. The transaction spends from a sanctioned address related to the Chinese illicit drug producers and traders.

The transaction had been in my mempool for 44 minutes at the time the block was found. With a fee of 252'639 sat and a size of 84'947 vByte it pays a feerate of 2.974'078 sat/vByte. SBI Crypto was aware of 16 transactions (see “Extra Transactions”) that paid a higher higher feerate (between 3 and 15 sat/vByte) totaling up to 4074 vByte. These displaced the sanctioned transaction and 12 other transactions (see “Missing Transactions”) from the block. This allowed the pool to pick the slightly smaller 300-input and one-output non-sanctioned consolidation transaction e6b44288.. paying a fee of 246'969 sat with a size of 83'042 vByte. This results in a feerate of 2.974'025 sat/vByte, which is minimally lower than the feerate of the sanctioned transaction but higher than the feerates of the 12 missing transactions.

As the sanctioned transaction would have been included rather close to the end of the block with the 16 extra transactions, it’s worthwhile to take a closer look if the sanctioned transaction a8a2e533.. would fit into the block after the 16 extra transactions.

In block 876590, the first transaction (e6b44288..) with a feerate lower than the sanctioned transaction starts after the first 3'659'828 WU of the block and weighs 332'167 WU. The sanctioned transaction a8a2e533.. weighs 339'785 WU (7618 WU more). The transaction e6b44288.. ends at 3'659'828 WU + 332'167 WU = 3'991'995 WU (8005 WU left in block) and the sanctioned transaction would end at 3'659'828 WU + 339'785 WU = 3'999'613 WU (387 WU left in block). Both don’t overflow the maximum block weight of 4 MWU, however, there is another threshold to consider.

The tail end of the package feerate distribution plot of block 876590. Shows that the sanctioned transaction would have been too large for the block as it would have overflown into the reserved weight.

When building block templates, Bitcoin Core reserves a bit of block weight for the block header and the coinbase transaction, which aren’t part of the template. Due to the bug described in issue #21950: Duplicate coinbase transaction space reservation in CreateNewBlock, Bitcoin Core currently reserves 2x 4000 WU by default. Subtracting the actual coinbase weight of 716 WU and the header weight of 332 WU from the reserved 8000 WU leaves us with a reserved but unused weight of 6952 WU. After the sanctioned transaction a8a2e533.., only 387 WU would have been left in the block, which is less than 6952 WU, and this explains why the sanctioned transaction was not picked. However, after transaction e6b44288.., there were still 8005 WU - 6952 WU = 1053 WU left to fill. These were filled with a 450 WU and a 437 WU transaction ¹. Leaving 1053 WU - 450 WU - 437 WU = 166 WU of unreserved block weight unused.

Block weight usage of block 876590:
320 WU # header weight
+ 12 WU # tx_count var_int weight
+ 716 WU # coinbase transaction weight
+ 3991834 WU # non-coinbase transaction weight
= 3992882 WU # weight of block 876590
+ 6952 WU # unused, reserved weight
+ 166 WU # unused, unreserved weight
= 4000000 WU # maximum consensus block weight

This assumes that the SBI Crypto pool uses the default value for the block weight and doesn’t set a custom value with -blockmaxweight. Based on the statistics posted by @ismaelsadeeq in Analyzing Mining Pool Behavior to Address Bitcoin Core’s Double Coinbase Reservation Issue, it seems like the majority of pools do use the default. In the two-year timeframe analyzed by @ismaelsadeeq, SBI Crypto never mined a block with a weight higher than the non-coinbase transaction weight Bitcoin Core fills by default. They did however mine a block with exactly 0 WU of unreserved block weight left. This indicates that use the default block weight reservation.

This explains why and how the sanctioned transaction was displaced in favor of higher feerate transactions from SBI Crypto’s block 876590 making it unlikely that they did filter this transaction. This is supported by the fact that SBI Crypto did include another sanctioned transaction 1a579eb6.. related to the same sanctioned entity in the same block.

ViaBTC Block 876614 (too young?)

The transaction e26f26d4.. is reported as missing from ViaBTC’s block 876614. It spends from a sanctioned address related to the Chinese illicit drug producers and traders. At the time the block was processed by the miningpool-observer tool, the transaction had only been in my mempool for 28 seconds. Along with the sanctioned transaction, 55 other transactions are missing due to being too “young” and range between an age of 25 seconds and an age of 87 seconds. The sanctioned transaction might not have been propagated to ViaBTC or used in a block template yet. It was likely not filtered by ViaBTC.

ViaBTC did mine the sanctioned transaction b3a25904.. 43 blocks before and the transaction 2951b5bb.. 73 blocks after block 876614.

F2Pool Block 876655 (filtered?)

F2Pool’s block 876655 is reported to be missing the transaction the sanctioned cb35835c... Along with the other transactions, this transaction is sanctioned by OFAC as it spends from an address related to the Chinese illicit drug producers and traders.

The transaction had been in my mempool for more than 17 minutes at the time F2Pool’s block arrived. With a feerate of 2.98 sat/vByte it pays more than enough fees to be included in the block. The mempool.space block-audit agrees with this and marks the sanctioned transactions as “removed”. This indicates that F2Pool could have filtered the transaction.

The mempool-space block-audit marking the sanctioned transaction cb35835c.. as removed.

F2Pool Block 876732 (filtered?)

The F2Pool block 876732 is reported to be missing the sanctioned transaction 9522108d.. which is spent from an address related to the Chinese illicit drug producers and traders. The transaction was in my node’s mempool for nearly 25 minutes before the F2Pool block arrived and paid a high enough feerate to be included in the block. The block-audit on mempool.space marks the sanctioned transactions as “removed” as well. This indicates that F2Pool could have filtered the transaction.

The mempool-space block-audit marking the sanctioned transaction 9522108d.. as removed.

F2Pool block 876883 (filtered?)

F2Pool’s block 876883 is reported to be missing the sanctioned transaction 41f4dfbe.. which spends from an address related to the Chinese illicit drug producers and traders. The transaction had been in my mempool for more than 6 minutes before the F2Pool found the block and paid a high enough feerate to be included in the block. The mempool.space block-audit marks the transaction as “removed”. This indicates that F2Pool could have filtered the transaction.

The mempool-space block-audit marking the sanctioned transaction 41f4dfbe.. as removed.

F2Pool block 877061 (filtered?)

In F2Pool’s block 877061 the sanctioned transaction 1a97f4c4.. is reported to be missing. As with the other transactions, it spends from an address related to the Chinese illicit drug producers and traders. It had been in my node’s mempool for close to 15 minutes. While the mempool-space block-audit labels the transaction with “marginal fee”, it paid enough fees and would have fit into the block. This indicates that F2Pool could have filtered the transaction.

Position of the sanctioned transaction 1a97f4c4.. if it would have been included in the block based on its feerate and weight.

F2Pool block 878458 (filtered?)

F2Pool’s block 878458 is reported to be missing the sanctioned transaction 5e2c59cb.. which spends from an address related to the Chinese illicit drug producers and traders. The transaction had been in my mempool for more than an hour. While the mempool.space block-audit marks the transaction as “marginal fee”, it does pay enough fees to be included in the first half of the block. This indicates that F2Pool could have filtered the transaction.

Position of the sanctioned transaction 1a97f4c4.. if it would have been included in the block based on its feerate and weight.

F2Pool block 878889 (block not full)

For F2Pool’s block 878889, the one-input and one-output transaction 6ad80e22.. is reported to be missing from the block. While the previously mentioned transactions were spent from a sanctioned address, this transaction sent a near-dust amount of 600 sat to an address related to a different Chinese national belonging to a different Drug Trafficking Organization sanctioned by OFAC.

The transaction was in my node’s mempool for more than 16 hours and would have fit into the F2Pool block. However, F2Pool only filled the block with only about 30% of the maximum allowed block weight. While the following F2Pool block 878894 was filled correctly again, the F2Pool blocks 878900 and 878901 were filled with less than 2.5% of the maximum block weight. As @mononautical suggested, these blocks might have been built by a node that was just restarted and didn’t have all transactions loaded into the mempool yet. This reported missing sanctioned transaction can not be used as evidence that F2Pool filters OFAC-sanctioned transactions.

Foundry USA block 878898 (displaced)

In Foundry USA’s block 878898, the sanctioned one-input and one-output transaction e415e518.. is reported to be missing from Foundry USA’s block 878898. Similar to the transaction before, it sends a near-dust amount of 600 sat to an address related to a Chinese Drug Trafficking Organization.

The transaction had been in my node’s mempool for more than 10 days and pays a feerate of 1.05 sat/vByte. However, Foundry USA knew about a large 26548 vByte transaction that pays a feerate of 1.07 sat/vByte and displaced 130 lower-feerate transactions from the block. This includes the sanctioned transaction. This missing sanctioned transaction can not be used as evidence that Foundry filters OFAC-sanctioned transactions.

Conclusion

This blog post analyzes 14 blocks for which OFAC-sanctioned transactions were reported as missing. Out of these, 11 blocks were mined by F2Pool, while SBI Crypto, ViaBTC, and Foundry USA each mined one of the blocks. The missing transactions from SBI Crypto’s block 876590 and Foundry USA’s block 878898 were displaced by higher feerate transactions, and the sanctioned transaction missing from ViaBTC’s block 876614 could have been too young to be included. While F2Pool’s block 875840 is missing a potentially too young transaction and F2Pool’s block 878889 was only 30% full indicating a problem with their block template, the other 9 analyzed F2Pool blocks indicate that F2Pool might be filtering OFAC-sanctioned transactions again.

After I identified four F2Pool blocks with missing sanctioned transactions in my previous analysis (November 2023), the founder of F2Pool confirmed that F2Pool was running a “tx filtering patch”. This patch was disabled for a while. While all pools with more than 5% hashrate mined an OFAC-sanctioned transaction in the past weeks ², F2Pool mined their most recent OFAC-sanctioned transaction over seven months ago on 2024-06-08.

Note that a mining pool operator has the power and right to choose which transactions to include in their blocks and which transactions to exclude. Personally, I want to know when and if pools filter transactions - not because I agree or disagree with US OFAC sanctions, but because I want to know how well Bitcoin’s censorship resistance holds up against pools trying to filter transactions. As all of the missing sanctioned transactions were picked up in the following blocks, I assume Bitcoin’s censorship resistance has been holding up quite well for now. We’ll see if, for example, stronger mining pool regulations being enforced in the future, will change this. Until then, let’s hope mining protocols like StratumV2 and DATUM, which allow miners to build and use their own templates, gain more adoption.

Not your templates, not your blocks.

See the last two transactions on the bottom of page 86 of block 876590 ↩︎
I purposefully leave this part as an exercise for the reader. Similar to how my miningpool-observer tool does not display when a pool mines an OFAC-sanctioned transaction, I don’t want pictures or tables from this blog post to be a potential source of information for regulators. Nonetheless, the data is publicly visible on the blockchain. If a regulator wants to find out, they will. If a reader wants to verify my claims, either take this as a small exercise (here is a list of OFAC-sanctioned Bitcoin addresses to start with) or feel free to reach out and I can share the methodology and potentially some scripts. ↩︎

OpenSats Work-Log 3

Thu, 31 Oct 2024 00:00:00 +0000

This is a copy of the 3rd work-log I sent to OpenSats for my LTS grant.

Disclaimer: Some information that is not (or not yet) meant to be published may have been redacted.

How did you spend your time?

Publications & Talks

Block Template Similarities between Mining Pools: Using the Stratum jobs I’ve collected with my stratum-observer project, I was able to show that the block templates from major mining pools often match. This indicates proxy pooling and suggests mining centralization.
Stats on compact block reconstructions: I published a delving post with data on compact block reconstructions. Specifically, I looked at reconstructions that don’t need an extra round trip for transaction requests. I also used this data to conceptually review the mempoolfullrbf=1-by-default PR. I plan to revisit this topic with recent data.
I authored two Bitcoin Core vulnerability disclosures I was involved with to some extend: Disclosure of DoS due to inv-to-send sets growing too large and Disclosure of hindered block propagation due to mutated blocks. I appeared on the brink.dev podcast to discuss on of these vulnerabilities. # Episode 6: Discussing Pre-25.0 Bitcoin Core Vulnerability Disclosures
I spoke at the eBPF Summit 2024 about how I’m using eBPF and the USDT tracepoints in Bitcoin Core to extract P2P network events to use them for, for example, anomaly detection: https://b10c.me/talks/#monitoring-bitcoin-p2p-network-attacks-and-anomalies-with-ebpf-and-usdt-tracepoints
At the “Bitcoin Burg Academy” I talked about Bitcoin Core development to an non-technical audience: https://b10c.me/talks/#bitcoin-core-development

Projects

peer-observer and infrastructure

To monitor for Bitcoin P2P anomalies and attacks, I run Bitcoin Core “honeynodes” (honeypot nodes). The nodes have additional monitoring attached that is used to record data and metrics. In the last three months, my focus has been on:

Developed a set of tools that can connect to a peer-observer websocket endpoint and display P2P events in real time.
Configured CJDNS connectivity on some of the monitoring nodes
Started a few nodes with a recent ASMap file to test and collect data on ASMap
Configured some nodes with ASan, UBSan and TSan to catch potential issues
All assumes() normally disabled in production nodes are treat as asserts to catch potential issues
I cleaned up a bunch of old Grafana dashboards and created a big playlist
General maintenance of the nodes and infrastructure.

Bitcoin Core

I attended the CoreDev event to catch up with other Bitcoin Core developers. I showcased some of my P2P monitoring tooling and got good feedback and ideas.
I have been GUIX-building and uploading signatures of GUIX build hashes for the recent Bitcoin Core releases.
I helped in another round of (demo) ASMap file creation and publication.
Helped out in exploring a nix-based CI setup for Bitcoin Core to reduce the maintenance burden and allow for faster deployment of more CI resources.
I’ve been keeping my currently open PRs rebased and ready for review.
When ever suitable, I left my feedback on Bitcoin Core PRs and issues.

misc

nix flake: I have a nix flake repository for nix packages and modules of Bitcoin software I use. Next to keeping the packages and modules up-to-date, I’ve added a package for my transactionfee-info rewrite, added a miningpool-observer module and integration test, an asmap-data package, and an auto update functionality for packages.
fork-observer: Based on user feedback from the Warnet project, I improved the UX a bit in #47. I also set up a testnet4 node and attached fork-observer to it: https://fork.observer/?network=4
stratum-observer: While I had planned to work on stratum v2 implementation in Q3 2024, I have postponed this work for now as there is no real stratum v2 pool yet to connect to. Using the collected stratum data to show the similarities between pools seemed more important.
btcffm.org: I’m maintaining the website of the Frankfurt Bitcoin meetup

What do you plan to work on next quarter?

further work on the peer-observer & infrastructure
- keep nodes up-to-date and running
- switch out the current nanomsg message queue for something like NATS: https://github.com/0xB10C/peer-observer/issues/56
- extract non-sensitive infrastructure parts into a separate nix flake and publish a public demo instance again
- collect and analyze data on the transactions requested for compact block reconstruction
continue to work on the projects mentioned above and in the last progress reports

Episode 6: Discussing Pre-25.0 Bitcoin Core Vulnerability Disclosures

Thu, 10 Oct 2024 00:00:00 +0000

I joined Gloria and Niklas on the brink.dev podcast to talk about the recently disclosed Bitcoin Core pre-25.0 vulnerabilities.

Episode 6: Discussing Pre-25.0 Bitcoin Core Vulnerability Disclosures

Block Template Similarities between Mining Pools

Mon, 16 Sep 2024 00:00:00 +0000

Different mining pools sending out the same or a similar block template to miners is an indicator for proxy pooling. Knowing about proxy pools is important when discussing mining pool centralization. To find similarities between mining pool block templates, I compare the Merkle branches pools sent in the stratum jobs and calculate a similarity score. This shows pools with similar templates and allows building a relationship graph between the pools.

In April 2024, I reported that pools like BTC.com, Binance Pool, Poolin, Braiins, and possibly other pools¹ sometimes have the same template and custom transaction prioritization as AntPool. While motivated by on-chain transaction flows, my reporting was mainly based on visual observations over a few days. To further back these claims with data and have a base for further analysis, I started to record the stratum jobs published by major Bitcoin mining pools. In this post, I use a similarity score to compare the pool templates and develop a pool relationship graph showing pools that share templates.

Looking at the merkle branches that mining pools send to miners as part of stratum jobs, it's clear that the BTCcom pool, Binance pool, Poolin, EMCD, Rawpool, and possibly Braiins* have exactly the same template and custom transaction prioritization as AntPool. https://t.co/KTjFWtTXEP pic.twitter.com/xhCrdvkOH8
— b10c (@0xB10C) April 17, 2024

Stratum jobs and Merkle branches

Stratum jobs don’t contain the full list of transactions included in the block template. A miner only needs to construct a block header, which can be done without knowing the full template contents. Modern miners exhaust the 32-bit nonce in the block header quite fast and can then either update the timestamp in the header, roll the version a la overt ASICBoost, or change the so-called extranonce in the coinbase transaction, which causes the Merkle root to change. For this, miners need the coinbase transaction, information about the extranonce, and the Merkle branches to calculate a new Merkle root.

The list of Merkle branches in stratum jobs contains just the information required to calculate the Merkle root. To build the Merkle root, the coinbase transaction is hashes together with the first Merkle branch, the result is then hashed with the second Merkle branch, which is then again hashed with the third Merkle branch. The Merkle root is reached once all Merkle branches have been hashed together.

How to construct a block header from a stratum job: Shows the stratum job on the left and a block header on the top. The individual transactions, the Merkle branches, and the Merkle tree are shown.

The first Merkle branch b0, which is hashed with the coinbase transaction, is the txid of the first non-coinbase transaction in the block. The second Merkle branch b1 consists of the txids of the third and fourth transaction hashed together. The third Merkle branch b2 consists of the hash of the fifth and sixth txid hashed together and the seventh and eighth txid hashed together. The number of transactions included in each merkle branch grows exponentially. While the third branch consists of 4 transactions, the eighth branch already includes 128 transactions.

Similarity-score

To measure the similarity of block templates by different pools, I record the stratum jobs published by major mining pools with my work-in-progress stratum-observer tool. While some pools offer multiple stratum endpoints distributed across the globe, I choose the endpoints located closest to me to reduce latency. Note that different stratum endpoints from the same pool might serve different jobs.

A naive approach to measure the similarity would be to compare the Merkle branches included in mining jobs by calculating the share of branches that match across two pools. For example, if two pools include 10 Merkle branches in their jobs, and the first 7 Merkle branches match, this would result in a 70% match score. However, as the number of transactions in the Merkle branches rises exponentially, the first Merkle branches only contain a few transactions while the later ones contain the majority. The result would show branch similarity, but wouldn’t reflect template similarity.

A weighted approach, where the first branches weigh less than the later branches closer reflects template similarity. Matching branches can be weighted by the maximum number of transactions they can contain. The weighted similarity score of the Merkle branches $A$ and $B$ is

$$ \sum_{A_i \text{ matches } B_i}^l {\frac{1}{2}^{1 + l - i}} $$

where $i$ is the one-indexed position in the list of Merkle branches and $l$ is the minimum length of $A$ and $B$. A similarity score of 100% indicates that the templates match while a similarity score of 0% indicates that no branch matches. The example from above, where the first 7 of 10 Merkle branches match, results in a 12.5% weighted similarity score.

The similarity score is strongly affected by transaction ordering. Two block templates may include the same transactions but as soon as the transaction order is slightly different, the Merkle branches won’t match and the similarity score won’t indicate similarity.

Evaluation

The described similarity score is applied to stratum job data collected between 2024-06-01 and 2024-09-12. The data consists of 690k stratum jobs across 24 pools and pool configurations. My test_0xB10C Pool_ used as a reference, AntPool, BTC.com, Binance Pool, Braiins, the CKPool solo pool, the DEMAND stratum v1 solo pool, F2Pool, Foundry, Luxor, MaraPool and a development endpoint Marapool (dev), Ocean Pool’s four template providers Ocean (default), Ocean (ordis), Ocean (core), Ocean (datafree), Poolin, the PyBlock solo pool, SBICrypto, SecPool, SigmaPool, SpiderPool, Ultimus, and ViaBTC.

Pools generally publish at least one new mining job every minute with a frequently used interval being 30 seconds. To be able to compare recent jobs by different pools to each other, I sampled and compared the most recent jobs every five minutes.

The following similarity matrix gives an overview of which templates are similar. The similarity scores shown in the matrix can be grouped into three groups. First, a high-similarity group with scores between 99% and 80% containing, for example, AntPool-Poolin with a 99% similarity score, and AntPool-BTC.com with a 98% similarity score. Secondly, a low-similarity group with scores between 35% and 20%. This group contains, for example, Braiins-Poolin with a score of 32%. The third group is the no-similarity group with a similarity score below 5%. This group contains, for example, the Ocean (datafree)-ViaBTC pool combination with a similarity score of 0%.

A similarity matrix showing the mean similarity score for all 276 pool combinations. Higher (yellow) means more similar.

High-similarity pools

The pool combinations in the high-similarity group regularly send out mining jobs based on the same template. The high-similarity score combinations are:

AntPool - Poolin: 99%
AntPool - BTC.com: 98%
BTC.com - Poolin: 99%
SecPool - SigmaPool: 97%
Braiins - Ultimus: 89%
SpiderPool - Binance Pool: 81%

Based on these combinations, a preliminary relationship graph can be constructed. A relationship between AntPool, BTC.com, Poolin, and other pools has been assumed before based on coinbase reward consolidations. Braiins and Ultimus Pool both have been seen consolidating coinbase rewards with AntPool and other pools in the past, too. The relationship between SigmaPool and SecPool isn’t surprising, as the SigmaPool stratum endpoint eu1.btc.sigmapool.com only publishes mining jobs with the Mined by SecPool tag in the coinbase transaction. The SigmaPool stratum endpoint is a proxy for the SecPool endpoint. SpiderPool mined its first block in the spring of 2024. The relationship with Binance Pool is interesting.

A preliminary relationship graph based on high similarity scores. This graph is updated below.

Similarity over time

By plotting the similarity scores of the high- and low-similarity pool combinations over time, it becomes apparent that the BTC.com and Binance Pool mining pools switched to a different template provider in the observed time frame. The BTC.com pool switched to the Braiins and Ultimus templates at 9 am UTC on 2024-06-18 for about 24 hours. Binance Pool switched from SpiderPool to the AntPool-Poolin-BTC.com template on 2024-08-23.

Pools with an average similarity score >10% plotted over time.

Binance Pool switching from SpiderPool to AntPool-Poolin-BTC.com can be observed in the coinbase transaction tags sent along with the stratum jobs too. Until 2024-08-23, the Binance Pool endpoint I was connected to sends out jobs with the SpiderPool/ coinbase tag. After the switch, it mainly includes the tag binance/. A relationship between Binance Pool and BTC.com has already been observed in CoinMetrics’s State-of-the-Network #249 “Following Flows V: Pool Cross-Pollination”. While my data does not contain any jobs with binance/ in the coinbase tag before 2024-08-23, there were certainly blocks mined with a Binance Pool tag. Possibly, different Binance Pool endpoints publish different jobs.

From 2024-08-23 on, the Binance Pool jobs sometimes also contain the tag Mined by SecPool. The Binance Pool endpoint seems to transparently, without changing the coinbase transaction, proxy the mining jobs from SecPool from time to time. This partly explains the similarity score of 10% between Binance Pool and SecPool.

Coinbase tags in stratum jobs by Binance Pool’s stratum endpoint

With this information, we can update the relationship graph.

Updated pool relationship graph.

When looking for switches in coinbase tags from other pools, Braiins and Luxor stand out. Braiins normally uses the /slush/ coinbase tag from its predecessor Slush. However, since mid-July 2024, they started to transparently proxy Foundry’s mining jobs from time to time. This partly explains the 10% similarity score between Braiins and Foundry. On 2024-08-23, the day Binance Pool switched from SpiderPool to AntPool-Poolin-BTC.com, Braiins transparently proxied the jobs from Binance Pool for a few hours. Note that the Binance Pool website shows a popup referring to UltimusPool as a “Strategic Business Partner” and mentions that UltimusPool, who have the same block templates as Braiins, provides “technical services” for Binance Pool.

Coinbase tags in stratum jobs by Braiins stratum endpoint

The Luxor stratum endpoint seems to transparently proxy the jobs from the SBICrypto pool from time to time. Interestingly, the Luxor and SBICrypto pool only have a similarity score of 2%.

Coinbase tags in stratum jobs by Luxor’s stratum endpoint

Low-similarity pools

To closer analyze the relationships in the low-similarity groups, it makes sense to inspect where the Merkle branches of the pools with 20%-30% similarity match. The number of Merkle branches in a job depends on the number of transactions in the template, which depends on the pool’s mempool. More transactions in the template mean more Merkle branches send in the mining job. Currently, most mining jobs include 12 or 13 Merkle branches, which corresponds to up to 4096 and 8092 transactions. The following graphic shows the share of matching Merkle branches for each branch in the jobs. For example, when comparing the Merkle branches of Ocean (core) and CKPool (solo), 70% of all Merkle branches matched at position 2 when the jobs contained 12 Merkle branches. The 70% can be found in the top-left chart, on x = 2 and y = 12.

Where do the Merkle branches match? Four examples.

The chart includes four hand-picked examples². The Ocean (core)- CKPool (solo) example in the top-left should be a good baseline for the Merkle branch similarity between likely well-connected but unmodified Bitcoin Core nodes. These two pools have a similarity score of 1%. While branch position 1 matches more than 80% of the time, around position 7 the branches only match in less than 10% of the cases. The last branch positions never match. This is likely caused by small, normally occurring differences in the pools mempools.

The Poolin-Antpool comparison in the top-right shows the extreme case when the Merkle branches almost always completely match. These two pools have a similarity score of 99%. For nearly all positions, 99% or more of the branches across all branch positions match.

The ViaBTC-Ocean (datafree) example in the bottom-left shows the other extreme when the Merkle branches rarely match. These pools have a similarity score of 0%. ViaBTC is known for its custom transaction prioritizations through its transaction accelerator and the Ocean datafree node policy filters out all data-carrying transactions. These block templates are expected to be as different as it gets. There are very few matches between the branches at position 1 and no matches at the later positions.

With a baseline and examples for both extremes, we can better categorize pool combinations from the low-similarity group. The example in the bottom right shows Braiins-AntPool with a similarity score of 32%. The first few branch positions nearly always match. At positions 8 and 9, around 50% of the branches still match. At the last positions, often more than 25% of the templates match. This is significantly better than the baseline and raises the question of why, for example, the Braiins-AntPool templates share more transactions than other pool combinations.

It has been previously observed that AntPool, Braiins, and other pools sometimes prioritize the same low-fee transactions by putting them at the beginning of the block at the same time. One explanation could be that the templates are built from two nodes connected to each other. Both nodes receive the same prioritization. Sometimes their mempools match, and slight mempool differences cause a lower similarity score.

Final pool relationship graph including high- and low-similarity data as well as temporary relationships. Percentages are the average similarity scores. Pools not included here don’t show any significant similarities to other pools.

Looking at the network share of these nine pools can help to put these relationships into perspective. In the past month, AntPool mined 24.8% of the blocks, Binance Pool 2.86%, SpiderPool 2.67%, SecPool 2.14%, Luxor 1.89%, Braiins 1.7%, BTC.com 0.82%, Poolin 0.4%, and UltimusPool 0.31%. In contrast, Foundry mined 31.31% of the blocks. The high-similarity pool combination of AntPool-BTC.com-Poolin has therefore a network share of 26.02%. Braiins and Ultimus Pool together have a share of 2%. All pools together have had a 37.6% share of the network hashrate over the past month, which is significantly larger than Foundry’s share. That said, while the block templates might be unusually similar between some of these pools, and some pools might be engaging as proxy pools for others here and there, it’s not proven that there is a single entity behind these nine pools. Yet, it adds more data points to the discussion around mining pool centralization.

@mononautical reported about coinbase transaction outputs from multiple pools, including AntPool, Braiins, Binance Pool, SecPool, and F2Pool spent in the same transaction. Yet, the F2Pool templates show no similarity to any of the other pools. However, it is known that F2Pool runs its own nodes and builds its own block templates. It’s expected that F2Pool templates don’t match with, for example, AntPools templates.

More insights can be extracted from the stratum job data. For example, looking at the job update arrival time might be interesting as the jobs from some pools arrive at the same time. It might be interesting to look at the custom transaction prioritizations and where these match across pools. The coinbase output value could be analyzed. Pools often having a similar coinbase output value might be peering with each other. Generally, it might make sense to directly peer two otherwise independent Bitcoin Core nodes to see how similar the templates get.

My post mentioned EMCDPool and RawPool. According to mempool.space, EMCDPool mined its last block over a year ago and RawPool has been inactive for three years. While their stratum endpoints still publish jobs, I didn’t include them in this analysis. ↩︎
A collection showing all possible 276 pool combinations can be found here (7.1 MB, ~3500x12000px). ↩︎

Monitoring Bitcoin P2P network attacks and anomalies with eBPF and USDT tracepoints

Wed, 11 Sep 2024 00:00:00 +0000

I spoke at the eBPF Summit 2024 about how I’m using eBPF and the USDT tracepoints in Bitcoin Core to extract P2P network events to use them for, for example, anomaly detection.

Recording (YouTube)

Bitcoin Core Development

Sat, 31 Aug 2024 00:00:00 +0000

At the “Bitcoin Burg Academy” I talked about Bitcoin Core development to an non-technical audience. Often when I talk to non-technical Bitcoiners, they don’t really know much about the Bitcoin Core open-source software project but are very interested to learn more about it. In this talk, I covered the basics starting from the beginning with Satoshi, to the early years without Satoshi, to how Bitcoin Core development works now. I detail how Bitcoin Core developers communicate, I briefly introduce GitHub, Issues, PRs, and review, and I discuss what jobs maintainers have and how Developer Funding works.

Slides (Google Slides)

OpenSats Work-Log 2

Wed, 31 Jul 2024 00:00:00 +0000

This is a copy of the 2nd work-log I sent to OpenSats for my LTS grant.

Disclaimer: Some information that is not (or not yet) meant to be published may have been redacted.

How did you spend your time?

Publications

I’ve been looking at mining pool behavior during forks while building out the stratum-observer stratum job collecting (see below). I regularly covered forks on mainnet with tweets (such as these 1, 2, and 3) and then ended up publishing a short-form explainer video and a blog post on Mining Pool Behavior during Forks.
I’ve looked into compact-block reconstructions on mainnet after another bitcoiner reached out asking for stats on them. I noticed that, since most miners have switched to -mempoolfullrbf=1 by default, enabling full-RBF in Bitcoin Core by default would help block propagation. I commented on this on the current “enable full-RBF by default” Bitcoin Core pull request.
I posted an extended version of the compact-block reconstruction stats on delving-bitcoin.

Projects

peer-observer & infrastructure

To monitor for Bitcoin P2P anomalies and attacks, I run 11 Bitcoin Core “honeynodes” (honeypot nodes) on four continents across three different hosting providers. All nodes have additional monitoring attached that is used to record data and metrics. As leaking the node IP addresses would defeat the purpose of the honeypot, the public interface https://public.peer.observer/ is redacted. I’ve been providing access to interested and trusted developers and community members on an ad-hoc basis.

Over the past months, I’ve been mentoring a Summer of Bitcoin mentee as he has been working on some of the initial anomaly detection and alerting setup. I’ve also been supporting him in his journey learning Rust and more about Bitcoin and Bitcoin Core internals.

Other things include:

On going maintenance of the nodes and infrastructure. For example, coordinating with the MIT DCI to increase disk size on one of the nodes, or setting up new ARM instances with more CPU and disk IO compared to the previous instances.
I build a tool that converts the peer-observer events (e.g. nodes connecting and disconnecting, P2P messages being send and received) to messages published on a websocket. This allows for realtime visualizatioins such as https://x.com/0xB10C/status/1797904155593548273
Working with my mentee, we noticed a few bugs setting his infrastructure up. For example, https://github.com/0xB10C/peer-observer/pull/22, which we fixed together
Build out further dashboards
Set up a Bitcoin Core v0.21.1 node to monitor eventual exploitation of Bitcoin Core :: Disclosure of remote crash due to addr message spam

stratum-observer (step 1)

Following my tweet about Antpool proxy pools, I started to build out proper tooling to observe stratum jobs by mining pools. In the past months, I primarily worked on step 1, which is support for stratum v1, a simple and WIP website showing stratum-job updates in real-time and collecting data on stratum jobs into a database for later analysis.

As other realtime monitoring solutions have popped up (e.g. stratum.work), I’ve decided to hold off on publishing my WIP website until it’s in better shape. The plan for step 2 is to build out support for stratum v2 and to expose more metrics and information on the frontend.

transactionfee-info

REDACTED

I picked up work on a new open-source Rust backend for transactionfee.info. The old Go backend (see transactionfee.info (2020 version)) is closed source and I’m not planning to maintain it any further. Over the last weeks, I tried to bring the new backend closer to being ready to replace the old backend. This will also allow interested contributors to contribute new charts and new metrics.

Bitcoin Core

Over the last few months, the Bitcoin Core guix.sigs repsoistory, containing signatures for release builds from developers, had problems with mismatching build hashes. These were only detected after being merged and used together with a release. I had build a CI job that reports mismatches, however, it turned out that getting the CI to comment on a PR was harder as initially thought. Together with willcl-ark, we fixed the automated GitHub action to comment a SHASUM summary on PRs as soon as they are opened.
I’ve been keeping my currently open PRs rebased and ready for review.

misc

I reworked the fork-observer front-end animations for new blocks and forks in https://github.com/0xB10C/fork-observer/pull/41
I’ve been keeping my collection of Nix modules and packages for software I’ve written up-to-date. I use this for my own setups. I started adding test for newly added packages and hope to add more tests for the older packages soon.
I’ve been reviewing nixpkgs PRs for Bitcoin related software - e.g. https://github.com/NixOS/nixpkgs/pull/322524

Plans for Next Quarter?

start to work on stratum-observer step 2
continue working with my Summer of Bitcoin mentee on peer-observer alerts and anomaly detection
continue to work on the projects mentioned above

Mining Pool Behavior during Forks

Mon, 15 Jul 2024 09:00:00 +0000

I have recently been looking at mining pool behavior during forks. Which block does a pool choose to mine on during a fork? Do they behave rationally and mine on their own block? In this post, I’ll detail the mining pool behavior during forks and give some recent examples of pool behavior.

I’ve also published a short-form video¹ on this topic on Twitter and YouTube.

In Bitcoin, a blockchain fork usually happens when two mining pools find a block based on the same previous block at roughly the same time. The fork is usually resolved with the next block building on top of one of the two fork-blocks. One block becomes part of the active chain (wins) while the other one becomes stale (loses).

A fork at height 45342: Pool A and Pool B both mined a block at height 45342. Pool B mined block 45343 on top of its own block and won. Pool A’s block 45343 became stale.

During a fork, the game theory is as follows: Pools that mined one of the fork blocks always want to mine on their own block and never on the competing fork-block. The chance of finding a block is equal to the share of network hashrate the pool has, and does not depend on who found the previous block. A pool with 5% of the network hashrate has a 5% chance to find a block on the competing pool’s block and a 5% chance of finding a block on his own block. However, mining on its own block, the pool can find the winning block and yield two blocks, the fork-block and the winning-block. Mining on the competing block, it can only yield the winning-block.

Pool B yields only one block when mining on top of Pool A’s block. However, it can yield two blocks when mining on its own block.

Pools that mined neither of the fork-blocks can freely choose which block to build on. By default, they usually end up mining on the block they validate first. Their chance to find a block is equal to their network hashrate. It does not matter which fork-block other pools are mining on.

Recent Examples

The block pools are mining on is sent along with the mining jobs the pools public stratum servers publish. To analyze pool behavior, I’ve been recording the stratum jobs during forks. Note that there are possibly multiple public or even private stratum endpoints that could publish different jobs. The data I have might be incomplete and not show the full picture.

ViaBTC and AntPool at height 848860

Originally posted here.

During the fork at height 848860 between ViaBTC and AntPool, AntPool started off by briefly mining an empty-block-job², a mining job with no transactions in the block, on their own block but then switched to mining on ViaBTC’s block. By switching to ViaBTC’s block they abandoned their block and lost the chance to find the fork- and the winning-block. However, it turned out that Foundry was mining on the AntPool block and found a block on it. Despite abandoning its block, AntPool was able to include one block in the active chain.

ViaBTC and AntPool at height 848860

AntPool and Foundry at height 848968

Originally posted here.

108 blocks later, AntPool and Foundry both mined a block at height 848968 resulting in another fork. Again, AntPool started off with an empty-block-job mining on their own block before switching to Foundry’s block ². Foundry found a block 848969 causing AntPool’s block 848968 to become stale.

AntPool and Foundry at height 848968

AntPool and Foundry at height 851170

Originally posted here.

Similarly, both Foundry and AntPool found a block at height 851170 and AntPool starts off with an empty-block-job² on its own block before switching to Foundry’s block. AntPool ended up finding a block on top of Foundry’s block and both AntPool and Foundry end up with one block each. If AntPool had stayed on their own block, they could have ended up mining both blocks.

AntPool and Foundry at height 851170

Foundry and AntPool at height 848477

Originally posted here.

The fork at height 848477 between Foundry and AntPool was different. All pools, including Foundry, started mining on top of AntPool’s block 848477. About 30 seconds³ in, Foundry switched to a different block at height 848477 mined by Foundry, and they end up finding block 848478 building on top of it. In this case, Foundry successfully switched to their own block, yielding two blocks in the active-chain while AntPool’s block became stale.

Foundry and AntPool at height 848477

Why isn’t AntPool mining on their own blocks?

The question of why AntPool, who have been around since 2015 and have mined 10% of all Bitcoin blocks in existence, are not switching to their own blocks during forks remains open. Aren’t forks frequent enough to bother? Probably, calling preciousblock along with the submitblock RPC when submitting a newly found block to their nodes would do the trick. Maybe some part of their pool software is incompatible with this?

2024-08-05: A previous version of this blog post was called “Mining Pool Game Theory during Forks”. I’ve since changed the title to “Mining Pool Behavior during Forks” as I think this better reflects the contents.

↩︎
AntPool often mines an empty-block-job for 30 seconds after a new block. These empty jobs aren’t based on a block template generated by a Bitcoin Core node and probably originate from a custom mining job creation software. When the job creation software learns about a new AntPool block, it immediately publishes an empty-block-job for it and the block is probably submitted to a Bitcoin Core node. Probably at around the same time, the Foundry block arrives and is validated on the Bitcoin Core node. A Bitcoin Core node, by default, will mine on the block it validated first. This means that if they don’t manually switch, Foundry’s blocks must have reached the nodes first. ↩︎ ↩︎ ↩︎
Foundry implemented a “switch-to-own-block” logic and my guess is that they didn’t bother, or forgot, to implement sending a new mining job for it. The switch happened with the next job update they sent out. Foundry’s job updates happen every 30 seconds. ↩︎

OpenSats Work-Log 1

Tue, 30 Apr 2024 00:00:00 +0000

This is a copy of the 1st work-log I sent to OpenSats for my LTS grant.

Disclaimer: Some information that is not (or not yet) meant to be published may have been redacted.

How did you spend your time?

Publications

ViaBTC’s mutated blocks without witness data: Last year, errors like ERROR: AcceptBlock: bad-witness-nonce-size in the Bitcoin Core debug.log caused confusion. As these were coming up again, I took a look into why they’re happening. Turns out this is a problem with ViaBTC’s pool implementation and nothing to be done on the Bitcoin Core side.
Vulnerability Disclosure: Wasting ViaBTC’s 60 EH/s hashrate by sending a P2P message: While looking at the code of the ViaBTC pool server to learn more about the mutated blocks, I noticed a security vulnerability and responsibly disclosed it.
Update on LinkingLion: Reduced activity and a statement by LionLink Networks: LionLink, the ISP where the LinkingLion connections originate from, issued a statement and subsequently the connections from LinkingLion dropped for a couple of hours before starting again. LinkingLion is still active and possibly linking IP addresses to Bitcoin transaction broadcasts.
Invalid F2Pool blocks 783426 and 784121 (April 2023): F2Pool mined two invalid blocks in early April 2023. I still had notes from looking into these and decided to publish them.
Following mononauts post about major mining pools sharing a custodian and directly mining to that custodian, I looked into the stratum jobs they send and found that the BTC.com pool, Binance pool, Poolin, EMCD, Rawpool, and possibly Braiins have exactly the same template and custom transaction prioritization as AntPool.
- https://twitter.com/0xB10C/status/1780611768081121700
- https://twitter.com/0xB10C/status/1783126815822717080
During the halving, I ran a Bitcoin Halving Monitoring Stream with the goal to keep track of possible selfish mining and reorgs due to the anticipated high reward of mining block 840000. While there weren’t any reorgs, there was a possible reorg attempt.

Projects

peer-observer & infrastructure

To monitor for Bitcoin P2P anomalies and attacks, I now run 12 Bitcoin Core “honeynodes” (honeypot nodes) on four continents across three different hosting providers. All nodes have additional monitoring attached that is used to record data and metrics. As leaking the node IP addresses would defeat the purpose of the honeypot, the public interface https://public.peer.observer/ is redacted. I’ve been providing access to interested and trusted developers and community members on an ad-hoc basis.

I’ll be mentoring someone as part of my Summer of Bitcoin project “peer-observer: Anomaly detection and alerting for Bitcoin Core P2P events”. The goal is to extend peer-observer with proper alerting and to experiment a bit with proper anomaly detection.

The infrastructure work for peer-observer includes (but not limited to):

Setting up four low powered ARM nodes in a new datacenter.
Decommissioning of two nodes used during early development in 2022 and 2023
Enabling detailed debug logging on the nodes and daily log rotation of debug.log files
Automated FTP backup of old debug.log files for future use
Use client certificate authentication instead of basic auth
Update nodes to Bitcoin Core 27.0rc1 release candidate as well as 26.1 and 25.2 release candidates
Rework Grafana dashboards and add a dashboard playlist for TV mode

fork-observer

After noticing and reporting an issue with a stuck btcd node connected to my fork-observer instance, I added an RSS feed for lagging nodes (to be able to easily alert on stuck nodes) and added an RSS feed for offline nodes. Also, exposed and started showing node implementation along with some general refactoring. For the halving stream, I added a fullscreen mode.

Bitcoin Core

I tested hebasto’s proposed Bitcoin Core build system change from CMake to autotools on NixOS: https://github.com/hebasto/bitcoin/issues/121
I opened PR #29636, #29877, #29549, have been keeping #26593 and #25832 up-to-date and 28998 was merged.
I’ve also been experimenting with a possible continuous benchmarking solution for the Bitcoin Core CI. See 27284.
I attended the CoreDev meeting in Berlin in early April and presented my peer-observer work. I also offered to help other developers with data/stats/insights for their proposals or PRs. This resulted in five developers reaching out during and after the event requesting data (mempool data, network-adjusted time data for 29623, benchmarking #29491, non-standard tx stats for the great consensus cleanup, orphan transaction stats and tooling, …).
GUIX builds and hash mismatch tooling: After submitting my reproducible GUIX build signatures for Bitcoin Core 25.2rc2, 27.0rc1, and 27.0 a binary hash-mismatch was noticed. This could be tracked down to me switching to a new build setup. As we don’t have any alerting for hash-mismatches, I PR’d a CI job that comments a summary of the hashes on each PR. The goal is to learn about future mismatches as early as possible to be able to investigate them.

misc

I’ve been keeping my collection of Nix modules and packages for software I’ve written up-to-date. I use this for my own setups. However, till I get around to adding tests and further service hardening, I don’t recommend anyone else using it.
I need a quick and dirty transaction dependency visualization and build tx-family-tree. A live version can be found here: https://0xb10c.github.io/tx-family-tree/ (this loads quite slow and isn’t really meant to be used)
I’ve started a dataset of non-standard transactions I’ll sporadically update. Generated with https://github.com/0xB10C/find-non-standard-tx.

Plans for Next Quarter?

continue to work on the projects mentioned above
build out a stratum job monitoring tool to provide everyone access to the pool’s job information (inspired by https://twitter.com/0xB10C/status/1780611768081121700)
Start to work with my Summer of Bitcoin mentee on peer-observer alerts and anomaly detection

Invalid F2Pool blocks 783426 and 784121 (April 2023)

Tue, 02 Apr 2024 00:00:00 +0000

My notes on the two bad-blk-sigops: too many sigops invalid blocks, 783426 and 784121, mined by F2Pool in April 2023.

On April 1st, 2023, F2Pool mined an invalid block at height 783426. Bitcoin Core nodes rejected the block with the reason bad-blk-sigops and the note too many sigops. On April 6th, 2023, F2Pool mined another bad-blk-sigops block at height 784121. Mining an invalid block with a valid proof-of-work is an expensive mistake. F2Pool lost more than USD $300k in block rewards with these two invalid blocks.

ERROR: ConnectBlock(): too many sigops
ERROR: ConnectTip: ConnectBlock 00000000000000000002ec935e245f8ae70fc68cc828f05bf4cfa002668599e4 failed, bad-blk-sigops

While invalid blocks are normally not relayed on the Bitcoin P2P network, these blocks were likely announced as BIP-152 compact blocks. To ensure fast block propagation, compact blocks are relayed across BIP-152 high-bandwidth connections before they are fully validated.

BitMex Research reported about the first invalid block and Sjors Provoost began analyzing it by counting sigops. A sigop is a Bitcoin script opcode that performs a signature check. For example, OP_CHECKSIG, OP_CHECKSIGVERIFY, OP_CHECKMULTISIG, and OP_CHECKMULTISIGVERIFY. As signature checks are computationally expensive, blocks are limited to 80000 sigops. The sigops in, for example, OP_IF ... OP_ENDIF branches are counted even if they aren’t executed. Similarly, sigops in coinbase inputs are counted but won’t ever be executed ¹.

With the activation of SegWit, the sigops limit was raised from 20000 to 80000. At the same time, the cost of sigops in pre-SegWit scripts (e.g., P2PKH, P2SH, ..) was quadrupled. For example, a OP_CHECKSIG in a P2PKH output is counted as 4 sigops but was counted as 1 sigop before SegWit activation. In P2WSH a OP_CHECKSIG is counted as 1 sigop.

Sjors counted 80003 sigops for the first invalid block. Three sigops more than the limit of 80000 sigops allows. While some joked the first block on April 1st was an “April F2Pool’s block”, on April 6th the second invalid F2Pool block arrived and made it clear that this wasn’t a one-time event nor an expensive April F2Pool’s joke.

The second invalid block also counted 80003 sigops. At this point, speculation arose that F2Pool was likely running custom software with a bug. Sjors guessed that F2Pool didn’t correctly count the sigops of the coinbase transaction. In a conversation with an F2Pool engineer, I learned that F2Pool had an old patch to Bitcoin Core, which reduced the sigops reserved for the coinbase transaction to a single sigop. This should probably act as a block-building optimization in cases where the sigops limit is the limiting factor to including a high-fee transaction. Before SegWit, 100 sigops were reserved for the coinbase, and as part of the SegWit changes, nBlockSigOpsCost was increased to 400 sigops.

Reserving only a single sigop in the coinbase was fine before SegWit when using a P2PKH output as the OP_CHECKSIG in the P2PKH output was counted as one sigop. After the activation of SegWit, F2Pool would either have to increase the reserved sigop count from one sigop to four sigops or switch to a P2WPKH output (no sigops) or a P2WSH output with a single sigop ².

Since increasing the coinbase reserved sigops count, F2Pool hasn’t mined another invalid block. However, they now mine blocks with two P2PKH outputs in the coinbase³. As these both cost 4 sigops, if they had hard-coded their reserved sigop count to 4, adding an extra P2PKH coinbase output might have caused an invalid block again.

The F2Pool engineer told me that F2Pool now reserves 100 sigops for the coinbase (400 is the default). This can be observed on-chain. In the recent past, F2Pool’s blocks 821336, 824444, 824931, and 824934 had a sigop count of 79907. Subtracting the 8 sigops from F2Pool’s coinbase from 79907 results in 79899 sigops. With 100 sigops reserved for the coinbase, this equals 79999 sigops. This matches the Bitcoin Core mining algorithm, which fills blocks to one sigop below the 80000 limit ⁴.

height	sigops	pool	hash
821336	79907	F2Pool	`00000000000000000000fa5888308cfcdf29ab782112bc071074081da9fcda3a`
824444	79907	F2Pool	`000000000000000000016e1916d655ad057b871654ba1006d267b2f7c8f6ab9f`
824931	79907	F2Pool	`00000000000000000002ba7ba36fb73ebc989eec6eed842a2e5da5a38e94fed4`
824934	79907	F2Pool	`0000000000000000000090b2fe804461fd6e2171da8c14c567f96ffdcb905fc0`

Other pools, such as Braiins⁵ (784123), MARA Pool (e.g. 820460), and ViaBTC (e.g. 824442) mined blocks with 79603 sigops. In these blocks, they use a P2PKH output with 4 sigops. With the default nBlockSigOpsCost of 400, these blocks also reached the Bitcoin Core miner limit of 79999 sigops (79603 - 4 + 400). Other pools that use a P2SH-P2WSH, P2WPKH, P2WSH, or even P2TR coinbase output, such as Foundry USA (e.g. 783426), Binance Pool (784124), AntPool (790224), Luxor (821174), BTC.com (822717), and SBI Crypto (824443) all mined blocks with 79599 sigops in the recent past. Here, the coinbase output does not have sigops, and they use the default coinbase reserved sigops count of 400.

An interesting case is the Ocean.xyz pool. Here, the largest miners are paid directly to the miner-specified address in the pool’s coinbase output. Thus, their coinbase sigops count is based on which addresses the miners specify. The pool only allows P2PKH, P2SH, SegWit, and Taproot addresses. If they allowed raw P2MS scripts, an attacker could fill the coinbase with 5 P2MS outputs having 80 sigops each to exhaust the default sigops reserved for the coinbase. This would produce an invalid block if the transactions in the block would have more than 79600 sigops. Similarly, 100 P2PKH outputs with 4 sigops each are required to exhaust the sigops reserved for the coinbase. Currently, all but one Ocean block have 20 coinbase outputs, which includes an OP_RETURN SegWit commitment and their pool fee output. At maximum, there were two P2PKH outputs with 4 sigops each in, for example, block 829513.

Looking at the valid blocks between height 760000 (2022-10-23) and 836428 (‎2024-03-26), there are only 99 blocks with more than 70000 sigops. In median, in this time frame, the blocks had 10348 sigops with the 25th percentile having 6407 and the 75th percentile having 13950 sigops.

Block sigops per height

If you found this interesting, you might also like my notes on the invalid MARA Pool block 809478.

For future reference, the invalid block at height 783426 had the hash 00000000000000000002ec935e245f8ae70fc68cc828f05bf4cfa002668599e4 (header, full block). The invalid block at height 784121 had the hash 000000000000000000046a2698233ed93bb5e74ba7d2146a68ddb0c2504c980d (header, full block).

This description is based on Pieter Wuille’s answer https://bitcoin.stackexchange.com/a/117359/63817 ↩︎
Mining pools often have large amounts of funds in the same coinbase output address. They might want to wait before switching to a new address version and script type until it’s more battle-tested. ↩︎
F2Pool’s coinbase transactions separate the first few sats of each block to an extra address in an additional P2PKH output. Under the Ordinal Theory, the first Satoshi of each block is an uncommon sat. There are marketplaces where these outputs are offered and sold for more than their bitcoin value. This is MEV. ↩︎
This was surprising to me. During validation and when connecting, Bitcoin Core checks that a block hasn’t more (> MAX_BLOCK_SIGOPS_COST) than 80000 sigops. So a block with 80000 sigops should be allowed. This was introduced in PR 7600. A review comment noted that this check isn’t correct. However, as the sigops limit is rarely ever reached and 400 sigops are reserved for the coinbase, an off-by-one error isn’t too big of a deal. ↩︎
Braiins mined block 824446 with 79599 sigops. They switched to a P2SH-P2WSH output for the coinbase reward. ↩︎

Update on LinkingLion: Reduced activity and a statement by LionLink Networks

Thu, 28 Mar 2024 09:00:00 +0000

This is an update on the LinkingLion entity, presumably linking Bitcoin transactions to IP addresses, I published about a year ago. Yesterday, LionLink Networks AS issued a statement on their non-affiliation with the LinkingLion entity and on the same day, LinkingLion activity significantly dropped.

Exactly a year ago, I published about LinkingLion, an entity that’s presumably linking transactions to IP addresses. The entity has been opening hundreds of connections per minute to my monitoring nodes for the last year.

On March 27, 2024, LionLink Networks AS, who are announcing the IP addresses ranges used by the entity, published a statement on LinkingLion.net. In the statement, the LionLink Networks AS denies any involvement with the LinkingLion entity other than announcing the IP addresses ranges. I have no reason to believe that the AS is involved with LinkingLion entity other being the common factor in announcing the IP addresses used by the LinkingLion entity. I understand that the play on words “LinkingLion”, being a combination of LionLink (the AS that announces IP addresses used by LinkingLion), “Linking” as in linking IP addresses to Bitcoin transactions, and Dandelion as privacy protocol helping against transaction linkage of transactions to IP addresses, is, from a public relations standpoint, unfortunately close to their company name.

To clarify, I don’t think the LionLink Networks AS is the LinkingLion entity.

Coincidentally, I’m happy to report that since yesterday (about 2024-03-27 3pm UTC), I’m not receiving connections from two of the three IPv4 ranges anymore, and I am seeing reduced numbers of connections from the third IPv4 range. I currently don’t have data on the IPv6 range at hand. Maybe the LinkingLion entity is switching IP ranges or adapting their strategy or software? Or it’s just a network degradation or outage with coincidental timing?

209.222.252.0/24: no new connections
91.198.115.0/24: no new connections
162.218.65.0/24: reduced connections to ~100/min with previously about 200/min - 400/min
2604:d500::/32: no data

Total new connections from LinkingLion per minute per /24 subnets on 2024-03-27

On the Bitcoin Core side, there’s work being done by vasild and reviewers to broadcast transactions via short-lived Tor or I2P connections, when possible. This should help against entities like LinkingLion. I personally haven’t been able to take a detailed look at this proposal.

Update 2024-03-29

LinkingLion is opening connections again.

Total new connections from LinkingLion per minute per /24 subnets on 2024-03-29

OpenSats supports Vasil, myself, and others with a Long-Term-Support grant to work Bitcoin Core and monitoring like these. Consider donating to OpenSats if this is work you find worth supporting. Additionally, the MIT DCI is providing me with server infrastructure that I use to monitor for P2P anomalies and attacks, like, for example, the behavior by the LinkingLion entity.

Vulnerability Disclosure: Wasting ViaBTC's 60 EH/s hashrate by sending a P2P message

Wed, 20 Mar 2024 09:00:00 +0000

In January, while investigating a misbehaving client on the Bitcoin P2P network, I found a vulnerability in ViaBTC’s, the fourth largest Bitcoin mining pool, SPV mining code that allowed a remote attacker to waste ViaBTC’s 60 EH/s hashrate by sending a single, crafted Bitcoin P2P message. I responsibly disclosed this to ViaBTC, and they awarded a bug bounty of 2000 USDT.

Improper Input Validation in the SPV mining module of the ViaBTC mining server (not fixed on GitHub, fixed in a closed-source version) allows a remote attacker to waste the pools hashrate by letting it mine on an old block (i.e. DoS) by sending a modified, old block via the P2P network.

ViaBTC’s SPV mining module and Bitcoin P2P client, bitpeer, did not check the merkle root of received blocks, allowing an attacker to make ViaBTC’s miners mine on an old block by sending a modified previous block. While ViaBTC fixed this vulnerability in their systems, the open source viabtc_mining_server is still affected.

ViaBTC Mining Server

In March 2021, ViaBTC published a version of its mining server on GitHub. While this repository hasn’t been updated since summer 2021, the underlying software, likely with a few modifications and patches, is still in use by ViaBTC. The mining server is made up of multiple modules. For the disclosed vulnerability, the interesting modules are the bitpeer module and the jobmaster module. The jobmaster module is responsible for producing mining jobs by talking to a Bitcoin Core node. The bitpeer module is a Bitcoin P2P client and connects to multiple other nodes on the Bitcoin P2P network. The bitpeer client broadcast ViaBTC’s newly found blocks to Bitcoin nodes and notifies the jobmaster about new blocks received over the network.

Infrastructure diagram of the ViaBTC mining server. Based on this overview.

SPV mining

Mining pools want to minimize the time they spent mining on an old block, when a new one has been found by another pool. To quickly switch to a new block, ViaBTC’s bitpeer module sends details, like the new block height and the block hash, to the jobmaster. If the new block height is current block height + 1, the jobmaster switches to SPV (Simple Payment Verification) mining mode. In SPV mining mode, a mining job is issued to miners without having validated the new block. Without having validated the transactions in the previous block, the SPV mining job can only be an empty block (only a coinbase transaction).

Vulnerability

The process_block() function in bp_peer.c is called when a new block message arrives over the P2P network. In this function, it’s first checked that the message contains a block header with enough proof-of-work (higher than the current difficulty requirement). Then, the BIP-34 block height is extracted from the coinbase. If this height is larger than the best know height, the block is send to the jobmaster by calling send_block_nitify().

Here, checking that the block header has a valid and enough proof-of-work to be an SPV-valid block defends against most attacks. The cost of mining a block header with enough proof-of-work is high. For an attacker, it does not make sense to spend that hashrate to attack ViaBTC with an otherwise invalid block. A rational actor would use the hashrate to mine for them selves.

However, ViaBTC only checks the header. They don’t verify that the transactions in the block match the merkle root in the header. This means, the coinbase transaction and thus the BIP-34 block height in the coinbase transaction can be modified. Since they use the height in the coinbase to determine if they should SPV mine, the pool can be tricked into mining on a valid, but old, header with an arbitrary coinbase transaction attached.

Exploit

To exploit this vulnerability, an attacker needs to send a crafted block message to a ViaBTC bitpeer client. ViaBTC runs multiple instances of bitpeer in different data centers distributed around the globe. Each instance opens connections to multiple listening nodes on the P2P Bitcoin network. On my nodes I counted zero to two connections from bitpeer instances per node. In total, I saw seven different IP addresses, which indicates there are at least seven instances running. These bitpeer peers can be detected as they always use the same fake user agent of /Satoshi:0.19.0.1/ and only set the WITNESS service flag. The IP addresses always belong to AWS-owned IP ranges. Connections from bitpeer instances can, for example, be listed with the following command.

$ bitcoin-cli getpeerinfo | jq '.[] | select(.subver == "/Satoshi:0.19.0.1/" and .services == "0000000000000008")'

A block header used for this attack should be from the same difficulty adjustment period to meet the difficulty requirements. It does not matter which coinbase transaction is chosen. The BIP-34 height in the coinbase input needs to be current network height + 1. To send this block to a bitpeer instance, the recently introduced, test-only sendmsgtopeer Bitcoin Core RPC call can be used.

$ bitcoin-cli sendmsgtopeer <peer-id> "block" <block hex>

Local and remote verification

To make sure the vulnerability is practicably exploitable before I disclose it, and not only a theoretical issue, I tested it against a local test setup using the viabtc_mining_server code available on GitHub.

I build, installed, and configured a bitpeer and jobmaster instance to use a local Bitcoin Core node. The bitpeer instance was configured to connect to a second local Bitcoin Core node via P2P. To be able to start the bitpeer and jobmaster instances I needed to comment out some startup actions for services I was unable to configure (mostly due to missing documentation). Once set up, I constructed a block composed of a real block header and a coinbase transaction encoding a height a few hundred blocks into the future. I send the block to the bitpeer instance, and it notified the jobmaster about the new block. I couldn’t tell from my patched jobmaster instance if it would have switched to SPV mode as it crashed due not being configured correctly. This made it hard to verify the vulnerability locally.

At this point I wasn’t sure if this vulnerability could even be exploited against ViaBTC. Surely they are running an updated and maintained version of their mining server? I contemplated trying to verify the vulnerability against ViaBTC’s production mining pool. As far as I could tell, there is no ViaBTC testnet pool available. I knew that ViaBTC would, by default, only SPV mine for 30 seconds, which made the possible damage done manageable: about $1000 of lost revenue per attack. If the vulnerability could be exploited, I could supply ViaBTC with log timestamps and the information about the block I send. Showing that the vulnerability can be exploited against their production pool increases the chances of them taking this seriously. I feared they wouldn’t take this seriously if I only a reported a vulnerability in unmaintained version of their mining server they uploaded to GitHub three years ago¹.

I ultimately decided to go ahead with trying to verify the vulnerability against the production pool. I attempted this only once, while taking great care not to unnecessarily harm ViaBTC or it’s customers.

I choose block 826284 mined a few hours earlier. This block has only a coinbase transaction and is comparatively quite small, which made it easy to handle in a text editor and on the command line. However, vulnerability can be exploited with larger blocks just as well. At the time, block 826337 was the most recent block. My Bitcoin Core node saw it at 2024-01-19 00:55:46. The 53 block difference between these blocks was big enough to make it clear to ViaBTC that a successful exploitation isn’t just a small reorg.

Block 826284 encodes an original block height of ac9b0c in the coinbase transaction. I set the height to e29b0c (826338; current height of 826337 + 1) in the coinbase using a text editor. Note that the BIP34 height in the coinbase is encoded in little-endian.

  original ac9b0c - 826284
current e19b0c - 826337
  modified e29b0c - 826338

The following shows the modified block 826284. The only difference to the original block is that the coinbase height has been modified from ac9b0c to e29b0c.

0000cd2765e111fa8f2870ae4fdaa564907e501ac5ab12d0cab6020000000000000000007908d44825dc7bc91fb883fa5b2
083f0f95641aa4ac518bb968c2f29ae61a00f4357a96569d80317397f083301010000000100000000000000000000000000
00000000000000000000000000000000000000ffffffff6403█e29b0c█2cfabe6d6d654b01255ebb409c165395395b3f3c2
301f032f53df45cba5ed5d266dc2c786010000000f09f909f092f4632506f6f6c2f73000000000000000000000000000000
00000000000000000000000000000000000000000500ae970800000000000422020000000000001976a914c6740a12d0a7d
556f89782bf5faf0e12cf25a63988ac1ebc4025000000001976a914c825a1ecf2a6830c4401620c3a16f1995057c2ab88ac
00000000000000002f6a2d434f524501a21cbd3caa4fe89bccd1d716c92ce4533e4d4733bdb2a04b4ccf74792cc6753c27c
5fd5f1d6458bf00000000000000002c6a4c2952534b424c4f434b3acd2e3ba1354794d09aabccd650c2155ae16cd9830cc9
b0d57aecd423005ba3a64940a53f

To track which previous block ViaBTC mines on, I set up a patched version of achow101’s stratum-watcher. This connects to the ViaBTC stratum server and listens for new mining jobs. My patch prints which previous block is specified in the mining jobs. If ViaBTC sends out a new mining job with the block hash of 826284, I’d know that ViaBTC is vulnerable.

I send the modified block to one of my bitpeer peers using the sendmsgtopeer RPC call on 2024-01-19 at 00:56:28 UTC. My Bitcoin Core node with net debug logging showed that the block was sent: sending block (409 bytes).

At the same time, I saw that the ViaBTC stratum servers I was connected to send out a new mining job switching to mining on block 0000000000000000000231524f6ba483c6d6e84b68622ec7128a7269bcb9a9d8. This meant I had successfully confirmed that the vulnerability is exploitable against the production ViaBTC mining pool. All other pools kept mining on 0000000000000000000005575548ec79ea5afb112f91422e12aad67080fda017.

[..]
00:56:25,877: btc.f2pool.com mining on 0000000000000000000005575548ec79ea5afb112f91422e12aad67080fda017
00:56:25,924: btc-eu.f2pool.com mining on 0000000000000000000005575548ec79ea5afb112f91422e12aad67080fda017
█ 00:56:28,678: btc.viabtc.io mining on 0000000000000000000231524f6ba483c6d6e84b68622ec7128a7269bcb9a9d8
█ 00:56:28,687: btc.viabtc.com mining on 0000000000000000000231524f6ba483c6d6e84b68622ec7128a7269bcb9a9d8
█ 00:56:28,699: btc.viabtc.com mining on 0000000000000000000231524f6ba483c6d6e84b68622ec7128a7269bcb9a9d8
█ 00:56:28,904: btc.viabtc.io mining on 0000000000000000000231524f6ba483c6d6e84b68622ec7128a7269bcb9a9d8
00:56:32,710: solo.ckpool.org mining on 0000000000000000000005575548ec79ea5afb112f91422e12aad67080fda017
00:56:35,854: stratum.kano.is mining on 0000000000000000000005575548ec79ea5afb112f91422e12aad67080fda017
[..]

Exactly 30 seconds later (ViaBTC’s default SPV-mining timeout), ViaBTC switched back to the correct previous block 0000000000000000000005575548ec79ea5afb112f91422e12aad67080fda017. In these 30 seconds, ViaBTC’s miners wasted about 1.8 sextillion hashes (1.8×10²¹ or 1.8 zeta hashes) mining on an old block. No block was found during these 30 seconds by neither ViaBTC nor another pool. If a ViaBTC miner had found an empty block, the block would have been invalid² as the height commitment in the coinbase would been 826338 and not 826285 as expected with the previous block hash in the header.

00:56:49,503: [..]slushpool.com mining on 0000000000000000000005575548ec79ea5afb112f91422e12aad67080fda017
00:56:51,422: [..]slushpool.com mining on 0000000000000000000005575548ec79ea5afb112f91422e12aad67080fda017
█ 00:56:58,645: btc.viabtc.io mining on 0000000000000000000005575548ec79ea5afb112f91422e12aad67080fda017
█ 00:56:58,677: btc.viabtc.com mining on 0000000000000000000005575548ec79ea5afb112f91422e12aad67080fda017
█ 00:56:58,721: btc.viabtc.com mining on 0000000000000000000005575548ec79ea5afb112f91422e12aad67080fda017
█ 00:56:58,899: btc.viabtc.io mining on 0000000000000000000005575548ec79ea5afb112f91422e12aad67080fda017
00:57:02,589: solo.ckpool.org mining on 0000000000000000000005575548ec79ea5afb112f91422e12aad67080fda017
00:57:05,867: stratum.kano.is mining on 0000000000000000000005575548ec79ea5afb112f91422e12aad67080fda017

I stopped digging further at this point and responsibly disclosed the vulnerability to ViaBTC.

Impact

The vulnerability has a denial-of-service impact with a measurable associated business cost. By default, ViaBTC miners are paid according to PPS+ (Pay Per Share Plus): this means, miners are paid even if ViaBTC does not find a block or isn’t rewarded for finding a block. When the vulnerability was discovered, ViaBTC had around 12% of the network hashrate. Assuming 144 blocks per day, that are 12 blocks per day found by ViaBTC. With each block worth at least 6.25 BTC, that’s 75 BTC or $3M USD per day at $40k USD/BTC. This is ~$35 USD per second. Exploiting the vulnerability once for 30 seconds costs ViaBTC at least $1000 USD.

When a bitpeer instance sends a SPV mining notification, it increases the best known height and does not notify for the same height again. This means, a single bitpeer instance can only be used once per block to exploit the vulnerability. However, since there are at least seven bitpeer instances running, in theory, a malicious actor (e.g., a competing mining pool) could have exploited the vulnerability at least seven times per block. A attack over a longer time-frame would be detectable not only by ViaBTC but also on e.g., fork.observer as ViaBTC would have an abnormal stale block rate. An attacker could also be more careful and exploit this sporadically over a longer time-frame, which could cause serious financial damage if undetected.

Communication with ViaBTC

ViaBTC offers a bug bounty program (mainly scoped for their viabtc.com website) where they list a Zendesk support email to report vulnerabilities to. I didn’t find a dedicated security contact or way to send an encrypted message. My responsible disclosure included all information I know about the vulnerability, details how I verified the vulnerability against the ViaBTC production pool, how they can verify it against a local setup, a discussion of a potential impact if exploited multiple times, a recommendation on how to fix the vulnerability, and some general notes on running old C code with seemingly no test or fuzz coverage for critical business infrastructure. I also included that it would be nice if they could notify potential other pools using the same software. Additionally, I choose to send the ViaBTC CEO and author of the ViaBTC mining server, Haipo Yang, a short summary of the vulnerability and it’s potential impact.

The next day, ViaBTC confirmed they’ve received the vulnerability. Three days later, ViaBTC assigned it severity level 1 of 3 and awarded a 500 USDT bounty. I followed up asking for a reasoning behind the level 1 classification. They re-classified the vulnerability as level 2 and raised the bug bounty reward to 2000 USDT. They also noted that their risk scoring system and monitoring had detected my verification attempt. Furthermore, they claim, a fix was implemented immediately, even before I reported the vulnerability. If true, then the classification as level 2 under “certain asset losses” makes sense. The vulnerability couldn’t have been exploited over a longer time range causing level 3 “serious asset loss” (i.e., ViaBTC paying their PPS+ miners out of their pockets and their PPLNS miners moving away due ViaBTC’s “bad luck”).

While ViaBTC responded quick, funneling the communication with the “Security Risk Team” through a support agent isn’t ideal. Their responses were short and at first I thought they don’t understand the vulnerability, maybe due to a noticeable language barrier. ViaBTC didn’t provide details on how they fixed the vulnerability (they, obviously, don’t need to) and didn’t respond to my offer to re-test the vulnerability. I choose to re-test nonetheless before publishing this disclosure. Ignoring the rough edges, it was still a positive experience working with them.

Timeline

2024-01-18: vulnerability detected and successfully exploited locally
2024-01-19:
- Confirmed ViaBTC production mining pool is vulnerable
- Responsible disclosure of the vulnerability to ViaBTC
- Disclosure of a vulnerability summary to Haipo Yang, ViaBTC CEO and pool software author
- Vulnerability was fixed immediately after being detected by ViaBTC’s “risk scoring system” (claimed by ViaBTC)
2024-01-20: ViaBTC confirms the vulnerability has been received
2024-01-24: ViaBTC classifies vulnerability as level 2 and awards a 2000 USDT bug bounty
2024-02-06: I re-tested that the vulnerability is indeed fixed
2024-03-20: Public disclosure

Notes

To my knowledge, only ViaBTC was affected. I hope ViaBTC informed other known users of their software about the vulnerability. I contacted a few people in the mining pool community who asked around if anyone is using the ViaBTC mining server, however it doesn’t seem there’s much usage besides ViaBTC. As far as I can tell, all bitpeer connections I receive are from ViaBTC. However, someone might be using the GitHub version to host a mining pool for another coin. I opened an issue in the repository to make potential users aware.

I can’t know for sure if this hasn’t been expoited in the wild, but I don’t have any reason to assume it was before I validated the vulnerability against the ViaBTC production mining pool. If ViaBTC’s “risk scoring system” really detected my attempt, it would likely have detected other attacks too.

Reflection

I think, it was the right decision to test if ViaBTC’s production pool is vulnerable. The claimed internal alarms made sure they took this vulnerability serious and fixed it before I could report it. Additionally, it removed the necessity for ViaBTC to verify a theoretical vulnerability in a test environment – they had confirmation that it works. Still, I could have been more careful when exploiting the vulnerability by choosing the current block to mine an empty block on. This would have cost them the transaction fees, but not the whole block reward. Yet, this might also have not been alerted by their “risk scoring system”.

Running the C code, as uploaded on GitHub, as a production mining server seems scary to me. As far as I can tell, there is no unit testing (there seems to be a few years of in-the-wild-usage-testing though…), no fuzz-testing, error handling seems to be done by returning a failed line number, etc. I personally wouldn’t run this in production. Especially not with 12% of Bitcoin’s hashrate. I suspect the vulnerability existed since Haipo originally wrote the ViaBTC mining server in 2016. While I briefly checked that there weren’t any scary buffer overflows in the code that accepts P2P messages before publishing this disclosure, I decided not to spent more time digging deeper. Usually, disclosing a vulnerability ends up bringing in more eyes on the code. In this case, it wouldn’t surprise me if there are more vulnerabilities to be found.

This once again confirms my impression that some mining pools aren’t as technologically advanced as one might think. Especially older pools might still be stuck with a bit of technical dept. However, ViaBTC seems to have adequate an alerting and monitoring setup.

Credits

Will Clark reminded me about the weird P2P network behavior of what we found out to be ViaBTC bitpeer P2P clients. Furthermore, I found the discussion with Will about the vulnerability and how to best handle it helpful. The MIT DCI provides me with servers I use to run Bitcoin Core nodes alongside some monitoring tools. Having access to multiple nodes was helpful in collecting the initial data about the bitpeer clients. My day-to-day work of, for example, monitoring Bitcoin network anomalies, was funded by Sprial and Human Rights Foundation when I found the vulnerability and is now funded by OpenSats while writing this disclosure. Without their support, I wouldn’t be able to work on Bitcoin full-time. Consider donating to them if you want to support my and others work on Bitcoin.

However, who knows who is still using this… ↩︎
A previous version of this post clailmed that the block would have been valid but stale. This was incorrect. AJ made me aware of this in https://twitter.com/ajtowns/status/1770780046707302514. ↩︎

ViaBTC's mutated blocks without witness data

Mon, 18 Mar 2024 00:00:00 +0000

I noticed multiple ERROR: AcceptBlock: bad-witness-nonce-size errors in the debug log of my Bitcoin Core node. These indicate that a block my node received is invalid and not accepted. It turned out that these are ViaBTC’s blocks, broadcast by their mining pool software, where transaction witness data is missing. In this post, I’ve written down my notes on this observation.

Some Bitcoin Core node operators might have noticed the following error in the log of their listening node:

ERROR: AcceptBlock: bad-witness-nonce-size, ContextualCheckBlock: invalid witness reserved value size
ERROR: ProcessNewBlock: AcceptBlock FAILED (bad-witness-nonce-size, ContextualCheckBlock: invalid witness reserved value size)

This error can be found on bitcointalk as early as November 2019 and was a topic in a bitcoin-core-dev IRC meeting in May 2023 as well as in multiple conversations I had. As part of my ongoing Bitcoin P2P network anomaly monitoring, I decided to look into the origin of this error.

The log message tells us that, while processing a newly received block, a block validation check failed with the reason bad-witness-nonce-size and the note: invalid witness reserved value size. Bitcoin Core¹ validates new blocks in four steps. First, the block header’s proof of work, the timestamp, and the block version are checked. As a second step, the block’s properties that don’t require context are checked. These checks verify that the merkle root matches the transactions, the block is not too large, that only the first transaction is a coinbase transaction, the transactions are consensus valid (inputs and outputs not empty, output value not too large, …), and that the block is under the sigops limit. The checks in the third step involve context. The transaction locktime, that the coinbase script starts with a block height, that the witness commitment in the coinbase transaction is valid, and that the block weight is below the maximum weight is checked. The fourth and final step checks, for example, the UTXO’s, input scripts, and block reward. If all checks are successful, the block is accepted.

The witness commitment is checked in step three. As specified in BIP 141, the witness commitment is the hash of the witness root hash and witness reserved value. The witness root hash is the root of a merkle tree using the witness transaction hashes, wtxid’s, as leaves. Compared to the merkle root in the block header, the witness root also commits to the transaction witness data. The witness reserved value is a 32 byte value stored in the coinbase witness. It’s currently unused, but can be used by future softforks to add a new commitment. The coinbase witness is required to be exactly one 32 byte element, otherwise, the bad-witness-nonce-size error is raised. The witness reserved value was previously called witness nonce and the error was likely never changed.

if (block.vtx[0]->vin[0].scriptWitness.stack.size() != 1
 || block.vtx[0]->vin[0].scriptWitness.stack[0].size() != 32) {
 return state.Invalid(BlockValidationResult::BLOCK_MUTATED, "bad-witness-nonce-size",
 strprintf("%s : invalid witness reserved value size", __func__));
}

The proof-of-work of the received blocks is valid, which means that someone invested a large amount of energy into mining the block. As Bitcoin Core only processes new blocks it does not yet know, this could indicate that my node received a block shortly after being mined. Maybe the block came directly from a mining pool?

Digging deeper, the Bitcoin Core debug.log, with the net category turned on, reveals that these blocks are sent to us with an unsolicited (we didn’t ask for this block in a getdata message) Bitcoin P2P block message. These block messages stand out, as blocks on the Bitcoin network are typically relayed as compact blocks². The peers sending these blocks are all inbound connections. The version message they send to us contains the user agent /Satoshi:0.19.0.1/, a version of 70015, and has only the NODE_WITNESS service flag set. The user agent is clearly fake as Bitcoin Core version 0.19.0.1 never sets only the NODE_WITNESS service flag. The peer’s IPv4 addresses all belong to IP ranges owned by AWS - but belong to different AWS regions. The logs also show the block hash of the received block. Looking up the block hashes in a block explorer shows that the blocks are all mined by ViaBTC and, apparently, a valid version of them was relayed and accepted by the network.

On GitHub, an old version of the ViaBTC mining server can be found. The user agent, version, and service flag match the observed peers. These connections are made using the ViaBTC “bitpeer” module. ViaBTC runs multiple bitpeer instances that connect to the Bitcoin P2P network to propagate ViaBTC’s blocks and feed blocks from other pools into their mining server.

To figure out why the blocks received from ViaBTC’s bitpeer peers don’t have a valid witness reserved value, I started recording the P2P communication between my Bitcoin Core node and the bitpeer clients. Inspecting the transactions in the received blocks showed that all transactions are serialized without the witness data. A SegWit commitment in the coinbase exists. Without witness data, the SegWit commitment in the coinbase transaction can’t be verified as the witness reserved value is missing. The verification fails with the reason: bad-witness-nonce-size.

The following shows the raw coinbase transaction from block 832535 mined by ViaBTC. The witness parts of the transaction, missing from blocks send by ViaBTC’s bitpeer clients, are colored in red. These are the SegWit maker 00 and flag 01, as well as the witness stack size 01, witness element size 20, and the witness reserved value 0000000000000000000000000000000000000000000000000000000000000000.

010000000001010000000000000000000000000000000000000000000000000000000000000000ffffffff620317b40c1b2f5669614254432f4d696e6564206279206b7a616c67736176652f2cfabe6d6d7182edded3ec31023b380b5ecb0cfdb8bc19b7b64b038620edc9f3f0d73115ad100000000000000010cdbc33017bb0ae8f449fc454faf2010000000000ffffffff0327609e26000000001976a914536ffa992491508dca0354e52f32a3a7a679a53a88ac00000000000000002b6a2952534b424c4f434b3a5f520faece8244c1a89cc42225671c5e95eabe95f00186984778b119005d76cd0000000000000000266a24aa21a9ede8f8b5f251110e1c7b7c6f9e4082514e245b6fdb10b06e8285835f8c80eec86c0120000000000000000000000000000000000000000000000000000000000000000000000000

While the ViaBTC bitpeer clients incorrectly broadcast the ViaBTC blocks without witness data, the blocks (with witness data) still somehow propagate through the network. Probably, ViaBTC’s Bitcoin Core nodes announce the blocks as compact blocks to their peers. It’s not clear to me where in the ViaBTC mining server the witness data is lost. In most cases I’ve observed, the block messages from ViaBTC bitpeer instances arrive after the compact blocks³. Thus, fixing this might only slightly improve the block propagation of ViaBTC’s blocks on the Bitcoin network. I’m not sure if the bitpeer clients are required at all to ensure the blocks propagate well.

I let ViaBTC know about this in late February 2024, and they thanked me for informing them. However, I don’t know if they are working on a fix for their mining server.

In the recently merged pull request #29412 an early mutation check for blocks received in a block message was added. The mutation check is now done right after receiving the block before doing any other block validation. If a mutation is detected, we disconnect the peer. Previously, we only checked blocks we hadn’t seen before for mutations. Now, all blocks are checked. This means we also disconnect more aggressively. Starting with the upcoming Bitcoin Core version 27.0, ViaBTC’s bitpeer clients will be disconnected for each mutated block they send.

Here specifically version 26.0 as the block processing will slightly change in version 27.0. ↩︎
The block message is however used during initial block download. ↩︎
Bitcoin Core only logs the error when the bitpeer block message arrives before the compact block message. ↩︎

Long-Term-Support Grant from OpenSats for 2024 and 2025

Thu, 01 Feb 2024 00:00:00 +0000

OpenSats supported my work on Bitcoin in 2024 and 2025 with a Long-Term-Support grant.

Annoucement

Spiral Work-Log Q4 2023

Wed, 31 Jan 2024 00:00:00 +0000

This is a copy of the Q4 2023 work-log I sent to Spiral for my grant.

Disclaimer: Some information that is not (or not yet) meant to be published may have been redacted.

In Q4 of 2023 and January 2024, among other things, I’ve worked on my fork-observer, the miningpool-observer and the peer-observer tools. I’ve also maintained my bitcoin-data related repositories, worked on macOS and FreeBSD support for Bitcoin Core’s tracing framework, and build a tool to find non-standard transactions. I’ve written blog posts about CPU usage of Bitcoin Core peers, about OFAC sanctioned transactions missing from F2Pool blocks, and a post about non-standard transactions being mined.

In more detail in no particular order:

Bitcoin Core CPU timings per relay-type

There is an ongoing effort in Bitcoin Core to increase the number of outbound block-relay-only connections from 2 to 8 with the goal of helping against network partition attacks. However, first the number of inbound slots need to be raised to accommodate more block-relay-only connections. More inbound slots also means higher resource usage. I’ve focused on CPU usage of Bitcoin Core peers, while others looked at memory and bandwidth usage.

I developed a framework to measure per peer CPU usage on a busy mainnet node. By recording the time spent in the SendMessage() and ProcessMessage() functions, it can be estimated how expensive adding more outbound and inbound block-relay-only connections are. As expected, block-relay-only connections are quite cheap. Full results can be found in https://delvingbitcoin.org/t/cpu-usage-of-peers/196/2, and I also left some commentary on the PR approach, which ended up allowing the authors to improve the approach.

fork-observer

When forks, or invalid blocks on the Bitcoin network happen, it’s important to know the involved parties. My fork.observer tool now tries to identify the miner, where possible. This also supports blocks on the default Signet.

Along with general maintenance, I’ve also added a feature that produces an RSS feed for invalid blocks because we saw three of them in 2023.

miningpool-observer

As some mining pools started censoring Inscriptions, I’ve invested some time into being able to detect inscriptions with add: detect ordinal inscription reveals in inputs and added an inscription tag to inscription transactions missing from blocks.

Also, I found it was time to revisit the miningpool-observer transaction package building code. I changed it to closer match Bitcoin Core’s ancestor package building code. This allows for a much cleaner feerate distribution graph.

Mining pool identification data needed to be manually updated now and then. I’ve changed this to automatically update if possible, and otherwise to fall back to older pool identification data. This means less ongoing maintenance work for me.

My mining-pool observer project also found six OFAC-sanctioned transactions missing from blocks. I’ve analyzed them and found that four of these were likely filtered by F2Pool. I’ve published a blog post about this, which was well received. F2Pool admitted the filtering and claimed they’d stop censoring.

peer-observer

I’ve got a request to check my mempool data to see if I’ve seen similar spikes in vByte/second being broadcast on the Bitcoin network as mempool.space did in September 2023. None of my nodes saw these spikes. However, it prompted me to add monitoring for it to my p2p monitoring nodes.

bitcoin-data

My call for more block-arrival timestamp data was mentioned in a Bitcoin Optech newsletter, and I received a few contributions adding more and older timestamps. I’ve also updated the stale-block dataset. Along with the work on improving miner detection in fork-observer and miningpool-observer, I’ve also been updating the mining-pools dataset, adding four new pools.

Bitcoin Core tracing support for macOS and FreeBSD

The current Bitcoin Core tracing framework only supports Linux. I’ve revisited this and found a way of extending this framework to macOS and FreeBSD without requiring new dependencies or similar. I’ve bought an old Mac mini to test this and have a branch where tracing on macOS is working. FreeBSD still needs more work. I plan to PR this against Bitcoin Core soon.

find-non-standard-transactions

I’ve got a request about non-standard transactions and MEV on Bitcoin. I had previously thought about a tool to find non-standard transactions being mined and took a shot at implementing a quick and dirty tool https://github.com/0xB10C/find-non-standard-tx. I’ve used this to check the last 100k blocks and wrote a blog post about it.

Miscellaneous

for fork-observer and miningpool-observer I needed to have an automatically updating source of pool identifications. I modified my rust-bitcoin-pool-identification library to support this.
With the release of Bitcoin Core v26, I contributed my GUIX build signatures for v26, and I’ve helped to review and fixed a bug in the Nix Bitcoin Core package. I’ve also reviewed an update of the btcd nix package, which I maintain.
I’ve decided to discontinue the https://bitcoin-dev.blog and have written about it here.
I quickly hacked together an Ocean.xyz mining pool template fee comparison tool.
Is anyone actually reading these updates? If yes, please send me an 🐘 emoji.
Responsibly disclosed a critical security vulnerability to a company in the Bitcoin space. Will publish more details sometime in the future.
I helped Fabian Jahr on his Bitcoin Core asmap work by running and reviewing the results of his new asmap tools: https://github.com/fjahr/asmap-data/issues/4
After the recent btcd vulnerability disclosure, I added up a few btcd nodes to my monitoring and detected that the vulnerability had been exploited on testnet a day after being published.

An overview of recent non-standard Bitcoin transactions

Mon, 29 Jan 2024 00:00:00 +0000

This blog post provides an overview of non-standard transactions that mining pools included in the last 117000 Bitcoin blocks.

Usually, before a Bitcoin transaction is included in the blockchain, it is relayed between nodes on the Bitcoin P2P network and cached in their mempools. To mitigate some transaction relay Denial-of-Service attacks, the Bitcoin Core node limits what transactions can be relayed. Transactions deemed as “standard” are relayed, while “non-standard” transactions are not relayed. In Bitcoin Core, these rules are known as the policy rules. Mining pools can receive non-standard transactions out-of-band or loosen the relay rules to accept them over the P2P network. Non-standard transactions can be, and frequently are, mined.

This post provides an overview of recent non-standard transactions included by mining pools. Frequent non-standard transactions might show policy rules that need to be adapted or changed, while ensuring no Denial-of-Service attacks are made possible. Having an overview of which non-standard transactions are mined and who mines them can serve as basis for future policy design. Additionally, vulnerabilities in Bitcoin software came to light, which require mining non-standard transactions to be exploited. Some of these vulnerabilities have been exploited, while others have been responsibly disclosed. Some mining pools try to extract more value from blocks than just the block reward by including non-standard transaction. This is generally known as Miner Extractable Value (MEV). A large mining pool might be able to extract more value by investing heavily into their transaction selection software to out-compete smaller mining pools. This causes further mining pool centralization.

Methodology

To detect non-standard transactions, two nodes are used. A data-node and a test-node. The data-node has access to the full blocks that should be checked¹. The test-node is synced to a height just below the test-start-height. It can be a pruned node. Now, beginning from the test-start-height, blocks are requested from the data-node. For each transaction in a block, excluding the coinbase transaction, we test if the transaction is accepted to the test-node’s mempool using the testmempoolaccept RPC call. If the transaction is rejected from the test-node’s mempool, then the transaction is non-standard. The rejection reason is recorded. If the transaction is accepted, it is submitted to the test-node with the sendrawtransaction RPC call. This makes sure child transactions aren’t rejected due to parents from the same block not being available. Once all transactions have been tested, the block is submitted to the test-node with the submitblock RPC call. This clears the mempool. The next block is tested until the chain tip is reached. This methodology is implemented in the quick and dirty tool github.com/0xB10C/find-non-standard-tx.

While this methodology reliably detects non-standard transactions, there are a few limitations. For example, there is only one reject reason returned for a transaction even if the transaction would be rejected for multiple reasons. Furthermore, descendants of rejected transactions can’t be added to the mempool as their inputs are missing. These have to be analyzed manually. The ordering in the block might not represent the order in which transactions entered the mempool. Usually, this isn’t a problem. However, for the CPFP carve-out rule to be applicable, the 25th descendant has to be added last. This results in one transaction being rejected with an “too-long-mempool-chain” when the CPFP carve-out rule was used (but also allows detecting and measuring the use of CPFP carve-out).

Non-standard transactions by rejection reason

Here, a start height of 710575 (2021-11-20) and an end height of 827622 (2024-01-27) was used, which results in about 117k blocks checked for non-standard transactions. The start height was chosen to be shorty after Taproot activation. A Bitcoin Core v26.0 node was used as test-node. The following overview does not cover all 20k non-standard transaction detected. Rather, it picks out a few interesting examples and tries to highlight the reasoning behind these non-standard transactions, where possible.

“bad-txns-nonstandard-inputs”

In January 2023, between block 771609 and block 773957, MaraPool included 16 non-standard transactions in their blocks. These were all rejected with the reason: bad-txns-nonstandard-inputs. In this case, the redeemScripts in the inputs have more sigops than allowed. This is related to Bitcoin Core PR #26348 by Sergio Demian Lerner, where he describes that the RSK sidechain migrated to a non-standard P2SH redeemscript with now more than 3000 BTC stuck on there. MaraPool helped RSK by mining the non-standard transactions. Sergio noted, that their tests on testnet passed. At the time, non-standard transactions were allowed by default on testnet. This was later changed in the Bitcoin Core PR #28354.

“tx-size”

As a protection against excessive resource usage, Bitcoin Core does not relay and rejects transactions larger than 100 kvB (400 kWU; MAX_STANDARD_TX_WEIGHT). These are rejected with the reason “tx-size”.

The Luxor pool mined the 985 kvB (3.94 MWU), zero-fee transaction 0301e0480b374b32851a9462db29dc19fe830a7f7d7a88b81612b9d42099c0ae 0301e0480b in block 774628 (2023-02-01). This transaction inscribes an 1803 × 1803 pixel JPEG showing a taproot wizard with a size of 3.9 MB and fills nearly the entire block.

Block 776884, mined by Terra Pool, includes the 850 kvB transaction b5a7e05f28d00e4a791759ad7b6bd6799d856693293ceeaad9b0bb93c8851f7f b5a7e05f28 , paying nearly 0.5 BTC in fees. The transaction inscribes a 1-minute MP4 music video showing a frog holding a drink. In block 777945, the ‎975 kvB transaction 79b91e594c03c8f06d70c44a288a88a413c540abca007829ca119686a7f979da 79b91e594c pays a fee of nearly 0.75 BTC to inscribe a 4000×5999 pixel WEBP image. The 992 kvB transaction 4af9047d8b4b6ffffaa5c74ee36d0506a6741ba6fc6b39fe20e4e08df799cf99 4af9047d8b , mined in block 786501, pays nearly 0.5 BTC in fees to inscribe a 2040×2653 pixel JPEG image of a Bitcoin Magazine cover with the face of Julian Assange.

F2Pool frequently includes larger-than-standard transaction in their blocks. These are mainly zero-fee mining pool payout transactions with one input and more than 3000 outputs. Examples are b460f758881bd1dd03da99ce6fb465637cbdcf6c6bb4664a4fedaf18d033e57f b460f75888 , 56c5b34eaca34967bc22dad74887339d94c620a6e33f3294e8ad2fbd24baa45e 56c5b34eac , and 8239fbb0da35b5f6bbbcff2a0c2b00d7b2f21efb47387bce6816b42e7fc3e61f 8239fbb0da .

There are two F2Pool transactions that are not pool payouts and stand out. The zero-fee transaction 7884712aae8685406c3b8f264fca151d76fc20ed3071d21275a6b6fc8dc2a82a 7884712aae has 800 inputs and 800 outputs, with a size just above 152 kvB. All inputs and outputs are 546 sat. This transaction moves F2Pool’s “uncommon sats”² to a different address. Transaction 73be398c4bdc43709db7398106609eea2a7841aaf3a4fa2000dc18184faa2a7e 73be398c4b has a size of 125 kvB and pays the complete input amount of 0.03682719 BTC ($750 USD at the time) as fees. The only output is an OP_RETURN output with the message “you'll run cln. and you'll be happy.”. This transaction exploited a consensus bug in btcd causing, for example, LND, to stop processing new blocks. This transaction was constructed by brqgoo.

“non-mandatory-script-verify-flag”

The transaction 00c1af8f7d9e2bfe854d1718db078eb6740920f702bc00d4a9f205d3a084c40e 00c1af8f7d is non-standard with the reason “non-mandatory-script-verify-flag (OP_SUCCESSx reserved for soft-fork upgrades)”. This transaction has a size of 104 vB and pays a fee of 5000 sat. It was also constructed by brqgoo and, similar to the previous transaction, also includes an OP_SUCCESS. It was, however, mined before the transaction that exploited the consensus bug in LND and might have been a test transaction. The only output is an OP_RETURN with a 32 byte value, which could be a hash commitment to the later exploited vulnerability.

possible hash commitment: 5d86c2f206e5d9eb8c797035039ffac92ed94f794f9dab764be0ba40491dec47

“scriptpubkey” & “multi-op-return”

In block 826775, F2Pool mined transaction c3dd9ae8b5e9811514ef15238e0de96db8cf2f17ea4bdc41942e8c591c961a25 c3dd9ae8b5 , which included a 100 byte OP_RETURN message. Similarly, in block 826796, F2Pool mined transaction ee8bbff798c5b07cfdf0a7379fb8ef2e05d91fcfac2daacb21de30c1f007bf79 ee8bbff798 with a 686 byte OP_RETURN message, in block 827419 17ec5d70668578d7e572eb9f821741759d1fe4ee4d6ca82ce35cb051d54bcf20 17ec5d7066 and 2c869583a8dc9109ac60760c9cfbfdf25e34416a0e4ad858f096a5a38e1eedda 2c869583a8 with 111 byte and 400 byte OP_RETURN messages. OP_RETURN is, by default, limited to 80 bytes of data. These transactions were rejected with the reason “scriptpubkey”. F2Pool also mined the transactions cb22f12d777e95e426fd31840171af453c600390276d29d307ef34dbaeda4387 cb22f12d77 and 6f660e9f1bbfcc8435593eb8ff70a501275ad6cdbaa536747bd9d55bdbeda65a 6f660e9f1b , which have five and two OP_RETURN outputs. As, by default, only one OP_RETURN output is allowed, these are non-standard with the reason “multi-op-return”.

While these transactions are non-standard when using Bitcoin Core, they are standard when using Peter Todd’s “Libre Relay” patch. In particular, these changes allow large OP_RETURN messages and multi-OP_RETURN transactions. F2Pool might be running the “Libre Relay” patches.

While writing this blog post, more non-standard “scriptpubkey” transactions were mined by F2Pool. For example, the 3.22 kvB transaction 8c9fe58186c199fa9f8d72c121a45270f70d3854e67238f3c7c319989a50921b 8c9fe58186 encodes a PNG image and 33a271fd7782754e3a73fd1b791be920898317f73b1c77cf6b94d9d718bce701 33a271fd77 a GIF. For more transactions, see this tweet thread.

“min relay fee not met”

By default, Bitcoin Core has a minimum relay fee of 1 sat/vByte. Transactions below this feerate are rejected and not relayed to other nodes. These are often sent to the mining pool out-of-band. Some mining pools might opt to mine their own zero-fee transactions. By creating zero-fee transactions and using, for example, the prioritisetransaction RPC to assign a virtual fee, they avoid their transaction from being relayed but still can get it accepted into their mempool. If they construct a transaction that pays a fee, they risk that it’s relayed to another pool and pays a competing mining pool. Additionally, depending on the mining pool’s payout and fee structure, including their own zero-fee transactions might shift the transaction fee costs to their miners. As the pool’s zero-fee transaction takes up block space, it reduces the transaction fees earned by their miners. The miners might not be aware of being exploited by the pool.

ECMD Pool mined b1360f73a7a990d5210ac43482ec4de83b76c3932522702dca6b9bb5f6468261 b1360f73a7 paying 7400 sat at 14 kvB resulting in a 0.52 sat/vB feerate. This transaction consolidates 96 inputs to the sending address and sends 43k USDT (Omnilayer) to Huobi³ in a nearly-dust output. The consolidated inputs don’t seem to stem from EMCD Pool themself. It’s unclear why the transaction did not pay more than the minimum relay fee of 1 sat/vByte. The output of the transaction was spent in the same block with a feerate of 89.1 sat/vB⁴. While this results in an effective feerate of over 2 sat/vB through CPFP, the first transaction would not have been relayed.

Transaction 9ffad0add040094e7d61dd021a2750eebd034042467b0ce0dbc3f4a4b3aac07d 9ffad0add0 , paying a feerate of 0.43 sat/vB with a fee of 7701 sat and a size of 17 kvB, was mined by SBICrypto. It consolidates 100 inputs into a single output. The addresses belong to the freebitco.in entity³. Why freebitco.in constructed a transaction paying below the minimum relay feerate is not known.

The transaction 4da9b2ab358a76c0dd9067b91f07078e25b77c82439fe58450ae206c3deb4464 4da9b2ab35 was mined by an unidentified pool⁵. It does not pay fees and script-path-spends a 2-of-2 multisig P2TR output. The parent transaction creates an OP_RETURN output with a valid but unused bech32 address bc1qypz2ynlgy3dmpqxxvap2s30wjfqcp24tk46p83. It’s unknown why the transaction does not pay fees.

The mining pool Terra Pool mined the zero-fee transaction 1a6466600e05ccf5b63ded29d4a1a067ce33b79cd1252754ead733f21fae1775 1a6466600e in block 811775 creating a 10k sat output. The output was directly consumed by the next, also a zero-fee, transaction f9aa65bb55baa3e2ba95deac89afc99d28934cdb613e72abe20b1f25a0c848ee f9aa65bb55 in the same block. It inscribes a “Clean Bitcoin Certificate”. In a block mined by a “Clean Incentive” pool, the transaction d2a769546e9182d8b23765850e3ef605eec48fe641822bcce86594417072c96b d2a769546e creates an additional 10k sat output. This output, and the previously inscribed certificate, are then consumed by another zero-fee transaction, ed8819d17dbd6ab0d934b24d1e6dadd6bcd7b91866a9a235a081576421c68022 ed8819d17d , which inscribes an accompanying JSON object with more information about the certificate. This is later moved in yet another zero-fee transaction 7cc18418b814a1e9fe731e630e638201c36bf45e1f20aa084e0182a43cdc8174 7cc18418b8 , where a collection of inscriptions is inscribed.

The same pool also mined the zero-fee transaction 699e9a96320465223172d739e0fb8723c5951314e4aba31d9ed7d1703b5b07d0 699e9a9632 , which has one input and one output. It spends a 1-of-2 P2SH output created in the same block. The 1-of-2 multisig is composed of two public keys with well-known private keys. The private keys are 1 and 2⁶. The transaction creating the multisig output has a feerate of 3.9 sat/vB and is the first transaction in the block after the coinbase. As the remaining transactions in the block pay a significantly higher feerate, this is an indication the transaction was manually prioritized by the pool. The pool might run a tool that checking for UTXOs spendable with well-known private keys, and tries to spend and mine them before someone else does ⁷.

MaraPool mined the zero-fee transaction 0026faeccb25b9837ccf9d8e7b88f676669967263bcd8d5de1b55fedb08a328d 0026faeccb , which script-path-spends a single P2TR output from a depth-2 merkle tree, revealing a 1-of-1 multisig. The parent transaction has an OP_RETURN output with the data cc1qjdexdhgqu95zfkvw20vfn3uh9r6s563y86qpsd. It’s unclear why MaraPool mined this zero-fee transaction.

F2Pool frequently mines zero-fee transactions. These are mainly consolidations of their coinbase outputs and pool payout transactions⁸. An example for a coinbase consolidation is transaction 11887bfc724d3486b7da3e70cf13cc992f16e25273b3a685a70a7890957310e5 11887bfc72 . It consolidates 21 inputs from 1KFHE7w8BhaENAswwryaoccDb6qcT6DbYY, the well-known F2Pool coinbase output address, into two outputs paying no fees.

However, it seems F2Pool is also using zero-fee transactions for payments to Huobi (44335aef14) and Binance (a5c062118a). While these are small, one-input two-output transaction and don’t take up much block space, they reduce the fees earned in the block and the payout distributed to F2Pool’s miners.

In block 730459, F2Pool mined the zero-fee, one-input one-output transaction 26a9e3872fadcee3144aae6629e887500d33813217b5447d2fb8e5ae6e5e0329 26a9e3872f , which moved 3900 BTC from 1J1F3U7gHrCjsEsRimDJ3oYBiV24wA8FuV to the same address. While this seems pointless at first, one explain could be that there existed an alternative, (pre-) signed transaction spending this UTXO. To invalidate this transaction, the UTXO was spent. The transactions 03e43f5e2877a48c253cd9e05be63805f2717c47bf1bc7c1b210e43309e6899c 03e43f5e28 and 68e129eb0a00cf8d9717e33ba213c9a9849989b9ccfe57b6ea0275c38d2f8cf0 68e129eb0a are similar occurrences.

F2Pool also transferred it’s “uncommon sats”² to another address in the‎38 kvB transaction 9f62795a4766409d7e32e2ca6a9e3565284cdbaa31a25687249e40cd80671049 9f62795a47 . This transaction has 200 inputs and 200 outputs worth 546 sat each. F2Pool sells “uncommon sats”², starting at 450 USD for a single 546 sat UTXO⁷.

“dust”

Bitcoin Core treats low value outputs as non-standard or “dust”, when the output value is lower than the cost of creating and spending it. Dust UTXOs aren’t economical to spend and pollutes the UTXO set. F2Pool is the mining pool that frequently mines transactions, creating “dust” outputs.

In the Stacks 2.0 mining protocol, stacks-miners try to get their commit transaction into the next block by paying a high fee. In their commit transaction, they include, as a proof-of-transfer, two outputs to two other stack miners. The higher the (bitcoin) amount of these outputs, compared to the other stack-miners competing for the same block, the higher the probability to mine the Stacks block and be rewarded. There is no minimal proof-of-transfer amount besides the Bitcoin dust limit. F2Pool is using their position as a mining pool to their advantage⁷. They filter all stacks block commits broadcast by other stacks-miners. This allows them to include only their own commit transaction in their block. In this zero-fee transaction, they pay dust outputs to two other stack-miners. This allows them to have a 100% probability of being rewarded while only having to spend a few sats.

F2Pool creates other dust outputs too. In the zero-fee transaction 638d8cf6da03eef9ccfdd6505782be98bb7c3b909dc650eef32a910dd0d74c2f 638d8cf6da , F2Pool splits their “uncommon sats”² from some of their coinbase payout outputs into separate UTXOs. While these separate UTXOs all have a value of 546 sat, there is one UTXO created with a value of 330 sat, which makes it a dust output. On the output, according to the ordinal theory, a portrait of a frog in a suit is inscribed. This inscription was inscribed about 40k blocks before F2Pool split these out. The inscription somehow made its way to one of F2Pool’s addresses.

F2Pool also mined the one-input two-output transaction d82e322d9dc3ccd7b69450cc29c42b9da2576aa4184e4d4620f45bb9e269ff27 d82e322d9d , which creates a 250 sat dust output. It’s unclear why this transaction was included by F2Pool. The transaction 38086f6079c9eeb1e1a637600645e99982281f5f8ee23dd9680d879b9e7da204 38086f6079 creates two dust outputs and was sent to F2Pool out-of-band. The story behind this transaction can be found on Twitter.

F2Pool mined the one-input and one-output transaction 2814d0a3c9e6dd5b88911a6280fc3899391f5c47072eb11593af2838160fad2f 2814d0a3c9 , paying a fee of 10k sat. The output has a value of zero, which is below the dust limit. This zero-value output is spent in transaction c1e0db6368a43f5589352ed44aa1ff9af33410e4a9fd9be0f6ac42d9e4117151 c1e0db6368 , which inscribes a text inscription you will use soma and you will like it and creates another zero-output. An inscription on a zero-value output broke the ordinals index. These transactions were created by supertestnet, who later released his code in a tool called breaker-of-jpegs.

There are plenty of other cases where F2Pool included transactions that create outputs below the dust limit. This could indicate that they use a lower dust limit than what’s set by default.

“too-long-mempool-chain”

By default, Bitcoin Core limits the number of in-mempool descendants a transaction can have to less than 25. An exception is the CPFP carve out rule, which allows a single, small child of the top most ancestor transaction. When these limits are exceeded, transactions are rejected with the “too-long-mempool-chain” reason. These transactions are usually standard on their own but aren’t accepted into the mempool at the same time.

In block 788839, ViaBTC included multiple ancestry sets with more than 200 transactions in total. These would have been rejected from a default mempool. These sets include the transactions 6cd3e204ae299952e30acf4a7c5c337a6a0310b5a58e081199b2d17c7aec504e 6cd3e204ae and 174f1695d537fd21455b3a195a3dac6c430c0060b986ecc8bea5fa8c6d32ef09 174f1695d5 . Both ancestry sets are related to minting BRC-20 tokens.

In block 789149, AntPool mined an ancestor transaction A d4431ed437bce8de22c1c134c6cd9427f5eb08e7b401d7152be936fed58848d0 d4431ed437 with 28 descendants as part of an BRC-20 token mint. With default Bitcoin Core rules, only 24 children would have been allowed.

‘A’ is an ancestor transaction with 28 descendants. The arrow direction shows a ‘is-parent-of’ relationship between the transactions.

In block 802625, F2Pool mined an ancestor A 1a4fa7cadc072f1937da9857531950069776e2742c7b15b04f3ef9292a46d50b 1a4fa7cadc with two descendants. The children are rejected from a default mempool with the reason “too-long-mempool-chain”. Here, the ancestry set is clearly under the limit of 25 transactions. However, the ancestor A has a size of 52 kvB, child 1 a size of 60 kvB, and 2 a size of‎70 kvB. The default Bitcoin Core configuration only allows an ancestry set to have a maximum size of 101 kvB (DEFAULT_ANCESTOR_SIZE_LIMIT_KVB). Transaction A and child 1 together already exceed this limit. These transactions seem related to the KuCoin exchange³.

‘A’ is an ancestor transaction with two descendants. The arrow direction shows a ‘is-parent-of’ relationship between the transactions.

As mentioned in the methodology limitations, occurrences of CPFP carve out are incorrectly rejected with “too-long-mempool-chain”. The ordering in the block is different from the valid mempool submission order. An example is 0b9900440d01600e0589a855fb9d9393860b65a3e0f76a8fc8915385d43ae3e9 0b9900440d mined by Binance Pool. This ancestor transaction A has a child transaction 1. Transaction 1 has transaction 2 through 24 as children. This is the maximum number of allowed children. Transaction 25 is a direct child of the ancestor A and has a size smaller than 1000 vB, which makes it a valid CPFP carve out. Future work could closer analyze how the CPFP carve out rule is being used on the network.

‘A’ is an ancestor transaction with 25 descendants. Transaction ‘25’ is an example of a CPFP carve out. Here, arrow direction shows a ‘is-parent-of’ relationship between the transactions.

Conclusion

Non-standard transactions are frequently included by mining pools. While some pools occasionally help community members by, for example, mining UTXOs stuck on non-standard scripts, other pools use their position as pool to extract more value from blocks. The F2Pool mining pool is clearly the pool having the least strict standardness rules and, from time to time, they accept to mine transactions that exploit vulnerabilities in Bitcoin related software. The recent non-standard transactions don’t immediately show standardness rules that need to be changed.

An alternative to a data node could be a block explorer API serving the full blocks. ↩︎
https://docs.ordinals.com/overview.html ↩︎ ↩︎ ↩︎ ↩︎
According to OXT.me. ↩︎ ↩︎ ↩︎
And it tries to double spent the 43k USDT. https://omniexplorer.info/search/05abfed93a56683ac909e1e0142c0f90f1180e04ba42b18841abcc7c820809d0 ↩︎
https://github.com/bitcoin-data/mining-pools/issues/102 ↩︎
The private key 1 corresponds to the address 1BgGZ9tcN4rm9KBzDn7KprQz87SZ26SAMH and the private key 2 to the address 1cMh228HTCiwS8ZsaakH8A8wze1JR5ZsP. The particular P2SH address used here is 38fEX6RbBBMmpu3nbbuULku1xyrrzqqqnE. This is a 1-of-2 and a 1-or-2 multisig! ↩︎
This is MEV. ↩︎ ↩︎ ↩︎
These are usually too large and listed under the tx-size section. ↩︎

CPU usage of Bitcoin Core peers

Wed, 29 Nov 2023 00:00:00 +0000

To help improve partition resistance, a medium-term goal is to increase the number block relay connections a Bitcoin Core node has (see #28462). However, how much resources do block-relay connections use? Surely they are cheaper than full-relay connections? This blog-post focuses on the CPU usage of Bitcoin Core peers.

I’ve looked into this for CPU usage of peers - delvingbitcoin.org and PR #28463. This post is cross-posted from my answer on delvingbitcoin.org.

Methodology

To measure CPU usage of peers in Bitcoin Core, I’ve opted to measure the time we spent in ProcessMessages() and SendMessages() per peer. I’ve added a tracepoint to the end of both functions and pass, for example, the peer id, the connection type, and function duration. I hook into these tracepoints with a simple bpftrace script that prints each SendMessages() and ProcessMessages() function call as CSV-formatted row. See this commit.

Setup

The underlying node is hosted on a VPS of a large cloud provider and has access to four AMD EPYC 7R32 cores (the message handling thread of Bitcoin Core is single-threaded and only uses one of these). The IP address of the node is well-known in the network. The node is pruned, which means no one was doing an IBD from the node. Other than pruning and debug logging enabled, the node is running with default parameters. The timing measurements posted here were all taken with all inbound slots filled (the per-peer measurements with about half-full inbound slots were similar). I’ve only looked at connections that that ended up being connected for more than a minute. This means, the data doesn’t cover short-lived feeler connections we made or received, and it doesn’t cover short-lived spy node connections that were evicted after a few seconds.

I’ve repeated these measurements on weekdays and a weekend in early November 2023. The resulting numbers differ slightly. This is likely related to, for example:

transaction broadcast rate on the network ⇾ with more transactions being broadcast, we spent more time validating transactions, which is expensive (see below)
number of inbound full-relay vs inbound block-relay-only connections ⇾ inbound full-relay are more expensive than inbound block-relay-only connections (see below)
…

Total time spent sending and processing messages

To start, this is the time spent per second in SendMessages() and ProcessMessages() summed up for all peers.

On November 4th and 5th, the weekend before, it averaged at around 56ms per second.

Weekend: Total time spent (by all connections) sending and processing messages per second?

On November 7th, a Tuesday, this averaged at about 32ms per second with 125 connections. This is on average about 17ms per second processing messages and 15ms per second sending messages.

November 7th: Total time spent (by all connections) sending and processing messages per second?

There were short periods with the total time per second reaching nearly 1000ms per second. This equals to 100% usage of one CPU core.

Per-peer time spent sending and processing messages

Looking at individual peers by connection direction and connection type shows which connections are cheaper and which are more expensive. I assume that an inbound connection sending me a version message with the f_realy flag set to false is an outbound block-relay-only connection by the peer. While -blocksonly nodes have the same fingerprint (link), I assume that these are rare and only marginally affect the numbers.

November 4th and 5th:

Weekend: On average, how long does a single connection spent in Send- and ProcessMessages() per second by connection and relay type?

November 7th:

Weekend: On average, how long does a single connection spent in Send- and ProcessMessages() per second by connection and relay type?

connection type	mean 4th+5th ㅤㅤㅤ	mean 7th ㅤㅤㅤㅤ	stdev 4th+5th ㅤㅤㅤ	stdev 7th
outbound full-relay	661.77µs	611.63µs	1378.43µs	2596.95µs
inbound full-relay	457.81µs	271.72µs	880.94µs	1061.78µs
outbound block-relay-onlyㅤ	94.62µs	86.14µs	24.67µs	158.18µs
inbound block-relay-only	96.84µs	84.34µs	77.94µs	76.31µs

ㅤ

The connections spend slightly less time on average on the 7th, but had a higher standard deviation. Likely related to differences in messages relayed on the network, however, I haven’t looked deeper into it.

Outbound full-relay connections are the most expensive connections here, taking more than 600µs per second on average. We currently only ever make 8 of these at a time. Inbound full-relay connections are cheaper than outbound full-relay connections, but still expensive compared to block-relay connections. However, we accept up to 114 inbound full-relay connections (typically about 91 due to some being block-relay-only, see also #28463 (comment)). Inbound and outbound block-relay-only connections spent just under 100µs sending and processing messages per second on average. These are the cheapest connections.

Time spent processing messages by relay type and connection direction

Since ProcessMessages() only ever processes one message at a time, we can measure the processing time by received message. SendMessages() might send zero, one, or multiple messages when called, which makes the same measurement harder.

November 7th:

November 7th: Boxen plot of time taken to process a received message by relay-type.

tx, addr, and addrv2 messages are only received and processed by full-relay peers. Especially tx messages are expensive with close to 0.5ms in median. While I received a few inv and getdata messages from block-relay-only peers, the majority stems from full-relay peers. Additionally, during this time-frame, all cmpctblock messages were received by full-relay peers.

Inbound version messages take slightly shorter to process for block-relay-only connections than for full-relay. @amiti suggested this might be related to initializing the data structures for transaction relay?

Learnings

On a modern server CPU core (AMD EPYC 7R32), Bitcoin Core usually spends less than 100ms (10%) of the time in the (single-thread) message handling thread with full inbound slots.
Very roughly, an outbound full-relay connection (~600µs per second) has about 6x the CPU usage of an outbound block-relay-only connection (~100µs per second). An inbound full-relay connection (~300µs per second) has 3x the CPU usage of an inbound block-relay-only connection (~100µs per second).
As to be expected: Time spent processing and sending messages per second is lower for block-relay-only connections compared to full-relay. The block-relay-only connections don’t process and send transactions. On the processing side, transaction relay is more expensive than address relay.

Increasing number of block-relay-only slots and connections

#28463 proposes to increase the number of inbound connection slots for block-relay-only connections. Currently, a node has about 91 full-relay and 23 block-relay-only inbound connections (80% and 20% of 114). As currently proposed, the PR increases this to about 113 full-relay and 75 block-relay-only connections (60% and 40% of 189 = 200 - 2 - 1 - 8).

Assuming 600µs for a outbound full-relay connection, 300µs for an inbound full-relay and 100µs for a block-relay connection, currently we are at 34.6ms per second and will be at 46.4ms (increase of 34%) with the proposed change. 6.6ms more due to the new full-relay connections and 5.2ms due to the block-relay-only connections. While spending 46.4ms per second in the message handling thread is probably fine, a more conservative change might be to leave the number of full-relay inbound slots largely untouched. Here, RAM and bandwidth usage should be considered too.

currently: $$ 8 \times 600µs + 2 \times 100µs + 91 \times 300µs + 23 \times 100µs = 34600µs = 34.6ms $$ as proposed: $$ 8 \times 600µs + 2 \times 100µs + 113 \times 300µs + 75 \times 100µs = 46400µs = 46.4ms $$

Since later increasing from 2 outbound block-relay connections to 8 as proposed in #28462 is only an increase of $ 6 \times 100µs $, I don’t see a problem with this from the CPU usage side.

Some notes

It would also be useful to have these measurements for Erlay. Maybe Erlay makes transaction relay a lot cheaper (or a lot more expensive?).
These measurements were made on a server CPU core, where a 34% increase is only about 11ms per second. However, the numbers might look very different on a nearly 70% slower Raspberry Pi 4 BCM2711 Core (see this comparison).
Since my node is pruned, this does not include IBD data, which might raise the average time spent in SendMessages() (for both full-relay and block-relay connections).
Time spent in function isn’t a perfect measurement of CPU usage. For example, when sending requested blocks, a big chunk of the time might be spent waiting on disk IO.

Thanks to the MIT DCI for sponsoring the node I’ve used to measure this (and five more for related purposes!).

Six OFAC-sanctioned transactions missing

Mon, 20 Nov 2023 00:00:00 +0000

My project, miningpool-observer, aims to detect when Bitcoin mining pools are not mining transactions they could have been mining. Over the past few weeks, it detected six missing transactions spending from OFAC-sanctioned addresses. This post examines whether these transactions were intentionally filtered because they spent from OFAC-sanctioned addresses or if there are other possible explanations for these transactions to be missing from blocks. I conclude that four out of six transactions were likely filtered.

In September and October 2023, the RSS feed of my miningpool-observer instance reported six blocks missing an OFAC-sanctioned transaction. One block was mined by the ViaBTC mining pool, another by the Foundry USA pool, and four by F2Pool. An OFAC-sanctioned transaction is a transaction spending from or paying to an address sanctioned by the US Department of the Treasury’s Office of Foreign Assets Control. I maintain a tool to extract a list of OFAC-sanctioned addresses from the Specially Designated Nationals (SDN) list published by the OFAC.

Several reasons could explain why a transaction might be absent from a block. Generally, transactions do not propagate equally through the network, and there is no global mempool to pick transactions from. Each node has its own set of valid transactions. A pool might also prioritize transactions for which it received an out-of-band payment. However, it might also deprioritize or filter transactions.

The goal here is to determine if the mining pool filtered any of these six OFAC-sanctioned transactions or if there are other possible explanations for their absence from the block¹. Note that mining pools are free to choose which transactions to include and which to leave out. However, to analyze Bitcoin’s censorship-resistant properties, it’s crucial to understand which pools and how many of them are filtering transactions.

I conclude that the reports from miningpool-observer indicating sanctioned transactions missing from blocks by ViaBTC and Foundry are likely false-positives and not the result of filtering. The transactions missing from F2Pool’s blocks are, however, likely filtered. All missing transactions have been picked up by other miners ².

Block 808660 by ViaBTC

Block 808660 ..866c79c5³, mined by ViaBTC on September 21, 2023, did not include transaction 262025e7..⁴. This transaction consolidates 100 inputs into one output. One of these inputs spends an output paid to 1ECeZBxCVJ8Wm2JSN3Cyc6rge2gnvD3W5K. This address was added to the OFAC’s SDN list on September 21st, 2021.

The transaction has a size of 14.7 kvB and pays a feerate of 25.18 sat/vByte. The output spent from the sanctioned address is 0.0002 BTC (20k sat) and was only created a day before. When ViaBTC mined block 808660, the transaction had been in my node’s mempool for about 75 minutes. It did not have any dependency on in-mempool transactions.

Feerate distribution of the block and the template for block 808660. Screenshot from miningpool.observer.

Examining the feerate distribution of block 808660 on miningpool.observer, reveals that ViaBTC occupied about 1 MWU worth of block space, of a total of 4 MWU, with prioritized transactions. These likely stem from the ViaBTC Bitcoin Transaction Accelerator. Prioritizing some transactions means that lower feerate transactions, such as the transaction spending from the sanctioned address here, don’t make it into the block. For this ViaBTC block, my miningpool-observer instance lists 24 large consolidation transactions at the end of the template that didn’t make it into this block.

List of large, missing consolidation transactions from block 808660

This leads to the conclusion that ViaBTC did not filter this transaction. It got displaced by other, prioritized transactions. This is supported by the fact that, three days later, ViaBTC mined a transaction ⁵ spending an output from the same sanctioned address in block 809181.

Block 813231 by Foundry USA

Block 813231 ..0a8528b6⁶, mined by Foundry USA on October 21st, 2023, did not include the transaction c9b57191..⁷. This transaction consolidates 150 inputs into one output. One of the inputs spends an output paid to 3PKiHs4GY4rFg8dpppNVPXGPqMX6K2cBML⁸. This address was added to the OFAC’s SDN list on April 14th, 2023.

As most of the 150 inputs are 2-of-3 multisig P2SH scripts, the missing transaction is large at 43842 vByte. It pays a feerate of 5.09 sat/vByte and had no dependencies on in-mempool transactions. This feerate was enough to place it at position 161 out of 2215 transactions in the template constructed by my Bitcoin Core node. However, this transaction, along with 18 other transactions, had only been in my mempool for around 30 seconds when I learned about block 812331 by Foundry USA. This makes it likely that Foundry didn’t have a chance to include the transaction in their block because they didn’t know about it yet.

It might take a few seconds for the transaction to propagate. Additionally, most pools only push new block templates to miners every 30 seconds, which then take a while to switch to the new job. Furthermore, the miningpool-observer tool requests new block templates every few seconds and does best-effort matching based on the minimum difference in missing and extra transactions (see Methodology in the FAQ). This makes false-positives for young transactions, probably up to around 60 seconds, possible.

The mempool.space block explorer also keeps track of differences between block templates and the final blocks broadcast by miners. They show that c9b57191.. is included in their template but missing from the actual block. The transaction is tagged “recently broadcasted” by them too.

Transaction missing from block 813231 by Foundry considered as ‘recently broadcast’ by mempool.space.

This leads to the conclusion that Foundry USA did not filter this transaction. The transaction was broadcast too late to be included in the mining job that ended up finding block 813231. In addition, Foundry USA also mined the next block at height 813232 and included the sanctioned transaction there.

Blocks 810727, 811791, 811920, and 813357 by F2Pool

F2Pool mined the block 810727 ..ccda1498 on October 5th, 2023, the blocks 811791 ..af4453d6 and 811920 ..00badf62 on October 12th, and the block 813357 ..63ac1669 on October 22nd ⁹. Each block is missing one sanctioned transaction¹⁰. Each of these transactions consolidates 150 2-of-3 multisig inputs into one output. For each transaction, one of the inputs spends an output paid to 3PKiHs4GY4rFg8dpppNVPXGPqMX6K2cBML. This is the same consolidation pattern and address as discussed in the previous section. None of the missing transactions had a dependency on in-mempool transactions.

Block 810727

In block 810727, F2Pool did not include the transaction c6a66836..¹⁰, which spends a sanctioned output. Due to the 150 2-of-3 multisig inputs, the transaction is rather large at 44017 vBytes. It pays a fee of 446260 sat and had been in my node’s mempool for nearly 4 hours when F2Pool mined block 810727. Instead of c6a66836.., F2Pool chose to include the transaction 907e1f45..¹¹. This transaction is also a consolidation transaction with 150 inputs and one output, but does not spend from a sanctioned output. It pays the same fee of 446260 sat but happens to be 3 vByte larger¹² at 44020 vByte. This means the missing transaction c6a66836.. has a slightly higher feerate than 907e1f45... When strictly sorting by feerate, the missing transaction should have been included. However, in practice, it’s unlikely that 3 vByte of additional block space will make a difference in the total fees in the block.

Comparison of the sanctioned transaction missing from F2Pool’s block 810727 to the extra transaction included. The extra transaction is 3 vBytes larger.

Block 811791

The sanctioned transaction aa001ce6..¹⁰ is missing from F2Pools block 811791. Similar to the previous consolidation transactions, this transaction has a size of 42459 vBytes (169836 WU). With a fee of 446260 sat it pays a feerate of 10.5 sat/vByte. When block 811791 arrived at the miningpool-observer node, the transaction had been in its mempool for four minutes.

In this block, it’s notable that five transactions with OP_RETURN Stacks block commitments are missing. F2Pool has, however, inserted their own Stacks block commitment. This happens regularly and has been reported before. Additionally, F2Pool is including two large, zero-fee transactions in their block. One consolidates previous F2Pool coinbase outputs, and the other is a payout transaction to miners. This is common behavior for blocks mined by F2Pool.

While these extra transactions take up more than 400 kWU of block space, there would still have been enough space to include transaction aa001ce6... The block includes 2.86 MWU of transactions below aa001ce6..’s feerate of 10.5 sat/vByte. This transaction, with about 170 kWU, would have fit into the block. On mempool.space, this transaction is marked as “removed”, which negatively affects their block health metric.

Feerate distribution by transaction packages in block 811791 including markers for the feerate and weight of the missing transaction.

Block 811920

In block 811920, F2Pool did not include transaction 1cb3d6bc..¹⁰, which spends a sanctioned output. This transaction is a large consolidation transaction as well. It has a size of 43630 vBytes (169836 WU) and, with a fee of 44660 sat it pays a feerate of 10.23 sat/vByte. When block 811920 arrived at the miningpool-observer node, the transaction had been in the node’s mempool for close to 2 minutes.

In block 811920, 1.44 MWU of transactions pay less than 10.23 sat/vByte. The 170 kWU transaction 1cb3d6bc.. would have fit into the block. As the transaction was in my node’s mempool for only close to two minutes, there is a possibility that it didn’t yet propagate to F2Pool when they were building their block template. The transaction is shown as “recently broadcast” on mempool.space, too. Usually, mining pools try to have good connectivity to the Bitcoin network. There is a high likelihood that the transaction was in F2Pool’s mempool if it was in mempool.space’s and miningpool.observer’s mempool.

Feerate distribution by transaction packages in block 811920 including markers for the feerate and weight of the missing transaction.

Block 813357

In F2Pool’s block 813357, the transaction e49cdb60..¹⁰, which spends a sanctioned output, is missing. This consolidation transaction has a size of 43053 vBytes (172209 WU). With a fee of 178504 sat it pays a feerate of 4.15 sat/vByte. When block 813357 arrived at the miningpool-observer node, the transaction had been in the node’s mempool for more than 25 minutes.

There are 684 kWU of transactions paying below 4.15 sat/vByte in block 813357. The 172 kWU transaction e49cdb60.. would have fit into the block. As the transaction was in my node’s mempool for more than 25 minutes, it’s unlikely that the transaction didn’t propagate to one of F2Pools nodes yet. The transaction was included in mempool-space’s template for block 813357 too.

Feerate distribution by transaction packages in block 813357 including markers for the feerate and weight of the missing transaction.

Conclusion on F2Pools blocks

The sanctioned transaction missing from block 810727 had a slightly higher feerate because it’s 3 vByte smaller than the included transaction. While, in this case, these 3 vBytes of extra block space wouldn’t have made a difference in total fees, the Bitcoin Core block templating algorithm would have chosen the transaction with the higher feerate. The large extra transactions included in block 811791 wouldn’t have had an effect on the sanctioned transaction missing from block 811791. It has likely been filtered from the block. The block audit on mempool.space agrees with this. There is a chance that F2Pool didn’t yet know about the missing sanctioned transaction from block 811920. However, 2 minutes should be enough for a large pool to receive a transaction. Especially since mempool.space and miningpool.observer knew about this transaction. It’s likely that this sanctioned transaction is missing because F2Pool filtered it. Similar to the missing transaction from block 811791, the missing transaction from block 813357 has likely been filtered by F2Pool.

These four missing sanctioned transactions lead to the conclusion that F2Pool is currently filtering transactions. Since we’ve only seen transactions spending from the single OFAC-sanctioned address 3PKiHs4GY4rFg8dpppNVPXGPqMX6K2cBML missing, we can’t tell if F2Pool is filtering this single address or all OFAC- sanctioned addresses.

Conclusion

This post discusses six Bitcoin transactions spending from OFAC-sanctioned addresses that the miningpool-observer tool detected as missing from blocks. The two transactions missing from the ViaBTC and Foundry USA pool blocks are false-positives and not filtered. The four OFAC-sanctioned transactions missing from the F2Pool blocks are likely filtered. This raises the question of why F2Pool, a pool with origins in Asia, is the first pool to filter transactions based on US OFAC sanctions.

The Bitcoin network, however, continues to work as normal. A single pool filtering transactions does not affect the censorship resistance of the Bitcoin network as a whole. Further monitoring of the transaction selection of mining pools allows identifying when more pools start to filter transactions based on, for example, OFAC sanctions. It also allows miners pointing their hashrate to these pools to make an informed decision on switching to a different pool if they don’t agree with a pool’s (unannounced) filtering policies.

Update 2023-11-23

After publishing this blog post, F2Pool Co-Founder @satofishi tweeted that F2Pool was indeed filtering these transactions. He followed up with a statement that F2Pool would disable the filtering for now. Both tweets have since been deleted. The second tweet is archived on archive.ph. After deleting the tweet that the transaction filtering patch is disabled for now, it is unclear if F2Pool is still filtering transactions. ²

@satofishi justifying F2Pool filtering transactions. This tweet was deleted. Note: No Bitcoin addresses by Vladimir Putin and Xi Jinping are sanctioned by OFAC.

satofishi claiming he will disable the F2Pool transaction filtering patch for now. This tweet was deleted.

As all blocks with missing transactions didn’t come close to the sigop limit of 80.000, these aren’t discussed here. ↩︎
Ammended on 2023-11-23 after publishing. ↩︎ ↩︎
ViaBTC block 808660:
- 000000000000000000017c18a76632d9e39e8c388ee1e4028ec75e50866c79c5
↩︎
Transaction missing from block 808660:
- 262025e73812fc68b6514ea366abf463147176c7867e5853f117aded58c30e0e
↩︎
The transaction cb9f2592.. mined in block 809181 by ViaBTC is an Omnilayer transaction transferring 1528 USDT deposited to this address in September, 2020. The output to the sanctioned address 1ECeZBxCVJ8Wm2JSN3Cyc6rge2gnvD3W5K for this transaction was created similarly to the transaction 262025e7.. missing from block 808660 in d11019a2...

I checked a few of these addresses and they all contained USDT balances on OmniLayer which were swept in these transactions. While this is a guess, it seems someone wanted to sweep the remaining USDT on a bunch of addresses, sent 20k sats to each of them, and messed up the sweep by consolidating the newly created outputs again in 262025e7... They then retried with d11019a2.. and successfully swept it with cb9f2592...

If that’s the case, then OFAC is likely missing a bunch of addresses by the same entity on their list. ↩︎
Block 813231 mined by Foundry has the header hash:
- 00000000000000000001740d5fbb8bbc0b93d4bf46ca2011f642e92a0a8528b6
↩︎
Missing sanctioned transaction from block 813231 has the txid:
- c9b5719131bfeac6378749243731c5e70f1ce56deabb7006a2b6539710866420
↩︎
Based on OXT.me data, this address belongs to an OKEX wallet. The consolidation transaction c9b57191.. is OKEX consolidating deposits. The output being consolidated is, according to OXT.me, a Hydra darknet market payout. Additional information can be found here.
↩︎
Block hashes:
- 810727: 0000000000000000000350ae5ee08a4415146612af59a20021efeaf2ccda1498
- 811791: 00000000000000000001631243b00b6c1019c0d833b6738e0c591dacaf4453d6
- 811920: 00000000000000000002efd0fc8801b149f505b125308a35c584ed2600badf62
- 813357: 00000000000000000000519c33dcdf5ca386524b2cbacb561f767e9663ac1669
↩︎
Missing, sanctioned transactions:
- 810727: c6a668364f19df0f2977f8ad7d0a3a73c5e32b55b6a7c650cafa37a5ab4b19f2
- 811791: aa001ce6e262b8b9042645ecdec9c84e9e2ad06f56dff6dd5ae42005fdea8da9
- 811920: 1cb3d6bcc650c2891b68e7b205d601bcf5158e30e1926d0fd0c8385cb456b37b
- 813357: e49cdb6075c49b8fc37b3e922038e2a3205d75a9a1fb4b69f3568707594c2d3e
↩︎ ↩︎ ↩︎ ↩︎ ↩︎
Slightly larger and thus lower feerate transaction F2Pool picked for block 810727:
- 907e1f45334652dd344cf846639f3f9a2ee11b5489e2ffc2660ea543881b1bce
↩︎
Likely because it has fewer low-r nonces in the signatures which makes the signatures larger. ↩︎

FFM BTC Meetup: New features in Bitcoin Core v26.0

Thu, 19 Oct 2023 00:00:00 +0000

The upcoming Bitcoin Core v26.0 release will include new experimental features like AssumeUTXO and P2P transport v2.

At the Frankfurt Bitcoin Meetup (German), I explained the Bitcoin Core development process, that for this release cycle contributors focused their review on specific projects, and when to expect the new release (after testing, where everyone can participate!). Then dove into the problems that AssumeUTXO and P2P transport provide solutions for and what these solutions are. This was followed by a small discussion.

No slides.

Spiral Work-Log Q3 2023

Mon, 16 Oct 2023 00:00:00 +0000

This is a copy of the Q3 2023 work-log I sent to Spiral for my grant.

Disclaimer: Some information that is not (or not yet) meant to be published may have been redacted.

In Q3, I finished my GitHub repository backup and mirroring project, helped with the fork-observer integration in Warnet, continued my work on the MIT DCI peer-observer setup, attended CoreDev and the BTC Azores unconference, implemented a hidden getrawaddrman RPC in Bitcoin Core, started working addrman-observer, wrote about the invalid Marathon block at height 809478, and worked on a NixOS modules workshop for bitcoin++ in Berlin.

In chronological order:

GitHub Metadata Backup and Mirror

This year, the Bitcoin Core project will have its 13th anniversary being hosted on GitHub. 13 years of issues and pull requests with critical design decisions and nuanced discussions hosted with a US-based company known for shutting down open-source software repositories when needing to follow DMCA and OFAC requests. While the medium-to-long term plan is to move off of GitHub, I’ve written a tool for incremental GitHub metadata backups as a short-to-medium-term alternative. To use and test the backups, I’ve set up a read-only metadata mirror generated from the backups (with github.com/0xb10c/github-metadata-mirror): e.g. https://mirror.b10c.me/bitcoin-bitcoin/.

Warnet + fork-observer

While I consider my fork-observer project to be work-in-progress, I helped the warnet Bitcoin network simulation project in setting up a local fork-observer instance and made sure attaching more than a 100 Bitcoin Core nodes works as intended.

MIT DCI machines for peer-observer

The MIT DCI generously provided me with access to six machines to run my work-in-progress Bitcoin P2P anomaly and attack detection tooling, peer-observer, on. I deployed Bitcoin Core nodes and related software on these machines.

CoreDev and BTC Azores

I flew to Terceira to attend the Bitcoin Core CoreDev meeting and the BTC Azores unconference. At CoreDev, I led a discussion session on (undisclosed) Bitcoin P2P network problems we’ve observed earlier this year. At the unconference, I enjoyed the discourse with the wider Bitcoin developer community.

`getrawaddrman` RPC + addrman-observer

During Bitcoin Core PR review and discussions at CoreDev, the idea for a getrawaddrman RPC call came up. I added this RPC in PR #28523, which was merged recently.This RPC enables introspection into the Bitcoin Core IP address manager. To visualize the output, I created a work-in-progress tool called addrman-observer. Bitcoin Core developers interested in the addrman provided me with good feedback, the warnet project would like to use it, and I want it for my peer-observer setup.

Marathon Pool invalid block

I wrote down my notes on the Marathon pool invalid block at height 809478 in Invalid MARAPool block 809478. In turn, I received good feedback from people in the mining industry on my fork-observer project.

bitcoin++23 Workshop: Writing a NixOS Module for your_app

For the nix+bitcoin focused edition of bitcoin++, I’ve prepared a 90 minute workshop enabling Bitcoin developers to write NixOS modules for their projects. The workshop can be found here.

bitcoin++23 Workshop: Writing a NixOS Module for your_app

Sat, 07 Oct 2023 00:00:00 +0000

Instructions for my Writing a NixOS Module for your_app workshop. Slides can be found here.

The workshop tries to convey the basics of writing a NixOS module. In the end, participants should be able to write a basic NixOS module including a systemd service for their project, be able to define and declare NixOS options, and be familiar with basic systemd hardening and running containers on NixOS.

A 360p recording of me giving the workshop can be found here.

Task 0 - Setup and starting the VM

This workshop makes use of GitHub Codespaces to spin up a personal environment to work in. GitHub currently offers 120 hours of free CPU time per month for Codespaces. All you need is a GitHub account. You can find the code for this workshop in 0xb10c/btcpp23-nixos-modules-workshop.

This will open a new tab with a VSCode interface. The codespace automatically starts a NixOS virtual machine. This might take a couple of minutes. It will let you know when the VM is ready.

A downside of using GitHub Codespaces is that we don’t have KVM support for our NixOS VM and have to fall back to emulation, which is considerably slower than a KVM VM would be. Running nixos-rebuild switch inside the NixOS VM is quite slow and not recommended for this workshop. Rather, run sh nixos-rebuild-vm.sh on the host, which deploys the configuration to the VM.

I’ve marked commands with host$ <command> when you should run them from the GitHub Codespaces shell. Commands marked with vm$ <command> should be run from the VM shell. You can use ssh vm to log in to the VM.

your_app

For the workshop, I’ve written a very basic Rust program called your_app. Imagine this is a project you’ve been working on, and you now want to write a NixOS module for it. your_app starts a web server on a user-defined port and responds to requests. To show how easy it is to interact with other NixOS modules and services, your_app communicates with the RPC interface of a Bitcoin Core node.

$ your_app --help
your_app 0.1.0
USAGE:
your_app --rpc-host <RPC_HOST> --rpc-port <RPC_PORT> --rpc-user <RPC_USER>
--rpc-password <RPC_PASSWORD> <SUBCOMMAND>
OPTIONS:
-h, --help
Print help information
--rpc-host <RPC_HOST>
The host of the Bitcoin Core RPC server
--rpc-password <RPC_PASSWORD>
A password for authentication with the Bitcoin Core RPC server
--rpc-port <RPC_PORT>
The port of the Bitcoin Core RPC server
--rpc-user <RPC_USER>
A user for authentication with the Bitcoin Core RPC server
-V, --version
Print version information
SUBCOMMANDS:
help Print this message or the help of the given subcommand(s)
server Run the app with a web server

Task 1 - First steps

In this task, we enable the Bitcoin Core service that ships with NixOS and learn how to inspect a systemd service.

1.1: Enable the regtest Bitcoin Core node

In the configuration.nix file you’ll find a bitcoind service called regtest. This service is defined in the services/networking/bitcoind.nix module¹. Searching for services.bitcoind on search.nixos.org shows the options that can be set. For this workshop, I’ve configured a Bitcoin Core node on a local regtest test network with an RPC server listening on port 18444.

To enable the Bitcoin Core node, change the enable = false; into enable = true;. For the changes to take effect, run sh nixos-rebuild-vm.sh from the host to rebuild the VM. NixOS will automatically generate, enable, and start the systemd service defined for this node in the NixOS bitcoind.nix module.

host$ sh nixos-rebuild-vm.sh

1.2: Using systemd tools

Once the system is rebuilt, you can inspect the service with the systemctl status command. The status command only shows the last few log lines. If you want to see more lines, use the journalctl tool.

vm $ systemctl status bitcoind-regtest.service

vm$ journalctl --pager-end --follow --unit bitcoind-regtest
vm$ journalctl -efu bitcoind-regtest

Questions:

How many log lines does systemctl status bitcoind-regtest show?
Using systemctl status bitcoind-regtest also shows the generated *.service file. Can you find the -datadir parameter passed to Bitcoin Core? Where is the datadir?

Task 2 - defining and declaring options

The NixOS module for your_app is located in modules/your_app/default.nix. I’ve already defined a your_app_server and a your_app_backup systemd service in the config section of the NixOS module. I’ve left a few comments on the options that already exist. The places where you’ll need to fill in something are marked with # FIXME: Task X.X. You’ve successfully completed the task when you can reach the your_app server web server from the host machine (from outside the VM).

2.1: Declare options for `your_app_server`

When running your_app server, it expects the following command line arguments from us:

--rpc-host and --rpc-port for the location of the Bitcoin Core RPC server to connect to
--rpc-user and --rpc-password for authenticating with the Bitcoin Core RPC server
and a port on which the web server will start to listen on

Your task is to declare options for these command line arguments in options.services.your_app in modules/your_app/default.nix:

use the mkOption function - documentation can be found in the NixOS manual on mkOption
there’s a list of types you can use in the NixOS manual Option Types section
think about reasonable defaults for the options. Defaulting to null helps NixOS complain when an option is not set by the user.

2.2: Using the declared options

We can use the values from the options declared in 2.1 to define options from NixOS modules such as, for example, the systemd services module. I’ve prepared a systemd service in systemd.services.your_app_server and have already defined a few options. It’s your task to fill in the command line arguments (marked with FIXME: 2.2) in serviceConfig.ExecStart with the options you defined in 2.1.

Hint 1: Help, where do I start?

ExecStart is defined with a multi-line string. Each line contains a new argument. You can insert Nix expressions into strings with ${ <nix expression> }. If you have defined an option called username in 2.1., you could access it with ${ cfg.username }. Here, cfg is short for config.services.your_app (see the let .. in at the top of the file).

Hint 2: error: cannot coerce an integer to a string

Integers and strings don’t mix well in Nix. You can, however, convert an integer to a string with a function provided by NixOS. See the toString function.

2.3 Enable the your_app service in `configuration.nix`

The your_app module is imported in configuration.nix. We can now enable the services.your_app.enable option by setting it to true. We also need to define the options we declared in 2.1:

set the port for the web server to 4242 (this is important, otherwise the port forwarding to the VM won’t work)
the Bitcoin Core RPC server listens on localhost
you can set the Bitcoin Core RPC server port to config.services.bitcoind."regtest".rpc.port
use the RPC user workshop and the password btcpp23berlin

2.4: Open the firewall

By default, NixOS has a firewall that blocks incoming packets. To be able to reach the web server from the host, you’ll need to open this port in the firewall. See the allowedTCPPorts option of the NixOS firewall for more information. Similar to accessing the Bitcoin Core RPC port, you can access the port from the your_app server service.

2.5: sh nixos-rebuild-vm.sh

To rebuild and apply the configuration, run sh nixos-rebuild-vm.sh. NixOS might complain about errors in your configuration or module. Try to fix them, or ask someone next to you for help. Looking at the logs of the your_app_server.service systemd service might help.

Once you’ve managed to switch to your new configuration, try accessing the your_app web server from your host system.

vm$ journalctl -efu your_app_server

host$ curl localhost:4242

Task 3 - Secrets, security, and hardening

This task covers basic security and systemd hardening. However, this is likely not enough for a production setup.

3.1: RPC password is world-readable!

Your NixOS system configuration is world-readable by everyone with access to the nix/store/. Additionally, the systemd service configurations are world-readable too. This means the RPC password set in 2.3 is now world-readable, too. Can you find it?

Questions:

Where did you find the RPC password?
How can this be avoided?

Hint for question 2

Search for passwordFile on search.nixos.org/options.

3.2: systemd-analyze security

Under the “Principle of Least Privilege”, our newly set up systemd service should only have the minimum needed privileges. Systemd offers a bunch of sandboxing and hardening features that we can use to reduce the privileges of the your_app service.

The default systemd service options are quite lax. You can use

vm$ systemd-analyze security

to let systemd list you an “exposure” score (good = 0 - 10 = bad) for all loaded services. A high “exposure” score does not mean that the service isn’t sandboxed. It also does not mean that the service is vulnerable to attacks. It indicates that there is likely room for improvement by applying additional hardening settings to the service. Likewise, a perfect score doesn’t mean the service is completely secure. A better score indicates that the service has fewer privileges.

Questions:

What exposure score is shown for bitcoind-regest.service and your_app_server.service?
Which service has the lowest score?

3.3: your_app hardening

There is room for improvement in the “exposure” score of your_app. You can use

vm$ systemd-analyze security your_app_server.service

to list hardening options that can be enabled to improve the score. For inspiration, take a look at the nix-bitcoin defaultHardening options.

Set the hardening options in systemd.services.your_app_server.serviceConfig in modules/your_app/default.nix. You need to do a sh nixos-rebuild-vm.sh for the changes to be applied. This will also restart the your_app_server.service. Check if it still starts and the web server is still reachable from the host. If not, you might have removed too many privileges.

host$ curl localhost:4242

Task 4 - Containers

Running software in containers is also possible on NixOS. NixOS supports declarative oci-containers (i.e., Docker containers) but also allows running imperative and declarative NixOS containers. OCI-containers can be useful if there’s software not (yet) packaged for Nix. NixOS containers might be useful if you want to run multiple instances of the same service on the same machine or need a place for a quick experiment.

4.1: OCI-Containers

To demonstrate running an OCI-container, we can use the nginxdemos/hello:plain-text image. In configuration.nix you’ll find a commented plaintext-hello definition under virtualisation.oci-containers.containers. Uncomment it and set the image (just use the image name above) and ports values (use "8000:80"). More options can be found here. You will need to do a sh nixos-rebuild-vm.sh to start the OCI container.

Test that the web server in the container is reachable with:

vm$ curl localhost:8000

Questions

Which backend is being used by default to run oci-containers? Docker or podman?

4.2: NixOS containers

NixOS containers are lightweight systemd-nspawn containers running NixOS. These can be defined imperatively and declaratively. Imperative containers are great for short-to-medium term experimental setups, while declarative containers can be used for long-running container setups.

Imperative NixOS container

To imperatively create and start a NixOS container named btcpp23 use:

vm$ nixos-container create btcpp23
vm$ nixos-container start btcpp23

You can see the container logs and login as root with the following commands. See Imperative NixOS Containers for more.

vm$ systemctl status container@btcpp23
vm$ nixos-container root-login btcpp23

Declarative NixOS container

An empty, auto-starting, declarative NixOS container might look like:

containers.empty = {
 autoStart = true;
 privateNetwork = true;
 config = { config, pkgs, ... }: {
 # An emtpy NixOS container.

 system.stateVersion = "23.05";
 };
};

Feel free to copy and paste this container into the configuration.nix file and rebuild the VM. You should be able to login with nixos-container root-login.

local install (not recommended)

0.1: Install `nixos-shell`

To utillize NixOS modules, we need a running NixOS system. In this workshop, we’ll start a NixOS qemu virtual machine with the nixos-shell tool. Don’t confuse this with the nix-shell command, which allows us to temporary bring Nix packages into our environment. We can however use nix-shell to install nixos-shell as it’s packaged in nixpkgs. This assumes you have Nix installed.

$ nix-shell -p nixos-shell

0.2: Clone the workshop repository

$ git clone https://github.com/0xB10C/btcpp23-nixos-modules-workshop.git

You can find the configuration for the nixos-shell VM in this repository. The vm.nix file defines qemu VM parameters such as the number of CPU cores to use, the amount of RAM to reserve, and the size of the VM’s disk. Feel free to leave all files as they are. You’ll only need to modify configuration.nix and the modules/your_app/default.nix module during the workshop.

0.3: Starting the VM and logging in

Inside the btcpp23-nixos-modules-workshop folder, start the VM by running nixos-shell. While initial VM setup might take a minute or two, all following starts should be faster.

You’ll be greeted with a message explaining how to login and how to quit the VM. Use the root user without a password to log in. To exit the VM, either use shutdown now to shut it down or Ctrl+a c and type quit.

0.4: Rebulding the system with `sh nixos-rebuild-vm.sh` (optional)

Skip this step if you plan to directly continue with Task 1.

Once logged in, you can rebuild the NixOS system from the configuration. Changes can be made to the configuration.nix from your favorite editor on the host. If you are setting up the VM up before the workshop. feel free to run sh nixos-rebuild-vm.sh once to rebuild the system.

This module allows running multiple Bitcoin Core instances at the same time, which makes it a bit harder to reason about as a NixOS module beginner. ↩︎

Workshop: Writing a NixOS module for YOUR_APP

Sat, 07 Oct 2023 00:00:00 +0000

I held a workshop about “Writing a NixOS module for YOUR_APP”.

Invalid MARAPool block 809478

Thu, 28 Sep 2023 00:00:00 +0000

Notes on the invalid Bitcoin mainnet block at height 809478 mined by experimental, in-house MARAPool mining pool software on September 27, 2023.

On September 26, 2023, I noticed that a testnet node attached to a fork-observer instance reported frequent invalid blocks. The nodes debug.log showed multiple messages similar to the following:

ERROR: ConnectBlock: Consensus::CheckTxInputs: aca785e8.., bad-txns-inputs-missingorspent, CheckTxInputs: inputs missing/spent
InvalidChainFound: invalid block=00000000.. height=2505274 log2_work=75.527022 date=2023-09-26T15:20:05Z
InvalidChainFound: current best=00000000.. height=2505273 log2_work=75.527015 date=2023-09-26T15:13:45Z
ERROR: ConnectTip: ConnectBlock 00000000.. failed, bad-txns-inputs-missingorspent, CheckTxInputs: inputs missing/spent
InvalidChainFound: invalid block=00000000.. height=2505274 log2_work=75.527022 date=2023-09-26T15:20:05Z
InvalidChainFound: current best=00000000.. height=2505273 log2_work=75.527015 date=2023-09-26T15:13:45Z
ERROR: AcceptBlockHeader: block 00000000.. is marked invalid

This isn’t concerning by itself, as the Bitcoin testnet can be weird at times due to its low mining difficulty. No one was losing money, as testnet coins do not have any value. Someone could be testing a new block template algorithm and generating invalid blocks. Since this stopped after a few blocks, I didn’t investigate deeper after posting about it on Twitter.

On September 27, fork-observer notified me about the invalid Bitcoin mainnet block 000000000000000000006840568a01091022093a176d12a1e8e5e261e4f11853 at height 809478. Invalid blocks on mainnet are rather uncommon. Someone is losing money when they use time and energy to grind the block header of an invalid block. I checked a few monitoring nodes, and all of them saw the block and considered it invalid. BitMex Research’s forkmonitor also lists the block as invalid and attributes the block to MARAPool.

While validating this block, Bitcoin Core reported:

ERROR: ConnectBlock: Consensus::CheckTxInputs: 66dfefcdc3eeec2745c11f511f6068d62f34c34c767ba0feed47f63f01ccc2d8,
bad-txns-inputs-missingorspent, CheckTxInputs: inputs missing/spent

This means, there was a problem validating the transaction 66dfefcd[..]¹. Specificially, a previous output refferenced in an input wasn’t found in the UTXO-set during block verification. This is usually either caused by a transaction ordering problem (output is being spent before it’s created) or a corrupt UTXO set. Problems like this always raise the question where the problem originates from and who else might be affected. Is this a bug only on the mining pool side? Or is this a bug in some version of Bitcoin Core possibly affecting the whole network?

Sjors Provoost pointed out that 66dfefcd[..], which is simple a one-input and one-output transaction, did end up being mined by Foundry USA in the competing, valid block 809484. The input of 66dfefcd[..] spends the first output of 7d18f0ee[..]². Looking at the invalid block with

bitcoin-cli getblock 000000000000000000006840568a01091022093a176d12a1e8e5e261e4f11853

(output) shows that 66dfefcd[..] is the 6th transaction while 7d18f0ee[..] is the 1454th. In a valid block, 7d18f0ee[..] should have been included before 66dfefcd[..]. This shows a transaction ordering problem. Bitcoin Core checks created block templates for validity before responding to a block template request. This check would have caught transaction ordering problems before the template was sent to miners.

Upon closer inspection, it’s noticeable that the coinbase transaction of the invalid block includes the string /MARA Pool (v092623)/. The v092623 part looks like a version number and could indicate software built on September 26, 2023 (MM-DD-YY), the day the invalid blocks on testnet first appeared. Checking the invalid blocks on testnet shows a similar but differently formatted string: /MARA Pool (v230812AUG)/. This could be August 12, 2023, when read as YY-MM-DD. While the date formats don’t match, the testnet and mainnet activity seem related.

@mononaut further inspected the invalid block and found 145 transactions that were included before their parent. They, too, came to the conclusion that this likely isn’t related to Bitcoin Core’s block template algorithm. Moreover, the transactions in the block were sorted by fee before mining, which breaks Bitcoin’s consensus rules if there are parent-child relationships in the transactions.

This is what MARA's invalid block at 809478 looks like:

- pink transactions no longer exist in the main chain
- blue transactions are invalid due to ordering (they spend an output from a transaction included later in the block) https://t.co/SJI1azOB5Z pic.twitter.com/5gY9TRA2eG
— mononaut (@mononautical) September 27, 2023

Marathon Digital Holdings later confirmed that they mined an invalid block and clarified that they are using a small portion of their hash rate to experiment with a new, internal mining pool software in a development environment. This software included a bug that caused it to produce an invalid block. They also made sure to clarify that this isn’t a problem with Bitcoin Core in any way. No other mining pool should be affected.

We can confirm that Marathon did mine an invalid block. We utilize a small portion of our hash rate to experiment with our development pool and research potential methods to optimize our operations. The error was the result of an unanticipated bug that came from one of our experiments. In no way was this experiment an attempt to alter Bitcoin Core in any way. Our team noticed the invalid block around the same time as the rest of the world, and we immediately corrected the error. This incident, while unintended, underscores the robust security of the Bitcoin network, which rejected and rectified the anomaly.
― @MarathonDH

They mention that their team noticed the invalid block at the same time as everyone else. It’s, however, unclear to me why they didn’t notice the six invalid blocks on testnet a day before switching to mining on mainnet.

If a mining pool (or any other company) is interested, I’d be happy to help set up an internal and private fork-observer (FOSS) instance that can be used to monitor what your nodes consider to be the valid chain tip and alert you on invalid blocks and reorgs.

To clarify, this bug emanated from Marathon's own internal development environment. It was not related to Marathon's production pool. It was also not related to Bitcoin Core. Bitcoin functioned exactly as designed by excluding the invalid block.
― @MarathonDH

Marathon ends their tweet with “Bitcoin functioned exactly as designed by excluding the invalid block”. I agree - this was just a 6.43701991 BTC (USD $170k at the time of writing) bug that could probably have been avoided.

66dfefcdc3eeec2745c11f511f6068d62f34c34c767ba0feed47f63f01ccc2d8 ↩︎
7d18f0eefce0497b5d0c9b61fdf816b7744587c7e5e57acc53de71d1dae59725 ↩︎

Support from MIT DCI for peer-observer

Fri, 01 Sep 2023 00:00:00 +0000

The MIT DCI supported my peer-observer project with six monitoring Bitcoin nodes for 1.5 years.

GitHub Metadata Backup and Mirror

Thu, 10 Aug 2023 00:00:00 +0000

This year, the Bitcoin Core project will have its 13th anniversary being hosted on GitHub. 13 years of issues and pull requests with critical design decisions and nuanced discussions hosted with a US-based company known for shutting down open-source software repositories when needing to follow DMCA and OFAC requests. While the medium-to-longterm plan is to move off of GitHub, I’ve written a tool for incremental GitHub metadata backups as a short-to-medium-term alternative. To use and test the backups, I’ve set up a read-only metadata mirror generated from the backups.

github-metadata-backup github-metadata-mirror Mirrors and backups Mirrors and backups (Onion Service)

Moving off of GitHub?

Moving the Bitcoin Core development process away from GitHub has repeatedly been a topic among Bitcoin Core contributors. GitHub being a single point of failure, its unreliability, and the spam combined with the lack of moderation tools have been frequent topics. However, moving away from GitHub also means finding a better alternative. Ideally, the alternative is decentralized or federated and easily self-hostable to avoid moving to the next single point of failure. This also raises questions about who will host and administer the platform. Who is a trustworthy sysadmin to protect the alternative from DOS and other attacks? A slow or unreachable platform does not help developer productivity. Undeniably, GitHub has a significant network effect. Requiring users to sign up on another platform to report an issue or submit a small patch might not work well. Good code-review tools and stable CI integrations are high on the developer wishlist.

While a perfect alternative might not exist, Bitcoin Core developer fjahr currently experiments with a self-hosted GitLab instance that synchronizes GitHub issues and pull requests in real-time. This is a hot-spare alternative to GitHub. It might not be the final medium-to-longterm alternative the project seeks. Still, it can act as an interim alternative, allowing developers to continue working in case of problems with GitHub.

GitHub metadata backups

Tangentially, having a standalone backup of the development history of the Bitcoin Core project is vital for the project’s future. In the case that GitHub de-platforms the project, a backup of issues and pull requests with comments and code review allows reading up on design decisions and discussions about smaller and more extensive changes. This is amplified by long-time Bitcoin Core developers leaving and new developers starting to contribute to the project. Some ideas and discussions of the last 13 years would otherwise be lost. The backup can also be imported into a GitHub alternative once the project agrees to move to a new platform.

There are already tools to back up GitHub metadata, like issues and pull requests, and a public GitHub repository containing a metadata backup. The GitHub user zw has been running his ghrip Perl script for the last nine years and has pushed nearly 30.000 incremental backup commits to his bitcoin-gh-meta backup repository. However, upon closer inspection, it turns out that the backups are incomplete. Pull-request reviews are missing from the backups. This is likely due to a change in the GitHub API since the Perl script was last touched nine years ago. Also, Bitcoin Core maintainer achow101 has a project called github-dl, which downloads the full git repository, including source code, release-assets, and a wiki, if present. The backups are, however, not incremental, and a single backup takes more than a day to complete.

My github-metadata-backup tool makes incremental metadata backups by writing a state file after the first full backup and then only re-requests the changed issues and pull requests on subsequent runs. While source code, release- assets, and the wiki are out-of-scope, the backup contains everything displayed in an issue or pull request on GitHub. The backup is written to disk as one JSON file per issue or pull request. These JSON files can be tracked in Git and periodically pushed to remote repositories (for example, hosted by GitHub - duh).

The github-metadata-backup tool is written in Rust and uses XAMPPRocky’s octocrab library. Next to the endpoints for issues and pull requests, the GitHub REST API timeline endpoint is used to fetch events in issues and pull requests. I had to add the timeline API endpoints to the octocrab library, as they weren’t implemented before. A GitHub access token is required to run the backup tool, as unauthenticated API requests are heavily rate-limited (60 requests per hour). The tool detects rate-limiting when authenticated and waits until the token isn’t rate-limited anymore. The initial backup takes a while as requests are frequently rate-limitied, but the following incremental backups are pretty fast and normally only take a few seconds.

Mirroring issues and pull requests

Having metadata backups of GitHub repositories is great. However, using and testing the backups is required to ensure they are up-to-date and complete. I’ve set up a script that transforms the JSON files into Markdown that the Hugo static-site generator can use to generate a read-only mirror of the repository metadata. Using the Bootstrap CSS framework, a GitHub-like look can be archived. Reusing the GitHub IDs for issues and pull request comments allows linking from the mirror directly to comments on GitHub. On the mirror, URLs to other issues and pull requests in the same repository are rewritten to the mirror. This allows to open linked issues and pull requests, even if GitHub is down or the repository has been removed.

I’m backing up and mirroring the bitcoin/bitcoin, the bitcoin/bips, the bitcoin-core/secp256k1, and the bitcoin-core/gui repositories. Let me know if I should consider adding other repositories, too. I’m focusing on GitHub repositories with comments on issues and pull requests that are a vital part of the Bitcoin and Bitcoin Core development history.

Bitcoin Core mirror Bitcoin BIPs mirror secp256k1 mirror Bitcoin Core GUI mirror

I’m also offering compressed archives of the backups for download. Feel free to download backups occasionally and store them on one of your disks. The mirror is also available via an onion service for the people who want or need to use it. GitHub itself doesn’t offer an onion service to access its site.

Mirrors and backups Mirrors and backups (Onion Service)

While I will host the backups and mirrors for a while, I’d welcome it if others put up backups and mirrors, too. The backup tool can easily be run on low-power hardware. The mirroring tool uses Hugo, which loads the complete JSON files into memory before generating the static pages. Processing large repositories like bitcoin/bitcoin uses quite a bit of RAM. I’d be happy to help anyone trying to set this up. There are also public Nix packages and NixOS modules I use. This includes automatic runs via systemd-timers, a commit after each backup run, and automated pushes to one or more git remotes. I am happy to share my configuration if someone wants to run this on NixOS.

The backups can also be used for data analysis and data mining¹. Number of new contributors, comments per contributor, busiest times, most-active contributor, and so on. Also, the comments can be used as training data for a language model. I won’t have the time to play around with the data for the next few months, but let me know if you do something with the data.

Start page of the bitcoin/bitcoin mirror

Pull-request #28053 on the bitcoin/bitcoin mirror

Pull-request #28053 merge on the bitcoin/bitcoin mirror

Issue #20227 on the bitcoin/bitcoin mirror

Reminds me of SpiegelMining – Reverse Engineering von Spiegel-Online (33c3) - english translation ↩︎

Grant from Human Rights Foundation

Tue, 09 May 2023 00:00:00 +0000

The Human Rights Foundation supported my work on Bitcoin with a developer grant.

Annoucement

"[Grant to @0xB10C] for their work on Bitcoin Core tracepoints, P2P monitoring, fork observer, mining pool observer, and Bitcoin data.

Funding supports 0xB10C's efforts to monitor the Bitcoin network for anomalies, improving network security and resiliency 🦾"

LinkingLion: An entity linking Bitcoin transactions to IPs?

Tue, 28 Mar 2023 00:00:00 +0000

This post describes and discusses the behavior of an entity I call LinkingLion. The entity opens connections to many Bitcoin nodes using four IP address ranges and listens to transaction announcements. This might allow the entity to link newly broadcast transactions to node IP addresses. The entity has been active in some capacity since 2018 and is also active on the Monero network using the same IP address ranges. The entity might be a blockchain analysis company collecting data to improve its products.

I previously observed an entity making multiple, short-lived connections per second to many nodes on the Bitcoin P2P network. I called this entity an “Inbound Connection Flooder” and wrote about my initial observation in this post. However, after closer inspection of the entity’s behavior, I think these short-lived connections are only a symptom of the primary goal. I suspect this entity is likely tracking transaction propagation to attempt to determine which node broadcasts which transaction to link transactions to IP addresses.

The entity uses IP addresses from three IPv4 /24 ranges and one IPv6 /32 range to connect to listening nodes on the Bitcoin network. These IP address ranges are all announced by AS54098, LionLink Networks. However, the ranges belong to different companies based on ARIN and RIPE registry information.

162.218.65.0/24: Fork Networking, LLC (forked.net) - ARIN Whois
209.222.252.0/24: Castle VPN, LLC (castlevpn.com) - ARIN Whois
91.198.115.0/24: Linama UAB (?) - RIPE Whois
2604:d500::/32: Data Canopy (datacanopy.com) - ARIN Whois

Fork Networking and Castle VPN are US-based companies owned by the same person. Fork Networking offers hosting and Colocation services, while Castle VPN is a VPN provider. Linama UAB is a Lithuanian company with no web presence. Data Canopy is a US-based company offering cloud and colocation data centers. Since the connections from these IP ranges share very similar behavior, I assume they are controlled or rented by the same entity. I’m calling the entity “LinkingLion” as the AS LionLink Networks is the common factor for these IPs, and I assume the entity is trying to link transactions to IP addresses.

Behavior

To analyze the behavior of LinkingLion, I recorded the network traffic between my node and the entity’s IP ranges for about five days in the first half of March 2023. In this timeframe, about 200.000 connections were opened to my node from the entity. In the following section, I’ll walk through the observed behavior.

Connection establishment and handshake

Out of the four IP ranges, the entity uses the following 812 addresses (list) to open TCP connections to many listening Bitcoin nodes on the network:

162.218.65.11 - 162.218.65.254 (244 addresses)
209.222.252.2 - 209.222.252.254 (253 addresses)
91.198.115.3 - 91.198.115.62 (60 addresses) + 91.198.115.114
2604:d500:4:1::2
2604:d500:4:1::3:2 - 2604:d500:4:1::3:fe (253 addresses)

It uses the full range of ephemeral ports (1024-65535), which deviates from the default behavior of many operating systems (most use a smaller subset). It can be observed that the same IP address repeatedly connects, in some cases more than 50 times, before the entity switches to another IP address in the same address range.

The entity establishes a TCP connection to our Bitcoin node and starts the version handshake by sending a version message. The version messages have obscure user agents like, for example, /bitcoinj:0.14.3/Bitcoin Wallet:4.72/, /Classic:1.3.4(EB8)/, or /Satoshi:0.13.2/. In total, 118 different user agents are used. Nearly all of these appear in version messages with the same frequency, which indicates that the user agents are picked from a list and are likely fake. The entity uses 0 as the nonce for all connections and sets the transaction relay flag to receive information about new transactions we know.

The block height sent in the version message does not match the block height known to the Bitcoin network. About 98% of the connections increment the block height precisely every 10 minutes. Since the average time between blocks has been less than 10 minutes over the last few months, the entity’s height lags behind the network’s best height. In the observed connections, two different height configurations can be identified as lagging by about 700 and 2100 blocks. I estimated that the entity’s and the network’s height for the connections lagging by about 700 blocks matched in late Q4 2022 or early Q1 2023, and the height for connections lagging by 2100 matched in Q3 2022. I assume this is the time the height was last configured. For about 2% of the connections, the height is always set to block 658501. These connections all originate from 2604:d500:4:1::2, 91.198.115.114, or 162.218.65.219.

My node responds with a version and a verack message acknowledging that it understood the entity’s version message. At this point, the entity is expected to respond with a verack to complete the handshake. However, the entity closes about 82% of connections without sending a verack message. These connections are short-lived, with a connection duration of only a few seconds. All IPv4 addresses besides 209.222.252.2 open these connections. However, only the IPv6 address 2604:d500:4:1::2 opens short-lived connections, while the other IPv6 addresses don’t.

Opening a short-lived connection and closing it right after receiving the version message is typical when checking if a node is reachable on a given address. The entity also learns metadata like which network services the node offers, what version the node has, and what height it considers the blockchain to be.

Communication

The remaining 18% of the opened connections receive a verack and stay open longer. After the handshake, a Bitcoin Core node sends a sendcmpct message indicating support for Compact Block Relay, a ping message, a feefilter message with the minimum feerate we’re interested in, and a getheaders message requesting new headers the peer might know. The entity responds with a pong message and continues to respond for the duration of the connection. It never initiates a ping itself.

From here on, two different behaviors can be observed. Either the entity listens for inv messages from us for up to 150 seconds (2 minutes and 30 seconds), or it sends us a getaddr and listens for inv and addr messages from us for up to 600 seconds (10 minutes) before closing the connection. We send 15 inv messages on average to the entity during the shorter, inv-only connections. During the longer inv-and-addr connections, we send an average of six addr-messages and 104 inv-messages. In the Bitcoin protocol, inv (inventory) messages are announcements that new blocks or transactions are available. Upon receiving an inv, a node might request the block or transaction if it doesn’t know about it yet. The entity never requests blocks or transactions.

The inv-only connection duration is similar for the three IPv4 address ranges. Many connections are closed after either 90 seconds or 150 seconds. The connections from the 253 addresses in the 2604:d500:4:1::3 IPv6 range are primarily closed after 150 seconds, while some are closed earlier, between 90 and 150 seconds. The connections from 2604:d500:4:1::2 are closed nearly uniformly between 0 and 90 seconds. Generally, there are no special IP addresses used only for longer or shorter connections. The only outliner is 209.222.252.2, which only makes the longer 150-second connections. The IP 162.218.65.219 is notable for making twice the number of connections than the other IPs in the same IP range. The inv-and-addr connections request addresses with a getaddr message and only stem from 2604:d500:4:1::2 and 91.198.115.114. These are closed just after being open for 600 seconds.

Mass inbound-eviction

A Bitcoin Core node has a limited number of inbound connection slots. A new inbound connection might evict an existing one when all slots are full. Some peers are protected from being evicted, for example, peers that send us blocks or transactions we didn’t know about. Bitcoin Core’s eviction logic might choose to evict a peer from the network group with the most connections out of the unprotected peers. Bitcoin Core calculates network groups based on the /16 subnet for IPv4 and /32 subnet for IPv6.

LinkingLion often has multiple open connections to a node simultaneously. Once the inbound connection slots are full, a new inbound connection might cause one of the connections by the entity to be evicted. The entity reacts by opening another connection to the node, causing yet another connection to be evicted. I described this as “Inbound Connection Flooder” due to the high frequency of multiple hundred connections per minute. What I described only happens when a node’s inbound connection slots are full.

Other behavior

As reported in this monero issue, the same IP ranges also open connections to nodes on the Monero network. One user reports that the entity also uses IP addresses like 91.198.115.74, while I only observed connections using the IP addresses 91.198.115.3 to 91.198.115.62. The IP address ranges the entity uses have all been added to an IP block list for Monero nodes.

I’ve only seen connections from the 162.218.65.0/24 IP range starting at 162.218.65.11. However, there are publicly accessible logs, for example, [1], [2], and [3] from Summer 2021 showing requests to web servers from 162.218.65.10 with a Java/1.8.0_292 user agent. It’s unclear if these requests are related to the entity.

Discussion

In the following section I discuss questions that came up after making the above observations.

Are the connections from the same legal entity?

It’s unclear if the described LinkingLion entity is a single entity or a group of legal entities. The connections share patterns across the different IP address ranges. For example, the connection durations for the inv-only and inv-and-addr connection types are similar across the IP address ranges. Additionally, the IP address ranges all use the same fake user agents. While this indicates that the same or similar software is used to open connections through the same IP address ranges, it does not confirm that only one legal entity is behind these connections. Furthermore, the three different height configurations sent via the version message (static at height 658501, lagging by about 2100 and 700 blocks) could indicate three different configurations or versions of the software run by either one or multiple entities.

Are the connections opened through a VPN service?

Based on ARIN registry information, the 209.222.252.0/24 IP range belongs to a company called CastleVPN. This could indicate that the connections are opened through a VPN service. The other IP ranges could also be used as VPN endpoints, which would explain why multiple software configurations share the same IP addresses. However, this theory remains unconfirmed for now.

What information does the entity learn about a node it targets?

The information the entity learns from a node can be categorized into metadata, inventory, and addresses. All connections learn about node metadata, which includes if and when a node is reachable or unreachable, which software version runs on this specific node and when it upgrades, which block height it considers the best and when it changes, and which services the node offers. For example, if the node is pruned or serves bloom or compact block filters.

Connections that complete the version handshake and stay connected learn about our node’s inventory, like transactions and blocks. The timing information, i.e., when a node announces its new inventory, is especially relevant. The entity is likely to first learns about our new wallet transaction from us. As the entity is connected to many listening nodes, it can use that information to link broadcast transactions to IP addresses.

About 2% of the LinkingLion connections also ask our node to send it the network addresses of other nodes on the network. The entity likely uses these to find new targets to connect to and to keep connections to all possible nodes open. There are known ways of trying to infer the network topology, for example, how many connections a node has or who its peers are, based on address propagation. Based on the small number of connections that request and learn about other network addresses, this doesn’t seem to be the goal of the addr messages here. Though, it is also possible to learn about the network topology by tracking transaction relay.

However, why does the entity open multiple short-lived connections from multiple IP ranges to a single node? Similar information could be extracted with less effort and without opening and closing connections frequently. This would have avoided much of the noise that caused me to look at this in detail.

How long has the entity been active for?

I personally first observed the entity in the Summer of 2022. However, the entity has been active for longer. In August 2020, Bitcoin Core developer @jonatack posted a review comment on GitHub, which included the peers currently connected to his node. Four inbound connections from 2604:d500:4:1::2 with fake user agents are visible. Similarly, a screenshot in Bitcoin Core PR #18402: gui: display mapped AS in peers info window from March 2020 shows a connection from the same IP address as peer 43.

On an IP address banlist previously maintained by Greg Maxwell, now only accessible via the Way Back Machine, the IP ranges 162.218.65.0/24 and 2604:d500:4:1::2/128 can be found. They first appeared on the archived list in March 2019 and weren’t present in September 2018. The other two IP ranges are not on the list. However, the IP range 23.92.36.0/24, also announced by AS54098 LionLink Networks, can be found there since September 2018. There are #bitcoin-core-dev IRC logs from February 18, 2018, discussing this IP range with multiple users mentioning that they have multiple connections from that IP range. A screenshot shows two inbound connections (id 214 and 246) from this IP range with the user agents /Satoshi:0.10.2/ and /bitcoinj:0.14.3/. It seems the entity has been active since early 2018 in some capacity. However, it’s unknown whether the entity was active the whole time. The lagging block heights, presumably set in Q3 2022 and Q1 2023, indicate that the entity is still, at least to some degree, maintaining the data collection.

Who is the entity?

Most Bitcoin P2P anomalies originate from individuals playing around with the open network, companies with profit motives, for example, selling data to other companies and law enforcement, or by (academic) researchers. In this case, it seems unlikely that an individual would sustain this over multiple years. The IP address ranges and servers cost money. An academic experiment is usually shorter, too, as papers eventually need to be published. Academic researchers might not use fake user agents. It makes sense for a company to pay for IP address ranges and servers if they can sell the collected data or enhance an existing product. This could be a company doing blockchain analysis.

What are possible preventions and solutions?

A short-term prevention might be to manually ban the IP address ranges used by the entity from making inbound connections to nodes. I’ve published a transparent and Open Source banlist with the first entry being this entity. Node operators that want to protect against the entity making connections to their node can use this banlist. However, it’s important to note that this banlist is entirely optional and centralized. Another possibility is to contact the abuse contacts of the IP range owners or AS54098 LionLink Networking.

Both of these methods, however, don’t solve the root problem. The entity can easily switch to new IP ranges or route traffic through a different AS. The root problem is that transactions can to be linked to IP addresses. Fixing this requires changes in the initial transaction broadcast and transaction rebroadcast logic on the Bitcoin network and in Bitcoin Core. Transactions are transmitted to peers with independent, exponential delays. An entity opening multiple concurrent inv-listening connections to a node can link transactions to the node’s IP address with a high success rate. A solution might be Dandelion (in particular, Dandelion++ or some modification of it), where transactions are first transmitted to another node, which then broadcasts it. Dandelion++ is beeing used in Monero since 2020. An implementation attempt in Bitcoin Core did not succeed primarily due to DoS and complexity concerns.

Transaction broadcast over privacy networks like Tor is not affected if done correctly. A strategy is to broadcast a transaction to a node on the Tor network using a fresh connection and then close the connection right after. Some Bitcoin wallets with a strong focus on privacy implement similar features. It is currently not implemented in the Bitcoin Core wallet. Tools like bitcoin-submittx might be helpful.

To summarize, an entity frequently opens connections from multiple IP ranges to many nodes on the Bitcoin network. Some characteristics, like the fake user agents and the block heights that increase precisely every 10 minutes, confirm that the connections do not originate from some misconfigured Bitcoin node but are custom clients. About 20% of the connections are used to listen to transaction announcements, allowing the entity to link newly broadcast transactions to IP addresses. The same IP addresses connect to nodes on the Monero network too.

Only a few details about the entity are known. The same IP ranges have been making connections since 2018 in some capacity. It’s unclear if the IP ranges are maybe endpoints of a VPN service. Similarly, if the entity is a single entity or a group of legal entities is unknown. The behavior could indicate financial motives. A possibility is a blockchain analysis company that wants to enrich its product with additional data. A short-term solution might be a banlist or reporting the entity’s behavior. Solving the root problem requires deeper changes to the P2P logic in bitcoin.

You can check if LinkingLion is connecting to your listening clearnet Bitcoin Core node by grepping for the addresses in the getpeerinfo output:

bitcoin-cli getpeerinfo | grep -E '162.218.65|209.222.240|91.198.115|2604:d500:4:1'

Update 2024-03-28: One year after publishing this blog post, the LinkLion Networks AS issues a statement that they aren’t affiliated with LinkingLion besides announcing their IP addresses. On the same day, the LinkingLion activity significantly drops. I’ve written an update about it here.

Grant from Spiral.xyz for 2023

Wed, 01 Feb 2023 00:00:00 +0000

Spiral.xyz supported my work on Bitcoin for the full year of 2023 with a developer grant.

Annoucement

2022 Review and 2023 Outlook

Tue, 17 Jan 2023 09:00:00 +0000

In this post, I revisit my plans and projects for 2022, and give an outlook for 2023. I plan to continue my current Bitcoin network monitoring efforts in 2023. I briefly touch on potentially tight open-source developer funding in 2023.

Projects

In 2022, I pushed 582 commits to GitHub, opened 72 issues, proposed 68 pull requests, spoke at four (un)conferences, and appeared on one podcast. However, I don’t think these numbers express what has kept me busy in 2022. In this post, I want to introduce some of my current projects, mention their status, discuss recent progress, and lay out planned work for 2023.

I was fortunate to be a Brink.dev grantee in 2022, which allowed me to focus on my projects with very little administrational overhead and no need to worry if and when I’ll receive my next grant payment. That’s how you should do open-source developer grants! When Brink asked me to summarize the projects I plan to work on in 2022, I replied with the following.

[..] I plan to continue work on tracepoints in Bitcoin Core, including interface tests, review, and the addition of further tracepoints. Another goal is to launch a Signet with regular reorgs and a tool to explore reorgs visually. After observing flooding attacks on the Bitcoin P2P network in the summer of 2021, I set out to build a P2P network monitoring and anomaly detection tool in 2022. The goal is to feed the observations into Bitcoin and Bitcoin Core development, improving the P2P network robustness. I also plan to write up further Mempool Observations and start a similar series of blog posts for P2P network observations.

Tracepoints for Bitcoin Core

We’ve been adding a tracing interface to Bitcoin Core. This allows for greater observability of process internals during development and production use. For example, we can attach to a tracepoint in the P2P network message processing and learn in real-time about which peer sends us which network messages. This opens up many new possibilities for debugging, monitoring, evaluating changes, and detecting anomalies or other problems. We mainly use eBPF and Userspace, Statically Defined Tracing (USDT) on Linux systems to hook into the tracepoints. I’ve previously introduced this topic in my blog post Userspace, Statically Defined Tracing support for Bitcoin Core.

There has been much progress on the Bitcoin Core tracing framework in 2022. After we added the tracing framework and followed up with the first tracepoints in 2021, I started 2022 by ensuring that the tracepoints are included in release builds and added functional tests for the tracepoints to provide a semi-stable interface between releases. Getting these tests to run in the CI containers was more complex than expected, as the CI’s container kernel headers often don’t match the host’s kernel headers. The headers are needed to compile the eBPF bytecode when using the BPF compiler collection (bcc). A solution was to run the tests in a VM. I opened a PR to add tracepoints for opened, closed, evicted, and misbehaving connections and helped with a PR for mempool tracepoints. End of November 2022, I opened a PR that would eliminate the (minimal) overhead from tracepoint argument computation for users not using the tracepoints. This would also allow expanding the tracepoints to, for example, pass serialized transactions to tracing scripts. Currently, we make sure to avoid potentially expensive computations just for the tracepoints not to affect the majority of users not using the tracepoints. While I think Bitcoin Core’s tracing framework is still experimental, it’s moving in the right direction. I’m trying to keep this list of tracing-related issues and pull requests up to date.

I had the opportunity to hold a Tracing Bitcoin Core workshop at bitcoin++ 2022 in Austin, Texas. Together with jb55, who initially proposed a PR using eBPF to trace Bitcoin Core, I did an impromptu talk on the tracing framework, and its use cases at BTCAzores 2022.

I plan to propose further tracepoints, for example, some which allow observing changes to the IP address manager’s state. I’ve also looked into the kernels libbpf, which is an alternative to the current bpftrace and bcc tooling to develop tracing scripts. There is the possibility to replace the current Python and bcc tracepoint interface tests with concise and faster unit tests using libbpf. Instead of spinning up a full node in the functional test, we could directly test the function with the tracepoints. It might make sense to extract the example tracing scripts from contrib/tracing/ into a separate tracing-tools repo which allows us to, for example, add CI testing for the scripts. Seemingly, with each new bpftrace version, one of the existing example bpftrace scripts break. An option might be to drop the bpftrace example scripts moving forward. I might do a video workshop or similar format that goes into the details of eBPF, USDT, and the Bitcoin Core tracing framework to transfer some of my knowledge about it.

P2P monitoring

As my most ambitious project of 2022, I’ve been working on a tool for passive P2P monitoring using honey-pot nodes to detect anomalies and attacks on the Bitcoin network. We want to be aware of these attacks, anomalies, potential bugs, and other problems to react to them if needed. This is a reaction to the addr message spam during the Summer of 2021, which was only noticed by coincidence. It would be good to have some monitoring for this.

On a technical level, the monitoring utilizes the tracepoints mentioned above to extract P2P events like receiving a message, a connection being opened, or a misbehaving peer. These events are published into a message queue to which other applications can subscribe. One example application is a Prometheus metrics endpoint used in a Grafana dashboard. A demo version of this is running on public.peer.observer.

In 2023, I’m planning to deploy more instances of this tool to different regions with different configurations: for example, a mix of networks (IPv4, IPv6, Onion, I2P, …), P2P features (block filter, bloom filter, P2Pv2, …), and bitcoind versions (releases, master, P2P PRs). I’m using NixOS for declarative (infrastructure as code) and reproducible system deployments, which reduces the infrastructure maintenance overhead to a minimum.

I also plan to build out a P2P data archival tool. Next to disk-space efficient storage of the P2P traffic, this should include functionality to read, replay, and filter archival data. There’s also the idea of a Grafana Live dashboard with real-time metrics. The work on this ties in with proposing further P2P-related tracepoints to Bitcoin Core.

This project includes a “detect ➞ maybe analyze ➞ maybe react” part, which happens ad-hoc when we notice an anomaly or problem. For example, I’ve looked into and have written about an anomaly I’ve come across: Inbound connection flooder down. As a reaction, there’s the possibility of publishing an open-source banlist (similar to Greg Maxwell’s old banlist) with the IPs of the inbound connection flooder. I have notes for a few anomalies that I haven’t analyzed closely yet. I hope to get to them at some point in 2023.

Reorgs on Signet and fork-observer

In 2021, I worked with a Summer of Bitcoin mentee on “Reorgs on Signet”. Testnet can often be unreliable with block storms and frequent reorgs. With signet, the idea is to have a network that can be reliably unreliable. A part of this is to have, for example, automatic reorgs to test Bitcoin software behavior. Though, something like this hasn’t been deployed yet.

While I initially planned to rebase the old branch where we added functionality for periodic reorgs to Bitcoin Core’s signet miner during the Summer of Bitcoin, I didn’t get to it as it didn’t seem like a priority. There hasn’t been much interest from the broader community to use this. In many Bitcoin software projects, the reorg behavior is probably either tested in a regtest test suite or not tested at all due to large reorgs being very infrequent nowadays on the Bitcoin mainnet. It’s unclear to me if a signet with scheduled reorgs would currently see much use.

One artifact from working on the signet miner script is a tool to visualize the header tree with its forks and stale tips. This tool is now called fork-observer. It collects chain tip information from Bitcoin Core and btcd nodes via the getchaintips RPC and then builds a header tree. A single fork-observer instance can handle multiple networks with multiple nodes.

The fork-observer project is still work in progress. While a large chunk of it is done, there are a few missing pieces, such as RSS feeds for stale and offline nodes and some UI improvements. Next to the RSS feeds, bots posting events to, e.g., nostr relays or Twitter are planned. Ideally, I want to combine the tool’s release with a custom signet with reorgs to show its potential.

The fork-observer tool only needs access to the getchaintips, getblockheaders, and getblockhash RPC calls. These can be whitelisted in Bitcoin Core for a fork-observer-specific user. This allows connecting to many external nodes via a private network like, for example, a WireGuard tunnel to one fork-observer instance. Though, it’s certainly possible to run a fork-observer instance on, for example, a nix-bitcoin, start9 Embassy, or an Umbrel node connected to a single mainnet node.

The tool is similar to BitMEX Research’s forkmonitor.info. I think fork-observer is easier to self-host and benefits from the header tree visualization. However, forkmonitor.info has additional features, for example, checking for supply mismatches and listing interesting lightning transactions. It should be possible to add some of the header tree visualizations to forkmonitor.info too, which I might attempt at some point.

The tool also works well together with a bitcoin-data project I’ve started. The goal is to collect historical stale block data with header information where possible. I’m purposefully holding off on publishing this data until a Bitcoin Core PR is merged and released. However, if you have a long-running node (>5 years), feel free to get in touch with me if you want to provide data. I’m using the fork-observer database to feed new data into the dataset.

You can find a development version of fork-observer at fork.observer.

mininigpool-observer

In 2021, I started working on a miningpool-observer project that provides insights into mining pool transaction selection. This has been running on miningpool.observer since May 6th, 2021, when Marathon Pool mined their first block, claiming not to include OFAC-sanctioned transactions. In Observing Bitcoin Mining Pools, I’ve written about the project in more detail.

My work on miningpool-observer in 2022 has been mainly maintenance, smaller features, and bug fixes. I don’t expect this to change in 2023. I’m thinking about a few changes that allow for easier self-hosting and reduce the maintenance burden going forward. There are a few dependencies, such as the ofac-sanctioned-digital-currency-addresses repository and the known-mining-pools repository, which I plan to work on further. We’re currently combining the efforts of my known-mining-pools and the mempool/mining-pools dataset. My goal is to have a high quallity dataset that applications can use attribute blocks to mining pools like some block explorers already do.

Once Stratum v2 gains wider adoption, and job negotiation with the pools is possible, it would be interesting to poke at some Stratum v2 mining pools with non-standard transactions and transactions to sanctioned addresses. This could be incorporated into the miningpool-observer at some later point, depending on Stratum v2 adoption.

bitcoin-data

I’ve set up the @bitcoin-data organization on GitHub to archive some public but ephemeral bitcoin data, which could be helpful for research purposes at some point.

There’s a block-arrival-times repository that contains timestamps for when a node first saw a block. These timestamps are ephemeral as the timestamps encoded in the block headers almost always don’t match the time the block was broadcast to the network. We can extract the block arrival times from, for example, Bitcoin Core’s debug.log files. For instance, I’ve needed block arrival times when looking at which block height was active in The stair-pattern in time-locked Bitcoin transactions. There are other applications, too, like measuring block propagation timings across the network. If you have an old debug.log, feel free to check out the adding data section. There are details on how to parse and add data to the repository. Otherwise, just send it to me.

Another repository is the bitcoin-stats-archive. This currently fetches, and archives [KIT DSN Bitcoin Monitoring] data and luke-jr’s Bitcoin node counts.

Other smaller projects for 2023

I run a lot of my infrastructure, including many of the projects listed above, on NixOS. This allows me to build and package my projects reproducibly and declaratively define what services run on which hosts. This has been very reliable for me, and I’ll continue to use this setup. I plan to share the Nix packages and modules for my projects, making it easier for others to self-host them. I still have to figure out how to publish these and how to incorporate this into my current infrastructure repository.

I’ve been running a mempool visualization website on mempool.observer for a few years now, and I’ve recently set up a quick and dirty extension for full-RBF monitoring on fullrbf.mempool.observer. I could extend the main site with proper visualizations of, for example, RBF replacements, transaction packages in the mempool, size- and time-based evictions, and more. Though, I’m not sure if I’ll get to it. I’ve recently seen that there’s ongoing work to, for example, incorporate [RBF replacement timelines] into the mempool.space project. My time might be better spent helping to review this work rather than building something similar from scratch.

Over the past years, I’ve been able to use my monitoring tools and the data I’ve collected to provide data-based feedback to proposed changes. For example, I could reproduce the simulated bandwidth savings promised by Erlay, run detailed IBD benchmarks for PR #20827, and visualize RBF replacement data to provide insights into an RBF edge case. I plan to help out with similar smaller projects wherever possible. Looking at network data can often reveal possible improvements, as with #26526 and #26527. There are also a few more Bitcoin Network Observations I’m planning to write.

I’ve set up a technical, open-content Bitcoin developer blog as an experiment in 2021. This experiment succeeds when people start (cross) posting their blog posts to it on their own, and it fails once I don’t think it’s relevant anymore and stop paying for the domain. For more information, see bitcoin-dev.blog. While the blog didn’t see much activity in 2022, there’s a plan to revive it in 2023.

I won’t be able to finish or even get to all projects I’ve listed here in 2023. Similar to the last years, ad-hoc projects will pop up, and others will lose their relevance for a while. I often finish the projects I’ve started but might put them on ice for a while if I need time to focus on something I deem to be more urgent.

Developer funding in 2023

Over the past years, there have often been more funding opportunities than qualified¹ open-source Bitcoin developers. However, this has changed with big donors affected by the current market situation. Among others, for example, Gemini’s superlunar fund, funding many important developers and projects, recently shut down. There’s a chance that we’ll see a few grantors unable to renew grants for existing developers resulting in an even tighter funding situation with more qualified developers than funding. Though, at the moment, there is still funding available. I recommend starting looking for alternatives as early as possible if you think your grantor might not be able to renew your grant.

Personally, I’ve found a replacement for my Brink grant with a Spiral developer grant, which allows me to continue working on the projects mentioned above.

0xB10C is a Bitcoin developer focusing on observability in the Bitcoin network. He has built monitoring tools for mining pool transaction selection, P2P network anomalies, transaction replacements, forks and reorgs, and Bitcoin protocol statistics extracted from the blockchain. He aims to feed insights gained from the monitoring tools and collected data into Bitcoin development. Learn more about his projects on b10c.me/projects.

However, there have been qualified developers unable to find funding in some instances. ↩︎

Inbound Connection Flooder Down (LinkingLion)

Wed, 16 Nov 2022 00:07:00 +0000

Over the past few months, I’ve repeatedly observed very short-lived P2P connections with fake user agents being made to my Bitcoin Core node in a high succession. This morning around 7:00 am UTC, these abruptly stopped.

Update 2023-03-28: I’ve inspected the behavior of these connections in detail and wrote about it in LinkingLion: An entity linking Bitcoin transactions to IPs?. The frequent connections and evictions reported here only happen when my inbound connection slots are full.

This morning, I noticed a sudden drop in inbound connections to one of my nodes monitored by a P2P monitoring tool I’ve been working on. The number of inbound connections dropped from 115 (all inbound connection slots used) to 85 in about 10 minutes. Drops in inbound connections could be caused by an incident at a cloud service provider like AWS, Google, Hetzner, or similar. However, I didn’t find any incident reports on relevant status pages.

Looking into it, I’ve also noticed a significant drop in new inbound connections per minute from a relatively high number of 40 to a more normal level of about three new inbound connections per minute. As my node’s connection slots are limited to the default 125 connections and 10 are by default used as outgoing connections, most of these 40 inbound connections per minute caused another connection to be evicted¹. Subsequently, the number of evicted and closed connections dropped after 7:00 am UTC too. The number of inbound connections per net_group² reveals that the new connections primarily originated from three net_groups.

These newly opened inbound connections sent version message with rather uncommon user agents (some listed below). The user agents all appear with about the same frequency indicating that these are just randomly picked from a list. With slightly more effort, the entity opening these connections could have picked out of a more realistic distribution of currently used user agents. This would have made them a bit less obvious. After 7:00 am UTC, the version messages mainly contained the expected /Satoshi:23.0.0/, /Satoshi:22.0.0/, and an occasional seed node scraper or bitnodes user agent.

/bitcoinj:0.14.3/Bitcoin Wallet:1.0.5/
/Satoshi:0.16.0/
/Satoshi:0.8.2.2/
/breadwallet:1.3.5/
/bitcoinj:0.14.3/Bitcoin Wallet:4.72/
/Satoshi:0.12.1(bitcore)/
/bitcoinj:0.14.4/Bitcoin Wallet:5.21-blackberry/
/Satoshi:0.10.0/
/Satoshi:0.15.0/
/bitcoinj:0.14.5/Bitcoin Wallet:5.38/
/Satoshi:0.11.1/
/breadwallet:0.6.2/
/bitcoinj:0.14.5/Bitcoin Wallet:5.35/
/bitcoinj:0.14.4/Bitcoin Wallet:5.22/
/Satoshi:0.14.2/UASF-Segwit:0.3(BIP148)/
/breadwallet:0.6.4/
/Satoshi:0.15.99/
/Satoshi:0.16.1/
/Satoshi:0.8.5/
/bitcoinj:0.13.3/MultiBitHD:0.4.1/
/Satoshi:0.12.1/
/Satoshi:0.14.2(UASF-SegWit-BIP148)/
/bitcoinj:0.14.4/Bitcoin Wallet:5.25/
/bitcoinj:0.14.3/Bitcoin Wallet:4.58.1-btcx/
/breadwallet:0.6.5/
/Bitcoin ABC:0.16.1(EB8.0)/
...

I’ve seen these connections by this entity as early as June this year. However, it might have been active before. Back then, I briefly recorded the Bitcoin P2P traffic between one of the IP addresses these connections were opened from and my node. The IP addresses belong to a US-based VPN service called CastleVPN³ and a hosting provider called Fork Networking. Both are likely products by the same company. It’s unclear to me if the connections are being passed through the VPN or if then originate from a server hosted by Fork Networking. A typical P2P message exchange between the entity (them) and my Bitcoin Core node (us) looked as follows.

In the received version message, the protocol version of 70015 and the user agent /Satoshi:1.0.5/ clearly mismatch. The nonce, typically used to detect if we’re connecting to ourselves, is always 0, only the service bit for NODE_NETWORK is set, and the starting height is set ten blocks into the future. I’ve observed both positive and negative deltas for the starting height. During the communication, we don’t reveal a lot of information to the entity opening the connection. However, by sending the getheaders message, we inform the entity about which headers we know and which block we consider as chain tip. I found it interesting that they even respond to our ping. The connection stays open until it reaches a timeout or we evict the peer.

I don’t know the goal of the entity opening the connections. They learn that there’s a listening Bitcoin node behind this IP address. However, that’s not information you’d need to query more than 40 times a minute. They also learn about which chain tip we currently consider valid via the getheaders message we send them, which is interesting if you are someone monitoring the network for reorgs. However, you would be fine with way fewer connections for this too. Someone interested in monitoring block propagation would favor long-lived connections to learn about the blocks by us notifying them.

If your goal is to block inbound connection slots on listening Bitcoin Core nodes, then you can reach this with this technique. My node’s 30 newly free inbound connection slots have been filling up with (hopefully) good peers over the day. Additionally, by constantly opening new inbound connections, you can evict some good inbound connections before they can send you a block or transaction that would protect them. Since the eviction logic also considers other heuristics to protect our peers, an attacker can only trick us into evicting only some of our peers.

I wonder if the entity behind this is actually malicious. There is always the possibility that anomalies stem from some misconfigured academic measurements. The entity doesn’t seem too sophisticated given that it, for example, picks uncommon user agents with the same frequency as commonly used user agents.

A simple protection against this is banning these IP addresses from opening connections with your Bitcoin Core node. There have been so-called banlists containing known spy nodes, mass connectors, and other abusive nodes in the past. While I don’t know about any currently maintained banlists, it should be possible to create a new banlist based on data collected with my P2P monitoring project.

I’d like to hear from other node operators if they observed a similar drop in inbound connections this morning around 7:00 am UTC. You can also check what peer ids are reported by getpeerinfo RPC. My node has currently seen over a million peers. Also, if you are interested in this topic, I’m currently proposing a change to Bitcoin Core that adds the tracepoints I’m using for opened, closed, evicted, and misbehaving connections. I’m always happy about more testing and review.

Update 2022-11-18: I’ve observed another drop in inbound connections at my node. However, the inbound flooding seems to have started up again. Other Bitcoin developers reported seeing many inbound connections being opened to their node too. Some reported also seeing these connections from the same three /24 IPv4 subnets owned by Fork Networking. Based on the leasing price of $256/m per /24 subnet, it can be assumed that this costs the entity at least $768 per month not including server hosting fees.

The Bitcoin Core connection eviction logic protects nodes that, for example, provided useful information about blocks and transactions to us. For more information about the eviction logic in Bitcoin Core, see, for example, https://bitcoincore.reviews/16756 and https://bitcoincore.reviews/20477. ↩︎
In Bitcoin Core, a net_group of an IPv4 address is currently defined as the /16 subnet (first two tuples of an IPv4 address; netmask 255.255.0.0). This might be superseded with asmap in the future. ↩︎
It seems their VPN is so secure that they don’t even see the need to use TLS on their website… http://www.castlevpn.com/ ↩︎

Are you a real Bitcoiner?

Sun, 18 Sep 2022 09:00:00 +0000

It’s well known that you are only a real Bitcoiner if you are a carnivore, eat red meat at least three times a week but never consume seed oils, lift heavy weights, are straight, white, unvaccinated, believe in the Christian god, got into Bitcoin before 2015, are a maximalist, stack sats regularly, and own at least 6.15 BTC and two or more guns with plenty of ammunition.

Say yes #Bitcoin Fundamentalism pic.twitter.com/DXwqH5JYsk
— Skyler Designer (@skyler_fs) September 17, 2022

Obviously, that’s not the case. Anyone claiming this is a self-proclaimed gatekeeper that hasn’t understood Bitcoin. There are no gates in Bitcoin. On the technical side, anyone can join the Bitcoin network, and the other peers don’t care what you bench, if or which god you believe in, and how much bitcoin you own. On the social side, it’s similar - your dietary preferences, religion, political affiliation, skin color, when you got interested in Bitcoin, and your sexual orientation doesn’t matter.

Bitcoin is for everyone. A Bitcoiner is someone interested in Bitcoin and bitcoin as a tool, money, or technology. Someone who understood that Bitcoin is open for everyone, even for people you don’t like. Everything else doesn’t matter. Bitcoin doesn’t care. You are probably not a Bitcoiner if you don’t get this. You are probably not a Bitcoiner if you tell others they are not a real Bitcoiner because they like something you don’t like.

If you consume only vegan products and are interested in Bitcoin or bitcoin, you are probably a vegan Bitcoiner. Or, if you only eat meat, you are probably a carnivore Bitcoiner. But don’t generalize and assume all Bitcoiners must be vegan or, in the latter case, carnivores. Similarly, you can be a Christian Bitcoiner, a gun-owner Bitcoiner, or an LGQBT Bitcoiner. Being a Bitcoiner doesn’t depend on your other preferences and life decisions.

Thanks @skyler_fs on twitter for the fitting, and provocative header image.

P2TR spending transactions missing from F2Pool and AntPool blocks (2021)

Wed, 24 Aug 2022 00:00:00 +0000

My miningpool-observer project aims to detect when mining pools don’t mine transactions they could have mined. Right after taproot activation, it caught that F2Pool and AntPool didn’t mine P2TR (Pay-to-Taproot) spending transactions. This post is a write-up of this observation.

Note: I’m documenting an observation from November 2021 here so I can link to it. This is no longer an issue. I’ve also discussed this in Monitoring Taproot Activation, which might give a better overview of the situation.

During Taproot Activation

The taproot soft-fork was activated with block 709632 mined on November 14th, 2021. Some Bitcoin developers had crafted P2TR spending transactions containing, for example, OP_RETURN outputs celebrating the soft-fork activation. With block 709631 being mined, Bitcoin Core nodes began to accept and relay these P2TR spending transactions. Developers broadcast their transactions, hoping to make it into block 709632.

F2Pool mined block 709632 about 18 minutes after 709631 was mined. This was the first block with the taproot rules active. However, it didn’t contain any purposefully crafted P2TR spending transaction, which should have been included. While people were discussing what could have happened, AntPool mined the two blocks 709633 and 709634. Both of which didn’t contain any P2TR spending transactions either.

Then, Foundry mined block 709635, which included all P2TR spending transactions waiting in the mempool. Over the following blocks, other mining pools like, for example, Poolin, SlushPool, BTC.com, and Luxor also included P2TR spending transactions.

However, it took a few days until F2Pool and AntPool mined their first P2TR spending transactions. We only learned from F2Pool why they didn’t start mining these transactions right after activation. We don’t know details about AntPools problems but assume it was a similar issue. F2Pool had upgraded its nodes at least a few weeks before activation. However, they were running Bitcoin Core with custom patches. One of these patches disconnected peers, which reported a version string starting with /Satoshi:0.1. Only Bitcoin Core v0.21.1 and v22.0¹ shipped with the taproot soft-fork logic. Both of these didn’t match F2Pools version string filter and were disconnected. After fixing this, F2Pool started mining P2TR spending transactions. AntPool might have fixed a similar issue and started to include P2TR spends, too.

Missing transactions

The miningpool-observer project compares a just mined block to a block template generated just before the block was found. The differences between the two can show which transactions the pool left out and which were included extra. Block templates and blocks often differ slightly as different nodes know about other transactions. However, transactions not making it into multiple blocks when they have been in multiple block templates could indicate filtering or censorship by pools.

This was the case with the P2TR spending transactions. They were included in the block templates but not in the blocks by F2Pool and AntPool. The miningpool.observer/missing page shows transactions missing from three or more blocks. This is the case with F2Pools taproot activation block 709632, and AntPools blocks 709633 and 709634. The following transactions were included by Foundry in block 709635.

For example, transaction 0bf67b.. ² was missing from these blocks. It spent a 6490 sat P2TR output and created an OP_RETURN and a P2SH output. The OP_RETURN output contains the message “∞/21million. Thanks Satoshi! —BitGo”. Similarly, the transaction 905ecd.. ³ spent the first Taproot 2-of-2 multisig output and includes an OP_RETURN output with the message Thx Satoshi! ∞/21mil First Taproot multisig spend -BitGo. Andrew Chow’s script path spend 37777d.. ⁴, which ended up being the second P2TR spend mined, was reported missing. Pieter Wuille’s vanity address taproot party taptap transaction 83c8e0.. ⁵, too.

This shows that the tool works when multiple pools do not include transactions. This also shows that Bitcoin works as intended. If at least some pools are mining P2TR spending transactions, the transactions will eventually be mined.

More Taproot activation history is documented in Monitoring Taproot Activation and in this bitcoin-dev mailing list thread “Taproot activates - A time/block line”.

Bitcoin Core decided to drop the zero in the beginning of the version starting with version v0.22. ↩︎
txid 0bf67b1f05326afbd613e11631a2b86466ac7e255499f6286e31b9d7d889cee7, miningpool.observer on archive.ph, blockstream.info, mempool.space ↩︎
txid 905ecdf95a84804b192f4dc221cfed4d77959b81ed66013a7e41a6e61e7ed530, miningpool.observer on archive.ph, blockstream.info, mempool.space ↩︎
txid 37777defed8717c581b4c0509329550e344bdc14ac38f71fc050096887e535c8, miningpool.observer on archive.ph, blockstream.info, mempool.space ↩︎
txid 83c8e0289fecf93b5a284705396f5a652d9886cbd26236b0d647655ad8a37d82, miningpool.observer on archive.ph, blockstream.info, mempool.space ↩︎

bitcoin++ workshop: Tracing Bitcoin Core v23.0

Wed, 08 Jun 2022 00:00:00 +0000

These are the tasks of my “Tracing Bitcoin Core v23.0” workshop for bitcoin++ 2022. They might not make much sense on their own as participants used a pre-setup VPS. The workshop slides can be found here. This branch was used during the workshop.

Prerequisites

You will only need an SSH client to participate in the workshop. I will provide you with a pre-setup VPS to work on the tasks. I’ll send you the IP for your VPS. However, I first need an SSH public key from you. Please make sure not to send the private key part of your SSH key! I’d recommend generating a new, temporary key pair for this workshop. You should be able to run the following command on Linux and macOS to generate a new ED25519 keypair.

$ ssh-keygen -t ed25519 -C "bpp22-workshop"
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/your_user/.ssh/id_ed25519): /home/your_user/.ssh/id_bpp22workshop
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/your_user/.ssh/id_bpp22workshop
Your public key has been saved in /home/your_user/.ssh/id_bpp22workshop.pub
The key fingerprint is:
SHA256:FaK3FinGErPr1nTG63msKLoVSU9syTPpQSGGdsnpfvZM bpp22-workshop
The keys randomart image is:
+--[ED25519 256]--+
+ snip +
+----[SHA256]-----+

Please send the /home/your_user/.ssh/id_bpp22workshop.pub file to me via e-mail, telegram, or similar. I’ll respond with your server name. Once the VPSs are set up, I’ll publish the IPs below.

server name (BIP39 word)	IP	assigned
satoshi	XXX.X.X.X
hedgehog	X.XX.XXX.XX	Y
pottery	X.XX.XX.XXX	N
mango	XX.XXX.XX.XXX	SV
lock	XX.XXX.XX.XXX	Ja
senior	XX.XXX.XXX.XXX	Ju
acid	X.XX.XXX.XXX	JD
rabbit	XX.XXX.XXX.XX	A
vehicle	XX.XXX.XXX.XX	B
piano	XX.XXX.XXX.XX	MS
universe	XX.XXX.XX.XXX	M
link	XX.XX.X.XXX	SP
gadget	XX.XXX.XXX.XXX	N
pumpkin	XX.XXX.XX.XX	L
sunny	XX.XXX.XXX.XX
gorilla	X.XX.XXX.XXX
…

Once you’ve got your IP address, you should be able to SSH into the server via the following command. You might need to specify the SSH private key file (without the .pub at the end).

$ ssh workshop@13.37.83.33 -i ~/.ssh/id_bpp22workshop

Additionally, you might need to set the correct permissions (only read+write for your current user) on the SSH private key.

chmod 600 /home/your_user/.ssh/id_bpp22workshop

Once you’ve logged in to the system, it’s time to get familiar with it. Feel free to navigate around, check out the specs and running processes with htop, and check the kernel and operating system information with uname -a. The kernel version is important for using a lot of the eBPF functionality. We need at least a Linux kernel of version 4.8 to use hook into the tracepoints in Bitcoin Core. The programs vim and nano are also installed.

Note that you have password-less sudo access on the system. We need a privileged user to do BPF syscalls and to read from BPF maps. I expect you to not abuse this power (e.g. deleting system files). Treat this server as you would treat a server you’ve rented. The server auto-shutdowns on excessive resource usage. Additionally, your disk space is limited (please don’t try to sync mainnet). I might not provision another server for you or restart your instance. This would effectively end your participation in the workshop.

For the following tasks, it could be helpful to take a look at the Bitcoin Core tracing documentation and examples:

Tracing documentation: doc/tracing.md
Tracing examples: contrib/tracing

Questions:

Which Linux kernel version does the provided server have?
How much memory does your server have?
Are you able to switch to the root user with sudo su?

Task 1 - Listing available tracepoints

There is a bitcoin directory in your home directory. This contains a checkout of the Bitcoin Core source code. I’ve also built Bitcoin Core from source for you to save some time during the workshop. During the ./configure step, it was detected that the SystemTap sys/sdt.h header file is present on the system. Bitcoin Core was built with tracepoints.

$ ./configure
[..]
Options used to compile and link:
 multiprocess = no
 with experimental syscall sandbox support = yes
 with libs = yes
 [..]
 with zmq = yes
 with test = yes
 with fuzz binary = yes
 with bench = yes
 use asm = yes
 USDT tracing = yes # <---- Tracepoints enabled!
 sanitizers =
 debug enabled = no
[..]

Task: Your task is to list the available tracepoints in the bitcoind binary. The Bitcoin Core tracepoint documentation (linked above) contains three methods to do so. Feel free to try them out. The bitcoind binary is located in /home/workshop/bitcoin/src/bitcoind.

Try finding one or two tracepoints in the Bitcoin Core source code. I’d recommend that you navigate to the /home/workshop/bitcoin/src directory and grep for TRACE in *.cpp files from there (grep TRACE *.cpp -R). The ripgrep tool (rg) is installed on the system too.

Questions:

Which tracepoints does the binary contain?
Which of the three methods shows you information about the arguments passed to the tracepoints?
Are all tracepoints documented in the tracing documentation?
Were you able to find a tracepoint in the Bitcoin Core source code?

Task 2 - Running `p2p_monitor.py`

The p2p_monitor.py script is intended to demonstrate the possibilities the tracepoints offer. It shows the inbound and outbound traffic between the Bitcoin Core node you’re hooking into and it’s peers in real-time. Under the hood, it uses the net:inbound_message and net:outbound_message tracepoints, the Python BCC wrapper, and curses to render a TUI (Text User Interface).

We’ll be hooking into a testnet Bitcoin Core node already running on the system. To execute BPF syscalls and read from BPF maps, we’ll need a privileged user. Here, we use the root user. You can become root by executing sudo su.

As root, navigate into the /home/workshop/bitcoin directory. From there, you can run the following command to start the p2p_monitor.py script. We pass the path of the system-wide bitcoind binary we want to hook into (not the one in the src/ directory).

$ python3 contrib/tracing/p2p_monitor.py $(which bitcoind)

You’ll be able to navigate with UP/DOWN and select a peer with SPACE to see individual P2P messages being exchanged. It might take a few seconds for peers to appear as there is generally a lot less traffic on testnet compared to mainnet. The script only learns about a connected peer once we send or receive a message from it. Keep the script running for a minute or two.

This is also documented in contrib/tracing/README.md - p2p_monitor.py if you want to revisit running this at a later point.

Questions:

How many peers is your node connected to?
Does your node have any inbound peers? What are the connection types of your peers?
Are you able to observe, for example, a handshake with a peer or a ping-pong?

Task 3 - Tracing a Unit Test

When building a bitcoind binary with tracepoints enabled, the Bitcoin Core unit tests and benchmarks include these tracepoints, too, if they call the code sections containing the tracepoints. When adding the tracepoints, we’ve made sure to check that the tracepoints cause only minimal overhead of a few CPU instructions if we don’t hook into the tracepoint.

eBPF is programmable. In the previous task, the tracing script sent the data passed in tracepoint arguments back to userspace. However, for example, we can further filter the data or use it in calculations. bpftrace has builtin functionality like, e.g., sum(), count(), and creating histograms with hist().

Bitcoin Core’s unit test suite has a test called coins_test/coins_cache_simulation_test. It runs the function SimulationTest(). This is a large randomized insert/remove simulation test for the in-memory UTXO cache of Bitcoin Core. We can trace this test with the utxocache tracepoints.

Run the test from inside the bitcoin directory (as workshop user, CTRL-D to exit from sudo su) with the following command.

$ ./src/test/test_bitcoin --run_test=coins_tests/coins_cache_simulation_test --log_level=test_suite

Task: Using the bpftrace script contrib/tracing/task-3-stub.bt as a starting point, add functionallity to count how many UTXOs are added and removed (spent) during the unit test. Additionally, create a histogram of the UTXO values being added and removed.

Hint: Where do I start?

Take a look at the documation for the bpftrace functions sum(), count(), and hist() linked above.

Start your tracing script in the bitcoin directory as root user. In another SSH session, run the unit test. Stop the tracing script once the test is done. This should print the bpftrace counters and histograms.

$ bpftrace contrib/tracing/task-3-stub.bt

Questions:

How many UTXOs are added and spent during the test?
What are the most common values of UTXOs added and removed?
Which binary is the bpftrace script hooking into?
What data does the fourth argument (i.e. arg3; zero-indexed) of the add and spent tracepoints contain? (Hint: it’s documented)

Task 4 - Running the tracepoint interface tests

We want the tracepoint API to be semi-stable. That means there shouldn’t be unexpected breaking changes to the API, and tracepoint arguments should be documented. However, as tracepoints expose internals, we might need to change or drop the tracepoints if we refactor or drop Bitcoin Core internals. We test them in the Bitcoin Core functional test suite written in Python to ensure that the tracepoints don’t break. We use the BCC Python wrapper.

Task: Run the interface functional tests. There are two ways of running the tests. You can either run them through test/functional/test_runner.py or execute the standalone Python scripts. I recommend running them as standalone Python scripts first to see the eventual reasons the tests are skipped. Once you’re sure they aren’t skipped (i.e., they pass), you can run them through test_runner.py.

Running the tests as standalone Python scripts:

$ python3 test/functional/interface_usdt_net.py
$ python3 test/functional/interface_usdt_utxocache.py
$ python3 test/functional/interface_usdt_validation.py
$ python3 test/functional/interface_usdt_coinselection.py

Running the tests through test_runner.py:

$ python3 test/functional/test_runner.py --filter=interface_usdt_*

Questions:

Do the tests pass?
We require permissions to do BPF syscalls and read BPF maps for the tests. What happens when you run the tests with the workshop user?
Is blindly running Python scripts downloaded from the internet as root user on your own machine a good idea?

Task 5 - Adding new tracepoints

Here, the goal is to get familiar with adding a new tracepoint to Bitcoin Core. You’ll find documentation on this in doc/tracing.md. This also includes a few guidelines and best practices. Make sure to familiarize yourself with the documentation and to ask questions if something is unclear!

For long-running tracing tools like bitcoind-observer it’s useful to know when the Bitcoin Core instance we’re hooking into is stopped and started. This can, for example, be used to reset the statistic counters on a restart. These tracepoints don’t need to pass specific data.

Task: Add two new tracepoints. The first should trigger when Bitcoin Core is shut down, and the second should trigger when Bitcoin Core is started. Think about a name for context and event (see docs on “Adding tracepoints to Bitcoin Core”) that give a good description for a user not familar with Bitcoin Core internals.

Hint: Where should I put the shutdown tracepoint?

The src/init.cpp file contains a function called Shutdown(). Take a look at this function.

Hint: Where should I put the start-up tracepoint?

The src/init.cpp file contains a function called AppInitMain(). Take a look at this function.

You’ll need to recompile Bitcoin Core for your tracepoints to appear in the bitcoind binary. In the /home/workshop/bitcoin directory, run the following command as the workshop user. This will bring the Bitcoin Core build dependencies into scope.

$ nix-shell

Then, you’ll only need to run make to compile Bitcoin Core. We don’t have to re-run the autogen and configure steps. Compiling Bitcoin Core might take a minute or two.

You can use the bpftrace script in contrib/tracing/task-5-stub.bt to test your tracepoints. However, you’ll first need to fill in the names you’ve choosen for context and event. Then, run the script with the following command as root user (sudo su).

$ bpftrace contrib/tracing/task-5-stub.bt

Open another SSH session to your VPS to start and stop Bitcoin Core. From the bitcoin directory, use the following command to start Bitcoin Core in regtest mode. Wait a few seconds and then stop it with CTRL-C.

$ ./src/bitcoind -regtest

What name did you choose for context and event?
Where did you choose to place your tracepoints in the functions resposible for startup and shutdown?
Is your shutdown tracepoint being triggered in case Bitcoin Core does not shutdown cleanly? If not, would a tracepoint for this make sense?

Workshop: Tracing Bitcoin Core v23.0

Wed, 08 Jun 2022 00:00:00 +0000

I’ve held a workshop showcasing the new tracepoints and tracing functionallity included in Bitcoin Core release v23.0.

0xB10C – Tracepoints and monitoring the Bitcoin network

Mon, 06 Jun 2022 00:00:00 +0000

Jonas, Murch, and I chat about monitoring Bitcoin mining pools and tracing Bitcoin Core.

anchor.fm Podcast Page

Monitoring Bitcoin mining pool transaction selection

Sun, 08 May 2022 00:00:00 +0000

Can we detect transaction censorship by mining pools on the Bitcoin network?

I spoke at the MIT Bitcoin Expo 2022 about mining pool transaction selection, my miningpool-observer project, and observed extra, missing, and conflicting transactions between my block templates and the blocks mined by pools. I conclude that we can detect large scale transcation censorship by but haven’t seen any concrete evidence for censorship attempts yet.

Grant from Bitcoin Suisse (not accepted)

Fri, 08 Apr 2022 00:00:00 +0000

Bitcoin Suisse suprise-announced that they had selected me for a Research Grant.

However, due to a exclusivity agreement in the Brink grant agreement, I wasn’t able to accept the grant. To the best of my knowledge, the grant payment was paid out to Brink instead.

Announcement

Advancing Bitcoin Workshop: Tracing Bitcoin Core v23.0

Wed, 02 Mar 2022 00:00:00 +0000

These are the tasks of my “Tracing Bitcoin Core v23.0” workshop I held at Advancing Bitcoin 2022. They might not make much sense on their own.

Slides from the workshop can be found here.

Prerequisites

You will only need an SSH client to participate in the workshop. I will provide you with a pre-setup VPS to work on the tasks. I’ll send you the IP for your VPS. However, I first need an SSH public key from you. Please make sure not to send the private key part of your SSH key! I’d recommend to generate a new, temporary key pair for this workshop. You should be able to run the following command on Linux and macOS to generate a new ED25519 keypair.

$ ssh-keygen -t ed25519 -C "advbit22-workshop"
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/your_user/.ssh/id_ed25519): /home/your_user/.ssh/id_advbit22workshop
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/your_user/.ssh/id_advbit22workshop
Your public key has been saved in /home/your_user/.ssh/id_advbit22workshop.pub
The key fingerprint is:
SHA256:FaK3FinGErPr1nTG63msKLoVSU9syTPpQSGGdsnpfvZM advbit22-workshop
The key's randomart image is:
+--[ED25519 256]--+
+ snip +
+----[SHA256]-----+

Please send the /home/your_user/.ssh/id_advbit22workshop.pub file to me. I’ll display contact information on the screen. Once the VPS is set up, I’ll send you the IP address. Feel free to take a break in the meantime. It might take a few minutes until I send you the IP address of your server.

Once you’ve got your IP address, you should be able to SSH into the server via the following command. You might need to specify the SSH private key file (without the .pub at the end).

$ ssh workshop@[your.ipv4.addr] -i /home/your_user/.ssh/id_advbit22workshop

Additionally, you might need to set the correct permissions (only read+write for your current user) on the SSH private key.

chmod 600 /home/your_user/.ssh/id_advbit22workshop

Note that you’ll be having password-less sudo access on the system. We need a privileged user to do BPF syscalls and to read from BPF maps. I expect you to not abuse this power (e.g. deleting system files). Treat this server as you would treat a server you own. The server auto-shutdowns on excessive resource usage. Additionally, your disk space is limited (please don’t try to sync mainnet). I might not provision another server for you or restart your instance. This would effectively end your participation in the workshop.

For the following tasks, it could be helpful to take a look at the Bitcoin Core tracing documentation and examples:

Tracing documentation: doc/tracing.md
Tracing examples: contrib/tracing

Questions:

Which Linux kernel version does the provided server have?
Are you able to switch to the root user with sudo su?

We’ll wait until everyone is able to login into their server before continuing to the next task.

Task 1 - Listing available tracepoints

$ ./configure
[..]
Options used to compile and link:
multiprocess = no
with experimental syscall sandbox support = yes
with libs = yes
[..]
with zmq = yes
with test = yes
with fuzz binary = yes
with bench = yes
use asm = yes
USDT tracing = yes # <---- Tracepoints enabled!
sanitizers =
debug enabled = no
[..]

Questions:

Which tracepoints does the binary contain?
Which of the three methods show you information about the arguments passed to the tracepoints?
Are all tracepoints documented in the tracing documentation?
Were you able to find a tracepoint in the Bitcoin Core source code?

Task 2 - Running `p2p_monitor.py`

python3 contrib/tracing/p2p_monitor.py $(which bitcoind)

This is also documented in contrib/tracing/README.md - p2p_monitor.py if you want to revisit running this at a later point.

Questions:

How many peers is your node connected to?
Does your node have any inbound peers? What are the connection types of your peers?
Were you able to observe, for example, a handshake with a peer or a ping-pong?

Task 3 - Tracing a Unit Test

Run the test from inside the bitcoin directory with the following command.

./src/test/test_bitcoin --run_test=coins_tests/coins_cache_simulation_test --log_level=test_suite

Task: Count how many UTXOs are added and removed (spent) during the unit test. Additionally, create a histogram of the UTXO values being added and removed. Use the bpftrace script contrib/tracing/taks-3-stub.bt as a starting point. Run the completed tracing script as root user (e.g. with sudo). In another SSH session, run the test. Stop the tracing script once the test is done. This will print the bpftrace counters and histograms.

Questions:

How many UTXOs are added and spent during the test?
What are the most common values of UTXOs added and removed?

Task 4 - Running the tracepoint interface tests

Task: Run the interface functional tests. There are two ways of running the tests. You can either run them through test/functional/test_runner.py or execute the standalone Python scripts. I recommend to running them as standalone Python scripts first to see the eventual reasons the tests are skipped. Once you’re sure they aren’t skipped (i.e., they pass), you can run them through test_runner.py.

Running the tests as standalone Python scripts:

$ python3 test/functional/interface_usdt_net.py
$ python3 test/functional/interface_usdt_validation.py
$ python3 test/functional/interface_usdt_utxocache.py

Running the tests through test_runner.py:

$ python3 test/functional/test_runner.py --filter=interface_usdt_*

Questions:

Do the tests pass?
We require permissions to do BPF syscalls and read BPF maps for the tests. What happens when you run the tests with the workshop user?
Is blindly running Python scripts downloaded from the internet as root user on your own machine a good idea?

If you want, you can leave a short review on the PR #24358. For example, if you think it would be good to test the tracepoints, you could leave a “Concept ACK”.

Task 5 - Adding new tracepoints

Task: Add two new tracepoints. The first should trigger when Bitcoin Core is shut down, and the second should trigger when Bitcoin Core is started. Think about a name for context and event (see documentation) that give a good description for a user not familar with Bitcoin Core internals. Hint: You’ll probably want to have a look inside src/init.cpp.

Spoiler: Help! Where should I put the shutdown tracepoint?

The `src/init.cpp` file contains a function called `Shutdown()`. Take a look into this function.

Spoiler: Help! Where should I put the shutdown tracepoint?

The `src/init.cpp` file contains a function called `AppInitMain()`. Take a look into this function.

You’ll need to recompile Bitcoin Core for your tracepoints to appear in the bitcoind binary. In the /home/workshop/bitcoin directory, run the following command as non-root (i.e. workshop) user. This will bring the Bitcoin Core build time dependencies into scope.

$ nix-shell

Then, you’ll only need to run make. We don’t have to re-run the autogen and configure steps. Compiling Bitcoin Core might take a minute or two.

$ sudo bpftrace contrib/tracing/task-5-stub.bt

What name did you choose for context and event?
Where did you choose to place your tracepoints in the functions resposible for startup and shutdown?
Is your shutdown tracepoint being triggered in case Bitcoin Core does not shutdown cleanly? If not, would a tracepoint for this make sense?

P2P monitoring

Mon, 28 Feb 2022 11:00:00 +0000

I presented my planned P2P monitoring project to CoreDev attendees.

Slides (reveal-js)

Grant from Brink for 2022

Tue, 01 Feb 2022 00:00:00 +0000

Brink.dev supported my work on Bitcoin for the full year of 2022 with a developer grant.

Annoucement

Extracting the Private Key from Schnorr Signatures that reuse a Nonce

Mon, 17 Jan 2022 09:00:00 +0000

Elliott (aka @robot__dreams) posted a Bitcoin-flavored cryptography challenge on Twitter. The goal is to extract the private key from two Schnorr signatures that reuse a nonce. I’ve recently reviewed and merged Kalle Rosenbaum’s cross-post of his Schnorr Basics post to bitcoin-dev.blog. His post shows, for example, why nonce reuse leaks the private key. The challenge was the perfect opportunity for me to apply the theory the blog post explains. This post is a write-up of how I solved the challenge.

Challenge

The challenge puts us into the role of an adversary. The goal is to extract the private key used in the creation of two Schnorr signatures. The nonce is reused in both signatures.

You're given Schnorr signatures on two different messages signed by the same private key. Fortunately for you (the adversary), the signer screwed up their implementation of BIP-340 and reused a nonce. Can you capitalize on this fatal error and extract the signer's private key?

Note: You may find it helpful to interpret some of the byte strings as ASCII, in order to check your work.
Public Key: 463F9E1F3808CEDF5BB282427ECD1BFE8FC759BC6F65A42C90AA197EFC6F9F26 Message 1: 6368616E63656C6C6F72206F6E20746865206272696E6B206F66207365636F6E Signature 1: F3F148DBF94B1BCAEE1896306141F319729DCCA9451617D4B529EB22C2FB521A32A1DB8D2669A00AFE7BE97AF8C355CCF2B49B9938B9E451A5C231A45993D920 Message 2: 6974206D69676874206D616B652073656E7365206A75737420746F2067657420 Signature 2: F3F148DBF94B1BCAEE1896306141F319729DCCA9451617D4B529EB22C2FB521A974240A9A9403996CA01A06A3BC8F0D7B71D87FB510E897FF3EC5BF347E5C5C1

Theory

We use the same notation as laid out in Schnorr basics. The Schnorr signature scheme used in Bitcoin uses the secp256k1 curve. The generator point $G$ and the curve group order $n$ for the secp256k1 curve are listed, along with the other curve parameters, in Standards for Efficient Cryptography.

We are looking for the private key $p$. The public key $P$, two messages, $m_1$ and $m_2$, and two Signatures $(R, s_1)$ and $(R, s_2)$ are given. Both signatures share the nonce commitment $R$, as they reuse the nonce $r$. The responses $s_1$ and $s_2$ are different in both signatures as they belong to distinct messages.

$$ \begin{align} p & \quad \text{private key} \\ P = pG & \quad \text{public key} \\ m & \quad \text{message} \\ r & \quad \text{random nonce} \\ R = rG & \quad \text{nonce commitment} \end{align} $$

The signature signs a challenge hash $e$ which is the hash the concatenation of $R$, $P$, and $m$.

$$ e = H(R||P||m) \tag{1} $$

The response $s$ is the addition of the nonce $r$ and $ep$.

$$ s = r + ep $$

To extract the private key $p$, we can set up an equation system with two equations and two unknowns. This is adapted from the Schnorr basics post but shown in more detail.

$$ \begin{align} I & \quad s_1 = r + e_1 p \\ II & \quad s_2 = r + e_2 p \end{align} $$

We substract $II$ from $I$.

$$ \begin{align} s_1 - s_2 &= r + e_1 p - (r + e_2 p) & \tag*{ negating $(r + e_2 p)$} \\ s_1 - s_2 &= r + e_1 p - r - e_2 p & \tag*{ $+r$ and $-r$ cancel out} \\ s_1 - s_2 &= e_1 p - e_2 p & \tag*{ extracting $p$ } \\ s_1 - s_2 &= p(e_1 - e_2) \tag*{ dividing by ($e_1 - e_2)$ } \\ \frac{s_1 - s_2}{e_1 - e_2} &= p \tag{2} \end{align} $$

We can calculate $e_1$ and $e_2$ and know $s_1$ and $s_2$ from the provided signatures. The nonces $r$ and $-r$ only cancel out because the nonce is reused. With no nonce reuse, $r_1$ and $r_2$ wouldn’t cancel out, and we couldn’t solve for $p$.

Implementation

We’ve implemented a solution in Python. The curve group order, messages, signatures, and the public key are Python’s integer type. Python integers can hold arbitrarily large numbers (PEP 237). Arithmetic on integers is modulo $n$, the curve group order.

# curve group order
n = 0xfffffffffffffffffffffffffffffffebaaedce6af48a03bbfd25e8cd0364141

# public key
P = 0x463F9E1F3808CEDF5BB282427ECD1BFE8FC759BC6F65A42C90AA197EFC6F9F26

# first message and signature
m1 = 0x6368616E63656C6C6F72206F6E20746865206272696E6B206F66207365636F6E
sig1 = 0xF3F148DBF94B1BCAEE1896306141F319729DCCA9451617D4B529EB22C2FB521A32A1DB8D2669A00AFE7BE97AF8C355CCF2B49B9938B9E451A5C231A45993D920

# second message and signature
m2 = 0x6974206D69676874206D616B652073656E7365206A75737420746F2067657420
sig2 = 0xF3F148DBF94B1BCAEE1896306141F319729DCCA9451617D4B529EB22C2FB521A974240A9A9403996CA01A06A3BC8F0D7B71D87FB510E897FF3EC5BF347E5C5C1

Bitcoin ECDSA signatures are DER-encoded and have an encoding overhead. Bitcoin Schnorr signatures have no overhead as they are just the concatenation of 32 bytes of $R$ and 32 bytes of $s$. See Evolution of the signature size in Bitcoin for more details. As the nonce $r$ is reused, the signatures share the same $R$.

# the first 256 bits of the signatures are R
R = sig2 >> 256

# the last 256 bits of the signatures sig1 and sig2 are the reponses s1 and s2
s1 = sig1 - (R << 256)
s2 = sig2 - (R << 256)

The next step is to calculate $e_1$ and $e_2$ using formula $(1)$.

$$ e = H(R||P||m) $$

BIP 340 defines that the hashes are tagged with a context-specific tag. This makes sure hashes used in one context can’t be reinterpreted in another context. The tag for challenges is SHA256("BIP0340/challenge") || SHA256("BIP0340/challenge"). We use the hashlib module provided by Python to calculate SHA256 hashes.

import hashlib
challenge_tag = hashlib.sha256(b"BIP0340/challenge").digest() * 2

As the hashlib.sha256 function expects parameters of type bytes we need to convert our integers. We define the int_to_bytes(i) helper function and calculate the challenge hashes $e_1$ and $e_2$. We treat integers as big-endian when converting them to and from byte arrays.

byteorder = "big"

def int_to_bytes(i):
 return i.to_bytes(32, byteorder, signed=False)

def challenge_hash(tag, R, P, m):
 return hashlib.sha256(tag + R + P + m).digest()

e1_hash = challenge_hash(challenge_tag, int_to_bytes(R), int_to_bytes(P), int_to_bytes(m1))
e2_hash = challenge_hash(challenge_tag, int_to_bytes(R), int_to_bytes(P), int_to_bytes(m2))

The hashlib.sha256 function returns a bytes object. To convert the result back to integers, we define the helper function bytes_to_int(b).

def bytes_to_int(b):
 return int.from_bytes(b, byteorder, signed=False)

e1 = bytes_to_int(e1_hash) % n
e2 = bytes_to_int(e2_hash) % n

We now have the challenge hashes $e_1$ and $e_2$ and the responses $s_1$ and $s_2$. These can be used in formula $(2)$

$$ p = \frac{s_1 - s_2}{e_1 - e_2} = \frac{i}{j} $$

to calculate $p$. We first calculate the dividend $i$ and the divisor $j$.

i = (s1 - s2) % n
j = (e1 - e2) % n

However, implementing modular division isn’t as straightforward as dividing the dividend by the divisor.

# incorrect modular division!
p_ = i / j

We need the modular multiplicative inverse $j^{-1} = x$ of the divisor for modular division. The modular inverse $x$ multiplied with the divisor $j$ is congruent to 1 modulo $n$.

$$ j \cdot x \equiv 1 \mod n $$

The modular inverse is defined if the $i$ and $n$ are co-prime. This is the case if the greatest common divisor ($\text{GCD}$) of $i$ and $n$ is $1$. Then, to perform modular division, we can multiply the divisor $i$ and $x$.

$$ \begin{align} \frac{i}{j} = i \cdot j^{-1} = i \cdot x \mod n \tag{3} \end{align} $$

This is implemented using the extended Euclidean algorithm ¹ to calculate the $\text{GCD}(j, n)$, and, if $j$ and $n$ are co-prime, also the modular multiplicative inverse $x$. We’ve adapted the recursive Python implementation of the extended Euclidean algorithm provided by wikibooks.org.

def eea(a, b):
 """return (g, x, y) such that a*x + b*y = g = gcd(a, b)"""
 if a == 0:
 return (b, 0, 1)
 else:
 b_div_a, b_mod_a = divmod(b, a)
 g, x, y = eea(b_mod_a, a)
 return (g, y - b_div_a * x, x)

(gcd, x, _) = eea(j, n)
if gcd != 1:
 raise Exception('GCD of divisor mod n is not 1. Modular inverse is not defined.')

We check that the $\text{GCD}$ of $i$ and $n$ is equal to $1$. If that’s the case, we can use formula $(3)$ to calculate $p$.

p = (i * (x % n)) % n

The variable p now holds the private key as an integer. The challenge mentioned checking our results by printing the ASCII encoded private key. We can convert the integer to bytes with the int_to_bytes(i) helper function we defined earlier. When printed, the bytes type automatically shows ASCII characters.

# The output is purposefully omitted here.
print(int_to_bytes(p))

Another way of checking if the extracted private key is correct is to calculate the corresponding public key and compare it to the provided public key. If the public keys match, the private key is correct. We use the fastecdsa Python module for elliptic curve scalar multiplication (a scalar multiplied with a point). The public key is $P = pG$.

from fastecdsa.curve import secp256k1

if (p * secp256k1.G).x == P:
 print("Congratulations, you found the private key")
else:
 print("Public keys don't match: the private key is incorrect!")

# prints: Congratulations, you found the private key

This solves the challenge. The complete Python code as a Jupyter Notebook can be found here and run online in Google Colab here.

Bonus

While not part of the original challenge, knowing the private key $p$ allows us to extract the nonce $r$. Solving formula $(1)$ for $r$.

$$ \begin{align} s = & \ r + ep \tag*{- ep} \\ s-ep = & \ r \end{align} $$

r1 = (((e1 * p) % n) - s1) % n
r2 = (((e2 * p) % n) - s2) % n
assert(r1 == r2)

To double-check, we can calculate the nonce commitment $R = rG$.

if (r2 * secp256k1.G).x == R:
 print("The provided and the calculated nonce commitment R match!")
else:
 print("Nonce commitments don't match!")

# prints: The provided and the calculated nonce commitment R match!

Thanks

I’ve enjoyed this challenge and wanted to thank Elliott for setting it up. For me, as someone who is far from being an expert in cryptography, it turned out to hit the sweet spot between challenging and still being solvable. Elliott already followed up with the second and harder challenge.

I wouldn’t have looked at this challenge if I hadn’t read Kalle Rosenbaum’s Schnorr basics post only a few days before. He mentions he attempts to explain Schnorr signatures at a level he appreciates and hopes the reader finds it valuable too. For me, this has been the case. I think it’s often worth it writing a deep-dive into topics that you spent a lot of time on.

I want to pay this forward with this post. There are likely some who can’t dedicate enough time to solve this challenge but who find my solution valuable.

Elliott mentioned that I could have used Fermat’s Little Theorem to find the modular inverse too. ↩︎

Bitcoin Core PR Review Club: #20827

Wed, 29 Dec 2021 17:00:00 +0000

I’ve prepared and moderated a Bitcoin Core PR review club meeting on luke-jr’s PR #20827: During IBD, prune as much as possible until we get close to where we will eventually keep blocks. This is an potential IBD speed-up for pruned nodes.

Bitcoin Core PR Review Club on #20827

Bitcoin Core PR Review Club: #23724

Wed, 15 Dec 2021 17:00:00 +0000

I’ve prepared and moderated a Bitcoin Core PR review club meeting on my PR #23724: add systemtap’s sys/sdt.h as depends for GUIX builds with USDT tracepoints. My goal was to get feedback and eventually reach consensus on having the tracepoints for Userspace, Statically Defined Tracing in GUIX, and release builds.

Monitoring Taproot Activation

Sun, 14 Nov 2021 00:00:00 +0000

In November 2021, the Taproot soft-fork activated on the Bitcoin network. I streamed my activation monitoring and helped pools not mining P2TR spends to fix their issues.

Soft-fork activations can keep Bitcoin developers awake at night. After all, up-to-date nodes reduce what is allowed under the consensus rules to a subset. If, for example, a block not adhering to a newly introduced rule is broadcast shortly after soft-fork activation, the updated nodes will reject it while the old nodes will still accept it. This could cause a chain split between up-to-date and old nodes - a situation we want to avoid.

Taproot getting activated meant that certain SegWit version 1 outputs now could only be spent according to the new consensus rules defined in BIP-341. Before activation, there were no consensus rules on how to spend these outputs. Bitcoin Core nodes wouldn’t accept and relay transactions spending P2TR (Pay-to-Taproot) outputs to their peers. You could, however, spend them directly asking a miner to include them, as I’ve demonstrated here.

Taproot Activation Livestream

I’ve set out to monitor the activation of the Taproot soft-fork scheduled for block 709632. I set up a machine that runs a Bitcoin Core node, connected it to a script I’ve written, and streamed the console output it generated. The left side of the screen showed blocks being mined with, for example, the pool name, information about the coinbase, and the number of transactions in the block. The right side showed new taproot transactions (transactions with P2TR inputs or P2TR outputs) entering my mempool. It showed the transaction size, feerate, inputs and outputs, and eventual OP_RETURN messages the transactions contained.

Screenshot of my Taproot Activation Livestream. Recent blocks on the left and recent taproot transactions on the right.

I expected users to create P2TR outputs in the hours before activation and broadcast P2TR spending transactions right after block 709631, where the policy rules changed to allow P2TR spends to be relayed. Then, the next block should include the first P2TR spends.

As expected, we saw P2TR output spends being broadcast right after block 709631. Some of them included an OP_RETURN message celebrating the activation.

After 18 minutes and 17 seconds, my node received block 709632 mined by F2Pool. Taproot is active, and this block could have contained P2TR spends. However, it didn’t. Why? Was there a bug? There were P2TR spends in my mempool, which should have been included. The YouTube live stream chat¹ started discussing what could have happened.

The next block was mined by AntPool about 10 minutes later. It contained five transactions creating P2TR outputs, but still, no P2TR spends. After 6 minutes and 30 seconds, AntPool found the next block. Again, this one did not include any P2TR spends. At this point, there were plenty of P2TR spending transactions in my mempool, which should have been included. Both pools previously signaled that they were ready for Taproot and upgraded their nodes.

Foundry USA mined block 709635 8 minutes and 20 seconds later. It contained all 16 P2TR spending transactions currently in my mempool, which made up 1.69% of all 945 transactions in this block. Foundry including P2TR spends in their block was a relief for everyone watching. This means miners were able to mine P2TR spends. However, it was unclear why F2Pool and AntPool hadn’t mined them.

Over the following blocks, it became clear that Poolin, SlushPool, BTC.com, and Luxor also included P2TR spends, reassuring that the soft-fork activated as planned. You can find the full live stream recording here:

Pools not mining P2TR spends

Over the next day, it became clear that F2Pool and AntPool still weren’t mining P2TR spends. It was unclear if this was a technical problem or if they didn’t like Taproot and were purposefully ignoring these transactions. When asked why F2Pool didn’t mine P2TR spends and hasn’t upgraded their infrastructure, the owner of F2Pool replied that they “Will upgrade soon”. I suspect they had upgraded but didn’t know why they weren’t mining P2TR spending transactions.

Looking at the pools that mined more than 3 blocks since taproot activation or included a P2TR spend, it's clear that F2Pool and AntPool are, very likely, NOT including P2TR spends. F2Pool already mentioned that they will upgrade their infrastructure soon. pic.twitter.com/qVOwZvWO4q
— b10c (@0xB10C) November 14, 2021

Shortly after, an F2Pool employee contacted me to help troubleshoot why they weren’t mining P2TR spends. He told me that they had upgraded their node a while ago. They asked if I could craft a P2TR spending transaction to test with testmempoolaccept on their node. The transaction was accepted by their node and showed that they indeed upgraded their node. I asked them if they could send me the output of bitcoin-cli getpeerinfo | jq .[].subver and if they are configuring their peers manually. It turned out that they were only connected to peers version /Satoshi:0.19.1/ or lower. This meant they were ready to mine P2TR spends but didn’t have any peers relaying P2TR spends to them. Only Bitcoin Core nodes version v0.21.1 or newer relay P2TR spends after activation. Other versions don’t know about the soft-fork and reject these.

It turned out that F2Pool was running up-to-date Bitcoin Core nodes but had applied an old, custom patch. This patch disconnected nodes which version string didn’t start with /Satoshi:0.1. Bitcoin Core nodes with version /Satoshi:0.21.*/ and /Satoshi:22.0/ were disconnected. It seems like they didn’t signal readiness in bad faith. They just didn’t know they weren’t ready for Taproot activation due to a legacy patch they had forgotten about. After fixing this, they mined their first P2TR spend on November 18th, four days after activation.

Update: F2Pool included their first P2TR spend (https://t.co/Kr9Z0HXdzn) today in block 710269.

F2Pool reached out, and we debugged the issue: Nodes are up-to-date but didn't have any >=v0.21.1 peers (due to custom patches and addnode) that relayed P2TR spends. That's fixed now. https://t.co/Tg9jmIUmFq pic.twitter.com/2y1WtYjt40
— b10c (@0xB10C) November 18, 2021

AntPool was now the only pool with a significant number of blocks (107 blocks mined since activation) that hadn’t included a P2TR spending transaction on the 18th. The communication with them turned out to be not as easy as with F2Pool. Alejandro De La Torre communicated with AntPool. They told him that only having connections to old peers “may be the issue”. They were running v0.21.1 and would soon upgrade to v22.0. It seems like they also weren’t signaling readiness in bad faith.

AntPool mined their P2TR spend in block 710494 on the November 20th, 2021. All pools that mined ten or more blocks since taproot activation had now included at least one P2TR spend. Alejandro and I summarized this in a bitcoin-dev mailing list post too.

Update 2: AntPool included their first P2TR spend (https://t.co/H7FcJnLxRb) in block 710494.

All pools that mined ten or more blocks since taproot activation have included at least one P2TR spend. https://t.co/oKqj8BHb6H pic.twitter.com/jHqCbPRlUS
— b10c (@0xB10C) November 20, 2021

Learnings

We’ve learned that it’s important to communicate that miners do not only need to upgrade their node but also should make sure you are at least connected to a few upgraded peers. Direct communication channels between mining pools and developers are sometimes also helpful.

I’ve received a lot of positive comments regarding the activation stream and the monitoring I did. I think it’s essential to have some monitoring in place, even if it’s only to see that everything went as smoothly as expected. Or, in case it doesn’t, to be able to react to it as soon as possible. And more generally, documenting soft-fork activations for future Bitcoin developers is a good habit. This ensures they know what to expect and might not repeat the same mistakes.

Sadly, the YouTube live-stream chat is lost as live-stream recordings longer than 12h aren’t kept once the stream finishes. ↩︎

Updates on USDT in Bitcoin Core

Thu, 14 Oct 2021 10:00:00 +0100

I updated about my recent work on Userspace, Statically Defined Tracing (USDT) support for Bitcoin Core, and showed examples. We discussed where debug logging and where USDT makes more sense and security and performance of USDT. We briefly touched on adding automated testing for the tracepoints. A few additional tracepoint ideas were proposed by different developers.

Blog post: Userspace, Statically Defined Tracing support for Bitcoin Core on my blog.

Update on Reorgs on SigNet

Wed, 13 Oct 2021 11:00:00 +0100

I updated about my recent work on reorgs on SigNet and showed a work-in-progress block tree visualization. Approach and reorg parameters were discussed.

Discussion: Reorgs on SigNet - Looking for feedback on approach and parameters on the bitcoin-dev mailing list.

Userspace, Statically Defined Tracing support for Bitcoin Core

Mon, 30 Aug 2021 09:00:00 +0200

This report updates on what 0xB10C, Coinbase Crypto Community Fund grant recipient, has been working on over the first half of his year-long Bitcoin development grant. This specifically covers his work on Userspace, Statically Defined Tracing support for Bitcoin Core. This report was published on the Coinbase blog too.

The reference implementation of the Bitcoin protocol rules, Bitcoin Core, is the most widely used software to interact with the Bitcoin network. Bitcoin Core is, however, a black box to most users. While information can be queried via the RPC interface or searched in the debug log, there is no defined interface for real-time insights into process internals. Yet, some users could benefit from more observability into their node. Hobbyists and companies running Bitcoin Core in production want to include their nodes in their real-time monitoring. Developers need visibility into test deployments to evaluate, review, debug, and benchmark changes. Researchers want to observe and analyze the behavior of nodes on the peer-to-peer network. Exchanges and other services handling large sums of bitcoin want to detect attacks and other anomalies early.

Peeking inside with Userspace, Statically Defined Tracing

The eBPF technology present in the Linux kernel can be used for observability into userspace applications. The technology allows running a small, sandboxed program in the Linux kernel, which can hook into predefined tracepoints in running processes. Once hooked into a tracepoint, the program is executed each time the tracepoint is reached. Tracepoints can pass data, for example, application state. Tracing scripts can further process the data. The practice of hooking into tracepoints in userspace applications is known as Userspace, Statically Defined Tracing (USDT). For example, these tracepoints are also included in PostgreSQL, MySQL, Python, NodeJS, Ruby, PHP, and libraries like libc, libpthread, and libvirt.

The static tracepoints can be leveraged by Bitcoin Core users wishing for more insights into their node. Adding USDT support did not require intrusive changes, and no custom tooling had to be written. When not used, the performance impact of the tracepoints is minimal to non-existent. Only privileged processes can hook into the tracepoints, no information leaks to other processes on the host. These properties make Userspace, Statically Defined Tracing a good fit for Bitcoin Core.

For example, I placed two tracepoints in the peer-to-peer message handling code of Bitcoin Core. For each inbound and outbound P2P message, the tracepoints pass information about the peer, the connection, and the message. This data can be filtered and processed by tracing scripts. As a demo, I have built a P2P Monitor that shows the communication between two peers in real-time. Users can find this script alongside other USDT examples in the contrib/tracing/ directory of the Bitcoin Core repository.

Realtime Bitcoin Core P2P monitoring with User-Space, Statically Defined Tracing (USDT) and eBPF.

This is one of the examples from https://t.co/1QGb7jgsnZ (with added IP masking). pic.twitter.com/E5mREYknEm
— b10c (@0xB10C) May 24, 2021

Use-cases for Userspace, Statically Defined Tracing

I list some use-cases for Userspace, Statically Defined Tracing I have thought about or worked on. With only three tracepoints merged, there is plenty of room for developers to add new tracepoints and get creative with tracing scripts. Issue #20981 contains discussion and ideas for additional tracepoints that can be implemented.

Researchers and developers can use the P2P message tracepoints to monitor P2P network anomalies in real-time. One example could be detecting the recent addr message flooding as reported in this bitcointalk.org post. The messages were announcing random IP addresses not belonging to nodes on the Bitcoin network. The flooding has been covered in detail by Grundmann and Baumstark. They discuss that the attacker could obtain the number of connected peers and learn about other addresses, including Tor addresses, the node is listening on. This would reduce the privacy of the node operator. It’s important to stay vigilant to these attacks, discuss them, and then, if needed, react to them.

Similarly, I have been instrumenting the Bitcoin Core network address manager with tracepoints. The addrman keeps track of gossiped network addresses for potential outbound peers connections a node makes. It’s designed to be resiliant against Eclipse Attacks, where a node only has connections to peers controlled by the attacker. The attacker can choose which information to feed to the node, enabling, for example, double-spending attacks. Information about the addresses in the addrman might help detect the build-up of an eclipse attack when combined with other data.

Additionally, these addrman tracepoints can be helpful during debugging and code review. To showcase this, I build a tool that visualizes the addresses in the addrman data structure based on the data submitted to the tracepoints.

Quiz: Which Bitcoin Core data-structure does this visualization show? pic.twitter.com/U6cnHBFWWT
— b10c (@0xB10C) June 21, 2021

A Prometheus metric exporter can also build on top of the tracepoints without requiring additional code in Bitcoin Core. There already exist RPC-based Prometheus exporters and projects like Statoshi. However, RPC-based exporters are limited by the information exposed via the RPC interface, and Statoshi is large a patch-set that requires maintenance on each Bitcoin Core release. I have published an experimental USDT-based exporter called bitcoind-observer that hooks into the three currently merged tracepoints and serves metrics in the Prometheus format. The exporter can be used by everyone currently running a Bitcoin Core node compiled with USDT support. A demo is available on bitcoind.observer. I’ve recently used this to benchmark the bandwidth usage of an implementation of Erlay: Bandwidth-Efficient Transaction Relay for Bitcoin. Initial results can be found here.

The already existing tracepoint validation:block_connected can be used to benchmarking block validation. This allows, for example, to compare the initial block download performance between different patches and can aid in detecting performance improvements and regressions. For example, the bitcoinperf project might benefit from such tracepoints. I’ve used the tracepoint to benchmark Martin Ankerls pull request #22702. If merged, the changes he proposes would result in a substantial block validation speed up and reduction in memory usage.

Next steps

I will collect further ideas for tracepoints and implement them alongside example tracing scripts and more tooling. This will also involve communicating with other Bitcoin and Bitcoin Core developers about which tracepoints could be helpful in their projects. An example is Antoine Riard’s cross-layer anomaly detection watchdog which he initially proposed as a new, internal module to Bitcoin Core. However, many of the required events and metrics can be collected by hooking into tracepoints. This means the watchdog could be an external runtime, which would speed up the watchdog development and requires less code and maintenance on the Bitcoin Core side.

If everything goes according to plan, the v23.0 release of Bitcoin Core, expected in early 2022, will include the first set of tracepoints. A goal is to enable USDT support in release builds by default, which still needs some work. Additionally, the tracepoint API should be semi-stable and thus needs testing.

In short: I have been adding tracepoints to Bitcoin Core that users can hook into to get insights into the internal state. The tracepoints are based on Linux kernel technology and do not require intrusive changes or custom tooling. The groundwork is done. Now further tracepoints can be added, and tooling can be written.

Different sources of funding are important. I accept community donations for my work here.

Contribution: Bitcoin Core Project

Wed, 28 Jul 2021 00:00:00 +0000

I list my contributions to the Bitcoin Core project and detail their context and background. Of course, not all contributions are worth mentioning here.

Contributing to the Bitcoin Core project is not limited to opening pull requests which add or change code. Even more, and arguably most important, is reviewing existing pull requests. The review helps discovering issues and potential problems. Acknowledging a pull request once all issues are addressed helps finding consensus on a proposed change. Bitcoin Core contributor Jon Atack thinks that a good ratio is to review 5 to 15 pull requests for each pull request one opens.

PR #22006: tracing: first tracepoints and documentation on User-Space, Statically Defined Tracing (USDT) by @0xB10C

merged on July 26th, 2021

This PR adds documentation for User-Space, Statically Defined Tracing (USDT) as well as three tracepoints (including documentation and usage examples). We can hook into these tracepoints via a Linux kernel technology called eBPF. The data passed into these tracepoints can then be processed with, for example, bpftrace, or bcc scripts (in Python or Rust).

I add three tracepoints. Two tracepoints for inbound and outbound P2P messages and a tracepoint for the block connection function. The Bitcoin Core USDT support is documented here and tracing script examples are listed here.

PR #19643: Add -netinfo peer connections dashboard by Jon Atack

A Bitcoin Core node communicates with other nodes over a peer-to-peer network. Nodes open outbound connections to other nodes, and some nodes allow inbound connections. The connection management happens under the hood. A user can query the information about the current peer-to-peer connections with the getpeerinfo RPC. It returns a JSON formatted response containing detailed information about the connection to each peer. This JSON response is intended to be machine-readable and does not provide a quick overview over the open connections for developers and node operators. A script reformatting the RPC response would be needed to produce a dashboard like overview out of the JSON response.

This is where PR #19643 by Jon Atack comes into play. It extends bitcoin-cli with a -netinfo command. The command wraps the getnetworkinfo and the getpeerinfo RPCs and displays information about the inbound and outbound connections. In its current form, the command takes a numerical argument ranging from zero to four. With zero it only displays the connection counts grouped by in- and outbound connections. Passing one displays a dashboard with detailed information about the connections, and passing four extends this information with both the IP address and the version of the connected peer.

$ ./src/bitcoin-cli -testnet -netinfo 1
Bitcoin Core v0.20.99.0-bf1f913c4 testnet - 70016/Satoshi:0.20.99/
Peer connections sorted by direction and min ping
<-> relay net mping ping send recv txn blk uptime id
out full ipv4 13 13 13 13 0 16 12
out block ipv4 16 19 35 35 16 15
out full ipv4 16 16 13 23 2 2 17 6
out block ipv4 32 32 36 36 16 14
out full ipv4 33 34 12 28 0 17 4
out full ipv4 34 34 13 25 0 16 10
out full ipv4 34 35 13 28 0 17 1
out full ipv4 38 38 12 25 1 17 5
out full ipv4 158 233 25 8 1 16 13
out full ipv4 174 236 21 12 9 16 7
ms ms sec sec min min min
ipv4 ipv6 onion total block-relay
in 0 0 0 0 0
out 10 0 0 10 2
total 10 0 0 10 2

This resolves the need for writing a custom script. The change itself only affects bitcoin-cli and not bitcoind, which is neat. By running watch bitcoin-cli -netinfo 4 a detailed dashboard is shown, which updates itself every other second. Jon’s PR can only be used with Bitcoin Core version v0.21.0 or higher as an extra field was added to the response to the getpeerinfo RPC in an earlier PR. I’ve archived a branch where it’s possible to use the netinfo command with older bitcoind versions. This is available here.

PR #18172: test: Transaction expiry from mempool by @0xB10C

merged on 18th February 2020

There are multiple reasons a transaction can be removed from the mempool. The two most common removal reasons are transaction confirmation and transaction replacement. Transactions that remain in the mempool for a long time are probably not attractive to miners. By default, these are removed after 336 hours (two weeks). This default timeout can be overwritten with the -mempoolexpiry=<n> command-line argument where <n> specifies the timeout in hours. This functionality is important as we don’t want our mempool to fill up with transactions that are unattractive to miners (either because they pay a low fee or through other quirks). Having a test for this behavior is important to make sure that the functionally does not break.

In PR #18172 I’ve added a test for the mempool expiry behavior. This test tests both the default and the user-definable timeout. A first transaction called parent is added to our test nodes mempool and its entry time is recorded. Then the time is forwarded using setmocktime(). This simulates the test nodes time and conveniently allows for testing time-dependent functionality without having to wait until that time passes. A second transaction called child which spends an output from the first transaction is broadcasted to the test node after the first half of the expiry timeout. Then the time is forwarded to five seconds before the expiry of the parent transaction. The parent transaction should still be in our test nodes mempool. Five seconds after the expiry timeout the parent transaction should not be in our mempool anymore. As the child transaction depends on the removed parent transaction it becomes invalid and should be removed as well.

On anyone-can-spend Pay-to-Taproot outputs before activation

Fri, 23 Jul 2021 10:24:20 +0200

While working on taproot support for transactionfee.info, it became apparent that there already exist a few Pay-to-Taproot (P2TR) outputs on mainnet. Anyone can spend these outputs, but they are non-standard and thus not relayed between nodes in the Bitcoin peer-to-peer network. However, a mining pool can include non-standard transactions in their block. We cover the six P2TR outputs already present on mainnet and explain why the outputs can be spent by anyone before taproot activates. With the help of the f2pool.com mining pool, four P2TR outputs are donated to brink.dev, a non-profit company focused on supporting open-source Bitcoin development.

Taproot is a softfork upgrade to the Bitcoin protocol and tightens the consensus rules. A softfork forbids something that was previously allowed once it activates. Taproot restricts how native SegWit outputs, with a witness version of 1 and a 32-byte witness program, can be spent. The spending rules for other witness versions, nested SegWit, and native SegWit outputs with a witness version of 1, but longer or shorter witness programs are unaffected.

Existing P2TR outputs on Bitcoin mainnet

While it is possible to create P2TR outputs before taproot activates, it is not recommended to do so. Likewise, wallets shouldn’t generate P2TR addresses before activation. Nonetheless, at the time of writing, there are already six P2TR outputs on Bitcoin mainnet.

The first P2TR output was created on December 17, 2019, in a withdrawal transaction from purse.io. Matthew Zipkin initiated a withdrawal of 5431 sat to a witness version 1 bech32 address to test sending support for witness version 1. He added his test results to the Bitcoin Optech Compatibility Matrix and provided a screenshot showing the withdrawal on the purse.io frontend. The public key encoded in the address is 0101010101010101010101010101010101010101010101010101010101010101, which is likely not generated from a private key. It is to be assumed that the private key to this public key is unknown, and the funds won’t be spendable once taproot activates.

The second P2TR output was created on January 28th, 2020. With a value of 700 sat, it is close to the default SegWit dust threshold of 294 sat. The creator of this output is unknown.

The third, fourth, and fifth P2TR outputs are test outputs created while checking send support to a witness v1 bech32 addresses that Pieter Wuille posted to the bitcoin-dev mailing list on October 19th, 2020. The third P2TR output has a value of 3656 sat and was created by Riccardo Casatta using the Blockstream Aqua wallet. The fourth P2TR output with a value of 50.000 sat was likely created by Mike Schmidt using the BRD wallet. It’s unknown who created the fifth P2TR output with a value of 100.000 sat.

The sixth P2TR output, with a value of 1324 sat, was created on July 7th, 2021. It is the only P2TR output created after taproot locked in on June 13th, 2021. The other P2TR outputs were created while it was still unclear when or if taproot would activate.

Why can P2TR outputs currently be spent by anyone?

Note (added on 2021-07-24): This post mentions anyone-can-spend outputs. While this term is commonly used to describe outputs with not yet enforced spending rules, it has been the source for a lot of misinformation about SegWit in the past ¹. A better name might be “outputs with not yet consensus enforced spending rules” or “unenforced outputs before activation”.

Before taproot activates, these six P2TR outputs can be spent by anyone. However, none of these P2TR outputs have been spent yet, as the spending transaction is currently considered non-standard. Non-standard transactions aren’t relayed on the Bitcoin network ². Bitcoin Core checks each input for standardness in the AreInputsStandard function before accepting a transaction to the mempool. If the spend UTXO is a P2TR output, the function currently returns false, indicating that the transaction has non-standard inputs. Nodes don’t accept the transaction to their mempool and it is not relayed to peers.

However, a non-standard transaction can still be valid when directly included in a block. Next to other checks, the transaction has to pass the script verification. Standardness is not checked. In Bitcoin Core, the VerifyScript function is responsible for script verification. The script verification for native SegWit spends consists of two parts: the script evaluation and the verification of the witness program.

During script evaluation, the scriptSig is evaluated first. The resulting stack is used as the initial stack state when evaluating the scriptPubKey from the UTXO referenced in the input. The script evaluation succeeds if it finishes without failing, the stack is not empty, and the top stack element casts to true (is not zero). When spending native SegWit outputs like, for example, P2TR outputs, the scriptSig is empty. The scriptPubKey contains the witness version and the witness program. As the scriptSig is empty, an empty stack is used for the scriptPubKey evaluation. There, the witness version and witness program are pushed onto the stack. The script evaluation finishes. As the stack isn’t empty and the top stack element is the witness program (which usually isn’t zero), the script evaluation will succeed. This behavior makes SegWit a softfork. SegWit transactions are valid for nodes without SegWit support as the top stack element is not empty.

A keypath P2TR spend that will be valid and standard once taproot activates. A anyone-can-spend P2TR spend will likely have an empty witness, but otherwise look the same.

As a next step, the witness program is verified if the witness version is known. If the witness version is unknown, the witness program is treated as valid ³. One of the first checks during verification of version 1 witness programs (i.e. when spending P2TR outputs) is if the SCRIPT_VERIFY_TAPROOT script verification flag is set. If unset, the verification succeeds before checking the witness program. This flag is only set when taproot is active. This causes taproot outputs to be anyone-can-spend before activation. The witness program isn’t verified, and the witness can be left empty.

Spending P2TR outputs before activation

We demonstrate the spending of P2TR outputs before the taproot softfork activates by constructing a non-standard transaction that is consensus valid. The mining pool f2pool.com helps by including the non-standard transaction in a block.

For our transaction, we pick the first, third, fourth, and fifth P2TR output out of the six available outputs discussed earlier. The first output with a value of 5.431 sat will be unspendable upon taproot activation and would otherwise remain in the UTXO-set forever. The third (3.656 sat), fourth (50.000 sat), and fifth (100.000 sat) P2TR output were sent in test transactions with the knowledge that the funds will most likely be lost or donated to charity. This allows us to spend a total of 159.087 sat. We leave the other two P2TR outputs (700 sat and 1.324 sat) for someone else to spend either before or after activation.

Our transaction contains two outputs. The first output donates the full input amount of 159.087 sat (about 50 USD at the time of writing) to brink.dev to support open-source Bitcoin development. The transaction purposefully doesn’t pay a miner fee to maximize the donation amount. The second output is an OP_RETURN output with a link to this blog post. This makes it possible for someone finding the anyone-can-spend transaction to learn more about why the P2TR outputs were spendable before Taproot activation.

We constructed the transaction with a quick and dirty Rust script. For Bitcoin Core to accept the non-standard transaction, this patch had to be applied to v0.21.1.

The transaction b10c007c60e14f9d087e0291d4d0c7869697c6681d979c6639dbd960792b4d41 was included in block 0000000000000000000f14c35b2d841e986ab5441de8c585d5ffe55ea1e395ad (height 692261) mined by f2pool.com.

Show decoded transaction

{
 "txid": "b10c007c60e14f9d087e0291d4d0c7869697c6681d979c6639dbd960792b4d41",
 "hash": "b10c007c60e14f9d087e0291d4d0c7869697c6681d979c6639dbd960792b4d41",
 "version": 1,
 "size": 234,
 "vsize": 234,
 "weight": 936,
 "locktime": 45324,
 "vin": [
 {
 "txid": "b53e3bc5edbb41b34a963ecf67eb045266cf841cab73a780940ce6845377f141",
 "vout": 0,
 "scriptSig": { "asm": "", "hex": "" },
 "sequence": 6817749
 },
 {
 "txid": "b48a59fa9e036e997ba733904f631b1a64f5274be646698e49fd542141ca9404",
 "vout": 0,
 "scriptSig": { "asm": "", "hex": "" },
 "sequence": 45324
 },
 {
 "txid": "7641c08f4bd299abfef26dcc6b477938f4a6c2eed2f224d1f5c1c86b4e09739d",
 "vout": 1,
 "scriptSig": { "asm": "", "hex": "" },
 "sequence": 45324
 },
 {
 "txid": "fd43650477e0ba6825ae0482a8b0b2b509d5443fa2a8cdd101872752a9171dd6",
 "vout": 1,
 "scriptSig": { "asm": "", "hex": "" },
 "sequence": 45324
 }
 ],
 "vout": [
 {
 "value": 0.00159087,
 "n": 0,
 "scriptPubKey": {
 "asm": "OP_HASH160 fb94f9a556bae3f98f44186e1ccaa4f5ff6a3187 OP_EQUAL",
 "hex": "a914fb94f9a556bae3f98f44186e1ccaa4f5ff6a318787",
 "reqSigs": 1,
 "type": "scripthash",
 "addresses": [
 "3QdFzfpU3u92GeRN4U5pLLezbBaHbj2ppr"
 ]
 }
 },
 {
 "value": 0.00000000,
 "n": 1,
 "scriptPubKey": {
 "asm": "OP_RETURN 68747470733a2f2f623130632e6d652f37",
 "hex": "6a1168747470733a2f2f623130632e6d652f37",
 "type": "nulldata"
 }
 }
 ]
}

Show raw transaction

010000000441f1775384e60c9480a773ab1c84cf665204eb67cf3e964ab341bbedc53b3eb50000000000d50768000494ca412154fd498e6946e64b27f5641a1b634f9033a77b996e039efa598ab400000000000cb100009d73094e6bc8c1f5d124f2d2eec2a6f43879476bcc6df2feab99d24b8fc0417601000000000cb10000d61d17a952278701d1cda8a23f44d509b5b2b0a88204ae2568bae077046543fd01000000000cb10000026f6d02000000000017a914fb94f9a556bae3f98f44186e1ccaa4f5ff6a3187870000000000000000136a1168747470733a2f2f623130632e6d652f370cb10000

Note (added on 2021-07-24): There is a downside to including a transaction spending P2TR outputs in a block before the taproot softfork activates. A few months or years after the softfork has activated, the activation height becomes fact and can be hardcoded into Bitcoin Core ⁴. If there would have been no spends from P2TR outputs on-chain before activation, the taproot activation height would have been irrelevant. The taproot rules could have been enforced starting with the Genesis block (or maybe starting at the SegWit activation). However, with the P2TR outputs spending transaction included in a block, the activation height needs to be kept as part of the consensus logic. This slightly increases the software complexity of Bitcoin nodes.

Changes on taproot activation

When taproot activates on block 709632, Bitcoin Core v0.21.1 and newer will start enforcing the taproot rules. Spending P2TR will only be possible by supplying a valid signature for the keypath spend or a by satisfying the script requirements for a scriptpath spend. However, older nodes that don’t know about the taproot softfork will continue to treat the P2TR spends as anyone-can-spend. This can become a problem if, for example, a mining pool forgets to upgrade and includes a transaction spending from a P2TR output without satisfying the taproot rules enforced by the network. The chain would split between upgraded nodes enforcing taproot and not upgraded nodes. It is recommended to upgrade production and payment handling nodes before taproot activates.

I like to thank f2pool.com for their cooperation. Without them including the non-standard transaction in a block, donating the P2TR outputs to brink.dev, wouldn’t have been possible. You can support me and my work here.

Some SegWit opponents claimed that the SegWit softfork could be reversed after activation and the outputs would then be anyone-can-spend. To learn more about this I recommend reading The Blocksize War – Chapter 5 – SegWit. ↩︎
A good summary on non-standard transactions and forward compatibility can be found in this comment by nullc on Hacker News. ↩︎
This allows introducing of new rules to currently unused witness versions via a future softforks. The witness program verification succeeds at nodes unaware of the softfork. ↩︎
See, for example, the notes from the Bitcoin Core PR Review Club on #19438: Introduce deploymentstatus for more information. ↩︎

Summer of Bitcoin 2021

Mon, 28 Jun 2021 00:00:00 +0000

This is a placeholder for the Summer of Bitcoin mentoring I did. I plan to fill this in, once I get the time to do so.

bitcoind-observer

Sun, 27 Jun 2021 00:00:00 +0000

My experimental bitcoind-observer tool is a Bitcoin Core Prometheus metrics exporter utilizing and demonstrating the newly added tracepoints in Bitcoin Core.

The idea for a Bitcoin Core Prometheus metrics exporter based on the new tracepoints originated in a discussion on potential tracepoints between jb55 and laanwj in the Bitcoin Core issue 20981. Similar tools exist, for example, Jameson Lopp’s website statoshi.info. However, one downside with Statoshi is the patch-set that has to be rebased with every release. This can be cumbersome work if there are, for example, refactors in Bitcoin Core.

jb55: I was looking at this a long time ago. In theory you could get the same functionality as statoshi (stats gathering via prometheus) with https://github.com/cloudflare/ebpf_exporter Would be an interesting project. source
laanwj: I agree! This would be my preferred way to do this kind of reporting, to be honest. It'd flexible and built into the operating system. It has low impact on the code and low impact on performance. It could potentially work for all software.

One reason Statoshi never got merged is because it's just too much hassle to integrate and maintain a Bitcoin Core specific framework for this. Not because it's not a great thing to have. source

The tracepoints in Bitcoin Core have a semi-stable API. Upgrading applications that utilize the tracepoints to a new Bitcoin Core release should be easy. The tracepoints are flexible enough that we don’t instrument Bitcoin Core with an additional interface for a single application. The tracepoints can be used in debugging, review, for education about internals, monitoring, and data collection in small and large projects.

The bitcoind-observer tool attaches to multiple tracepoints in Bitcoin Core and transforms the tracepoint arguments to Prometheus metrics. It uses the bcc (BPF compiler collection) Rust crate to interact with BPF. For me, it also serves as a test bed for working with the tracepoints in Rust and for evaluating newly proposed tracepoints.

I build an experimental Bitcoin Core Prometheus metrics exporter based on the eBPF and USDT tracepoints I add in PR #22006 (https://t.co/1QGb7jgsnZ).https://t.co/RTLdhCLEf6

Grafana demo available on https://t.co/uEgE7d7Z7q
— b10c (@0xB10C) July 27, 2021

And block connection timings during IBD and data about validated blocks/second, transactions/second, inputs/second, and sigops/second.https://t.co/syfJ0pEAJZ pic.twitter.com/sGuIv3NavH
— b10c (@0xB10C) July 27, 2021

I’ve used the bitcoind-observer to provide feedback to Gleb for his work on Erlay by measuring the bandwidth consumption between erlay and non-erlay nodes. My measurements match Glebs calculations showing that we can archive the hoped bandwidth savings in this specific setup.

Bitcoin Core PR Review Club: #22006

Wed, 26 May 2021 17:00:00 +0000

I’ve prepared and moderated a Bitcoin Core PR review club meeting on my PR #22006 Tracing: first tracepoints and documentation on User-Space, Statically Defined Tracing (USDT). My goal was to familiarize Bitcoin Core contributors with the concept of Userspace, Statically Defined Tracing, and to get review on the first tracepoints, examples, and documentation.

Regulatory conform mining on einundzwanzig Podcast (German)

Fri, 14 May 2021 00:00:00 +0000

Dennis Reimann and I chat about regulatory conform mining in the context of my miningpool.observer project.

Podcast (German)

miningpool-observer: Observing Bitcoin Mining Pools

Thu, 06 May 2021 00:00:00 +0000

My miningpool-observer project aims to bring transparency to mining pool transaction selection. The tool can detect missing and extra transactions in blocks. One goal is to detect censorship by mining pools.

The idea for this project originated while reading the article Bitcoin Miner Transaction Fee Gathering Capability by BitMex Research. They touch on a new North American mining pool called Marathon that announced that they plan to filter out transactions that are not OFAC compliant. While this wouldn’t cause immediate problems for Bitcoin, as there are plenty of other pools mining these transactions, it’s still valuable to get an overview of mining pool censorship on the Bitcoin network.

Marathon Digital Holdings announced in a press release on March 30th, 2021, that they have licensed technology to set up a mining pool that allows them to filter transactions. Around the same time, I started my work on miningpool-observer. The goal was to have the tool ready from the day Marathon starts to mine blocks to check if they are leaving transactions out.

On May 5th, Marathon published a press release stating they have directed all of their hashrate to the new pool. On May 6th, they mined their first block 682170. The coinbase message stated:

MARA Pool mined its first 'clean' block today.https://t.co/v7WzKrzt9P pic.twitter.com/qTQOoQsq0D
— b10c (@0xB10C) May 6, 2021

As planned, I published my mingingpool-observer project on the same day.

In response to block #682170, mined by MARA Pool, I'm announcing a project I've been working on. I hope this brings transparency into mining pool transaction selection.

https://t.co/QrMdv9gADB

This compares block templates generated by my node to the blocks mined by pools.
— b10c (@0xB10C) May 6, 2021

Inspecting the block on miningpool.observer reveals that they’ve mined 166 transactions which the node powering miningpool-observer would have mined too, included 12 extra transactions, and had 5 missing transactions. These 5 missing transactions were only seen 12 seconds before the block was mined, which indicates that the pool might not have seen them yet as most pools update the block they are mining about every 30 seconds. The 12 extra transactions the pool include consist of the coinbase transaction and 11 low feerate transactions my node didn’t consider as it knew about the 5 missing transactions that paid a higher feerate. It was also pointed out that the block did include transactions sending to the Russian Darknet market Hydra Market.

Similarly, the next Marathon Pool blocks did not filter transactions to or from OFAC-compliant addresses.

1/ A quick thread on “compliant” blocks mined so far by MARA group.

It’s important to remember that missing transactions are not necessarily censored, but could just be due to propagation delays or other normal network behavior.
— Seth For Privacy (@sethforprivacy) May 14, 2021

On May 31st, Marathon Digital Holdings published a third press release stating they’ve ceased to filter transactions as they’re upgrading to Bitcoin Core v0.21.1 to signal taproot. We didn’t see Marathon Pool leave out and include any OFAC-sanctioned transactions during these 25 days of mining OFAC-compliant blocks, likely due to OFAC-sanctioned transactions being rare.

I’ve given a talk at the MIT Bitcoin Expo 2022 about this project. I discuss implementation details and other findings there. See here for slides and a recording.

This summary was written in December 2022.

Grant from Coinbase for 2021

Mon, 01 Feb 2021 00:00:00 +0000

Coinbase Giving provided a Bitcoin developer grant my work on Bitcoin in 2021.

0xB10C plans to build and improve existing open source tools, including a web interface visualizing forks on, for example, the new Signet test network. His work builds on his previous efforts launching mempool.observer and transactionfee.info. Additionally, he will continue to publish research articles on how the Bitcoin network is being used, provide review on Bitcoin Core PRs, and help improve functional testing."

Evolution of the signature size in Bitcoin

Tue, 10 Nov 2020 00:00:00 +0000

Digital signatures are an essential building block of the Bitcoin protocol and account for a large part of the data stored on the blockchain. We detail how the size of the encoded ECDSA signatures reduced multiple times over the last years and how the proposed Schnorr signature compares to the length of the currently used ECDSA signatures.

Digital signatures in Bitcoin transactions are located in the SigScript for inputs that are not spending SegWit or in the Witness for SegWit spending inputs. They consist of the encoded r and s values and a so-called SigHash flag, which specifies which part of the transaction the signature signs. The r and s values are both 256 bit (32-byte) integers.

DER Encoded ECDSA Signatures

Since its first version, the Bitcoin client relied on OpenSSL for signature validation and encoding. ECDSA signatures are encoded with the Distinguished Encoding Rules (DER) defined in the ANS.1 encoding rules. While the DER encoding only allows for exactly one way of representing a signature as a byte sequence, the OpenSSL library deemed derivations from the DER standard as valid. When this changed in the OpenSSL library, it caused some nodes with the newer OpenSSL version to reject the chain from nodes using an older version of the library. BIP-66 proposed a consensus soft fork where only signatures strictly following the DER encoding are accepted.

A DER-encoded ECDSA signature starts with a 0x30 identifier marking a compound structure. Next is a length byte containing the length of the structure followed by the compound structure itself. The compound contains the r- and s- values as integers. These are marked with the integer identifier 0x02 and followed by a length byte defining the respective lengths of the values.

Format of a DER-encoded Bitcoin signature with SigHash flag

While the ANS.1 encoding requires a signed integer, the r and s-values in ECDSA are expected to be unsigned integers. This causes an issue when the first bit of the r- or s-values are set. To solve this, the value is prepended with a 0x00 byte. Thus the unsigned integer is encoded as a positive, signed integer. A value with the first bit set is referred to as high and a value with an unset first bit as low.

A 73-byte high-r and high-s Bitcoin ECDSA signature

The r and s-values are random. When both values are high (both have their first bit set), they both require a prepended 0x00 byte. With two extra bytes, the encoded r- and s-values and the SigHash flag result in a total signature length of 73 bytes. The probability of both values being high in the same signature is 25%. Until early 2014, a distribution of about 25% 73-byte, 50% 72-byte, and about 25% of 71-byte signatures can be observed on the blockchain. In the 72-byte signatures, one of the two values is high and the other one is low. In a 71-byte signature, both values have to be low.

The share of the signatures with a high-s value started to reduce the release of Bitcoin Core v0.9.0 in March 2014. This release contained a change to the Bitcoin Core wallet to only create low-s signatures. With the release of Bitcoin Core v0.10.3 and v0.11.1 in October 2015, high-s signatures were made non-standard to completely remove the malleability vector. This forbids transactions with high-s values from being relayed or used in mining. Starting December 2015, nearly all transactions on the blockchain have only low-s values in their signatures.

A 72-byte high-r and low-s Bitcoin ECDSA signature

Between December 2015 and early 2018, the signatures on the blockchain are nearly evenly split between 72 and 71 bytes in length. The 72-byte signatures have a low-s and a high-r value, which requires a prepended 0x00 byte. The 71-byte signatures are low-r and low-s.

End of August 2017, the SegWit soft-fork activated. SegWit moves the contents of the SigScript, which contains, for example, the signature, into the Witness. While the witness gets discounted when calculating the transaction weight, the signature size on the blockchain remains the same.

A 71-byte low-r and low-s Bitcoin ECDSA signature

The Bitcoin Core v0.17.0 release in October 2018 included an improvement to the Bitcoin Core wallet to produce only 71-byte signatures. By re-signing a transaction with a different nonce, a new r-value can be grinded until a low value is found. The technique has been adopted by other projects such as the NBitcoin library and the Electrum Bitcoin Wallet.

Schnorr Signatures

BIP-340 introduces Schnorr signatures for Bitcoin and BIP-341 proposes a new SegWit version 1 output type and its spending rules based on Schnorr signatures, Taproot, and Merkle branches. Unlike ECDSA signatures, the Schnorr signatures are not DER-encoded.

Format of a Bitcoin Schnorr signature

Schnorr signatures contain the 32 byte r-value followed by the 32 byte s-value. The most commonly used SigHash flag SIGHASH_ALL is assumed by default and does not need to be set explicitly. Other SigHash flags are placed after the s-value. Schnorr signatures with the default SIGHASH_ALL flag have a length of exactly 64 byte. Signatures with a different SigHash flag are 65 byte long.

A 64-byte SIGHASH_ALL Bitcoin Schnorr signature

Compared to ECDSA signatures, Schnorr signatures are between 6 and 9 byte shorter. These savings stem from the removed encoding overhead and the default SigHash flag. With a Schnorr signature adoption of 20%, and assuming all of the 800.000 inputs spent per day contain only a single signature, more than 1MB of blockchain space is saved per day.

A Layman’s Guide to a Subset of ASN.1, BER, and DER - An RSA Laboratories Technical Note - 1993
Create the shortest transaction - Eternity Wall Blog - 2017
Exact probabilities of obtaining a DER encoded signature of a certain length - Eternity Wall Blog

This article was re-published on the Advancing Bitcoin blog and Hackernoon. Furthermore, it was translated into Chinese by Chen Bo Yu and Hsu Tzu Hsiu who published their translation on 链闻 ChainNews and Binance Blockchain Research Institute.

Support from Samourai Wallet for mempool.observer

Thu, 01 Oct 2020 00:00:00 +0000

Samourai Wallet supported my work on mempool.observer with a one-time donation that payed for hosting costs.

Following the Blockchain.com feerate recommendations

Mon, 13 Jul 2020 11:00:00 +0200

Transactions sent with Blockchain.com wallets make up for about a third of all Bitcoin transactions. A methodology to identify these transactions is described and used. Insights about the wallet-usage are derived from the resulting dataset. The privacy implications and possible improvements are discussed.

One of the first observations made when building the Bitcoin Transaction Monitor was that many transactions precisely follow the recommendations of a feerate estimator. These transactions appear as horizontal bands, which rise and sink as the feerate recommendations change.

Most of these transactions share the same fingerprint. Only P2PKH outputs are spent. No SegWit and neither multisig are spent. With every transaction, either one or two outputs are created. When two outputs are created, then at least one of them is a P2PKH output. The transactions are not time-locked, have a version of one, and do not signal BIP-125 replaceability. However, all are BIP-69 compliant.

This matches the fingerprint of the Blockchain.com wallets: namely a Web, an iOS, and an Android wallet. The wallets can only receive and spend P2PKH outputs. While users can pay to all address formats¹, the change-output, if created, is a P2PKH output. The wallets construct the transactions with a locktime of zero and a transaction version of one. The inputs and outputs are all lexicographically sorted as specified by BIP-69.

The wallets use the Blockchain.com feerate estimator, which is publicly accessible via an API. The API returns two feerate estimates: priority and regular. The priority feerate aims for confirmation in the next hour and the regular feerate for confirmation in an hour or more. By default, the wallets follow the recommendations closely. Users can set a custom feerate, but a warning is displayed.

Methodology

Combining the feerate estimates and the transaction fingerprints makes it possible to identify transactions sent with one of the Blockchain.com wallets. While the majority of the Blockchain.com transactions pay exactly the recommended feerate, some under- or overpay by a fixed percentage. This is caused by incorrect assumptions about the transaction size during the calculation of the transaction fee. The transaction fee is the product of the targeted feerate and the assumed transaction size. The final and actual transaction size is only known after adding the signature to the transaction.

fee = target feerate × assumed transaction size

All underpaying transactions have two outputs. However, during the fee calculation, the size of a one-output transaction is assumed. For example, for a P2PKH 1in ⇒ 2out transaction (226 bytes), the size of a 1in ⇒ 1out transaction (192 bytes) is used. This incorrect assumption results in the transaction only paying around 85% (192 byte / 226 byte) of the recommended feerate. As the transaction inputs make up for a large part of the transaction size, the effect is smaller for transactions with more inputs. This behavior was only present in the Blockchain.com Web wallet. A fix was released on April 21st, 2020.

The overpaying transactions all have a single output. For these, a second output is assumed during the fee calculation. To calculate the fee of a P2PKH 1in ⇒ 1out transaction (192 bytes), the size of a 1in ⇒ 2out transaction (226 bytes) is used. This results in the transaction paying about 118% (226 byte / 192 byte) of the recommended feerate. Similar to the underpaying transactions, the effect is smaller for transactions with more inputs. These transactions are assumed to originate from the Blockchain.com iOS wallet. This has not yet been confirmed.

Out of the set of transactions with the Blockchain.com wallet fingerprint, the transactions paying the feerate recommended by the Blockchain.com feerate estimator are selected. Transactions broadcast on April 19th, 2020, are shown. The y-axis is centered around the regular recommendation, which was 3 sat/vbyte for most of the day. Between 12:00 UTC and 17:00 UTC, the regular recommendation briefly jumped to 4 sat/vbyte for a few minutes each. On other days the feerate recommendations are usually more volatile. April 19th is a Sunday. Sundays are known for less network activity compared to weekdays. This day has been specifically chosen to showcase the methodology.

Identifying Blockchain.com wallet transactions with this methodology is not assumed to be perfectly accurate or reliable. For example, transactions send with a custom feerate can not be identified and are false negatives. Transactions constructed by different wallets that pay a similar feerate and share the fingerprint could be identified as false positives. When the recommended feerate is volatile, which is often the case for the priority recommendation (for example, shortly after the daily BitMEX broadcast), then some transactions might pay a feerate not recoded by the Bitcoin Transaction Monitor. Additionally, the wallets could construct a transaction using an older recommendation, which is different from the recommendation at the time the transaction is broadcast. These transactions are false negatives as well.

Observations

The described methodology is used to identify the transactions send with Blockchain.com wallets between April 1st and May 20th, 2020. The resulting dataset spans over 50 days and contains about 4 million transactions. These pay a total fee of 445.73 BTC and account for about 1.34 GB of block space. Roughly two-thirds of the Blockchain.com wallet transactions target the regular feerate while the remaining third targets the priority feerate.

Roughly the same number of outputs are created as are spend. Blockchain.com wallet transactions have either a single payment-output or a payment-output and a change-output. As the change-outputs are always P2PKH outputs, it is possible to determine the payment-output type. Out of all outputs created about 31.7% are P2PKH, 23.3% are P2SH, 0.34% are P2WPKH, and less than 0.01% are P2WSH payment-outputs. The remaining 45.5% are P2PKH change-outputs. The most commonly used input-output combinations are P2PKH ⇒ P2PKH + P2PKH with 33%, P2PKH ⇒ P2SH + P2PKH with 26%, and P2PKH ⇒ P2PKH with around 7%.

Users of the Blockchain.com wallet are most active between 15:00 UTC and 18:00 UTC and least active between 4:00 UTC and 5:00 UTC. At around 5:00 UTC, the number of transactions per minute starts to rise. At this time it is 8am in Moscow, and 7am in central Europe. Between 5:00 UTC and 10:00 UTC, the number of transactions per minute rises from about 30 to just above 60. The transactions per minute remain constant until rising again at noon UTC, which is 8am on the US east coast. The daily maximum is reached at around 16:00 UTC with just above 75 transactions per minute. From there on, the activity declines until reaching the minimum number of transactions per minute at around 4:00 UTC again.

The transactions broadcast per minute with Blockchain.com wallets are shown. The error bands show the standard deviation. The time between 8am and 8pm is marked for central Asia, Europe, and eastern US timezones.

Reportedly, Blockchain.com claims that their wallets are responsible for one-third of all Bitcoin transactions. They publish the daily number of transactions sent by their wallets. This lead to a discussion on the accuracy and correctness of these numbers. The described dataset can be used to verify this claim. The number of daily transactions in the dataset and the published numbers can be compared. The total number of transactions sharing the fingerprint with the Blockchain.com wallet transactions acts as an upper-bound. The total transactions per day are retrieved from transactionfee.info to calculate Blockchain.com’s share of the network.

The daily transaction count published by Blockchain.com translates into a network share of 30% to 35%. The share of the transactions with the same fingerprint, the upper-bound, is on average about three absolute percent higher. The share of the identified transactions in the dataset is about four to five absolute percent lower than the Blockchain.com reported numbers at around 27% on average. The transactions account for about 13% of the daily fees paid, and 20% of the daily block space used.

However, the numbers reported by Blockchain.com still lie in a reasonable range. There are multiple reasons why the described dataset could contain fewer transactions than are reported by Blockchain.com. Some users might send transactions with a custom feerate. These are not picked up by the described methodology. Furthermore, it’s not clear if the reported numbers include transactions send with the Blockchain.com Wallet API. The API allows users to construct transactions sending to multiple recipients which are not accounted for in the described dataset.

With the knowledge that the Blockchain.com Web wallet underpaid the recommended feerate for transactions with two outputs, and the iOS wallet overpays on transactions with one output, the wallet’s shares can be estimated. For this, the assumption that the ratio of two-output to one-output transactions is similar in all wallets must hold. The Web wallet accounts for one-third and the iOS wallet for half of the Blockchain.com wallet transactions. The Android wallet probably accounts for a majority of the remaining 17%. However, this can not be verified as no data is indicating the share of the Android wallet.

Between April 1st and April 22nd, the two-output transactions send with the Web wallet made up for about a third of all two-output transactions send with Blockchain.com wallets. The shown mean is weighted with the transaction counts. A fix released on April 21st resolved the underpaying behavior for two-output transactions in the Web wallet. It took a few days until the release got deployed.

Between April 1st and May 20th, the one-output transactions send with the iOS wallet made up for about half of all one-output transactions constructed by Blockchain.com wallets. The shown mean is weighted with the transaction counts per input-output combination. It’s unclear why the iOS wallet would account for 60% to 70% of the 4+in ⇒ 1out transactions.

The iOS wallet overpays and the Web wallet underpaid the recommended feerate for some input-output combinations. This is noticeable in the time it takes for a transaction to confirm when targeting the regular feerate recommendation. The overpaying transactions fill up most of the block space, and transactions paying or underpaying the recommended feerate are only included in later blocks. In median, a one-output transaction send with the iOS wallet confirms the fastest. Two-output transactions sent with the Web wallet took the longest. The Web wallet’s effect was the strongest for transactions with one-input and two-outputs, which is the most commonly used input-output combination. These only paid about 85% of the recommended feerate. Transactions send with the Android wallet, iOS wallet transactions with two outputs or Web wallet transactions with one output confirm after the iOS wallet transactions with one-output and before the Web wallet transactions with two-outputs.

Box plots for the different input-output combinations show the distribution of the times it took for the transactions to confirm. The data ranges from April 1st to April 22nd, 2020. On April 21st, the fix for the Web wallet was released. The median time to confirmation is annotated. Not outliners are shown in the box plots.

Paying a slightly higher feerate than the Blockchain.com iOS wallet pays for a one-output transactions could be a good trade-off between a fast confirmation and paying a low transaction fee. An advanced user could target a feerate of, for example, about 120% of the regular recommendation. A miner would include the transaction in a block before any of the Blockchain.com transactions are included. Targeting, for example, 102% of the regular recommendation could be an option too. This would be cheaper, but the transaction might take longer to confirm as the overpaying iOS wallet transactions confirm first. The effectiveness of these strategies might be reduced when the feerate recommendations are volatile during high activity hours.

Taking a closer look at the transactions paying the recommended feerate shows transactions with P2SH outputs pay a slightly higher feerate than transactions with P2PKH outputs. In the Blockchain.com wallets, the assumed size of a single P2PKH output with 34 bytes is used in the fee calculation. P2SH outputs, with 32 bytes, are slightly smaller. Transactions with a P2SH output pay for two extra bytes, when the P2PKH output size is used. This results in transactions with P2SH outputs paying a slightly higher feerate. Something similar can be observed for transactions with a P2WPKH output, which have a size of 31 bytes. These pay for three bytes they do not have. P2WSH outputs take up 43 bytes, and thus transactions with a P2WSH output slightly underpay the recommended feerate as they are not paying for 9 bytes.

Users sending their funds to other wallets or services by creating a transaction that pays to a P2SH or a P2WPKH output unknowingly pay a minimally higher fee than they would have to. On average, these transactions confirm earlier. Transactions with P2WSH outputs pay slightly less than the recommendation and take longer to confirm on average. These effects are probably most noticeable during high activity hours.

Transactions paying exactly the recommended feerate of 3 sat/vbyte on April 19th, 2020, are shown. Between 12:00 UTC and 15:00 UTC, the recommended feerate was briefly at 4 sat/vbyte or more. This is out-of-bounds and not shown. The most common input-output combinations are annotated. It’s visible that transactions with P2SH outputs, here marked in orange, pay a slightly higher feerate that P2PKH transactions. Transactions with P2WSH outputs are out-of-bounds on this graph, and transactions with more than four outputs are not shown.

Some transactions with the same input-output combinations appear multiple times at different feerates and have slightly different sizes. Low and high R-values in the ECDSA signatures can cause a one-byte size difference per input. Some transactions have the same input-output combination and the same size but pay a different fee, even when targeting the same feerate. This is caused by the iOS and Android wallet assuming a different P2PKH input size as the Web wallet. The Web wallet uses 147 bytes, and the Android and iOS wallet both use 149 bytes. P2PKH inputs are usually either 147 or 148 bytes. This depends on the R-value in the signature being either low or high. The sizes assumed by the Android and iOS wallet are incorrect. P2PKH inputs with 149 bytes were only possible when high-S values were allowed in the standardness rules before October 2015. More history on the lengths of ECDSA signatures can be found here.

Privacy

The more information a passive observer can derive from a Bitcoin transaction and public metadata, the worse the impact on the privacy of the sender and the recipient. Being able to identify the sending wallet is an information leak. To improve the privacy of Blockchain.com wallet users and to reduce the effectiveness of the described methodology, the recommended feerate should not be followed as closely, and the wallet fingerprint should be broadened.

A key part of reliably identifying Blockchain.com wallet transactions is to select the transactions that pay exactly the recommended feerates. By introducing randomness in feerates, the transactions mix with non-Blockchain.com wallet transactions. This increases the false-positive rate of the described methodology making it less reliable.

The term anonymity pool is used to describe the set of transactions with the same wallet fingerprint. The more wallets construct transactions with the same fingerprint, the harder it gets to identify the wallet a transaction originates from. This improves the privacy of all Bitcoin users. While the Blockchain.com anonymity pool consists of often more than 100000 transactions per day, the Blockchain.com wallets make up for at least 80% of these transactions. The fingerprint can be broadened in different ways, which increases the anonymity pool size and thus decreases the Blockchain.com’s share. This would have a positive effect on the privacy of Blockchain.com wallet users and the privacy of all other Bitcoin users. To broaden the fingerprint of the Blockchain.com wallet, it could, for example, support receiving to and spending from different address types, time-lock some of the created transactions to the current block height, or set a random transaction version when constructing the transaction.

For advanced users, it might be possible to hide their transactions in the anonymity pool of the Blockchain.com transactions. This could be done by mimicking the Blockchain.com wallet fingerprint and paying exactly the recommended feerate. If done correctly, somebody trying to wallet fingerprint transactions with the described methodology would false positively identify the transaction as sent by a Blockchain.com wallet. Blockchain.com could track which transactions were constructed by one of their wallets and which only try to mimic their wallets. This information could be sold to adversaries.

Personal note: I value the privacy of the individual. I won’t publish the transactions or txids I identified as Blockchain.com transactions. However, a motivated passive listener could easily use the outlined methodology to start tagging Blockchain.com users. I publish the above intending to raise awareness about the issue. Especially the awareness of Blockchain.com, users of the Blockchain.com wallets, and developers of other wallets closely following a feerate estimator. If you think I should have disclosed this privately before publishing it, please let me know - either in private or by calling me out publicly.

While my Mempool Observations are fun to write, the process is very time-consuming. You can support me here, which enables me to spend more of my time researching and writing. If you work for a company interested in improving your on-chain foot- and fingerprint, then I’d be more than happy to chat. If you are working on improving Bitcoin privacy, especially on removing wallet fingerprints, then I’d like to contribute the things I’ve learned starring at my Bitcoin Transaction Monitor. My contact information can be found at the bottom of this page.

I realized that I was not able to send to a bech32 P2WPKH address in the Android wallet when trying to withdraw funds used to test the wallet behavior. It is however possible to send to a bech32 address in the Web wallet. ↩︎

The daily BitMEX broadcast at 13:08 UTC

Mon, 04 May 2020 13:08:30 +0000

At around 13:00 UTC every day, BitMEX, a cryptocurrency exchange and derivative trading platform, broadcasts multiple megabytes of large transactions into the Bitcoin network. This affects the transaction fees paid during European afternoons and US business hours. The transaction size could be greatly reduced by implementing current industry standards in the BitMEX wallet. Once activated, utilizing Schnoor and Taproot combined with output batching seems to be the most promising for improving the transaction count and size.

The observation that BitMEX broadcasts transactions every day at around 13:00 UTC is not novel. The transactions are mainly withdrawals initiated by BitMEX users and some internal UTXO consolidations¹. As part of their wallet security practice, BitMEX reviews and processes all withdrawals by hand. They claim that doing so multiple times a day would be infeasible and worry that spreading the transaction broadcast over the day, which would lighten the burden on the network, could decrease their user’s experience.

BitMEX transactions have a unique fingerprint which originates from their in-house multi-signature wallet solution. All transactions strictly spend P2SH outputs with 3-of-4 multisig redeem scripts. The four public keys used in the redeem script are uncompressed. The hashed and encoded redeem script result in addresses with the prefix ‘3BMex’². BitMEX does not spend SegWit outputs, and all transactions have a version of 2. These properties make it straightforward to recognize BitMEX transactions on the chain and in the mempool.

Observations

A dataset consisting of the transactions observed by the Bitcoin Transaction Monitor between September 2019 and March 2020 is used. In these six months, BitMEX broadcast around 415 000 transactions into the Bitcoin network. Summed together, these take up around 593 MB and pay a total miner fee of 181 BTC. This represents about 2.8% of the total bytes and 3.8% of the total fees broadcast in this period.

The median broadcast contains 2209 transactions, 5688 3-of-4 multi-signature inputs, pays 0.95 BTC in miner fees, and has a total size of 3.16 MB. The input count, the miner fees, and the total broadcast size grow linear with the transaction count. On weekends, fewer transactions are broadcast than on weekdays, and thus the input count, total miner fee, and broadcast size are usually less.

With more than 500 bytes each, the 3-of-4 multi-signature inputs account for most of the transaction size. About 26% of the transactions have one, 36% have two, 32% have three, 3% have four, and about 2% have five or more inputs. The transactions have either one (46%) or two outputs (54%).

The miner fees for withdrawals are not paid by BitMex. They are deducted from the withdrawing users. The transaction feerate does not depend on the input count nor the transaction size. Users choose a miner fee when withdrawing. However, they do not know how many inputs the final withdrawal transaction will spend and thus can not reason about the transaction size and required feerate. All observed transaction fees are a multiple of 10 000 satoshi, which is the minimum step size the withdrawal frontend allows. 10 000 satoshi is the smallest, and most commonly (44%) observed withdrawal fee. It is followed by 100 000 satoshi (30%), 20 000 satoshi (17.5%), and 50 000 satoshi (3%). The most commonly observed feerates, which result from the combination of a user-picked fee and an algorithmically chosen number of inputs, are 17 (24%), 9 (19%), 60 (16%), 12 (10%), 88 (8%) and 18 sat/vbyte (8%).

BitMEX starts processing withdrawals at 13:00 UTC. It takes a few minutes before the transactions are broadcast to the network. On some days, the first transactions can be observed as early as 13:05 UTC. In median, the first transaction arrives at 13:08:30 UTC. It takes about 2:14 minutes in median until all transactions arrive on weekdays and 1:57 minutes in median on weekends.

On three days in the observed six months, the broadcast was delayed by a few minutes. These broadcasts are out-of-bounds for this plot. On the 1st of November, 2449 transactions were broadcast starting at 13:40 UTC, on the 14th of December, 1715 transactions were broadcast at 13:35 UTC, and on the 7th of February, 2794 transactions were broadcast at 13:26 UTC.

The 25th percentile of broadcast transactions takes about 10 minutes to confirm, the 50th percentile about 27 minutes and the 75th for about 71 minutes. The 80th percentile takes about two hours, the 86th for more than three hours, the 92nd over five hours, and the 99th percentile over ten hours.

Effects

Broadcasting multiple megabytes of transactions at various feerate levels has immediate and noticeable effects on the network. The feerate estimators adjust their recommendations, and the wallets using these recommendations set a higher feerate when constructing a transaction. The minimum feerate for block-inclusion rises and the time-to-confirmation spikes.

Feerate estimators correctly react to the thousands of large transactions spread over different feerate levels. They recommend paying a higher feerate to outbid the BitMEX transactions, which take up a significant part of the available space in the next blocks. The median estimated feerates for inclusion in the next, in the next three and the next six blocks, increase sharply at around 13:00 UTC. The next block estimate stays at a high level for a few hours. The three and six block estimates continue to increase to a maximum between 16:00 and 17:00 UTC.

The median estimated feerate is calculated by aggregating multiple estimates. For each feerate estimator, the estimates between September 2019 and March 2020 are grouped by minute of day³. The median feerate estimate is picked and averaged with the median estimates of other estimators for the block target. A list of public feerate estimators with their block targets can be found here.

The feerates of the newly arriving transactions react to the increased estimates. The average feerate sharply increases at around 13:00 UTC and continues to rise over the next hours. At about 15:30 UTC, the daily maximum is reached. From there on, the feerate starts to decrease slowly. It takes until around 21:00 UTC before it sinks below the level of the initial spike. At 13:00 UTC, it is afternoon in Europe and morning in the US. The higher network activity at this time likely amplifies and lengthens the effect of the BitMex broadcast. However, most of the near-vertical increase is presumably caused by BitMEX alone. The effect of the US business hours is visible as a slow increase and decrease over multiple hours.

The feerate relative to midnight UTC is grouped by minute of day. The group average is plotted.

Calculating the additional fees Bitcoin users pay as a result of the BitMEX broadcast is hard. There is no data for days without a broadcast to compare to. However, the magnitude of the effect can be estimated. Therefore, the following assumption about the effect is made: The BitMEX broadcast causes an average feerate increase by 4 sat/vbyte between 13:00 UTC and 21:00 UTC. The blue area in the figure above visualizes the assumption. On average, 42.5 vMB of non-BitMEX transactions arrive in the eight hours between 13:00 UTC and 21:00 UTC. If 4 additional satoshi are paid for each broadcast vbyte, then a total of 1.7 BTC of additional fees are paid by Bitcoin users per day due to the BitMEX broadcast. This represents about 17% of the total fees arriving during the eight hours and about 6.8% of the total daily transaction fees. However, it is unclear if the assumption holds. The estimate can only show the magnitude of the average daily effect. One or even multiple days without a BitMEX broadcast would be needed to further study it.

Both the median block feerate and the 5th percentile block feerate spike and stay at an elevated level before slowly declining to pre-broadcast levels at around 22:00 UTC. Transactions send with a low feerate might take a few hours until they are included in a block during this period. The 5th percentile block feerate represents nearly the minimum feerate included a block while not picking up, for example, the parents of child-pays-for-parent transactions.

The median and 5th percentile block feerate are calculated from the blocks in the blockchain. The observed block arrival time (not the miner timestamp in the block header) is used to group the blocks by minute of day. The group median and the 5th percentile are plotted.

The median time-to-confirmation, the time between a transaction being first observed and its confirmation in a block, spikes between 13:00 and 14:00 UTC and remains at a slightly elevated level until around 21:00 UTC.

Transactions are grouped by the minute of day of their arrival time. The median time-to-confirmation for each minute of day is plotted. Only transactions seen in the mempool and a block are used. Replaced transactions are ignored.

Improvements

The effects can be minimized by reducing the number of bytes broadcast to the Bitcoin network. This can be achieved both by decreasing the transaction count and size of the individual transactions.

The four uncompressed public keys used in every BitMEX input could be replaced by compressed public keys. An uncompressed public key is 65 bytes long and can be encoded in 33 bytes as a compressed public key by leaving out redundant data. Wallets started using compressed public keys as early as 2012, and by 2017 more than 95% of all public keys added to the blockchain per day are compressed⁴. By using compressed public keys, BitMEX could reduce their transaction size by as much as 23%.

Transaction batching could help to minimize the amount of 3-of-4 multisig change outputs being created. Spending the 3-of-4 multisig outputs makes up for most of the block space used by BitMEX. Every output created needs to be spent by supplying three signatures of around 71 bytes each and a redeem script with four uncompressed public keys of 65 bytes each. This totals at around 532 bytes per input. However, an even more block space-efficient alternative would be to use a non-multisig wallet for user deposits, which are periodically consolidated into a multisig wallet. Spending a few high-value 3-of-4 multisig outputs for multiple withdrawals batched together greatly reduces the overall transaction count. Additionally, BitMEX operational costs for manually reviewing and processing a few high-value transactions per day would likely be lower than they currently are for reviewing a few thousand lower-value withdrawals.

Spending SegWit would help as well. The large scripts used in the 3-of-4 P2SH multisig inputs would be placed in the witness data. There they have less effect on the transaction weight. In December 2019, BitMEX mentioned that their priority lies on upgrading their wallet to use P2SH wrapped SegWit. They estimated around 65% in transaction weight savings for an average withdrawal transaction.

Once activated, BitMEX users and the whole network could greatly benefit from BitMEX utilizing Schnorr and Taproot. The three ECDSA signatures in the inputs of BitMEX transactions with around 71 bytes each can be replaced by a single 64 bytes aggregate signature. The four uncompressed public keys of 65 bytes each can be replaced by a single 32 byte tweaked public key. This public key would be an aggregate of the three most commonly used public keys. It could be tweaked for additional spending conditions⁵. A single P2TR (Pay-to-Taproot) input does account for around 57 vbytes. This is an 89% reduction when compared to the current input size. Combined with output batching, this could drastically minimize the on-chain footprint of the BitMEX broadcast. Additionally, Taproot would remove the unique fingerprint of BitMEX transactions, which would, in turn, increase the privacy of BitMEX users.

Conclusion

By using the fingerprint of BitMEX transactions, their footprint on the Bitcoin network is observed and discussed. The daily broadcast has a significant impact on the Bitcoin network and user fees. By utilizing scaling techniques, some of which have been industry standards for multiple years, the impact could be reduced. BitMEX is stepping in the right direction by planning to switch to nested SegWit. They, however, shouldn’t stop there.

Disclaimer: I’ve written this article to educate and inform. It’s published without bad intentions and no malice aforethought against BitMEX. I have not been paid to cover this topic by neither BitMEX nor a competitor nor anybody else. The information and data herein have been obtained from sources I believe to be reliable. I make no representation or warranty as to its accuracy, completeness, or correctness. This article has not been extensively peer-reviewed.

These usually don’t contain many inputs, which is untypical for consolidations. On second thought, these might as well be withdrawals to other accounts on BitMEX. ↩︎
I suspect that BitMEX reuses three of the four public keys of their multi-signature setup for multiple addresses (redeem scripts). A fourth and extra public key is used to iterate the redeem scripts until a ‘3BMex’ prefixed vanity address is found. This is clever but wastes 1+65 bytes for each UTXO spent. Additionally, it might not have any security benefit if the fourth private key is not kept. ↩︎
Minute of day: Imagine a vector with 24*60 (1440) entries. The first would be for minute 00:00, the second for 00:01 the sixty-third for minute 01:02, and the last for minute 23:59. ↩︎
I assume that BitMEX currently accounts for the majority uncompressed public keys added to the blockchain per day. ↩︎
If the second footnote holds, then the tweak might not even be required. Bonus: It could, however, be used to create vanity addresses prefixed with, for example, 'bc1mex' ('bc1bmex' is not possible as the character 'b' can’t be used in the data part of the Bech32 address format). ↩︎

The stair-pattern in time-locked Bitcoin transactions

Mon, 27 Apr 2020 13:00:00 +0000

Some of the regularly used Bitcoin wallets, for example, the Bitcoin Core wallet and the Electrum Bitcoin Wallet, set the locktime of newly constructed transactions to the current block height. This is as an anti-fee-sniping measure and visible as a stair-like pattern when plotting time-locked transactions by their mempool arrival time and locktime. The plot, however, reveals transactions time-locked to a future block height. These should, usually, not be relayed through the Bitcoin network.

Bitcoin transactions can contain so-called time locks as a time-based validity condition. Different types of time locks are used in Bitcoin. The focus here lies on transactions time-locked with a lock-by-block-height to an absolute block height by utilizing the nLocktime field. This field holds a 32-bit unsigned integer and is present in every Bitcoin transaction. A nLocktime value between 0 and 500000000 specifies a lock-by-block-height. The Bitcoin consensus rules define that a transaction with a lock-by-block-height of n can only be included in a block with a height of n + 1 or higher. Likewise, a block with the height n can only include transactions with a locktime of n - 1 or less. At least one input of the transaction must have a nSequence below 0xFFFFFFFF for the time lock to be enforced. Otherwise, the lock is ignored. The Bitcoin Core software does not relay transactions with an enforced locktime higher than the current block height.

Some Bitcoin wallets specify a lock-by-block-height with the current height when creating a new transaction. This makes a potentially disruptive mining strategy, called fee-sniping, less profitable. Fee-sniping is currently not used by miners but could cause chain reorgs in the future. Since the block subsidy declines to zero over time, transaction fees will become the main monetary incentive for miners. If a recent block contains a transaction paying a relatively high fee, then it could be profitable for a larger miner to attempt to reorg this block. By mining a replacement block, the sniping miner can pay out the transaction fees to himself, as long as his block ends up in the strong-chain. The fee-sniping miner is not limited to pick the same transactions as the miner he is trying to snipe. He would want to maximize his profitability by picking the highest feerate transactions from both the to-be-replaced-block and the recently broadcast transactions currently residing in the mempool. The more hashrate a fee-sniping miner controls, the higher is the probability that he will win the block-race. Miners have the risk of losing such a block-race, leaving them without reward.

A constant backlog of transactions paying a similar or even the same feerate would be needed to incentivize miners to move the chain forward. When mining a next block yields roughly the same revenue as sniping the previous, then a rational miner would not start a block-race. However, setting the current block height as a lock-by-block-height forbids a fee-sniping miner to use transactions from the mempool his replacement block. It creates an incentive to move the chain forward to be able to include the next batch of time-locked transactions in a block. While this does not fully mitigate fee-sniping, it reduces profitability. The more transactions enforce a time lock to the current block height, the less profitable fee-sniping is.

Example: Making fee-sniping less profitable by setting the current block height as a lock-by-block-height. Block b2, with a height of n + 1, pays its miner a high fee. Another miner, Eve, attempts to snipe block b2 with block b2*. The mempool includes two transactions paying a relatively high fee of one bitcoin each. However, Eve can only include transaction tx1 in block b2*. Transaction tx2 can not be included in b2* as it is only valid in a block with a hight of n + 2 or more.

When the block height increases, the locktime newly constructed transactions should increase as well. Transactions with the locktime set to the current block height should appear as a stair-pattern when plotted by mempool arrival time and locktime.

Observation

Transactions observed by the Bitcoin Transaction Monitor project on December 17, 2019 are plotted. A stair-pattern of transactions with a lock-by-block-height is visible.

The scatterplot shows a dot for each transaction with a locktime between 608507 and 608519 that was broadcast on December 17, 2019, between 12:06 and 14:08 UTC. The radius represents the feerate the transaction paid. Blocks found in the displayed timeframe (height 608509 to 608519) are drawn.

Some transactions are locked to a height below the current block height. These are likely not broadcast immediately after their creation. This happens, for example, with high-latency mixers and some CoinJoin implementations. Additionally, the Bitcoin Core wallet randomly chooses a locktime up to 100 blocks below the current height for 10% of the signed transactions to increase the privacy of the not immediately broadcast transactions.

The plot, however, does show transactions with two different locktimes arriving at the same time. Transactions time-locked to the current block height and transactions time-locked to the next block height. Between block 608511 and block 608512, for example, transactions with a locktime of 608511, the current height, and 608512, the next block height, arrived. These should, normally, not be relayed trough the network.

Interpretation

The transactions time-locked to the next block height have inputs with a nSequence number of 0xFFFFFFFF. The lock-by-block-height is thus not enforced. All of these transactions have common properties and share a somewhat unique fingerprint. All are spending P2SH, nested P2WSH, or P2WSH inputs and are BIP-69 compliant (inputs and outputs ordered lexicographically). Most of them pay a feerate of 12.3 sat/vbyte and all spending 2-of-3 multisig inputs. This leads to the assumption that these transactions are constructed with the same wallet or by the same entity.

The same transactions as above are shown but transactions spending multisig inputs are highlighted in blue.

The fingerprint, especially spending 2-of-3 multisig, limits the number of wallets or entities which the transactions could originate from. Reaching out to a possible entity, which prefers to remain unnamed, proved to be successful. They confirmed the two off-by-one-errors. Firstly, the transactions should be time-locked to the current block height and not the next block height. Secondly, at least one of the inputs should have a nSequence number of 0xFFFFFFFF-1 or less for the time-lock to be enforced. A fix for this has been released in early 2020. However, it will take a while before all instances of the currently deployed software are upgraded.

Between September 2019 and March 2020, about 10.9 million (19%) out of the 57.49 million total transactions set a locktime. Out of these, 1.16 million transactions had an unenforced locktime. That represents 2% of the total and about 10% of all transactions with a locktime. More than 98% of the transactions with an unenforced time-lock share the same fingerprint as the transactions observed on the 17th of December. The remaining 21577 transactions are likely broadcast by other entities. Out of these, 92.5% have an unenforced lock-by-block-height and 7.5% an unenforced lock-by-block-time. Some transactions with unenforced locktimes might have valid use-cases while others unknowingly set an unenforced locktime.

Conclusion

Some bitcoin wallets reduce the profitability of fee-sniping by time-locking transactions to the next block. This appears as a stair-pattern when plotting the arrival time and the locktime of transactions. The plot reveals transactions with locktimes higher than the current block height. These should, usually, not be relayed through the Bitcoin network. This leads to the discovery that a large entity transacting on the Bitcoin network does not properly set the nSequence field of their transaction inputs. This allowed for the off-by-one-error in the transaction locktimes to remain unnoticed.

I thank David Harding for commenting on an early draft of this article. I found the background information he provided to be very valuable. Any errors that remain are my own.

BIP42 on einundzwanzig Podcast (German)

Wed, 01 Apr 2020 00:00:00 +0000

Dennis Reimann and I chat about BIP42. We talk about the background of BIP42, Satoshi’s implementation of the subsidy function, the bug in the implementation and the fix which enables a finite monetary supply for Bitcoin.

Podcast (German)

Interview on einundzwanzig Podcast (German)

Fri, 20 Mar 2020 00:00:00 +0000

Dennis Reimann and I talk about my projects, the Chaincode Labs Residency, and more on the German einundzwanzig Podcast.

Podcast (German)

transactionfee.info (2020 version)

Fri, 31 Jan 2020 00:00:00 +0000

The website transactionfee.info shows Bitcoin protocol layer statistics. This includes statistics about Bitcoin transactions, their in- and outputs, about blocks and Bitcoin scripts. The project is a joint effort with Bitrefill CEO @ziggamon.

The transactionfee.info website was initially designed in 2018 to raise awareness about the inefficient use of block space by exchanges, services, and wallets. Read about the 2018 version here

We iterated and released the 2020 version as we had reached our goal of raising awareness and saw the usage of the fee efficiency checker decline over 2019. blockstream.info can be used as an alternative to our fee efficiency checker for now.

Some of the new charts I'm excited about below:
https://t.co/VHtSX8o8Fc
— b10c (@0xB10C) January 31, 2020

I was surprised that there is still somebody using uncompressed PubKeys https://t.co/LB8vXLtekv

(yes, for example you @brbtcoficial)
— b10c (@0xB10C) January 31, 2020

Contribution: Colab version of the Optech Schnorr / Taproot Workshop

Wed, 27 Nov 2019 00:00:00 +0000

Bitcoin Optech created a workshop explaining the Schnorr and Taproot upgrade to engineers. However, users needed to compile a patched version of Bitcoin Core with Taproot support and download and set up the Jupyter notebooks.

This causes friction for users who want to quickly try out the notebooks. I’ve contributed the option to run the notebooks in Google Colab, a cloud-based Jupyter notebook environment. This reduced the setup time to less than 20 seconds and the only requirement is a Google account. The links to the notebooks in the Google Cloud can be found in the Bitcoin Optech Schnorr / Taproot Workshop repository.

Our schnorr/taproot workshop notebooks are now available on Google Colab so you can play with taproot without having to clone and build a bitcoind locally: https://t.co/zfeH9YmrLO

Thanks to @0xB10C for setting these up!
— Bitcoin Optech (@bitcoinoptech) November 27, 2019

If you have not tried out the schnorr/taproot workshop demos yet, the Google Colab option has made it really easy!

Thanks @0xB10C ! https://t.co/Eu0hwTovT6
— Mike Schmidt (@bitschmidty) December 5, 2019

Frequently Asked Questions: Bitcoin Transaction Monitor

Thu, 10 Oct 2019 09:09:09 +0000

The Bitcoin Transaction Monitor provides deeper insights into the usage of the Bitcoin network by showing transactions by time and feerate. This post answers frequently asked questions about the Bitcoin Transaction Monitor itself.

Bitcoin Transaction Monitor: transactions plotted over time and feerate

Why did you build a Bitcoin Transaction Monitor?

With Bitcoin, a permissionless network has been created, where everybody can join. Companies, services, and individual users broadcast transactions to the network. Plotting these transactions by arrival time and feerate reveals interesting activity patterns. The Bitcoin Transaction Monitor is built to visualize, share and inform about these patterns. I hope it lets us gain deeper insights into the usage of the Bitcoin network.

Where do you get the data about the transactions from?

I run a Bitcoin Core node connected to the Bitcoin network, which passes valid transactions to memod (mempool observer daemon) over the ZMQ-interface. Each passed transaction is processed by memod and written into a database.

Doesn’t the Transaction Monitor reveal private information about transactions ?

The Bitcoin Transaction Monitor shows activity and usage patterns of the Bitcoin network. This information can be (and probably is already) used by bad actors to weaken the privacy or even completely depseudonymize transactions. Yet this information broadcasted on the Bitcoin network is entirely public. Raising the awareness of what transactions can reveal is far more valuable than hiding public information.

If I can build a Transaction Monitor in my free time that visualizes this data and could run on your laptop, what can a motivated bad actor do with far more resources?

What are future ideas for the Bitcoin Transaction Monitor?

Provided I have the time and come up with an efficient architecture I’d like to archive and display historical data. Additionally, providing a live visualization of incoming transactions would be interesting. There is a GitHub issue which has a few feature suggestions.

Project: Bitcoin Transaction Monitor GitHub issue with feature suggestions

Bitcoin Transaction Monitor

Thu, 10 Oct 2019 00:00:00 +0000

Whenever you, an exchange or somebody else sends a Bitcoin transaction, it gets broadcast to all nodes in the Bitcoin network. Each broadcast transaction is represented by a dot on the Bitcoin Transaction Monitor scatterplot. The transactions are arranged by the time of arrival at my Bitcoin node and its feerate (fee per size). The plot reveals activity patterns of wallets, exchanges, and users transacting on the Bitcoin network.

Building the Transaction Monitor

I first got the idea of plotting Bitcoin transactions by their arrival time and feerate as I was working on my mempool.observer project. Plotting the output of Bitcoin Core’s getrawmempool RPC generated chart below. While there are big white areas of confirmed transactions, there are definitely activity patterns visible. This sparked my interest and I started to dive deeper.

plotting the output of the getrawmempool RPC of Bitcoin Core

My first goal was to find a way to efficiently extract the incoming transactions. Polling the mempool via the getrawmempool RPC was not an option. The RPC can run for multiple seconds if the mempool holds a few thousand transactions which is not too uncommon. I started profiling the RPC but found no obvious way to speed it up. Additionally, by polling, I would miss the confirmed transactions that entered the mempool between my last poll and a new block.

Bitcoin Core can be configured to publish transactions that enter the mempool via a ZMQ interface. These ZMQ messages contain the raw binary Bitcoin transaction. However, as I previously noted in my blog post Plotting the Bitcoin Feerate Distribution, Bitcoin transactions don’t contain the fee they pay as an explicit value. The fee is implicitly set by leaving a bit of the previous output amount to the miner when creating the new outputs. This means that I would have had to query the transaction fee for every transaction that arrived. As this would have made the project quite resource hungry I started thinking about my alternatives.

The best performing alternative I found was to patch my Bitcoin Core instance and to create a custom ZMQ publisher that sends the transaction and the fee. This approach allowed me to just subscribe to my newly added ZMQ publisher and be able to extract this data. The biggest downside here is probably that it meant that this creates a big hurdle for somebody wanting to self-host the Transaction Monitor.

The next step was to build a backend that keeps the last few thousand transactions and attach an API to it for retrieval in a frontend. I choose a Redis sorted-set and used the transaction arrival timestamp as score. This allowed me to quickly retrieve the most recent entries while being able to drop older transactions. I ended up implementing a 30-second cache and gzipped the JSON responses to speed up API calls even more. All in all, this allowed me to reduce the average response time to around 700ms (from more than 12s when starting off) and to respond to concurrent requests with nearly no increase in the response time.

For maximal flexibility, I choose D3.js to visualize the data in the frontend. D3.js comes with a steep learning curve but was the only library that allowed for the interactivity and performance I aimed for. At first, I tried to draw the dots for the transactions as objects in an SVG. However, this is slow with multiple thousand transactions drawn. The alternative was to use an HTML canvas that basically acts as a bitmap image. By using a Canvas the interactivity and the data-bindings that D3.js offers are lost. I ended up using a Quadtree (a tree data structure where each internal node has four children) to find transactions close to the user’s mouse pointer which enabled me to restore the interactivity while keeping the performance high.

I wanted to be able to filter transactions by their properties and thus I wrote a Golang library that allows me to answer questions about raw Bitcoin transactions. This library is called rawtx (project page).

The Bitcoin Transaction Monitor lets you now highlight transactions based on the block they are included in.https://t.co/q1sbHn4Tqy pic.twitter.com/hQcKvocg6O
— b10c (@0xB10C) January 15, 2020

Privacy Implications

I reflected a bit about the privacy implications before publishing the Transaction Monitor. For somebody familiar with the Bitcoin ecosystem, it’s companies, wallets, and users, it’s fairly trivial to attribute some transactions to entities using the Transaction Monitor. This was a point for not publishing it. However, I think there is a bigger total gain for the community in raising the awareness that the pseudonymity sets are small. Users and companies often leave distinct mempool-fingerprints. I’ve asked the following question in my Frequently Asked Questions: Bitcoin Transaction Monitor post as well:

Everything I display is public information. If I can build a Transaction Monitor in my free time that visualizes this data and could run on your laptop, what can a motivated bad actor do with far more resources?

Prior Art

basil00’s TxMon, archived
Conor Scott’s draw_mempool (master), screenshots
Jan Vornberger’s bitcoinmonitor.com

How is the Bitcoin network being used?

I've build a Bitcoin Transaction Monitor to gain deeper insights on the Bitcoin network usage. Transactions are plotted by time and feerate, which reveals interesting activity patterns. https://t.co/CWgyPpdjJo
— b10c (@0xB10C) October 10, 2019

This is a neat new visualization of recent bitcoin transactions that makes it abundantly clear that some services are still hard coding their fee rates - there are no known fee estimators that recommended over 60 satoshis / vbyte during this period. https://t.co/VKJ2ivg6xE pic.twitter.com/0MoYtWkvco
— Jameson Lopp (@lopp) October 10, 2019

ShiftCrypto sponsoring mempool.observer

Tue, 01 Oct 2019 00:00:00 +0000

ShiftCrypto sponsored some of my work on mempool.observer.

rawtx library

Sat, 28 Sep 2019 23:53:07 +0000

The rawtx Golang module helps you (and me) to answer questions about raw Bitcoin transactions, their inputs, outputs, and scripts. I use the rawtx package for example in my Bitcoin Transaction Monitor and transactionfee.info projects.

The package has a high unit test coverage (> 90%) and is battle-tested (all mainnet Bitcoin transactions have been analyzed at least once). However, I’d strongly advise to not use it in a consensus-critical environment.

rawtx on GitHub Documentation on GoDoc

The Incomplete History of Bitcoin Development

Sun, 04 Aug 2019 09:09:09 +0000

To fully understand the rationale behind the current state of Bitcoin development, knowledge about historical events is essential. This blog post highlights selected historical events, software releases and bug fixes before and after Satoshi left the project. It additionally contains a section about the current state of Bitcoin development. The linked timeline provides extra detail for each event.

I wasn’t following the Bitcoin space when a majority of these events happened. A big part of the timeline is adapted from a talk John Newbery gave on the History and Philosophy of Bitcoin Development. The title of this blog post is supposed to remind you that it can’t and doesn’t include every event. History is in the beholder’s eye. History evolves. If you have a suggestion about something missing or want to propose a change, please create an issue in the open-source project bitcoin-development-history, which is used to generate the attached timeline.

With Satoshi

The timeline tells a story beginning in early 2007. Satoshi Nakamoto starts working on Bitcoin. The peer-to-peer electronic cash system with no trusted third party. A system only controlled by the software which its users run.

Early on contributors join Satoshi working on Bitcoin. These new contributors add, next to many other things, support for Linux and macOS to the project. Over the summer of 2010 Satoshi authors a few critical software changes. For example, checkpoints are introduced as a safeguard against malicious peers broadcasting low difficulty chains. A node enforcing these checkpoints rejects chains that don’t include specific block hashes at specific heights. Checkpoints are hard-coded by Satoshi alone which in theory allows Satoshi to control which chain the network follows.

A few days after adding checkpoints Satoshi releases the first consensus change in version v0.3.3. Satoshi urges users to upgrade. Over the next month, multiple minor versions are released. One of them fixes a critical overflow bug. This bug was exploited to create two high-value UTXOs. Satoshi advises miners to reorg the chain containing the blocks with the malicious transactions.

A week later Satoshi introduces an alert system to inform node operators about similar bugs and problems in the network. The alert system includes a safe mode. The safe mode, once triggered, disables all money handling RPC methods in the entire network. Only Satoshi can create valid network alerts by signing them with a private key. Some users raise the question of what could happen when a second party, for example, a government, gets access to this key.

Satoshi has a lot of power over the Bitcoin network at this point. The main concern here isn’t that Satoshi would turn evil and try to destroy the network, but rather that there shouldn’t be such a single point of failure in a decentralized system.

In December 2010 Satoshi opens his last thread on bitcointalk announcing the removal of the safe mode. Satoshi writes in one of his last emails: »I’ve moved on to other things. It’s in good hands with Gavin and everyone.« Some might argue Satoshi stepping away from Bitcoin is one of his greatest contributions.

Without Satoshi

Around the same time, the development process moves from SVN to GitHub, which leads to longtime contributors like TheBlueMatt, sipa, laanwj and gmaxwell joining the project. In mid-2011 the BIP process for Bitcoin Improvement Proposals is introduced. In the last quarter of 2011 and the first months of 2012, the community discusses various proposals, which would allow the receiver of a transaction to specify the script needed to spend it. Out of them, P2SH is merged.

In fall 2012 the Bitcoin Foundation is announced. The Bitcoin Foundation hopes to achieve for Bitcoin what the Linux Foundation does for Linux. Some people raise the fear of development centralization in the announcement thread.

Bitcoin version v0.8.0 is released in spring 2013. Two weeks after the release an unexpected hardfork splits the network in nodes that have and haven’t yet upgraded. The hardfork is resolved fairly quickly. Different miners react by shifting their hashpower to the chain valid for both upgraded and not upgraded nodes.

In late 2013 the Bitcoin software is rebranded to Bitcoin Core. In the following year companies such as Chaincode and Blockstream are founded. Later the MIT Digital Currency Initiative joins Chaincode and Blockstream by paying developers and researchers to work on Bitcoin. In February 2015 Joseph Poon and Tadge Dryja release the first draft of the Lightning Whitepaper. The next year Luke Dashjr revises the BIP process with BIP 2 and the Bitcoin Core release v0.13.0 includes SegWit as a softfork. In November 2016 the alert system is retired and in August 2017 SegWit gets activated. The year 2019 brings a new company, Square Crypto, sponsoring Bitcoin development. In May Pieter Wuille proposes BIP taproot.

The current state of Bitcoin development

Over the years the Bitcoin development culture became more decentralized, well-defined and rigorous. There are currently six Bitcoin Core maintainers, distributed over three continents. Only they can merge commits by contributors. Before commits get merged, however, they have to go through a review process, which has gotten a lot stricter.

For example, a competing proposal, to the earlier mentioned P2SH, was OP_EVAL. The pull request implementing OP_EVAL was merged at the end of 2011. It had only one reviewer, though it changes consensus-critical code. Russell O’Connor opened an issue criticizing parts of the implementation and that such a big and consensus-critical change should have had a lot more review and testing.

This fueled an ongoing discussion on how to ensure higher code quality through more testing and review. Today each pull request should at least be reviewed by multiple developers. If a change touches security-critical or even consensus-critical code, the review process needs many reviewers, a lot of testing and usually spans over multiple months. John Newbery, an active Bitcoin Core contributor, told me that there is “no chance a consensus change would be merged with a single reviewer today”.

A lot of work went into automated testing. There are unit-tests written in C++ and functional-test written in Python. Every non-trivial change should update existing tests or add new ones to the frameworks. Next to unit- and functional-tests, there is an initiative to do fuzz-testing on Bitcoin Core and a benchmarking framework to monitor code performance. The website bitcoinperf.com, for example, offers both a Grafana and a codespeed interface visualizing periodic benchmarking results.

A well-defined release process has been put together over the years. Major releases of Bitcoin Core are scheduled every six months. The schedule includes a translation process, a feature freeze and usually multiple release candidates. Recent efforts by Cory Fields and Carl Dong aim to increase the Bitcoin Core build system security with deterministic and bootstrappable builds. The new build system might not be fully ready for the v0.19.0 release of Bitcoin Core this fall but could provide greater build security in the future.

Conclusion

Over the past ten years, the Bitcoin development culture has changed a lot. Moving from being very centralized around Satoshi to being more decentralized with more than a thousand GitHub contributors in 2018. It has become clear that high standards for code review, code quality, and security are needed. These standards are followed and constantly improved.

I think to fully understand the rationale behind the current state of Bitcoin development, knowledge about historical events is essential. There is a timeline with more events attached below. Some suggested further reading could be The Tao Of Bitcoin Development written by Alex B., The Bitcoin Core Merge Process written by Eric Lombrozo and the blog post by Jameson Lopp: Who Controls Bitcoin Core?

Acknowledgments

I’m thankful for John Newbery helping me to shape and review this blog post. He did a lot of the historical digging for his History and Philosophy of Bitcoin Development talk, which this blog post is based upon. I’m also very grateful for Chaincode Labs inviting me to their 2019 Summer Residency where I met a lot of awesome people, learned a ton and started working on this blog post and the timeline.

Timeline

Timeline: Historical events in the development of Bitcoin

Sun, 04 Aug 2019 09:09:09 +0000

To fully understand the rationale behind the current state of Bitcoin development, knowledge about historical events is essential. I created an open-source project containing the data for a timeline of historical developments in Bitcoin. Most data points are adopted from a talk John Newbery gave on the History and Philosophy of Bitcoin Development. I’ve used this timeline in my blog post The Incomplete History of Bitcoin Development.

This timeline based on an open-source project called bitcoin-development-history. If you have a suggestion about something missing or want to propose a change, please open an issue there.

A List of Public Bitcoin Feerate Estimation APIs

Sat, 29 Jun 2019 14:41:14 +0000

My search for a list of public Bitcoin feerate estimation APIs ended without any real results. Jameson Lopp has a section on feerate estimators on his bitcoin.page and Antoine Le Calvez’s dashboard txstats.com provides a visualization of different estimation APIs. But that is not what I was looking for. That’s why I compiled this list.

I opted to only include publicly advertised feerate estimation APIs by e.g. payment processors and block explorers. I’m purposefully leaving out list APIs by wallets, such as Mycelium and Trezor because their APIs are not publicly advertised. Additionally, I’m leaving out Bitcoin Core’s feerate estimates via the estimatesmartfee RPC. I don’t consider the RPC publicly reachable as in reachable over the web by everyone (some services wrap the estimatesmartfee RPC however).

The following list of public feerate APIs is lexicographically sorted.

bitcoiner.live API

The bitcoiner.live API provides sat/vByte estimates for confirmation in half an hour, 1 hour, 2 hours, 3 hours, 6 hours, 12 hours and 24 hours. It’s reachable under https://bitcoiner.live/api/fees/estimates/latest.

{
 "timestamp": 1563456789,
 "estimates": {
 "30": {
 "sat_per_vbyte": 12.0,
 [ ... ]
 },
 "60": {
 "sat_per_vbyte": 12.0,
 [ ... ]
 },
 "120": {
 "sat_per_vbyte": 8.0,
 [ ... ]
 },
 [ ... ]
 }
}

BitGo API

Bitgo’s feerate API is reachable under https://www.bitgo.com/api/v2/btc/tx/fee and there is documentation available here. The API returns estimates for different block targets in sat/kB. (sat/kB / 1000 = sat/Byte)

{
 "feePerKb": 61834,
 "cpfpFeePerKb": 61834,
 "numBlocks": 2,
 "confidence": 80,
 "multiplier": 1,
 "feeByBlockTarget": {
 "1": 64246,
 "2": 61834,
 "3": 56258,
 [ ... ]
 }
}

Bitpay Insight API

The API of Bitpay’s Insight instance is available under https://insight.bitpay.com/api/utils/estimatefee?nbBlocks=2,4,6. With the parameter nbBlocks the confirmation target in the next n blocks can be specified. Feerates are in BTC/kB. (BTC/kB x 100000 = sat/Byte)

{
 "2": 0.00051894,
 "4": 0.00047501,
 "6": 0.00043338
}

Blockchain.info API

Blockchain.info recommends sat/Byte feerates via https://api.blockchain.info/mempool/fees. They provide a regular and a priority feerate. Additionally a minimum and maximum feerate are included.

{
 "limits": {
 "min": 2,
 "max": 79
 },
 "regular": 4,
 "priority": 53
}

Blockchair API

The Blockchair API offers a transaction fee suggestion in sat/Byte via https://api.blockchair.com/bitcoin/stats. While their API is publicly available for occasional requests, they require an API key for more and periodical requests. You can read more about the API here.

{
 "data": {
 [ ... ]
 "suggested_transaction_fee_per_byte_sat": 1
 },
 [ ... ]
}

BlockCypher API

The BlockCypher API includes a low, medium and high feerate estimate in https://api.blockcypher.com/v1/btc/main. The feerates are in sat/kB. (sat/kB / 1000 = sat/Byte)

{
 [ ... ]
 "high_fee_per_kb": 41770,
 "medium_fee_per_kb": 25000,
 "low_fee_per_kb": 15000,
 [ ... ]
}

Blockstream.info API

Blockstream.info offers an API returning feerates for different confirmation targets in sat/vByte under https://blockstream.info/api/fee-estimates. The API is documented here.

{
 "2": 32.749,
 "3": 32.749,
 "4": 24.457,
 "6": 20.098,
 "10": 18.17,
 "20": 10.113,
 "144": 1,
 "504": 1,
 "1008": 1
}

BTC.com API

BTC.com offers a feerate estimate for the next block in sat/Byte under https://btc.com/service/fees/distribution.

{
 "tx_size": [ ... ],
 "tx_size_count": [ ... ],
 "tx_size_divide_max_size": [ ... ],
 "tx_duration_time_rate": [ ... ],
 "fees_recommended": {
 "one_block_fee": 14
 },
 "update_time": "1563456789"
}

earn.com API

The API behind bitcoinfees.earn.com is reachable under https://bitcoinfees.earn.com/api/v1/fees/recommended. Feerate estimates for the fastest confirmation, a confirmation in half an hour and a hour are shown in sat/Byte.

{
 "fastestFee": 44,
 "halfHourFee": 44,
 "hourFee": 4
}

Please let me know if you know a feerate estimation API I should add to the list.

How Bitcoin Core works: LoadMempool() and DumpMempool()

Fri, 21 Jun 2019 00:00:00 +0000

I’ve talked about the LoadMempool() and DumpMempool() functions of Bitcoin Core at the 2019 Chaincode Labs Summer Residency seminar. These functions are used to write and read the mempool.dat file.

Slides (Google Slides) Slides (PDF)

BIP-42: A finite monetary supply for Bitcoin

Thu, 20 Jun 2019 00:00:00 +0000

I’ve talked about BIP-42 at the 2019 Chaincode Labs Summer Residency seminar. With BIP-42 a fix for a bug in the block subsidy calculation of Bitcoin Core is described. This BIP is meant as an april-fools joke and will only become relevant in 2262.

Slides (Google Slides) Slides (PDF)

BIP-125: Opt-in Full Replace-by-Fee Signaling

Fri, 14 Jun 2019 00:00:00 +0000

I’ve presented about BIP-125 Replace-by-Fee at the 2019 Chaincode Labs Summer Residency seminar. This talk was about the motivation, the history, other proposals and the specification of BIP-125 Replace-by-Fee.

Slides (Google Slides) Slides (PDF)

Building mempool.observer - mempool stats and visualizations

Fri, 14 Jun 2019 00:00:00 +0000

I’ve presented my mempool.observer project at the 93rd NYC BitDevs Socratic Seminar. This included past challenges, current todo’s and future ideas as well as a demo of the site.

Slides (PDF)

mempool.observer (2019 version)

Sat, 01 Jun 2019 23:53:07 +0000

The mempool.observer website displays visualizations about my Bitcoin mempool. For example, a visualization of my current mempool and the historical mempool of my node is shown. The idea is to provide information about the current mempool state to a Bitcoin user with a seemingly stuck and longtime-unconfirmed transaction. Additionally, the site can be used for double-checking feerate estimates before sending a transaction.

I started working on the 2019 version in April 2019. The 2019 version is a full rewrite of mempool.observer - only the idea and license remained. The goal is to offer way more than the 2017 version did, but built on a foundation with performance and maintainability in mind as this was a problem in the 2017 version. Timothy Lim @timothyylim helped out by contributing to the frontend.

Screenshot: current mempool (on a sunday morning)

mempool.observer Source Code on GitHub

In October 2020 I released the Bitcoin Transaction Monitor as a sub-project of mempool.observer. This project is tracked on its own page.

Screenshot: The mempool over the last two hours

Screenshot: recent blocks and the time between them

mempool-dat

Thu, 16 May 2019 00:00:00 +0000

A Golang package that can deserialize Bitcoin Core’s mempool.dat files. This is a toy project. I developed this to learn more about Golang and the mempool.dat file format by Bitcoin Core.

This Go package parses Bitcoin Core’s mempool.dat files. These are automatically written since Bitcoin Core v0.14.0 on shutdown and can be written manually by calling the RPC savemempool since Bitcoin Core v0.16.0.

The package offers access to the mempool.dat’s

- header
- version
- number of transactions
- mempool entries
- raw transaction parsed as (https://godoc.org/github.com/btcsuite/btcd/wire#MsgTx)
- first seen timestamp
- the feeDelta
- and the not-parsed mapDeltas as byte slices

Source on GitHub Documentation on GoDoc

Slides (Google Slides)

c-lightning plugin: csvexportpays

Sat, 02 Mar 2019 00:00:00 +0000

A toy plugin for c-lightning to export all payments made with a c-lightning node to a .csv file. I build this a few days after Blockstream released the plugin support in c-lightning v0.7 to showcase how simple it is to build plugins.

Source on GitHub

screenshot of the plugin in action

I've build a simple python plugin to show how easy it is to build a plugin for @Blockstream's v0.7 release of c-lightning.

Export a your c-lightning payments into a CSV file directly from lightning-cli. ⚡

Clone it at https://t.co/h8XPm86APk #clightning pic.twitter.com/QAaaZlXwxF
— b10c (@0xB10C) March 2, 2019

Support from Bitrefill for transactionfee.info

Tue, 01 Jan 2019 00:00:00 +0000

Bitrefill supported my work on transactionfee.info (now mainnet.observer).

lnplays.com

Sat, 23 Jun 2018 00:00:00 +0000

Konstantin Nick (@sputn1ck) and I build lnplays.com for the Lighting Hackday in June 2018. You could play Pokémon via the Lightning Network build. Pressing the buttons of the GameBoy to move the player and to interact with the world would generate a lightning invoice. Paying that invoice would send the pressed button to the game-backend and the user could see the action over the live stream. The site not longer up, but parts of it can still be seen on archive.org.

screenshot of the user interface of lnplays.com

Thanks for the amazing #LightningHackday @fulmolightning. Now all of you should have time playing Pokemon on https://t.co/IudNbyY3i9 made by @0xB10C and me
— kon (@sputn1ck) June 25, 2018

Plotting the Bitcoin Feerate Distribution

Sun, 18 Mar 2018 12:30:00 +0000

How did the median Bitcoin feerate evolve from 2013 to 2015? When were the feerate spikes in 2017? Visualizing the Bitcoin feerate distribution per block was on my todo list since I’ve started working on the first version of my mempool.observer project in mid-2017. But acquiring the data wasn’t as easy as I first thought.

Update January 2020: The feerate chart have are not included the 2020 version of transactionfee.info so some links in this post might be broken.

In this post, I describe my unsuccessful approaches and explain my final solution. I’ll go into beginner to intermediate technical stuff on Bitcoin. I hope this post might help people who are just starting out on small Bitcoin projects (i.e. me a few months ago).

feerate distribution over time

Motivation

Our site transactionfee.info lets a user check the fee efficiency of a bitcoin transaction they’ve send or received. Besides SegWit and output batching we wanted to provide an indicator when a service drastically overpays the feerate. We needed data on the feerate distribution per block to determine what a low, but reasonable feerate would have been to be included in the same block. David Harding suggested plotting the feerate across ten-percentile intervals to see which percentile is suitable.

Querying feerate data over the RPC interface

My first idea was to query each block over the Bitcoin Core RPC interface. The query getblock blockhash 2 returns the corresponding block with all its transactions as JSON. For the feerate distribution per block, I’d need to iterate over each transaction and calculate its feerate. The feerate is specified as the transaction fee divided by the transaction vsize $$(2)$$. The fee is the sum of all input amounts deducted by the sum of all output amounts $$(1)$$.

fee = sum(input amounts) - sum(output amounts)
feerate = fee / vsize

However, querying each block with its transactions is not enough. An input only references a previous transaction output (TXO). It doesn’t contain the actual input amount. This can be solved by simply querying the TXO. RPC-call batching brings huge performance improvements here.

I coded a small python script and used a local testnet node to generate a first data set. Everything worked fine and I quickly had a basic chart setup to share my initial proof of concept. However, I didn’t think about the testnet versus mainnet performance of that script. Since blocks on mainnet contain more transactions and more inputs per transaction the script ran ages querying data from my HDD. I eventually stopped it and started looking for alternative approaches.

Alternatives

Reading the raw .dat files

I looked at multiple blk*.dat-file parsers to build a TXO-index for quicker lookups. This turned out to be a dead end. I wanted to automatically update the data every few hours on a small VPS, but the output of the index would probably have been more than a hundred gigabytes. This was not an option for me.

Google’s BigQuery Bitcoin Blockchain data set

When I stumbled over Google’s BigQuery Bitcoin Blockchain data set I thought this would solve all my data problems, but was quickly disappointed with the limited data that’s available. I miss the reference to the TXO in the input table. The data set might be well suited for other projects and one plus point definitely is the ten-minute update interval, but I, at least for this project, have no use for it.

BlockSci

Then I found out about BlockSci: “A high-performance tool for blockchain science and exploration”. It seems to do exactly what I wanted to do. At the time I looked into it, it didn’t use the vsize for feerate calculation. This would have resulted in an incorrect feerate for SegWit transactions. However, a few days ago v0.4.0 was released which uses the vsize for feerate calculations. If you are interested in working with data from the bitcoin blockchain I can highly recommend taking a look at their paper.

Patching Bitcoin Core and -reindexing

As a final solution, I included a simple LogPrintf in validation.cpp of Bitcoin Core. I print an identifier to grep for followed by the block height and the feerate for each non-coinbase transaction into the debug.log separated by a comma. The patch diff is only a few lines. After compiling and a full -reindex of the blockchain my debug.log contained all blocks with the feerate of each transaction. Grepping the blocks from the debug.log leaves me with about a gigabyte of raw data (around 100MB gziped).

log-file with the feerates for the transactions in block 510851

Using the dataset

I have a pruned node running on a small VPS and insert the percentile data into a database. I then automatically generate the .csv-files which are displayed on transactionfee.info/charts. In my next step, I’ll connect the database to an indicator on transactionfee.info.

I guess I’ll be using the raw data to experiment a bit with Gnuplot in the future. I have a neat idea for that. And retrospectively rating feerate estimation algorithms might be an idea, too. Feel free to contact me if you are interested in the data set.

transactionfee.info (2018 version)

Mon, 22 Jan 2018 00:00:00 +0000

We build transactionfee.info in 2018 to raise awareness about the inefficient use of block space by exchanges, services, and wallets. The project is a joint effort with Bitrefill CEO @ziggamon.

In early 2018 the price of bitcoin surged to 20.000 USD. At that time we saw more than 400.000 daily transactions on the Bitcoin network which caused a steep increase in transaction fees. A lot of exchanges, services and wallets did not use the block space efficiently. We build transactionfee.info to raise community awareness.

input field of the 2018 version transactionfee.info site

Only a few exchanges and services made use of payment batching or spend SegWit outputs in 2018. This meant, for example, that every withdrawal was costly for exchanges as the withdrawal transaction takes up a lot of space in the blockchain. Often the transaction fees or even a higher fixed fee was deducted from the user’s withdrawal amount. Our site allowed users to check the fee efficiency of their transactions and their wallets. It calculated how much fees could have been saved by using, for example, payment batching or by spending SegWit outputs. Green checkmarks for fee- and space-saving behavior and red crosses for inefficient and wasteful behavior.

WayBackMachine: 26th August 2018

Check out https://t.co/H6LeHNaZax.

It’s a new tool I’ve helped create (code by @0xB10C). Helps you see how well optimized bitcoin transactions are.

Just paste a tx id and see if you’re overpaying for your bitcoin transactions and withdrawals.

Feedback welcome! pic.twitter.com/4jUMGky7SS
— Sergej Kotliar (@ziggamon) January 22, 2018

Samourai users are paying the lowest transaction fees according to https://t.co/ntB5Sz3mww - Don't use wallets that abuse their users. No Android device? Use @GreenAddress - If you don't want to use Samourai install @ElectrumWallet - This isn't marketing, this is common sense. pic.twitter.com/vGhSnf9shv
— Samourai Wallet (@SamouraiWallet) February 4, 2018

Later, we added a subsection with charts about payment and feerate metrics to the site. We offered all data under a CC0 “No Rights Reserved” license. The transactionfee.info website has appeared in numerous online news articles since then, one, for example, being thenextweb.com: Bitcoin’s median transaction fee lowest since 2011 — nearing BCH. The transactionfee.info data even made its way into academia.

Presenting “Payments” per day, a more accurate representation of real economic activity on the Bitcoin network.

(It’s a count of all outputs that are not change outputs back to the sender)https://t.co/kyuTMXz09A

Props @0xB10C pic.twitter.com/5Z1d5PQ0gQ
— Sergej Kotliar (@ziggamon) February 11, 2018

The potential block space efficiency gains from transaction batching are even greater than from implementing SegWit. Thus far the services that have implemented batching are imperceptible in the grand scheme of things. Plenty of room for improvement! https://t.co/uPSWiaXdUT pic.twitter.com/bDj85dmXQ7
— Jameson Lopp (@lopp) February 22, 2018

New feature on https://t.co/2G9sgi1yL5: Bitcoin transaction fees distribution by percentile.

You can see the fee rates rising and and falling and which fees are “high” vs “low” at any given time.https://t.co/usCSbZZQIH pic.twitter.com/L9ntWw0oNv
— Sergej Kotliar (@ziggamon) March 31, 2018

Due to low usage of the fee efficiency checker over 2019 and us reaching our goal to raise awareness we decided to overhaul the site and released a second iteration in 2020.

The 300 MB default maxmempool Problem

Mon, 18 Dec 2017 16:16:16 +0000

Unconfirmed transactions are quite a hassle for bitcoin users. I recently came across an interesting problem which is not the usual “my transaction is stuck” problem.

In the last weeks of 2017, the number of unconfirmed transactions has again reached all-time highs. Periods with over 100k transactions in my mempool were not unusual. Bitcoin Core limits the system memory allocated for storing unconfirmed transactions to 300 MB by default. This serves as an anti-DoS feature. Fully used 300 MB of RAM equal about 110 MB of actual raw transaction data. This default value can be changed with the -maxmempool <n> option where n is the number of megabytes allocated to store unconfirmed transactions.

 $ bitcoin-cli getmempoolinfo
 {
 "size": 123803, # amount of tx
 "bytes": 107840125, # raw tx bytes
 "usage": 298526272, # tx in ram bytes
 "maxmempool": 300000000, # maxmempool in bytes
 "mempoolminfee": 0.00016486 # min tx-fee to get accepted
 }

Once 300 MB of system memory is filled, Bitcoin Core starts dropping low-feerate transactions from the mempool for high-feerate transactions. Additionally, a mempoolminfee-threshold is set to prevent new low fee transactions from entering the mempool.

A node operator can increase or decrease the maxmempool option to better fit their system. I assume that for example, some block-explorers have increased their nodes maxmempool. I know that Jochen Hoenicke, for example, has a node running with a maxmempool of 2GB.

The Problem

Small differences in the mempool are common. However, with an increased maxmempool a nodes mempool can differ vastly from a majority of other nodes on the network. These nodes, especially when they power a block-explorer, still many keep unconfirmed transactions the majority of the network has already dropped. This can be irritating for users.

I came across a particular form of this problem, when a friend asked me, why he can see a certain low-fee transaction on bitaps.com, but not on blockchain.info. My local node, with a default 300 MB maxmempool, responded with an error on calling the getrawtransaction RPC with the txid.

 $ bitcoin-cli getrawtransaction <txid>
 error code: -5
 error message:
 No such mempool transaction.

However, using the bitaps.com API to query the raw transaction worked fine. I tried rebroadcasting the raw transaction with sendrawtransaction. This failed.

 $ bitcoin-cli sendrawtransaction <rawtx>
 error code: -25
 error message:
 Missing inputs

Looking at the missing input I noticed that it references an unconfirmed change output from a transaction with equally low fees. However, I again wasn’t able to find the parent transaction for the input in my local mempool or on blockchain.info. Only bitaps.com had it. And this transaction again referenced an unconfirmed low-fee output as an input.

With the context of the maxmempool-limit this started to make sense.

My take on what happened

My take on what happened is, that bitaps.com has a higher maxmempool-limit than e.g. blockchain.info and therefore does not drop the transactions. A user checking bitaps.com sees the transaction, a user checking blockchain.info doesn’t.

The real issue appears when the original sender is not aware of this and has a peer with a larger-than-default mempool. He unknowingly continues to chain transactions to an unconfirmed parent transaction that the vast majority of the network rejects. The broadcasted transaction spends an output that references a dropped transaction and therefore is invalid.

David Harding pointed out that the sender could use BIP-125 Replace-By-Fee to update an unconfirmed transaction with new outputs:

mempool.observer (2017 version)

Sat, 07 Oct 2017 23:53:07 +0000

The mempool.observer website displays statistics about my Bitcoin mempool. This covers the 2017 version which I iterated on in 2019.

The mempool.observer website was my first open-source Bitcoin project. I started working on it in mid-2017 and officially released it on the 7th of October 2017. At the end of 2017, the Bitcoin transaction fees sky-rocketed which resulted in a lot of traffic to my site. The high transaction fees were caused by a flood of transactions as the price of bitcoin rose to $20.000. I regularly ran into problems with long-running Python scripts due to querying and processing mempool on a low-end VPS. Due to time constraints, I wasn’t able to improve the backend performance. This resulted in version 2017 of mempool.observer dying down in 2018 due to not being maintained. I started working on a next version of the site in 2020 with performance and maintainability as key goals.

Source Code on GitHub for 2017 version 2017 version on archive.ph

Screenshot: Low-feerate transactions are evicted from my mempool during the high-fee event in 2017. My mempool contains more that 100k transactions.

version 1.0.0 of memo is live! 👓https://t.co/9mQp1L4CzF https://t.co/NMBMD3gMUX
— mempool.observer (@mempoolobserver) October 7, 2017

b10c's blog: Bitcoin Developer on b10c's blog

localprobe: detect Bitcoin nodes running on the same machine as your (Firefox) browser

Recap: A network wide DoS bug in Bitcoin Core

Monitoring Workshop at Vinteum BDL 2026

OpenSats Report 8

What did you work on?

Publications / Podcasts / Talks

BNOC

peer-observer

tooling (peer-observer/peer-observer)

infrastructure library (peer-observer/infra-library)

Bitcoin Core

bitcoin/bitcoin

bitcoin-core/guix.sigs

0xb10c/mainnet-observer

misc

rust-bitcoin/bip324

nixos/nixpkgs

0xb10c/nix

rust-bitcoin/corepc

0xb10c/github-metadata-backup

bitcoin-data/block-arrival-times

bitcoin-data/stale-blocks

OpenSats Work-Log 8

What did you work on?

Publications / Podcasts / Talks

Bitcoin Network Operations Collective

peer-observer

tooling (peer-observer/peer-observer)

infrastructure library (peer-observer/infra-library)

Bitcoin Core

bitcoin/bitcoin

bitcoin-core/guix.sigs

0xb10c/mainnet-observer

misc

rust-bitcoin/bip324

nixos/nixpkgs

0xb10c/nix

rust-bitcoin/corepc

0xb10c/github-metadata-backup

bitcoin-data/block-arrival-times

bitcoin-data/stale-blocks

Bitcoin Network Monitoring with b10c: SLP707

Monitoring Bitcoin’s P2P network: Insights from the peer-observer Project

OpenSats Work-Log 7

How did you spend your time?

Publications & Talks

Bitcoin Core

peer-observer

Tool

Infrastructure

Bitcoin Network Operations Collective (bnoc.xyz)

Misc

What do you plan to work on next quarter?

Support from LocalhostResearch for peer-observer

OpenSats Work-Log 6

How did you spend your time?

Publications:

mainnet-observer

peer-observer

fork-observer

misc

peer-observer: A tool and infrastructure for monitoring the Bitcoin P2P network for attacks and anomalies

Motivation

The peer-observer tool

Interfaces

Extractors and Tools

logger tool

metrics tool

websocket tool

The peer-observer infrastructure

Some peer-observer findings

A Bitcoin Network Operations Collective

Notes on 'DoS due to inv-to-send sets growing too large' from May 2023

Observation

Effect

Cause

Fix

Reflection

peer-observer: A tool and infrastructure for monitoring the Bitcoin P2P network for attacks and anomalies

`logger` tool

`metrics` tool

`websocket` tool