This is a copy of the 4th work-log I sent to OpenSats for my LTS grant.

What did you work on?

Publications

• My mininpool-observer project detected 15 OFAC sanctioned transactions missing from blocks. After analyzing them to rule out false-positives, I concluded that F2Pool is filtering transactions with OFAC sanctioned addresses again: https://b10c.me/observations/13-missing-sanctioned-transactions-2024-12/
• I was asked by a fellow developer if I happen to have any insights into Bitcoin Core orphanage overflows. To make them available to everyone, I posted them to delving: Stats on orphanage overflows
• I updated my stats on compact block reconstructions on delving with the newest data. This sparked new discussion and interest from other developers: https://delvingbitcoin.org/t/stats-on-compact-block-reconstructions/1052/5

Bitcoin Core self-hosted CI

Inspired by discussions at the last Bitcoin CoreDev meeting in fall 2024, I started looking into a NixOS based configuration for the Bitcoin Core self-hosted CI runners. At the time, I discovered that the self-hosted runners were running under a privileged user, which could easily stop/vandalize/otherwise negatively affect the underlying machine. CI tasks weren’t isolated from the machine and from other tasks. Note that a CI is basically one of your servers where you give someone from the internet Remote-Code-Execution access. Additionally, the CI token could easily be leaked, which allowed anyone to spawn new, and potentially malicious CI runners for the Bitcoin Core project.

This made is it worthwhile to spend a bit of time looking into a potential CI runner setup that is more secure, properly isolates CI tasks, and doesn’t leak the CI token. By choosing NixOS, the CI runners can be configured once as infrastructure-as-code, and then easily deterministically replicated across multiple hosts.

To isolate individual CI jobs from each other, I choose to run a ephemeral QEMU VM for each job. For this, the cirrus-ci runner used by Bitcoin Core needs to stop after it completed a single job (ephemeral mode). I opened a PR for this, but sadly, there hasn’t been much review activity by the Cirrus-CI folks: https://github.com/cirruslabs/cirrus-cli/pull/813.

The hardest part is to do caching of CI build inputs in ephemeral runners. The current Bitcoin Core CI runners aren’t ephemeral because caching dependency sources and built dependency artifacts, docker base images and task specific docker, previous releases, and ccache artifacts is important for both CI performance and resilience against e.g. network problems or rate-limiting. Managing these across CI jobs can be challenging, especially when a single CI job shouldn’t be able to clear the cache. I’ve written down some notes about this in https://github.com/bitcoin/bitcoin/issues/30852#issuecomment-2558198430.

The project is about 85% done, but still requires some work to get to a production ready-level. Based on discussions with other Bitcoin Core developers, the current CI situation might have been improved since last fall: CI jobs don’t run as privileged user anymore and other safeguards have been put into place. The plan is to present my CI setup at the upcoming CoreDev and evaluate if it makes sense to invest more time and energy into this project in the short-term.

I’ve published the CI-runner setup in https://github.com/0xB10C/bitcoin-core-cirrus-runner and my infrastructure in https://github.com/0xB10C/bitcoin-core-cirrus-runner-infra.

During development, I noticed that it is time consuming to manually parse the Bitcoin Core CI logs. To help during my development and to provide stats for the Bitcoin Core project, I’ve build a tool and website that parses and gives an overview over the recent tasks. The repository can be found at https://github.com/0xB10C/bitcoin-core-ci-stats and the website here https://0xb10c.github.io/bitcoin-core-ci-stats/.

On the Bitcoin Core side, I initially proposed https://github.com/bitcoin/bitcoin/pull/31377, which was then superseded by https://github.com/bitcoin/bitcoin/pull/31545. This makes the caching of the Bitcoin Core CI docker images possible.

Bitcoin Core

• Opened a “Tracepoint Interface Tracking Issue” to track open tasks that could be done to improve the tracepoint interface https://github.com/bitcoin/bitcoin/issues/31274
• Got two of my long-open Bitcoin Core tracepoint PRs merged: https://github.com/bitcoin/bitcoin/pull/26593 and https://github.com/bitcoin/bitcoin/pull/25832
• Did some testing on “ci: detect outbound internet traffic generated while running tests #31349” and found that on some systems the tests still make DNS requests, which can be used to identify developers running the tests https://github.com/bitcoin/bitcoin/pull/31349#issuecomment-2499335672
• Helped Suhas with tracing related changes in https://github.com/bitcoin/bitcoin/pull/31122#discussion_r1820876152
• Quick fixup https://github.com/bitcoin/bitcoin/pull/31419

peer-observer

A tool used to monitor for attacks and anomalies by hooking into the Bitcoin Core tracepoints.

• With Bitcoin Core # tracing: Only prepare tracepoint arguments when actually tracing #26593 merged, I opened Support PID / PID file to attach tracepoints to bitcoind #58 and addressed it in add: cli arg to specify a bitcoind PID #61 and add: use bitcoind PID in peer-observer #75. This made sure peer-observer stayed compatible with the changes to Bitcoin Core.
• Added a feature to warn and exit if the peer-observer extractor didn’t receive any events for a while. Implemented in https://github.com/0xB10C/peer-observer/pull/63
• Refactored peer-observer to use a NATS message server instead of an NNG (nanomessage) PUB-SUB. This allows to have multiple publishers instead of a single one. I’ve been planning to work on a RPC publisher in combination with the current tracepoint publisher. https://github.com/0xB10C/peer-observer/pull/76
• A python tool to record getblocktxn msgs used for https://delvingbitcoin.org/t/stats-on-compact-block-reconstructions/1052/21?u=0xb10c: https://github.com/0xB10C/peer-observer/pull/106
• websocket tool maintenance: https://github.com/0xB10C/peer-observer/pull/100
• nix-shell maintenance: https://github.com/0xB10C/peer-observer/pull/96

fork-observer

Tool to visualize forks and reorgs on various Bitcoin networks.

• testnet4 reorg mining improvements for sjors: show header difficulty and highlight difficulty 1 blocks #50
• esplora backend (blockstream.info, mempool.space) support https://github.com/0xB10C/fork-observer/issues/52 and https://github.com/0xB10C/fork-observer/pull/55
• added a legend explaining the tip status colors: https://github.com/0xB10C/fork-observer/issues/53 and https://github.com/0xB10C/fork-observer/pull/54
• to display changes quicker to reduce desync, the backend now notifies the frontend about all changes: https://github.com/0xB10C/fork-observer/pull/57
• btcd would send chain tips in random order, which confused fork-observer into thinking it the btcd tips changed. Fixed by sorting the tips before comparing them: https://github.com/0xB10C/fork-observer/pull/58

miningpool-observer

Transparency for Mining Pool Transaction Selection

• general maintenance update rawtx-rs and bitcoin-pool-identification dependecies #86
• add missing-sanctioned page https://github.com/0xB10C/miningpool-observer/pull/89
• Use unique but non-sequential block id in database for easier importing of backups: https://github.com/0xB10C/miningpool-observer/issues/91 and https://github.com/0xB10C/miningpool-observer/pull/92
• remove dead block explorer links https://github.com/0xB10C/miningpool-observer/issues/88

my nix packages

Collection Nix packages and NixOS modules of software I’ve written or software I use. Allows others to easily run my tools too.

• integration test for asmap.dat files/package https://github.com/0xB10C/nix/pull/71
• added a features that checks for new versions of packages in a nightly CI run and automatically opens a PR for the new version with me as a reviewer in https://github.com/0xB10C/nix/pull/63
• other maintenance and package updating: https://github.com/0xB10C/nix/commits?author=0xB10C&since=2024-11-01&until=2025-01-31

Misc

• maintained my ofac-sanctioned-digital-currency-addresses repo with, for example, https://github.com/0xB10C/ofac-sanctioned-digital-currency-addresses/pull/18
• further maintenance of btcffm.org (https://github.com/btcffm/website-btcffm/pull/16, https://github.com/btcffm/website-btcffm/pull/17, https://github.com/btcffm/website-btcffm/pull/15)
• contributed my GUIX signatures for two Bitcoin Core release candidates: https://github.com/bitcoin-core/guix.sigs/pull/1484, https://github.com/bitcoin-core/guix.sigs/pull/1455

What do you plan to work on next quarter?

• Evaluate (e.g. at CoreDev) if makes sense to continue working on the self-hosted Bitcoin Core CI runners or if other projects might have a higher priority for me for now.
• Continue working on open issues for my current projects like fork-observer, miningpool-observer, peer-observer and others
• Continue working on the Bitcoin Core tracing interface. See e.g. https://github.com/bitcoin/bitcoin/issues/31274
• Further analyzing data and research posts on delving (e.g. compact block reconstruction and more)