In this post, I revisit my plans and projects for 2022, and give an outlook for 2023. I plan to continue my current Bitcoin network monitoring efforts in 2023. I briefly touch on potentially tight open-source developer funding in 2023.

Projects

In 2022, I pushed 582 commits to GitHub, opened 72 issues, proposed 68 pull requests, spoke at four (un)conferences, and appeared on one podcast. However, I don’t think these numbers express what has kept me busy in 2022. In this post, I want to introduce some of my current projects, mention their status, discuss recent progress, and lay out planned work for 2023.

I was fortunate to be a Brink.dev grantee in 2022, which allowed me to focus on my projects with very little administrational overhead and no need to worry if and when I’ll receive my next grant payment. That’s how you should do open-source developer grants! When Brink asked me to summarize the projects I plan to work on in 2022, I replied with the following.

[..] I plan to continue work on tracepoints in Bitcoin Core, including interface tests, review, and the addition of further tracepoints. Another goal is to launch a Signet with regular reorgs and a tool to explore reorgs visually. After observing flooding attacks on the Bitcoin P2P network in the summer of 2021, I set out to build a P2P network monitoring and anomaly detection tool in 2022. The goal is to feed the observations into Bitcoin and Bitcoin Core development, improving the P2P network robustness. I also plan to write up further Mempool Observations and start a similar series of blog posts for P2P network observations.

Tracepoints for Bitcoin Core

We’ve been adding a tracing interface to Bitcoin Core. This allows for greater observability of process internals during development and production use. For example, we can attach to a tracepoint in the P2P network message processing and learn in real-time about which peer sends us which network messages. This opens up many new possibilities for debugging, monitoring, evaluating changes, and detecting anomalies or other problems. We mainly use eBPF and Userspace, Statically Defined Tracing (USDT) on Linux systems to hook into the tracepoints. I’ve previously introduced this topic in my blog post Userspace, Statically Defined Tracing support for Bitcoin Core.

There has been much progress on the Bitcoin Core tracing framework in 2022. After we added the tracing framework and followed up with the first tracepoints in 2021, I started 2022 by ensuring that the tracepoints are included in release builds and added functional tests for the tracepoints to provide a semi-stable interface between releases. Getting these tests to run in the CI containers was more complex than expected, as the CI’s container kernel headers often don’t match the host’s kernel headers. The headers are needed to compile the eBPF bytecode when using the BPF compiler collection (bcc). A solution was to run the tests in a VM. I opened a PR to add tracepoints for opened, closed, evicted, and misbehaving connections and helped with a PR for mempool tracepoints. End of November 2022, I opened a PR that would eliminate the (minimal) overhead from tracepoint argument computation for users not using the tracepoints. This would also allow expanding the tracepoints to, for example, pass serialized transactions to tracing scripts. Currently, we make sure to avoid potentially expensive computations just for the tracepoints not to affect the majority of users not using the tracepoints. While I think Bitcoin Core’s tracing framework is still experimental, it’s moving in the right direction. I’m trying to keep this list of tracing-related issues and pull requests up to date.

I had the opportunity to hold a Tracing Bitcoin Core workshop at bitcoin++ 2022 in Austin, Texas. Together with jb55, who initially proposed a PR using eBPF to trace Bitcoin Core, I did an impromptu talk on the tracing framework, and its use cases at BTCAzores 2022.

I plan to propose further tracepoints, for example, some which allow observing changes to the IP address manager’s state. I’ve also looked into the kernels libbpf, which is an alternative to the current bpftrace and bcc tooling to develop tracing scripts. There is the possibility to replace the current Python and bcc tracepoint interface tests with concise and faster unit tests using libbpf. Instead of spinning up a full node in the functional test, we could directly test the function with the tracepoints. It might make sense to extract the example tracing scripts from contrib/tracing/ into a separate tracing-tools repo which allows us to, for example, add CI testing for the scripts. Seemingly, with each new bpftrace version, one of the existing example bpftrace scripts break. An option might be to drop the bpftrace example scripts moving forward. I might do a video workshop or similar format that goes into the details of eBPF, USDT, and the Bitcoin Core tracing framework to transfer some of my knowledge about it.

P2P monitoring

As my most ambitious project of 2022, I’ve been working on a tool for passive P2P monitoring using honey-pot nodes to detect anomalies and attacks on the Bitcoin network. We want to be aware of these attacks, anomalies, potential bugs, and other problems to react to them if needed. This is a reaction to the addr message spam during the Summer of 2021, which was only noticed by coincidence. It would be good to have some monitoring for this.

On a technical level, the monitoring utilizes the tracepoints mentioned above to extract P2P events like receiving a message, a connection being opened, or a misbehaving peer. These events are published into a message queue to which other applications can subscribe. One example application is a Prometheus metrics endpoint used in a Grafana dashboard. A demo version of this is running on public.peer.observer.

In 2023, I’m planning to deploy more instances of this tool to different regions with different configurations: for example, a mix of networks (IPv4, IPv6, Onion, I2P, …), P2P features (block filter, bloom filter, P2Pv2, …), and bitcoind versions (releases, master, P2P PRs). I’m using NixOS for declarative (infrastructure as code) and reproducible system deployments, which reduces the infrastructure maintenance overhead to a minimum.

I also plan to build out a P2P data archival tool. Next to disk-space efficient storage of the P2P traffic, this should include functionality to read, replay, and filter archival data. There’s also the idea of a Grafana Live dashboard with real-time metrics. The work on this ties in with proposing further P2P-related tracepoints to Bitcoin Core.

This project includes a “detect ➞ maybe analyze ➞ maybe react” part, which happens ad-hoc when we notice an anomaly or problem. For example, I’ve looked into and have written about an anomaly I’ve come across: Inbound connection flooder down. As a reaction, there’s the possibility of publishing an open-source banlist (similar to Greg Maxwell’s old banlist) with the IPs of the inbound connection flooder. I have notes for a few anomalies that I haven’t analyzed closely yet. I hope to get to them at some point in 2023.

Reorgs on Signet and fork-observer

In 2021, I worked with a Summer of Bitcoin mentee on “Reorgs on Signet”. Testnet can often be unreliable with block storms and frequent reorgs. With signet, the idea is to have a network that can be reliably unreliable. A part of this is to have, for example, automatic reorgs to test Bitcoin software behavior. Though, something like this hasn’t been deployed yet.

While I initially planned to rebase the old branch where we added functionality for periodic reorgs to Bitcoin Core’s signet miner during the Summer of Bitcoin, I didn’t get to it as it didn’t seem like a priority. There hasn’t been much interest from the broader community to use this. In many Bitcoin software projects, the reorg behavior is probably either tested in a regtest test suite or not tested at all due to large reorgs being very infrequent nowadays on the Bitcoin mainnet. It’s unclear to me if a signet with scheduled reorgs would currently see much use.

One artifact from working on the signet miner script is a tool to visualize the header tree with its forks and stale tips. This tool is now called fork-observer. It collects chain tip information from Bitcoin Core and btcd nodes via the getchaintips RPC and then builds a header tree. A single fork-observer instance can handle multiple networks with multiple nodes.

The fork-observer project is still work in progress. While a large chunk of it is done, there are a few missing pieces, such as RSS feeds for stale and offline nodes and some UI improvements. Next to the RSS feeds, bots posting events to, e.g., nostr relays or Twitter are planned. Ideally, I want to combine the tool’s release with a custom signet with reorgs to show its potential.

The fork-observer tool only needs access to the getchaintips, getblockheaders, and getblockhash RPC calls. These can be whitelisted in Bitcoin Core for a fork-observer-specific user. This allows connecting to many external nodes via a private network like, for example, a WireGuard tunnel to one fork-observer instance. Though, it’s certainly possible to run a fork-observer instance on, for example, a nix-bitcoin, start9 Embassy, or an Umbrel node connected to a single mainnet node.

The tool is similar to BitMEX Research’s forkmonitor.info. I think fork-observer is easier to self-host and benefits from the header tree visualization. However, forkmonitor.info has additional features, for example, checking for supply mismatches and listing interesting lightning transactions. It should be possible to add some of the header tree visualizations to forkmonitor.info too, which I might attempt at some point.

The tool also works well together with a bitcoin-data project I’ve started. The goal is to collect historical stale block data with header information where possible. I’m purposefully holding off on publishing this data until a Bitcoin Core PR is merged and released. However, if you have a long-running node (>5 years), feel free to get in touch with me if you want to provide data. I’m using the fork-observer database to feed new data into the dataset.

You can find a development version of fork-observer at fork.observer.

mininigpool-observer

In 2021, I started working on a miningpool-observer project that provides insights into mining pool transaction selection. This has been running on miningpool.observer since May 6th, 2021, when Marathon Pool mined their first block, claiming not to include OFAC-sanctioned transactions. In Observing Bitcoin Mining Pools, I’ve written about the project in more detail.

My work on miningpool-observer in 2022 has been mainly maintenance, smaller features, and bug fixes. I don’t expect this to change in 2023. I’m thinking about a few changes that allow for easier self-hosting and reduce the maintenance burden going forward. There are a few dependencies, such as the ofac-sanctioned-digital-currency-addresses repository and the known-mining-pools repository, which I plan to work on further. We’re currently combining the efforts of my known-mining-pools and the mempool/mining-pools dataset. My goal is to have a high quallity dataset that applications can use attribute blocks to mining pools like some block explorers already do.

Once Stratum v2 gains wider adoption, and job negotiation with the pools is possible, it would be interesting to poke at some Stratum v2 mining pools with non-standard transactions and transactions to sanctioned addresses. This could be incorporated into the miningpool-observer at some later point, depending on Stratum v2 adoption.

bitcoin-data

I’ve set up the @bitcoin-data organization on GitHub to archive some public but ephemeral bitcoin data, which could be helpful for research purposes at some point.

There’s a block-arrival-times repository that contains timestamps for when a node first saw a block. These timestamps are ephemeral as the timestamps encoded in the block headers almost always don’t match the time the block was broadcast to the network. We can extract the block arrival times from, for example, Bitcoin Core’s debug.log files. For instance, I’ve needed block arrival times when looking at which block height was active in The stair-pattern in time-locked Bitcoin transactions. There are other applications, too, like measuring block propagation timings across the network. If you have an old debug.log, feel free to check out the adding data section. There are details on how to parse and add data to the repository. Otherwise, just send it to me.

Another repository is the bitcoin-stats-archive. This currently fetches, and archives [KIT DSN Bitcoin Monitoring] data and luke-jr’s Bitcoin node counts.

Other smaller projects for 2023

I run a lot of my infrastructure, including many of the projects listed above, on NixOS. This allows me to build and package my projects reproducibly and declaratively define what services run on which hosts. This has been very reliable for me, and I’ll continue to use this setup. I plan to share the Nix packages and modules for my projects, making it easier for others to self-host them. I still have to figure out how to publish these and how to incorporate this into my current infrastructure repository.

I’ve been running a mempool visualization website on mempool.observer for a few years now, and I’ve recently set up a quick and dirty extension for full-RBF monitoring on fullrbf.mempool.observer. I could extend the main site with proper visualizations of, for example, RBF replacements, transaction packages in the mempool, size- and time-based evictions, and more. Though, I’m not sure if I’ll get to it. I’ve recently seen that there’s ongoing work to, for example, incorporate [RBF replacement timelines] into the mempool.space project. My time might be better spent helping to review this work rather than building something similar from scratch.

Over the past years, I’ve been able to use my monitoring tools and the data I’ve collected to provide data-based feedback to proposed changes. For example, I could reproduce the simulated bandwidth savings promised by Erlay, run detailed IBD benchmarks for PR #20827, and visualize RBF replacement data to provide insights into an RBF edge case. I plan to help out with similar smaller projects wherever possible. Looking at network data can often reveal possible improvements, as with #26526 and #26527. There are also a few more Bitcoin Network Observations I’m planning to write.

I’ve set up a technical, open-content Bitcoin developer blog as an experiment in 2021. This experiment succeeds when people start (cross) posting their blog posts to it on their own, and it fails once I don’t think it’s relevant anymore and stop paying for the domain. For more information, see bitcoin-dev.blog. While the blog didn’t see much activity in 2022, there’s a plan to revive it in 2023.

I won’t be able to finish or even get to all projects I’ve listed here in 2023. Similar to the last years, ad-hoc projects will pop up, and others will lose their relevance for a while. I often finish the projects I’ve started but might put them on ice for a while if I need time to focus on something I deem to be more urgent.

Developer funding in 2023

Over the past years, there have often been more funding opportunities than qualified¹ open-source Bitcoin developers. However, this has changed with big donors affected by the current market situation. Among others, for example, Gemini’s superlunar fund, funding many important developers and projects, recently shut down. There’s a chance that we’ll see a few grantors unable to renew grants for existing developers resulting in an even tighter funding situation with more qualified developers than funding. Though, at the moment, there is still funding available. I recommend starting looking for alternatives as early as possible if you think your grantor might not be able to renew your grant.

Personally, I’ve found a replacement for my Brink grant with a Spiral developer grant, which allows me to continue working on the projects mentioned above.

0xB10C is a Bitcoin developer focusing on observability in the Bitcoin network. He has built monitoring tools for mining pool transaction selection, P2P network anomalies, transaction replacements, forks and reorgs, and Bitcoin protocol statistics extracted from the blockchain. He aims to feed insights gained from the monitoring tools and collected data into Bitcoin development. Learn more about his projects on b10c.me/projects.