bitcoin++ workshop: Tracing Bitcoin Core v23.0Wednesday, June 8, 2022
These are the tasks of my “Tracing Bitcoin Core v23.0” workshop for bitcoin++ 2022. They might not make much sense on their own as participants used a pre-setup VPS. The workshop slides can be found here. This branch was used during the workshop.
You will only need an SSH client to participate in the workshop. I will provide you with a pre-setup VPS to work on the tasks. I’ll send you the IP for your VPS. However, I first need an SSH public key from you. Please make sure not to send the private key part of your SSH key! I’d recommend generating a new, temporary key pair for this workshop. You should be able to run the following command on Linux and macOS to generate a new ED25519 keypair.
$ ssh-keygen -t ed25519 -C "bpp22-workshop" Generating public/private ed25519 key pair. Enter file in which to save the key (/home/your_user/.ssh/id_ed25519): /home/your_user/.ssh/id_bpp22workshop Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/your_user/.ssh/id_bpp22workshop Your public key has been saved in /home/your_user/.ssh/id_bpp22workshop.pub The key fingerprint is: SHA256:FaK3FinGErPr1nTG63msKLoVSU9syTPpQSGGdsnpfvZM bpp22-workshop The keys randomart image is: +--[ED25519 256]--+ + snip + +----[SHA256]-----+
Please send the
/home/your_user/.ssh/id_bpp22workshop.pub file to me via
e-mail, telegram, or similar. I’ll respond with your server name. Once the
VPSs are set up, I’ll publish the IPs below.
|server name (BIP39 word)||IP||assigned|
Task 0 - Login and get familiar with the system
Once you’ve got your IP address, you should be able to SSH into the server via
the following command. You might need to specify the SSH private key file
.pub at the end).
$ ssh email@example.com -i ~/.ssh/id_bpp22workshop
Additionally, you might need to set the correct permissions (only read+write for your current user) on the SSH private key.
chmod 600 /home/your_user/.ssh/id_bpp22workshop
Once you’ve logged in to the system, it’s time to get familiar with it.
Feel free to navigate around, check out the specs and running processes with
htop, and check the kernel and operating system information with
The kernel version is important for using a lot of the eBPF functionality.
We need at least a Linux kernel of version 4.8 to use hook into the tracepoints
in Bitcoin Core. The programs
nano are also installed.
Note that you have password-less
sudo access on the system. We need a
privileged user to do BPF syscalls and to read from BPF maps. I expect you to
not abuse this power (e.g. deleting system files). Treat this server as you
would treat a server you’ve rented. The server auto-shutdowns on excessive
resource usage. Additionally, your disk space is limited (please don’t try to
sync mainnet). I might not provision another server for you or restart your
instance. This would effectively end your participation in the workshop.
For the following tasks, it could be helpful to take a look at the Bitcoin Core tracing documentation and examples:
- Tracing documentation: doc/tracing.md
- Tracing examples: contrib/tracing
- Which Linux kernel version does the provided server have?
- How much memory does your server have?
- Are you able to switch to the
Task 1 - Listing available tracepoints
There is a
bitcoin directory in your home directory. This contains a checkout
of the Bitcoin Core source code. I’ve also built Bitcoin Core
from source for you to save some time during the workshop. During the
./configure step, it was detected that the SystemTap
sys/sdt.h header file
is present on the system. Bitcoin Core was built with tracepoints.
$ ./configure [..] Options used to compile and link: multiprocess = no with experimental syscall sandbox support = yes with libs = yes [..] with zmq = yes with test = yes with fuzz binary = yes with bench = yes use asm = yes USDT tracing = yes # <---- Tracepoints enabled! sanitizers = debug enabled = no [..]
Task: Your task is to list the available tracepoints in the
binary. The Bitcoin Core tracepoint documentation (linked above) contains three
methods to do so. Feel free to try them out. The
bitcoind binary is located
Try finding one or two tracepoints in the Bitcoin Core source code. I’d
recommend that you navigate to the
*.cpp files from there (
grep TRACE *.cpp -R).
The ripgrep tool (
rg) is installed on the system too.
- Which tracepoints does the binary contain?
- Which of the three methods shows you information about the arguments passed to the tracepoints?
- Are all tracepoints documented in the tracing documentation?
- Were you able to find a tracepoint in the Bitcoin Core source code?
Task 2 - Running
p2p_monitor.py script is intended to demonstrate the possibilities the
tracepoints offer. It shows the inbound and outbound traffic between the Bitcoin
Core node you’re hooking into and it’s peers in real-time. Under the hood, it
tracepoints, the Python BCC wrapper, and curses to render a TUI (Text User
We’ll be hooking into a testnet Bitcoin Core node already running on the system.
To execute BPF syscalls and read from BPF maps, we’ll need a privileged
user. Here, we use the root user. You can become root by executing
As root, navigate into the
/home/workshop/bitcoin directory. From there, you
can run the following command to start the
p2p_monitor.py script. We pass the
path of the system-wide
bitcoind binary we want to hook into (not the one in
$ python3 contrib/tracing/p2p_monitor.py $(which bitcoind)
You’ll be able to navigate with UP/DOWN and select a peer with SPACE to see individual P2P messages being exchanged. It might take a few seconds for peers to appear as there is generally a lot less traffic on testnet compared to mainnet. The script only learns about a connected peer once we send or receive a message from it. Keep the script running for a minute or two.
This is also documented in contrib/tracing/README.md - p2p_monitor.py if you want to revisit running this at a later point.
- How many peers is your node connected to?
- Does your node have any inbound peers? What are the connection types of your peers?
- Are you able to observe, for example, a handshake with a peer or a ping-pong?
Task 3 - Tracing a Unit Test
When building a
bitcoind binary with tracepoints enabled, the Bitcoin Core
unit tests and benchmarks include these tracepoints, too, if they call the code
sections containing the tracepoints. When adding the tracepoints, we’ve made
sure to check that the tracepoints cause only minimal overhead of a few CPU
instructions if we don’t hook into the tracepoint.
eBPF is programmable. In the previous task, the tracing script sent the
data passed in tracepoint arguments back to userspace. However, for example,
we can further filter the data or use it in calculations.
builtin functionality like, e.g.,
count(), and creating
Bitcoin Core’s unit test suite has a test called
coins_test/coins_cache_simulation_test. It runs the function
SimulationTest(). This is a large randomized insert/remove
simulation test for the in-memory UTXO cache of Bitcoin Core.
We can trace this test with the
Run the test from inside the
bitcoin directory (as
CTRL-D to exit from
sudo su) with the following command.
$ ./src/test/test_bitcoin --run_test=coins_tests/coins_cache_simulation_test --log_level=test_suite
Task: Using the
contrib/tracing/task-3-stub.bt as a
starting point, add functionallity to count how many UTXOs are added and removed
(spent) during the unit test. Additionally, create a histogram of the UTXO
values being added and removed.
Hint: Where do I start?
Take a look at the documation for the bpftrace functions
hist() linked above.
Start your tracing script in the
bitcoin directory as root user. In another
SSH session, run the unit test. Stop the tracing script once the test is done.
This should print the
bpftrace counters and histograms.
$ bpftrace contrib/tracing/task-3-stub.bt
- How many UTXOs are added and spent during the test?
- What are the most common values of UTXOs added and removed?
- Which binary is the bpftrace script hooking into?
- What data does the fourth argument (i.e.
arg3; zero-indexed) of the
spenttracepoints contain? (Hint: it’s documented)
Task 4 - Running the tracepoint interface tests
We want the tracepoint API to be semi-stable. That means there shouldn’t be unexpected breaking changes to the API, and tracepoint arguments should be documented. However, as tracepoints expose internals, we might need to change or drop the tracepoints if we refactor or drop Bitcoin Core internals. We test them in the Bitcoin Core functional test suite written in Python to ensure that the tracepoints don’t break. We use the BCC Python wrapper.
Task: Run the interface functional tests. There are two ways of running
the tests. You can either run them through
or execute the standalone Python scripts. I recommend running them as
standalone Python scripts first to see the eventual reasons the tests are
skipped. Once you’re sure they aren’t skipped (i.e., they pass), you can run
Running the tests as standalone Python scripts:
$ python3 test/functional/interface_usdt_net.py $ python3 test/functional/interface_usdt_utxocache.py $ python3 test/functional/interface_usdt_validation.py $ python3 test/functional/interface_usdt_coinselection.py
Running the tests through
$ python3 test/functional/test_runner.py --filter=interface_usdt_*
- Do the tests pass?
- We require permissions to do BPF syscalls and read BPF maps for the tests. What happens when you run the tests with the
- Is blindly running Python scripts downloaded from the internet as root user on your own machine a good idea?
Task 5 - Adding new tracepoints
Here, the goal is to get familiar with adding a new tracepoint to Bitcoin Core. You’ll find documentation on this in doc/tracing.md. This also includes a few guidelines and best practices. Make sure to familiarize yourself with the documentation and to ask questions if something is unclear!
For long-running tracing tools like bitcoind-observer it’s useful to know when the Bitcoin Core instance we’re hooking into is stopped and started. This can, for example, be used to reset the statistic counters on a restart. These tracepoints don’t need to pass specific data.
Task: Add two new tracepoints. The first should trigger when
Bitcoin Core is shut down, and the second should trigger when Bitcoin Core
is started. Think about a name for
event (see docs on “Adding
tracepoints to Bitcoin Core”) that give a good description for a user not
familar with Bitcoin Core internals.
Hint: Where should I put the shutdown tracepoint?
src/init.cpp file contains a function called
Shutdown(). Take a look at this function.
Hint: Where should I put the start-up tracepoint?
src/init.cpp file contains a function called
AppInitMain(). Take a look at this function.
You’ll need to recompile Bitcoin Core for your tracepoints to appear in the
bitcoind binary. In the
/home/workshop/bitcoin directory, run the following
command as the
workshop user. This will bring the Bitcoin Core build
dependencies into scope.
Then, you’ll only need to run
make to compile Bitcoin Core. We don’t have
to re-run the autogen and configure steps. Compiling Bitcoin Core might take a
minute or two.
You can use the
bpftrace script in
contrib/tracing/task-5-stub.bt to test
your tracepoints. However, you’ll first need to fill in the names you’ve
event. Then, run the script with the following
command as root user (
$ bpftrace contrib/tracing/task-5-stub.bt
Open another SSH session to your VPS to start and stop Bitcoin Core. From the
bitcoin directory, use the following command to start Bitcoin Core in regtest
mode. Wait a few seconds and then stop it with
$ ./src/bitcoind -regtest
- What name did you choose for
- Where did you choose to place your tracepoints in the functions resposible for startup and shutdown?
- Is your shutdown tracepoint being triggered in case Bitcoin Core does not shutdown cleanly? If not, would a tracepoint for this make sense?