These are the tasks of my “Tracing Bitcoin Core v23.0” workshop for bitcoin++ 2022. They might not make much sense on their own as participants used a pre-setup VPS. The workshop slides can be found here. This branch was used during the workshop.

Prerequisites

You will only need an SSH client to participate in the workshop. I will provide you with a pre-setup VPS to work on the tasks. I’ll send you the IP for your VPS. However, I first need an SSH public key from you. Please make sure not to send the private key part of your SSH key! I’d recommend generating a new, temporary key pair for this workshop. You should be able to run the following command on Linux and macOS to generate a new ED25519 keypair.

$ ssh-keygen -t ed25519 -C "bpp22-workshop"
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/your_user/.ssh/id_ed25519): /home/your_user/.ssh/id_bpp22workshop
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/your_user/.ssh/id_bpp22workshop
Your public key has been saved in /home/your_user/.ssh/id_bpp22workshop.pub
The key fingerprint is:
SHA256:FaK3FinGErPr1nTG63msKLoVSU9syTPpQSGGdsnpfvZM bpp22-workshop
The keys randomart image is:
+--[ED25519 256]--+
+      snip       +
+----[SHA256]-----+

Please send the /home/your_user/.ssh/id_bpp22workshop.pub file to me via e-mail, telegram, or similar. I’ll respond with your server name. Once the VPSs are set up, I’ll publish the IPs below.

Task 0 - Login and get familiar with the system

Once you’ve got your IP address, you should be able to SSH into the server via the following command. You might need to specify the SSH private key file (without the .pub at the end).

$ ssh workshop@13.37.83.33 -i ~/.ssh/id_bpp22workshop

Additionally, you might need to set the correct permissions (only read+write for your current user) on the SSH private key.

chmod 600 /home/your_user/.ssh/id_bpp22workshop

Once you’ve logged in to the system, it’s time to get familiar with it. Feel free to navigate around, check out the specs and running processes with htop, and check the kernel and operating system information with uname -a. The kernel version is important for using a lot of the eBPF functionality. We need at least a Linux kernel of version 4.8 to use hook into the tracepoints in Bitcoin Core. The programs vim and nano are also installed.

Note that you have password-less sudo access on the system. We need a privileged user to do BPF syscalls and to read from BPF maps. I expect you to not abuse this power (e.g. deleting system files). Treat this server as you would treat a server you’ve rented. The server auto-shutdowns on excessive resource usage. Additionally, your disk space is limited (please don’t try to sync mainnet). I might not provision another server for you or restart your instance. This would effectively end your participation in the workshop.

For the following tasks, it could be helpful to take a look at the Bitcoin Core tracing documentation and examples:

• Tracing documentation: doc/tracing.md
• Tracing examples: contrib/tracing

Questions:

1. Which Linux kernel version does the provided server have?
2. How much memory does your server have?
3. Are you able to switch to the root user with sudo su?

Task 1 - Listing available tracepoints

There is a bitcoin directory in your home directory. This contains a checkout of the Bitcoin Core source code. I’ve also built Bitcoin Core from source for you to save some time during the workshop. During the ./configure step, it was detected that the SystemTap sys/sdt.h header file is present on the system. Bitcoin Core was built with tracepoints.

$ ./configure
[..]
Options used to compile and link:
  multiprocess    = no
  with experimental syscall sandbox support = yes
  with libs       = yes
  [..]
  with zmq        = yes
  with test       = yes
  with fuzz binary = yes
  with bench      = yes
  use asm         = yes
  USDT tracing    = yes  # <---- Tracepoints enabled!
  sanitizers      =
  debug enabled   = no
[..]

Task: Your task is to list the available tracepoints in the bitcoind binary. The Bitcoin Core tracepoint documentation (linked above) contains three methods to do so. Feel free to try them out. The bitcoind binary is located in /home/workshop/bitcoin/src/bitcoind.

Try finding one or two tracepoints in the Bitcoin Core source code. I’d recommend that you navigate to the /home/workshop/bitcoin/src directory and grep for TRACE in *.cpp files from there (grep TRACE *.cpp -R). The ripgrep tool (rg) is installed on the system too.

Questions:

1. Which tracepoints does the binary contain?
2. Which of the three methods shows you information about the arguments passed to the tracepoints?
3. Are all tracepoints documented in the tracing documentation?
4. Were you able to find a tracepoint in the Bitcoin Core source code?

Task 2 - Running p2p_monitor.py

The p2p_monitor.py script is intended to demonstrate the possibilities the tracepoints offer. It shows the inbound and outbound traffic between the Bitcoin Core node you’re hooking into and it’s peers in real-time. Under the hood, it uses the net:inbound_message and net:outbound_message tracepoints, the Python BCC wrapper, and curses to render a TUI (Text User Interface).

We’ll be hooking into a testnet Bitcoin Core node already running on the system. To execute BPF syscalls and read from BPF maps, we’ll need a privileged user. Here, we use the root user. You can become root by executing sudo su.

As root, navigate into the /home/workshop/bitcoin directory. From there, you can run the following command to start the p2p_monitor.py script. We pass the path of the system-wide bitcoind binary we want to hook into (not the one in the src/ directory).

$ python3 contrib/tracing/p2p_monitor.py $(which bitcoind)

You’ll be able to navigate with UP/DOWN and select a peer with SPACE to see individual P2P messages being exchanged. It might take a few seconds for peers to appear as there is generally a lot less traffic on testnet compared to mainnet. The script only learns about a connected peer once we send or receive a message from it. Keep the script running for a minute or two.

This is also documented in contrib/tracing/README.md - p2p_monitor.py if you want to revisit running this at a later point.

Questions:

1. How many peers is your node connected to?
2. Does your node have any inbound peers? What are the connection types of your peers?
3. Are you able to observe, for example, a handshake with a peer or a ping-pong?

Task 3 - Tracing a Unit Test

When building a bitcoind binary with tracepoints enabled, the Bitcoin Core unit tests and benchmarks include these tracepoints, too, if they call the code sections containing the tracepoints. When adding the tracepoints, we’ve made sure to check that the tracepoints cause only minimal overhead of a few CPU instructions if we don’t hook into the tracepoint.

eBPF is programmable. In the previous task, the tracing script sent the data passed in tracepoint arguments back to userspace. However, for example, we can further filter the data or use it in calculations. bpftrace has builtin functionality like, e.g., sum(), count(), and creating histograms with hist().

Bitcoin Core’s unit test suite has a test called coins_test/coins_cache_simulation_test. It runs the function SimulationTest(). This is a large randomized insert/remove simulation test for the in-memory UTXO cache of Bitcoin Core. We can trace this test with the utxocache tracepoints.

Run the test from inside the bitcoin directory (as workshop user, CTRL-D to exit from sudo su) with the following command.

$ ./src/test/test_bitcoin --run_test=coins_tests/coins_cache_simulation_test --log_level=test_suite

Task: Using the bpftrace script contrib/tracing/task-3-stub.bt as a starting point, add functionallity to count how many UTXOs are added and removed (spent) during the unit test. Additionally, create a histogram of the UTXO values being added and removed.

Start your tracing script in the bitcoin directory as root user. In another SSH session, run the unit test. Stop the tracing script once the test is done. This should print the bpftrace counters and histograms.

$ bpftrace contrib/tracing/task-3-stub.bt

Questions:

1. How many UTXOs are added and spent during the test?
2. What are the most common values of UTXOs added and removed?
3. Which binary is the bpftrace script hooking into?
4. What data does the fourth argument (i.e. arg3; zero-indexed) of the add and spent tracepoints contain? (Hint: it’s documented)

Task 4 - Running the tracepoint interface tests

We want the tracepoint API to be semi-stable. That means there shouldn’t be unexpected breaking changes to the API, and tracepoint arguments should be documented. However, as tracepoints expose internals, we might need to change or drop the tracepoints if we refactor or drop Bitcoin Core internals. We test them in the Bitcoin Core functional test suite written in Python to ensure that the tracepoints don’t break. We use the BCC Python wrapper.

Task: Run the interface functional tests. There are two ways of running the tests. You can either run them through test/functional/test_runner.py or execute the standalone Python scripts. I recommend running them as standalone Python scripts first to see the eventual reasons the tests are skipped. Once you’re sure they aren’t skipped (i.e., they pass), you can run them through test_runner.py.

Running the tests as standalone Python scripts:

$ python3 test/functional/interface_usdt_net.py
$ python3 test/functional/interface_usdt_utxocache.py
$ python3 test/functional/interface_usdt_validation.py
$ python3 test/functional/interface_usdt_coinselection.py

Running the tests through test_runner.py:

$ python3 test/functional/test_runner.py --filter=interface_usdt_*

Questions:

1. Do the tests pass?
2. We require permissions to do BPF syscalls and read BPF maps for the tests. What happens when you run the tests with the workshop user?
3. Is blindly running Python scripts downloaded from the internet as root user on your own machine a good idea?

Task 5 - Adding new tracepoints

Here, the goal is to get familiar with adding a new tracepoint to Bitcoin Core. You’ll find documentation on this in doc/tracing.md. This also includes a few guidelines and best practices. Make sure to familiarize yourself with the documentation and to ask questions if something is unclear!

For long-running tracing tools like bitcoind-observer it’s useful to know when the Bitcoin Core instance we’re hooking into is stopped and started. This can, for example, be used to reset the statistic counters on a restart. These tracepoints don’t need to pass specific data.

Task: Add two new tracepoints. The first should trigger when Bitcoin Core is shut down, and the second should trigger when Bitcoin Core is started. Think about a name for context and event (see docs on “Adding tracepoints to Bitcoin Core”) that give a good description for a user not familar with Bitcoin Core internals.

You’ll need to recompile Bitcoin Core for your tracepoints to appear in the bitcoind binary. In the /home/workshop/bitcoin directory, run the following command as the workshop user. This will bring the Bitcoin Core build dependencies into scope.

$ nix-shell

Then, you’ll only need to run make to compile Bitcoin Core. We don’t have to re-run the autogen and configure steps. Compiling Bitcoin Core might take a minute or two.

You can use the bpftrace script in contrib/tracing/task-5-stub.bt to test your tracepoints. However, you’ll first need to fill in the names you’ve choosen for context and event. Then, run the script with the following command as root user (sudo su).

$ bpftrace contrib/tracing/task-5-stub.bt

Open another SSH session to your VPS to start and stop Bitcoin Core. From the bitcoin directory, use the following command to start Bitcoin Core in regtest mode. Wait a few seconds and then stop it with CTRL-C.

$ ./src/bitcoind -regtest

1. What name did you choose for context and event?
2. Where did you choose to place your tracepoints in the functions resposible for startup and shutdown?
3. Is your shutdown tracepoint being triggered in case Bitcoin Core does not shutdown cleanly? If not, would a tracepoint for this make sense?