Before the first probe

Two launches, two preloaded libraries

A shared library is a file of compiled code that can be loaded into a program. On Linux these files commonly end in .so. The dynamic linker is startup code that finds and loads the libraries a program needs.

LD_PRELOAD is an environment variable that asks the dynamic linker to load an additional library when it starts a dynamically linked program. bpftime uses it to put its code into each of our two processes.

The bpftime command-line tool wraps this mechanism for the common workflow. In the first terminal, bpftime load launches your loader with the syscall-server library. In the second, bpftime start launches the target with the agent.

# Terminal 1: the BPF loader lives here.
bpftime load ./loader ./probe.bpf.o ./target

# Terminal 2: the target and agent live here.
bpftime start ./target

The loader in this lab is a C program using libbpf, a library for loading and attaching BPF programs. Ordinarily those loading operations go to the kernel. The preloaded libbpftime-syscall-server.so handles the supported operations through bpftime instead. Despite its name, this “server” is library code inside the loader.

The other library, libbpftime-agent.so, goes inside the target. It opens the shared experiment state, prepares the BPF programs, and uses Gum to install their function hooks.

The command named load takes an executable loader command. In this example loader opens probe.bpf.o. Passing the BPF ELF object itself as though it were a native executable confuses two different build products.

For function hooks, the lower-level form is:

# Terminal 1
LD_PRELOAD="$HOME/.bpftime/libbpftime-syscall-server.so" \
  ./loader ./probe.bpf.o ./target

# Terminal 2
LD_PRELOAD="$HOME/.bpftime/libbpftime-agent.so" ./target

These commands are alternatives to the CLI form, not additional steps to run afterward. Both processes must use compatible bpftime builds, the same name for the shared-memory region, and the same target file for attachment resolution.

A description becomes a live hook

The agent must turn the loader’s request into something executable in this process. It opens the existing shared memory, prepares the requested BPF programs and their helpers, finds the target function, and asks the selected hooking library to install the hook.

On Ubuntu, this startup path uses the GNU C library, called glibc. bpftime’s wrapper establishes the order bpftime_agent_main → original main. It does not establish that every kind of application initialization happens after agent setup. Constructors and library startup have their own ordering, and hooks can fire during initialization in other workloads. Our lab’s first do_work invocation is deliberately in main, after the wrapper has run.

The target also has to find the function in its own memory. A compiled file’s code can be loaded at different addresses in different runs. The loader therefore describes a location in a binary or library; the target determines where that location sits in its own address space. This avoids assuming that an address observed in the loader means the same thing in the target.

What changes with late attachment?

bpftime attach PID injects the native agent into an existing process and invokes its entrypoint. The target has already passed main startup, so the preload wrapper is not how the agent gets control. It still needs access to compatible shared state and still must prepare and install the requested hooks.

Injection permissions depend on ownership and the host’s ptrace/security policy. A successfully injected library is only one milestone; it does not prove that a symbol resolved, a BPF program prepared, or a hook fired. For a first run, use preload and a small dynamically linked C target. Static binaries and managed runtimes need their own startup and calling-context treatment.

Install a deliberately small function-hook build

The lab targets Ubuntu 24.04 Linux, x86-64 or ARM64, with a dynamically linked C application. Use a Linux VM or container if your development machine runs another OS. Two terminals must share that same Linux environment; two independent containers do not automatically share the required memory namespace.

This recipe uses LLVM, a compiler toolkit, to translate the BPF programs into native code for the target CPU. It builds the CPU function-hook components; it does not install the optional syscall-transformer or GPU path. Selecting the execution engine matters: one engine can reject instructions that another supports. Our probe uses an atomic add.

sudo apt-get update
sudo apt-get install -y build-essential git cmake clang llvm llvm-dev pkg-config \
  libelf-dev zlib1g-dev libssl-dev libboost-all-dev binutils-dev \
  libyaml-cpp-dev ca-certificates curl unzip

mkdir bpftime-first-call
cd bpftime-first-call
git clone --filter=blob:none https://github.com/eunomia-bpf/bpftime.git
cd bpftime
git checkout 25e250d651ad1d2a2608bbbaeaf7b9ee90cca401
git submodule update --init --recursive

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
  -DBPFTIME_LLVM_JIT=ON -DBPFTIME_UBPF_JIT=OFF \
  -DBPFTIME_BUILD_KERNEL_BPF=OFF -DBUILD_BPFTIME_DAEMON=OFF
cmake --build build --target bpftime-cli-cpp bpftime-agent \
  bpftime-syscall-server --parallel 2

mkdir -p "$HOME/.bpftime"
cp build/tools/cli/bpftime \
  build/runtime/agent/libbpftime-agent.so \
  build/runtime/syscall-server/libbpftime-syscall-server.so \
  "$HOME/.bpftime/"
export PATH="$HOME/.bpftime:$PATH"
bpftime --help
cd ..

Keep the source/build tree: the lab links against its libbpf and uses its headers. clang -target bpf needs a Clang distribution that includes the BPF backend; Ubuntu’s package does. CMake downloads matching Frida development archives, so network access to upstream release assets is also required.

Run your first extension

Download the source bundle into the bpftime-first-call directory that now contains the bpftime checkout.

Download the five-call lab · source ZIP

# Run from bpftime-first-call, beside the bpftime checkout.
curl -fLO https://notes-9fl.pages.dev/posts/bpftime/lab.zip
unzip lab.zip
cd lab
make BPFTIME_SRC=../bpftime
./target
nm -n ./target | grep ' do_work$'
objdump -d --disassemble=do_work ./target

The plain target prints its PID and five results: 1, 4, 7, 10, 13. nm confirms that the target contains a named function. objdump shows its original instructions on your architecture. Neither tool shows the live patch merely by reading the executable on disk.

Now open two terminals in this same lab directory. The commands set BPFTIME_VM_NAME=llvm to choose the BPF execution engine and BPFTIME_GLOBAL_SHM_NAME to name the shared experiment state. Use a shared-memory name dedicated to this exercise in both. Stop the loader before starting a fresh experiment with another name.

# Terminal 1 — stay here while terminal 2 runs.
export PATH="$HOME/.bpftime:$PATH"
export BPFTIME_VM_NAME=llvm
export BPFTIME_GLOBAL_SHM_NAME=bpftime_first_call
bpftime load ./loader ./probe.bpf.o "$(pwd)/target"

Wait for Registered do_work entry + return and initially zero counters. That output says the loader registered its requests. The target has not executed a probe yet.

# Terminal 2 — same Linux environment and lab directory.
export PATH="$HOME/.bpftime:$PATH"
export BPFTIME_VM_NAME=llvm
export BPFTIME_GLOBAL_SHM_NAME=bpftime_first_call
bpftime start ./target

Terminal 2 prints the same five function results. Terminal 1 should eventually show entries=5 returns=5. Intermediate snapshots can differ because reads happen while the target is running. Running another target while the loader stays alive adds five more calls.

# Terminal 1, after the target has finished:
entries=5 returns=5

# Press Ctrl-C in terminal 1 to close the loader’s links and object.
# This is the end of this preload lab: its target has already exited.

There are three different compiled files in this experiment. The target and loader are ordinary CPU executables. probe.bpf.o contains BPF instructions and map definitions; it needs bpftime to execute its programs. This .o uses the ELF file format, which lets a compiler package code together with names and other information that a loader can read.

The source files have corresponding jobs:

obj = bpf_object__open_file("probe.bpf.o", NULL);
bpf_object__load(obj);
ep = bpf_object__find_program_by_name(obj, "on_entry");
rp = bpf_object__find_program_by_name(obj, "on_return");

struct bpf_uprobe_opts opts = {
    .sz = sizeof(opts), .func_name = "do_work"
};
entry = bpf_program__attach_uprobe_opts(
    ep, -1, target_binary, 0, &opts);
opts.retprobe = true;
leave = bpf_program__attach_uprobe_opts(
    rp, -1, target_binary, 0, &opts);

fd = bpf_object__find_map_fd_by_name(obj, "counts");
bpf_map_lookup_elem(fd, &key, &value);

Whose calls do these counters count?

The attach calls pass pid = -1: this lab requests no specific PID. A matching target needs its agent, access to this namespace, and a resolvable module/function attachment before its calls contribute. An ordinary unpreloaded target does not join the experiment just because it runs the same executable.

The two map entries are shared aggregate counters for this experiment, not one map per target. Multiple participating targets or threads add to the same values; the atomic increment prevents concurrent increments from overwriting each other. Separate process/thread attribution requires keys that encode that identity. The two totals also do not correlate individual invocations or prove argument handling. This lab does not test PID-specific filtering.

To hook your own C function, use a target with a suitable non-inlined symbol, change .func_name = "do_work" in the loader, rebuild, and pass the correct target binary path. Restart the loader and then launch your target with the agent. If your probe reads arguments or return values, use the correct register/context layout for the target architecture; our counter intentionally avoids that extra dependency.

Locate the missing stage

Debug this as a chain of milestones. “The loader started” and “the target printed output” are each compatible with a probe that never ran.

For a live target, gdb -p PID followed by disassemble /r do_work inspects the in-memory instructions, subject to debugger permissions. Compare them with objdump on disk. That answers a different question from inspecting BPF bytecode with llvm-objdump -d probe.bpf.o.

For research, first prove that your event fires and original behavior survives. Then measure the plain target, empty probes, and the real extension with matched builds and workload. Keep logging outside the timed event path. A five-call correctness check is not an overhead benchmark.

The call, with its history restored

The loader opened an object and recorded an attachment. Startup put a native agent in the target. The agent realized those descriptions as executable probe code and a live hook. Then an ordinary call followed a local detour, updated shared state, and returned its original result. The observer printed that result’s evidence later.

The same five calls still return the same five results. We have added two BPF programs that count what happened around them.

For a small next experiment, change the return BPF program to immediately return 0;. Rebuild, stop the old loader, and restart both commands with a new shared-memory name in both terminals. The expected final counts become entries=5 returns=0. The application’s five results should remain unchanged. When you introduce a native helper, a different runtime, or a new attachment backend, identify the new boundary before carrying your old assumptions across it.