Move the observer

Start with a program we can see

Save this as target.c. It calls do_work five times, one second apart. Each call multiplies its argument by three and adds one.

#include <stdio.h>
#include <unistd.h>

/* Keep a real, discoverable function entry for the first experiment. */
__attribute__((noinline)) int do_work(int x)
{
    volatile int y = x * 3;
    return y + 1;
}

int main(void)
{
    printf("target pid=%ld\n", (long)getpid());
    fflush(stdout);
    for (int i = 0; i < 5; i++) {
        printf("do_work(%d) = %d\n", i, do_work(i));
        fflush(stdout);
        sleep(1);
    }
    return 0;
}

noinline asks the compiler to keep do_work as a separate function. That gives our tracing tool a real function entry to find later. The volatile local also keeps the computation visible in this deliberately simple example.

cc -O0 target.c -o target
./target

-O0 turns optimization off for this first experiment, keeping the function easy to recognize. The first output line identifies the running process. The remaining lines show what the function returned:

target pid=12345
do_work(0) = 1
do_work(1) = 4
do_work(2) = 7
do_work(3) = 10
do_work(4) = 13

We want a second program to report entries=5 returns=5. The application should keep printing exactly the results above. The interesting part is that do_work contains no counting code.

Add something before the function runs

If we were willing to edit the application, the idea would be straightforward:

record_entry();
result = do_work(x);
record_return();

A hook gives us a way to arrange that extra execution without adding those calls to the source. A tool changes how the running program reaches the function: it makes a small detour through our extra code, then lets the original function continue.

The piece of extra code that records an observation is called a probe. In this example we want one probe when the function begins and another when it returns. The hook gets our code called; the probe decides what to record.

The function still computes 13. We have added observations around that computation. There are several ways to arrange this detour; the choice determines where our probe code runs.

Where should the counter run?

Linux already provides a mechanism for observing a userspace program—a program such as our C application, running outside the kernel. A uprobe selects an instruction in that program, often the start of a function. When execution reaches it, Linux can run an attached probe. A uretprobe observes the function returning.

The names describe what we observe: a userspace function. In the conventional Linux mechanism, the probe code itself runs in the kernel. A breakpoint at the selected instruction transfers control into the kernel; the kernel runs the probe and then resumes the application.

One way to write that probe is eBPF, shortened to BPF throughout this series. You write a small program, commonly in a restricted form of C, and compile it into BPF instructions. These instructions need software that knows how to execute them. Linux provides that execution machinery in the kernel.

bpftime provides BPF execution machinery inside the application instead. With its ordinary function hooks, the call takes a detour through probe code in the same process. It can then continue into the original function.

This is the main idea behind bpftime’s function tracing: keep the ability to write BPF probes, but execute them in the process being observed. We still need a way to make the function take that detour.

Frida and Gum: the part that makes the detour

Frida is a toolkit for inspecting and changing a program while it runs. For example, you can ask it to report the argument passed to a function, run code when that function begins, or change an argument before the function uses it. You can do this without rebuilding the application.

Frida includes a C library called Gum. Gum provides the low-level function-hooking machinery: it can alter a function’s instructions in memory so that execution visits additional code and then continues through the original function. “Frida-Gum” refers to this library.

bpftime uses Gum for this job. When Gum reports that do_work has been entered, bpftime runs our BPF entry probe. When Gum reports a normal return, bpftime runs our BPF return probe.

You may see Frida tutorials controlled by Python, with the hook behavior written in JavaScript. bpftime uses Gum’s C interface for the function hook described here and supplies BPF programs as its probe code. You do not need to write a Frida script to run the bpftime lab.

If you want to try Frida on its own, its function tutorial shows a small program, a hook, and the resulting output. For our story, the role to carry forward is specific: Gum gets the extra code called; bpftime executes the BPF probe.

Give the counter a home—and someone to read it

Where do those five increments live? BPF programs use data structures called maps to keep state. Our map is a two-element array: entry 0 stores the number of function entries; entry 1 stores the number of returns.

The BPF programs update this array. A second program reads it and prints the results. We call that second program the loader, because it also opens the BPF program file and asks bpftime to attach its programs to do_work.

The loader is a separate Linux process. For it to read the same counters that the target updates, bpftime places this map’s data in shared memory: memory that both processes can access. The loader reads the counters periodically; the probe does not need the loader to execute anything on each call.

That leaves one practical question: our original application has neither Gum nor a BPF execution engine. How do they get into it? bpftime supplies a shared library, libbpftime-agent.so, that is loaded into the application. This is the agent. It is library code running inside the target, where it can install hooks and run BPF programs.

The software that executes BPF programs and provides operations they can call is often called the runtime. That word names a responsibility here: turn a BPF program into something that can run when Gum reaches our hook.

The loader also uses a bpftime library, libbpftime-syscall-server.so. It lets a loader written for the usual BPF loading interfaces put its requests into bpftime. Part 3 will show both libraries being loaded, and explain why they belong in different processes.

Why make this move?

Consider an experiment where you need to compare several probes around the same function while keeping the service binary unchanged. bpftime lets you keep writing BPF programs and using BPF maps, while changing the function event’s route so the probe executes inside the service.

Moving probe execution into the application can avoid the kernel breakpoint path for that event. Whether it helps your workload depends on the hook, the probe, and the machine. The probe still takes time, and it runs on the application thread that reached the function.

There are limits to the idea. A probe observing a userspace function cannot thereby observe every kernel event. An operation available to kernel BPF is not automatically available to bpftime. And putting code in the application means that code can delay or disturb the application; the word “BPF” does not by itself establish isolation.

For now, the architecture fits in one sentence: the loader asks for a probe; the agent installs it in the target; Gum redirects the call; bpftime executes the BPF program; the loader reads the resulting map.