1. Bibliographic Information

• Title: bpftime: userspace eBPF Runtime for Uprobe, Syscall and Kernel-User Interactions
• Authors: Yusheng Zheng, Tong Yu, Yiwei Yang, Yanpeng Hu, Xiaozheng Lai, Andrew Quinn.
• Affiliations: The authors are from the eunomia-bpf Community, University of California, Santa Cruz, ShanghaiTech University, and South China University of Technology. This mix of community and academic contributors suggests a project grounded in both practical open-source development and rigorous research.
• Journal/Conference: The paper is available on arXiv, a repository for electronic preprints of scientific papers.
• Publication Year: 2023
• Abstract: The paper introduces bpftime, a userspace eBPF runtime designed to overcome the performance bottlenecks of kernel-based eBPF, particularly for uprobes, which suffer from context switch overhead. bpftime uses binary rewriting to implement uprobe and syscall hooks directly in userspace. This approach yields a 10x performance improvement for uprobes compared to kernel equivalents. The runtime can be injected into any running process without restarts or recompilation. It also features shared-memory eBPF maps for inter-process communication and maintains compatibility with existing eBPF toolchains like clang and libbpf, including support for CO-RE. The work aims to improve eBPF's performance, versatility, and security by moving its execution to userspace.
• Original Source Link: The paper is an arXiv preprint, available at https://arxiv.org/abs/2311.07923v2. As a preprint, it has not yet completed a formal peer-review process for publication in a journal or conference.

2. Executive Summary

• Background & Motivation (Why):

  • Core Problem: The standard eBPF framework, while powerful, executes within the Linux kernel. This leads to two significant issues. First, operations that bridge userspace and kernel, like uprobes (tracing user-level functions), incur substantial performance overhead due to repeated context switches (switching the CPU from running a user process to running the kernel and back). Second, kernel eBPF requires elevated (root) privileges, which expands the system's attack surface and poses security risks, such as container escapes or kernel exploits.
  • Existing Gaps: Previous userspace eBPF runtimes (e.g., uBPF, rbpf) demonstrated the potential of this approach but were incomplete. They lacked crucial features like dynamic uprobe and syscall attachment, required manual code changes and recompilation for integration, had inefficient data sharing mechanisms (maps), and were often incompatible with the standard eBPF toolchain (libbpf, clang).
  • Innovation: bpftime introduces a novel approach that combines a syscall-compatible userspace eBPF runtime with a dynamic injection mechanism. It uses binary rewriting to implement hooks directly within the target process's memory, completely avoiding kernel context switches for uprobes. This allows bpftime to be attached to any running process on the fly.

• Main Contributions / Findings (What):

  1. High-Performance Userspace eBPF Runtime: The paper presents bpftime, a general-purpose runtime built with an LLVM-based JIT compiler for high performance. It includes an efficient shared-memory implementation of eBPF maps.
  2. 10x Faster Uprobes: By eliminating the two context switches required by kernel uprobes, bpftime's userspace uprobes are over 10 times faster, making high-frequency tracing practical for latency-sensitive applications.
  3. Seamless Runtime Injection: bpftime can be injected into any running process without requiring the process to be restarted or its source code to be modified. This is a major usability improvement over previous solutions.
  4. Full Toolchain Compatibility: It maintains compatibility with the existing eBPF ecosystem, including clang for compilation, libbpf as a loader library, and CO-RE (Compile Once - Run Everywhere) for portability. This means existing eBPF applications can run in userspace with bpftime without modification.
  5. Enhanced Security Model: By running in userspace without root privileges, bpftime significantly reduces the kernel attack surface and provides a more secure way to instrument applications.

3. Prerequisite Knowledge & Related Work

To understand this paper, several foundational concepts are essential.

• Foundational Concepts:

  • eBPF (Extended Berkeley Packet Filter): Think of eBPF as a tiny, efficient virtual machine inside the Linux kernel. It allows developers to write small, event-driven programs that can run safely within the kernel's privileged context. These programs can be attached to various hooks (e.g., network events, system calls) to monitor, trace, or even modify system behavior without changing the kernel source code.
  • Context Switch: A context switch is the process where the CPU stops executing one process (e.g., a user application) and starts executing another (e.g., the kernel). This involves saving the state of the first process and loading the state of the second. It's a computationally expensive operation. Traditional kernel uprobes require two context switches for each function call being traced: one from userspace to the kernel to run the eBPF program, and one back to userspace to resume the application. This overhead is the primary bottleneck bpftime addresses.
  • Uprobe (User-level Probe): A dynamic tracing mechanism that allows you to execute a piece of code (like an eBPF program) whenever a specific function in a userspace application is called. The traditional implementation places a software interrupt (int3) at the function's entry point, which traps into the kernel.
  • Syscall Tracepoint: A stable hook in the kernel that allows an eBPF program to run whenever a specific system call (e.g., open, read, write) is executed by any process on the system.
  • Binary Rewriting: The technique of modifying a program's executable machine code instructions while it is running. bpftime uses this to replace the first few instructions of a target function with a jump or call to its own code, effectively "hooking" the function without kernel involvement.
  • JIT (Just-In-Time) Compilation: A technique where source code or intermediate bytecode (like eBPF bytecode) is compiled into native machine code at runtime, just before it is executed. This offers much better performance than interpreting the bytecode.

• Previous Works & Technological Evolution:

  • The paper first acknowledges the power and limitations of kernel eBPF. The standard workflow, shown in Image 1, highlights the bpf syscall as the central point of interaction between userspace applications and the kernel's eBPF runtime. The diagram clearly shows the context switch needed for a uprobe to trap into the kernel.

    该图像是图1，展示了内核态eBPF运行时的工作流程示意图，描述了从eBPF程序源码到目标进程的运行机制及上下文切换过程，体现了用户态和内核态之间的交互。

  • Early userspace eBPF projects like uBPF and rbpf are cited as pioneers. They provided the first eBPF interpreters and JITs outside the kernel but were limited. They couldn't dynamically attach to running processes and lacked support for uprobes and syscall hooks, making them unsuitable for general-purpose tracing.

  • The paper contrasts eBPF with WebAssembly (Wasm), another popular userspace virtual machine. While Wasm excels at portability and sandboxing for entire applications (with a strong focus on security via Software Fault Isolation), it often requires manual integration and can have high performance costs for interacting with the host system. In contrast, eBPF is designed for performance and deep system interaction, making it better suited for fine-grained tracing and monitoring.

  • The paper also situates itself in the context of Dynamic Binary Instrumentation (DBI) tools like Pin and Frida. While these tools also allow runtime code modification, they typically lack the built-in safety verifier of eBPF and the high-performance, structured data aggregation capabilities of eBPF maps. bpftime essentially combines the power of DBI with the safety and ecosystem of eBPF.

4. Methodology (Core Technology & Implementation)

The core of the paper is the design of the bpftime runtime.

• Principles and Goals: The primary goal is to create a userspace eBPF runtime that is fast, compatible, flexible, and secure.

  1. Userspace Execution: Move eBPF execution out of the kernel to eliminate context-switch overhead.
  2. Kernel Compatibility: Be a drop-in replacement, supporting existing eBPF applications and tools (libbpf) without modification.
  3. Dynamic Hooks: Provide uprobe and syscall hooking that can be attached to running processes.
  4. Performance & Extensibility: Use a high-performance JIT compiler and design for cross-platform potential.

• Architectural Overview: The bpftime architecture, shown in Image 2, consists of two main components operating entirely in userspace:

  1. Syscall-Compatible Library (bpftime-syscall.so): This library intercepts the bpf() system calls made by a standard eBPF user application (e.g., one using libbpf). Instead of passing them to the kernel, it handles them in userspace. When the application loads an eBPF program or creates a map, this library places the program's bytecode and map definitions into a shared memory region.

  2. Attachment Agent (bpftime-agent.so): This is a shared library containing the eBPF virtual machine (VM) and JIT compiler. It is injected into the target process that needs to be traced. Once injected, the agent reads the eBPF programs and map configurations from the shared memory, compiles the programs with its JIT, and uses binary rewriting to attach them to the specified functions or syscalls within the target process.

     When a hooked function is called, control is transferred directly to the JIT-compiled eBPF program within the agent, all within the same process and address space. This avoids any kernel interaction.

     该图像是论文中的示意图，展示了内核eBPF运行时的工作流程，涵盖了从eBPF程序源代码到用户空间与内核空间交互的整体过程。

• Steps & Procedures:

  1. An eBPF application (e.g., a monitoring tool) starts. It's configured to use bpftime (e.g., via LD_PRELOAD).
  2. The application uses libbpf to load an eBPF program. The bpftime-syscall.so library intercepts the bpf() syscall.
  3. The library allocates shared memory and stores the eBPF bytecode and map definitions there.
  4. A separate control plane process tells the bpftime injector to attach to a target application.
  5. The injector uses ptrace to pause the target process and forces it to load the bpftime-agent.so library.
  6. The agent initializes, connects to the shared memory, finds the eBPF program, and JIT-compiles it.
  7. The agent identifies the target function's address (e.g., malloc) and uses binary rewriting to install a hook that redirects execution to the JIT-compiled eBPF code.
  8. The target process is resumed. Now, every call to malloc will first trigger the eBPF program.

• Hook Design Details:

  • Function Hooks (Uprobes): bpftime uses inline hooking. It saves the first few bytes of the target function's machine code and overwrites them with a call or jump instruction pointing to the eBPF agent's dispatcher. The dispatcher saves the CPU register state (which contains the function arguments), executes the eBPF program, and then restores the original instructions and register state to resume the original function's execution.
  • System Call Hooks: Hooking syscalls is trickier. On ARM, the process is similar to function hooking. However, on the x86-64 architecture, the syscall instruction is only two bytes long, which is too short to be replaced with a standard 5-byte jump instruction. To solve this, bpftime uses the zepoline method. This clever technique finds an executable page of memory filled with zeros (the "zero page") and writes a call instruction there. The original two-byte syscall instruction is then replaced with an instruction that jumps to this call in the zero page, which in turn redirects execution to the bpftime runtime.

• Security Architecture: bpftime is designed with a multi-layered security model to prevent abuse.

  • SP1: Verifier-Ensured Safety: eBPF programs are statically analyzed by a verifier (either the kernel's verifier or a userspace one) before execution. This ensures the program is safe: it won't crash the host process, it can't enter infinite loops, and it can only access memory it's explicitly given access to (e.g., via function arguments passed by the hook).
  • SP2: Runtime Memory Protection: The memory belonging to the bpftime agent is protected (e.g., set to read-only) to prevent the host application from maliciously modifying it.
  • SP3: Segregated Shared Memory: Shared memory is partitioned. The agent only has read-only access to program metadata but can read/write to map data. This prevents a compromised agent in one process from tampering with the eBPF programs intended for another.
  • SP4: Unprivileged Kernel eBPF Map Access: For scenarios where userspace eBPF needs to communicate with kernel eBPF programs, bpftime allows access to kernel eBPF maps without requiring the target process to have CAP_SYS_ADMIN privileges. This is achieved by having a privileged control plane create the maps and pin them to the BPFFS (e.g., at /sys/fs/bpf). The unprivileged target process can then access these maps via their file descriptors, using standard file permissions to control access.

5. Experimental Setup

The paper evaluates bpftime by answering four key questions related to performance, efficiency, compatibility, and security.

• Datasets/Workloads:

  • Micro-benchmarks: A series of small, targeted tests to measure specific performance aspects. For hook performance, this involves calling a hooked empty function repeatedly. For runtime efficiency, it includes benchmarks for integer math (log2_int), loops (prime), memory operations (memcpy), and control flow (switch).
  • Real-World Programs: Programs from the bcc-tools suite, such as malloc.py (traces memory allocations) and opensnoop.py (traces open() syscalls), were used to test compatibility.

• Evaluation Metrics:

  • Latency (ns): This metric measures the time overhead introduced by a single hook invocation. It is defined as the total time taken to execute a hooked function call minus the time taken to execute the original, unhooked function. It is measured in nanoseconds (ns). A lower value is better.
  • Instruction Count (#Inst): The number of CPU instructions executed by the hook mechanism. This provides a hardware-agnostic measure of the hook's complexity. A lower value is generally better.

• Baselines:

  • Kernel eBPF: The standard, in-kernel implementation of uprobes and syscall tracepoints serves as the primary performance baseline.
  • Other Userspace Runtimes: For VM efficiency, bpftime with its LLVM JIT is compared against:
    • ubpf: A userspace eBPF interpreter.
    • rbpf: A userspace eBPF JIT compiler written in Rust.
    • WASM: A WebAssembly runtime, representing an alternative VM technology.
    • Native: The performance of the benchmark code compiled directly to native machine code, representing the theoretical performance ceiling.

6. Results & Analysis

• Core Results: Hook Performance The paper's most significant finding is the dramatic performance improvement for uprobes. The results from Table 1 are transcribed below.

  Probe Types	Kernel (ns)	User (ns)	#Inst
  Uprobe	3224.17	314.57	4
  Uretprobe	3996.80	381.27	2
  Syscall Tracepoint	151.83	232.58	4
  Embedding Runtime	Not available	110.01	N/A

  Analysis:

  • Uprobe/Uretprobe: bpftime's userspace uprobe is over 10x faster than the kernel uprobe (314 ns vs. 3224 ns). This is a direct result of eliminating the two expensive context switches. The low latency makes it feasible to trace functions that are called very frequently without impacting application performance.
  • Syscall Tracepoint: Interestingly, bpftime's userspace syscall hook is slightly slower than the kernel's syscall tracepoint (232 ns vs. 151 ns). This is because the kernel's tracing mechanism for syscalls is highly optimized and integrated deep within the syscall handling path. In contrast, bpftime's hook is more generic and incurs a small overhead from its binary rewriting mechanism. However, the performance is still in the same order of magnitude and acceptable for many use cases.
  • Embedding Runtime: This shows the low overhead of simply having the runtime present in a process.

• Core Results: Runtime Efficiency Image 3 shows the performance of bpftime's LLVM JIT compared to other runtimes on various micro-benchmarks.

  

  Analysis:

  • Across all benchmarks (strcmp, log2_int, prime, simple, memcpy, switch, memory_a_plus_b), bpftime-llvm (the dark blue bar) consistently demonstrates the best performance among all userspace eBPF and Wasm runtimes.
  • Its performance is very close to that of native code (the yellow bar), especially in compute-intensive tasks like prime and log2_int. This highlights the efficiency of the LLVM JIT backend.
  • Both ubpf (interpreter) and WASM are significantly slower, showcasing the performance advantage of JIT compilation. bpftime is a clear winner in terms of raw execution speed.

• Compatibility Analysis The paper reports that real-world eBPF applications from bcc-tools, like malloc and opensnoop, can be run with bpftime in userspace without any code modifications. This is a critical result, as it proves that bpftime successfully emulates the kernel's bpf() syscall interface and is compatible with the libbpf library, ensuring easy adoption for developers already familiar with the eBPF ecosystem.

• Security Assessment The analysis confirms the security benefits proposed in the design. By moving eBPF execution to an unprivileged userspace context, bpftime eliminates the need for root access for tracing applications. This drastically reduces the kernel attack surface, mitigating risks like container escapes via kernel vulnerabilities in the eBPF subsystem.

7. Conclusion & Reflections

• Conclusion Summary: The paper successfully introduces and validates bpftime, a high-performance, compatible, and more secure userspace eBPF runtime. Its key innovation is the use of dynamic binary rewriting to implement uprobes and syscall hooks entirely in userspace, leading to a 10x performance gain for uprobes by avoiding kernel context switches. By maintaining compatibility with existing eBPF toolchains, bpftime lowers the barrier to adopting userspace eBPF for observability, monitoring, and security applications. The project is open-sourced, encouraging community collaboration.

• Limitations & Future Work: The paper itself does not explicitly list limitations, but some can be inferred:

  • Platform Specificity: The zepoline technique for syscall hooking is specific to the x86 architecture. Supporting other architectures like ARM64 for syscalls would require different implementation strategies.
  • JIT Complexity: While fast, an LLVM-based JIT adds a significant dependency and complexity. The paper mentions a simpler handcrafted JIT for constrained devices, but its performance is not detailed.
  • Peer Review: As an arXiv preprint, the work has not yet undergone formal peer review, which is a standard part of academic validation.
  • Hooking Fragility: The binary rewriting approach can be fragile. It might fail if the target application also uses a JIT compiler or if other hooking tools have already modified the target function's code.

• Personal Insights & Critique: bpftime is an excellent piece of systems engineering that addresses a very real-world problem. The 10x performance gain for uprobes is not just an incremental improvement; it's a game-changer that enables new use cases for high-frequency tracing in performance-critical environments like financial trading systems or high-throughput web servers.

  The decision to maintain compatibility with libbpf is strategically brilliant. It allows the project to leverage the entire existing eBPF ecosystem, ensuring immediate usability and a smoother adoption curve. Developers don't need to learn a new API; they can use tools they already know.

  The most compelling aspect is the "seamless injection" capability. The ability to attach a powerful tracer to any running process without restarts is the holy grail for production debugging and live monitoring. bpftime delivers this with an elegant and robust architecture.

  Overall, bpftime represents a significant step forward in making eBPF technology more accessible, performant, and secure. It effectively bridges the gap between powerful but risky kernel-level tracing and safer but less capable userspace alternatives.