Productivity & Career

Systems Performance Ch. 13-15: perf, Ftrace, & BPF Tracing

Otama
3 min read
Progress 12 / 13
Table of Contents

Introduction

[!NOTE] This article was generated by feeding my chaotic, personal reading notes of this massive textbook into an AI to restructure them into a readable blog post.

We’ve organized survival tactics for CPU, Memory, File Systems, Disks, and Networks. Now, we’ve finally reached the technical highlight of the book: “Chapter 13: perf,” “Chapter 14: Ftrace,” and “Chapter 15: BPF.” These chapters support the most advanced levels of performance analysis.

When a system slows down, guessing parameters or blindly running benchmarks in production is for amateurs. What we should really be doing is safely observing internal states without negatively impacting the target system.

In this post, we’ll dive into the internal structures and field-usage strategies of the “Three Sacred Treasures” of modern Linux performance analysis: perf, Ftrace, and BPF (BCC/bpftrace).

1. Exposing CPU and Hardware Truths: “perf” (Ch. 13)

perf is the official Linux profiler. Despite being a large-scale user-level program, it’s a unique entity included directly in the Linux kernel source tree. It reveals which code paths are consuming CPU resources and why threads are yielding the CPU.

  • Profiling and Flame Graphs: perf excels at sampling stack traces. By running perf record to generate perf.data and using perf script to textify it, you can expose exactly where CPU time is being spent in both the kernel and user space. This is the powerful foundation for Flame Graphs.
  • Analyzing Hardware Counters (PMCs): Another strength is microarchitectural analysis, such as cache misses and stall cycles. perf stat can measure cycle counts and instructions, automatically calculating “shadow stats” like Instructions Per Cycle (IPC).
  • Versatile Event Sources: Beyond hardware events, it supports static and dynamic tracing via tracepoints, kprobes, uprobes, and USDT. perf trace acts as a lightweight strace alternative, tracking system calls and block I/O latency in real-time. It’s incredibly handy for quick field investigations.

2. The Artisan’s Tool for Kernel Depths: “Ftrace” (Ch. 14)

If you need to know how kernel code paths branch and which functions are sluggish, Ftrace is your tool. Since it’s operated directly through the filesystem under /sys/kernel/tracing, you don’t need to install special packages. It works everywhere, from embedded systems to massive servers.

  • Function Graph Tracer: The crown jewel of Ftrace. It displays kernel function call hierarchies (parent to child) graphically and measures the execution time of each. You can identify the culprit child function slowing down a parent in an instant.
  • Powerful hist Triggers and Synthetic Events: You can generate event histograms directly by writing to files, keyed by PID or stack trace. Furthermore, “synthetic events” allow you to combine arguments from multiple events to calculate things like delta time (latency) using nothing but shell scripts.
  • Using Frontend Tools: Manipulating raw files can be tedious. In the field, standard practice is to use frontends like trace-cmd or single-purpose toolsets like perf-tools (e.g., execsnoop, iolatency).

3. The Ultimate Magic for Production: “eBPF” (Ch. 15)

Old-school tracers often killed production performance by transferring every event to user space. BPF is a revolution, allowing for direct and safe data aggregation (like histogramming) within the kernel.

Depending on your goal, you’ll use one of two main frontends:

  • BCC (Complex, Polished Toolset): Powerful, ready-made tools written in C and Python like biolatency (disk I/O histograms) or tcplife. You don’t need to memorize options; just run them to get perfect, noise-free output.
  • bpftrace (The Survival Knife): A high-level language inspired by awk and C for writing short custom programs or one-liners on the fly. By combining “built-in variables” (like pid or comm) with “map functions” (like count(), sum(), hist()), you can freely analyze unique application-specific events that existing tools can’t handle.

Conclusion

Reading these three chapters drives home the point that to solve those “mysterious delays,” we must choose the right tool for the specific situation.

Related Posts