a linux trace/probe tool.
官網：[https://sourceware.org/systemtap/](https://sourceware.org/systemtap/)
?
### 用戶空間
?
SystemTap探測用戶空間程序需要utrace的支持，3.5以上的內核版本默認支持。
對于3.5以下的內核版本，需要自己打相關補丁。
更多信息：[http://sourceware.org/systemtap/wiki/utrace](http://sourceware.org/systemtap/wiki/utrace)
需要：
debugging information for the named program
utrace support in the kernel
?
**(1) Begin/end**
探測點：
進程/線程創建時
進程/線程結束時
?
process.begin
process("PATH").begin
process(PID).begin
?
process.thread.begin
process("PATH").thread.begin
process(PID).thread.begin
?
process.end
process("PATH").end
process(PID).end
?
process.thread.end
process("PATH").thread.end
process(PID).thread.end
?
**(2) Syscall**
探測點：
系統調用開始
系統調用返回
process.syscall
process("PATH").syscall
process(PID).syscall
?
process.syscall.return
process("PATH").syscall.return
process(PID).syscall.return
?
可用的進程上下文變量：
$syscall // 系統調用號
$argN ($arg1~$arg6) // 系統調用參數
$return // 系統調用返回值
?
**(3) Function/statement**
探測點：
函數入口處
函數返回處
文件中某行
函數中的某個標簽
process("PATH").function("NAME")
process("PATH").statement("*@FILE.c:123")
process("PATH").function("*").return
process("PATH").function("myfunc").label("foo")
?
**(4) Absolute variant**
探測點：
進程的虛擬地址
process(PID).statement(ADDRESS).absolute
?
A non-symbolic probe point uses raw, unverified virtual addresses and provide no $variables.
The target PID parameter must identify a running process and ADDRESS must identify a valid instruction address.
This is a guru mode probe.
?
**(5) Target process**
探測點：
動態鏈接庫中的函數(比如glibc)
Target process mode (invoked with stap -c CMD or -x PID) implicitly restricts all process.* probes to the given child
process.
If PATH names a shared library, all processes map that shared library can be probed.
If dwarf debugging information is installed, try using a command with this syntax:
probe process("/lib64/libc-2.8.so").function("...") { ... }
?
**(6) Instruction probes**
探測點：
單條指令
指令塊
?
process("PATH").insn
process(PID).insn
?
process("PATH").insn.block
process(PID).insn.block
?
The .insn probe is called for every single-stepped instruction of the process described by PID or PATH.
The .insn.block probe is called for every block-stepped instruction of the process described by PID or PATH.
Using this feature will significantly slow process execution.
?
統計一個進程執行了多少條指令：
stap -e 'global steps; probe process("/bin/ls").insn {steps++}; probe end {printf("Total instruction: %d\n", steps)}' \
??? -c /bin/ls
?
**(7) 使用**
gcc -g3 -o test test.c
stap -L 'process("./test").function("*")' // 顯示程序中的函數和變量
?
調試等級：
Request debugging information and also use level to specify how much information. The default level?is 2.
Level 0 produces no debug information at all. Thus, -g0 negates -g.
Level 1 produces minimal information, enough for making backtraces in parts of the program that you don't
plan to debug. This includes descriptions of functions and external variables, but no information about local
variables and no line numbers.
Level 3: includes extra information, such as all the macro definitions present in the program.
?
### 高級功能
?
**(1) 自建腳本庫**
A tapset is just a script that designed for reuse by installation into a special directory.
Systemtap attempts to resolve references to global symbols (probes, functions, variables) that are not defined
within the script by a systematic search through the tapset library for scripts that define those symbols.
A user may give additional directories with the -I DIR option.
?
構建自己的庫：
1. 創建庫目錄mylib，添加兩個庫文件
time-default.stp
~~~
function __time_value() {
return gettimeofday_us()
}
~~~
time-common.stp
~~~
global __time_vars
function timer_begin(name) {
__time_vars[name] = __time_value()
}
function timer_end(name) {
return __time_value() - __time_vars[name]
}
~~~
2. 編寫應用腳本
tapset-time-user.stp
~~~
probe begin {
timer_begin("bench")
for(i=0; i<1000; i++) ;
printf("%d cycles\n", timer_end("bench"))
exit()
}
~~~
3. 執行
stap -I mylib/ tapset-time-user.stp
?
**(2) 探測點重命名**
主要用于在探測點之上提供一個抽象層。
Probe point aliases allow creation of new probe points from existing ones.
This is useful if the new probe points are named to provide a higher level of abstraction.
格式：
probe new_name = existing_name1, existing_name2[, ..., existing_nameN]
{
??? prepending behavior
}
實例：
~~~
probe syscallgroup.io = syscall.open, syscall.close,
syscall.read, syscall.write
{
groupname = "io"
}
probe syscallgroup.process = syscall.fork, syscall.execve
{
groupname = "process"
}
probe syscallgroup.*
{
groups[execname() . "/" . groupname]++
}
global groups
probe end
{
foreach (eg in groups+)
printf("%s: %d\n", eg, groups[eg])
}
~~~
**(3) 嵌入C代碼**
SystemTap provides an "escape hatch" to go beyond what the language can safely offer.
嵌入的C代碼段用%{和%}括起來，執行腳本時要加-g選項。
提供一個THIS宏，可以用于獲取函數參數和保存函數返回值。
實例：
~~~
%{
#include <linux/sched.h>
#include <linux/list.h>
%}
function process_list()
%{
struct task_struct *p;
struct list_head *_p, *_n;
printk("%-20s%-10s\n", "program", "pid");
list_for_each_safe(_p, _n, ¤t->tasks) {
p = list_entry(_p, struct task_struct, tasks);
printk("%-20s%-10d\n", p->comm, p->pid);
}
%}
probe begin {
process_list()
exit()
}
~~~
stap -g embeded-c.stp
dmesg可看到打印出的所有進程。
?
C代碼用%{ ... %}括起來，可以是獨立的一個段，可以作為函數的一部分，也可以只是一個表達式。
?
**(4) 已有腳本庫**
SystemTap默認提供了非常強大的腳本庫，主要類別如下：
Context Functions
Timestamp Functions
Time utility functions
Shell command functions
Memory Tapset
Task Time Tapset
Secheduler Tapset
IO Scheduler and block IO Tapset
SCSI Tapset
TTY Tapset
Interrupt Request (IRQ) Tapset
Networking Tapset
Socket Tapset
SNMP Information Tapset
Kernel Process Tapset
Signal Tapset
Errno Tapset
Device Tapset
Directory-entry (dentry) Tapset
Logging Tapset
Queue Statistics Tapset
Random functions Tapset
String and data retrieving functions Tapset
String and data writing functions Tapset
Guru tapsets
A collection of standard string functions
Utility functions for using ansi control chars in logs
SystemTap Translator Tapset
Network File Storage Tapsets
Speculation
?
### 實現原理
?
**(1) SystemTap腳本的執行流程**
?
**pass1**
During the parsing of the code, it is represented internally in a parse tree.
Preprocessing is performed during this step, and the code is checked for semantic and syntax errors.
?
**pass2**
During the elaboration step, the symbols and references in the SystemTap script are resolved.
Also, any tapsets that are referenced in the SystemTap script are imported.
Debug data that is read from the DWARF(a widely used, standardized debugging data format) information,
which is produced during kernel compilation, is used to find the addresses for functions and variables
referenced in the script, and allows probes to be placed inside functions.
?
**pass3**
Takes the output from the elaboration phase and converts it into C source code.
Variables used by multiple probes are protected by locks. Safety checks, and any necessary locking, are
handled during the translation. The code is also converted to use the Kprobes API for inserting probe points
into the kernel.
?
**pass4**
Once the SystemTap script has been translated into a C source file, the code is compiled into a module that
can be dynamically loaded and executed in the kernel.
?
**pass5**
Once the module is built, SystemTap loads the module into the kernel.
When the module loads, an init routine in the module starts running and begins inserting probes into their
proper locations. Hitting a probe causes execution to stop while the handler for that probe is called.
When the handler exits, normal execution continues. The module continues waiting for probes and executing
handler code until the script exits, or until the user presses Ctrl-c, at which time SystemTap removes the
probes, unloads the module, and exits.
?
Output from SystemTap is transferred from the kernel through a mechanism called relayfs, and sent to STDOUT.
?
**(2) 從用戶空間和內核空間來看SystemTap腳本的執行**
?
**(3) kprobes**
斷點指令(breakpoint instruction)：__asm INT 3，機器碼為CC。
斷點中斷(INT3)是一種軟中斷，當執行到INT 3指令時，CPU會把當時的程序指針(CS和EIP)壓入堆棧保存起來，
然后通過中斷向量表調用INT 3所對應的中斷例程。
INT是軟中斷指令，中斷向量表是中斷號和中斷處理函數地址的對應表。
INT 3即觸發軟中斷3，相應的中斷處理函數的地址為：中斷向量表地址 + 4 * 3。
?
A Kprobe is a general purpose hook that can be inserted almost anywhere in the kernel code.
To allow it to probe an instruction, the first byte of the instruction is replaced with the breakpoint
instruction for the architecture being used. When this breakpoint is hit, Kprobe takes over execution,
executes its handler code for the probe, and then continues execution at the?next instruction.
?
**(4) 依賴的內核特性**
kprobes/jprobes
return probes
reentrancy
colocated (multiple)
relayfs
scalability (unlocked handlers)
user-space probes
?
