feat(layers/dtrace): Support User Statically-Defined Tracing(aka USDT) on Linux

[Zheaoli]

This feature is an experimental feature

TL;DR;

The PR will make the debug and tracing the opendal process easier in the Linux platform. The people can use so many modern tools like bpftrace to trace the internal state in the process

USDT is a way to allow people to pre-define some static hooks. Those hooks could be used to fetch some internal information when the process runs.

I will use this PR to describe the detailed behavior of the DTrace in the OpenDAL.

When people enabled layers-dtrace feature and added the layer to their code like the following code:

use anyhow::Result;
use opendal::services::Fs;
use opendal::Operator;
use opendal::layers::DTraceLayer;

#[tokio::main]
async fn main() -> Result<()> {
    let mut builder = Fs::default();

    builder.root("/tmp");

    // `Accessor` provides the low level APIs, we will use `Operator` normally.
    let op: Operator = Operator::new(builder)?.layer(DTraceLayer{}).finish();
    
    let path="/tmp/test.txt";
    for _ in 1..100000{
        let bs = vec![0; 64 * 1024 * 1024];
        op.write(path, bs).await?;
        op.read(path).await?;
    }
    Ok(())
}

Then we compile the code, and use readelf -n ${target_binary} to get the binary detail, we will find some information like following below

Displaying notes found in: .note.stapsdt
  Owner                Data size        Description
  stapsdt              0x00000039       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: create_dir_start
    Location: 0x00000000000f8f05, Base: 0x0000000000000000, Semaphore: 0x00000000003649f8
    Arguments: -8@%rax
  stapsdt              0x00000037       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: create_dir_end
    Location: 0x00000000000f9284, Base: 0x0000000000000000, Semaphore: 0x00000000003649fa
    Arguments: -8@%rax
  stapsdt              0x0000003c       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_list_start
    Location: 0x00000000000f9487, Base: 0x0000000000000000, Semaphore: 0x0000000000364a28
    Arguments: -8@%rax
  stapsdt              0x0000003a       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_list_end
    Location: 0x00000000000f9546, Base: 0x0000000000000000, Semaphore: 0x0000000000364a2a
    Arguments: -8@%rax
  stapsdt              0x0000003c       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_read_start
    Location: 0x00000000000f9740, Base: 0x0000000000000000, Semaphore: 0x0000000000364a18
    Arguments: -8@%rax
  stapsdt              0x0000003a       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_read_end
    Location: 0x00000000000f97fd, Base: 0x0000000000000000, Semaphore: 0x0000000000364a1a
    Arguments: -8@%rax
  stapsdt              0x0000003c       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_stat_start
    Location: 0x00000000000f99f3, Base: 0x0000000000000000, Semaphore: 0x0000000000364a20
    Arguments: -8@%rax
  stapsdt              0x0000003a       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_stat_end
    Location: 0x00000000000f9ab9, Base: 0x0000000000000000, Semaphore: 0x0000000000364a22
    Arguments: -8@%rax
  stapsdt              0x0000003d       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_write_start
    Location: 0x00000000000f9cec, Base: 0x0000000000000000, Semaphore: 0x0000000000364a1c
    Arguments: -8@%rax
  stapsdt              0x0000003b       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_write_end
    Location: 0x00000000000f9df2, Base: 0x0000000000000000, Semaphore: 0x0000000000364a1e
    Arguments: -8@%rax
  stapsdt              0x0000003e       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_delete_start
    Location: 0x00000000000f9fdc, Base: 0x0000000000000000, Semaphore: 0x0000000000364a24
    Arguments: -8@%rax
  stapsdt              0x0000003c       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_delete_end
    Location: 0x00000000000fa0e6, Base: 0x0000000000000000, Semaphore: 0x0000000000364a26
    Arguments: -8@%rax
  stapsdt              0x00000042       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_create_dir_start
    Location: 0x00000000000fa250, Base: 0x0000000000000000, Semaphore: 0x0000000000364a14
    Arguments: -8@%rax
  stapsdt              0x00000040       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_create_dir_end
    Location: 0x00000000000fa30c, Base: 0x0000000000000000, Semaphore: 0x0000000000364a16
    Arguments: -8@%rax
  stapsdt              0x00000033       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: list_start
    Location: 0x00000000000fa869, Base: 0x0000000000000000, Semaphore: 0x0000000000364a0c
    Arguments: -8@%rax
  stapsdt              0x00000031       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: list_end
    Location: 0x00000000000faac7, Base: 0x0000000000000000, Semaphore: 0x0000000000364a0e
    Arguments: -8@%rax
  stapsdt              0x00000033       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: read_start
    Location: 0x00000000000fb0a4, Base: 0x0000000000000000, Semaphore: 0x00000000003649fc
    Arguments: -8@%rax
  stapsdt              0x00000031       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: read_end
    Location: 0x00000000000fb321, Base: 0x0000000000000000, Semaphore: 0x00000000003649fe
    Arguments: -8@%rax
  stapsdt              0x00000033       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: stat_start
    Location: 0x00000000000fb8eb, Base: 0x0000000000000000, Semaphore: 0x0000000000364a04
    Arguments: -8@%rax
  stapsdt              0x00000031       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: stat_end
    Location: 0x00000000000fbb66, Base: 0x0000000000000000, Semaphore: 0x0000000000364a06
    Arguments: -8@%rax
  stapsdt              0x00000034       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: write_start
    Location: 0x00000000000fc6e5, Base: 0x0000000000000000, Semaphore: 0x0000000000364a00
    Arguments: -8@%rax
  stapsdt              0x00000032       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: write_end
    Location: 0x00000000000fc954, Base: 0x0000000000000000, Semaphore: 0x0000000000364a02
    Arguments: -8@%rax
  stapsdt              0x00000035       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: delete_start
    Location: 0x00000000000fcf30, Base: 0x0000000000000000, Semaphore: 0x0000000000364a08
    Arguments: -8@%rax
  stapsdt              0x00000033       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: delete_end
    Location: 0x00000000000fd2aa, Base: 0x0000000000000000, Semaphore: 0x0000000000364a0a
    Arguments: -8@%rax
  stapsdt              0x00000036       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: presign_start
    Location: 0x00000000000fd87b, Base: 0x0000000000000000, Semaphore: 0x0000000000364a10
    Arguments: -8@%rax
  stapsdt              0x00000034       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: presign_end
    Location: 0x00000000000fdaf6, Base: 0x0000000000000000, Semaphore: 0x0000000000364a12
    Arguments: -8@%rax

We describe the offset, the argument of hook, and so many information else in each section of the text. For example, the write_start hook, we can find the offset is 0x00000000000fc6e5 and we can use gdb to verify it.

We will figure out the asm in 0x5555556506e5 is NOP (the process virtual address started at 0x555555554000 on my machine, and 0x555555554000+0x00000000000fc6e5=0x5555556506e5). The NOP instruction code will be replaced by the interrupt code when the hook is attached (In x86_64, the interrupt code is INT3)

We can also set a breakpoint at this address.

We will find the code paused on line 146 in dtrace.rs. This is exactly what we want

The USDT can help us debug it more easily and make it easier to trace it.

╭─   manjusaka@manjusaka-garuda   ~/Documents/projects/rust-demo                                                               02:14:06  
╰─ sudo bpftrace -e ' 
usdt:/home/manjusaka/Documents/projects/rust-demo/target/debug/demo:opendal:write_start { 
    @start[tid,str(arg0)]=nsecs;
}

usdt:/home/manjusaka/Documents/projects/rust-demo/target/debug/demo:opendal:write_end { 
    printf("Write %s use %d nsecs\n", str(arg0), nsecs - @start[tid,str(arg0)]);
    delete(@start[tid,str(arg0)]);
}' -p 385697      
Attaching 2 probes...
Write tmp/test.txt use 59609 nsecs
Write tmp/test.txt use 56546 nsecs
Write tmp/test.txt use 53352 nsecs
Write tmp/test.txt use 57003 nsecs
Write tmp/test.txt use 58479 nsecs
Write tmp/test.txt use 52876 nsecs
Write tmp/test.txt use 56553 nsecs
Write tmp/test.txt use 59414 nsecs
Write tmp/test.txt use 101736 nsecs
Write tmp/test.txt use 54204 nsecs
Write tmp/test.txt use 63272 nsecs
Write tmp/test.txt use 64367 nsecs
Write tmp/test.txt use 56620 nsecs
Write tmp/test.txt use 56536 nsecs
Write tmp/test.txt use 55383 nsecs
Write tmp/test.txt use 52547 nsecs
Write tmp/test.txt use 50337 nsecs
Write tmp/test.txt use 52908 nsecs
Write tmp/test.txt use 59654 nsecs
Write tmp/test.txt use 64653 nsecs
Write tmp/test.txt use 66440 nsecs

We can use many other tools to trace it.

[Xuanwo]

All our probe names should follow the same rule. I raised some of them but not all. I feel like it's better to follow the same rules across the whole project.

[Xuanwo]

blocking_reader_read_error

[Xuanwo]

Thanks a lot!