No output in uprobe

[sebastiaoamaro]

Hi everyone I am running a simple uprobe:

int main() {
    int workload_size = 10000000;
    for(int i=0;i < workload_size;i++){
        my_test_function();
        sleep(1);
   }
}
void my_test_function(){
    printf("Running \n");
    fflush(stdout);
}

SEC("uprobe/my_uprobe")
int handle_uprobe(struct pt_regs *ctx) {
    bpf_printk("In probe please \n");	
    return 0;
}

For a simple C function called my_test_function. When using libbpf-c I have no problems and I see the bpf_printk output in the tracelog, however when using libbpf-rs (version 0.23), ubuntu 22.04, kernel version 6.5.0-1025-oem, I see no output.
Below is the userspace code:

          let uprobe = skel.progs_mut().handle_uprobe().attach_uprobe(
              false,
              *pid as i32,
              binary_path.clone(),
              symbol_location as usize,
          );
          match uprobe {
              Ok(uprobe) => {
                  println!("Inserted probe with name {}", function);
              }
              Err(e) => {
                  println!("Failed to insert uprobe error: {}", e);
              }
          }

The probe is correctly attached, the symbol_location value is the same as the one in libbpf-c.
Is this a problem on my end?
Thanks in advance.

[danielocfb]

libbpf-rs is just a wrapper around libbpf. Make sure that both libbpf versions match for apples:apples comparison; e.g., by not using vendored libbpf with Rust code.

One random guess I have is that your kernel is lacking a fix for PID filtering (I don't have a link right now) and you are using an old libbpf in C that does not yet use optimized USDT attach by using multi-uprobe attach internally. More recent versions may do that and because of the broken kernel PID filtering that may not work properly. That is likely the case with libbpf-rs 0.23. libbpf/libbpf@d9f9fd5 is a fix for libbpf. Alas, cutting a libbpf-sys release containing it seems to amount to a major exercise in patience...libbpf/libbpf-sys#92

To verify that this is indeed the issue, try not providing a PID to the attach call.

[d-e-s-o]

Assuming this is the issue, you should now be able to upgrade to libbpf-sys-1.4.3+v1.4.5 and it should address the problem

[sebastiaoamaro]

Hi @danielocfb and @d-e-s-o, thank you both for the quick responses.
I provided 0 as a pid to the attach call and the result was the same.

I ran this cargo add libbpf-sys@=1.4.3+v1.4.5.
And now have libbpf-sys = "=1.4.3" in Cargo.toml, however, still no prints in tracelog.

[d-e-s-o]

Probably different issue then. Please try with the same libbpf version you used in the C program.

[anakryiko]

@sebastiaoamaro to disable PID filtering you need to provide -1, not zero. Zero is "current process only", which is still filtering by PID.

[danielocfb]

Reporter missing in action, nothing is pointing to a bug in the library. Closing.

[sebastiaoamaro]

Hi,
Sorry for the delayed response.
I ran with this Cargo.toml:

[dependencies]
anyhow = "1.0"
libbpf-rs = "0.19"
libc = "*"
structopt = "0.3"
ctrlc = "3.1"
object = "0.25"
plain = "0.2"
libloading = "0.7.3"
rand = "0.8"
nix = "0.24.1"
libbpf-sys = "=1.1.1"
[build-dependencies]
libbpf-cargo = "0.13"

I tried with libbpf-sys 1.1.1,1.12.0 and the most recent one previously suggested, and it shows the same behaviour.

[danielocfb]

Can you provide complete reproducible example?

[sebastiaoamaro]

Hi,
Here is the example.
In the test.sh file an absolute directory is needed, and in the main.rs as well.
After that running test.sh is enough.
uprobetest.zip

[danielocfb]

Thanks. I think you should really be using more up-to-date version of libbpf-rs and libbpf-cargo. We can't really support multiple versions, but certainly not two year old stuff. Second, in a newer version libbpf-rs already supports attaching by symbol name. No need to have all this custom logic in you shared object and whatnot (though of course you may need it for other reasons not obvious from the example).

That being said, your issue is that you are immediately dropping the Link after attachment. So you'll attach and then detach, without giving the probe a chance to trigger. See https://docs.rs/libbpf-rs/0.24.2/libbpf_rs/struct.Link.html for semantics.

--- src/main.rs
+++ src/main.rs
@@ -45,8 +45,8 @@ fn main() -> Result<()>{
       binary_path.clone(),
       symbol_location as usize,
   );
-  match uprobe {
-      Ok(uprobe) => {
+  match &uprobe {
+      Ok(..) => {
           println!("Inserted probe with name {}", function);
       }
       Err(e) => {

[sebastiaoamaro]

Oh, thanks for the quick answer and for the help!
I will update to the newer version, and use the opts.
Thank you all for the comments and for the help!