All commit authors signed the Contributor License Agreement.

cc @pablogsal. I went ahead and submitted this PR for #136459 but please let me know what you think!

For now, the trampoline depens on the mmap.

But it has record about the mmap performance issue on macOS.

I think we need a benchmark here

FYI

1. tmm1/node@3542e4e

2. https://chromium-review.googlesource.com/c/v8/v8/+/6385655/7/src/diagnostics/perf-jit.cc

3. https://bugzilla-dev.allizom.org/show_bug.cgi?id=1827214

Thanks for you comment but I don't think we need any benchmark. We are using mmap all over the place and that's what the allocators, the JIT and many other things do. The trampoline also batches allocations in big chunks so I think we are fine. If the problem is on samply side that's their challenge to figure out there not here. If they need some sort of hook they should mention it here but is unlikely we will add anything that's not generic.

Edit: ah I think you are referring to the mmap to tell Perf about the maps file no? Sorry I thought you were referring to the jit trampoline compiler that itself uses mmap to get the chunks.

@canova i am currently on vacation until EOW I will answer more when I have some time later today or tomorrow but I would love to get macOS support. I have some questions I think we need to answer first to ensure we provide a good UX if we are adding more profilers.

@Zheaoli

For now, the trampoline depens on the mmap.

But it has record about the mmap performance issue on macOS.

I think we need a benchmark here

FYI

1. [tmm1/node@3542e4e](https://github.com/tmm1/node/commit/3542e4e2944f2fe1132ded1052e223f54c90e4bf)

2. https://chromium-review.googlesource.com/c/v8/v8/+/6385655/7/src/diagnostics/perf-jit.cc

3. https://bugzilla-dev.allizom.org/show_bug.cgi?id=1827214

Ah I think you are right. I actually looked for this and only found one mmap inside the perf_trampoline.c file and that was for a memory arena so thought this change was not needed and didn't think much of if after. But looking at it again, I think it's this one that we shouldn't mmap:

I updated the PR to remove this on macOS, let me know what you think!

@pablogsal

@canova i am currently on vacation until EOW I will answer more when I have some time later today or tomorrow but I would love to get macOS support. I have some questions I think we need to answer first to ensure we provide a good UX if we are adding more profilers.

Thanks! I'm glad that you would like to support it on macOS. And I'm happy to discuss the details further and update the code later. (Enjoy your vacation!)

@canova One thing I am concerned is that for perf I had to add jit support because nothing compiles with frame pointers in the wild which means that unless people compile Python themselves (and take a big-ish hit unfortunately) the frame-pointer version is suboptimal. What's the status of this for samply? Does it also support the perf jit interface? Can you ping here the samply maintainer if the answer is nuanced?

Samply maintainer here - I don't understand the question, I'm afraid. What do you mean by "add jit support", and what do you mean by "the perf jit interface"?

Samply supports unwinding with DWARF information if available, but only for regular code mapped from a binary / shared library, not for JIT code. When it encounters JIT code during unwinding, it falls back to using frame pointers for that frame. It does not make use of any JIT_CODE_UNWINDING_INFO records from the jitdump.

Samply maintainer here - I don't understand the question, I'm afraid. What do you mean by "add jit support", and what do you mean by "the perf jit interface"?

Samply supports unwinding with DWARF information if available, but only for regular code mapped from a binary / shared library, not for JIT code. When it encounters JIT code during unwinding, it falls back to using frame pointers for that frame. It does not make use of any JIT_CODE_UNWINDING_INFO records from the jitdump.

Perf supports two modes to deal with JIT compilers which is what we are leveraging to make it work with Python via the trampolines:

• A simple perf maps based interface described here: https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/jit-interface.txt
• A much more complex JIT interface described here: https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/jitdump-specification.txt

We do implement both. The reason we implement both is that the simple maps interface works as long as perf can unwind through the JIT frames. Perf uses libunwind or eelfutils for this and both choke when Python is not compiled with frame pointers. Check https://docs.python.org/3/howto/perf_profiling.html#how-to-obtain-the-best-results for more info.

Unfortunately, nobody compiles Python with frame pointers because it's very slow. See #96174. Not only that, even if you compile with frame pointers most of the time wheels and binary packages are not compiled with frame pointers so this is sort of not very useful in the wild unless you have control over your full ecosystem.

To deal with this we implement the much more complex JIT interface. This allows us to provide to perf DWARF for the trampolines so we can pass the eh_frame values and the unwinding information. perf then uses these via some horrible code that creates a single ELF file per trampoline (yuk!) and then uses that to interpret the samples it took. The problem with this mode is that to do this perf must dump the entire stack to disk to analyze it later. Unfortunately Python C stack is very big so this not only is slow but almost all the time you need the MAX size of the dump. This ends with gigantic files and a much slower compilation. You can see our support for this in https://github.com/python/cpython/blob/main/Python/perf_jit_trampoline.c

I hope this gives some background for the question.

I really want to know if simply is going to be able cope with the general case in general in Linux and Mac for all arches that we support.

I am asking this among other things because we already had problems with code that doesn't have frame pointers around the trampolines. See #130856.

Also, we probably need a buildbot and tests for this to ensure this doesn't break in the future

You can see our support for this in https://github.com/python/cpython/blob/main/Python/perf_jit_trampoline.c

Thanks for the pointer! Wow, I feel slightly nauseous after reading the part about having to to space out the trampolines by the size of the unwind info... Anyway:

samply supports both perf.map file and jitdump. But as I said above, it does not respect the JIT unwind info. Instead, when finding the return address for a JIT frame, samply always uses framepointer unwinding. Looking at the actual assembly for the trampolines, I can see the following:

• On aarch64, the trampoline actually has a frame pointer - the stp x29, x30, [sp, -16]!; mov x29, sp instructions make it so that the x29 ("fp") register is set to the location where the caller's framepointer and the return address are stored. So samply's approach should just work.
• On x86_64, the trampoline does not set up a frame record. But it also doesn't clobber the rbp register. So: If the python C functions are compiled with framepointers, then unwinding can still resume, but the immediate caller of the JIT frame will be missing. And if the python C functions do not use framepointers, then unwinding will likely fail, because unwinding the caller using dwarf information usually requires an accurate rsp value, which samply won't have unless it starts respecting the jitdump unwind info.

For samply's current approach to work on x86_64, the x86_64 trampoline just needs to set up a framepointer - add a push rbp, mov rbp rsp at the start and a pop rbp at the end, adjust the jitdump dwarf to match (for Linux perf), and that should be it, no? Having a framepointer in the trampoline doesn't require using framepointers when compiling the python C code; it's a per-function decision. So IMO such a change would be very much orthogonal to #96174.

Thanks a lot @mstange, that makes perfect sense and really clarifies the situation!

If we’re going to add samply support, we need to ensure it works reliably on both aarch64 and x86_64 across both macOS and Linux. Based on your analysis, we should definitely add frame pointers to the x86_64 trampoline in this PR to ensure consistent behavior across architectures.

I can help set up some buildbot infrastructure to ensure all of this works properly with samply once the PR is ready for more comprehensive testing.

I also think we should have full end-to-end documentation that covers how to use this with Python, similar to our existing perf docs. We should probably make the docs more generic to cover both perf and samply workflows, since users will likely want guidance on both tools depending on their platform and preferences.

@canova would you be willing to add the frame pointer setup to the x86_64 trampoline and the DWARF as part of this PR? It seems like the right time to ensure sample works everywhere and we can advertise it properly

Thanks for the pointer! Wow, I feel slightly nauseous after reading the part about having to to space out the trampolines by the size of the unwind info...

I feel you: I was certainly nauseous when I had to debug that nonsense for hours and hours and even more when I had to implement the hack to "fix" it.

Edit: ah I think you are referring to the mmap to tell Perf about the maps file no? Sorry I thought you were referring to the jit trampoline compiler that itself uses mmap to get the chunks.

Yes, I mean maps file here. Sorry for confusing

I updated the PR to remove this on macOS, let me know what you think!

SGTM

@mstange @pablogsal Thanks for the investigation!

@canova would you be willing to add the frame pointer setup to the x86_64 trampoline and the DWARF as part of this PR? It seems like the right time to ensure sample works everywhere and we can advertise it properly

Sure that sounds good to me, I'll update the PR soon with this and add some more tests.

Would you like me to update the documentation in this PR as well?

@mstange @pablogsal Thanks for the investigation!

@canova would you be willing to add the frame pointer setup to the x86_64 trampoline and the DWARF as part of this PR? It seems like the right time to ensure sample works everywhere and we can advertise it properly

Sure that sounds good to me, I'll update the PR soon with this and add some more tests.

Would you like me to update the documentation in this PR as well?

This should do the trick, but needs checking and a fix for CET:

diff --git a/Python/asm_trampoline.S b/Python/asm_trampoline.S
index 616752459ba..a14e68c0e81 100644
--- a/Python/asm_trampoline.S
+++ b/Python/asm_trampoline.S
@@ -12,9 +12,10 @@ _Py_trampoline_func_start:
 #if defined(__CET__) && (__CET__ & 1)
     endbr64
 #endif
-    sub    $8, %rsp
-    call    *%rcx
-    add    $8, %rsp
+    push   %rbp
+    mov    %rsp, %rbp
+    call   *%rcx
+    pop    %rbp
     ret
 #endif // __x86_64__
 #if defined(__aarch64__) && defined(__AARCH64EL__) && !defined(__ILP32__)
diff --git a/Python/perf_jit_trampoline.c b/Python/perf_jit_trampoline.c
index 2ca18c23593..671b56e0846 100644
--- a/Python/perf_jit_trampoline.c
+++ b/Python/perf_jit_trampoline.c
@@ -401,10 +401,12 @@ enum {
     DWRF_CFA_nop = 0x0,                    // No operation
     DWRF_CFA_offset_extended = 0x5,        // Extended offset instruction
     DWRF_CFA_def_cfa = 0xc,               // Define CFA rule
+    DWRF_CFA_def_cfa_register = 0xd,      // Define CFA register
     DWRF_CFA_def_cfa_offset = 0xe,        // Define CFA offset
     DWRF_CFA_offset_extended_sf = 0x11,   // Extended signed offset
     DWRF_CFA_advance_loc = 0x40,          // Advance location counter
-    DWRF_CFA_offset = 0x80                // Simple offset instruction
+    DWRF_CFA_offset = 0x80,               // Simple offset instruction
+    DWRF_CFA_restore = 0xc0               // Restore register
 };

 /* DWARF Exception Handling pointer encodings */
@@ -868,17 +870,22 @@ static void elf_init_ehframe(ELFObjectContext* ctx) {
          * conventions and register usage patterns.
          */
 #ifdef __x86_64__
-        /* x86_64 calling convention unwinding rules */
+        /* x86_64 calling convention unwinding rules with frame pointer */
 #  if defined(__CET__) && (__CET__ & 1)
-        DWRF_U8(DWRF_CFA_advance_loc | 8);    // Advance location by 8 bytes when CET protection is enabled
-#  else
-        DWRF_U8(DWRF_CFA_advance_loc | 4);    // Advance location by 4 bytes
+        DWRF_U8(DWRF_CFA_advance_loc | 4);    // Advance past endbr64 (4 bytes)
 #  endif
-        DWRF_U8(DWRF_CFA_def_cfa_offset);     // Redefine CFA offset
-        DWRF_UV(16);                          // New offset: SP + 16
-        DWRF_U8(DWRF_CFA_advance_loc | 6);    // Advance location by 6 bytes
-        DWRF_U8(DWRF_CFA_def_cfa_offset);     // Redefine CFA offset
-        DWRF_UV(8);                           // New offset: SP + 8
+        DWRF_U8(DWRF_CFA_advance_loc | 1);    // Advance past push %rbp (1 byte)
+        DWRF_U8(DWRF_CFA_def_cfa_offset);     // def_cfa_offset 16
+        DWRF_UV(16);
+        DWRF_U8(DWRF_CFA_offset | DWRF_REG_BP); // offset r6 at cfa-16
+        DWRF_UV(2);
+        DWRF_U8(DWRF_CFA_advance_loc | 3);    // Advance past mov %rsp,%rbp (3 bytes)
+        DWRF_U8(DWRF_CFA_def_cfa_register);   // def_cfa_register r6
+        DWRF_UV(DWRF_REG_BP);
+        DWRF_U8(DWRF_CFA_advance_loc | 3);    // Advance past call *%rcx (2 bytes) + pop %rbp (1 byte) = 3
+        DWRF_U8(DWRF_CFA_def_cfa);            // def_cfa r7 ofs 8
+        DWRF_UV(DWRF_REG_SP);
+        DWRF_UV(8);
 #elif defined(__aarch64__) && defined(__AARCH64EL__) && !defined(__ILP32__)
         /* AArch64 calling convention unwinding rules */
         DWRF_U8(DWRF_CFA_advance_loc | 1);        // Advance location by 1 instruction (stp x29, x30)

@canova I have made a PR to add the frame pointers and the DWARF as it was slighly more tricky than I though: #136500

@canova I have made a PR to add the frame pointers and the DWARF as it was slighly more tricky than I though: #136500

Ah cool, thank you! I can rebase this PR on top of yours.

@canova @mstange can you confirm that samply works as expected with #136500 ?

Would you like me to update the documentation in this PR as well?

Yep please.

@canova @mstange can you confirm that samply works as expected with #136500 ?

Just tested your patch using samply. I can verify that it fixes the stack walking! Here are before and after profiles:
Before your patch / After your patch

@canova @mstange do you know why the Python frames appear duplicated? Is that a samply bug?

It's a samply bug, yes - or rather a workaround that's not necessary for Python because there's only a single native function for each Python function (just the trampoline itself). The duplication was initially added for JS JITs which can compile the same function multiple times with different JIT tiers, and it was useful to both have a single call node per JS function and to have a separate call node for each tier.

@pablogsal I just added two new commits for documentation and testing.

For documentation, I made the perf profiling page more generic and added a section for samply. Started small, so I can shape it along the way with the feedback I get.

For the test, we already have some test coverage already from previous tests for perf. But now I added some samply tests as well. (they might fail on x86_64 since I haven't rebased on top of your patch yet. I'll do it once it's merged) Let me know what you think about them. Currently samply tests will be skipped if samply is not installed. I think you were talking about setting up some buildbot tasks. Would that be possible to install samply?