LLDB gives incorrect callstack using the PDB or YAML output

[Laurens van Run]

Using the PDB or YAML output (and then convert it to PDB using llvm-pdbutil) gives always a strange callstack in LLDB. There is a constant reference to __dbk_fcall_wrapperin stead of the real method.

(lldb) thread backtrace 1
  thread #1
    frame #0: 0x7565113c win32u.dll`NtUserWaitMessage + 12
    frame #1: 0x00a605e6 Tester.exe`__dbk_fcall_wrapper + 2927526
    frame #2: 0x00a5f5b8 Tester.exe`__dbk_fcall_wrapper + 2923384
    frame #3: 0x02791c90 Tester.exe`__dbk_fcall_wrapper + 33538640
    frame #4: 0x02791cf7 Tester.exe`__dbk_fcall_wrapper + 33538743
    frame #5: 0x03e092d2 Tester.exe`__dbk_fcall_wrapper + 57096338
    frame #6: 0x03e0931f Tester.exe`__dbk_fcall_wrapper + 57096415
    frame #7: 0x05753071 Tester.exe`Tester + 93
    frame #8: 0x76d07d59 kernel32.dll`BaseThreadInitThunk + 25
    frame #9: 0x779ab74b ntdll.dll`RtlInitializeExceptionChain + 107
    frame #10: 0x779ab6cf ntdll.dll`RtlClearBits + 191

‌

[Anders@Melander]

I have no experience with LLDB so I can’t really speculate as to what the cause might be.

One thing to look at is what method LLDB uses to read the PDB file:

• Does it use Microsoft’s msdia140.dll?
• Does it use its own functions?

msdia140.dll is pretty much a black box and even if the data in the PDB file and its format is now known we don’t know exactly how msdia140.dll uses it.

However, if LLDB uses its own functions then we can look at the source to see what areas of the data format to investigate for possible problems.

[Laurens van Run]

AFAIK LLDB uses its own functions to read the PDB file. I have also tried it with the LLDB_USE_NATIVE_PDB_READER environment variable (see this blog post: Debugging Wine with LLDB and VSCode | Andy Hippo (werat.dev)), but that does not make any difference.

[Anders@Melander]

Thanks for the link. That explained very well how it works.

If LLDB neither works with nor without LLDB_USE_NATIVE_PDB_READER then the problem is likely to be something other than the PDB file. From the linked blog post it seems that LLDB is picky about the stack frames of the process being debugged. Please ensure that you are compiling with:

• Optimization: False
• Stack frames: True

And linking with:

• Map file: Detailed

You might also verify that LLDB is actually finding and reading the PDB file at all. Process Monitor can be used for that.

You can also try to find a copy of cvdump.exe (the MS Debug Information Dumper) and run that against your PDB file. If it can read it without problems then the problem is elsewhere.

[Laurens van Run]

Thanks for thinking along! I have no copy of cvdump.exehere, so I could not test it. But llvm-pdbutilcan also read the PDB file and output method information:

C:\Users\....\Embarcadero\Studio\Projects\Win32\Debug>llvm-pdbutil pretty Project11.pdb --with-name=TForm18.MyMethod
Summary for C:\Users\...\Embarcadero\Studio\Projects\Win32\Debug\Project11.pdb
  Size: 1069056 bytes
  Guid: {7A4C6B0E-FB69-40FB-A33F-32A7CBD60BFA}
  Age: 1
  Attributes: HasPrivateSymbols
---SYMBOLS & TYPES BY NAME---
[1 occurrences] - TForm18.MyMethod

  public [0x001d6518] TForm18.MyMethod

The PDB-file is actually read by LLDB, I have confirmed this with ProcMon. If I remove the pdb-file, the calls in LLDB (image lookup -vn <name>) fail without any result. So, it is read correctly, but not interpreted correctly. Looking up the method that is called (TForm.MyMethod) it gives an incorrect/strange Summary. I also noticed that the rangethat LLDB shows seems to be my whole program.

(lldb) image lookup -vn __dbk_fcall_wrapper
1 match found in C:\Users\l.vanrun\OneDrive - Global Data Exchange B.V\Documenten\Embarcadero\Studio\Projects\Win32\Debug\Project11.exe:
        Address: Project11.exe[0x00411394] (Project11.exe..text + 66452)
        Summary: Project11.exe`__dbk_fcall_wrapper
         Module: file = "C:\Users\l.vanrun\OneDrive - Global Data Exchange B.V\Documenten\Embarcadero\Studio\Projects\Win32\Debug\Project11.exe", arch = "i386"
    CompileUnit: id = {0x00001493}, file = "C:\Projects\Olympus\tp\runtime\rtl\sys\delayhlp.cpp", language = "c++"
      LineEntry: [0x00481394-0x0048139b): SysInit.pas:1189
         Symbol: id = {0x00000002}, range = [0x00481394-0x0064d740), name="__dbk_fcall_wrapper"

(lldb) image lookup -vn TForm18.MyMethod
1 match found in C:\Users\l.vanrun\OneDrive - Global Data Exchange B.V\Documenten\Embarcadero\Studio\Projects\Win32\Debug\Project11.exe:
        Address: Project11.exe[0x005d6518] (Project11.exe..text + 1922328)
        Summary: Project11.exe`__dbk_fcall_wrapper + 1855876
         Module: file = "C:\Users\l.vanrun\OneDrive - Global Data Exchange B.V\Documenten\Embarcadero\Studio\Projects\Win32\Debug\Project11.exe", arch = "i386"
         Symbol: id = {0x00000002}, range = [0x00481394-0x0064d740), name="__dbk_fcall_wrapper"

‌

[Anders@Melander]

I think I’ll need to look at your data. Can I ask you to attach the following:

• The output from map2pdb with the -v -debug switches (i.e. redirect to a file)
• Your map file
• The PDB file
• The exe file

‌

[Anders@Melander]

Btw, cvdump.exe is here:

https://github.com/microsoft/microsoft-pdb/blob/master/cvdump/cvdump.exe

[Laurens van Run]

Sure thing, see the requested files attached.

‌

[Laurens van Run]

And the souce too.

[Anders@Melander]

I’ve been unable to get anywhere with this one (and I’m sure you’ve moved on) so I’m closing it for now as there’s nothing I can do.

Your map file did manage to surface an overflow error in the pdb writer, so the effort wasn’t completely in vain. I’ll push the fix for that in a jiffy.

[Anders Melander]

Fix for overflow in MSF free-block-map writer. Refs #6

→ <<cset 6ad8fc336909>>

[Laurens van Run]

Thanks for your effort. Yes, I moved on and decided to push more on Embarcadero to also move on with modern tools. Hopefully all this effort will pay off some time in the near future