This seems to be an issue with libunwind and pointer authentication. Works as expected when adding arm64.nopauth to the kernel command line.
This might be fixed in libunwind 1.8.0. There is a similar report in the libunwind tracker which reports the issue as fixed with 1.8.0: access_mem in aaarch64 coredump · Issue #260 · libunwind/libunwind · GitHub