CC @ambv @ned-deily

Unfortunately this warning also propagates to all extension modules built with the given Python.

@ronaldoussoren

-bundle_loader doesn't work with shared library builds (including the framework build), as the option does exactly as described in the documentation: It looks for symbols in the loader executable, not in shared library dependencies of that binary. I found this when looking into -bundle_loader to get link time errors when using non-existing symbols instead of only getting those when using an extension.

Linking with libpython is also problematic because on macOS shared libraries are referenced by absolute path which means linking with libpython ties the extension to a specific interpreter (e.g. makes it impossible to build an extension using the Python.org installer and load it using a homebrew install). For the user this will result in seemingly random crashes (due to libpython being loaded twice while only one is initialised).

I haven't spend time on looking into this particular issue yet, with some luck there's another option beyond disabling chained lookups.

FYI.. https://issues.guix.gnu.org/issue/57849

Looks like using -Wl,-w will help.

FYI.. https://issues.guix.gnu.org/issue/57849

Looks like using -Wl,-w will help.

That doesn't really help, it just suppresses the warning.

You can pass -Wl,-no_fixup_chains to disable them entirely for now, but that flag is undocumented and who knows if it will exist in the future.

You can pass -Wl,-no_fixup_chains to disable them entirely for now, but that flag is undocumented and who knows if it will exist in the future.

It will likely won't because chained fixups are the new linker information for the dynamic symbol tables. So we will be just delaying the inevitable

Another workaround is to target an older deployment target (before 11.0), but that's also not a long term solution.

For static builds using -Wl,-bundle_loader,$(BUILDPYTHON) -Wl,-undefined,error appears to work. I've manually patched the Makefile for some extensions and that does result in a build without warnings and I can load those extensions. What I've not checked yet is if those extensions work with a python installed into a different prefix.

This doesn't work with dynamic builds though, even when using BLDLIBRARY= -L. -Wl,-reexport-lpython$(LDVERSION) when linking $(BUILDPYTHON).

Note that just linking extensions with libpython isn't a realistic option because that results in C extensions that target a specific python prefix and hence would loose interoperability for wheels build with our installer and homebrew.

What does work for a dynamic build (again with a tweaked $(BLDLBIRARY): -Wl,-flat_namespace -Wl,-bundle_loader,./python.exe -Wl,-undefined,error, that is add -Wl,-flat_namespace. That's not ideal though because the default two-level namespaces are more efficient (and faster).

Something similar should also work for framework builds (but using a different reexport flag), but I haven't tested this yet. Mostly because I'm not happy with the link flags I've used for a shared build.

All of this was tested by manual patching of the Makefile, I haven't looked into tweaking the configure script and makefile template yet.

Hi — this issue has also been raised in the pybind11 and CMake projects, including a fix similar to what was suggested above by @ronaldoussoren.

Links:

• pybind11 issue tracker ticket: Change macOS handling of undefined symbols pybind/pybind11#4301
• CMake issue tracker ticket: https://gitlab.kitware.com/cmake/cmake/-/issues/24044

All of this raises a few questions (pybind/pybind11#4301 (comment)) that would be good to figure out together with somebody who is familiar with the intricacies of Darwin's dynamic linker.

Hi @wjakob,

We are currently waiting for some clarifications from Apple. Will follow up with anything we find.

Anyone with Apple clout could push this PR into their attention sphere: apple-opensource/ld64#1

It basically says "here's a library; make any symbol that is resolved here dynamic_lookup and in the flat namespace". Which is…exactly what is wanted here.

It basically says "here's a library; make any symbol that is resolved here dynamic_lookup and in the flat namespace". Which is…exactly what is wanted here.

Correct me if I'm wrong, but it seems to me like this PR targets a different use case: it enables dynamic_lookup selectively for chosen symbols to avoid the "sledgehammer" solution of changing the defaults for all undefined symbols.

While the issue discussed here is that the very mechanism that drives dynamic_lookup no longer works in binaries using the new "chained fixups" linker feature (which is automatically enabled for newer macOS deployment targets).

One more data point in case it is useful: explicitly enumerating all relevant Python symbols using the -U command line arguments (wjakob/nanobind@bb211d1) also works and the warnings are gone. But it leaves me wondering:

• Does it actually fix the issue? The manpage says that dynamic_lookup is still used in that case.
• The linker gets a huge command line argument (3.5KB), which seems quite ugly and would need to be tweaked for every Python version. I could not find a -U_from_file <filename> argument to batch-inform ld about the symbols to accept as being undefined.

I could not find a -U_from_file <filename> argument to batch-inform ld about the symbols to accept as being undefined.

ld response files might be useful here (though it is documented under Rarely used Options):

     @response_file_path
             Inserts contents of file at response_file_path into arguments. This allows for linker command line args to be store in a
             file.  Note: ld is normally invoked through clang, and clang also interprets @file on the command line.  To have clang
             ignore the @file and pass it through to ld, use -Wl,@file.

While the issue discussed here is that the very mechanism that drives dynamic_lookup no longer works in binaries using the new "chained fixups" linker feature (which is automatically enabled for newer macOS deployment targets).

Maybe? But if -U works, that's what this should do (IIRC it is, but it has been a long time since I poked it). Of course, the code is also unbuilt because…I have no idea what the instructions are in the first place.

Anyone with Apple clout could push this PR into their attention sphere: apple-opensource/ld64#1

It basically says "here's a library; make any symbol that is resolved here dynamic_lookup and in the flat namespace". Which is…exactly what is wanted here.

I am not sure that anyone at apple monitors that repo. If you check it has no pull requests and nobody ever answered an issue

Yeah, I'm aware. I had contacts at Apple, but they've since moved elsewhere…

I did a bit more investigation with the -U linker argument that enables specifying individual symbols as undefined. The gist is that despite not showing the ominous warning about "chained fixups", it seems to produce shared libraries with the exact same binding table entries as the previous standard solution (-undefined dynamic_lookup).

Links:

• Change macOS handling of undefined symbols pybind/pybind11#4301 (comment)
• Change macOS handling of undefined symbols pybind/pybind11#4301 (comment)

The third option being discussed (-undefined suppress -flat_namespace) apparently has severe drawbacks that were pointed out by others in the discussion thread.

Somebody from Apple needs to come here and confirm that this warning is a fluke. Otherwise, we have a problem 🙈

Somebody from Apple needs to come here and confirm that this warning is a fluke. Otherwise, we have a problem 🙈

Yeah, we are unlikely to proceed with anything in CPython unless we have direct authoritative confirmation from Apple that everything is working as we expect and we are not going to have problems down the line.

Hi @wjakob,

We are currently waiting for some clarifications from Apple. Will follow up with anything we find.

My first attempt at contacting Apple failed. Asking developer support resulted in getting forwarded to the forumes to ask about system extensions...

I'll try again after November 14th, before that my agenda is too full to quickly respond.

Somebody from Apple needs to come here and confirm that this warning is a fluke. Otherwise, we have a problem 🙈

My current plan is to try to contact someone from Apple (if he still works there), and if that fails use one of the code-level support tickets included in a paid developer subscription.