Hi,

This bug was originally reported at
https://bugs.launchpad.net/bugs/898172

ctypes/utils.py provides a find_library function which amongst other things will scan the ldconfig -p output on linux to find libraries by name. It applies some logic to filter out incompatible libraries, however the logic is mainly based on uname output which is incorrect.

We noticed because the new Debian/Ubuntu armhf ports have a slightly different ldconfig -p output than the armel ports; one gets ",hard-float" in the output, e.g.:
ld-linux.so.3 (libc6,hard-float) => /lib/arm-linux-gnueabihf/ld-linux.so.3

there's provision in find_library to allow for certain strings when uname returns certain names:
mach_map = {
'x86_64-64': 'libc6,x86-64',
'ppc64-64': 'libc6,64bit',
'sparc64-64': 'libc6,64bit',
's390x-64': 'libc6,64bit',
'ia64-64': 'libc6,IA-64',

but this is incorrect for multiple reasons:
a) this requires setting utsname properly before running a 32-bits python on a 64-bits kernel (e.g. "linux32 ./foo.py" instead of just "./foo.py"); this shouldn't be needed and breaks 32-bits userspace installations with a 64-bits kernel
b) uname output can be anything really, e.g. i486, i586, i686 etc. on 32-bits x86, or armv5l, armv6l, armv7l etc. on ARM
c) uname output doesn't indicate userspace ABI, a single kernel can support multiple ABIs; for instance ARM kernels can support EABI and OABI (old ABI) syscall ABIs at the same time, and even with the same syscall ABI like EABI the userspace calling conventions might allow for multiple ABIs to be present on the filesystem -- for instance soft-float and hard-float userspace calling conventions

I've attached a patch to ctypes/utils.py in the Launchpad bug which I'll also attach here. It will work for either soft-float or hard-float, but not if "ldconfig -p" lists both types of libraries (as will be the case with biarch or multiarch systems).

It is extremely hard to reproduce correct glibc semantics in find_library, and a linux implementation would necessarily become extremely glibc and linux specific. One possible way is to look at /proc/$pid/maps output to find information about the ABI of the currently running program, and then ask the runtime linker (ld.so) to check whether a given library is compatible or not (--verify). Another way would be to run ldd on sys.executable to find the runtime linker or libc. This is all extremely fragile and linux andglibc specific, and will likely fail in special cases.

Finally, one needs to wonder whether offering "find_library" as an API isn't calling for trouble; dlopen() requires one to state which SOVER should be used, e.g. dlopen("libmagic.so.1"), not dlopen("magic"). Allowing the first SOVER to be used means that the behavior is not determinstic and also means that people wont think of binary compatibility when implementing ctypes-based bindings. I would personally prefer if this API was deprecated and if we recommended for upstreams to use ctypes.cdll.LoadLibrary("libmagic.so.1") constructs.

Cheers,

While I'm at it, find_library also tries creating temp files when running gcc and other issues mention trouble running gcc or propose running ld:
http://bugs.python.org/issue9998
http://bugs.python.org/issue5289

IMHO, calling binutils/gcc is troublesome, it's not necessarily installed on production systems and creating tempfiles when running a program just to locate a library is fragile at best.

This issue is still up to date. Most impacted are distos like for example Buldroot (http://buildroot.org/). Usual production rootfs provide neither gcc nor /sbin/ldconfig nor objdump. So find_library() is doomed from the very beginning.

Such packages like pyusb, pyudev etc. rely on find_library() and are not usable in such environment. Pyusb developers had to create an extra backend to overcome this issue (https://github.com/walac/pyusb/pull/29).

So common solution would be very useful for such environments.

I am still encountering this issue when using pyudev on Python 3.8.5.

This issue was resolved via 8c52027 around the time that this bug was filed.

Thanks for finding that!

The message before yours seems to contradict it; we could add tests (monkey-patching the output of the subprocess call) to be sure either way.

Debian has been carrying the patch to fix this forever. The patch was written by @lool, the bug submitter.

That patch got mangled in the 3.x transition to the point that it was entirely noop. I went back to find out what the story was, and found that the commit I referenced had made the critical change that resolved the bug described here. find_library does not break in either hard or soft float 32-bit ARM environments. _findSoname_ldconfig (the function that was patched) also behaves correctly.

I can see why they didn't, previously, but the commit I referenced fixed the issue. The patch here only would have fixed the issue for hard-float ARM. Nobody ever noticed that, because it resolved separately.

I don't know what bug @trevornewton above ran into, but I don't think it was the bug described here.