gh-86642: Allow cross compiling on darwin #23523

isuruf · 2020-11-26T18:53:04Z

This was used for providing macos-arm64 builds for conda where we are using cross compiling exclusively for all macos-arm64 builds

https://bugs.python.org/issue42476

Issue: Allow cross compiling python for macos-arm64 from macos-x86_64 #86642

ronaldoussoren

Please add documentation on how to cross build, without that it is hard to review this PR. This would preferably added to Mac/README.rst. In particular, I get the impression that cross building involves more than just specifying the target tuple to the configure script.

The change to platform.py is not acceptable as this affects a lot more than just the build. A random script using the platform module shouldn't get different answers when SDKROOT is present in the environment.

And out of interest: Why are you cross building to arm64? The build supports universal2 binaries out of the box, which run on both mac architectures and don't require cross compiling.

ronaldoussoren · 2020-11-26T20:00:46Z

Lib/platform.py

@@ -409,7 +409,12 @@ def win32_ver(release='', version='', csd='', ptype=''):
 def _mac_ver_xml():
    fn = '/System/Library/CoreServices/SystemVersion.plist'
    if not os.path.exists(fn):
-        return None
+        if 'SDKROOT' in os.environ:


This affects more than just the build. Why is this necessary?

To install wheels when cross-compiling, this is needed to make sure wheels know what platform that it is targetting.
Setting SDKROOT is supported by clang and I thought python should support that too. I can remove it and figure out a way to monkeypatch it if you think it is too controversial.

I'm -1 on this particular change, see also my latest comment in the main discussion.

ronaldoussoren · 2020-11-26T20:03:22Z

configure.ac

@@ -437,6 +437,18 @@ if test "$cross_compiling" = yes; then
 			_host_cpu=$host_cpu
 		esac
 		;;
+	*-*-darwin*)
+		case "$host_cpu" in


Why are there two cases?

Thanks for the review. Fixed now.

ronaldoussoren · 2020-11-26T20:05:42Z

setup.py

@@ -62,7 +62,7 @@ def get_platform():
 HOST_PLATFORM = get_platform()
 MS_WINDOWS = (HOST_PLATFORM == 'win32')
 CYGWIN = (HOST_PLATFORM == 'cygwin')
-MACOS = (HOST_PLATFORM == 'darwin')
+MACOS = (HOST_PLATFORM.startswith('darwin'))


Why is this change needed? HOST_PLATFORM will be darwin on macOS.

HOST_PLATFORM is set to _PYTHON_HOST_PLATFORM which will be darwin-arm64 when cross compiling.

ronaldoussoren · 2020-11-26T20:06:42Z

configure.ac

@@ -2480,7 +2481,8 @@ case $ac_sys_system/$ac_sys_release in
    }
    ]])],[ac_osx_32bit=yes],[ac_osx_32bit=no],[ac_osx_32bit=yes])

-    if test "${ac_osx_32bit}" = "yes"; then
+    if test -z "${MACOSX_DEFAULT_ARCH}"; then


Why add this test? I guess you are setting this variable when cross building.

On a second look, what is LIBTOOL_CRUFT doing? Is it necessary?

I haven't looked at this part of the build in a long time, so I don't know. I'm fairly sure that the configure script can be cleaned up a lot, even just the macOS related parts, but that should be out of scope for this PR.

Use of LIBTOOL_CRUFT was removed in 2503249. I'll send a separate PR.

isuruf · 2020-11-26T20:28:30Z

Please add documentation on how to cross build, without that it is hard to review this PR. This would preferably added to Mac/README.rst. In particular, I get the impression that cross building involves more than just specifying the target tuple to the configure script.

Will add docs

isuruf · 2020-11-26T20:30:01Z

And out of interest: Why are you cross building to arm64? The build supports universal2 binaries out of the box, which run on both mac architectures and don't require cross compiling.

Because at conda-forge, we don't want a universal2 binary to keep the size of the python installation smaller. We also don't build the dependencies of python as universal binaries, so it would require more work.

ned-deily

We already have a mechanism for "cross-compiling" on macOS, we just don't call it that:

 --with-universal-archs=ARCH
                          specify the kind of universal binary that should be
                          created. this option is only valid when
                          --enable-universalsdk is set; options are:
                          ("universal2", "32-bit", "64-bit", "3-way", "intel",
                          "intel-32", "intel-64", or "all") see Mac/README.rst

Note that intel-32 and intel-64 produce single architecture binaries. Rather than using the GNU method of specifying cross-compilation just for this one case on macOS, I would prefer to see adding an additional universal-archs option, say arm64, and then doing the minimum fixups necessary, like skipping extension loading and specifying a build interpreter, to truly make building without a common architecture work.

Ideally,it should also support the reverse case: building Intel-64 on arm Macs.

bedevere-bot · 2020-11-28T22:15:52Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

isuruf · 2020-11-28T22:39:17Z

@ned-deily, wouldn't that allow only cross compiling for different macos architectures within macos? My proposal would allow building for macos-arm64 from linux-x86_64.

ned-deily · 2020-11-28T22:52:07Z

We don't support building a complete Python for macOS on anything but macOS today using Apple's Developer Tools. The current Python build system is criticized as being overly complex as it stands today and the whole area of cross-compilation is among the least documented and tested, with a few select architectures having been shoehorned in. There have been calls to fund redoing Python's build system using on a standard build tool, like autotools or cmake or ..., that support cross-compilation and other features out of the box. (As far as I know, such calls are currently on hold pending the return of PSF finances to normal after the cancellations of PyCons due to the pandemic.) I don't think we should shoehorn in yet another hack on top of all the other hacks we already have there (speaking as someone who put some of those hacks there in the first place :).

ned-deily · 2020-11-28T23:50:43Z

@ronaldoussoren, what's your opinion?

ned-deily · 2020-11-29T02:19:07Z

Also, FWIW, and I'm not a lawyer (and I don't play one on the Internet), the copies of Apple Developer agreements that I see include a clause to the effect that: "You agree not to install, use or run the Apple SDKs on any non-Apple-branded computer".

ronaldoussoren · 2020-11-29T09:36:04Z

I'm in favour of adding single-architecture variants to --with-universal-archs, but that won't solve the problem this PR intends to solve: --enable-universalsdk --with-universal=archs=.... only works when at least one of the specified architectures can be used on the build machine (that is host-python == build-python), while the condaforge use case entails building arm64 on an x86_64 (and requires host-python != build-python).

This PR is acceptable in principal, as long as it doesn't complicate the build too much and it is clear that this build option is basically unsupported by the core devs. That said, the version of the PR that I reviewed earlier is not in a mergeable state yet.

I definitely don't like the change to platform.py, that is a backward incompatible change that affects way more than just building. I don't want to end up in a situation where "platform.mac_ver()" does not report the version of the currently running version of macOS due to an environment variable.

BTW. I'll probably create a PR for the --with-univeral-archs change when the dust settles on the univeral2 work (both here, in my own projects and packaging related projects).

ronaldoussoren · 2020-11-29T09:37:51Z

Lib/platform.py

@@ -409,7 +409,12 @@ def win32_ver(release='', version='', csd='', ptype=''):
 def _mac_ver_xml():
    fn = '/System/Library/CoreServices/SystemVersion.plist'
    if not os.path.exists(fn):
-        return None
+        if 'SDKROOT' in os.environ:


I'm -1 on this particular change, see also my latest comment in the main discussion.

ronaldoussoren · 2020-11-29T09:40:08Z

configure.ac

@@ -2480,7 +2481,8 @@ case $ac_sys_system/$ac_sys_release in
    }
    ]])],[ac_osx_32bit=yes],[ac_osx_32bit=no],[ac_osx_32bit=yes])

-    if test "${ac_osx_32bit}" = "yes"; then
+    if test -z "${MACOSX_DEFAULT_ARCH}"; then


I haven't looked at this part of the build in a long time, so I don't know. I'm fairly sure that the configure script can be cleaned up a lot, even just the macOS related parts, but that should be out of scope for this PR.

ronaldoussoren · 2020-11-29T09:40:42Z

setup.py

@@ -62,7 +62,7 @@ def get_platform():
 HOST_PLATFORM = get_platform()
 MS_WINDOWS = (HOST_PLATFORM == 'win32')
 CYGWIN = (HOST_PLATFORM == 'cygwin')
-MACOS = (HOST_PLATFORM == 'darwin')
+MACOS = (HOST_PLATFORM.startswith('darwin'))


ned-deily · 2020-11-29T10:37:21Z

I'm in favour of adding single-architecture variants to --with-universal-archs, but that won't solve the problem this PR intends to solve: --enable-universalsdk --with-universal=archs=.... only works when at least one of the specified architectures can be used on the build machine (that is host-python == build-python), while the condaforge use case entails building arm64 on an x86_64 (and requires host-python != build-python).

Yes, which is what I meant by "would prefer to see adding an additional universal-archs option, say arm64, and then doing the minimum fixups necessary, like skipping extension loading and specifying a build interpreter, to truly make building without a common architecture work". In other words, add the universal arch option and use the specification of a build Python to turn off extension module load checking in the build as is done in the pr and anything else that can't be performed without using the built Python. That generalizes the pr to any case of macOS archs. It doesn't allow for cross builds on non-macOS hosts but I don't think that is a use case we should support, at least in the existing build system.

github-actions · 2020-12-30T00:37:45Z

This PR is stale because it has been open for 30 days with no activity.

isuruf · 2021-09-16T18:41:05Z

cc @FFY00

isuruf · 2021-09-16T18:46:45Z

This PR is acceptable in principal, as long as it doesn't complicate the build too much and it is clear that this build option is basically unsupported by the core devs.

I've changed this PR so that it is very minimal.

furkanonder · 2023-05-09T15:23:28Z

@isuruf Could you sign the CLA and resolve the conflict?

isuruf · 2023-05-09T17:04:05Z

@furkanonder, done

Misc/NEWS.d/next/macOS/2020-11-29-14-38-17.bpo-42476.MSsU2h.rst

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

ned-deily

Thanks for simplifying the PR. But it still doesn't address @ronaldoussoren's earlier request:

Please add documentation on how to cross build, without that it is hard to review this PR.

As it stands, it is still not clear exactly how someone would use this and under what circumstances it is intended to work.

bedevere-bot · 2023-05-09T19:42:14Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

nightscape · 2023-06-15T06:54:51Z

@isuruf I'd be interested in this as well. I'm trying to build a JavaCPP ARM64 binding to CPython and unfortunately Github Actions still don't have Mac ARM64 runners...
Can you give any pointers of what's left to do?

the-knights-who-say-ni added the CLA signed label Nov 26, 2020

bedevere-bot added the awaiting review label Nov 26, 2020

isuruf changed the title ~~bpo42476: Allow cross compiling on darwin~~ bpo-42476: Allow cross compiling on darwin Nov 26, 2020

ronaldoussoren reviewed Nov 26, 2020

View reviewed changes

ned-deily requested changes Nov 28, 2020

View reviewed changes

bedevere-bot removed the awaiting review label Nov 28, 2020

bedevere-bot added the awaiting changes label Nov 28, 2020

ronaldoussoren requested changes Nov 29, 2020

View reviewed changes

isuruf force-pushed the cross-darwin branch 2 times, most recently from 496f7cd to 6fe3dd7 Compare November 29, 2020 14:25

github-actions bot added the stale Stale PR or inactive for long period of time. label Dec 30, 2020

ned-deily self-assigned this Sep 16, 2021

isuruf force-pushed the cross-darwin branch from a124a4a to 0f4f9fc Compare September 22, 2021 02:21

isuruf mentioned this pull request Sep 22, 2021

bpo-45258: search for -isysroot in addition to --sysroot #28501

Closed

isuruf force-pushed the cross-darwin branch from 4e7a31d to 959e9e6 Compare September 24, 2021 00:12

ezio-melotti removed the CLA signed label Jul 13, 2022

github-actions bot removed the stale Stale PR or inactive for long period of time. label Aug 1, 2022

Allow cross compiling for Darwin

94e3a6f

isuruf force-pushed the cross-darwin branch from 959e9e6 to d385005 Compare May 9, 2023 15:34

isuruf requested review from erlend-aasland and corona10 as code owners May 9, 2023 15:34

📜🤖 Added by blurb_it.

5c935c6

isuruf force-pushed the cross-darwin branch from d385005 to 5c935c6 Compare May 9, 2023 15:39

AA-Turner changed the title ~~bpo-42476: Allow cross compiling on darwin~~ gh-86642: Allow cross compiling on darwin May 9, 2023

bedevere-bot added awaiting review and removed awaiting changes labels May 9, 2023

AA-Turner reviewed May 9, 2023

View reviewed changes

Misc/NEWS.d/next/macOS/2020-11-29-14-38-17.bpo-42476.MSsU2h.rst Outdated Show resolved Hide resolved

isuruf and others added 2 commits May 9, 2023 13:39

Fix typos in news

3b62383

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

Merge branch 'main' into cross-darwin

ba174b2

ned-deily requested changes May 9, 2023

View reviewed changes

bedevere-bot removed the awaiting review label May 9, 2023

bedevere-bot added the awaiting changes label May 9, 2023

nightscape mentioned this pull request Jun 14, 2023

Build cpython for Mac OSX ARM64 bytedeco/javacpp-presets#1373

Merged

Merge branch 'main' into cross-darwin

ccb04cd

May	JUN	Jul
	05
2024	2025	2026

Uh oh!

gh-86642: Allow cross compiling on darwin #23523

Are you sure you want to change the base?

gh-86642: Allow cross compiling on darwin #23523

Conversation

isuruf commented Nov 26, 2020 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ronaldoussoren left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

isuruf commented Nov 26, 2020

Uh oh!

isuruf commented Nov 26, 2020

Uh oh!

ned-deily left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Nov 28, 2020

Uh oh!

isuruf commented Nov 28, 2020

Uh oh!

ned-deily commented Nov 28, 2020

Uh oh!

ned-deily commented Nov 28, 2020

Uh oh!

ned-deily commented Nov 29, 2020

Uh oh!

ronaldoussoren commented Nov 29, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ned-deily commented Nov 29, 2020

Uh oh!

github-actions bot commented Dec 30, 2020

Uh oh!

isuruf commented Sep 16, 2021

Uh oh!

isuruf commented Sep 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

furkanonder commented May 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

isuruf commented May 9, 2023

Uh oh!

Uh oh!

ned-deily left a comment

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented May 9, 2023

isuruf commented Nov 26, 2020 •

edited by bedevere-bot

Loading

ned-deily left a comment •

edited

Loading

isuruf commented Sep 16, 2021 •

edited

Loading

furkanonder commented May 9, 2023 •

edited

Loading