X Tutup
The Wayback Machine - https://web.archive.org/web/20241209100520/https://github.com/python/cpython/issues/127405
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set sys.abiflags on Windows to make check free-threaded/debug builds easier #127405

Open
XuehaiPan opened this issue Nov 29, 2024 · 6 comments · May be fixed by #127406
Open

Set sys.abiflags on Windows to make check free-threaded/debug builds easier #127405

XuehaiPan opened this issue Nov 29, 2024 · 6 comments · May be fixed by #127406
Labels
build The build process and cross-build OS-windows type-feature A feature request or enhancement

Comments

@XuehaiPan
Copy link
Contributor

XuehaiPan commented Nov 29, 2024

Feature or enhancement

Proposal:

Currently, the sys.abiflags attribute is only set on Unix systems. On Windows, there are no sys.abiflags set (i.e. hasattr(sys, 'abiflags') -> False). It is hard to tell if Python is in free-threaded or debug builds. Although developers can use something like sysconfig.get_config_var('Py_GIL_DISABLED'). It is not well documented and is not convenient.

if not sys.platform.startswith('win'):
self.assertIsInstance(sys.abiflags, str)

cpython/Lib/test/test_sys.py

Lines 1294 to 1297 in 3afb639

@test.support.cpython_only
@unittest.skipUnless(hasattr(sys, 'abiflags'), 'need sys.abiflags')
def test_disable_gil_abi(self):
self.assertEqual('t' in sys.abiflags, support.Py_GIL_DISABLED)

It would be nice if we could set ABIFLAGS in pyconfig.h and enable sys.abiflags on Windows. This would simplify much code in the wild world and also in the CPython repo.

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Linked PRs

@XuehaiPan XuehaiPan added the type-feature A feature request or enhancement label Nov 29, 2024
@XuehaiPan XuehaiPan linked a pull request Nov 29, 2024 that will close this issue
@picnixz picnixz added OS-windows build The build process and cross-build labels Nov 29, 2024
@zooba
Copy link
Member

zooba commented Dec 2, 2024

I'm not convinced this is non-impactful elsewhere. We would need to make sure that anywhere that uses something "defined" by sys.abiflags is actually using those flags (e.g. the debug flag doesn't currently appear in ABI tags, and so we may be breaking the definition by putting this attribute here).

The lack of documentation for experimental builds is somewhat deliberate right now. It's being documented outside of the main docs, as we don't want to confuse users with features that are not officially part of CPython. If you want more info about how to work with free-threaded builds, you'll have to find that (I don't remember where it is, but the discussion forums are a good place to start looking).

@XuehaiPan
Copy link
Contributor Author

XuehaiPan commented Dec 2, 2024

I'm not convinced this is non-impactful elsewhere. We would need to make sure that anywhere that uses something "defined" by sys.abiflags is actually using those flags

Here is the search result of the following statements on GitHub:

Issue on the cpython GitHub repo: Cross compiling issue on Windows


(e.g. the debug flag doesn't currently appear in ABI tags, and so we may be breaking the definition by putting this attribute here).

The debug flag is included in the ABI tags. Here is the result that the Python is configured with --with-pydebug.

$ python3 -c 'import sysconfig; print(sysconfig.get_config_var("EXT_SUFFIX"))'
.cpython-314td-darwin.so

d is included in the abiflags for debug builds: https://docs.python.org/3/using/configure.html#python-debug-build

3.3.8. Python Debug Build

A debug build is Python built with the --with-pydebug configure option.

Effects of a debug build:

  • Display all warnings by default: the list of default warning filters is empty in the warnings module.
  • Add d to sys.abiflags. <====
  • Add sys.gettotalrefcount() function.
    ...

A fresh installation built with --with-pydebug --disable-gil:

$ python3 -VV
Python 3.14.0a2+ experimental free-threading build (heads/main:c46acd35888, Dec  3 2024, 01:17:32) [Clang 16.0.0 (clang-1600.0.26.4)]
$ python3 -c 'import sys; print(sys.abiflags)'
td
$ python3 -c 'import sysconfig; print(sysconfig.get_config_var("EXT_SUFFIX"))'
.cpython-314td-darwin.so

@zooba
Copy link
Member

zooba commented Dec 4, 2024

The debug flag is included in the ABI tags.

Not on Windows - we use a _d suffix on the name rather than putting it in the ABI (historical reasons, but not easily changed).

>>> sysconfig.get_config_var("EXT_SUFFIX")
'_d.cp314-win_amd64.pyd'

There are a number of differences like this between Windows and other platforms. It's really meaningless to test any of this on non-Windows and then assume it applies cross-platform, it really won't. (Also don't put too much faith in the docs. They're largely written by Linux developers for Linux developers, and only rarely branch out into other platforms.)

@XuehaiPan
Copy link
Contributor Author

The debug flag is included in the ABI tags.

Not on Windows - we use a _d suffix on the name rather than putting it in the ABI (historical reasons, but not easily changed).

>>> sysconfig.get_config_var("EXT_SUFFIX")
'_d.cp314-win_amd64.pyd'

There are no ABIFLAGS in C and sys.abiflags in Python on Windows.

$ .\arm64\python3.14t_d.exe -c 'import pprint, sysconfig; pprint.pprint(sysconfig.get_config_vars())'
{'BINDIR': 'C:\\Users\\PanXuehai\\Projects\\cpython\\PCbuild\\arm64',
 'BINLIBDEST': 'C:\\Users\\PanXuehai\\Projects\\cpython\\Lib',
 'EXE': '.exe',
 'EXT_SUFFIX': '_d.cp314t-win_arm64.pyd',
 'INCLUDEPY': 'C:\\Users\\PanXuehai\\Projects\\cpython\\Include',
 'LDLIBRARY': 'python314t_d.dll',
 'implementation_lower': 'python',
 'installed_base': 'C:\\Users\\PanXuehai\\Projects\\cpython',
 'installed_platbase': 'C:\\Users\\PanXuehai\\Projects\\cpython',
 'platbase': 'C:\\Users\\PanXuehai\\Projects\\cpython',
 'platlibdir': 'DLLs',
 'prefix': 'C:\\Users\\PanXuehai\\Projects\\cpython',
 'projectbase': 'C:\\Users\\PanXuehai\\Projects\\cpython',
 'py_version': '3.14.0a2+',
 'py_version_nodot': '314',
 'py_version_nodot_plat': '314t-arm64',
 'py_version_short': '3.14',
 'srcdir': 'C:\\Users\\PanXuehai\\Projects\\cpython',
 'userbase': 'C:\\Users\\PanXuehai\\AppData\\Roaming\\Python'}

I wonder if adding something that did not exist before would break historical things. EXT_SUFFIX is calculated in C here which does not rely on ABIFLAGS (not exists):

#ifdef MS_WINDOWS
#include <windows.h>
typedef FARPROC dl_funcptr;
#ifdef _DEBUG
# define PYD_DEBUG_SUFFIX "_d"
#else
# define PYD_DEBUG_SUFFIX ""
#endif
#ifdef Py_GIL_DISABLED
# define PYD_THREADING_TAG "t"
#else
# define PYD_THREADING_TAG ""
#endif
#ifdef PYD_PLATFORM_TAG
# define PYD_SOABI "cp" Py_STRINGIFY(PY_MAJOR_VERSION) Py_STRINGIFY(PY_MINOR_VERSION) PYD_THREADING_TAG "-" PYD_PLATFORM_TAG
#else
# define PYD_SOABI "cp" Py_STRINGIFY(PY_MAJOR_VERSION) Py_STRINGIFY(PY_MINOR_VERSION) PYD_THREADING_TAG
#endif
#define PYD_TAGGED_SUFFIX PYD_DEBUG_SUFFIX "." PYD_SOABI ".pyd"
#define PYD_UNTAGGED_SUFFIX PYD_DEBUG_SUFFIX ".pyd"
#else
typedef void (*dl_funcptr)(void);
#endif

@zooba
Copy link
Member

zooba commented Dec 5, 2024

I wonder if adding something that did not exist before would break historical things.

Yes, this is my concern, but I'm far more concerned about 3rd party code using sys.abiflags to calculate things that currently don't depend on ABIFLAGS and won't also be updated.

This seems much like the discussion we had about using cp3.10 instead of cp310, where eventually we decided to not make the change because it would break practically everyone who made any assumption about the tag at all for limited benefit.

Being able to add t here would be an actual benefit, but I'm not sure it's worth the risk of breakage for 2-3 versions (when it should go away again when we go back to a single ABI). If there are other ABI flags worth adding that would be permanent, we should consider it, but I'm not aware of any right now.

@effigies
Copy link
Contributor

effigies commented Dec 8, 2024

Being able to add t here would be an actual benefit, but I'm not sure it's worth the risk of breakage for 2-3 versions (when it should go away again when we go back to a single ABI). If there are other ABI flags worth adding that would be permanent, we should consider it, but I'm not aware of any right now.

sys.abiflags has been around since 3.2 and was empty from 3.8-3.12. It doesn't seem to be a problem to leave it as an empty string for years on end, and it provides a standard place to make a mark if there is ever another need for multiple ABIs.

For a concrete use-case, I'm trying to patch tox-gh-actions to detect free-threaded builds (ymyzk/tox-gh-actions#199), which need slightly different configurations from the regular GIL builds. It's breaking for Windows, and falling back to 3.13 for Windows just means I can't test free-threaded Python on Windows unless I find some other contortion. For the moment, I will probably parse EXT_SUFFIX, but that hardly seems less brittle than a sequence of character codes that may be empty.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build The build process and cross-build OS-windows type-feature A feature request or enhancement
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants
X Tutup