cc @vstinner

As usual, the documentation does not require a NEWS (I hope I remember correctly).

I'm aware that this function returns a utf-8. Perhaps describing the function in this way would make it more robust (at the very least, we won't need to update its documentation if we modify it in the future).

I'd link to sys.getdefaultencoding, copy the doc from there, and add a .. impl-detail:: that it's always utf-8.

Perhaps this should be soft-deprecated.

Alright, something interesting happened. sys.getdefaultencoding always returns utf-8, but it's description says:

Return the name of the current default string encoding used by the Unicode implementation.

Should I modify its description? e.g., Return a "utf-8" string constant.

copy the doc from there, and add a .. impl-detail:: that it's always utf-8.

In other words, should I add .. impl-detail:: tag to sys.getdefaultencoding to point it to the PyUnicode_GetDefaultEncoding function? But here's the interesting part, it uses a completely new implementation, like the one below:

static PyObject *
get_utf8_unicode(void)
{
    _Py_DECLARE_STR(utf_8, "utf-8");
    PyObject *ret = &_Py_STR(utf_8);
    return Py_NewRef(ret);
}

Even so, do I still need to add this tag?

Hm, looking further, looks like the “current default string encoding used by the Unicode implementation” is what str.encode() uses. In Python 2, it was ascii; in Python 3 it's utf-8.

>>> import sys
>>> sys.getdefaultencoding()
'ascii'
>>> u'á'.encode()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 0: ordinal not in range(128)

>>> import ctypes
>>> ctypes.pythonapi.PyUnicodeUCS4_SetDefaultEncoding("utf-8")  # hack!
0

>>> sys.getdefaultencoding()
'utf-8'
>>> 'á'.encode()
'\xc3\xa1'

It might be good to clarify this in the sys.getdefaultencoding docs, e.g.:

Return "utf-8". This is the name of the default string encoding, used in methods like str.encode.

(So, UTF-8 is not a CPython implementation detail.)

it uses a completely new implementation

That's needed to get a Python object, as opposed to a char*

Hmm, maybe I should open a separate PR to do it.

@encukou Please review! I hope it meets your expectations. #130353

I'd prefer including the sys.getdefaultencoding change here, so that the PyUnicode_GetDefaultEncoding doc can refer to it.

Thanks @rruuaanng for the PR, and @encukou for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

Thanks @rruuaanng for the PR, and @encukou for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

GH-130511 is a backport of this pull request to the 3.13 branch.

Merged. Thank you, @rruuaanng!

GH-130512 is a backport of this pull request to the 3.12 branch.