X Tutup
The Wayback Machine - https://web.archive.org/web/20231214201706/https://github.com/python/cpython/issues/81276
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature Request]: Add zstd support in tarfile #81276

Open
evan0greenup mannequin opened this issue May 30, 2019 · 9 comments
Open

[Feature Request]: Add zstd support in tarfile #81276

evan0greenup mannequin opened this issue May 30, 2019 · 9 comments
Labels
3.10 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@evan0greenup
Copy link
Mannequin

evan0greenup mannequin commented May 30, 2019

BPO 37095
Nosy @gustaebel, @lilydjwg, @serhiy-storchaka, @animalize, @websurfer5, @erlend-aasland

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-05-30.03:42:48.084>
labels = ['type-feature', 'library', '3.10']
title = '[Feature Request]: Add zstd support in tarfile'
updated_at = <Date 2021-10-26.10:45:36.871>
user = 'https://bugs.python.org/evan0greenup'

bugs.python.org fields:

activity = <Date 2021-10-26.10:45:36.871>
actor = 'yan12125'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-05-30.03:42:48.084>
creator = 'evan0greenup'
dependencies = []
files = []
hgrepos = []
issue_num = 37095
keywords = []
message_count = 7.0
messages = ['343945', '356498', '373583', '373634', '374123', '375472', '376095']
nosy_count = 11.0
nosy_names = ['lars.gustaebel', 'daniel.ugra', 'lilydjwg', 'serhiy.storchaka', 'wicher', 'malin', 'Jeffrey.Kintscher', 'evan0greenup', 'erlendaasland', 'Jerrod Frost', 'Anatol Pomozov']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue37095'
versions = ['Python 3.10']
@evan0greenup
Copy link
Mannequin Author

evan0greenup mannequin commented May 30, 2019

Zstandard is getting more and more popular. It could be awesome if tarfile support this compression format for .tar.zst file.

@evan0greenup evan0greenup mannequin added 3.7 (EOL) end of life stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels May 30, 2019
@tirkarthi tirkarthi added 3.8 only security fixes 3.9 only security fixes and removed 3.7 (EOL) end of life labels May 30, 2019
@JerrodFrost
Copy link
Mannequin

JerrodFrost mannequin commented Nov 12, 2019

Curious about this as well.

@AnatolPomozov
Copy link
Mannequin

AnatolPomozov mannequin commented Jul 13, 2020

Is there any progress with this feature development?

Arch Linux uses Python tar library for its toolset. Arch devs are looking to add ZSTD support to the toolset but it needs this feature to be implemented.

@ambv ambv added 3.10 only security fixes and removed 3.8 only security fixes 3.9 only security fixes labels Jul 13, 2020
@animalize
Copy link
Mannequin

animalize mannequin commented Jul 14, 2020

Add zstd support in tarfile

This requires the stdlib to contain a Zstandard module.

You can ask in the Idea forum:
https://discuss.python.org/c/ideas

@serhiy-storchaka
Copy link
Member

The tarfile module supports arbitrary compressions by using the stream mode. You only need to use a third-party library which provides zstd support.

Recent versions of the tar utility has options to explicit support of new compressions: --lzip, --lzma, --lzop, --zstd, so corresponding modes can be added to the tarfile module. But it needs to include the support of these compressions in the stdlib. It should be discussed on the Python-ideas mailing list.

https://mail.python.org/mailman3/lists/python-ideas.python.org/

@animalize
Copy link
Mannequin

animalize mannequin commented Aug 15, 2020

There are two zstd modules on pypi:

https://pypi.org/project/zstd/
https://pypi.org/project/zstandard/

The first one is too simple.

The second one is powerful, but has too many APIs:
ZstdCompressorIterator
ZstdDecompressorIterator
ZstdCompressionReader
ZstdCompressionWriter
ZstdCompressionChunkerIterator
(multi-thread compression)

IMO these are not necessary for stdlib.

In addition, it needs to add something, such as the max_length parameter, and a ZstdFile class that can be integrated with the tarfile module. These workloads are not big.

I looked at the zstd API, it's a bit simpler than lzma/bz2/zlib. If spend a month, should be able to make a zstd module for stdlib. Then discuss the detailed API on Python-Ideas.

I once wanted to do this job, but it seems my time does not allow it. If anyone wants to do this work, please reply here.

FYI, Python 3.10 schedule:
3.10.0 beta 1: 2021-05-03 (No new features beyond this point.)

@animalize
Copy link
Mannequin

animalize mannequin commented Aug 30, 2020

I have spent two weeks, almost complete the code, a preview:
https://github.com/animalize/cpython/pull/8/files

Write directly for stdlib, since there are already zstd modules on pypi.
In addition, the API of zstd is simple, not as complicated as lzma.

Can also use these:
1, argument clinic
2, multi-phase init
3. internal function _PyLong_AsInt

@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
@TheAssassin TheAssassin mentioned this issue May 15, 2022
Closed
@Techcable
Copy link

Techcable commented Sep 8, 2022 •
edited

@animalize wrote a pyzstd module that closely matches the gzip/lama API

The other main contender zstandard is very advanced, but doesn't try to adapt to the stdlib tarfile API....

@dralley
Copy link

dralley commented May 25, 2023

@animalize The PR you created is between branches on your own fork, is there any chance you could submit that PR against CPython upstream?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.10 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement
Projects
Compression issues
Status: No status
Tarfile issues
Status: No status
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ambv @dralley @serhiy-storchaka @tirkarthi @Techcable
X Tutup