X Tutup
The Wayback Machine - https://web.archive.org/web/20201105205406/https://github.com/systemd/casync/pull/221
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix curle partial file on downloading index #221

Open
wants to merge 2 commits into
base: master
from

Conversation

@gportay
Copy link
Contributor

@gportay gportay commented Jul 15, 2019

Hi,
This is a proposal to fix the issue mentioned in the second commit. This solution as a drawnback
There is an alternative solution. It consists in fixing the issue in casync process by updating the state machine to download the file after seeding.
In my solution, the size of the index is limited to CA_PROTOCOL_SIZE_MAX (1610241024). Is it enough for an index file?
@poettering, @keszybz, what do you think?

gportay added 2 commits Jun 20, 2019
This commit adds the parameter userdata to the function acquire_file()
to specify the private data given to cURL that is not necessarly the
CaRemote object.
This commit fixes a download issue. Now, the casync-http process writes
the whole index in one shot upon the download is completed, instead of
streaming it to the casync process.

The transfer may end with CURLE_PARTIAL_FILE when the download of the
index does not complete before casync starts seeding.

That issue is reproducible when the seed operation is very long. A large
device block of 9.5GB is good enough to reproduce the issue.

In this situation, the cURL transfer starts in the casync-http process
and the write_index() function streams the data received to the casync
process through stdout.

In the meanwhile, the casync process starts the long seeding operation
that causes the casync-http process to hold on the transfer as the
poller blocks it in the function process_remote().

Many minutes later, when the seeding operation ends, the poll in
casync-http returns and the function robust_curl_easy_perform() returns
CURLE_PARTIAL_FILE (which stands for "Transferred a partial file" in a
human-readable format, according to curl_easy_strerror()).

The commit does not call the poller anymore in the middle of the
transfer of the index, as for chunks. It appends the data to the realloc
buffer instead. Then, it sends the whole data upon the transfer is
complete.

The commit reuses the function write_chunk() to download both index and
chunk. It renames that function to write_buffer() that is more generic.
The buffer is then written to casync using the former function
write_index() that is freed from cURL stuff.

This commit fixes the issue below:

        $ casync extract http://localhost/rootfs-9.5GB.caibx /dev/sda6 -v
        Failed to acquire http://localhost/rootfs/rootfs-9.5G.caibx
        Failed to run synchronizer: Broken pipe
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked issues

Successfully merging this pull request may close these issues.

None yet

1 participant
You can’t perform that action at this time.
X Tutup