Skip to content

fix(download): ignore Content-Length when Content-Encoding is set#1857

Open
jlaportebot wants to merge 1 commit into
httpie:masterfrom
jlaportebot:fix/download-content-encoding-gzip
Open

fix(download): ignore Content-Length when Content-Encoding is set#1857
jlaportebot wants to merge 1 commit into
httpie:masterfrom
jlaportebot:fix/download-content-encoding-gzip

Conversation

@jlaportebot
Copy link
Copy Markdown

Summary

When a server returns Content-Encoding: gzip (or any non-identity encoding), requests/urllib3 transparently decompresses the response body. However, Content-Length reflects the compressed size per RFC 9110 §8.6. This mismatch caused HTTPie to flag complete downloads as "Incomplete download" because the decompressed bytes exceeded Content-Length.

Example

Server returns:

Content-Length: 5084527
Content-Encoding: gzip

HTTPie currently reports:

Incomplete download: size=5084527; downloaded=42846965

This is a false positive — the download is actually complete; 42846965 is the decompressed size.

Fix

Set total_size to None when a non-identity Content-Encoding is present, preventing the size comparison that produces the false "incomplete" error. This matches the behavior of curl and wget, which correctly trust Content-Length as the compressed size and simply save the file.

Test Plan

  • Added test_download_with_content_encoding_gzip: verifies a download with Content-Encoding: gzip and decompressed body larger than Content-Length is not flagged as incomplete
  • Added test_download_with_content_encoding_identity_uses_content_length: verifies Content-Encoding: identity still uses Content-Length for progress
  • Added test_download_with_content_encoding_gzip_interrupted: edge case test
  • All 38 existing download tests continue to pass

Fixes #1642
Also addresses #423 (the existing FIXME comment about Content-Encoding: gzip)

When a server returns Content-Encoding: gzip (or any non-identity
encoding), requests/urllib3 transparently decompresses the response
body. However, Content-Length reflects the compressed size per
RFC 9110 §8.6. This mismatch caused HTTPie to flag complete downloads
as "Incomplete download" because the decompressed bytes exceeded
Content-Length.

Fix by setting total_size to None when a non-identity Content-Encoding
is present, preventing the size comparison that produces the false
'incomplete' error. This matches the behavior of curl and wget.

Fixes httpie#1642
Also addresses httpie#423 (the existing FIXME comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🐛 Bug Report: http --download misinterprets Content-Length when Content-Encoding: gzip is set

1 participant