Skip to content

[Storage] Encode text as UTF-8 when uploading an object via stream - #2235

Open
shaynhornik wants to merge 1 commit into
apache:trunkfrom
shaynhornik:fix/storage-stream-upload-text-chunks
Open

shaynhornik wants to merge 1 commit into
apache:trunkfrom
shaynhornik:fix/storage-stream-upload-text-chunks

Conversation

@shaynhornik

Copy link
Copy Markdown

[Storage] Encode text as UTF-8 when uploading an object via stream

Description

Fixes #1802. upload_object_via_stream with a stream that yields text (a StringIO, or a file opened in text mode) passes the stream to the HTTP client as is, while the object hash is calculated from the UTF-8 encoded data. What the client does with text depends on the urllib3 version (checked against a local HTTP server):

Stream with non-latin-1 text urllib3 1.26 urllib3 2.x
StringIO UnicodeEncodeError (the traceback in the issue) no error, but Content-Length is the character count, so the upload is truncated
Text-mode file UnicodeEncodeError correct

This affects drivers that upload a stream in one request through StorageDriver._upload_object, such as Google Storage and CloudFiles.

Change: _upload_object now passes the stream through a new _get_stream_request_data():

  • A text-mode file-like object (io.TextIOBase) is read in CHUNK_SIZE blocks and encoded as UTF-8 lazily, so the stream is not buffered in memory.
  • str chunks from a plain iterator are encoded too. urllib3 already sends those as UTF-8, so this part only makes the behaviour explicit.
  • A binary file-like object is passed as the same object, so the client can still work out the content length.

_hash_buffered_stream is unchanged, so the hash and bytes_transferred now match what is sent.

Things to know:

  • A text-mode file-like object is now sent with Transfer-Encoding: chunked instead of a Content-Length, since the encoded size isn't known without another pass over the stream. That is the path generators already take.
  • A custom file-like object whose read() returns str without being an io.TextIOBase is not covered.
  • The OSS driver has its own copy of _upload_object and is not changed here. S3 multipart, Azure Blobs, Backblaze B2 and Atmos already encode through read_in_chunks.
  • Tested with unit tests and a local HTTP server, not against real Google Storage or CloudFiles.

Status

Done, ready for review.

Checklist (tick everything that applies)

  • Code linting (required, can be done after the PR checks)
  • Documentation (CHANGES.rst)
  • Tests
  • ICLA (required for bigger changes)

Run locally: libcloud/test/storage (972 passed, 12 skipped; also with urllib3 1.26.20), the full suite with tox's flags (7043 passed, 17 skipped), black, isort, flake8, mypy and rstcheck. The two new text tests fail on trunk; the binary pass-through test passes on both.

Generated-by: Claude Code (Claude Fable 5.2). I reviewed and tested the change.

🤖 Generated with Claude Code

A stream which yields text, such as a file opened in text mode, was
passed to the HTTP client as is. The client encoded it as latin-1, which
fails with UnicodeEncodeError for other characters, or sent it with a
content length counted in characters, while the object hash was already
calculated from the UTF-8 encoded data. Text chunks are now encoded as
UTF-8 in the shared upload code before they are sent, and binary streams
are passed through unchanged.

Generated-by: Claude Code (Claude Fable 5.2)
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

"UnicodeEncodeError: 'latin-1' codec can't encode character, ordinal not in range" on GCS upload_object_via_stream

1 participant