Determine this is the right repository
Summary of the issue
When handling a doubly gzipped file (the first test file in the code below), the reader fails to reach the end of file.
API client name and version
google-cloud-storage 3.14.1
Reproduction steps: code
file: main.py
from google.cloud.storage import Blob
from google.cloud.storage import Bucket
from google.cloud.storage import Client
from google.cloud.storage.fileio import BlobReader
client = Client("apache-beam-testing")
bucket = Bucket(client=client, name="apache-beam-samples")
test_files = [
"textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz",
"textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz",
"textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz",
"textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz",
"textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz",
"textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz",
"textio/textio-test-data.default.1k.txt",
"textio/textio-test-data.default.1k.txt.gz",
"textio/textio-test-data.gzip-local.1k.txt.gz",
]
for raw_download in [False, True]:
print(f"-- raw_download={raw_download} --")
for test_file in test_files:
blob = Blob(test_file, bucket=bucket)
reader = BlobReader(blob, raw_download=raw_download)
a = reader.read(16777216)
b = reader.read(16777216)
print(f"[{test_file}] first_read:{len(a)}, second_read:{len(b)}")
Reproduction steps: supporting files
Run the code directly.
Reproduction steps: actual results
-- raw_download=False --
[textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:101000 <--- problem
[textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.default.1k.txt] first_read:101000, second_read:0
[textio/textio-test-data.default.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.gzip-local.1k.txt.gz] first_read:101000, second_read:0
-- raw_download=True --
[textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:76478 <--- problem
[textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.default.1k.txt] first_read:101000, second_read:0
[textio/textio-test-data.default.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.gzip-local.1k.txt.gz] first_read:76484, second_read:0
Reproduction steps: expected results
-- raw_download=False --
[textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.default.1k.txt] first_read:101000, second_read:0
[textio/textio-test-data.default.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.gzip-local.1k.txt.gz] first_read:101000, second_read:0
-- raw_download=True --
[textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.default.1k.txt] first_read:101000, second_read:0
[textio/textio-test-data.default.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.gzip-local.1k.txt.gz] first_read:76484, second_read:0
OS & version + platform
MacOS, Linux
Python environment
Python 3.12
Python dependencies
google-cloud-storage 3.14.1
Additional context
No response
Determine this is the right repository
Summary of the issue
When handling a doubly gzipped file (the first test file in the code below), the reader fails to reach the end of file.
API client name and version
google-cloud-storage 3.14.1
Reproduction steps: code
file: main.py
Reproduction steps: supporting files
Run the code directly.
Reproduction steps: actual results
Reproduction steps: expected results
OS & version + platform
MacOS, Linux
Python environment
Python 3.12
Python dependencies
google-cloud-storage 3.14.1
Additional context
No response