Skip to content

google-cloud-storage: Failed to handle doubly-gzipped files #18423

Description

@shunping

Determine this is the right repository

  • I determined this is the correct repository in which to report this bug.

Summary of the issue

When handling a doubly gzipped file (the first test file in the code below), the reader fails to reach the end of file.

Image

API client name and version

google-cloud-storage 3.14.1

Reproduction steps: code

file: main.py

from google.cloud.storage import Blob
from google.cloud.storage import Bucket
from google.cloud.storage import Client

from google.cloud.storage.fileio import BlobReader

client = Client("apache-beam-testing")
bucket = Bucket(client=client, name="apache-beam-samples")

test_files = [
    "textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz",
    "textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz",
    "textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz",
    "textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz",
    "textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz",
    "textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz",
    "textio/textio-test-data.default.1k.txt",
    "textio/textio-test-data.default.1k.txt.gz",
    "textio/textio-test-data.gzip-local.1k.txt.gz",
]

for raw_download in [False, True]:
  print(f"-- raw_download={raw_download} --")
  for test_file in test_files:
    blob = Blob(test_file, bucket=bucket)
    reader = BlobReader(blob, raw_download=raw_download)
    a = reader.read(16777216)
    b = reader.read(16777216)
    print(f"[{test_file}] first_read:{len(a)}, second_read:{len(b)}")

Reproduction steps: supporting files

Run the code directly.

Reproduction steps: actual results

-- raw_download=False --
[textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:101000   <--- problem
[textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.default.1k.txt] first_read:101000, second_read:0
[textio/textio-test-data.default.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.gzip-local.1k.txt.gz] first_read:101000, second_read:0
-- raw_download=True --
[textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:76478   <--- problem
[textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.default.1k.txt] first_read:101000, second_read:0
[textio/textio-test-data.default.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.gzip-local.1k.txt.gz] first_read:76484, second_read:0

Reproduction steps: expected results

-- raw_download=False --
[textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.default.1k.txt] first_read:101000, second_read:0
[textio/textio-test-data.default.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.gzip-local.1k.txt.gz] first_read:101000, second_read:0
-- raw_download=True --
[textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.default.1k.txt] first_read:101000, second_read:0
[textio/textio-test-data.default.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.gzip-local.1k.txt.gz] first_read:76484, second_read:0

OS & version + platform

MacOS, Linux

Python environment

Python 3.12

Python dependencies

google-cloud-storage 3.14.1

Additional context

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    triage meI really want to be triaged.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions