What happened
When a VMDisk (or any CDI DataVolume with source.http) points at a large object served from a provisioned Bucket's S3 endpoint, the CDI importer fails partway through the download:
Unable to write file from dataReader: stream error: stream ID 3; INTERNAL_ERROR; received from peer
failed to pull image: ... Unable to transfer source data to scratch space
The importer retries and CrashLoops; the import never completes.
Root cause
The SeaweedFS S3 gateway (port 8333, TLS) aborts large streaming GET responses after ~25-40 MB (non-deterministic). Reproduced directly against the ClusterIP from inside the cluster (no ingress involved), same client, comparing layers:
| Source |
Full streaming GET of an ~840 MB object |
seaweedfs-filer:8888 (plain HTTP, file API) |
full object, stable |
seaweedfs-s3:8333 (HTTPS, S3 API) |
aborts at ~25-40 MB, OpenSSL SSL_read: unexpected eof while reading |
seaweedfs-s3:8333 over plain HTTP |
HTTP 400 (endpoint is TLS-only) |
Notes:
Impact
Any CDI http-import of a large image (VM disk images, etc.) from a Bucket's S3 URL fails. Small objects (< ~25 MB) import fine, which masks the problem until real images are used.
Environment
- Cozystack v1.6.1-rc.1
- seaweedfs chart 4.31.0 (
weed version 4.31)
- CDI v1.64.0
Steps to reproduce
- Create a
Bucket; upload a large (~800 MB) object.
- From any in-cluster pod:
curl -k https://seaweedfs-s3.<ns>.svc:8333/<bucket>/<object> -o /dev/null -> aborts with SSL unexpected EOF after tens of MB.
- Same object via filer:
curl http://seaweedfs-filer.<ns>.svc:8888/buckets/<bucket>/<object> -o /dev/null -> completes.
Workaround
Serve large objects through the filer HTTP API (8888) instead of the S3 endpoint (8333). Range-based clients are unaffected.
What happened
When a
VMDisk(or any CDIDataVolumewithsource.http) points at a large object served from a provisioned Bucket's S3 endpoint, the CDI importer fails partway through the download:The importer retries and CrashLoops; the import never completes.
Root cause
The SeaweedFS S3 gateway (port 8333, TLS) aborts large streaming GET responses after ~25-40 MB (non-deterministic). Reproduced directly against the ClusterIP from inside the cluster (no ingress involved), same client, comparing layers:
seaweedfs-filer:8888(plain HTTP, file API)seaweedfs-s3:8333(HTTPS, S3 API)OpenSSL SSL_read: unexpected eof while readingseaweedfs-s3:8333over plain HTTPHTTP 400(endpoint is TLS-only)Notes:
RangeGET returns 206 correctly) — only a long single streaming response breaks, so the stored object is intact.BucketInfoadvertises anhttps://S3 endpoint, so consumers hit the failing TLS path by default (possibly related: fix(bucket): endpoint helper unconditionally forces https:// — original scheme lost for provisioned buckets #3317).Impact
Any CDI
http-import of a large image (VM disk images, etc.) from a Bucket's S3 URL fails. Small objects (< ~25 MB) import fine, which masks the problem until real images are used.Environment
weed version 4.31)Steps to reproduce
Bucket; upload a large (~800 MB) object.curl -k https://seaweedfs-s3.<ns>.svc:8333/<bucket>/<object> -o /dev/null-> aborts with SSL unexpected EOF after tens of MB.curl http://seaweedfs-filer.<ns>.svc:8888/buckets/<bucket>/<object> -o /dev/null-> completes.Workaround
Serve large objects through the filer HTTP API (8888) instead of the S3 endpoint (8333). Range-based clients are unaffected.