feat(gizmosql): add - #289
Conversation
|
|
||
| ### gizmosql | ||
|
|
||
| You can query the event stream's parquet lakehouse using any Arrow Flight SQL client at gizmosql:31337. |
There was a problem hiding this comment.
Let's mention Grafana as the UI for gizmosql queries here too.
|
|
||
| ### redpanda-connect | ||
|
|
||
| You can track the event stream ingestion using the pipeline's readiness endpoint. |
There was a problem hiding this comment.
What's "the pipeline's readiness endpoint"?
| r2-account-id: | ||
| # The event stream analytics' cloud storage account identifier. | ||
| file: ~~/artifacts/secrets/r2-account-id.secret | ||
| r2-aws-credentials-access-key: | ||
| # The event stream analytics' cloud storage secret access key. | ||
| file: ~~/artifacts/secrets/r2-aws-credentials-access-key.secret | ||
| r2-aws-credentials-access-key-id: | ||
| # The event stream analytics' cloud storage access key identifier. | ||
| file: ~~/artifacts/secrets/r2-aws-credentials-access-key-id.secret |
There was a problem hiding this comment.
Let's check if it's possible to define these secrets on the shared src/development/compose.yaml or in the redpanda-connect compose file (where the credentials are used to write – writing comes before reading which gizmosql allows then). But as these credentials are used by multiple services, they should fit the shared compose file. This should be supported by dargstack already or added to it if not.
| ports: # dargstack:dev-only | ||
| - 31337:31337 # dargstack:dev-only |
There was a problem hiding this comment.
I think the port definition can be removed as a whole.
| # Tombstone messages (Debezium deletes) carry no payload and must be dropped. | ||
| - bloblang: | | ||
| root = if this.payload.after.or(this.after) == null { deleted() } | ||
| - mapping: | | ||
| root = this.payload.after.or(this.after) | ||
| root.operation = this.payload.op.or(this.op) |
There was a problem hiding this comment.
Not sure if this handles all tombstone messages correctly, I'd need to research a bit more.
| output: | ||
| aws_s3: | ||
| bucket: test | ||
| endpoint: ${S3_ENDPOINT_URL} |
There was a problem hiding this comment.
Environment variable to be set as secret.
| redpanda-connect: | ||
| deploy: | ||
| update_config: | ||
| order: stop-first |
There was a problem hiding this comment.
Could this be start-first? I'd need to research how redpanda-connect works.
2 participants
This pull request introduces an end-to-end event stream analytics pipeline, enabling Redpanda event data to be ingested, stored as Parquet files in cloud storage, and queried using Arrow Flight SQL via GizmoSQL. The changes add new services, configuration, and documentation to support this workflow.
Event Stream Analytics Pipeline
redpanda-connectservice, which ingests events from Redpanda, transforms the data, and writes Parquet files to S3-compatible cloud storage (R2). Includes configuration for S3 credentials and data schema.gizmosqlservice, which allows querying the event stream's Parquet lakehouse using Arrow Flight SQL. Includes secure credential handling and a custom entrypoint script for environment setup and templating.