Skip to content

Add an Event Hubs Capture, Event Grid and Functions sample - #110

Open
DrisDary wants to merge 2 commits into
mainfrom
feat/eventhubs-eventgrid-sample
Open

Add an Event Hubs Capture, Event Grid and Functions sample#110
DrisDary wants to merge 2 commits into
mainfrom
feat/eventhubs-eventgrid-sample

Conversation

@DrisDary

Copy link
Copy Markdown
Contributor

Motivation

The existing Event Hubs sample covers the hot path: events arrive, a consumer scores them, alerts go out. Nothing covers the cold path, which is how most production Event Hubs estates actually handle bulk analytics — Capture archives the stream to Blob Storage in Avro, Event Hubs announces each archive with Microsoft.EventHub.CaptureFileCreated, and a Function App picks the archive up and aggregates it.

That shape is worth a sample of its own because the reasoning is not obvious: the archive is already batched, ordered and durable, so aggregation reads whole windows instead of guessing batch boundaries, and it can be replayed from Blob Storage at any time. It is also the first sample to exercise Event Grid system topics, an Event Grid subscription with an Event Hubs destination, and a Function App driven by an Event Hubs trigger.

Changes

New sample at samples/eventhubs-eventgrid/python, with the three deployment paths the other samples offer (scripts, Bicep, Terraform), all producing identical resource names.

  • src/functions/function_app.py — the CaptureProcessor: an Event Hubs trigger with Cardinality.MANY, which downloads each archive, decodes the Avro with fastavro, aggregates per device (count, min/mean/max temperature, excursions against a configurable limit) and writes summaries to a curated hub through an Event Hubs output binding.
  • src/producers/telemetry_producer.py — publishes device telemetry, with a flag to make one device breach the temperature limit so the excursion path is actually exercised rather than assumed.
  • scripts/deploy.sh (nine steps), validate.sh (23 checks), run-pipeline.sh (the five-stage end-to-end demo), cleanup.sh, plus read_curated.py and roundtrip_check.py.
  • run-samples.sh — three CI entries, one per deployment path.
  • Root README.md — one row for the new sample.

A few decisions worth calling out for review:

  • The Function App uses namespace-level connection strings, not entity-level. An entity-level string carries EntityPath, which would pin the output binding to the notifications hub instead of the curated one.
  • Storage connection strings spell out BlobEndpoint/QueueEndpoint/TableEndpoint rather than using EndpointSuffix, because the Functions host cannot parse a suffix that carries a port.
  • Connection strings are read with the CLI after deployment rather than emitted as template outputs, so secrets stay out of the deployment history.
  • skip_empty_archives is on, so Capture does not raise a notification for a window that saw no readings.

Tests

Verified end to end against an emulator built from the companion localstack-pro branch, running each deployment path on a fresh emulator (deleting a resource group is not sufficient — it orphans the Event Hubs namespace):

Path deploy validate pipeline
scripts 0 23/23 0
bicep 0 23/23 0
terraform 0 23/23 0

All three produced the same six device summaries, with the seeded excursion device correctly flagged (DEV-0003 … excursions 8), confirming the assertions are load-bearing rather than decorative.

validate.sh and run-pipeline.sh assert rather than narrate: every stage fails the script if the thing it claims did not happen, including that Capture actually wrote an archive, that the notification was delivered, and that summaries were produced. The CI entries run deploy + validate for every path and additionally run-pipeline for the scripts path, so a sample that stops working turns the build red instead of passing quietly.

Related

Requires the emulator changes in the companion localstack-pro PR — CaptureFileCreated system events, Event Grid delivery to an Event Hubs destination, and the Python Function App dependency fix. This sample cannot pass CI until those ship in the image CI pulls, so this PR should merge after that one is released.

SMF-790

@DrisDary DrisDary self-assigned this Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant