diff --git a/README.md b/README.md index ba70232..935e782 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,7 @@ This repository contains comprehensive sample projects demonstrating how to deve | [ACI and Blob Storage](./samples/aci-blob-storage/python/README.md) | Azure Container Instances with ACR, Key Vault, and Blob Storage | | [Azure Service Bus with Spring Boot](./samples/servicebus/java/README.md) | Azure Service Bus used by a Spring Boot application | | [Event Hubs Fraud Detection Pipeline](./samples/eventhubs/python/README.md) | Real-time payment stream processing with Event Hubs (AMQP, Kafka and HTTPS ingestion, Capture, Schema Registry), an Event Hubs-triggered Function App, Key Vault, Storage and a Web App dashboard | +| [Event Hubs Cold-Path Automation](./samples/eventhubs-eventgrid/python/README.md) | Event Hubs Capture raises `Microsoft.EventHub.CaptureFileCreated` to an Event Grid system topic, a subscription delivers it into a second event hub, and an Event Hubs-triggered Function App decodes each Avro archive and writes per-device summaries | ## Sample Structure diff --git a/run-samples.sh b/run-samples.sh index ad060d3..7a73690 100755 --- a/run-samples.sh +++ b/run-samples.sh @@ -31,6 +31,7 @@ PURGE_DOCKER="${PURGE_DOCKER:-0}" SAMPLES=( "samples/servicebus/java|bash scripts/deploy.sh" "samples/eventhubs/python|bash scripts/deploy.sh|bash scripts/validate.sh && bash scripts/run-pipeline.sh" + "samples/eventhubs-eventgrid/python|bash scripts/deploy.sh|bash scripts/validate.sh && bash scripts/run-pipeline.sh" "samples/function-app-front-door/python|bash scripts/deploy_all.sh --name-prefix testafd|" "samples/function-app-managed-identity/python|bash scripts/user-managed-identity.sh|bash scripts/validate.sh && bash scripts/test.sh" "samples/function-app-service-bus/dotnet|bash scripts/deploy.sh|bash scripts/validate.sh && bash scripts/call-http-trigger.sh" @@ -48,6 +49,7 @@ SAMPLES=( TERRAFORM_SAMPLES=( "samples/servicebus/java/terraform|bash deploy.sh" "samples/eventhubs/python/terraform|bash deploy.sh|bash ../scripts/validate.sh" + "samples/eventhubs-eventgrid/python/terraform|bash deploy.sh|bash ../scripts/validate.sh" "samples/function-app-managed-identity/python/terraform|bash deploy.sh" "samples/function-app-service-bus/dotnet/terraform|bash deploy.sh" "samples/function-app-storage-http/dotnet/terraform|bash deploy.sh" @@ -63,6 +65,7 @@ TERRAFORM_SAMPLES=( BICEP_SAMPLES=( "samples/servicebus/java/bicep|bash deploy.sh" "samples/eventhubs/python/bicep|bash deploy.sh|bash ../scripts/validate.sh" + "samples/eventhubs-eventgrid/python/bicep|bash deploy.sh|bash ../scripts/validate.sh" #"samples/web-app-sql-database/python/bicep|bash deploy.sh" "samples/function-app-managed-identity/python/bicep|bash deploy.sh" "samples/function-app-service-bus/dotnet/bicep|bash deploy.sh" diff --git a/samples/eventhubs-eventgrid/python/README.md b/samples/eventhubs-eventgrid/python/README.md new file mode 100644 index 0000000..ed27aba --- /dev/null +++ b/samples/eventhubs-eventgrid/python/README.md @@ -0,0 +1,165 @@ +# Cold-path automation: Event Hubs Capture → Event Grid → Functions + +A cold-chain monitoring pipeline where **Azure tells you when a batch is ready**, instead of your +code polling for it. + +Devices publish temperature readings into an event hub. Event Hubs Capture archives the stream to +Blob Storage as Avro. The moment an archive lands, Event Hubs raises +`Microsoft.EventHub.CaptureFileCreated` to Event Grid, an Event Grid subscription delivers that +notification **into a second event hub**, and a Function App triggered on that hub downloads the +archive, aggregates it per device, and writes summaries to a curated hub. + +``` + devices ──▶ telemetry hub ──── Capture (60s window) ────▶ Avro archive in Blob Storage + │ │ + │ Microsoft.EventHub.CaptureFileCreated + │ ▼ + │ Event Grid system topic + │ │ + │ subscription, EventHub destination + │ ▼ + │ capture-notifications hub + │ │ + │ Event Hubs trigger + │ ▼ + └───────────── archive read back ──────────── Function App + │ + Event Hubs output binding + ▼ + curated hub +``` + +## Why this shape + +The obvious way to process archives is a timer that lists a container and looks for new blobs. +That is a poll: it is late by up to its interval, it re-lists everything each run, and it needs +its own bookkeeping to avoid double-processing. + +Here the platform does the telling: + +| Concern | How the pipeline handles it | +|---|---| +| **When is a batch ready?** | Event Hubs says so — `CaptureFileCreated` carries the file URL, the partition, and the exact sequence range it contains | +| **How does the notification reach the processor?** | Event Grid delivers it into an **event hub**, so there is no public webhook to expose and no HTTP endpoint to keep available | +| **What if the processor is down?** | The notification sits in the hub for the retention window; the processor resumes from its checkpoint and catches up | +| **Is the aggregation correct?** | Readings are keyed by device, so a device's readings share a partition — and therefore share an archive, in order | +| **Can it be re-run?** | The archive is durable in Blob Storage; replaying is reading a file, not replaying a stream | + +That last column is what makes it a *cold* path: it runs on whole, closed windows rather than on +whatever happened to be in the last batch. + +## What it demonstrates + +| Capability | Where | +|---|---| +| **Event Hubs Capture** to Avro in Blob Storage | `deploy.sh` step 4, `bicep/modules/event-hubs.bicep` | +| **`Microsoft.EventHub.CaptureFileCreated`** system event | Raised by Event Hubs; consumed in `src/functions/function_app.py` | +| **Event Grid system topic** over a namespace | `deploy.sh` step 7, `bicep/modules/event-grid.bicep` | +| **Event Grid subscription with an EventHub destination** | Same — this is what makes the notification a stream | +| **Event Hubs trigger** with `Cardinality.MANY` | `function_app.py` — batched delivery | +| **Event Hubs output binding** | `function_app.py` — writes the curated summaries | +| **Partition keys for correctness** | `telemetry_producer.py` — a device's readings stay together | +| **Consumer groups** | The processor reads the notifications hub with its own group | +| **Least-privilege SAS** | Send-only for producers, namespace-wide Listen-only for the demo scripts | +| **Avro decoding from Blob Storage** | `function_app.py` `_read_archive` | + +## Prerequisites + +- The LocalStack Azure emulator running, and `lstk az start-interception` run. +- [lstk](https://github.com/localstack/lstk) (`brew install localstack/tap/lstk` or + `npm install -g @localstack/lstk`), which routes the Azure CLI to the emulator. +- Python 3.12+ with `src/producers/requirements.txt` installed. +- `jq` and `zip`. + +> **The emulator must be able to resolve `cdn.functions.azure.com` from the function container.** +> The Event Hubs trigger lives in the Functions extension bundle, which the host downloads on +> every cold container. If the emulator's DNS server cannot forward external queries, the host +> aborts and the trigger silently never fires. Starting the emulator with `DNS_ADDRESS=0` +> sidesteps it. + +## Usage + +```bash +cd samples/eventhubs-eventgrid/python + +bash scripts/deploy.sh # nine steps: storage, hubs, Capture, Event Grid, Function App +bash scripts/validate.sh # numbered checks over every capability above +bash scripts/run-pipeline.sh # the end-to-end demo + +bash scripts/cleanup.sh # when you are done +``` + +`deploy.sh` writes `scripts/.deployment-env` with the resource names and connection strings; +the other scripts source it, and so can you: + +```bash +source scripts/.deployment-env +cd src/producers && python telemetry_producer.py --count 200 --devices 8 +``` + +### Infrastructure as code + +The same topology through either IaC path; both then publish the application code with the CLI, +and both write the same `scripts/.deployment-env`, so the demo runs after either one: + +```bash +cd bicep && bash deploy.sh # or: +cd terraform && bash deploy.sh + +bash ../scripts/validate.sh +bash ../scripts/run-pipeline.sh +``` + +## The demo, step by step + +`scripts/run-pipeline.sh` asserts each stage rather than narrating it — it exits non-zero, and +prints the processor's container logs, the moment the chain breaks: + +1. **Publish telemetry.** 120 readings across 6 devices, keyed by device id. One device + (`DEV-0003` by default) also emits readings above the temperature limit. +2. **Wait for Capture.** Polls the blob container until a new archive appears. Nothing downstream + can happen until a window closes, so this sets the tempo. +3. **Show the notification.** Reads `capture-notifications` — the hub Event Grid delivered into. +4. **Read the curated summaries.** Polls the `curated` hub for the per-device aggregates the + processor produced, and fails if the excursion device is missing from them. +5. **Recap** the path an event took. + +## Application + +### Producer (`src/producers/telemetry_producer.py`) + +Publishes cold-chain readings with the device id as the **partition key**, one batch per device. +That is not a throughput trick: it is what guarantees a device's readings land in one archive, in +order, so the archive-level aggregation is exact. + +```bash +python telemetry_producer.py [--count 120] [--devices 6] [--excursion-device DEV-0003] +``` + +### Processor (`src/functions/function_app.py`) + +An Event Hubs-triggered function over `capture-notifications`. For each notification it: + +1. parses the Event Grid **batch array** (Event Grid always delivers an array — "an array with a + single event" by default), +2. downloads the archive named by `data.fileUrl`, +3. decodes the Avro records, +4. aggregates per device (count, min/mean/max, excursions), carrying the archive's sequence range + through so a downstream consumer can spot a gap between windows, +5. writes one summary per device via the **output binding**. + +Two details that are easy to get wrong, both commented in the code: + +- The trigger declares `cardinality=func.Cardinality.MANY`. The default delivers a bare + `EventHubEvent` and the handler fails with *"'EventHubEvent' object is not iterable"*. +- The handler annotates `List[func.EventHubEvent]` from `typing`. The builtin `list[...]` makes + the Functions worker reject the function as an *"invalid non-type annotation"*, and it never + loads at all. + +## Notes + +- The first `deploy.sh` run pulls the Functions build image and can take several minutes. +- `deploy.sh` is idempotent: every resource is created only when it does not already exist. +- Capture's minimum interval is 60 seconds, so the demo always takes at least that long. +- `skip_empty_archives` is on, so a window with no readings raises no event — otherwise the + processor would be woken for empty files. diff --git a/samples/eventhubs-eventgrid/python/bicep/deploy.sh b/samples/eventhubs-eventgrid/python/bicep/deploy.sh new file mode 100644 index 0000000..48a0cf5 --- /dev/null +++ b/samples/eventhubs-eventgrid/python/bicep/deploy.sh @@ -0,0 +1,165 @@ +#!/bin/bash + +# ============================================================================= +# Deploys the cold-path pipeline with Bicep, then publishes the application code. +# Bicep provisions the infrastructure; the Function App package is pushed with the Azure CLI, as +# in the other samples. +# ============================================================================= + +PREFIX='local' +LOCATION='westeurope' +RESOURCE_GROUP_NAME="${PREFIX}-ehgrid-rg" +DEPLOYMENT_NAME='eventhubs-eventgrid-coldpath' +FUNCTION_ZIP='capture_processor.zip' + +CURRENT_DIR="$(cd "$(dirname "$0")" && pwd)" +SRC_DIR="$(cd "$CURRENT_DIR/../src" && pwd)" +cd "$CURRENT_DIR" || exit 1 + +fail() { + echo "ERROR: $1" + exit 1 +} + +echo "=== Creating the resource group ===" +az group create --name "$RESOURCE_GROUP_NAME" --location "$LOCATION" \ + --only-show-errors 1>/dev/null || fail "could not create the resource group" +echo "Resource group [$RESOURCE_GROUP_NAME] ready." + +echo "" +echo "=== Validating the Bicep template ===" +az deployment group validate \ + --resource-group "$RESOURCE_GROUP_NAME" \ + --template-file main.bicep \ + --parameters main.bicepparam \ + --only-show-errors 1>/dev/null || fail "the Bicep template did not pass validation" + +echo "" +echo "=== Deploying the template ===" +az deployment group create \ + --name "$DEPLOYMENT_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" \ + --template-file main.bicep \ + --parameters main.bicepparam \ + --only-show-errors 1>/dev/null || fail "the Bicep deployment failed" +echo "Template deployed." + +echo "" +echo "=== Reading the deployment outputs ===" +outputs=$(az deployment group show --name "$DEPLOYMENT_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" --query properties.outputs \ + --output json --only-show-errors) + +EVENTHUB_NAMESPACE_NAME=$(echo "$outputs" | jq -r '.eventHubNamespaceName.value') +TELEMETRY_HUB_NAME=$(echo "$outputs" | jq -r '.telemetryHubName.value') +NOTIFICATION_HUB_NAME=$(echo "$outputs" | jq -r '.notificationHubName.value') +CURATED_HUB_NAME=$(echo "$outputs" | jq -r '.curatedHubName.value') +CAPTURE_CONSUMER_GROUP=$(echo "$outputs" | jq -r '.captureConsumerGroup.value') +STORAGE_ACCOUNT_NAME=$(echo "$outputs" | jq -r '.storageAccountName.value') +CAPTURE_CONTAINER_NAME=$(echo "$outputs" | jq -r '.captureContainerName.value') +SYSTEM_TOPIC_NAME=$(echo "$outputs" | jq -r '.systemTopicName.value') +SUBSCRIPTION_NAME=$(echo "$outputs" | jq -r '.eventSubscriptionName.value') +FUNCTION_APP_NAME=$(echo "$outputs" | jq -r '.functionAppName.value') +TEMPERATURE_LIMIT=$(echo "$outputs" | jq -r '.temperatureLimit.value') + +[[ -n "$FUNCTION_APP_NAME" && "$FUNCTION_APP_NAME" != "null" ]] || fail "could not read the deployment outputs" +echo "Namespace: $EVENTHUB_NAMESPACE_NAME" +echo "Function App: $FUNCTION_APP_NAME" +echo "System topic: $SYSTEM_TOPIC_NAME -> $SUBSCRIPTION_NAME" + +# ----------------------------------------------------------------------------- +echo "" +echo "=== Wiring the connection strings ===" +# ----------------------------------------------------------------------------- +# Done here rather than in the template: connection strings are built from keys that only exist +# once the resources do, and anything a template emits as an output is kept in the deployment +# history. Reading them with the CLI keeps the secrets out of both. +STORAGE_KEY=$(az storage account keys list --account-name "$STORAGE_ACCOUNT_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" --query "[0].value" -o tsv --only-show-errors) +STORAGE_BLOB=$(az storage account show -n "$STORAGE_ACCOUNT_NAME" -g "$RESOURCE_GROUP_NAME" \ + --query primaryEndpoints.blob -o tsv --only-show-errors) +STORAGE_QUEUE=$(az storage account show -n "$STORAGE_ACCOUNT_NAME" -g "$RESOURCE_GROUP_NAME" \ + --query primaryEndpoints.queue -o tsv --only-show-errors) +STORAGE_TABLE=$(az storage account show -n "$STORAGE_ACCOUNT_NAME" -g "$RESOURCE_GROUP_NAME" \ + --query primaryEndpoints.table -o tsv --only-show-errors) +# Explicit endpoints, not EndpointSuffix: the Functions host cannot parse a suffix with a port. +STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=${STORAGE_ACCOUNT_NAME};AccountKey=${STORAGE_KEY};BlobEndpoint=${STORAGE_BLOB};QueueEndpoint=${STORAGE_QUEUE};TableEndpoint=${STORAGE_TABLE}" + +SEND_CONNECTION=$(az eventhubs eventhub authorization-rule keys list --name telemetry-send \ + --eventhub-name "$TELEMETRY_HUB_NAME" --namespace-name "$EVENTHUB_NAMESPACE_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" --query primaryConnectionString -o tsv --only-show-errors) +LISTEN_CONNECTION=$(az eventhubs namespace authorization-rule keys list --name pipeline-listen \ + --namespace-name "$EVENTHUB_NAMESPACE_NAME" --resource-group "$RESOURCE_GROUP_NAME" \ + --query primaryConnectionString -o tsv --only-show-errors) +# Namespace-level for the bindings: an entity-level string carries EntityPath, which would pin the +# output binding to the notifications hub instead of the curated one. +NAMESPACE_CONNECTION=$(az eventhubs namespace authorization-rule keys list \ + --name RootManageSharedAccessKey --namespace-name "$EVENTHUB_NAMESPACE_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" --query primaryConnectionString -o tsv --only-show-errors) + +[[ -n "$SEND_CONNECTION" ]] || fail "could not read the send connection string" +[[ -n "$LISTEN_CONNECTION" ]] || fail "could not read the listen connection string" +[[ -n "$NAMESPACE_CONNECTION" ]] || fail "could not read the namespace connection string" + +az functionapp config appsettings set --name "$FUNCTION_APP_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" \ + --settings \ + "AzureWebJobsStorage=$STORAGE_CONNECTION_STRING" \ + "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING=$STORAGE_CONNECTION_STRING" \ + "STORAGE_CONNECTION_STRING=$STORAGE_CONNECTION_STRING" \ + "EVENTHUB_NOTIFICATION_CONNECTION=$NAMESPACE_CONNECTION" \ + "EVENTHUB_CURATED_CONNECTION=$NAMESPACE_CONNECTION" \ + "ENABLE_ORYX_BUILD=true" \ + "SCM_DO_BUILD_DURING_DEPLOYMENT=true" \ + --only-show-errors 1>/dev/null || fail "could not set the function app settings" +echo "Configured the processor." + +# ----------------------------------------------------------------------------- +echo "" +echo "=== Publishing the application code ===" +# ----------------------------------------------------------------------------- +cd "$SRC_DIR/functions" || fail "missing src/functions" +rm -f "$CURRENT_DIR/$FUNCTION_ZIP" +zip -r "$CURRENT_DIR/$FUNCTION_ZIP" function_app.py host.json requirements.txt 1>/dev/null +cd "$CURRENT_DIR" || exit 1 + +echo "Deploying the capture processor (the first run can take several minutes)..." +az functionapp deploy \ + --resource-group "$RESOURCE_GROUP_NAME" \ + --name "$FUNCTION_APP_NAME" \ + --src-path "$FUNCTION_ZIP" \ + --type zip 1>/dev/null || fail "could not deploy the function app" +rm -f "$FUNCTION_ZIP" + +# ----------------------------------------------------------------------------- +echo "" +echo "=== Verifying the deployment ===" +# ----------------------------------------------------------------------------- +az eventhubs eventhub show --name "$TELEMETRY_HUB_NAME" \ + --namespace-name "$EVENTHUB_NAMESPACE_NAME" --resource-group "$RESOURCE_GROUP_NAME" \ + --query "{name:name, partitions:partitionCount, capture:captureDescription.enabled}" \ + --output table --only-show-errors || fail "the telemetry hub is not readable" + +# Mirror scripts/deploy.sh so the demo runs after a Bicep deployment too. +cat >"$CURRENT_DIR/../scripts/.deployment-env" </dev/null; then + echo "Resource group [$RESOURCE_GROUP_NAME] deleted." +else + echo "WARNING: could not delete resource group [$RESOURCE_GROUP_NAME] (it may not exist)." +fi + +rm -f "$CURRENT_DIR/.deployment-env" "$CURRENT_DIR"/*.zip +echo "Removed local deployment artifacts." diff --git a/samples/eventhubs-eventgrid/python/scripts/deploy.sh b/samples/eventhubs-eventgrid/python/scripts/deploy.sh new file mode 100644 index 0000000..fa9447e --- /dev/null +++ b/samples/eventhubs-eventgrid/python/scripts/deploy.sh @@ -0,0 +1,365 @@ +#!/bin/bash + +# ============================================================================= +# Cold-path automation: Event Hubs Capture -> Event Grid -> Event Hubs -> Functions +# +# Deploys the whole chain: +# telemetry hub (Capture -> Avro in Blob) +# -> Microsoft.EventHub.CaptureFileCreated raised to an Event Grid system topic +# -> subscription with an EventHub destination -> capture-notifications hub +# -> Function App (Event Hubs trigger) decodes the archive and aggregates it +# -> curated hub (Event Hubs output binding) +# +# Everything runs against the LocalStack Azure emulator. Run 'lstk az start-interception' first. +# ============================================================================= + +PREFIX='local' +SUFFIX='telemetry' +LOCATION='westeurope' +RESOURCE_GROUP_NAME="${PREFIX}-ehgrid-rg" +EVENTHUB_NAMESPACE_NAME="${PREFIX}-ehns-${SUFFIX}" + +TELEMETRY_HUB_NAME='telemetry' +NOTIFICATION_HUB_NAME='capture-notifications' +CURATED_HUB_NAME='curated' +TELEMETRY_PARTITION_COUNT=4 +NOTIFICATION_PARTITION_COUNT=2 +CURATED_PARTITION_COUNT=2 +RETENTION_HOURS=24 + +# Capture flushes on whichever comes first. 60s is the minimum Azure allows, and it sets the +# tempo of the whole demo: nothing downstream happens until a window closes. +CAPTURE_INTERVAL_SECONDS=60 +CAPTURE_SIZE_LIMIT_BYTES=10485760 +CAPTURE_CONTAINER_NAME='telemetry-archive' + +STORAGE_ACCOUNT_NAME="${PREFIX}ehgrid${SUFFIX:0:4}" +SYSTEM_TOPIC_NAME="${PREFIX}-ehns-systopic" +SUBSCRIPTION_NAME='capture-to-eventhub' +# Azure's topic type for an Event Hubs namespace as an event source. +EVENTHUB_TOPIC_TYPE='Microsoft.Eventhub.Namespaces' + +FUNCTION_APP_NAME="${PREFIX}-ehgrid-processor" +APP_SERVICE_PLAN_NAME="${PREFIX}-ehgrid-plan" +CAPTURE_CONSUMER_GROUP='capture-processor' +TEMPERATURE_LIMIT=80 + +SEND_RULE_NAME='telemetry-send' +LISTEN_RULE_NAME='pipeline-listen' + +FUNCTION_ZIP='capture_processor.zip' + +CURRENT_DIR="$(cd "$(dirname "$0")" && pwd)" +SRC_DIR="$(cd "$CURRENT_DIR/../src" && pwd)" +cd "$CURRENT_DIR" || exit 1 + +fail() { + echo "ERROR: $1" + exit 1 +} + +step() { + echo "" + echo "=== $1 ===" +} + +# ----------------------------------------------------------------------------- +step "[1/9] Resource group" +# ----------------------------------------------------------------------------- +az group create --name "$RESOURCE_GROUP_NAME" --location "$LOCATION" \ + --only-show-errors 1>/dev/null || fail "could not create the resource group" +echo "Resource group [$RESOURCE_GROUP_NAME] ready." + +# ----------------------------------------------------------------------------- +step "[2/9] Storage account for Capture archives" +# ----------------------------------------------------------------------------- +if ! az storage account show --name "$STORAGE_ACCOUNT_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" --only-show-errors &>/dev/null; then + az storage account create \ + --name "$STORAGE_ACCOUNT_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" \ + --location "$LOCATION" \ + --sku Standard_LRS \ + --kind StorageV2 \ + --only-show-errors 1>/dev/null || fail "could not create storage account" + echo "Created storage account [$STORAGE_ACCOUNT_NAME]." +else + echo "Storage account [$STORAGE_ACCOUNT_NAME] already exists." +fi + +STORAGE_KEY=$(az storage account keys list --account-name "$STORAGE_ACCOUNT_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" --query "[0].value" -o tsv --only-show-errors) +STORAGE_BLOB=$(az storage account show -n "$STORAGE_ACCOUNT_NAME" -g "$RESOURCE_GROUP_NAME" \ + --query primaryEndpoints.blob -o tsv --only-show-errors) +STORAGE_QUEUE=$(az storage account show -n "$STORAGE_ACCOUNT_NAME" -g "$RESOURCE_GROUP_NAME" \ + --query primaryEndpoints.queue -o tsv --only-show-errors) +STORAGE_TABLE=$(az storage account show -n "$STORAGE_ACCOUNT_NAME" -g "$RESOURCE_GROUP_NAME" \ + --query primaryEndpoints.table -o tsv --only-show-errors) +[[ -n "$STORAGE_KEY" ]] || fail "could not read the storage account key" + +# Explicit endpoints rather than EndpointSuffix: the Functions host cannot parse a suffix that +# carries a port, which it does against the emulator. +STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=${STORAGE_ACCOUNT_NAME};AccountKey=${STORAGE_KEY};BlobEndpoint=${STORAGE_BLOB};QueueEndpoint=${STORAGE_QUEUE};TableEndpoint=${STORAGE_TABLE}" + +az storage container create --name "$CAPTURE_CONTAINER_NAME" \ + --connection-string "$STORAGE_CONNECTION_STRING" --only-show-errors 1>/dev/null || + fail "could not create the capture container" +echo "Blob container [$CAPTURE_CONTAINER_NAME] ready for Capture archives." + +# ----------------------------------------------------------------------------- +step "[3/9] Event Hubs namespace (Standard tier - Capture needs it)" +# ----------------------------------------------------------------------------- +if ! az eventhubs namespace show --name "$EVENTHUB_NAMESPACE_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" --only-show-errors &>/dev/null; then + az eventhubs namespace create \ + --name "$EVENTHUB_NAMESPACE_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" \ + --location "$LOCATION" \ + --sku Standard \ + --only-show-errors 1>/dev/null || fail "could not create the Event Hubs namespace" + echo "Created Event Hubs namespace [$EVENTHUB_NAMESPACE_NAME]." +else + echo "Event Hubs namespace [$EVENTHUB_NAMESPACE_NAME] already exists." +fi + +NAMESPACE_ID=$(az eventhubs namespace show --name "$EVENTHUB_NAMESPACE_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" --query id -o tsv --only-show-errors) +STORAGE_ACCOUNT_ID=$(az storage account show --name "$STORAGE_ACCOUNT_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" --query id -o tsv --only-show-errors) + +# ----------------------------------------------------------------------------- +step "[4/9] Three hubs: telemetry (with Capture), notifications, curated" +# ----------------------------------------------------------------------------- +if ! az eventhubs eventhub show --name "$TELEMETRY_HUB_NAME" \ + --namespace-name "$EVENTHUB_NAMESPACE_NAME" --resource-group "$RESOURCE_GROUP_NAME" \ + --only-show-errors &>/dev/null; then + az eventhubs eventhub create \ + --name "$TELEMETRY_HUB_NAME" \ + --namespace-name "$EVENTHUB_NAMESPACE_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" \ + --partition-count "$TELEMETRY_PARTITION_COUNT" \ + --retention-time-in-hours "$RETENTION_HOURS" \ + --enable-capture true \ + --capture-interval "$CAPTURE_INTERVAL_SECONDS" \ + --capture-size-limit "$CAPTURE_SIZE_LIMIT_BYTES" \ + --skip-empty-archives true \ + --destination-name EventHubArchive.AzureBlockBlob \ + --storage-account "$STORAGE_ACCOUNT_ID" \ + --blob-container "$CAPTURE_CONTAINER_NAME" \ + --archive-name-format '{Namespace}/{EventHub}/{PartitionId}/{Year}/{Month}/{Day}/{Hour}/{Minute}/{Second}' \ + --only-show-errors 1>/dev/null || fail "could not create hub [$TELEMETRY_HUB_NAME]" + echo "Created [$TELEMETRY_HUB_NAME] with $TELEMETRY_PARTITION_COUNT partitions and Capture enabled." +else + echo "Event hub [$TELEMETRY_HUB_NAME] already exists." +fi + +for hub_spec in "$NOTIFICATION_HUB_NAME:$NOTIFICATION_PARTITION_COUNT" "$CURATED_HUB_NAME:$CURATED_PARTITION_COUNT"; do + hub_name="${hub_spec%%:*}" + partitions="${hub_spec##*:}" + if ! az eventhubs eventhub show --name "$hub_name" \ + --namespace-name "$EVENTHUB_NAMESPACE_NAME" --resource-group "$RESOURCE_GROUP_NAME" \ + --only-show-errors &>/dev/null; then + az eventhubs eventhub create \ + --name "$hub_name" \ + --namespace-name "$EVENTHUB_NAMESPACE_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" \ + --partition-count "$partitions" \ + --retention-time-in-hours "$RETENTION_HOURS" \ + --only-show-errors 1>/dev/null || fail "could not create hub [$hub_name]" + echo "Created [$hub_name] with $partitions partitions." + else + echo "Event hub [$hub_name] already exists." + fi +done + +# ----------------------------------------------------------------------------- +step "[5/9] Consumer group for the processor" +# ----------------------------------------------------------------------------- +if ! az eventhubs eventhub consumer-group show \ + --consumer-group-name "$CAPTURE_CONSUMER_GROUP" --eventhub-name "$NOTIFICATION_HUB_NAME" \ + --namespace-name "$EVENTHUB_NAMESPACE_NAME" --resource-group "$RESOURCE_GROUP_NAME" \ + --only-show-errors &>/dev/null; then + az eventhubs eventhub consumer-group create \ + --consumer-group-name "$CAPTURE_CONSUMER_GROUP" \ + --eventhub-name "$NOTIFICATION_HUB_NAME" \ + --namespace-name "$EVENTHUB_NAMESPACE_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" \ + --only-show-errors 1>/dev/null || fail "could not create consumer group" + echo "Created consumer group [$CAPTURE_CONSUMER_GROUP] on [$NOTIFICATION_HUB_NAME]." +else + echo "Consumer group [$CAPTURE_CONSUMER_GROUP] already exists." +fi + +# ----------------------------------------------------------------------------- +step "[6/9] Least-privilege authorization rules" +# ----------------------------------------------------------------------------- +# Producers may only Send to the telemetry hub. +if ! az eventhubs eventhub authorization-rule show --name "$SEND_RULE_NAME" \ + --eventhub-name "$TELEMETRY_HUB_NAME" --namespace-name "$EVENTHUB_NAMESPACE_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" --only-show-errors &>/dev/null; then + az eventhubs eventhub authorization-rule create \ + --name "$SEND_RULE_NAME" --eventhub-name "$TELEMETRY_HUB_NAME" \ + --namespace-name "$EVENTHUB_NAMESPACE_NAME" --resource-group "$RESOURCE_GROUP_NAME" \ + --rights Send --only-show-errors 1>/dev/null || fail "could not create rule [$SEND_RULE_NAME]" + echo "Created send-only rule [$SEND_RULE_NAME] on [$TELEMETRY_HUB_NAME]." +fi + +# The demo scripts read every hub, so this one is namespace-wide - but Listen only. +if ! az eventhubs namespace authorization-rule show --name "$LISTEN_RULE_NAME" \ + --namespace-name "$EVENTHUB_NAMESPACE_NAME" --resource-group "$RESOURCE_GROUP_NAME" \ + --only-show-errors &>/dev/null; then + az eventhubs namespace authorization-rule create \ + --name "$LISTEN_RULE_NAME" --namespace-name "$EVENTHUB_NAMESPACE_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" --rights Listen \ + --only-show-errors 1>/dev/null || fail "could not create rule [$LISTEN_RULE_NAME]" + echo "Created namespace-wide listen-only rule [$LISTEN_RULE_NAME]." +fi + +SEND_CONNECTION=$(az eventhubs eventhub authorization-rule keys list --name "$SEND_RULE_NAME" \ + --eventhub-name "$TELEMETRY_HUB_NAME" --namespace-name "$EVENTHUB_NAMESPACE_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" --query primaryConnectionString -o tsv --only-show-errors) +LISTEN_CONNECTION=$(az eventhubs namespace authorization-rule keys list --name "$LISTEN_RULE_NAME" \ + --namespace-name "$EVENTHUB_NAMESPACE_NAME" --resource-group "$RESOURCE_GROUP_NAME" \ + --query primaryConnectionString -o tsv --only-show-errors) +NAMESPACE_CONNECTION=$(az eventhubs namespace authorization-rule keys list \ + --name RootManageSharedAccessKey --namespace-name "$EVENTHUB_NAMESPACE_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" --query primaryConnectionString -o tsv --only-show-errors) + +[[ -n "$SEND_CONNECTION" ]] || fail "could not read the send connection string" +[[ -n "$LISTEN_CONNECTION" ]] || fail "could not read the listen connection string" +[[ -n "$NAMESPACE_CONNECTION" ]] || fail "could not read the namespace connection string" + +# ----------------------------------------------------------------------------- +step "[7/9] Event Grid system topic on the namespace" +# ----------------------------------------------------------------------------- +# A system topic is how a subscriber reaches the events an Azure resource raises about itself. +# Its source is the namespace, and Event Hubs raises Microsoft.EventHub.CaptureFileCreated to it. +if ! az eventgrid system-topic show --name "$SYSTEM_TOPIC_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" --only-show-errors &>/dev/null; then + az eventgrid system-topic create \ + --name "$SYSTEM_TOPIC_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" \ + --location "$LOCATION" \ + --topic-type "$EVENTHUB_TOPIC_TYPE" \ + --source "$NAMESPACE_ID" \ + --only-show-errors 1>/dev/null || fail "could not create the system topic" + echo "Created system topic [$SYSTEM_TOPIC_NAME] over [$EVENTHUB_NAMESPACE_NAME]." +else + echo "System topic [$SYSTEM_TOPIC_NAME] already exists." +fi + +NOTIFICATION_HUB_ID=$(az eventhubs eventhub show --name "$NOTIFICATION_HUB_NAME" \ + --namespace-name "$EVENTHUB_NAMESPACE_NAME" --resource-group "$RESOURCE_GROUP_NAME" \ + --query id -o tsv --only-show-errors) +[[ -n "$NOTIFICATION_HUB_ID" ]] || fail "could not read the notification hub id" + +# The subscription's destination is an event hub, so the notification becomes a stream the +# Functions host can trigger on - no webhook to expose, no polling. +if ! az eventgrid system-topic event-subscription show --name "$SUBSCRIPTION_NAME" \ + --system-topic-name "$SYSTEM_TOPIC_NAME" --resource-group "$RESOURCE_GROUP_NAME" \ + --only-show-errors &>/dev/null; then + az eventgrid system-topic event-subscription create \ + --name "$SUBSCRIPTION_NAME" \ + --system-topic-name "$SYSTEM_TOPIC_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" \ + --endpoint-type eventhub \ + --endpoint "$NOTIFICATION_HUB_ID" \ + --only-show-errors 1>/dev/null || fail "could not create the event subscription" + echo "Created subscription [$SUBSCRIPTION_NAME] -> event hub [$NOTIFICATION_HUB_NAME]." +else + echo "Event subscription [$SUBSCRIPTION_NAME] already exists." +fi + +# ----------------------------------------------------------------------------- +step "[8/9] Function App (Event Hubs trigger -> archive processor)" +# ----------------------------------------------------------------------------- +if ! az functionapp show --name "$FUNCTION_APP_NAME" --resource-group "$RESOURCE_GROUP_NAME" \ + --only-show-errors &>/dev/null; then + az functionapp create \ + --name "$FUNCTION_APP_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" \ + --storage-account "$STORAGE_ACCOUNT_NAME" \ + --consumption-plan-location "$LOCATION" \ + --runtime python \ + --runtime-version 3.12 \ + --functions-version 4 \ + --os-type Linux \ + --only-show-errors 1>/dev/null || fail "could not create the function app" + echo "Created function app [$FUNCTION_APP_NAME]." +else + echo "Function app [$FUNCTION_APP_NAME] already exists." +fi + +# The processor imports fastavro and azure-storage-blob to decode an archive, so the deployment +# has to run a build that installs requirements.txt. Without these two settings the package is +# copied as-is, the imports fail at invocation time, and the notification looks like it was never +# processed. +az functionapp config appsettings set \ + --name "$FUNCTION_APP_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" \ + --settings \ + "AzureWebJobsStorage=$STORAGE_CONNECTION_STRING" \ + "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING=$STORAGE_CONNECTION_STRING" \ + "STORAGE_CONNECTION_STRING=$STORAGE_CONNECTION_STRING" \ + "ENABLE_ORYX_BUILD=true" \ + "SCM_DO_BUILD_DURING_DEPLOYMENT=true" \ + "EVENTHUB_NOTIFICATION_CONNECTION=$NAMESPACE_CONNECTION" \ + "EVENTHUB_CURATED_CONNECTION=$NAMESPACE_CONNECTION" \ + "NOTIFICATION_HUB_NAME=$NOTIFICATION_HUB_NAME" \ + "CURATED_HUB_NAME=$CURATED_HUB_NAME" \ + "CAPTURE_CONSUMER_GROUP=$CAPTURE_CONSUMER_GROUP" \ + "TEMPERATURE_LIMIT=$TEMPERATURE_LIMIT" \ + --only-show-errors 1>/dev/null || fail "could not set function app settings" +echo "Configured function app settings." + +cd "$SRC_DIR/functions" || fail "missing src/functions" +rm -f "$CURRENT_DIR/$FUNCTION_ZIP" +zip -r "$CURRENT_DIR/$FUNCTION_ZIP" function_app.py host.json requirements.txt 1>/dev/null +cd "$CURRENT_DIR" || exit 1 + +echo "Deploying the processor (the first run pulls the build image and can take a few minutes)..." +az functionapp deploy \ + --resource-group "$RESOURCE_GROUP_NAME" \ + --name "$FUNCTION_APP_NAME" \ + --src-path "$FUNCTION_ZIP" \ + --type zip 1>/dev/null || fail "could not deploy the function app" +rm -f "$FUNCTION_ZIP" +echo "Deployed the capture processor." + +# ----------------------------------------------------------------------------- +step "[9/9] Connection details" +# ----------------------------------------------------------------------------- +cat >"$CURRENT_DIR/.deployment-env" < $CAPTURE_CONTAINER_NAME)" +echo " notifications hub: $NOTIFICATION_HUB_NAME (Event Grid delivers CaptureFileCreated here)" +echo " curated hub: $CURATED_HUB_NAME (device summaries)" +echo "System topic: $SYSTEM_TOPIC_NAME -> subscription '$SUBSCRIPTION_NAME'" +echo "Function App: $FUNCTION_APP_NAME (consumer group '$CAPTURE_CONSUMER_GROUP')" +echo "" +echo "Next steps:" +echo " bash scripts/validate.sh verify every deployed capability" +echo " bash scripts/run-pipeline.sh run the end-to-end demo" diff --git a/samples/eventhubs-eventgrid/python/scripts/read_curated.py b/samples/eventhubs-eventgrid/python/scripts/read_curated.py new file mode 100644 index 0000000..aab78ad --- /dev/null +++ b/samples/eventhubs-eventgrid/python/scripts/read_curated.py @@ -0,0 +1,71 @@ +"""Read the curated hub and print the device summaries the processor produced. + +Used by run-pipeline.sh as the end-to-end proof: a summary here means Capture flushed, Event Grid +delivered the notification, and the function decoded the archive. +""" + +import json +import os +import sys +import threading + +from azure.eventhub import EventHubConsumerClient + +RECEIVE_TIMEOUT_SECONDS = int(os.environ.get("CURATED_RECEIVE_TIMEOUT", "45")) + + +def main() -> int: + connection_string = os.environ.get("EVENTHUB_LISTEN_CONNECTION_STRING", "") + hub_name = os.environ.get("CURATED_HUB_NAME", "curated") + if not connection_string: + print("EVENTHUB_LISTEN_CONNECTION_STRING is not set") + return 1 + + summaries: list[dict] = [] + found = threading.Event() + + def on_event(context, event): + if event is None: + return + try: + summaries.append(json.loads(event.body_as_str())) + except ValueError: + return + found.set() + + consumer = EventHubConsumerClient.from_connection_string( + connection_string, consumer_group="$Default", eventhub_name=hub_name + ) + + def receive() -> None: + with consumer: + consumer.receive(on_event=on_event, starting_position="-1", max_wait_time=5) + + receiver = threading.Thread(target=receive, daemon=True) + receiver.start() + found.wait(timeout=RECEIVE_TIMEOUT_SECONDS) + # Give slightly longer for the remaining partitions once the first summary arrives. + if found.is_set(): + threading.Event().wait(5) + try: + consumer.close() + except Exception: + pass + receiver.join(timeout=10) + + if not summaries: + print("SUMMARIES_NONE") + return 1 + + print(f"SUMMARIES_OK {len(summaries)}") + for summary in sorted(summaries, key=lambda item: item.get("device_id", "")): + print( + f" {summary.get('device_id')}: {summary.get('reading_count')} readings, " + f"min {summary.get('min_temperature')} / mean {summary.get('mean_temperature')} / " + f"max {summary.get('max_temperature')}, excursions {summary.get('excursion_count')}" + ) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/samples/eventhubs-eventgrid/python/scripts/roundtrip_check.py b/samples/eventhubs-eventgrid/python/scripts/roundtrip_check.py new file mode 100644 index 0000000..c22d2e3 --- /dev/null +++ b/samples/eventhubs-eventgrid/python/scripts/roundtrip_check.py @@ -0,0 +1,95 @@ +"""Data-plane checks used by validate.sh. + +Two modes: + +* default - publish a uniquely marked event with the send-only rule and read it back + with the listen-only rule, proving both credentials and the round trip. +* --partitions - read runtime metadata for every partition, proving the management + endpoint of the data plane answers. + +Prints a single line starting with ROUNDTRIP_OK / PARTITIONS_OK on success so the shell +script can branch on it. +""" + +import os +import sys +import threading +from uuid import uuid4 + +from azure.eventhub import EventData, EventHubConsumerClient, EventHubProducerClient + +EVENT_HUB_NAME = os.environ.get("EVENT_HUB_NAME", "payments") +RECEIVE_TIMEOUT_SECONDS = 45 + + +def partitions_check() -> int: + conn = os.environ.get("EVENTHUB_LISTEN_CONNECTION_STRING") or os.environ.get( + "EVENTHUB_SEND_CONNECTION_STRING", "" + ) + if not conn: + print("no connection string available") + return 1 + client = EventHubProducerClient.from_connection_string(conn, eventhub_name=EVENT_HUB_NAME) + with client: + partition_ids = client.get_partition_ids() + total = 0 + for partition_id in partition_ids: + properties = client.get_partition_properties(partition_id) + total += properties["last_enqueued_sequence_number"] + 1 + if "--total" in sys.argv: + print(total) + else: + print(f"PARTITIONS_OK {len(partition_ids)} partitions, {total} events so far") + return 0 + + +def roundtrip_check() -> int: + send_conn = os.environ.get("EVENTHUB_SEND_CONNECTION_STRING", "") + listen_conn = os.environ.get("EVENTHUB_LISTEN_CONNECTION_STRING", "") + if not send_conn or not listen_conn: + print("send and listen connection strings are both required") + return 1 + + marker = f"validate-{uuid4().hex}" + partition_key = "VALIDATION" + + producer = EventHubProducerClient.from_connection_string(send_conn, eventhub_name=EVENT_HUB_NAME) + with producer: + batch = producer.create_batch(partition_key=partition_key) + batch.add(EventData(marker)) + producer.send_batch(batch) + + found = threading.Event() + consumer = EventHubConsumerClient.from_connection_string( + listen_conn, consumer_group="$Default", eventhub_name=EVENT_HUB_NAME + ) + + def on_event(context, event): + if event is not None and event.body_as_str() == marker: + found.set() + + def receive() -> None: + # starting_position "-1" means "from the beginning of the partition". + with consumer: + consumer.receive(on_event=on_event, starting_position="-1", max_wait_time=5) + + receiver = threading.Thread(target=receive, daemon=True) + receiver.start() + found.wait(timeout=RECEIVE_TIMEOUT_SECONDS) + try: + consumer.close() + except Exception: + pass + receiver.join(timeout=10) + + if found.is_set(): + print(f"ROUNDTRIP_OK sent and received {marker[:18]}...") + return 0 + print(f"event {marker} was not received within {RECEIVE_TIMEOUT_SECONDS}s") + return 1 + + +if __name__ == "__main__": + if "--partitions" in sys.argv or "--total" in sys.argv: + raise SystemExit(partitions_check()) + raise SystemExit(roundtrip_check()) diff --git a/samples/eventhubs-eventgrid/python/scripts/run-pipeline.sh b/samples/eventhubs-eventgrid/python/scripts/run-pipeline.sh new file mode 100644 index 0000000..ca569fa --- /dev/null +++ b/samples/eventhubs-eventgrid/python/scripts/run-pipeline.sh @@ -0,0 +1,189 @@ +#!/bin/bash + +# ============================================================================= +# Cold-path automation - end-to-end demo +# +# 1. publish telemetry into the Capture-enabled hub +# 2. wait for a Capture window to flush an Avro archive to Blob Storage +# 3. Event Hubs raises CaptureFileCreated; Event Grid delivers it to a hub +# 4. the Function App decodes the archive and writes per-device summaries +# 5. read the curated hub and show the summaries +# +# Every step is an assertion: the script exits non-zero the moment the chain +# breaks, and prints the processor's container logs so the reason is visible. +# A clean exit means the whole pipeline ran. +# +# Run scripts/deploy.sh first. Everything is local to the emulator. +# ============================================================================= + +CURRENT_DIR="$(cd "$(dirname "$0")" && pwd)" +SRC_DIR="$(cd "$CURRENT_DIR/../src" && pwd)" +PYTHON_BIN="${PYTHON_BIN:-python3}" + +READING_COUNT="${READING_COUNT:-120}" +DEVICE_COUNT="${DEVICE_COUNT:-6}" +EXCURSION_DEVICE="${EXCURSION_DEVICE:-DEV-0003}" +# Capture flushes on a 60s window; allow for the scan cycle behind it. +CAPTURE_WAIT_SECONDS="${CAPTURE_WAIT_SECONDS:-180}" +# The processor is a cold container on first run: it downloads its extension bundle before the +# first invocation, so the summary can take minutes to appear the first time. +SUMMARY_WAIT_SECONDS="${SUMMARY_WAIT_SECONDS:-420}" + +if [[ ! -f "$CURRENT_DIR/.deployment-env" ]]; then + echo "ERROR: scripts/.deployment-env not found. Run 'bash scripts/deploy.sh' first." + exit 1 +fi +# shellcheck disable=SC1091 +source "$CURRENT_DIR/.deployment-env" + +step() { + echo "" + echo "============================================================" + echo "$1" + echo "============================================================" +} + +function_logs() { + command -v docker >/dev/null 2>&1 || return 1 + local container + container=$(docker ps --format "{{.Names}}" | grep -E "^ls-.*${FUNCTION_APP_NAME}" | head -1) + [[ -n "$container" ]] && docker logs "$container" 2>&1 +} + +fail() { + echo "" + echo "============================================================" + echo "PIPELINE FAILED: $1" + echo "============================================================" + local logs + logs=$(function_logs) + if [[ -n "$logs" ]]; then + echo "" + echo "Processor container logs (last 40 lines):" + echo "$logs" | tail -40 | sed 's/^/ /' + elif command -v docker >/dev/null 2>&1; then + echo "" + echo "No processor container is running - the Functions host never started." + fi + exit 1 +} + +archive_count() { + az storage blob list \ + --container-name "$CAPTURE_CONTAINER_NAME" \ + --connection-string "$STORAGE_CONNECTION_STRING" \ + --query "length(@)" --output tsv --only-show-errors 2>/dev/null || echo 0 +} + +# ----------------------------------------------------------------------------- +step "Step 1/5 - Publish telemetry into the Capture-enabled hub" +# ----------------------------------------------------------------------------- +ARCHIVES_BEFORE=$(archive_count) +echo "Archives already in '$CAPTURE_CONTAINER_NAME': ${ARCHIVES_BEFORE:-0}" + +cd "$SRC_DIR/producers" || fail "missing src/producers" +"$PYTHON_BIN" telemetry_producer.py \ + --count "$READING_COUNT" \ + --devices "$DEVICE_COUNT" \ + --excursion-device "$EXCURSION_DEVICE" || + fail "the telemetry producer could not publish to '$TELEMETRY_HUB_NAME'" +cd "$CURRENT_DIR" || fail "could not return to the scripts directory" + +echo "" +echo "Readings are keyed by device, so each device's readings land on one partition - and" +echo "therefore inside one archive, in order. That is what makes per-device aggregation exact." + +# ----------------------------------------------------------------------------- +step "Step 2/5 - Wait for Capture to flush an Avro archive" +# ----------------------------------------------------------------------------- +echo "Capture writes on a 60s window (up to ${CAPTURE_WAIT_SECONDS}s allowed)..." +DEADLINE=$((SECONDS + CAPTURE_WAIT_SECONDS)) +ARCHIVES_AFTER=$ARCHIVES_BEFORE +while [ $SECONDS -lt $DEADLINE ]; do + ARCHIVES_AFTER=$(archive_count) + if [[ "${ARCHIVES_AFTER:-0}" -gt "${ARCHIVES_BEFORE:-0}" ]]; then + break + fi + sleep 10 +done +if [[ "${ARCHIVES_AFTER:-0}" -le "${ARCHIVES_BEFORE:-0}" ]]; then + fail "Capture wrote no new archive to '$CAPTURE_CONTAINER_NAME' within ${CAPTURE_WAIT_SECONDS}s" +fi + +echo "Capture wrote $((ARCHIVES_AFTER - ARCHIVES_BEFORE)) new archive(s):" +az storage blob list \ + --container-name "$CAPTURE_CONTAINER_NAME" \ + --connection-string "$STORAGE_CONNECTION_STRING" \ + --query "[].{name:name, bytes:properties.contentLength}" \ + --output table --only-show-errors 2>/dev/null | head -8 | sed 's/^/ /' + +# ----------------------------------------------------------------------------- +step "Step 3/5 - Event Hubs raises CaptureFileCreated, Event Grid delivers it" +# ----------------------------------------------------------------------------- +echo "Reading '$NOTIFICATION_HUB_NAME', where the subscription's EventHub destination delivers..." +NOTIFICATIONS=$(EVENTHUB_LISTEN_CONNECTION_STRING="$EVENTHUB_LISTEN_CONNECTION_STRING" \ + EVENT_HUB_NAME="$NOTIFICATION_HUB_NAME" \ + "$PYTHON_BIN" "$CURRENT_DIR/roundtrip_check.py" --partitions 2>/dev/null | tail -1) +echo " $NOTIFICATIONS" +echo "" +echo "Nothing polled Blob Storage and no webhook was exposed: the notification arrived as a" +echo "stream event, which is why a plain Event Hubs trigger can consume it." + +# ----------------------------------------------------------------------------- +step "Step 4/5 - The processor decodes the archive and curates it" +# ----------------------------------------------------------------------------- +echo "Waiting for the first device summary (up to ${SUMMARY_WAIT_SECONDS}s on a cold container)..." +DEADLINE=$((SECONDS + SUMMARY_WAIT_SECONDS)) +SUMMARY_OUTPUT="" +while [ $SECONDS -lt $DEADLINE ]; do + SUMMARY_OUTPUT=$(EVENTHUB_LISTEN_CONNECTION_STRING="$EVENTHUB_LISTEN_CONNECTION_STRING" \ + CURATED_HUB_NAME="$CURATED_HUB_NAME" \ + "$PYTHON_BIN" "$CURRENT_DIR/read_curated.py" 2>/dev/null) + if [[ "$SUMMARY_OUTPUT" == SUMMARIES_OK* ]]; then + break + fi + sleep 15 +done + +if [[ "$SUMMARY_OUTPUT" != SUMMARIES_OK* ]]; then + fail "no device summary reached '$CURATED_HUB_NAME' within ${SUMMARY_WAIT_SECONDS}s - the archive was never processed" +fi + +echo "$SUMMARY_OUTPUT" | sed 's/^/ /' + +echo "" +echo "Fraud-free but not signal-free: '$EXCURSION_DEVICE' should report excursions above" +echo "${TEMPERATURE_LIMIT}C, and every other device none." +if ! echo "$SUMMARY_OUTPUT" | grep -q "$EXCURSION_DEVICE"; then + fail "the summaries do not mention '$EXCURSION_DEVICE', so the archive was only partly processed" +fi + +# ----------------------------------------------------------------------------- +step "Step 5/5 - What just happened" +# ----------------------------------------------------------------------------- +cat <<'SUMMARY' + telemetry hub devices published, keyed by device id + | + | Capture (60s window) + v + Avro archive in Blob durable, ordered, replayable + | + | Microsoft.EventHub.CaptureFileCreated + v + Event Grid system topic + | + | subscription with an EventHub destination + v + capture-notifications a stream, not a webhook + | + | Event Hubs trigger + v + Function App downloads the archive, aggregates per device + | + | Event Hubs output binding + v + curated hub one summary per device +SUMMARY + +echo "" +echo "Pipeline demo complete: every stage ran and was verified." diff --git a/samples/eventhubs-eventgrid/python/scripts/validate.sh b/samples/eventhubs-eventgrid/python/scripts/validate.sh new file mode 100644 index 0000000..50ea08d --- /dev/null +++ b/samples/eventhubs-eventgrid/python/scripts/validate.sh @@ -0,0 +1,181 @@ +#!/bin/bash + +# ============================================================================= +# Verifies every capability the cold-path pipeline deploys, in the order an +# operator would check them: Event Hubs, then Capture, then the Event Grid +# wiring that connects them, then the workloads and the data plane. +# ============================================================================= + +PREFIX='local' +SUFFIX='telemetry' +RESOURCE_GROUP_NAME="${PREFIX}-ehgrid-rg" +EVENTHUB_NAMESPACE_NAME="${PREFIX}-ehns-${SUFFIX}" +TELEMETRY_HUB_NAME='telemetry' +NOTIFICATION_HUB_NAME='capture-notifications' +CURATED_HUB_NAME='curated' +CAPTURE_CONTAINER_NAME='telemetry-archive' +CAPTURE_CONSUMER_GROUP='capture-processor' +STORAGE_ACCOUNT_NAME="${PREFIX}ehgrid${SUFFIX:0:4}" +SYSTEM_TOPIC_NAME="${PREFIX}-ehns-systopic" +SUBSCRIPTION_NAME='capture-to-eventhub' +FUNCTION_APP_NAME="${PREFIX}-ehgrid-processor" +SEND_RULE_NAME='telemetry-send' +LISTEN_RULE_NAME='pipeline-listen' +TELEMETRY_PARTITION_COUNT=4 + +CURRENT_DIR="$(cd "$(dirname "$0")" && pwd)" +PYTHON_BIN="${PYTHON_BIN:-python3}" + +PASS_COUNT=0 +FAIL_COUNT=0 + +check() { + local description="$1" + local command="$2" + echo -n " Checking $description... " + if eval "$command" &>/dev/null; then + echo "OK" + PASS_COUNT=$((PASS_COUNT + 1)) + else + echo "FAIL" + FAIL_COUNT=$((FAIL_COUNT + 1)) + fi +} + +check_output() { + local description="$1" + local command="$2" + local expected="$3" + echo -n " Checking $description... " + local output + output=$(eval "$command" 2>/dev/null) + if echo "$output" | grep -q "$expected"; then + echo "OK" + PASS_COUNT=$((PASS_COUNT + 1)) + else + echo "FAIL (expected '$expected', got '$(echo "$output" | head -1)')" + FAIL_COUNT=$((FAIL_COUNT + 1)) + fi +} + +echo "============================================================" +echo "Validating the Capture -> Event Grid -> Functions pipeline" +echo "============================================================" + +echo "" +echo "--- Part 1: Event Hubs ---" +echo "" + +echo "[1] Namespace and hubs" +check "resource group exists" \ + "az group show --name $RESOURCE_GROUP_NAME" +check "Event Hubs namespace exists" \ + "az eventhubs namespace show --name $EVENTHUB_NAMESPACE_NAME --resource-group $RESOURCE_GROUP_NAME" +check_output "namespace is Standard tier (Capture requires it)" \ + "az eventhubs namespace show --name $EVENTHUB_NAMESPACE_NAME --resource-group $RESOURCE_GROUP_NAME --query sku.name -o tsv" \ + "Standard" +for hub in "$TELEMETRY_HUB_NAME" "$NOTIFICATION_HUB_NAME" "$CURATED_HUB_NAME"; do + check "hub '$hub' exists" \ + "az eventhubs eventhub show --name $hub --namespace-name $EVENTHUB_NAMESPACE_NAME --resource-group $RESOURCE_GROUP_NAME" +done +check_output "telemetry hub has $TELEMETRY_PARTITION_COUNT partitions" \ + "az eventhubs eventhub show --name $TELEMETRY_HUB_NAME --namespace-name $EVENTHUB_NAMESPACE_NAME --resource-group $RESOURCE_GROUP_NAME --query partitionCount -o tsv" \ + "^${TELEMETRY_PARTITION_COUNT}$" +echo "" + +echo "[2] Capture configuration" +check_output "Capture is enabled on the telemetry hub" \ + "az eventhubs eventhub show --name $TELEMETRY_HUB_NAME --namespace-name $EVENTHUB_NAMESPACE_NAME --resource-group $RESOURCE_GROUP_NAME --query captureDescription.enabled -o tsv" \ + "true" +check_output "Capture encoding is Avro" \ + "az eventhubs eventhub show --name $TELEMETRY_HUB_NAME --namespace-name $EVENTHUB_NAMESPACE_NAME --resource-group $RESOURCE_GROUP_NAME --query captureDescription.encoding -o tsv" \ + "Avro" +check_output "Capture targets the archive container" \ + "az eventhubs eventhub show --name $TELEMETRY_HUB_NAME --namespace-name $EVENTHUB_NAMESPACE_NAME --resource-group $RESOURCE_GROUP_NAME --query captureDescription.destination.blobContainer -o tsv" \ + "$CAPTURE_CONTAINER_NAME" +check "consumer group '$CAPTURE_CONSUMER_GROUP' exists on the notifications hub" \ + "az eventhubs eventhub consumer-group show --consumer-group-name $CAPTURE_CONSUMER_GROUP --eventhub-name $NOTIFICATION_HUB_NAME --namespace-name $EVENTHUB_NAMESPACE_NAME --resource-group $RESOURCE_GROUP_NAME" +echo "" + +echo "[3] Authorization rules (least privilege)" +check_output "telemetry send rule grants only Send" \ + "az eventhubs eventhub authorization-rule show --name $SEND_RULE_NAME --eventhub-name $TELEMETRY_HUB_NAME --namespace-name $EVENTHUB_NAMESPACE_NAME --resource-group $RESOURCE_GROUP_NAME --query \"join(',', rights)\" -o tsv" \ + "^Send$" +check_output "pipeline listen rule grants only Listen" \ + "az eventhubs namespace authorization-rule show --name $LISTEN_RULE_NAME --namespace-name $EVENTHUB_NAMESPACE_NAME --resource-group $RESOURCE_GROUP_NAME --query \"join(',', rights)\" -o tsv" \ + "^Listen$" +echo "" + +echo "--- Part 2: Event Grid wiring ---" +echo "" + +echo "[4] System topic over the namespace" +check "system topic exists" \ + "az eventgrid system-topic show --name $SYSTEM_TOPIC_NAME --resource-group $RESOURCE_GROUP_NAME" +check_output "system topic is an Event Hubs namespace topic type" \ + "az eventgrid system-topic show --name $SYSTEM_TOPIC_NAME --resource-group $RESOURCE_GROUP_NAME --query topicType -o tsv" \ + "[Ee]venthub" +check_output "system topic sources the Event Hubs namespace" \ + "az eventgrid system-topic show --name $SYSTEM_TOPIC_NAME --resource-group $RESOURCE_GROUP_NAME --query source -o tsv" \ + "$EVENTHUB_NAMESPACE_NAME" +echo "" + +echo "[5] Event subscription delivering to an event hub" +check "event subscription exists" \ + "az eventgrid system-topic event-subscription show --name $SUBSCRIPTION_NAME --system-topic-name $SYSTEM_TOPIC_NAME --resource-group $RESOURCE_GROUP_NAME" +check_output "its destination is an EventHub endpoint" \ + "az eventgrid system-topic event-subscription show --name $SUBSCRIPTION_NAME --system-topic-name $SYSTEM_TOPIC_NAME --resource-group $RESOURCE_GROUP_NAME --query destination.endpointType -o tsv" \ + "EventHub" +echo "" + +echo "--- Part 3: Workloads and data plane ---" +echo "" + +echo "[6] Function App" +check "function app exists" \ + "az functionapp show --name $FUNCTION_APP_NAME --resource-group $RESOURCE_GROUP_NAME" +check_output "it is configured with the notifications hub" \ + "az functionapp config appsettings list --name $FUNCTION_APP_NAME --resource-group $RESOURCE_GROUP_NAME --query \"[?name=='NOTIFICATION_HUB_NAME'].value\" -o tsv" \ + "$NOTIFICATION_HUB_NAME" +check_output "it is configured with the curated hub" \ + "az functionapp config appsettings list --name $FUNCTION_APP_NAME --resource-group $RESOURCE_GROUP_NAME --query \"[?name=='CURATED_HUB_NAME'].value\" -o tsv" \ + "$CURATED_HUB_NAME" +echo "" + +echo "[7] Storage" +check "storage account exists" \ + "az storage account show --name $STORAGE_ACCOUNT_NAME --resource-group $RESOURCE_GROUP_NAME" +echo "" + +echo "[8] Data plane round trip" +echo -n " Checking the telemetry hub accepts and returns an event... " +ROUNDTRIP=$(EVENTHUB_SEND_CONNECTION_STRING=$(az eventhubs eventhub authorization-rule keys list \ + --name "$SEND_RULE_NAME" --eventhub-name "$TELEMETRY_HUB_NAME" \ + --namespace-name "$EVENTHUB_NAMESPACE_NAME" --resource-group "$RESOURCE_GROUP_NAME" \ + --query primaryConnectionString -o tsv --only-show-errors) \ + EVENTHUB_LISTEN_CONNECTION_STRING=$(az eventhubs namespace authorization-rule keys list \ + --name "$LISTEN_RULE_NAME" --namespace-name "$EVENTHUB_NAMESPACE_NAME" \ + --resource-group "$RESOURCE_GROUP_NAME" \ + --query primaryConnectionString -o tsv --only-show-errors) \ + EVENT_HUB_NAME="$TELEMETRY_HUB_NAME" \ + "$PYTHON_BIN" "$CURRENT_DIR/roundtrip_check.py" 2>&1 | tail -1) +if [[ "$ROUNDTRIP" == ROUNDTRIP_OK* ]]; then + echo "OK (${ROUNDTRIP#ROUNDTRIP_OK })" + PASS_COUNT=$((PASS_COUNT + 1)) +else + echo "FAIL ($ROUNDTRIP)" + FAIL_COUNT=$((FAIL_COUNT + 1)) +fi +echo "" + +echo "============================================================" +echo "Validation results: $PASS_COUNT passed, $FAIL_COUNT failed" +echo "============================================================" + +if [ $FAIL_COUNT -eq 0 ]; then + echo "PASS: the pipeline is deployed and every capability responded." + exit 0 +else + echo "FAIL: review the output above." + exit 1 +fi diff --git a/samples/eventhubs-eventgrid/python/src/functions/function_app.py b/samples/eventhubs-eventgrid/python/src/functions/function_app.py new file mode 100644 index 0000000..92c5992 --- /dev/null +++ b/samples/eventhubs-eventgrid/python/src/functions/function_app.py @@ -0,0 +1,189 @@ +"""Cold-path automation: process each Capture archive the moment it lands. + +The hot path reacts to individual events. This is the *cold* path, and it is driven entirely by +Azure telling us when a batch is ready: + + telemetry hub --(Capture, 60s windows)--> Avro archive in Blob Storage + | + Event Hubs raises Microsoft.EventHub.CaptureFileCreated + | + Event Grid system topic + | + subscription with an EventHub destination + | + capture-notifications hub + | + THIS FUNCTION (Event Hubs trigger) + | + downloads the archive, aggregates it, and writes + one summary per device to the curated hub + (Event Hubs output binding) + +Why this shape rather than a plain stream processor: the archive is already batched, ordered and +durable, so the aggregation reads whole windows instead of guessing batch boundaries, and it can +be replayed from Blob Storage at any time. That is the pattern behind Microsoft's own "stream big +data into a data warehouse" tutorial. +https://learn.microsoft.com/en-us/azure/event-grid/event-schema-event-hubs +""" + +import json +import logging +import os +from datetime import UTC, datetime +from typing import Any, List +from urllib.parse import unquote, urlparse + +import azure.functions as func + +app = func.FunctionApp() + +CAPTURE_FILE_CREATED = "Microsoft.EventHub.CaptureFileCreated" +# A reading at or above this is reported as an excursion on the device's summary. +TEMPERATURE_LIMIT = float(os.environ.get("TEMPERATURE_LIMIT", "80")) + + +@app.function_name(name="CaptureProcessor") +@app.event_hub_message_trigger( + arg_name="notifications", + event_hub_name="%NOTIFICATION_HUB_NAME%", + connection="EVENTHUB_NOTIFICATION_CONNECTION", + consumer_group="%CAPTURE_CONSUMER_GROUP%", + # Cardinality.MANY delivers a list; the default (ONE) delivers a bare EventHubEvent and the + # handler below would fail with "'EventHubEvent' object is not iterable". + cardinality=func.Cardinality.MANY, +) +@app.event_hub_output( + arg_name="curated", + event_hub_name="%CURATED_HUB_NAME%", + connection="EVENTHUB_CURATED_CONNECTION", +) +def CaptureProcessor(notifications: List[func.EventHubEvent], curated: func.Out[List[str]]) -> None: + """Turn every CaptureFileCreated notification into per-device summaries.""" + summaries: list[str] = [] + + for notification in notifications: + for event in _event_grid_batch(notification): + if event.get("eventType") != CAPTURE_FILE_CREATED: + # The subscription is scoped to the namespace, so unrelated system events would + # arrive here too if Azure ever adds more for Event Hubs. + continue + data = event.get("data") or {} + try: + readings = _read_archive(data["fileUrl"]) + except Exception: + # A summary is derived data: losing one archive must not park the notification + # and re-deliver it forever. Log loudly and move on. + logging.exception("CAPTURE_ARCHIVE_UNREADABLE url=%s", data.get("fileUrl")) + continue + + summaries.extend( + json.dumps(summary) for summary in _summarize(readings, data, event["subject"]) + ) + + logging.info("CAPTURE_BATCH notifications=%d summaries=%d", len(notifications), len(summaries)) + if summaries: + curated.set(summaries) + + +def _event_grid_batch(notification: func.EventHubEvent) -> list[dict[str, Any]]: + """Parse one delivered message into the list of Event Grid events it carries. + + Event Grid always delivers a JSON array - "an array with a single event" by default, longer + only when batching is enabled on the subscription. + https://learn.microsoft.com/en-us/azure/event-grid/delivery-and-retry + """ + body = notification.get_body().decode("utf-8") + try: + parsed = json.loads(body) + except ValueError: + logging.warning("NOTIFICATION_NOT_JSON body=%s", body[:200]) + return [] + if isinstance(parsed, dict): + # Tolerated so the function still works if a producer hand-posts a single event. + return [parsed] + return parsed + + +def _read_archive(file_url: str) -> list[dict[str, Any]]: + """Download a Capture archive and decode its Avro records into telemetry dicts.""" + import io + + import fastavro + from azure.storage.blob import BlobServiceClient + + connection_string = os.environ["STORAGE_CONNECTION_STRING"] + container, blob_name = _split_blob_url(file_url) + + service = BlobServiceClient.from_connection_string(connection_string) + payload = service.get_container_client(container).download_blob(blob_name).readall() + + readings = [] + for record in fastavro.reader(io.BytesIO(payload)): + body = record.get("Body") + if not body: + continue + try: + readings.append(json.loads(bytes(body).decode("utf-8"))) + except ValueError: + continue + return readings + + +def _split_blob_url(file_url: str) -> tuple[str, str]: + """Split a Capture fileUrl into (container, blob name). + + The URL is whatever the service put in the event, so the path is taken apart rather than + rebuilt from configuration - the container name is not assumed to be the one we deployed. + """ + path = unquote(urlparse(file_url).path).lstrip("/") + container, _, blob_name = path.partition("/") + return container, blob_name + + +def _summarize( + readings: list[dict[str, Any]], capture_data: dict[str, Any], subject: str +) -> list[dict[str, Any]]: + """Aggregate one archive's readings into a summary per device.""" + by_device: dict[str, list[dict[str, Any]]] = {} + for reading in readings: + device_id = reading.get("device_id") + if device_id: + by_device.setdefault(device_id, []).append(reading) + + summaries = [] + for device_id, device_readings in sorted(by_device.items()): + temperatures = [ + float(reading["temperature"]) + for reading in device_readings + if reading.get("temperature") is not None + ] + if not temperatures: + continue + excursions = [value for value in temperatures if value >= TEMPERATURE_LIMIT] + summaries.append( + { + "device_id": device_id, + "source_hub": subject, + "partition_id": capture_data.get("partitionId"), + "archive_url": capture_data.get("fileUrl"), + # The archive's own sequence range: a downstream consumer can use it to detect a + # gap between windows without re-reading the hub. + "first_sequence_number": capture_data.get("firstSequenceNumber"), + "last_sequence_number": capture_data.get("lastSequenceNumber"), + "reading_count": len(temperatures), + "min_temperature": round(min(temperatures), 2), + "max_temperature": round(max(temperatures), 2), + "mean_temperature": round(sum(temperatures) / len(temperatures), 2), + "excursion_count": len(excursions), + "temperature_limit": TEMPERATURE_LIMIT, + "summarized_at": datetime.now(UTC).isoformat(), + } + ) + logging.info( + "DEVICE_SUMMARY device=%s readings=%d max=%.2f excursions=%d", + device_id, + len(temperatures), + max(temperatures), + len(excursions), + ) + return summaries diff --git a/samples/eventhubs-eventgrid/python/src/functions/host.json b/samples/eventhubs-eventgrid/python/src/functions/host.json new file mode 100644 index 0000000..b0b2ce1 --- /dev/null +++ b/samples/eventhubs-eventgrid/python/src/functions/host.json @@ -0,0 +1,20 @@ +{ + "version": "2.0", + "logging": { + "logLevel": { + "default": "Information" + } + }, + "extensionBundle": { + "id": "Microsoft.Azure.Functions.ExtensionBundle", + "version": "[4.*, 5.0.0)" + }, + "extensions": { + "eventHubs": { + "maxEventBatchSize": 10, + "batchCheckpointFrequency": 1, + "prefetchCount": 100 + } + }, + "functionTimeout": "00:05:00" +} diff --git a/samples/eventhubs-eventgrid/python/src/functions/requirements.txt b/samples/eventhubs-eventgrid/python/src/functions/requirements.txt new file mode 100644 index 0000000..cadeaf9 --- /dev/null +++ b/samples/eventhubs-eventgrid/python/src/functions/requirements.txt @@ -0,0 +1,3 @@ +azure-functions +azure-storage-blob==12.24.0 +fastavro==1.10.0 diff --git a/samples/eventhubs-eventgrid/python/src/producers/requirements.txt b/samples/eventhubs-eventgrid/python/src/producers/requirements.txt new file mode 100644 index 0000000..f58d611 --- /dev/null +++ b/samples/eventhubs-eventgrid/python/src/producers/requirements.txt @@ -0,0 +1,4 @@ +azure-eventhub==5.15.0 +azure-identity==1.19.0 +azure-storage-blob==12.24.0 +fastavro==1.10.0 diff --git a/samples/eventhubs-eventgrid/python/src/producers/telemetry_producer.py b/samples/eventhubs-eventgrid/python/src/producers/telemetry_producer.py new file mode 100644 index 0000000..911cec9 --- /dev/null +++ b/samples/eventhubs-eventgrid/python/src/producers/telemetry_producer.py @@ -0,0 +1,111 @@ +"""Publish cold-chain telemetry into the Event Hubs stream that Capture archives. + +Every reading is keyed on the device id, so a device's readings always land on the same partition +and therefore appear in the same Capture archive in order. That is what lets the downstream +processor aggregate per device from a single file instead of stitching partitions together. + + python telemetry_producer.py [--count 120] [--devices 6] [--excursion-device DEV-0003] +""" + +import argparse +import json +import os +import random +import sys +from datetime import UTC, datetime +from uuid import uuid4 + +from azure.eventhub import EventData, EventHubProducerClient + +SITES = ["depot-lisbon", "depot-porto", "depot-madrid"] +# Normal cold-chain range; the excursion device deliberately breaches the upper limit. +NORMAL_RANGE = (2.0, 8.0) +EXCURSION_RANGE = (81.0, 95.0) + + +def progress(message: str) -> None: + print(f"[{datetime.now().strftime('%H:%M:%S')}] {message}", flush=True) + + +def require_env(name: str) -> str: + value = os.environ.get(name) + if not value: + raise SystemExit( + f"Environment variable {name} is not set. " + f"Run 'source scripts/.deployment-env' (written by scripts/deploy.sh) first." + ) + return value + + +def make_reading(device_id: str, temperature: float) -> dict: + return { + "reading_id": str(uuid4()), + "device_id": device_id, + "site": random.choice(SITES), + "temperature": round(temperature, 2), + "humidity": round(random.uniform(30.0, 65.0), 2), + "battery_pct": random.randint(40, 100), + "recorded_at": datetime.now(UTC).isoformat(), + } + + +def main() -> None: + parser = argparse.ArgumentParser(description="Publish cold-chain telemetry readings.") + parser.add_argument("--count", type=int, default=120, help="number of readings to publish") + parser.add_argument("--devices", type=int, default=6, help="number of distinct devices") + parser.add_argument( + "--excursion-device", + help="device id that also emits out-of-range readings, so the summary reports excursions", + ) + args = parser.parse_args() + + connection_string = require_env("EVENTHUB_SEND_CONNECTION_STRING") + hub_name = require_env("TELEMETRY_HUB_NAME") + + device_ids = [f"DEV-{index:04d}" for index in range(1, args.devices + 1)] + readings = [ + make_reading(random.choice(device_ids), random.uniform(*NORMAL_RANGE)) + for _ in range(args.count) + ] + if args.excursion_device: + readings += [ + make_reading(args.excursion_device, random.uniform(*EXCURSION_RANGE)) for _ in range(8) + ] + + producer = EventHubProducerClient.from_connection_string( + connection_string, eventhub_name=hub_name + ) + progress(f"connected to '{hub_name}' over AMQP") + + # One batch per device: the partition key travels with the batch, so a device's readings stay + # together and in order, which is what the archive-level aggregation depends on. + by_device: dict[str, list[dict]] = {} + for reading in readings: + by_device.setdefault(reading["device_id"], []).append(reading) + + sent = 0 + with producer: + for device_id, device_readings in by_device.items(): + batch = producer.create_batch(partition_key=device_id) + for reading in device_readings: + event = EventData(json.dumps(reading)) + event.properties = {"device_id": device_id, "site": reading["site"]} + try: + batch.add(event) + except ValueError: + producer.send_batch(batch) + sent += len(batch) + batch = producer.create_batch(partition_key=device_id) + batch.add(event) + producer.send_batch(batch) + sent += len(batch) + + excursions = sum(1 for reading in readings if reading["temperature"] >= 80) + progress( + f"published {sent} readings from {len(by_device)} device(s); " + f"{excursions} above the temperature limit" + ) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/samples/eventhubs-eventgrid/python/terraform/deploy.sh b/samples/eventhubs-eventgrid/python/terraform/deploy.sh new file mode 100644 index 0000000..bfedf15 --- /dev/null +++ b/samples/eventhubs-eventgrid/python/terraform/deploy.sh @@ -0,0 +1,108 @@ +#!/bin/bash + +# ============================================================================= +# Deploys the cold-path pipeline with Terraform, then publishes the application code. +# Terraform provisions the infrastructure; the Function App package is pushed with the Azure CLI, +# as in the other samples. +# ============================================================================= + +CURRENT_DIR="$(cd "$(dirname "$0")" && pwd)" +SRC_DIR="$(cd "$CURRENT_DIR/../src" && pwd)" +FUNCTION_ZIP='capture_processor.zip' + +cd "$CURRENT_DIR" || exit 1 + +fail() { + echo "ERROR: $1" + exit 1 +} + +echo "=== Initializing Terraform ===" +terraform init -input=false || fail "terraform init failed" + +echo "" +echo "=== Validating the configuration ===" +terraform validate || fail "terraform validate failed" + +echo "" +echo "=== Planning ===" +terraform plan -input=false -out=tfplan || fail "terraform plan failed" + +echo "" +echo "=== Applying ===" +terraform apply -input=false -auto-approve tfplan || fail "terraform apply failed" + +echo "" +echo "=== Re-applying (the configuration must converge without errors) ===" +terraform apply -input=false -auto-approve || fail "the second apply failed" + +# A perfectly idempotent second apply reports "No changes". Against the emulator a few resources +# still report drift because some properties are accepted on create but not echoed back on read, +# so the provider re-sends them every run. That is an emulator parity gap rather than a problem +# with this configuration: the re-apply converges and changes nothing observable. +DRIFT=$( + terraform plan -input=false -detailed-exitcode >/dev/null 2>&1 + echo $? +) +if [[ "$DRIFT" == "0" ]]; then + echo "Second apply is fully idempotent (no changes)." +else + echo "Note: the plan still reports in-place updates for properties the emulator does not" + echo "echo back on read. The configuration converges; nothing is recreated." +fi + +# ----------------------------------------------------------------------------- +echo "" +echo "=== Publishing the application code ===" +# ----------------------------------------------------------------------------- +RESOURCE_GROUP_NAME=$(terraform output -raw resource_group_name) +FUNCTION_APP_NAME=$(terraform output -raw function_app_name) + +cd "$SRC_DIR/functions" || fail "missing src/functions" +rm -f "$CURRENT_DIR/$FUNCTION_ZIP" +zip -r "$CURRENT_DIR/$FUNCTION_ZIP" function_app.py host.json requirements.txt 1>/dev/null +cd "$CURRENT_DIR" || exit 1 + +echo "Deploying the capture processor (the first run can take several minutes)..." +az functionapp deploy \ + --resource-group "$RESOURCE_GROUP_NAME" \ + --name "$FUNCTION_APP_NAME" \ + --src-path "$FUNCTION_ZIP" \ + --type zip 1>/dev/null || fail "could not deploy the function app" +rm -f "$FUNCTION_ZIP" + +# ----------------------------------------------------------------------------- +echo "" +echo "=== Verifying the deployment ===" +# ----------------------------------------------------------------------------- +EVENTHUB_NAMESPACE_NAME=$(terraform output -raw eventhub_namespace_name) +TELEMETRY_HUB_NAME=$(terraform output -raw telemetry_hub_name) + +az eventhubs eventhub show --name "$TELEMETRY_HUB_NAME" \ + --namespace-name "$EVENTHUB_NAMESPACE_NAME" --resource-group "$RESOURCE_GROUP_NAME" \ + --query "{name:name, partitions:partitionCount, capture:captureDescription.enabled}" \ + --output table --only-show-errors || fail "the telemetry hub is not readable" + +# Mirror scripts/deploy.sh so the demo runs after a Terraform deployment too. +cat >"$CURRENT_DIR/../scripts/.deployment-env" <