Symptoms
ServiceControl 6.18.2 and above throws an ArgumentException when the Dead Letter Queue custom check runs when using Azure Service Bus transport with Managed Identity
Who's affected
Anyone using ServiceControl 6.18.2 and above with Azure Service Bus transport and Managed Identity
Root cause
The ASB DLQ custom check was constructing its own management client rather than relying on the connection settings to create one. This meant that Managed Identity configuration was not being correctly applied
Confirmed workarounds
None
Backported to
Original bug report
Describe the bug
When ServiceControl runs on the Azure Service Bus transport authenticated with a managed identity, the transport itself works correctly, but the "Dead Letter Queue" custom check (category: Transport) fails on every run. The check constructs its client directly from the raw connection string:
// src/ServiceControl.Transports.ASBS/DeadLetterQueueCheck.cs (line 34 in 6.18.2; still present on master)
var managementClient = new ServiceBusAdministrationClient(connectionString);
That constructor only supports SAS credentials, so the check bypasses the ASBS transport's own authentication machinery (ConnectionStringParser / AuthenticationMethod / TokenCredentialAuthentication) that makes managed identity work everywhere else. Both documented managed-identity connection string forms fail:
ConnectionString = bare fully-qualified namespace (<ns>.servicebus.windows.net): the transport connects fine, but the check throws a FormatException from the ServiceBusAdministrationClient constructor.ConnectionString = Endpoint=sb://<ns>.servicebus.windows.net/;Authentication=Managed Identity;ClientId=<clientId> (the documented managed-identity form, which the transport accepts): the transport connects fine, but the check throws System.ArgumentException from the same constructor — the connection-string-only constructor has no token-credential path, and on a namespace with disableLocalAuth=true there is no SAS to fall back to.
On an AAD-only namespace (disableLocalAuth=true, which organizational policy increasingly mandates) the check is therefore permanently red in ServicePulse. We found no configuration switch to suppress it; it re-fails hourly, which trains operators to ignore the one check that would flag real message loss in the <endpoint>.staging DLQ.
Observed error
With the documented managed-identity connection string form, every check run (hourly, per repeatAfter) logs:
Error|ServiceControl.CustomChecks.InternalCustomChecksHostedService|`ServiceControl.Transports.ASBS.DeadLetterQueueCheck` implementation failed to run|System.ArgumentException: The connection string used for an Service Bus client must specify the Service Bus namespace host and either a Shared Access Key (both the name and value) OR a Shared Access Signature to be valid. (Parameter 'connectionString')
at Azure.Messaging.ServiceBus.Administration.ServiceBusAdministrationClient..ctor(String connectionString, ServiceBusAdministrationClientOptions options)
at Azure.Messaging.ServiceBus.Administration.ServiceBusAdministrationClient..ctor(String connectionString)
at ServiceControl.Transports.ASBS.DeadLetterQueueCheck.PerformCheck(CancellationToken cancellationToken) in /_/src/ServiceControl.Transports.ASBS/DeadLetterQueueCheck.cs:line 34
at ServiceControl.CustomChecks.InternalCustomCheckManager.Run(CancellationToken cancellationToken) in /_/src/ServiceControl/CustomChecks/InternalCustomChecks/InternalCustomCheckManager.cs:line 43
ServicePulse shows the check as failed with "ServiceControl.Transports.ASBS.DeadLetterQueueCheck implementation failed to run."
Versions and environment
- ServiceControl 6.18.2 (error + audit + monitoring instances), ServicePulse 2.9.1
- Official Docker images (
particular/servicecontrol*) on Azure Container Apps, --setup-and-run
- Transport
NetStandardAzureServiceBus; namespace: Standard tier, disableLocalAuth=true
- User-assigned managed identity with Azure Service Bus Data Owner on the namespace;
ClientId pinned in the connection string
Everything else works with this setup: ingestion (error + audit + monitoring), queue creation at setup, retries. Only this check fails.
To reproduce
- Create an ASB namespace with local auth disabled (
disableLocalAuth=true) and grant a managed identity Azure Service Bus Data Owner.
- Run the ServiceControl error instance container with
TRANSPORTTYPE=NetStandardAzureServiceBus and CONNECTIONSTRING=Endpoint=sb://<ns>.servicebus.windows.net/;Authentication=Managed Identity;ClientId=<clientId>.
- Observe the instance run normally, then the "Dead Letter Queue" custom check fail on its first run (hourly thereafter). With the bare-FQNS connection string form, the instance logs a
FormatException from the check instead.
Expected behavior
The check should honor the transport's configured authentication method — e.g. construct ServiceBusAdministrationClient(fullyQualifiedNamespace, tokenCredential) when the connection settings resolve to token-credential auth, exactly as the transport connection does. The parsing/credential machinery already exists in ServiceControl.Transports.ASBS; the check just doesn't use it.
A possible CI note: the ASBS transport-test leg currently provisions a SAS connection string, so this path has no coverage. Since ci.yml already runs azure/login before the transport tests, a managed-identity variant could plausibly reuse the runner's credential via DefaultAzureCredential against the provisioned namespace.
Related
Symptoms
ServiceControl 6.18.2 and above throws an ArgumentException when the Dead Letter Queue custom check runs when using Azure Service Bus transport with Managed Identity
Who's affected
Anyone using ServiceControl 6.18.2 and above with Azure Service Bus transport and Managed Identity
Root cause
The ASB DLQ custom check was constructing its own management client rather than relying on the connection settings to create one. This meant that Managed Identity configuration was not being correctly applied
Confirmed workarounds
None
Backported to
Original bug report
Describe the bug
When ServiceControl runs on the Azure Service Bus transport authenticated with a managed identity, the transport itself works correctly, but the "Dead Letter Queue" custom check (category: Transport) fails on every run. The check constructs its client directly from the raw connection string:
That constructor only supports SAS credentials, so the check bypasses the ASBS transport's own authentication machinery (
ConnectionStringParser/AuthenticationMethod/TokenCredentialAuthentication) that makes managed identity work everywhere else. Both documented managed-identity connection string forms fail:ConnectionString= bare fully-qualified namespace (<ns>.servicebus.windows.net): the transport connects fine, but the check throws aFormatExceptionfrom theServiceBusAdministrationClientconstructor.ConnectionString=Endpoint=sb://<ns>.servicebus.windows.net/;Authentication=Managed Identity;ClientId=<clientId>(the documented managed-identity form, which the transport accepts): the transport connects fine, but the check throwsSystem.ArgumentExceptionfrom the same constructor — the connection-string-only constructor has no token-credential path, and on a namespace withdisableLocalAuth=truethere is no SAS to fall back to.On an AAD-only namespace (
disableLocalAuth=true, which organizational policy increasingly mandates) the check is therefore permanently red in ServicePulse. We found no configuration switch to suppress it; it re-fails hourly, which trains operators to ignore the one check that would flag real message loss in the<endpoint>.stagingDLQ.Observed error
With the documented managed-identity connection string form, every check run (hourly, per
repeatAfter) logs:ServicePulse shows the check as failed with "
ServiceControl.Transports.ASBS.DeadLetterQueueCheckimplementation failed to run."Versions and environment
particular/servicecontrol*) on Azure Container Apps,--setup-and-runNetStandardAzureServiceBus; namespace: Standard tier,disableLocalAuth=trueClientIdpinned in the connection stringEverything else works with this setup: ingestion (error + audit + monitoring), queue creation at setup, retries. Only this check fails.
To reproduce
disableLocalAuth=true) and grant a managed identity Azure Service Bus Data Owner.TRANSPORTTYPE=NetStandardAzureServiceBusandCONNECTIONSTRING=Endpoint=sb://<ns>.servicebus.windows.net/;Authentication=Managed Identity;ClientId=<clientId>.FormatExceptionfrom the check instead.Expected behavior
The check should honor the transport's configured authentication method — e.g. construct
ServiceBusAdministrationClient(fullyQualifiedNamespace, tokenCredential)when the connection settings resolve to token-credential auth, exactly as the transport connection does. The parsing/credential machinery already exists inServiceControl.Transports.ASBS; the check just doesn't use it.A possible CI note: the ASBS transport-test leg currently provisions a SAS connection string, so this path has no coverage. Since
ci.ymlalready runsazure/loginbefore the transport tests, a managed-identity variant could plausibly reuse the runner's credential viaDefaultAzureCredentialagainst the provisioned namespace.Related
Authentication=Managed Identityfor the transport connection at startup (closed, 4.21.8) but did not cover this custom check.