Skip to content

Report deployment and storage environment data in the usage report - #5828

Open
johnsimons wants to merge 2 commits into
masterfrom
john/telemetry
Open

Report deployment and storage environment data in the usage report#5828
johnsimons wants to merge 2 commits into
masterfrom
john/telemetry

Conversation

@johnsimons

Copy link
Copy Markdown
Member

The usage report already carries throughput, versions and whether audit and monitoring are in
use. It says nothing about how the instance is actually deployed or what it stores data in, so
questions like "how many customers run in containers", "who has moved off embedded RavenDB" or
"is anyone using S3 body storage with an IAM role" cannot be answered from the reports we receive.

This adds environment data covering the host, the persistence, the body storage and the
instance's configuration posture.

Keys

Group Keys
Host Host.Model (Container / WindowsService / Console), Host.Orchestrator, Host.OSPlatform, Host.OSVersion, Host.Architecture, Host.RuntimeVersion, Host.ProcessorCount, Host.AvailableMemoryGB
Persistence Persistence.Type, Persistence.RavenServer (Embedded / External), Persistence.Hosting, Persistence.ServerVersion, Persistence.FullTextSearch
Body storage Persistence.BodyStorage.Type, Persistence.BodyStorage.Auth, Limits.MaxBodySizeToStore
Posture Security.Authentication, Security.RoleBasedAuthorization, Security.Https, Features.MessageEditing, Features.ExternalIntegrationsPublishing, Features.ForwardErrorMessages, Features.EmailNotifications, Retention.ErrorHours, Retention.AuditHours, Retention.EventsHours

Keys are dotted strings, following the Features.IntegratedServicePulse key that was already
there. EnvironmentDataType is left alone rather than grown, so adding a datum no longer means
changing a contract.

No customer data

Every value is a fixed enum member, a boolean, a count or a version number. Nothing derived from
a host name, URL, connection string, bucket or container name, managed identity client id, file
path, machine name or endpoint name is emitted. RuntimeInformation.OSDescription is
deliberately not used because on Linux it carries a distro build string that identifies a
specific image. An acceptance test asserts the report carries no value matching the machine name,
and a unit test asserts no S3 bucket name or access key reaches the report.

Cloud database detection

Persistence.Hosting is resolved by asking the engine where it can, and by classifying the
configured host name where it cannot.

SQL Server answers definitively through SERVERPROPERTY('EngineEdition'), where 5, 8 and 11 mean
Azure SQL Database, Managed Instance and Synapse. PostgreSQL has no equivalent, so the probe
checks for the administrative roles that each managed offering creates and a self-hosted server
does not: azure_pg_admin, rds_superuser and cloudsqlsuperuser. Both probes run with a five
second command timeout and fall back to host suffix matching on any failure, so a report is never
held up by a database that will not answer.

Persistence.ServerVersion is the engine major version only. SQL Server's minor is always zero,
and Azure SQL Database reports major 12 regardless of the engine actually running, so
major.minor would add a digit that carries no information. Patch level, if it is ever wanted,
belongs in a separate key sourced from ProductUpdateLevel.

@warwickschroeder warwickschroeder left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think, if you havent already, we should test the probes using real cloud resources to ensure we arent getting false information back. It would be better to get an "Unavailable" or "Unknown" that a false "Azure SQL Database", for example.

Comment thread src/ServiceControl.Persistence/DatabaseHostClassifier.cs Outdated
Comment thread src/ServiceControl/ServiceControlErrorInstanceEnvironmentDataProvider.cs Outdated
Comment thread src/ServiceControl/ServiceControlErrorInstanceEnvironmentDataProvider.cs Outdated
Comment thread src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerDatabaseHostingProbe.cs Outdated
Comment thread src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerDatabaseHostingProbe.cs Outdated
…hosting classification

Introduces deferred reading for environment data so that a failure to fetch a specific datum doesn't discard the rest of a provider's output. Refines database hosting probes to better distinguish between confirmed self-hosted instances and unknown private hostnames, and adds a source field to track whether the classification was determined by a probe, configuration, or connection string.
@johnsimons

Copy link
Copy Markdown
Member Author

using real cloud resources to ensure we aren't getting false information back

I am spinning up temporary cloud infrastructure so we can check each probe against a real Azure SQL, Azure PostgreSQL, RDS and Cloud SQL instance before this ships.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants