Skip to content

Latest commit

 

History

History
213 lines (165 loc) · 13.6 KB

README.md

File metadata and controls

213 lines (165 loc) · 13.6 KB
Raw

Azure LogAnalytics exporter

license DockerHub Quay.io Artifact Hub

Prometheus exporter for Azure LogAnalytics Kusto queries with configurable fields and transformations.

azure-loganalytics-exporter can query configured workspaces or all workspaces in one or multiple subscriptions. The exporter can also cache metrics and servicediscovery information to reduce requests against workspaces and Azure API.

Usage

Usage:
  azure-loganalytics-exporter [OPTIONS]

Application Options:
      --log.debug                     debug mode [$LOG_DEBUG]
      --log.devel                     development mode [$LOG_DEVEL]
      --log.json                      Switch log output to json format [$LOG_JSON]
      --azure.environment=            Azure environment name (default: AZUREPUBLICCLOUD) [$AZURE_ENVIRONMENT]
      --azure.servicediscovery.cache= Duration for caching Azure ServiceDiscovery of workspaces to reduce API calls (time.Duration)
                                      (default: 30m) [$AZURE_SERVICEDISCOVERY_CACHE]
      --azure.resource-tag=           Azure Resource tags (space delimiter) (default: owner) [$AZURE_RESOURCE_TAG]
      --loganalytics.workspace=       Loganalytics workspace IDs [$LOGANALYTICS_WORKSPACE]
      --loganalytics.concurrency=     Specifies how many workspaces should be queried concurrently (default: 5) [$LOGANALYTICS_CONCURRENCY]
  -c, --config=                       Config path [$CONFIG]
      --server.bind=                  Server address (default: :8080) [$SERVER_BIND]
      --server.timeout.read=          Server read timeout (default: 5s) [$SERVER_TIMEOUT_READ]
      --server.timeout.write=         Server write timeout (default: 10s) [$SERVER_TIMEOUT_WRITE]

Help Options:
  -h, --help                          Show this help message

for Azure API authentication (using ENV vars) see https://docs.microsoft.com/en-us/azure/developer/go/azure-sdk-authentication

Configuration file

HTTP Endpoints

Endpoint Description
/query Query tester
/metrics Default prometheus golang metrics
/probe Execute loganalytics queries against workspaces (set on commandline/env var)
/probe/workspace Execute loganalytics queries against workspaces (defined as parameter)
/probe/subscription Execute loganalytics queries against workspaces (using servicediscovery)

HINT: parameters of type multiple can be either specified multiple times and/or splits multiple values by comma.

/probe parameters

uses predefined workspace list defined as parameter/environment variable on startup

GET parameter Default Required Multiple Description
module no no Filter queries by module name
cache no no Use of internal metrics caching (time.Duration)
parallel $LOGANALYTICS_PARALLEL no no Number (int) of how many workspaces can be queried at the same time

/probe/workspace parameters

uses dynamically passed workspaces via HTTP query parameter

GET parameter Default Required Multiple Description
module no no Filter queries by module name
workspace yes yes Workspace IDs which are probed
cache no no Use of internal metrics caching (time.Duration)
parallel $LOGANALYTICS_PARALLEL no no Number (int) of how many workspaces can be queried at the same time

/probe/subscription parameters

uses Azure service discovery to find all workspaces in one or multiple subscriptions

GET parameter Default Required Multiple Description
module no no Filter queries by module name
subscription yes yes Uses all workspaces inside subscription
filter no no Advanced filter for resource | {filter} | project id, customerId=properties.customerId ResoruceGraph query (available with 23.6.0)
cache no no Use of internal metrics caching (time.Duration)
parallel $LOGANALYTICS_PARALLEL no no Number (int) of how many workspaces can be queried at the same time
optional false no no Do not fail, if service discovery did not find any workspaces

Global metrics

available on /metrics

Metric Description
azure_loganalytics_status Status if query was successfull (per workspace, module, metric)
azure_loganalytics_last_query_successfull Timestamp of last successfull query (per workspace, module, metric)
azure_loganalytics_query_time Summary metric about query execution time (incl. all subqueries)
azure_loganalytics_query_results Number of results from query
azure_loganalytics_query_requests Count of requests (eg paged subqueries) per query
azure_loganalytics_workspace_query_count Count of discovered workspaces per module

AzureTracing metrics

(with 22.2.0 and later)

Azuretracing metrics collects latency and latency from azure-sdk-for-go and creates metrics and is controllable using environment variables (eg. setting buckets, disabling metrics or disable autoreset).

Metric Description
azurerm_api_ratelimit Azure ratelimit metrics (only on /metrics, resets after query due to limited validity)
azurerm_api_request_* Azure request count and latency as histogram

Settings

Environment variable Example Description
METRIC_AZURERM_API_REQUEST_BUCKETS 1, 2.5, 5, 10, 30, 60, 90, 120 Sets buckets for azurerm_api_request histogram metric
METRIC_AZURERM_API_REQUEST_ENABLE false Enables/disables azurerm_api_request_* metric
METRIC_AZURERM_API_REQUEST_LABELS apiEndpoint, method, statusCode Controls labels of azurerm_api_request_* metric
METRIC_AZURERM_API_RATELIMIT_ENABLE false Enables/disables azurerm_api_ratelimit metric
METRIC_AZURERM_API_RATELIMIT_AUTORESET false Enables/disables azurerm_api_ratelimit autoreset after fetch
azurerm_api_request label Status Description
apiEndpoint enabled by default hostname of endpoint (max 3 parts)
routingRegion enabled by default detected region for API call, either routing region from Azure Management API or Azure resource location
subscriptionID enabled by default detected subscriptionID
tenantID enabled by default detected tenantID (extracted from jwt auth token)
resourceProvider enabled by default detected Azure Management API provider
method enabled by default HTTP method
statusCode enabled by default HTTP status code

Examples

see example.yaml for general ingestion metrics (number of rows per second and number of bytes per second per table)

see example.aks-single.yaml for AKS namespace ingestion metrics (number of rows per second and number of bytes per AKS namespace) sending queryies to each LogAnalaytics workspace individually (single mode)

see example.aks-multi.yaml for AKS namespace ingestion metrics (number of rows per second and number of bytes per AKS namespace) sending one query against multiple LogAnalaytics workspaces (multi mode)

more examples of result processing can be found within azure-resourcegraph-expoter (uses same processing library)

Config file:

queries:
  - metric: azure_loganalytics_operationstatus_count
    query: |-
      Operation
      | summarize count() by OperationStatus
    fields:
      - name: count_
        type: value

Metrics:

# HELP azure_loganalytics_operationstatus_count azure_loganalytics_operationstatus_count
# TYPE azure_loganalytics_operationstatus_count gauge
azure_loganalytics_operationstatus_count{OperationStatus="Succeeded",workspaceId="xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx",workspaceTable="PrimaryResult"} 1

Prometheus configuration

predefined workspaces (at startup via parameter/environment variable)

- job_name: azure-loganalytics-exporter
  scrape_interval: 1m
  metrics_path: /probe
  params:
    cache: ["10m"]
    parallel: ["5"]
  static_configs:
  - targets: ["azure-loganalytics-exporter:8080"]

dynamic workspaces (defined in prometheus configuration)

- job_name: azure-loganalytics-exporter
  scrape_interval: 1m
  metrics_path: /probe/workspace
  params:
    workspace:
      - xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx
      - xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx
      - xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx
    cache: ["10m"]
    parallel: ["5"]
  static_configs:
  - targets: ["azure-loganalytics-exporter:8080"]

find workspaces with servicediscovery via subscription

- job_name: azure-loganalytics-exporter
  scrape_interval: 1m
  metrics_path: /probe/subscription
  params:
    subscription:
      - xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx
      - xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx
      - xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx
    cache: ["10m"]
    parallel: ["5"]
  static_configs:
  - targets: ["azure-loganalytics-exporter:8080"]