Skip to main content
Version: Next

Grafana

Certified

Important Capabilities​

CapabilityStatusNotes
Column-level Lineage✅Enabled by default
Detect Deleted Entities✅Enabled by default
Extract Ownership✅Enabled by default
Extract Tags✅Enabled by default
Platform Instance✅Enabled by default
Table-Level Lineage✅Enabled by default

This plugin extracts metadata from Grafana and ingests it into DataHub. It connects to Grafana's API to extract metadata about dashboards, charts, and data sources. The following types of metadata are extracted:

  • Container Entities:

    • Folders: Top-level organizational units in Grafana
    • Dashboards: Collections of panels and charts
    • The full container hierarchy is preserved (Folders -> Dashboards -> Charts/Datasets)

  • Charts and Visualizations:

    • All panel types (graphs, tables, stat panels, etc.)
    • Chart configuration and properties
    • Links to the original Grafana UI
    • Custom properties including panel types and data source information
    • Input fields and schema information when available

  • Data Sources and Datasets:

    • Physical datasets representing Grafana's data sources
    • Dataset schema information extracted from queries and panel configurations
    • Support for various data source types (SQL, Prometheus, etc.)
    • Custom properties including data source type and configuration

  • Lineage Information:

    • Dataset-level lineage showing relationships between:
      • Source data systems and Grafana datasets
      • Grafana datasets and charts
    • Column-level lineage for SQL-based data sources
    • Support for external source systems through configurable platform mappings

  • Tags and Ownership:

    • Dashboard and chart tags
    • Ownership information derived from:
      • Dashboard creators
      • Technical owners based on dashboard UIDs
      • Custom ownership assignments

The source supports the following capabilities:

  • Platform instance support for multi-Grafana deployments
  • Stateful ingestion with support for soft-deletes
  • Fine-grained lineage at both dataset and column levels
  • Automated tag extraction
  • Support for both HTTP and HTTPS connections with optional SSL verification

Concept Mapping​

Source ConceptDataHub ConceptNotes
"grafana"Data Platform
Grafana FolderContainerSubtype Folder
Grafana DashboardContainerSubtype Dashboard
Grafana Panel/VisualizationChartVarious types mapped based on panel type (e.g., graph → LINE, pie → PIE)
Grafana Data SourceDatasetCreated for each panel's data source
Dashboard OwnerCorp UserDerived from dashboard UID and creator
Dashboard TagsTagSupports both simple tags and key:value tags

Compatibility​

The connector supports extracting metadata from any Grafana instance accessible via API. For SQL-based data sources, column-level lineage can be extracted when the queries are parseable. The connector supports various panel types and their transformations, and can work with both standalone Grafana instances and those integrated with other platforms.

For optimal lineage extraction from SQL-based data sources:

  • Queries should be well-formed and complete
  • Database/schema information should be properly configured in the connection settings
  • The platform mapping should be configured to match your data sources

Prerequisites:​

  1. A running Grafana instance
  2. A service account token with permissions to:
    • Read dashboards and folders
    • Access data source configurations
    • View user information

CLI based Ingestion​

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.


source:
    type: grafana
    config:
        # Coordinates
        platform_instance: production # optional
        env: PROD # optional
        url: https://grafana.company.com
        service_account_token: ${GRAFANA_SERVICE_ACCOUNT_TOKEN}

        # SSL verification for HTTPS connections
        verify_ssl: true # optional, default is true

        # Source type mapping for lineage
        connection_to_platform_map:
            postgres:
                platform: postgres
                database: grafana  # optional
                database_schema: grafana  # optional
                platform_instance: database_2  # optional
                env: PROD  # optional
            mysql_uid_1:  # Grafana datasource UID
                platform: mysql
                platform_instance: database_1  # optional
                database: my_database  # optional
sink:
  # sink configs

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
service_account_token ✅
string(password)
Grafana API token
url ✅
string
URL of Grafana instance (e.g. https://grafana.company.com)
ingest_owners
boolean
Whether to ingest owners from Grafana dashboards and charts
Default: True
ingest_tags
boolean
Whether to ingest tags from Grafana dashboards and charts
Default: True
page_size
integer
Number of grafana entities to query through each Grafana API call
Default: 100
platform_instance
string
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://datahubproject.io/docs/platform-instances/ for more details.
platform_instance_map
map(str,string)
verify_ssl
boolean
Verify SSL certificate for secure connections (https)
Default: True
env
string
The environment that all assets produced by this connector belong to
Default: PROD
connection_to_platform_map
map(str,PlatformConnectionConfig)
Any source that produces dataset urns in a single environment should inherit this class
connection_to_platform_map.key.platform_instance
string
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://datahubproject.io/docs/platform-instances/ for more details.
connection_to_platform_map.key.platform ❓
string
Upstream platform code (e.g. postgres, ms-sql)
connection_to_platform_map.key.database
string
Database name
connection_to_platform_map.key.database_schema
string
Schema name
connection_to_platform_map.key.env
string
The environment that all assets produced by this connector belong to
Default: PROD
stateful_ingestion
StatefulStaleMetadataRemovalConfig
Base specialized config for Stateful Ingestion with stale metadata removal capability.
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Code Coordinates​

  • Class Name: datahub.ingestion.source.grafana.grafana_source.GrafanaSource
  • Browse on GitHub

Questions

If you've got any questions on configuring ingestion for Grafana, feel free to ping us on our Slack.