Databricks administrator’s has more one tool to ensure management of tags in Serverless resources.

Tue purpose of Budget policies is tagging serverless workloads with policies for cost attribution.

Effective spend management is crucial to optimizing cost efficiency for serverless compute. By using tags, administrators can group billing records based on cost centers, projects, or other relevant categories. This strategy ensures comprehensive visibility into costs associated with each tag, simplifying budget management across departments and projects.

It will be applied for the following resources:

• Notebook Serverless
• Job Serverless
• DLT Serverless

All of these items can be queried by using the system tables combined with Genie. Let’s explore the step-by-step:

1 – Create a Budget policy

a) Setting -> Compute -> Budget Policies

2- Assign to a notebook

3 – Assign to a job using Serverless compute

4 – Query all the given policies through the SQL Query Editor

--list usage by month for a given budget policy
--by mozucadata.com

SELECT
  date_format(usage_date, 'yyyy-MM') as usage_date,
  system.billing.usage.sku_name,
  custom_tags ['BudgetPolicyName'] AS BudgetPolicyName,
  system.billing.usage.usage_unit,
  sum(round(usage_quantity)) AS quantity,
  SUM(
    round(
      system.billing.usage.usage_quantity * system.billing.list_prices.pricing.effective_list.default,
      2
    )
  ) AS cluster_spend
FROM
  `system`.billing.usage
  JOIN system.billing.list_prices ON system.billing.usage.sku_name = system.billing.list_prices.sku_name
WHERE
  custom_tags ['BudgetPolicyName'] is NOT NULL --custom_tags['project'] = 'campaign-prediction'
GROUP BY
  ALL

I am also recommend using this field to filter only serverless products:

billing.usage.product_features.is_serverless = TRUE

The result should return something like this:

Sometimes I realized the custom_tags ['BudgetPolicyName'] does not appear properly, but you can find by the tag created:

To ensure you are looking into correctly, it is optional to add another filter:

AND billing_origin_product in ('DLT', 'JOBS', 'INTERACTIVE')

Those filters (billing_origin_product and is_serverless) ensure restrict access to the scope of budget policies.

5 – Save the query to a Genie space

Budget policies have the following limitations:

• Existing notebooks, jobs, and Delta Live Tables pipelines are not automatically assigned policies after their owners are granted access to a policy. Policies are required on new assets, but existing assets must be updated to add a policy.
• Updates to tags won’t be reflected in new pipeline updates if the pipeline is in Development mode. The changes take 24 hours to propagate.
• It can take some time to reflect in the system tables
  

I hope you enjoy it!