Terraform module for Databricks Azure Workspace (Part 1)

❗️ Important

👉 This Terraform module assumes you have logged into Azure using az login and have Contributor access to the subscription.

➡️️ Part 2: Terraform module for Databricks Workspace management

Versions

• Module tested for Terraform 1.1.4
• databrickslabs/databricks provider version 0.4.6
• Azure provider version 2.94.0.
• main branch: Provider versions may not be pinned to keep up with Terraform releases.
• tags releases: Tags are pinned with versions (use ).

---

Usage

Option 1:

terrafrom init
terraform plan -var='teamid=tryme' -var='prjid=project1'
terraform apply -var='teamid=tryme' -var='prjid=project1'
terraform destroy -var='teamid=tryme' -var='prjid=project1'

Note: With this option please take care of remote state storage

Option 2:

Recommended method (stores remote state in S3 using prjid and teamid to create directory structure):

• Create python 3.6+ virtual environment

python3 -m venv <venv name>

• Install package:

pip install tfremote --upgrade

• Set below environment variables:

export TF_AZURE_STORAGE_ACCOUNT=tfstatexxxxx # Output of remote_state.sh
export TF_AZURE_CONTAINER=tfstate # Output of remote_state.sh
export ARM_ACCESS_KEY=xxxxxxxxxx # Output of remote_state.sh

• Update examples directory with required values.

• Run and verify the output before deploying:

tf -c=azure plan -var='teamid=foo' -var='prjid=bar'

• Run below to deploy:

tf -c=azure apply -var='teamid=foo' -var='prjid=bar'

• Run below to destroy:

tf -c=azure destroy -var='teamid=foo' -var='prjid=bar'

NOTE:

• Read more on tfremote

Databricks workspace creation

module "databricks_workspace" {
  source = "git::git@github.com:tomarv2/terraform-databricks-azure-workspace.git"

  resource_group_settings = [
    {
      name     = "demo1"
      location = "westus2"
  }]
  # ---------------------------------------------
  # Note: Do not change teamid and prjid once set.
  teamid = var.teamid
  prjid  = var.prjid
}

Please refer to examples directory link for references.

References

- Terraform module for Databricks AWS Workspace

Requirements

Name	Version
terraform	>= 1.0.1
azurerm	~> 3.21.1
databricks	1.2.1
external	~> 2.2
random	~> 3.1

Providers

Name	Version
azurerm	~> 3.21.1
databricks.created_workspace	1.2.1
external	~> 2.2
random	~> 3.1

Modules

Name	Source	Version
resource_group	git::git@github.com:tomarv2/terraform-azure-resource-group.git	v0.0.12

Resources

Name	Type
azurerm_databricks_workspace.this	resource
databricks_token.pat	resource
random_string.naming	resource
azurerm_client_config.current	data source
external_external.current_user	data source

Inputs

Name	Description	Type	Default	Required
custom_parameters	https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/databricks_workspace	any	null	no
custom_tags	Extra custom tags	any	null	no
customer_managed_key_enabled	Is the workspace enabled for customer managed key encryption? If true this enables the Managed Identity for the managed storage account. Possible values are true or false. Defaults to false. This field is only valid if the Databricks Workspace sku is set to premium. Changing this forces a new resource to be created	bool	false	no
databricks_token_lifetime_seconds	Validity of the databricks token in seconds	number	86400	no
deploy_resource_group	feature flag to deploy this resource or not	bool	false	no
infrastructure_encryption_enabled	Is the Databricks File System root file system enabled with a secondary layer of encryption with platform managed keys? Possible values are true or false. Defaults to false. This field is only valid if the Databricks Workspace sku is set to premium. Changing this forces a new resource to be created	bool	false	no
load_balancer_backend_address_pool_id	Resource ID of the Outbound Load balancer Backend Address Pool for Secure Cluster Connectivity (No Public IP) workspace. Changing this forces a new resource to be created	string	null	no
managed_services_cmk_key_vault_key_id	Customer managed encryption properties for the Databricks Workspace managed resources(e.g. Notebooks and Artifacts). Changing this forces a new resource to be created.	string	null	no
network_security_group_rules_required	Does the data plane (clusters) to control plane communication happen over private link endpoint only or publicly? Possible values AllRules, NoAzureDatabricksRules or NoAzureServiceRules. Required when public_network_access_enabled is set to false. Changing this forces a new resource to be created	string	null	no
prjid	Name of the project/stack e.g: mystack, nifieks, demoaci. Should not be changed after running 'tf apply'	string	n/a	yes
public_network_access_enabled	Allow public access for accessing workspace. Set value to false to access workspace only via private link endpoint. Possible values include true or false. Defaults to true. Changing this forces a new resource to be created	bool	true	no
region	The region where the resources are created	string	"westus2"	no
resource_group_name	Resource Group name	string	null	no
resource_group_settings	Resource Group name	map(any)	null	no
sku	The sku to use for the Databricks Workspace. Possible values are standard, premium, or trial. Changing this can force a new resource to be created in some circumstances	string	"standard"	no
teamid	Name of the team/group e.g. devops, dataengineering. Should not be changed after running 'tf apply'	string	n/a	yes
workspace_name	Specifies the name of the Databricks Workspace resource. Changing this forces a new resource to be created	string	null	no

Outputs

Name	Description
databricks_host	databricks workspace url
databricks_sku	The sku to use for the Databricks Workspace. Possible values are standard, premium, or trial. Changing this can force a new resource to be created in some circumstances
databricks_token	Value of the newly-created token
databricks_token_lifetime_hours	Token validity
databricks_workspace_id	databricks workspace id
managed_resource_group_name	databricks managed resource group name
nonsensitive_databricks_token	Value of the newly-created token
resource_group_name	databricks resource group name