Configuring network tags in cdap_profile for CDAP application instance

[blackvvine]

We are trying to configure the Network Tags used by the ephemeral Dataproc clusters created by Cloud Data Fusion, in order to set up correct firewall configurations.

Manually, this can be configured via System Admin -> Configuration -> System Compute Profiles -> [create new] -> Cluster Metadata -> Network Tags or in a deployed pipeline, from Configure -> Compute Config -> (select profile) -> Customize -> Cluster Metadata -> Network Tags.

I understand the CDF provider for Terraform does have the Dataproc Profile module which also accepts arbitrary Dataproc Cluster Properties. However, the network tags are not configurable via cluster properties, but rather the --tags argument.

Is there a workaround for achieving this? Note that the configuration pertains to ephemeral clusters created by Cloud Data Fusion during its lifetime, so the name/ID of the target GCE VMs is unknown beforehand and it has to be configured in the Dataproc Profile.

[umairidris]

@blackvvine I know we synced offline on this, did you end up resolving it?

[blackvvine]

@umairidris we couldn't find a workaround so we ended up using a specific IP range instead of network tags. I think adding the capability to specify network tags is still worthwhile for other users.