Bug: ipConfigurations field is not populated in PrivateEndpoint status

[wjf3121]

Version of Azure Service Operator

ASO V2 v2.6.0

Describe the bug
When a private endpoint provisioned by ASO doesn't populate the ipConfigurations field in its status. We expect this field to be present so that we can get the private ip of the provisioned endpoint.

apiVersion: network.azure.com/v1api20220701
kind: PrivateEndpoint
metadata:
  annotations:
    serviceoperator.azure.com/latest-reconciled-generation: "1"
    serviceoperator.azure.com/operator-namespace: azureserviceoperator-system
    serviceoperator.azure.com/resource-id: /subscriptions/0000-00000-000000/resourceGroups/debug-rg/providers/Microsoft.Network/privateEndpoints/test
  creationTimestamp: "2024-03-28T07:06:26Z"
  finalizers:
  - serviceoperator.azure.com/finalizer
  generation: 1
  name: test
  namespace: default
  resourceVersion: "11865582"
  uid: fbf76c6d-3593-4f03-83c7-e8f43b66dc1b
spec:
  azureName: luketestpe
  location: central india
  owner:
    armId: /subscriptions/0000-00000-000000/resourceGroups/debug-rg
  privateLinkServiceConnections:
  - name: debug-service
    privateLinkServiceReference:
      armId: /subscriptions/0000-00000-000000/resourceGroups/debug-rg/providers/Microsoft.Network/privateLinkServices/debug-service
  subnet:
    reference:
      armId: /subscriptions/0000-00000-000000/resourceGroups/debug-rg/providers/Microsoft.Network/virtualNetworks/debug-vpc/subnets/debug-subnet
status:
  conditions:
  - lastTransitionTime: "2024-04-01T06:40:39Z"
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: Ready
  customNetworkInterfaceName: ""
  etag: W/"a21197e5-51ed-4142-93f3-f8dc27ebdb32"
  id: /subscriptions/0000-00000-000000/resourceGroups/dev-azr-inc-aks-a-rg/providers/Microsoft.Network/privateEndpoints/test
  location: centralindia
  name: luketestpe
  networkInterfaces:
  - id: /subscriptions/0000-00000-000000/resourceGroups/debug-rg/providers/Microsoft.Network/networkInterfaces/test.nic.437f5635-6ee4-4e52-8aef-985b7c4a6017
  privateLinkServiceConnections:
  - etag: W/"a21197e5-51ed-4142-93f3-f8dc27ebdb32"
    id: /subscriptions/3f59ac6a-d708-49ec-9490-a3c786d63455/resourceGroups/dev-azr-inc-aks-a-rg/providers/Microsoft.Network/privateEndpoints/luketestpe/privateLinkServiceConnections/dev-azr-inc-mgmt-aks-demosrc
    name: debug-service
    privateLinkServiceConnectionState:
      actionsRequired: None
      description: Approved
      status: Approved
    privateLinkServiceId: /subscriptions/0000-00000-000000/resourceGroups/debug-rg/providers/Microsoft.Network/privateLinkServices/debug-service
    provisioningState: Succeeded
    type: Microsoft.Network/privateEndpoints/privateLinkServiceConnections
  provisioningState: Succeeded
  subnet:
    id: /subscriptions/0000-00000-000000/resourceGroups/debug-rg/providers/Microsoft.Network/virtualNetworks/debug-vpc/subnets/debug-subnet
  type: Microsoft.Network/privateEndpoints

To Reproduce
Steps to reproduce the behavior:
Create a privatelink through ASO can check its status in the yaml output.

Expected behavior
ipConfigurations can be retrieved so that we can know the IP info of the private endpoint.

[super-harsh]

Hey, the IpConfiguration field is to specify the static ip address (specified by the user).

Although, we do get CustomDNSConfig from ARM for other first party services e.g. StorageAccount, etc which looks like below:

customDnsConfigs:                                                                                                                                                                   
   - fqdn:  asosamplestorage.blob.core.windows.net    
     ipAddresses:                                                                                                                                                                       
     - 10.0.0.4 

However, DNS configuration is not available for private endpoints connected to a private link service. Hence, its not present there. If I check the resource I created in portal, I see Network Interface address if that's what you're looking after to be exposed in the status?

[wjf3121]

Hi @super-harsh

First of all, thanks for following up on this issue!

If I check the resource I created in portal, I see Network Interface address if that's what you're looking after to be exposed in the status?

Pretty much, I would like to get the private ip address of the provisioned endpoint otherwise there's no way for us to connect to it without setting up the private dns.

FWIW, if I use the azurerm_private_endpoint in Azure Terraform provider to provision a private endpoint, I can get the IP through azurerm_private_endpoint.this.private_service_connection[*].private_ip_address.

[super-harsh]

ipConfiguration field in privateEndpoint is used for providing static ip address to be used and is set by the user. You can try adding this field in your spec(as below) when you create a resource and specify an address you'd want to register within subnet. That way, it'll be present in the status.

  ipConfigurations:
    - name: myip
      privateIPAddress: 10.0.0.4

[wjf3121]

ipConfiguration field in privateEndpoint is used for providing static ip address to be used and is set by the user. You can try adding this field in your spec(as below) when you create a resource and specify an address you'd want to register within subnet.

Hi @super-harsh

While this works, it means we need to pre-allocate the IP ourselves before creating the private endpoint. However, in our case, we simply want Azure to auto-assign a private IP for us and we can the the assigned IP after the private endpoint creation. Looks like this is not possible at this point.

[matthchr]

You're correct that this isn't supported currently. The reason why boils down to the fact that the PrivateEndpoint doesn't include any fields describing the IP associated with the PE. Instead the IP information can be retrieved from networking by issuing a request like:
https://management.azure.com/subscriptions/sub-id/resourcegroups/rg/providers/Microsoft.Network/privateEndpoints/pe?api-version=2021-05-01&$expand=networkInterfaces, note the $expand=networkInterfaces. ASO is autogenerated and as such doesn't have knowledge by default about these sorts of custom "tricks" to get extra information about a particular resource.

There are a few ways we can resolve this problem. The one that makes the most sense to me is to add a new field privateEndpoint.spec.operatorSpec.configMaps.privateIPAddress, which you could configure to write the private IP to a configMap of your choosing, like so:

spec:
    operatorSpec:
        configMaps:
            privateIPAddress:
                name: "my-confimap"
                key: "privateip"

Would this meet your needs?

[RSE132]

@matthchr I am facing similar issue where the expectation is ASO (v2.7.0) to create a record in the private dns. I tried using PrivateEndpointsPrivateDnsZoneGroup even though it successfully create the resource but the corresponding a record is not created.

apiVersion: network.azure.com/v1api20220701
kind: PrivateEndpointsPrivateDnsZoneGroup
metadata:
  annotations:
    serviceoperator.azure.com/credential-from: landingzone-aso-credential
    serviceoperator.azure.com/operator-namespace: azureserviceoperator-system
    serviceoperator.azure.com/reconcile-policy: manage
  name: private-sandbox-cluster-private-dns-zone-group
  namespace: landing-zone
spec:
  azureName: private-sandbox-cluster-Private-dns-zone-group
  owner:
    name: mop-prod-we01-logs-private-sandbox-cluster-pe
  privateDnsZoneConfigs:
  - name: mop-prod-we01-logs-private-sandbox-cluster-pe
    privateDnsZoneReference:
      group: network.azure.com
      kind: PrivateDnsZone
      name: private-sandbox-cluster-privatelink.maersk.io

I believe this is because the current kind: PrivateEndpoint cannot populate ipConfigurations. At the same time based on your response where you mentioned that you can populate the IP address in a configMap, I wondering can the ASO operator use this ipaddress and create a record in the privatednszone using the dns sone information from the PrivateEndpointsPrivateDnsZoneGroup

Alternatively can an optional field be added to kind: PrivateEndpoint to provide dns zone information based on which a record could be created on the respective private dns zone ?