Securing service traffic in Azure – Part 2

In “Securing service traffic in Azure – Part 1” I talked about using private endpoints to secure traffic between your Azure services. In this part, I explain another way to secure your traffic between Azure services using service endpoints.

So why not just use private endpoints?

As I demonstrated in Part 1, by configuring a private endpoint, I successfully secured traffic between my function app and my data source, so why would we use something else? The two primary reasons are:

  • Purpose – You should only ever deploy the minimum required services to support your environment. Private endpoints are designed to provide “private” routable access to a service running in Azure. This means traffic from anywhere within your network environment can access this endpoint. If your only requirement is to secure traffic between services in Azure, then the routing capability is not required.
  • Cost – For each private endpoint there is a cost required to run the endpoint, approximately $9USD/month. As usage of endpoints increase, so to do your costs scale and this can add up quickly over time.

If our only purpose is to secure traffic between services in Azure, then this is where service endpoints provide an alternative.

The Test Environment

Like Part 1 of this series, for simplicity, I will continue to use a test environment that consists of:

  • A storage account hosting a CSV file of data.
  • A PowerShell function app based on a http trigger that will retrieve the data from the CSV file.

Securing the storage account with a service endpoint

This time, the first step to securing the storage account is to use the firewall to disable public access and restrict access to selected virtual networks:

In this case I am connecting to the subnet used in Part 1 of this series. Similar to disabling public network access completely, by restricting to specific virtual networks, I have now disabled access from the internet:

and my function app no longer works:

Securing the service traffic in Azure by linking the function app to the storage account

Just like with private endpoints, I need to link the function app to the service endpoint that was created when I restricted the access to specific virtual network subnets. This is done in exactly the same method for private endpoints by enabling “virtual network integration” and linking it to the same subnet used for the storage account.

Now our environment will look like this:

  • The function app is integrated with the virtual network
  • The storage account has a delegation to the same subnet as the virtual network integration
  • Storage access is now only accessible within the Azure fabric and not across the open internet

And now, my function app is again returning data from the file on the storage account:

In part 3, I will demonstrate how to present a protected address with traffic secured between each layer.

Securing service traffic in Azure – Part 1

A question I come across a lot is how to secure traffic between services running in Azure. In this multi-part series, I will look at using two different approaches for securing service traffic in Azure using a Function App and a storage account as my example services.

The test environment

For the purposes of this post, I am going to use a simple function app that communicates with a storage account and retrieves the content of a file. Granted, I could get the content directly from the storage account, but this would not demonstrate the required functionality. Therefore the demo environment will consist of:

  • A storage account hosting a CSV file full of data.
  • A PowerShell function app based on a http trigger that will retrieve the data from the CSV file.
  • And an account with contributor rights to the resource group in which everything is being created.
Function App accessing storage file

When called, our function app will write the contents of the CSV to the browser:

Function app returned data

Concerns about the described environment

Now while our communications are over HTTPS and using access keys (for the sake of this demonstration), concerns are sometimes raised that the communication, encrypted or not, is over open internet. In addition, for this communication to occur over the open internet, our data source (the storage account) requires a public endpoint. All it takes is a simple misstep and our data could be exposed to the internet.

So how do we restrict public access to our source and ensure communications occur over a more secure channel?

Microsoft provide two mechanisms to secure the connection for the data source:

In this post, I am going to focus on private endpoints.

Securing the storage account with a private endpoint

Our first step is to secure the storage account from public access by disabling “Public Network Access” in the Firewall and Virtual Networks tab:

securing the storage account

With this committed, you can see that our function app no longer returns the content of the data file:

Securing Azure storage account prevents services from accessing it

To allow secured access to our data store, I am now going to need to create a private endpoint for the storage account. For this demo, I am going to call the instance “storage-endpoint” and attach it to the blob resource:

and now I need a virtual network to attach the endpoint to. For this demo, I have created a virtual network and added a subnet “privateips” to place private endpoints in:

I am accepting the defaults for DNS:

and then creating the private endpoint. As I accepted the default DNS settings, this will also create a privatelink DNS zone to host the A record for the private endpoint:

I have now successfully secured the blob storage from the internet:

However, as demonstrated earlier, our function app also has no access. I will now show you how to remedy that.

Connecting the function app to the private endpoint

Before we go any further, it is worth noting that the SSL certificate for the storage account is against “apptestrmcn0.blob.core.windows.net” and I have just created a private endpoint with the name “apptestrmcn0.privatelink.blob.core.windows.net“. Any service trying to connect to the private endpoint is going to fail due to a certificate mismatch. Not to worry, if I try resolving the original name of the storage account from an internal host in the same virtual network, you will see that the FQDN also maps to the new private endpoint:

As the storage account has now been isolated to private endpoint traffic only, I need to connect the Function App to the virtual network too. This is achieved via a “VNet integration” which relies on service delegations. Service delegations require a dedicated subnet per delegation type, and I therefore cannot use the same subnet as our private endpoint. For the sake of simplicity, I am going to use a separate subnet call “functionapps” within the same Virtual Network to demonstrate the functionality.

VNET integration is configured via the Networking tab for the function app:

When I select VNET integration, I am going to then select our subnet for function apps:

When Iclick connect, the service delegation will happen in the background if not already delegated and the function app will get a connection to the subnet. In the next window, as our function app’s sole purpose is to fetch data from the blob, I will disable internet traffic for the function app and apply the change.:

This is the last of the configuration. It should look like this:

And as you can see, our function app is, again, able to retrieve data the storage account:

In “Securing service traffic in Azure – Part 2” I will look at the alternative to private endpoints.

Azure firewall basic is now generally available

Firewall

On the 15th March, 2023 Microsoft announced the general availability of Azure firewall basic.

Azure firewall is a cloud native network security service that provides threat protection for cloud workloads running in Azure. It is a stateful service offering both east/west and north/south protection along with high availability and scalability. Azure firewall is available in 3 SKU’s; Standard, Premium and now Basic. All 3 versions provide the following features:

  • Built-in high availability
  • Availability Zones
  • Application FQDN filtering rules
  • Network traffic filtering rules
  • FQDN tags
  • Service tags
  • Threat intelligence
  • Outbound SNAT support
  • Inbound DNAT support
  • Multiple public IP addresses
  • Azure Monitor logging
  • Certifications

And while premium has additional features such as TLS inspection and IDPS, the Basic SKU has the following limitations:

  • Supports Threat Intel alert mode only.
  • Fixed scale unit to run the service on two virtual machine backend instances.
  • Recommended for environments with an estimated maximum throughput of 250 Mbps.

Where Azure Firewall Basic comes into its own is in cost to run. The Basic pricing model is designed to provide essential protection to SMB customers at an affordable price point for low volume workloads.

Approximate Costs

At the time of writing this article, the approximate retail costs for running Azure Firewall are:

SKUCost
Basic$0.592 (AU) per deployment hour
or
$432.16 (AU) per month
Standard$1.871 (AU) per deployment hour
or
$1,365.83 (AU) per month
Premium$2.619 (AU) per deployment hour
or
$1,911.87 (AU) per month
Recommended retail costs for running Azure Firewall

As you can see, Azure Firewall Basic is considerably cheaper than the Standard or Premium SKU’s just to turn on. But as mentioned previously, it is only for small workloads. The processing costs for data through Azure firewall basic are roughly 4 times more expensive.

If we look at processing 100GB in an hour the running costs would look like:

SKUCost per GBProcessing costTotal cost
(inc run cost)
Basic$0.098 (AU)$9.80 (AU)$10.39 (AU)
Standard$0.024 (AU)$2.40 (AU)$4.27 (AU)
Premium$0.024 (AU)$2.40 (AU)$5.02 (AU)
Recommended retail data processing costs

Clearly, sustained high workloads are much more expensive through the Basic SKU as opposed to the Standard or Premium SKU’s. The basic SKU is cost cheaper only when customers are processing less than 9,520GB per month, or 13GB per hour.

Recommendation

The new pricing model provides a much cheaper option for SMB customers to secure essential workloads at an affordable price where data volumes are low.

New enhanced connection troubleshoot for Azure Networking

On the 1st March, 2023, Microsoft announced “New enhanced connection troubleshoot” for Azure Network watcher has gone GA. Previously Azure Network Watcher provided specialised stand alone tools for use with network troubleshooting but these have now been consolidated into one place with additional tests and actionable insights to assist with troubleshooting.

Complex network paths
Network Troubleshooting can be difficult and time consuming.

With customers migrating advanced, high-performance workloads to Azure, it’s essential to have better oversight and management of the intricate networks that support these workloads. A lack of visibility can make it challenging to diagnose issues, leaving customers with limited control and feeling trapped in a “black box.” To enhance your network troubleshooting experience, Azure Network Watcher combines these tools with the following features:

  • Unified solution for troubleshooting all NSG, user defined routes, and blocked ports
  • Actionable insights with step-by-step guide to resolve issues
  • Identifying configuration issues impacting connectivity
  • NSG rules that are blocking traffic
  • Inability to open a socket at the specified source port
  • No servers listening on designated destination ports
  • Misconfigured or missing routes

These new features are not available via the portal at the moment:

connection troubleshoot via portal does not display enhanced connection troubleshoot results
Connection Troubleshooting via the portal

The portal will display that there are connectivity issues, but will not provide the enhanced information. This is accessible via PowerShell, Azure CLI and the Rest API. I will now show the real reason this is not working.

Accessing “enhanced connection troubleshoot” output via PowerShell

I am using the following PowerShell to test the connection between the two machines:

$nw = get-aznetworkwatcher -location australiaeast
$svm = get-azvm -Name Machine1
$dvm = get-azvm -Name Machine2
Test-AzNetworkWatcherConnectivity -NetworkWatcher $nw -SourceId $svm.Id -DestinationId $dvm.Id -DestinationPort 445

This returns the following JSON:

ConnectionStatus : Unreachable
AvgLatencyInMs   :
MinLatencyInMs   :
MaxLatencyInMs   :
ProbesSent       : 30
ProbesFailed     : 30
Hops             : [
                     {
                       "Type": "Source",
                       "Id": "a49b4961-b82f-49da-ae2c-8470a9f4c8a6",
                       "Address": "10.0.0.4",
                       "ResourceId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/CONNECTIVITYTEST/providers/Microsoft.Compute/virtualMachines/Machine1",
                       "NextHopIds": [
                         "6c6f06de-ea3c-45e3-8a1d-372624475ced"
                       ],
                       "Issues": [
                         {
                           "Origin": "Local",
                           "Severity": "Error",
                           "Type": "GuestFirewall",
                           "Context": []
                         }
                       ]
                     },
                     {
                       "Type": "VirtualMachine",
                       "Id": "6c6f06de-ea3c-45e3-8a1d-372624475ced",
                       "Address": "172.16.0.4",
                       "ResourceId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/CONNECTIVITYTEST/providers/Microsoft.Compute/virtualMachines/Machine2",
                       "NextHopIds": [],
                       "Issues": []
                     }
                   ]

As you can see, the issues discovered are explained in more detail, in this case, the local firewall is affecting the communication. If we check the local Defender firewall, we can see there is a specific rule blocking this traffic:

Blocked outbound protocols

If we remove the local firewall rule, connectivity is restored:

ConnectionStatus : Reachable
AvgLatencyInMs   : 1
MinLatencyInMs   : 1
MaxLatencyInMs   : 2
ProbesSent       : 66
ProbesFailed     : 0
Hops             : [
                     {
                       "Type": "Source",
                       "Id": "f1b763a1-f7cc-48b6-aec7-f132d3fdadf8",
                       "Address": "10.0.0.4",
                       "ResourceId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/CONNECTIVITYTEST/providers/Microsoft.Compute/virtualMachines/Machine1",
                       "NextHopIds": [
                         "7c9c103c-44ab-4fd8-9444-22354e5f9672"
                       ],
                       "Issues": []
                     },
                     {
                       "Type": "VirtualMachine",
                       "Id": "7c9c103c-44ab-4fd8-9444-22354e5f9672",
                       "Address": "172.16.0.4",
                       "ResourceId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/CONNECTIVITYTEST/providers/Microsoft.Compute/virtualMachines/Machine2",
                       "NextHopIds": [],
                       "Issues": []
                     }
                   ]

The enhanced connection troubleshoot can detect 6 fault types:

  • Source high CPU utilisation
  • Source high memory utilisation
  • Source Guest firewall
  • DNS resolution
  • Network security rule configuration
  • User defined route configuration

The first four faults are returned by the Network Watcher Agent extension for Windows as demonstrated above. The remaining two faults are from the Azure fabric. As you can see below, when a Network Security Group is misconfigured on the source or destination, our issue returns, but the output displays clearly where and which network security group is at fault:

ConnectionStatus : Unreachable
AvgLatencyInMs   :
MinLatencyInMs   :
MaxLatencyInMs   :
ProbesSent       : 30
ProbesFailed     : 30
Hops             : [
                     {
                       "Type": "Source",
                       "Id": "3cbcbdbe-a6ec-454f-ad2e-946d6731278a",
                       "Address": "10.0.0.4",
                       "ResourceId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/CONNECTIVITYTEST/providers/Microsoft.Compute/virtualMachines/Machine1",
                       "NextHopIds": [
                         "29e33dac-45ae-4ea3-8a9d-83dccddcc0eb"
                       ],
                       "Issues": []
                     },
                     {
                       "Type": "VirtualMachine",
                       "Id": "29e33dac-45ae-4ea3-8a9d-83dccddcc0eb",
                       "Address": "172.16.0.4",
                       "ResourceId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/CONNECTIVITYTEST/providers/Microsoft.Compute/virtualMachines/Machine2",
                       "NextHopIds": [],
                       "Issues": [
                         {
                           "Origin": "Inbound",
                           "Severity": "Error",
                           "Type": "NetworkSecurityRule",
                           "Context": [
                             {
                               "key": "RuleName",
                               "value": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/ConnectivityTest/providers/Microsoft.Network/networkSecurityGroups/Ma
                   chine2-nsg/SecurityRules/DenyAnyCustom445Inbound"
                             }
                           ]
                         }
                       ]
                     }
                   ]

In addition to the fault detection, IP Flow is also a part of the enhanced connection troubleshoot, providing a list of hops to a service. An excerpt of a trace to a public storage account is below:

PS C:\temp> Test-AzNetworkWatcherConnectivity -NetworkWatcher $nw -SourceId $svm.Id -DestinationAddress https://announcementtest.blob.core.windows.net/test1 -DestinationPort 443

ConnectionStatus : Reachable
AvgLatencyInMs   : 1
MinLatencyInMs   : 1
MaxLatencyInMs   : 1
ProbesSent       : 66
ProbesFailed     : 0
Hops             : [
                     {
                       "Type": "Source",
                       "Id": "23eb09fd-b5fa-4be1-83f2-caf09d18ada0",
                       "Address": "10.0.0.4",
                       "ResourceId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/CONNECTIVITYTEST/providers/Microsoft.Compute/virtualMachines/Machine1",
                       "NextHopIds": [
                         "78f3961c-9937-4679-97a7-4a19f4d1232a"
                       ],
                       "Issues": []
                     },
                     {
                       "Type": "PublicLoadBalancer",
                       "Id": "78f3961c-9937-4679-97a7-4a19f4d1232a",
                       "Address": "20.157.155.128",
                       "NextHopIds": [
                         "574ad521-7ab7-470c-b5aa-f1b4e6088888",
                         "e717c4bd-7916-45bd-b3d1-f8eecc7ed1e3",
                         "cbe6f6a6-4281-402c-a81d-c4e3d30d2247",
                         "84769cde-3f92-4134-8d48-82141f2d9bfd",
                         "aa7c2b73-0892-4d15-96c6-45b9b033829c",
                         "1c3e3043-98f2-4510-b37f-307d3a98a55b",
                         "b97778cb-9ece-4e87-bf6d-71b90fac3847",
                         "cb92d16d-d4fe-4233-b958-a4d3dbe78303",
                         "ec9a2753-3a60-4fce-9d92-7dbbc0d0219d",
                         "df2b1a3e-6555-424c-8e48-5cc0feba3623"
                       ],
                       "Issues": []
                     },
                     {
                       "Type": "VirtualNetwork",
                       "Id": "574ad521-7ab7-470c-b5aa-f1b4e6088888",
                       "Address": "10.124.144.2",
                       "NextHopIds": [],
                       "Issues": []
                     },
                     {
                       "Type": "VirtualNetwork",
                       "Id": "e717c4bd-7916-45bd-b3d1-f8eecc7ed1e3",
                       "Address": "10.124.146.2",
                       "NextHopIds": [],
                       "Issues": []
                     },

Centralising the troubleshooting tools under one command is obviously a great enhancement, but by also providing increased visibility into configurations or system performance make this a great update for your troubleshooting toolbox.

Azure Premium SSD V2 disks

Azure Premium SSD V2 disks generally available

Microsoft have recently announced the release of Azure Premium SSD V2 disks as generally available for select regions, but what does this mean for you?

Azure Premium v2 disks uncouple IOPs and throughput metrics from disk size, meaning you can adjust the IOPS, throughput, and capacity according to your workload needs. Premium SSD v2 disks are designed to handle performance-sensitive and general-purpose workloads that require low average read and write latency, high IOPS, and high throughput. Whereas with Azure Premium and Azure Ultra disks, IOPS and throughput are fixed by disk size. This can make Premium SSD V2 disks an efficient and cost-effective option for running and scaling transaction-intensive workloads.

Achieving High IO loads with Premium SSD

Assuming you have a highly transactional 100GB database generating a sustained load of 10,000 IOPs with a throughput of 200MB / sec and an appropriate compute configuration to support this. To achieve this with Premium SSD disks would require 4 x P20 disks or 2 x P30 disks presented to your compute and striped using a volume set. This however would also see overkill in both available space and data transfer.

4 x P202 x P30
Size2,048GB2,048GB
IOPs9,200 IOPs10,000 IOPs
Throughput600 MB/sec400 MB/sec
Achieving High IO loads with Premium SSD

Note that the above configuration only takes into account performance and does not address any concerns such as redundancy and data protection.

Achieving High IO loads with Premium SSD V2 disks

As V2 disks are configurable across all 3 parameters, the allocated disk is much more aligned with the need of the workload. Also, all three parameters can be adjusted as workload parameters change through growth. For the example above, a Premium SSD V2 disk would look like:

RequiredIncludedAdditional
Size100GBn/an/a
IOPs10,0003,0007,000
Throughput (MB/sec)200 12575
Achieving High IO loads with Premium V2 SSD

According to the Microsoft article for Managed Disks Pricing, this configuration would represent a significant saving in cost for the disk as well as reduced overhead. Like the previous example, this example only takes into account performance and does not address any concerns such as redundancy and data protection.

Current limitations of Azure Premium SSD v2 disks

As of the writing of this article, there are some limitations to Premium v2 disks. Firstly, they are only available in the following regions:

  • US East
  • West Europe

The other limiting factors are:

  • Disks are only available on Locally Redundant Storage.
  • Snapshots are not supported
  • Encryption capabilities are very limited
  • Azure Backup / Site Recovery aren’t supported for VM’s with V2 disks

I would expect these limitations to change over time.

Why Azure Premium SSD v2 disks should be on your radar

Current Premium SSD Disk configurations are fixed in their Size/ IOPs/Throughput ratios and therefore require additional overhead and creativity to achieve specific workloads for high performing systems and usually come with wasted resources that you are paying for.

With the introduction of Premium SSD V2 disks, you can have better control over your configuration and therefore only pay for what you need. As reductions in operating costs and management overhead is something that benefits all users and should be something watched for in your Azure deployment region.