The Top 5 Networking Patterns Data Professionals Should Know (Microsoft Azure)
Learn when to use public endpoints, service endpoints, private endpoints, and the hub-and-spoke by balancing tradeoffs between security, cost, and complexity. Check out the Network Maturity Model!
Read time: 8 minutes
In this article we introduce and explain the five most relevant networking patterns for data platform design. We walk through increasingly more mature solutions for protecting your data and Azure resources, and discuss when to use each by discussing the tradeoffs between security, cost, and complexity.
Introduction
Most data engineers live and breathe data pipelines and transformation. You can probably build a new ETL pipeline while half asleep but if your data and platform are exposed to the public internet then you’re not building production-ready solutions.
In the current cloud-centric world, understanding network architecture and patterns is no longer just for network engineers. As data professionals, it’s now a fundamental skill that separates a quick prototype from an enterprise ready data platform. Understanding both Identity and Access Management and networking will empower you to build data platforms that are protected against data breaches, compliances failures, or becoming a chaotic and expensive cloud environment.
If you’re new to the topic of networking, I suggest checking out our article covering Azure Fundamentals before continuing with this article.
This articles walks through the 5 networking patterns that I think every data engineer should know:
Public Endpoints and Firewalls
Service Endpoints (basic security)
Private Endpoints (the enterprise standard)
Hub-and-Spoke Architecture (scaling your network solution)
Hybrid Connectivity (combining your on-premises and Azure environments)
We’ll cover when to use them, trade-offs to be aware of, and practical examples that you can apply to your own projects.
1. Public Endpoints and Firewalls
The most straightforward way to connect to an Azure service, like an Azure SQL Database or an Azure Storage Account, is through its public endpoint. A data engineer might connect directly from their laptop to an Azure service in order to test a new idea or build a quick proof of concept. This is the quick and dirty method.
This approach is inherently insecure as it exposes the Azure resource to the public internet meaning people can access your data and even use the Azure resources you are paying for.
To add a basic layer of security and make it so that not just anyone can connect to your Azure resources, you can configure firewall rules that restrict access to a specific set of IP addresses, by creating an IP allowlist. This helps to ensure that only authorized users, such as those on corporate network, can connect to the service. See Figure 1.
Pros: Easy to set up and let’s you quickly test pipelines.
Cons: All traffic travels over the public internet, making it hard to enforce enterprise-level security policies. This introduces a security risk if not managed carefully and creates an overhead of having to maintain the IP allowlist.
This pattern is suitable for development and prototyping but shouldn’t be used for production workloads!
2. Service Endpoints (basic security)
When you need to move beyond IP allowlists, the first and most common step is to use Service Endpoints. These provide a secure and direct connection between your Virtual Network (VNet) and supported Azure services such as Azure Storage or Azure SQL Database.
By enabling a Service Endpoint, you ensure that all traffic to the Azure service originates from your private network and exclusively travels over the secure Microsoft Azure backbone, never touching the public internet. See Figure 2.
Pros: Significantly enhances security by keeping traffic off the public internet. It’s relatively simple to configure and is a cost effective way to secure PaaS Services.
Cons: Access is limited to resources within a specific VNet (and peered VNets). It doesn’t provide a private IP address for the service, and it is less flexible than Private Endpoints for more complex scenarios like connecting from on-premises networks. This doesn’t support cross-region scenarios well.
This pattern is ideal for securing PaaS services in straightforward, single-region architectures where simplicity is a priority.
3. Private Endpoints (enterprise standard)
When your workloads handle sensitive data, even the security provided by Service Endpoints isn’t enough, and we need to use Private Endpoints. Private Endpoints assign a private IP address from within your own Virtual Network (VNet) directly to the Azure service, such as an Azure Storage Account or Azure SQL Database.
This let’s us pull the associated service off of the public internet by making it a part of our private network. All traffic to/from the service is routed through this private IP, meaning it is secure by default and completely inaccessible from the public internet and any malicious actors. See Figure 3.
Setting up Private Endpoints is more complex than other methods because it requires careful integration with you DNS (Domain Name System) to ensure that traffic is correctly routed to the private IP address within our VNet, and not the public one.
Pros: The highest level of security as it disables public internet access entirely. It allows for granular network control and direct connections from on-premises networks via VPN or ExpressRoute.
Cons: The DNS requirement makes this more complex to configure. This method is more expensive than using Service Endpoints because of the additional resources (like network interfaces) needed.
This pattern is the standard for production environments handling sensitive data like financial records, PII, or healthcare information.
4. Hub-and-Spoke Architecture
As an organization grows, managing dozens of individual, isolated VNets for different projects becomes unfeasible. This typically results in expensive, chaotic environments. The Hub-and-Spoke architecture solves this by organizing your network into a centralized model that is secure, scalable, and cost-effective.
In this architecture, a central Hub VNet acts as the single point of control. It contains all your shared services, such as your primary firewall, logging and monitoring tools, and gateways for on-premises connectivity (see Figure 4). Individual Spoke VNets are then connected to this Hub with each containing resources for a specific workload or team.
Here’s the important bit all traffic between spokes, or to and from the internet, is forced to pass through the central Hub. This central port of call allows you to inspect and manage all traffic from a single, powerful firewall, ensuring consistent policy enforcement across the entire organization. Using a Hub-and-Spoke pattern lets you give individual teams the autonomy to manage their own spoke environments whilst having centralized governance.
Pros: Centralized security and control with significant cost savings by consolidating expensive resources like firewalls. Clear separation of concerns between central IT and teams working on different workloads.
Cons: More complex to set up initially and requires careful planning of your IP address space to avoid overlaps between the hub and all connected spokes.
This pattern is the standard for enterprise-scale deployments where multiple teams or business units are building solutions in Azure.
5. Hybrid Connectivity (On-Prem + Azure)
Although cloud is the established standard, many organizations still have critical data sources running in their on-premises datacenters, such as legacy SQL Server databases, SAP systems, or file shares. As a data engineer, you’ll often be tasked with building pipelines that securely pull this data into Azure for data integrations and modern analytics. Hybrid connectivity patterns act as your bridge to your on-premises network with your Azure VNet, effectively making the cloud a secure extension of your datacenter.
There are two ways to achieve this. A VPN Gateway creates an encrypted tunnel over the public internet, a cost-effective solution for smaller-scale needs. For mission-critical workloads that require high bandwidth and reliability, ExpressRoute provides a dedicated, private fiber connection directly between your datacenter and Microsoft's global network.
Understanding these two technologies and when to use them is an important skill for data engineers as data migrations and replications will always be in-demand!
Pros: Enables secure access to on-premises data sources from Azure as if they were in the same network. It facilitates data migrations to the cloud.
Cons: Complex to set up and requires close collaboration between cloud and on-premises networking teams. Express route can be particularly expensive.
This pattern is a necessity for any enterprise with a significant on-premises data platform that is beginning it's cloud journey.
Putting It All Together: Network Maturity Model
The networking patterns we've discussed are not an all or nothing choice. Instead, they represent a maturity model that a data platform can evolve through over time.
You don’t need a fortress for a simple proof of concept. The key is to select the pattern that matches the security, compliance, and scale requirements of your workload at its current stage whilst understanding the path forward as your needs change.
Starting a proof of concept? Use Public Endpoints with a Firewall
Your development phase involves fast paced iteration and you want to access Azure services from your laptop to test a new idea. By using a firewall to your IP address, you get a basic layer of security without too much complexity.
Moving your workload to production? Start with Service Endpoints
The first major step to securing your data platform. This lets you ensure traffic between your VNet and your PaaS services travels exclusively over the private Azure backbone. It’s a simple and cost effective way to take your data off the public internet.
Handling sensitive data or scaling across teams? Adopt Private endpoints with the Hub-and-Spoke Model.
The enterprise standard makes Azure resources part of your private network and lets you disable public access entirely. This architecture gives you centralized control over networking traffic and security policies whilst giving teams autonomy in their own spoke Virtual Networks.
Integrating with on-premises infrastructure? Implement Hybrid Connectivity
Whether you’re migrating legacy data or building a solution that spans your datacenter and the cloud. To make your Azure environment a true extension of your existing network you’ll need a VPN Gateway or ExpressRoute.
Choosing the right networking pattern is about making informed tradeoffs between security, cost, and complexity. My advice is to start with the simplest pattern that meets your immediate needs and evolve towards maturity from there. Don’t over-engineer your initial design.
Thanks for reading! Feel free to follow us on LinkedIn here and here. See you next time! To receive new posts and support our work, become a free or paid subscriber today.
If you enjoyed this newsletter, comment below and share this post with your thoughts.