Azure IAM Anti-Patterns: Avoid Putting Your Data at Risk
Identity and Access Management (IAM) is the backbone of a secure cloud environment. Get it wrong, and your data, pipelines, and data platform are at risk. Here's how to avoid the most common mistakes.
Read time: 9 minutes
1. Introduction
In our last post, we walked through the core concepts behind a well-organized Azure environment by structuring our environment with Management Groups, Subscriptions, and Resource Groups. Building on that foundation, we now need to think about how we secure our environment - protecting resources and data from external threats and internal mishaps. In this article, we’ll zoom in on one of the most important aspects of cloud governance: Identity and Access Management (IAM).
If you’ve been a data engineer for a while, you’ve likely run into one of two extremes:
Being blocked from doing your job because you don’t have sufficient permissions to the resources you need.
Being given overly broad permissions, exposing your organization to unnecessary risk.
Proper IAM implementation is all about striking a balance. Done well, it enables each type of user to work effectively while ensuring the environment stays secure and compliant.
When thinking about how to explain IAM, I find it helps to frame it as:
Who: The person, group or service that needs access to a resource.
How: The way they prove their identity (authentication).
What: The specific actions they are allowed to perform (authorization).
Why: The business or security reason for granting that access.
I find this lens helps to cut through the jargon. When I create or evaluate an IAM solution, I do so by answering these four questions. In this article, I’ll use this framing to introduce the core concepts of IAM, highlight common mistakes, and show how to apply best practices using a real-world example.
2. Core Concepts
Before we get into specific mistakes engineers make with IAM, I want to introduce core concepts that underpin any identity strategy. These principles are not just security buzzwords, they’re practical guardrails that determine how people and services interact with Azure resources. By understanding these concepts, you’ll be able to recognize when IAM is applied correctly, spot gaps in your own environment, and build solutions that follow Identity and Access best practices.
Authentication and Authorization
I like to use the analogy of boarding a plane when explaining authentication and authorization.
Authentication is the process of proving you are who you claim to be. At an airport, this is like showing your passport at the security checkpoint. The agent checks your ID to confirm you are the person on the flight ticket.
Authorization is the process of determining what you are allowed to do. After you’ve passed the security checkpoint and verified your identity, you receive a boarding pass. The boarding pass specifies what you are allowed to do: the plane you can board and seat to sit in. You can’t board a different flight!
Principle of Least Privilege
Grant the absolute minimum level of access and permissions necessary to perform the required actions, and nothing more.
Zero Trust Model
Assume no user, device, or application is trusted by default. Require strict verification of identity for every access request before granting access. Never trust, always verify.
Role-Based Access Control (RBAC)
Permissions are assigned via roles (e.g. Storage Blob Data Contributor). These roles are attached to security groups and users inherit permissions via group membership.
Managed identities
Services also need access. Instead of hardcoding secrets, Azure issues resources their own identity, a Managed Identity. This allows secure access to other services without needing credentials to be stored in code.
Audibility and Traceability
Every action should be attributable to a specific identity. By using individual accounts and group-based roles, you can see exactly who did what, when. This is essential for compliance and incident response.
3. Common IAM Anti-Patterns for Data Engineers
Now that we’ve covered the core concepts, let’s look at common examples of how they’re violated in day-to-day work. Avoiding these anti-patterns is one of the most effective ways to improve your data platform’s security posture.
Each mistake usually stems from prioritizing short-term convenience over long-term security and stability. As data engineers, we need to get better at avoiding the path of least resistance!
3.1 Overly Permissive Roles
This is the most frequent anti-pattern: granting excessive permissions beyond what’s necessary. For example, giving an ETL job’s service account administrator-level access to a data lake when it only needs to read from one location and write to another. This shortcut is taken to avoid friction as figuring out the exact minimum permissions can be time-consuming. It feels quicker to request a broad role like Contributor once than to get blocked and go through the approval process again for missing permissions.
The Security Risk
If these credentials are compromised, an attacker can read, modify, or delete vast amounts of data. Even a well-meaning (or not properly trained) team member with broad permissions could accidentally delete production data.
The Secure Solution
Adhere to the Principle of Least Privilege by creating and assigning granular roles that grant only the precise permissions required for the intended actions.
Example
A common mistake is to assign an Azure Data Factory instance the Contributor role over an entire resource group. This allows it to manage virtual machines, networking, and other unrelated resources. This is a direct violation of the Principle of Least Privilege, a failure of Authorization, and while it does use RBAC, it ignores the granular control available.
Instead, assign the Storage Blob Data Contributor role on the specific Azure Storage Account so it can read and write only where needed.
3.2 Hardcoding Credentials
Hardcoding means embedding static access keys, connection strings, or passwords in source code or configuration files. It often happens as it’s the quickest way to make a connection work. Setting up a secure alternative like Azure Key Vault or a Managed Identity requires extra steps that can feel like a detour when facing a tough deadline.
The Security Risk
Secrets become visible in version control systems (like Git), CI/CD logs and packaged applications. Anyone with read access to the codebase can get the credentials and access your resources. Rotating keys also becomes a nightmare as every instance must be found, changed, and redeployed.
The Secure Solution
Never store secrets in code. Use Azure Key Vault to securely store and retrieve credentials at runtime. For Azure resources, use Managed Identities, which provide temporary, automatically-rotated credentials to the resource itself.
Example
A developer pastes a Storage Account’s connection string directly into an Azure Databricks notebook. This creates a massive weak point for Authentication and explicitly violates the Zero Trust Model by storing secrets in plain text, implicitly trusting the environment and anyone with code access.
The fix: store the connection string in Azure Key Vault and have the Databricks notebook retrieve it using a Managed Identity that has been granted access to that one specific secret.
3.3 Shared Accounts
Multiple engineers (sometime even entire teams) share one IAM user or service account. An entire engineering team sharing a single administrative logic often stems from a desire for simplicity or bad legacy habits where managing individual accounts was considered administrative overhead.
The Security Risk
This destroys accountability. When something goes wrong (like a production table being dropped), it’s impossible to trace the specific individual or service responsible for the action. Auditing, incident response, and forensics are nearly impossible.
The Secure Solution
Enforce individual accountability. Every user must login with their own Azure Active Directory (AD) account and every service or pipeline should have its own unique Service Principal or Managed Identity.
Example
An entire data engineering team shares a single SQL administrator login, like sql-admin, to manage a production Azure SQL Database. A critical table is accidentally deleted with a DROP TABLE command and audit logs only show that sql-admin performed the action. This completely breaks Audibility and Traceability. While the shared account can be authenticated, you lose the ability to truly authorize or audit the actions of the individual behind it.
By using Azure AD integration every action in the database audit log is tied to a specific, identifiable user account, providing clear accountability.
3.4 Not Auditing and Rotating Keys
For a new project, access keys and IAM roles are created once and forgotten about - never reviewed, rotated, or removed. This is the classic set it and forget it mistake. This is especially dangerous because stale credentials still work, you just don’t notice them. Auditing and rotating keys is seen as administrative burden and easily deprioritized.
The Security Risk
Over time, privilege creep causes permissions to accumulate, and long-lived, static keys become a significant risk. It can provide an attacker with a persistent and often undetected backdoor into your systems.
The Secure Solution
Implement lifecycle management for all credentials. Conduct regular access reviews to prune unnecessary permissions, rotate keys automatically, and enforce strict expiration policies.
Example
A CI/CD Service Principal has a client secret set to never expire. Two years later, the pipeline has been decommissioned, but the secret still exists, representing a forgotten and unmonitored entry point. Stale credentials pose a severe Authentication risk. It directly violates the Zero Trust Model's principle of ‘always verify’ by allowing a state of permanent, un-audited trust. Effective Audibility isn't just about logging; it's about using those logs to maintain a healthy RBAC implementation.
Use Azure AD Access Reviews, enforce expiration, and rotate keys automatically with Azure Functions or similar tooling.
3.5 Sprawl and Lack of Documentation
Without proper strategy, teams end up accumulating dozens of poorly named, undocumented, and overlapping custom roles. As new projects start, engineers create one-off roles for specific tasks and the environment becomes littered with roles like DataTeadRead, DataTeamReadOnly, and ADF-Temp-Role.
Security Risk
Role sprawl makes effective governance impossible because no one knows which roles are safe to assign, modify or delete. Engineers either create new, redundant roles or they assign overly permissive ones because they’re unsure what existing roles do. This can lead to massive privilege creep, making an unauditable attack surface where forgotten roles provide unknown levels of access.
Secure Solution
Treat IAM roles like application code. Maintain a central, documented catalog of custom roles, ideally in a version control system; I call this Roles-as-Code. Use built-in roles whenever possible and enforce a clear process for creating, documenting, and retiring custom roles.
Example
An engineer needs read access to a production data lake but unclear roles like DataTeamRead and DataTeamReadOnly are confusing and no-one knows what they do. To be safe, they create a brand-new custom role, worsening the mess by adding another redundant role to the pile. This chaotic state completely undermines Audibility and makes it impossible to maintain the Principle of Least Privilege.
Migrate custom role definitions into a central Git repository. Each role definition includes a description, a designated owner, and a justification. Before creating a new role, an engineer must first check the repository for an existing one that meets their needs. The team also schedules a quarterly review to audit and prune unused or redundant roles from the catalog, avoiding sprawl and improving traceability.
4. Conclusion
Effective Identity and Access Management (IAM) is a foundational pillar of a secure, well-governed data platform. It marks the shift from simply building data systems that work to architecting solutions that are secure, scalable, and governable. As we've seen, getting IAM right isn't about memorizing every role in Azure; it's about internalizing a set of principles like Least Privilege and Zero Trust, and applying them consistently.
Key Takeaways:
Ensure the right people and services have the right access at the right scope.
Embrace Core Principles: Design with a Principle of Least Privilege and a Zero Trust mindset. Use Role-Based Access Control (RBAC) to manage permissions.
Avoid Anti-Patterns: Steer clear of granting overly permissive roles, hardcoding secrets in your code, using shared accounts, or letting credentials go stale.
Leverage Modern Tools: Use Managed Identities to eliminate hardcoded credentials for Azure services and store all other secrets in Azure Key Vault.
Establish Governance: Maintain a clean, documented set of roles (Roles-as-Code) and conduct regular access reviews to ensure permissions remain relevant.
Focus on audibility and accountability for compliance and incident response.
Your next step doesn't have to be a complete overhaul. Start small by using a Managed Identity instead of a connection string, or reviewing the permissions on one of your key Service Principals. Each small, deliberate step you take to improve your IAM practices builds a more robust and trustworthy data platform.
If you’re unhappy with how your Azure environment is running, reach out to us here to see how we can help. Two of our core services include Azure environment audits and building data/AI solutions.
Thanks for reading! Feel free to follow us on LinkedIn here and here. See you next time! To receive new posts and support our work, become a free or paid subscriber today.
If you enjoyed this newsletter, comment below and share this post with your thoughts.