A Month in the Life of a Data Platform Engineer in Aviation
In the past month I have worked on a new production environment whilst keeping existing data integrations healthy. This is a real look at what data engineering in aviation actually involves.
Read time: 6 minutes
Instead of a deep-dive on a single topic, I wanted to write something in the spirit of those “day in the life” videos popular some 10 years ago - an honest look at the variety of work that fills my month. I’m a data platform engineer working in the aviation domain where I architect and build a centralized data platform on Microsoft Azure. Our platform supports mission-critical operations across multiple airports, each with its own business teams and executive stakeholders.
The job market is very competitive right now, and I’m fortunate to have consistent remote work that’s technically challenging and varied. No two months are the same and this one is a good example of the range of problems I get to work on.
This month had two distinct threads running in parallel:
A large one-off project focused on building out a proper production environment
Business-as-usual data platform work across multiple workstreams
Let’s get into it.
Deploying a Proper Dev/Prod Environment
I started working with this client in Q4 2025. From day one it was obvious they’d built a genuinely impressive business intelligence capability and moved fast… But some best practices had fallen by the wayside. They’d been operating out of a single Synapse Workspace, using separate databases for dev and prod rather than separate workspaces entirely.
This month I continued a large effort to create a brand new Synapse Workspace that will become the dedicated production environment. That sounds simple, but the deployment involved:
Standardizing the release process for promoting solutions from dev to prod
Replicating SQL pools schema and data for dedicated and on-demand
Replicating Blob storage accounts in our prod subscription
Replicating Synapse objects across environments
With this foundation in place, we’re now running robust quality assurance and quality control tests between the two environments so we can officially migrate all consumer-facing data products to use the new production warehouse as their source.
Owning the deployment process end-to-end means being accountable for everything that breaks during the migration. Things always break. But with true environment separation in place, we can now safely build new data integrations in dev without worrying about displacing production datasets and downstream consumers. This pattern is a standard best practice that we are now putting into place.
Navigating Private Networks for Power BI
Alongside deploying the new production environment, we’ve been auditing it to ensure it meets the client’s updated internal security policies. This has meant wading into regulated networking configurations, private endpoints, and locked-down infrastructure.
One specific task focused on reconfiguring our Power BI reports to route through a Network Gateway to access a database inside a private Synapse workspace. It works now, but getting there required carefully managing a brief service disruption and keeping stakeholders informed throughout.
My approach here is to always communicate early and often. I’d rather we send one message too many than have an executive discover an outage themselves. A silent disruption erodes trust. But a well-communicated one where you’ve flagged the risk, explained the why, and confirmed remediation plan (with timeline) actually goes a long way in building trust.
Working in aviation reinforces this: the tolerance for unexplained failures is very low.
Building a Pipeline Monitoring Solution
We run a complex pipeline ecosystem; cloud and on-premises resources, internal and external systems, spanning the full data lifecycle from ingestion through to reporting.
The honest trigger for this was frustration. Pipeline failures were only being caught when a developer manually checked the Synapse Monitor blade. We had no proactive visibility. Off-the-shelf tools exist, but we needed something that fit our ways of working across an integrated team of consultants, data scientists, and project managers.
So I built a custom monitoring solution in Python which is hosted as a serverless Azure Function App. It pulls pipeline run data from the Synapse API and sends an automated daily email summarizing failed pipelines and key statistics from the previous 24 hours.
Every morning in standup, we go through that email together. Failed pipelines become user stories with the priority determined by severity. It’s a small tool (that didn’t take long to build) but has had disproportionately large impact on how our team operates.
New Data Integrations
Our team is always balancing ensuring that existing data flows remain healthy with building new data integrations. We deal with all sorts of data sources, everything from structured data served via API, temporal semi-structured data served via vendor managed SFTP servers through to unstructured survey data dropped in Blob storage.
For every new data integration, I start with understanding the business case, architecting the technical solution and then connecting, and aligning with both internal consumers and external vendor before writing any code. Getting those conversations right early is the ultimate separator between successful and failed delivery.
In terms of time, a new integration typically breaks down like this:
Requirements capture - 20%
Architecture design - 20%
Implementation - 30%
QA/QC - 30%
This split might not work for everyone but for me, the focus is always on understanding the use case and definition of good upfront.
Getting to wear both the data architect and data engineer hats and working closely with our data quality analyst is one of my favorite parts of the job. I add the most value by sitting at the interface of system design, business problems, and stakeholders management.
Performance & Cost Management
As data operations scale, two things demand constant attention; performance and cost management.
Performance management
A cloud environment has a lot of moving parts: integration runtimes, gateways, dedicated and serverless databases. Each needs to be sized and scheduled appropriately. I’ve spent significant time this month thinking about when jobs run relative to when the data gets consumed, so we can load-balance throughout the day rather than hammering resources all at once.
Cost management
Splitting dev and prod into standalone workspaces means more pipelines running and more spend. I regularly audit job schedules and compute sizing to make sure nothing is over-provisioned. This covers everything from SQL database sizing and Spark pools to Synapse integration runtimes and data storage tiering within our ADLS accounts.
On the security side, formalizing our dev/prod separation prompted a full review of secrets management. I have been creating Key Vaults, updating private network integrations, and configuring DNS zones so credentials never leave our private networking boundary. I’ve been writing lots of documentation to capture all of these changes into Standard Operating Procedures so that our team can scale safely with our defined set best-practices.
The cost audit and optimization work this month reduced projected spend signfincantly. At this scale, that’s a meaningful number and it’s the kind of outcome that’s easy to overlook because it shows up as money not spent.
Wrapping Up
My key takeaway after writing this article is just how much data engineering work is invisible when it goes well:
The Power BI reports load.
The migration completes cleanly.
The monitoring email lands in inboxes and no one has to chase anything down.
A lot of data engineering is building the systems and solutions so that others can do their jobs without friction.
Working in aviation adds a layer of seriousness to all of it. These aren’t vanity dashboards. The data flowing through this platform supports real operational decisions across multiple airports. Especially important given recent world events.
If you work in data engineering or cloud infrastructure, I’d love to hear what your month looked like in the comments. If you’re curious about any of the technical details above, feel free to reach out to me on LinkedIn.
Thanks for reading! Feel free to follow us on LinkedIn here and here. See you next time! To receive new posts and support our work, become a free or paid subscriber today.
If you enjoyed this newsletter, comment below and share this post with your thoughts.