As a (primarily) backend developer, I’ve spent a lot of time working with APIs, databases, and application logic—but data engineering operates on a different scale. A scale that is honestly kind of intimidating. Instead of focusing on individual transactions, it’s about designing efficient pipelines, managing large datasets, and working with distributed systems. At least, I think that’s what it’s about—I’ll figure it out as I go and document the process here.
This blog isn’t a step-by-step guide or a definitive roadmap. Instead, it’s a collection of lessons, challenges, and insights as I explore data engineering. I’ll cover key concepts, compare backend development with data engineering, and dive into tools like Airflow, FiveTran, and data warehouses. Along the way, I’ll build and break things, troubleshoot issues, and work on building an understanding of this field.
If you’re a dev curious about data engineering—or just someone interested in learning through experience—I hope this blog gives you useful insights, or at least a relatable journey. Let’s see where this path leads!