In the admittedly short time I’ve been a data analyst at The Data School, and even before that, I’ve been far more engaged by the back-end technical side of data.

My earliest projects involved extracting my personal data to load or analyse in different ways. I’m a programmer by nature and my curiosities find me getting my hands dirty with pretty technical stuff.

My approaches to project work usually involve a bit of SQL and Python with a focus on moving and transforming data from source to destination.

I’ve known this for a while but after 13 weeks of intense training I discovered that data engineering is a strength that makes me unique amongst my colleagues.

In the next two years of my time with The Data School I want to go all-in and set myself up to make Data Engineering the next stage of my career.

With that in mind, what is a Data Engineer and how are they different to a Data Analyst?

In this blog post I want to explore the similarities and difference between the two roles, and how they collaborate to generate insights.

Data Engineering and Data Analytics are two different fields that both deal with data; though in different ways.

  • Data Engineers focus on developing and maintaining systems for collecting and storing data, as well as creating pipelines to Extract, Transform, and Load data from various sources to a relevant destination.
  • Data Analysts generally, but not always, work on datasets that have been prepared for analysis and focus on generating insights that guide business decision making through visualisations and reports.

Here are a few of the key differences between Data Engineers and Data Analysts:

  1. Data Engineers are focused on the technical side of data; managing, building and maintaing date pipelines and infrastructure. Data Analysts on the other hand are focused on finding stories, and insights in data to guide individuals and corporations in their decision making.
  2. Data Engineers are responsible for designing and implementing systems for collecting, processing and storing data in databases or data warehouses. Data Analysts focus on using data stored in databases and data warehouses to interpret data and identify trends or patterns.
  3. Data Engineers often work with large amounts of raw structured and unstructured data, preparing them in a way that enables efficient storage or analysis to enable other functions within the organisation. Data Analysts are also trained in mining unstructured data although their outputs are designed to enable business decision making functions.
  4. Data Engineers are typically focused on the “backend” of data management, whereas Data Analysts focus more on the “frontend” analysis of data.
  5. Data Engineers use tools that are more similar to developers, with a focus on coding languages such as Python and SQL, as well as big data frameworks such as Apache Spark. Data Analysts on the other hand use visualisation software such as Tableau, and PowerBI, as well as low-code/no-code tools such as Alteryx.
  6. Data Engineers and Data Analysts both commonly use databases and are usually proficient in SQL. However, Data Engineers are more involved in the lower level creation and administration of databases whereas Data Analysts mostly query existing databases.

There is a large amount of overlap in the roles of Data Engineers and Data Analysts in the data ecosystem, although there is a huge difference in their approaches.

In my opinion, both professions should have some fundamental knowledge in what the other does as both must work closely together on projects.

I personally love solving the kind of problems that Data Engineers tackle on a daily basis, and while I enjoy creating stunning visualisations that tell stories and drive insights; I can’t seem to shake the desire to get my hands dirty with the technical side.

Whilst I will continue to pursue Data Engineering, I don’t want to lose sight of the importance of Data Analytics because I love being able to make data answer questions and tell stories.

Until next time, stay tuned for more content delivered direct from my keyboard, and think about the different roles that you can play in our wonderful world of data.

Love,

Dan Lawson

Daniel Lawson
Author: Daniel Lawson

Right off the bat I can tell you that I’m not your average data analyst. I’ve spent most of my career running my own business as a photographer and videographer, with a sprinkling of Web Development and SEO work as well. My approach to life and work is very T-shaped, in that I have a small set of specific skills complemented by a very broad range of interests; I like to think of myself as a dedicated non-specialist. Data Analytics, and Programming, started as a hobby that quickly grew into a passion. The more I learned the more I looked for opportunities to pull, manipulate, and join data from disparate sources in my life. I learned to interact with REST APIs for services I used, personal data from services I use like Spotify, and health data captured by my devices. I learned SQL to create and query databases, as well as analyse SQLite files containing my iMessages and Photos data on my Mac. Every technique I learned opened up more possibilities; now I’m hooked and there’s no turning back. Learn More About Me: https://danlsn.com.au