What is the difference between an Analytics Engineer and a Data Engineer?
Comparison between cooking data and plating it up ready for serving
This is part of a series where we compare professions. We interviewed Harrison Ford (analytics engineering) and Daniel Baron (data engineering) to find out more!
What made you want to get into analytics engineering (AE) / data engineering (DE)?
Harrison: I became an AE organically — I began my career in statistics/analytics, then by taking on problems outside the scope of my role I found I had an interest in data modelling and warehousing. When I started my career in data the term AE was unheard of and I became one by gravitating towards interesting problems and then solving them.
Daniel: I’ve always enjoyed solving problems and had a curiosity of things I don’t understand. Having strengths in maths & science led me to completing a computer science degree, during which I found DE appealing to me due to it always being on the cutting edge of what technology can do in a commercial setting.
What does your typical day look like?
Daniel: I start the day by catching up on admin before attending our team stand up, after which I’ll often have a call with one of my teammates to refine any work. After lunch on alternate days we have tech slots — think design sessions, demos or ways of working workshops. They can be really good for getting help, aligning the team to resolve recurring problems, or promoting a cool new tool you found on reddit. After that I try to keep my afternoons free for ‘Makers Time’ — essentially, time to focus and get stuff done.
Harrison: We have a very similar way of working to DE — at Gousto we are all members of the Data Platform team — after all, so we share many of the same ceremonies. Within our current feature team (which is currently all analytics engineers, by chance rather than design) we pair a lot in our work so much of my day is filled with collaboratively solving problems. Our day-to-day work is very varied and can range from working on our self-service products, to working with business users of our data to figure something specific out.
What skills are most crucial for your job?
Harrison: Collaboration, problem-solving and communication would be my top three. A healthy dose of knowing SQL, dbt and data modelling best practices helps too! Like with DE, there is a fair amount of technical work so it is key to be comfortable solving difficult problems.
Daniel: Strictly speaking, this is a technical role and so an appreciation of how to use tech to drive objectives and key results — you have to be highly analytical to solve problems. I think the most overlooked, yet equally important skill, is how you collaborate and work with others — this is crucial to delivering anything of significance!
What tools do you use for your job?
Daniel: People usually expect a list like Python, Spark, AWS here but I’d probably start with Google Meets, Miro, Confluence and Slack. Before a single line of code can be written we need to be able to effectively collaborate to refine and design what it is we’re building. We need effective ways to engage with our stakeholders to understand what it is we need to do to help drive the business forward. Then of course, that eventually translates itself somewhere into our tech stack which mainly consists of Databricks running on AWS with Spark Structured Streaming. We have broad use of AWS services as a standard at Gousto and we have a great platform that allows us to deploy these in just a few lines!
Harrison: My answer is very similar to Dan’s — knowing how to effectively use everyday tools like Miro and Slack are invaluable and will save you a lot of time! Specific to the AE role at Gousto is dbt and Databricks, which we use daily, and sometimes we can be found working with data pipelines or a BI dashboard. One of the great things about working in the data team here is you aren’t siloed into a specific role — you have the freedom (and are encouraged) to learn new skills.
Who are your key stakeholders?
Harrison: A stakeholder for us is anyone who uses the data we are providing via our models. This could be a data scientist, a data analyst, a business intelligence developer or anyone who is using a dashboard/analysis built off of our models. We also have stakeholders in the form of people looking to learn about and use dbt in their day-to-day work. Our dbt-as-a-product project has been ramped up in recent weeks as we look forward to onboard as many people as possible to enable them to self-serve their own models!
Daniel: As a recipe box company we have a growing network of factories, each producing a ton of data by the minute. I spent the first nine months of my career at Gousto aligned with our Supply tribe (now called Unbox) and therefore my key stakeholders were mainly centred around our factories. I’ve since moved into our core data platform team focused on ingesting web event data, so our stakeholders here are more interested in things such as app experimentation.
How do your two professions interact?
Daniel: For me, AE is the other side of the same coin and I’ve certainly worked on projects where we’ve almost been joined at the hip! Different projects require different ratios of AE:DE, but I found myself working heavily with AEs when focused on the delivery of factory KPIs. This meant I had to ensure the correct data was provisioned and cleaned prior to the AE picking up a KPI that required said data.
Harrison: We rely on DE to provide accurate data in the lake for us to base modelling on. Without their pipelines we would have no data! We will often jump on calls with each other to resolve issues regardless of the titled ‘role’ — the line between the two gets blurred as we pick up skills from each other. Indeed we’ve even had people migrate from one discipline to the other — we’re encouraged to follow our own interests.
What advice would you give to someone coming into your profession?
Harrison: Keep asking questions, no-one expects the new AE to be an expert in the subject matter or the technology (regardless of level). Over time you should be looking to learn things about the data you’re working with. The great thing about working with data is the diverse range of companies that you can work with! Try to stay up to date with the latest technologies, becoming an expert in yesterday’s technology is probably not ideal. If a process feels slow and painful, find a way to make it faster!
Daniel: DE can be challenging due to having to work with a vast set of technologies, which can feel overwhelming at times, especially when you’re more junior. Don’t let this worry you too much, but make sure you take the time to try new things out and if you find something of value, don’t hesitate to try and push the idea forward. As you progress, don’t let your technical competencies be your only competencies otherwise you’ll quickly stagnate. Find yourself a good mentor and try and focus on your softer skills for a bit, this will really change how you interact with others, making your working relationships more fruitful, increasing your ability to influence and ultimately, help you to get things done!
Check out more stories from Gousto and make sure to follow us here, so you catch the next instalment in our series of “What’s the difference between…”.
While you’re waiting for that you can check out this post about how Data Platform team go about setting their objectives.
