More and more, companies are seeking a new role to add to their data team—the analytics engineer. But what does an analytics engineer actually do? Is the term just a rebranding for a role that already existed? Or is it truly filling a gap across data teams?
In this post, we take a closer look at why the role came about, how it fits within the broader data team, and what the future may hold for analytics engineering.
What is analytics engineering?
Analytics engineering serves as the bridge between data analytics and data engineering.
Why? Because it requires oscillating between the data infrastructure managed by data engineers and the business initiatives managed by data analysts. The ultimate goal of an analytics engineer is to produce high-quality datasets for use in a variety of data initiatives across the organization. In practice, what that looks like is preparing, transforming, or modeling raw data in data warehouses or other data repositories to align with objectives and/or to ingest it into business-friendly applications.
Hence, analytics engineers must have a foundational knowledge of data architecture, but must also be in constant communication with the business.
How did analytics engineering come about?
The emergence of analytics engineering stems from a significant shift in ETL, or the “extract, transform, and load” process.
Today, instead of first extracting data, then transforming it for storage, and, finally, loading it into said data warehouse or repository, many organizations are following an ELT process. That is to say, the loading of the data into data warehouses happens before the transformation. Under ELT, data is moved from its source to staging and then into the data warehouse.
The benefits of an ELT approach are faster data availability, lower maintenance, and quicker loading time—all of which are benefits more well-suited to the size and pace at which data is generated today.
An ELT process also allows data engineers to solely focus on the “E” and “L,” with the idea of relegating the “T” to business groups that have the right context of how this data should be transformed.
While the convergence of data engineering and data analytics teams at this “T” point in the data pipeline was, and continues to be, an admirable goal, it can be a tricky balancing act in practice. Without clear ownership in either direction, it can be difficult for data engineering and data analytics teams to divide up responsibilities while still reserving the majority of their work day on data engineering or data analytics tasks, respectively.
Instead, many organizations have begun to see the decoupling of the “T” from the “E” and the “L” as an opportunity to hire new data workers that would specialize in that part of the data pipeline specifically, and interface between both data engineers and data analysts.
In that way, data engineers could better focus on the work of getting data flowing into the warehouse, data analysts could focus on the important dashboarding and visualization work that drives much of today’s business decisions, and analytics engineers could work between the two.
How do analytics engineers stack up against other data workers?
To more clearly understand how analytics engineers fit within the broader data team, here’s how the role sits between a data engineer and a data analyst:
Data engineer
A data engineer is largely focused on data infrastructure. They build the systems that collect and manage data, as well as the data pipelines that move data from one place to another.
A data engineer is always looking to optimize data warehouse performance; how can data load more quickly, be processed at greater speeds, or achieve greater quality levels, for example.
While a data engineer has relatively little need to understand the ins and outs of data initiatives, identifying more broadly how data will be used by the business can help inform a data engineer’s work.
Data analyst
A data analyst, on the other hand, is solely focused on using data to create value for the business through dashboards, visualizations, or reports. They are always attuned to how existing data can answer a question or help drive a decision.
Sometimes, data analysts will be distributed within departments (such as a dedicated marketing analyst or sales analyst); other times, organizations will create a unique data analytics team that functions like its own business.
Data analysts typically work within BI tools, such as Tableau or Looker, and have greater strengths in business acumen than in coding languages or more technical data skills. However, most will at least have a basic understanding in SQL.
Analytics engineer
Analytics engineers must be both technology and business focused. In general, analytics engineers are database-savvy, but also have business and data analytics experience.
They should also be able to work with data engineers to help define back-end requirements for data products, as well as build robust solutions that surface critical data from a vast and diverse collection of datasets.
At the same time, they should be able to collaborate with business teams to understand their data and reporting requirements, design and improve data models, and define successful analytics outcomes. They must be adept at understanding the nuanced ways that different departments need to leverage data.
Their ultimate goal, as stated earlier, is to work in tandem with both data engineering and data analytic teams to collect, manage, and convert raw data for use in analytics initiatives.
What skills do analytics engineers need?
In order to fulfill their role, what are some of the essential skills that analytics engineers must have?
Data transformation
Since analytics engineers sit at the “T” junction of ELT, they must, of course, be experts in transforming data, whether through code or through data engineering platforms, such as Alteryx Designer Cloud. They should be able to design, own, and implement scalable algorithms that automate repetitive data transformation processes.
Data modeling
A large part of analytic engineering involves modeling data, which means that a solid understanding of coding languages is a must—at the very least, more so than the average data analyst. Analytic engineers must have strong SQL skills and experience writing Python,
Data documentation
It is critical that analytics engineers ensure that data is well-documented, not only so that everyone in the organization adheres to the same definitions and language, but also so that data is readily searchable. The analytics engineers should be monitoring data pipelines to ensure this consistency, and the first to spot and alert downstream business teams of any documentation issues, should they arise.
Experience with modern data tools
Experience with leading modern data technologies, such as dbt, BigQuery, and Snowflake allow analytics engineers to put this best strategy in place. Designer Cloud is also a plus, as it provides an accelerated platform to transform data and build data platforms. Similarly, analytics engineers must have experience with Tableau or other data visualization tools to be able to properly communicate data to the business.
Strong collaboration & communication
As is probably evident, this is not a job where you can be heads down for too long; analytics engineers must have strong collaboration and communication skills that allow them to navigate between data engineers and data analysts. They must also be clear in their communication as to how and why data will be transformed so that business groups are aligned in how it meets specific requirements.
What is the average salary of an analytics engineer?
According to ZipRecruiter, the average salary of an analytics engineer is $111,480. This is, of course, dependent on a wide variety of factors, not least of which include the location, size of the company, and previous experience of the candidate.
Certainly, an analytics engineer would have much room for growth and, through handling bigger projects and eventually taking on a management position, could earn as much as $203,000.
What does the future hold for analytics engineering?
While this is a relatively new role, it has a tremendous amount of potential. It plays heavily into the new ELT strategy—which appears as though it’s here to stay—and offers a new lateral career option for data analysts who want to develop more solid technical skills or data engineers who want to lean more heavily into the business side of data.
While this role didn’t exist ten years ago, neither did much of today’s modern data technologies that have taken a strong foothold in the market. The role reflects a shifting culture toward greater data collaboration and a need for increased data availability. The data world is moving fast—it’s no wonder that new roles are being created to keep up.