Job Market

High demand for data and tech professionals is not a new trend to see in the labour market. The recent pandemic made us rely on technology more than ever before. It has also brought more attention to job dissatisfaction issues resulting in occurrence of the Great Resignation. More than half of businesses reported being unable to meet demands due to labour shortages.[i] As employees started to place more value on work-life balance, which is often broken when job demands exceed job resources, we saw increase in the number of job vacancies in tech.

Burnout in Data Engineering

Whilst trying to learn more about challenges in the data industry, I came across the survey conducted by DataKitchen and data.world. The results show that 97% of data engineers feel burnt-out. Reasons for burnout include inefficient workflows, manual processes and constantly changing requests from stakeholders. [ii]

Figure 1 – Sources of burnout identified by data engineers and managers.
Source: DataKitchen & data.world 2021

Growing Industry Challenges

Exponential growth in the industry and emergence of big data have contributed to a massive increase in the volume and complexity of data. Data professionals need to spend a significant amount of time on manual and repetitive processes related to data preparation, pipeline maintenance or finding and fixing errors. While these tasks help to achieve governance, they also reduce agility.

Evolving customer expectations mean there is a need for rapid results in order to meet business objectives and ensure customer satisfaction. However, in efforts to deliver against stakeholder requests quickly, data quality measures and testing are often reduced to a minimum, meaning the product is likely to contain errors that could damage trust in the team and data, and deem insights meaningless. ⁱⁱ

Even though teams consist of dedicated and competent people, the dynamic of the relatively new discipline causes struggles with conflicting priorities, insufficient resources, lack of communication and inefficient processes. All of which contribute to decrease in productivity and job satisfaction, also leading to worsening employee retention rates. ⁱⁱ

Employee Retention

According to the survey results, 70% of data engineers are likely to quit in the coming 12 months. That does not only mean changing companies but leaving the field entirely, which shows it is an industry-wide problem and not specific to a company. ⁱⁱ

Figure 3 – Data engineers expressed intentions to leave their job or change careers entirely.
Source: DataKitchen & data.world 2021

Harnham Salary Guide shows the average time in a role among UK respondents in 2020 was 2 years. [iii] Decreasing average tenure and employee retention are becoming big challenges, and salary often is not enough to make an effective counteroffer or attract talent. [iv]

What does it mean for a business?

Inability to address employee retention issues and find new talent quickly puts more pressure on the current employees, which consequently accelerates the problem causing more resignations. Companies lose valuable team members and often turn to cheap and temporary solutions in efforts to meet the demand. This approach however will damage overall business performance and slow down product delivery. While the technical debt is building up, reduced quality of a product and services will negatively impact customer satisfaction and churn rate.

How to address these problems?

The main solution identified by DataKitchen is DataOps, which is a set of practices that aim to improve quality of data products while reducing time to see value. The core principles are collaboration, communication, automation and universal applicability of data flows, continuous integration, and agility. Implementing these principles helps to optimise workflows and productivity of the team members, and consequently increases delivery speed without sacrificing governance. ⁱⁱ

Shifting from on-premise to cloud computing is another way to increase agility. Since a lot of services are managed by the cloud providers, it frees data team from maintenance tasks allowing them to focus on meaningful work. Unlimited number of servers and instances for different environments can speed up processes – multiple people are able to work on different parts and test features in parallel, which makes fixing errors faster and drives continuous integration and delivery.

While the volume of data is expected to grow to more than 180 zettabytes over the next 5 years [v]; the pace of stakeholder requests and user expectations are not likely to slow down. To set a team up for success and meet market demands, more attention needs to be given to the design of workflows that allow efficient collaboration. Working toward a solution should be the top priority of every business wanting to effectively adapt to changes.

At Rockborne, we are aware of the challenges faced by the industry and understand difficulties related to finding right candidates quickly. Our Graduate Program addresses the demand of data professionals and the skills shortage to help businesses transform outcomes with diverse data talent. On top of our intensive bootcamp, Rockborne consultants go through a custom training adjusted to the needs of a client. Find out how Rockborne can help your business today and create long-term benefits of the partnership.

References:

[i][i] Changing trends and recent shortages in the labour market, UK: 2016 to 2021. Office for National Statistics

[ii] 2021 Data Engineering Survey: Burned-out data engineers call for DataOps. DataKitchen & data.world

[iii] UK Data & Analytics Salary Guide 2020. Harnham

[iv] UK Data & Analytics Salary Guide 2021. Harnham

[v] Amount of data created, consumed, and stored 2010-2025. Statista Research Department