What is data engineering?

The aim is insight#

The aim of data engineering is insight.

But what is insight? Insight is information analysed in the light of intent.

[!SHARE:data-engineering-definition] Data engineering, therefore, is an activity in which business intent is applied to data to create first information, then insight.

Why is it engineering#

There are other activities besides data engineering that seek insight. These include staff consultation, customer feedback, and market research. In the case of government agencies, these activities include overt and covert means such as diplomacy and espionage.

Data engineering distinguishes itself from other means of gaining insight through the nature of its methodology.

Like other engineering disciplines, it focuses on systematic rigour, efficiency, and reproducibility—achieving these through the application of scientific and mathematical principles. As with other engineering disciplines, data engineering arose in today’s technological society as an effective tool to tackle complexity.

Where data engineering is different#

Despite similarities in method, data engineering differs from other engineering disciplines in crucial ways.

Easy substitutes#

There are easy substitutes to data engineering.

The business engaging a data engineer is seeking insight, and in many instances, there are seemingly simple alternatives to serious data engineering. For example, business clients may choose to rely on their own judgement or use one of the many available self-service data tools to rapidly create a visually stunning report with little effort.

This is not the case in other disciplines. The client who engages a bridge engineer needs a bridge. The client who engages a software engineer needs software. There are no easy alternatives to achieving these aims, and therefore no easy alternatives to the engineering effort.

Open-ended questions#

The search for business insight is open-ended.

If a client asks for a bridge, the bridge is the end of the engagement. A bridge is not likely to extend into a longer bridge, nor is the client likely to settle for half a bridge. This is not the case with business insight: a question provokes another question, and a question may be half answered.

This interminability of aim means that a data engineering project is likewise open-ended. In no other engineering discipline is the engineer working with such a degree of open-endedness.

Ambiguous goals#

The search for business insight is ambiguous.

Quite often, when stakeholders begin a data engineering project, they do not know what business insights they are after. Partly this is because business areas first need to discover the information before they know what questions to ask. Partly it is due to the wide-ranging and sometimes conflicting business intents—are we concerned about profit, cost, market share, or negative externalities?

And partly it is because of the wide range of stakeholders who need to be engaged—senior executives and operational staff have valid but different concerns—yet the data engineer is expected to meet all interests.

This ambiguity of aim is akin to the challenge of planning a new city suburb to satisfy the needs of all residents. Again, this is a unique challenge for a data engineer.

The fluid nature of data engineering#

The search for business insights is shaped by three intrinsic characteristics:

• the availability of easy substitutes
• the open-endedness of the search
• the ambiguity of aim

We summarise these as the fluid nature of data engineering.

Certainly, the data engineering team can shut its eyes to these problems and insist on hard-and-fast requirements from stakeholders. This approach is unlikely to lead to quality outcomes. Instead, the data engineer’s approach must adapt to the fluid nature of business insights.

At the end of the day, technical approach alone is insufficient. Instead, the data engineering team has an unparalleled need to combine flexibility, endurance of vision, and the ability to negotiate.

Moreover, compared to other engineering contexts, the non-technical capabilities sit in a small team—sometimes within a sole data engineer. A data engineering team must confidently wield these non-technical capabilities if it is to achieve excellence in an organisation.

The search for business insights is both demanding in complexity and fluid in nature. This dual aspect makes data engineering one of the most engaging and rewarding disciplines.

Key ideas

Data engineering is an activity in which business intent is applied to data to create first information, then insight.

The availability of easy substitutes, the open-endedness of the search, and the ambiguity of aim define the fluid nature of data engineering.