Data and organisations

Data and organisational self-understanding#

Data is a fragment of reality. This includes the reality of the environment in which they operate, but also the reality of the organisation itself.

This has implications for how data engineering work is done in practice. The veteran data engineer instinctively works within the entanglement between data and organisation.

To understand this, it is useful to visualise an organisation as a feedback loop.

The organisation agency cycle#

The central premise “Data is a fragment of reality captured by process” can be interpreted as a feedback loop around three components: business processes, digital systems, and captured data. This relationship is illustrated in Figure 1.

Like many cycles, an organisation’s feedback loop can become vicious or virtuous. Business intent is the determinant. An organisation moving toward its intent is in a virtuous cycle; an organisation drifting away from its intent is in a vicious cycle.

A key point of the cycle is that data is not passive “facts and figures” from the world. Data consists of records of the world the organisation is interacting with, and records of that interaction. The feedback loop is therefore a model of how an organisation exercises its goals in the world.

The implication is that what the organisation does is shaped by what it sees, and what it sees is shaped by what it does. Data is the medium of this double movement.

Consequently, data is central to an organisation’s self-understanding. Data engineering is not one capability among others. It is the capability through which other capabilities become visible.

Implications for data engineering#

There are two important implications for data engineering work.

First, recorded data is not simply “what is in the system.” It is downstream of business processes and digital capture. The data engineer is always working inside this context, and this context shapes the scope and options for action within a project.

Second, the data engineer’s work itself is not neutral. The output of data engineering is often the first time an organisation encounters the outcome of its own work in a stable form. For this reason, the output may be contested, especially when multiple business areas are involved. Interpretation becomes inseparable from the work: what is good, what is bad, what is reportable, what is not, which record is authoritative, and which definition should be canonical?

[!SHARE:meaning-negotiation] Organisations do not simply use data. Data is also where meaning is negotiated.

The data engineer cannot stand outside this negotiation. The engineer can help the organisation navigate meaning with expertise, defer to consensus without judgement, or allow the work to fail because meaning was never clear.

The agency cycle therefore shows why no part of data work is neutral. The business process that defines the digital system is not neutral. The digital capture is not neutral. Most importantly, the data engineer’s work to create feedback is not neutral. All are evaluated by intent. This is why data engineering reshapes data in light of intent.

Key ideas

What the organisation does is shaped by what it sees; what it sees is shaped by what it does. Data is the medium of this loop.

Organisations do not simply use data; data is also where meaning is negotiated.