Closing essay: Hallmarks of quality

This essay was written years before this book for the data science and engineering team at the Department of Agriculture, Fisheries and Forestry. It is reproduced mostly as it was originally written. Its terms do not always match the rest of the book, but it represents an accurate summary of its ideas.

As developers, and broadly as a team with the goal of delivering products, we are under the constant temptation to build products that answer to the demands of the day.

These demands are often driven by emotions we feel towards other people. They may be happiness at appreciation from a client, or fear of reprimand if we do not deliver on time.

However, the instinctive reactions of personalities, especially of personalities who barely understand our craft, are not reliable guides to quality. These reactions may be justified. For example, if a project has been dragging on too long, then the people around us may be right to be angry. But they cannot be relied upon as guides to the quality of work.

Our work is of quality if it enables the organisation to make sound, positive decisions.

Nevertheless, this is too abstract a goal for day-to-day work. Even other measures such as product usage or sentiment analysis of feedback are too distant to help developers make decisions in their daily work. We must seek other guides.

Our purpose is to lay down hallmarks of quality that help a team recognise and celebrate quality. They are nothing new, but are time-honoured practices.

As developers, we must deliver the features, but also complete the following:

1. Expressive entities
2. Well-written explanation
3. Thoughtful unit-tests
4. Monitor assumptions
5. Anticipate errors
6. Adhere to patterns
7. Optimised code

These are the hallmarks of quality. When done well, we have not only delivered a feature but delivered quality work. The work will be high value, long lasting, will win trust and, importantly, give us pride in the work we do. Not everyone will carry out all aspects in every instance, but as a team, these are non-negotiables to quality.

Expressive entities#

When looking at data, it is easy to forget we are dealing with a real world. Yet the real world is what decision makers deal with. All models, whether they are simulation models, data models, or machine learning models, are models of the world.

Correspondence to the world by the agent trying to influence it is therefore the final arbitrator of whether a data model is successful.

Expressive entities refer to the idea that the tables and relationships we create in the data warehouse should not merely reflect the data as we found it, but correspond strongly to real-world business processes. It is expressive because the consumer or reviewer of the model can clearly recognise the world the model is trying to approximate. If a competent layperson cannot easily relate the model to the real world, then the model is not expressive.

The process of creating expressive entities happens at a local level when we use table and column names that are business meaningful. At the level of a single model, we often say that we are “crafting” entities or tables. At the warehouse level across several models, we often say that we are conforming entities.

A data model is not a quality data model unless it is a useful representation of the world. A data model is, therefore, a worldview—more than a worldview, but not less. A developer without a strong worldview cannot create a sound data model.

In a rapid development environment with pressure to deliver, crafting and conforming entities are difficult tasks. They are hard because it is hard to think clearly about the world in a systematic way and then articulate it through data.

There is a temptation to keep moving on to the next feature. Moreover, the benefits of crafting entities are long term and not easily understood by stakeholders. People understand testing and monitoring assumptions, but what does a “conceptual model” mean? However, the implication of inexpressive entities is that no one has spent time thinking about the world and the processes through which we try to influence it.

A quality data model is only achievable by a developer who has thought seriously about the world, how data relates to it, and how to organise the world in a way that makes sense for decisions. The ability to create data entities that are accurate and expressive of the world is an infallible mark of quality.

Well-written explanation#

Documentation is particularly important for data analysis, as opposed to, say, game programming, because data is a projection of real-world processes and data code acts as an interpreter between data and the world. This interpretation has a plain-English equivalent. Documentation is not simply “code comments” from one developer to another. It is an articulation to oneself and to others of how the output reflects reality.

The writer C.S. Lewis said, “If you cannot say something in common language, you either do not believe in it or you do not understand it.” If we cannot explain the logic behind our code in plain English, why should anyone believe we know what we are doing?

The primary purpose of a well-written explanation of our code is to articulate the logic back to ourselves. What are the entities we created? What do they really mean? What assumptions have we made? Writing a clear explanation is to step back from the code and do the work again from a different angle. Often, in articulating an explanation, one finds new ways of approaching the problem or discovers assumptions that were missed.

The secondary purpose of documentation is to win trust. To non-developers, our work is a black box. A well-written explanation is part of the transparency that is integral to trust. Business areas also gain a clearer understanding of the quality of work and effort put into development. This builds trust.

A developer does not need to write documentation alone. It is the one task where other people can be intimately involved. Sharing and collaborating on documentation with subject matter experts are avenues to clarify concepts and listen to feedback.

It is interesting to see a shift of habit as developers mature. A new developer often writes code with no documentation. Over time, the developer gains the habit of writing documentation after completing code, then the habit of writing documentation together with code. When mature, it is common for a developer to start with code documentation for complex scenarios, usually for circulation. There is nothing surprising about this. The transition merely reflects the fact that the developer has mastered patterns in the work environment and is shifting focus from the mechanics of code-writing to the reality behind what the code is about.

Thoughtful unit-tests#

At the minimum, writing proper unit-tests means that a developer has solved the problem more than once and from different angles. This is enough to justify writing the test. Often, in writing a test, the developer discovers a logical error or edge case that escaped attention during development.

In the long term, tests exist because the world is not static. What is true at the time of writing the code may not be true the day after. The data itself can change, or the code itself may change. Sometimes these changes can lead to significant errors that lead to incorrect decisions or harm trust.

In isolated instances, one code element may have a low probability of going wrong due to external changes. When running several hundred thousand lines of code against several thousand entities, and a dozen developers are changing several hundred lines per week, the probabilities of error add up.

Tests are about quality, not quantity or coverage. Poorly written tests can lead to false confidence.

Resist the temptation to delay a test. If instinct says a code should be tested, do not delay it to the next week. It is astonishing how often the test of which we think, “I probably should write this test, but I will do this a bit later,” is the one that would have caught an error on the week’s product release.

Assumptions are monitored#

All reasoning involves assumptions. Dealing with incomplete data especially requires assumptions. Assumptions are the X in “If X, then Y.” Assumptions are the shortcut to Y. They allow one to defer insignificant problems to a future date.

The world is not static. Assumptions valid at the time of writing the code may be invalid the next day. When assumptions fail, the consequence is always negative. This is because if the failure of the assumption has no impact, then the assumption was not necessary in the first place.

The failure of assumptions is often incremental rather than sudden. The world and its data usually drift from where we thought it would be, rather than making a drastic break.

This is particularly relevant with the rise of AI—models that work today will behave differently as the world changes.

Whether they are sudden or gradual, we must monitor assumptions to mitigate the impact for the inevitable day when they fail. If we interpret fault-tolerance as the ability to go ahead despite errors, then assumptions monitoring is a form of macro fault-tolerance.

Developers often assume uniqueness and existence of data. Without such assumptions, there can be no analysis. But what if the product table is not unique by [Product ID] tomorrow because the application had to add a [Product sub-type]? This is quite possible if the source table came from a non-relational system. If this happens, our logic will fall apart. On the other hand, we become paralysed by risks if we constantly assume source tables can change drastically. Therefore, automated treatment of common types of assumptions should be built into any mature process.

The key step to monitoring assumptions is to know when we are making one. More often, the issue is not that we are not monitoring the assumption, but that we are not conscious of making it. We assume the source tables will load in the daily ELT process. What if one of them does not load, or if they load but not in the order we expect? Another common type of assumption is the use of hard-coded values in code.

Once assumptions have been identified, it is a matter of being disciplined and monitoring them through an automated framework.

Anticipate errors#

Tests and assumptions fall under the general category of error handling. The question which concerns a seasoned developer is not whether the code runs today, but how it may fail tomorrow. Many can write code against the actual, but writing code against the possible is a whole new level. One can often identify a superior developer by an instinctive, second-nature obsession with how the code may fail.

Even the most seemingly inconsequential actions can trigger an avalanche of unintended consequences. By learning from these errors and improving the resilience of the process, these errors decrease. However, any large structural change is still likely to cause unintended effects. The first task of a developer, then, is to expect errors to happen and to anticipate specific ones. A developer must assume that things will always go wrong.

The primary purpose of error handling is to limit damage and prevent catastrophic ripple effects. If we make it a rule to expect errors, the easiest response is to limit the damage through fault-tolerant code. This is the task of preventing or silencing errors.

The second purpose of error handling is to reveal silent errors. Not all errors cause panic. In fact, the most dangerous errors are silent ones. They can be silent by intention or by accident. In either case, a human should be alerted to apply a fix, or the code should be designed to recover automatically before damage propagates. This is the task of detecting or recovering gracefully from errors.

The third purpose of error handling is for the team to grow continuously. Each error reveals a blind spot. When errors are handled properly, the team will be able to learn from the error and improve processes. This may mean implementing new automated checks or rethinking the algorithm from scratch. With each error, the system becomes more mature and more resilient. This is the task of learning from errors. Learning from errors requires an open acknowledgement of failure and a willingness to share the lesson with others. This is why humility is important for developers.

Error handling is hard because the developer needs to look beyond the now and anticipate changes in the world. It is also challenging because error handling requires an end-to-end understanding of the data architecture. The ability to anticipate, prevent, silence, detect, recover gracefully, and learn from error is a hallmark of genius. This should apply both to technical situations—what happens to the join if the column is null?—and to business rules—does the logic given by the business make sense in all circumstances?

A delivery team that can continuously learn from errors is unstoppable. While error handling is challenging, a new developer can start by adopting the habit of expecting errors.

Adherence to patterns#

There are slavish adherences to patterns that are counterproductive. However, patterns, whether they are coding styles, naming conventions, architecture, or the use of Git, are an indispensable part of a strong team. There are bad teams that follow patterns, but there are no good teams that do not have them.

The primary purpose of patterns is to pass on hard-earned wisdom acquired by the team. In a complex environment, there are solutions that are difficult to arrive at, counter-intuitive, or invisible to the untrained eye. This wisdom has been acquired by predecessors at great cost. It is not proper for developers to remain ignorant of their value due to laziness in observation or a naive confidence in their own judgement.

The second purpose of patterns is to create a consistent experience for end users. This consistent experience ranges from design language such as table names to more abstract experiences such as joining a set of unfamiliar tables. This consistency of experience over a body of work is a felt quality that cannot be overstated. This hallmark of quality is achievable only by a team where every developer plays their part in adhering to the whole.

The third purpose of patterns is to support thinking by filtering out mental noise. Yes, CamelCase is no better than snake_case for table names. We could have used [Datetime created] as well as [Creation datetime] or indeed [Created datetime]. In isolation, whether text files have hard tabs, or 4-space, or 3-space soft tabs matters little. On the other hand, when the same standard decision is applied across the codebase, the pattern creates an important stability. In fact, when patterns are consistently applied, anomalies stand out—even anomalies which have nothing to do with the pattern. During review of code that is not written in accordance with a usual pattern, the reviewer is forced to carefully parse and interpret every line and every word.

These mental noises reduce the probability of detecting an error. It is astonishing how, after the developer fixes some code indentation, previously unseen logical errors stand out.

Judicious use of patterns in a team will advance quality by promoting solutions that survived the test of time. Patterns also accelerate delivery by removing developer uncertainty and simplifying decision making. On the other hand, an unthinking adherence to patterns through a copy-and-paste mindset can lead to overengineering or stifle innovation. To strike a balance between individual innovation and accumulated wisdom, team members should see themselves as participants in a conversation of innovation in which each team member can contribute with an attitude of “inquire, not debate.”

Optimised code#

Performant code carries out a task with as little server resource as possible. Elegant code promotes readability and simplicity. Taken together, we say the code has been optimised.

The primary purpose of optimising code is to support long-term maintainability. In a rapid development environment, a multitude of non-performant code can easily overwhelm the server. If code is not elegant, we increase the burden of changing the code. In isolated instances, the problem may be insignificant. Over several hundred thousand lines of code, the problem compounds.

A direct way to write performant code in a data warehouse is to use incremental extract logic. This is by no means easy. It can be difficult to do accurately when integrating several datasets. Incremental extract can influence overall design. For example, one may be selective about what tables to use in a single script to avoid a dependency that would ruin the potential to incrementally extract.

The secondary purpose of optimising code is to think about the problem again. It is harder, and takes more thought, to write simple code than complicated code. Often, in making code elegant, the developer sees the concept in a different light. The process of a re-think can lead to conceptual breakthroughs.

Not all good developers can write optimised code, but bad developers can never write elegant, optimised code. Optimised code is one of the harder hallmarks to achieve. We can all improve with practice.

Three aspects of quality#

The hallmarks describe three aspects of data-based code: expressiveness, error handling and elegant code.

Expressiveness#

Creating expressive entities, well-written documentation.

Expressiveness is grounded in the fact that data is not a real-world entity in and of itself but is a projection of real-world processes onto computer databases. Such projections are often fuzzy, imperfect, and contingent. When we create expressive entities, our goal is to reconstruct a clear and useful picture of the world from which the data came.

Well-written documentation serves the same purpose by articulating in plain language how a data model relates to the real world. Analysts who continually take data at face value—each row in the Task table is a task, right?—or mechanically apply statistical algorithms from a textbook are divorced from the reality behind the data.

Error handling#

Anticipating errors, thoughtful unit-tests, monitoring assumptions.

Error handling is to code what seat belts and safety brakes are to cars, or monitoring and kill-switches are to a power plant. They are not nice-to-haves but indispensable components of a complex computational environment. Unit tests and monitoring assumptions all help to mitigate against the inevitable day when an error occurs. A developer creating code without considering what may go wrong is akin to a builder who adds a floor to a building without strengthening the foundations—it will work, for now.

Elegant code#

Adherence to patterns, optimised code.

Elegant code is key to sustainable growth. In a complex and fast-moving environment, we cannot afford spaghetti code or free-for-all approaches to common problems.

Performance-tuned code is obvious for the simple reason that computational resources are finite.

Final words#

These hallmarks of quality are not revolutionary. They have been around since code development began. The issue is not that developers do not know about them but that they do not do them because of lack of discipline or pressures to deliver.

There is the idea that doing work properly costs time and delays delivery. The retort is that doing work improperly costs more time. Writing good tests helps capture errors earlier and reduces the time it takes to fix things. Crafted entities vastly reduce the time required to deliver complex features because the model becomes more powerful. Writing a clear explanation helps everyone get on board, establishes trust and builds momentum. On the other hand, technical debt quickly comes back with a vengeance. At the level of a single development cycle, doing the work properly may increase development time by 20 to 40%. Over the course of several months, the dividends of quality work pay off and enable faster delivery than hasty development could ever achieve.

The developer is constantly under pressure from deadlines. However, if one does not have time to do the work well, the solution is not to do the work poorly. The best way to achieve the hallmarks is simply to consider these as part of the work itself, not something in addition to the work. Once we make the mental leap, it becomes more natural to build these into daily work and into the pace of projects.