On automation: Full Automation Fallacy

Automation level taxonomies

Every automation level taxonomy going back 50 years looks similar: "computer does everything" is the final level:

Sheridan & Verplank (1978)

Taxonomies got refined over time but final level is always "computer does everything". Here is a version 20 years later:

Endsley & Kaber (1999)

There are even newer more refined taxonomies but they are all variations on a the theme. I even wrote my own 10y ago. The problem with these taxonomies is that they focus on the task and are wrong in practice even at the task level. Computer never does "everything".

Automated systems are manual

What we call "manual work" is almost always already automated - i.e. any person if front of the computer is obviously using some level of computer automation. Even highly automated systems are manual:

Even highly automated systems such as electric power networks, need human beings for supervision, adjustment, maintenance, expansion and improvement. Therefore one can draw the paradoxical conclusion that automated systems still are man-machine systems, for which both technical and human factors are important.

Bibby et all (1975) Man's Role in Control Systems

Human factor cannot be designed away

The designer who tries to eliminate the operator still leaves the operator to do the tasks which the designer cannot think how to automate.

Lisanne Bainbridge (1983), Irony of Automation

Because automation leaves the operator to deal with the increasingly rare and complex edge cases, manual work is always becoming more and more complex over time. This is also why there is always more and more "manual work" left to do.

Those who worry about "But AGI!" should familiarize themselves with "wicked problems" - problems that cannot be formulated, cannot be "finished", don't have enumerable solutions, essentially unique, and have a number of other devilishly difficult attributes. These problems are not going away anytime soon and will provide room for human factor forever.

Human factor is about managing failure

John Gall (1975) Systemantics: How Systems Work and Especially How They Fail

Theory of Errors

This sounds bad, but it is so common that we don't even realize that our systems are often named after failure modes: debt collection, waste disposal, sanction screening, etc.

Human factors don’t disappear; they shift:

• from continuous control → rare, high-stakes interventions
• from hands-on operation → monitoring, diagnosis, and trust calibration
• from skill-at-use → skill-at-recovery
• from individual error → organizational/maintenance/configuration error

Automation levels are about firm boundaries

Human factor cannot be removed, but it can be outsourced to increase efficiency (via economies of scale, economies of scope, learning rates, etc). Levels of automation are then about:

• Operational authority: who acts in the moment?
• Goal-setting: who chooses objectives/constraints?
• Change authority: who can modify the system (software, models, parameters)?
• Accountability: who is responsible when it fails?
• Recoverability: who can detect/stop/recover?

Robotaxis have tele-operators monitoring and recovering. Cloud vendors have incident teams, server capacity planners, etc. It is the fact that these are completely externalized that creates "computer does everything" magic feeling.

Thus, designing software that helps move high-fidelity man-machine interfaces across firm boundaries is what automation is all about.

Originally published on LinkedIn.