top of page

CFEngine’s Star Trek and AI origins--A 30 year anniversary tribute

CFEngine is 30 years old this year. For a piece of software, that’s been quite a long life. A lot has happened on its journey to arrive at where we are today, but most users have now forgotten how it emerged from a young lad’s dreams of artificial intelligence. I’ve written at length about the science in my book In Search of Certainty, and even some biographical notes. but grab a coffee and gather around, and I’ll tell you a bit more.

Me with Nick Anderson from CFEngine

Today, CFEngine is widely known as the configuration management tool that was replaced by Puppet, Chef, and then Ansible. “I know engineers, they love to change things!” (said Dr McCoy in Star Trek The Motion Picture). True, we love new tools, but it’s not accurate to say that CFEngine has been replaced. It’s alive and well. More importantly, calling CFEngine only configuration management isn’t quite accurate either. CFEngine was imagined to be much more than what we’ve now come to understand as configuration management. It was designed in the age before virtual machines and cloud computing, and yet its principles are still sound and applicable to much of the IT world today. To do everything that CFEngine did in modern IT systems over the years (configuration, monitoring, network routing, text processing, network orchestration, etc), you would need half a dozen different bits of software with some challenging integration on top.

I wrote CFEngine at the University of Oslo in 1993 after a deep discussion with the system administrator at the department of physics about the complexity of the shell and Perl scripts we were then using to automate the fifty or so servers we had. The physics department was one of the most demanding IT environments at the university for Unix-like operating systems. There were many flavours of Unix then: SunOS4, SunOS5 (which became Solaris), HP-UX with its Sinclair Spectrum-like rubber keyboards, Apollo workstations, IBM’s AIX in multiple versions, OSF1 on the DEC alphas, Ultrix, and more. Linux was still a pipe dream at that time.

These flavours of Unix were all quite different from one another–some based on BSD Unix, some based on System V. On top of this, everyone at the university had special needs! Today we try to make systems as similar as possible in order to manage scale, but CFEngine was designed to handle diversity and variability without breaking the bounds of human effort. It was putting the user needs ahead of limitations imposed for managerial convenience. It was as much about knowledge management as configuration management. Biology can handle diversity, why not technology?

A colleague at the university computing service USIT had written some impressive daily and weekly shell scripts to manage the university’s computing service’s interests on the machines. The idea of running the same software regularly, like an error correction loop was intriguing. Most software installed once and then went into a hands-off monitoring mode. I was intrigued by these maintenance scripts–not least the huge complexity that went just into dealing with the different syntax between systems. Half the scripts were if-then-else tests to figure out the precise version. One could hardly see what they were actually doing for all the checking.

What we were talking about was a software robot that lived not in the physical world, but whose environment was the abstract state space of operating systems. A declarative language could be used to express desired intent, but every machine had to have the robotic chops to be responsible for its own state. Back then, you never knew if the network would be up or down. The robot wouldn’t manipulate blocks or chess pieces as one thought in the 80s, but rather system files and processes.

One of the major challenges was the configuration of difficult subsystems like Berkeley Sendmail, which involved complex text editing as well as process management. One couldn’t simply install standard files because they wouldn’t work across all machines and they wouldn’t respect the special needs of what local users wanted. It would be best if people kept their hands off the hosts, but you could never guarantee that, so it was bad-form to just overwrite stuff to impose change. From this experience, the concept of autonomy and promises versus impositions eventually evolved in Promise Theory. CFEngine’s text editing language remains one of the most sophisticated models of automated editing for files, working convergently while working around what others might have done (what we now tend to call idempotently, which is not quite accurate).

Old notes on convergence in CFEngine. Today a new generation name drops semi-lattices and idempotence.

Just running shell commands wasn’t going to cut it either as some software does today for relatively homogeneous Linux. For one thing, all the options of shell commands were different across the Unices. A software agent that could manage a computer as well as a human needed to have complex manipulative skills and cause as little downtime disruption as possible.

The answer was obvious: one should separate the cognitive or sensoryenvironmental concerns and hide all that checking from the language of intent to reveal the purpose. We discussed the idea of a Domain Specific Language to make everything crystal clear, and over the Christmas break I wrote one. It was an ad hoc affair to begin with, which went through several revisions, but it did the job of exposing the intent rather than the technicalities. Simplicity was what system administrators wanted then (the story is different now, in an age of developers).

A doodle I made in a meeting in 2009 accurately shows CFEngine as an integrator of independent pieces in a larger game :) of configuration !

The endless if-then-else statements of scripts were replaced by a set-algebra evaluator for policy rule relevance that was likened to Prolog. I didn’t know much about AI research then, except for what I’d read in Douglas Hoftadter’s Gödel Escher Bach. I started intuitively, and only later came to try to put the ideas on a proper academic footing. Still, I’m still surprised at how often I rediscover that the right way to solve modern problems is to do what I did intuitively in CFEngine.

As a physicist, with an interest in computers, the challenge of regulating a system had several dimensions. I didn’t just want to run commands to be forgotten, like a Markov process. I wanted to understand the dynamic stability of the machines and measure their behaviour. I was intrigued by the idea of artificial life, and I imagined a system like the one in Star Trek where you could ask the computer for a diagnostic and everything would heal itself. In an early online manual, I even quoted with some self-irony from the original episode of Star Trek The Ultimate Computer, about Dr Daystrom’s M5 “duotronic” computer.


bottom of page