Network integrity:

• So far, we've considered what happens when one has one machine.
  • practical to build from scratch.
  • practical to do everything by hand.
  • best practices: what you decide to do to build the system each time.
• It's time to consider instead what happens when one has a network of machines.
  • no longer practical to build by hand.
  • no longer practical to do things from scratch.
  • best practices: automated methods for building stations the same way each time.

Network needs:

• homogeneity: every machine acts the same way.
• scalability: can add hosts without increasing workload.
• contingency: can react to unexpected changes.
• reproducibility: can replace a failed host with an identical copy.

Is there is truth no beauty?

• Homogeneity is boring. But it's better than all-nighters!
• Key to understanding the value of homogeneity: lifecycle cost.

Lifecycle cost

• Cost of running a machine, cradle-to-grave, from initial purchase to throwing it in the dumpster. Includes cost of:
  • hardware
  • software
  • maintenance
  • upgrades
  • troubleshooting
  • rebuilds
  • downtime
  • contingencies (expected cost = prob(risk)*cost(risk))
• Reality: almost all these costs are negligible. The significant costs include:
  • troubleshooting/rebuilds: require (expensive) human labor.
  • downtime: lost work of client (can't enter data, can't make sales, etc)

Effect of heterogeneity:

• doesn't increase hardware or software cost.
• drastically increases troubleshooting costs (every machine is different)
• indirectly increases downtime and downtime cost.
• makes it more difficult to react to contingencies.

Why heterogeneity doesn't work.

• As number of homogeneous machines increases, what one has to remember to manage them doesn't increase.
• As number of heterogeneous machines increases, what one has to remember varies with the number of unique configurations.

Naivete doesn't work:

• Naive approach:
  • Do everything by hand.
  • Tune till it works.
  • If systems differ, compare them and make them conform by hand.
  • Back up everything.
  • If a host dies, restore from backup.
• Realistically:
  • Too much work to do everything by hand.
  • Can't expect to be able to force systems into agreement manually.
  • Can't expect to be able to back up every computer on a network.

Do the math:

• We have 2 TeraBytes of data storage on the fileservers.
• We back this up daily. It costs about $1000 for a month's worth of tapes.
• We have 100 workstations with about 18 GB of local storage apiece.
• If we back these up, we have to store 1.8 terabytes of data, at a rate of approx. $1000 a month of tapes.
• And most of the information on these machines is just copies of software available on CDRoms.
• Better strategy: be able to return these machines to their current configuration without backup of the whole machine.
• Observation: there are very few differences between machine configurations.
• Practice: put variations into /var (why do you think it's called that?)
• Instead of backing up the whole machines, just back up the differences, including the contents of /var.

Changing files

• At the most basic level, configuration requires editing and copying files.
• Problem is: this is almost an assembly language. Far removed from the intent of management.
• Problem of semantic distance: file copying and editing is far removed from:
  • the commands a user might issue to configure a system.
  • what tools like make and rpm actually do.

Hierarchy of automation approaches:

• documentation: do it by hand.
• scripting: do same thing automatically.
• file abstraction: determine what is done to files by scripts, assure that automatically.
• service abstraction: describe services in terms of the assurances one must have in order to provide them, e.g., www.

Cloning

• Most systems look mostly the same.
• We can thus clone one system from another, and then make changes to individualize it as needed.
• Customizations just require changing a few files, not the whole disk.

Baselines

• One approach to cloning is baselining.
  • Build every machine exactly the same way.
  • This forms a baseline configuration.
  • Document changes needed to change the baseline into a desirable system.
• If a system dies,
  • Rebuild to baseline configuration.
  • Make changes again.

Principle of convergence

• Decide what it means to be correctly configured.
• Change anything different from this.
• If it ain't broke, don't fix it.

Simple convergence: rdist

• rdist: the remote distribution utility.
• distributes files from a master server to local hosts.
• server push: master sends files to slaves.

   +---------------+  distribution +----------------+
   |               | ------------> |                |
   | master server |   requests    | client machine |
   |               |               |                |
   |  /etc/motd    |    status     |  /etc/motd     |
   +---------------+ <-----------  +----------------+
                        reports
  

• Configuration file is Distfile (patterned after Makefile):

   #[key][source]                     -> [ hosts  ]
   motd: /admin/rdist/master/etc/motd -> ( presto apex forte )
         install /etc/motd; # directives must start with TAB
   #[tab][command][destination]
  

• /admin/rdist/master/etc/motd - local master copy of file.
• /etc/motd - remote destination for master.
• presto, apex, forte - hosts upon which to place the file.

Rdist directives:

• MACRO=value: set a macro
• install /etc/motd: put the file in question into /etc/motd
• notify couch@eecs.tufts.edu: mail a report of actions to this person
• except foo: omit copying files named foo if copying a directory.
• special "/usr/lib/newaliases" : run a command (as root) on the remote machine.

Example:

• If Distfile is

   SUN= allegro presto apex forte
   aliases: /etc/mail/aliases -> (${SUN}) 
         install /etc/mail/aliases
         special "/usr/sbin/newaliases"
         notify couch@eecs.tufts.edu
  

• And we invoke :

   rdist -m forte aliases
  

  then rdist will do the following things.
  1. check whether aliases is out of date.
  2. if out of date:
     1. copy the new one over (adjust date to that of copy).
     2. invoke local command /usr/sbin/newaliases.
     3. mail a message to me saying it was done.

Rdist strengths:

• won't update files unless needed (out of date or wrong protection)
• easy to invoke.
• very mature software (developed over many years)

Rdist Bugs (features):

• hangs forever if a target host is down.
• requires that the distributing host (master) have root privilege to rsh to the target host(Can fix this by customizing rdist using kerberos).
  • This means putting a record for root on the master host into /root/.rhosts on every host.
  • So if a hacker compromises the master host, he has root on the whole network!
• There are several versions of rdist that don't talk to one another very well!

Usage problems:

• must maintain a centralized list of client hosts.
• must maintain the database of files and targets.
• must maintain a rather large set of keys describing distribution actions.
• must remember a rather complex set of configuration commands.

Integrity problems:

• rdist can't work in reverse.
• if you modify files as root, rdist can walk on your changes unless you keep master files 'in sync'.
• you can't necessarily easily identify the files you need to propogate back to the master.
• you have to remember what you modify locally, and copy each modification to the server.

Personal testament

• Used rdist for many years.
• Came to several dead-ends; hit fundamental limits of rdist.
• Final solution: replace rdist with cfengine.

One solution: wrappers (front-ends)

• rdist didn't do what I wanted.
• very difficult to implement another rdist.
• to add features to a command, 'wrap' it with a command that calls it.
• This wrapper can have its own databases and builds the Distfile used by rdist automatically.

Case study: wrapping rdist

• needed simpler interface.
• needed revision control
• result: distr0.0 (couch)

Simple example of wrapping:

• interpret a database of files to distribute.
• generate an rdist control file Distfile by what amounts to macro processing.
• invoke rdist on the generated Distfile.
• largo:/admin/rdist/Dist.make: generate a Distfile for rdist from a database. This reads:
• /admin/rdist/Dist.cats: categories for distribution actions:

   _allegro
  

  shorthand for __allegro allegro: creates a one-machine category.

   __sol2x allegro presto apex forte vitesse 
  

  all solaris 2.x boxes (for some x=4,5,6,7)

   __sol2x_no_allegro presto apex forte vitesse 
  

  all of these except allegro, which must be treated specially.
• /admin/rdist/Dist.keys: actual file distribution commands:

   # file                                        keyword
   __sol2x_no_allegro/etc/mail/aliases           aliases
  

  • largo:/admin/rdist/__sol2x_no_allegro/etc/mail/aliases: source file
  • presto:/etc/mail/aliases: one target file
• Other distribution examples:

   # file                                        keyword
   __sol2x/etc/motd                              motd
  

  On every sol2x box, place this /etc/motd.

   __sol2x_no_allegro/etc/netgroup               netgroup
  

  Likewise on every sol2x box except allegro, put this /etc/netgroup.
• Based upon this information, Dist.make creates Distfile:

   _allegro = ( allegro )
   __sol2x = ( allegro presto apex forte vitesse )
   __sol2x_no_allegro = ( presto apex forte vitesse )
   motd: __sol2x/etc/motd -> ( ${__sol2x} )
         install /etc/motd;
   aliases: __sol2x_no_allegro/etc/mail/aliases -> ( ${__sol2x_no_allegro} )
         install /etc/mail/aliases;
   netgroup: __sol2x_no_allegro/etc/netgroup -> ( ${__sol2x_no_allegro} )
         install /etc/netgroup;
  

Experience

• worked for awhile.
• better than rdist alone.
• limited capabilities.
  • couldn't archive.
  • rdist still hangs forever if machines are down.
• exponential category explosion!
  • needed one directory hierarchy per machine type.
  • difficult to reconcile differences between files for machines in complex heterogeneous networks.
  • need an unlimited number of category names.
• blew away whole network with one mistake.
  • rdist configures whole network at once.
  • There has to be something better!

The flexible approach:

• hide features we don't need, add syntactic sugar, and actually call rdist: distr
• better syntax for host lists:

   sol2x= allegro presto apex forte vitesse
  

  These hosts run solaris 2.x for some x.

   sol2x_tftp= presto
  

  Presto supports the trivial file transfer protocol.

   sol2x_no_tftp= $sol2x - $sol2x_tftp
  

  The other machines don't.
• slightly better syntax for distribution commands:

   #cat master file     target           host list (with additions and deletions)
   auto etc/auto_master /etc/auto_master $sol26_nocache $sol27 - largo 
  

  In category auto, distribute Master/etc/auto_master as /etc/auto_master on all solaris 2.6 machines without cache space and all solaris 2.7 machines except for largo.

   inetd etc/inet/inetd.conf+2.5 /etc/inet/inetd.conf $sol25 - allegro presto agony
  

  In category inetd, distribute Master/etc/inet/inetd.conf+2.5 as /etc/inet/inetd.conf on all solaris 2.5 machines except for allegro, presto, and agony.

Distr behavior

• pings hosts to figure out whether they're up, then calls limited copies of rdist.
• keeps archives locally of distributed files.
• capable of copying backward from slave to master for archiving purposes.
• slow but safe.

Prototypes, archives, masters and slaves

• Distr operates on four kinds of data.
• PROTOTYPES are files that we plan to rdist.
• ARCHIVES are numbered 0000-9999 and hold revisions of prototypes. (as close as I ever got to a network version of CVS)
• MASTERS are seperate working files which are copied to prototypes when it's time to make changes.
• SLAVES are files on remote hosts that rdist updates.

Data movement

• Distr commands allow one to move data between these types as follows:

   (hand-edit)  :     (sent)==rdist=>(remote)
     MASTERS===m2p===>PROTOS<---s2p---SLAVES
        |       :      | ^       :      |        
        |       : -p2a | | -a2p  :      |        
        |       :      V |       :      |        
        +------m2a-->ARCHIVES<--s2a-----+   
                :   (0000-9999)  :
                :                :
  

  where : denotes possible filesystem boundaries, (PROTOS and ARCHIVES are always stored in the same filesystem).
• -m2p : master to prototype
• -p2a : prototype to archive
• -a2p : archive to prototype
• -p2s : prototype to slave
• -s2p : slave to prototype

Typical usage:

• edit masters
• archive current versions with -s2a.
• force download masters into slaves with -m2s.
• recover from stupid errors with -a2s.

Realities

• we did this for four years.
• Everyone here loved it.
• I threw it away! Scrapped the whole project!
• Several opposing forces caused its demise:
  • cost of configuration.
  • service level administration.
  • client-pull distribution strategy.

Problems with rdist/distr

• master/slave relationship: all configuration is on the master
  • inflexible
  • difficult to make special cases.
  • can't get slave contents up to master for archiving.
• server push is unscaleable.
  • configuring n machines takes n steps.
  • machines must be up first.
  • if a machine is down it gets skipped.
• limited command availability.
  • can only execute commands after distribution.
  • no way to make backups of current contents of overwritten files.
• certain files are always unique on every host and rdist works poorly.
  • /etc/passwd
  • /etc/services
• hosts know who they are and can decide which files they need.
  • don't need to build databases of host capabilities.
  • much less work to configure.

Service-level administration

• file contents are the wrong level at which to enforce system integrity.
• know which services the system should provide.
• each service modifies several files.

Cfengine

• I switched management of the network from rdist to cfengine
• written by collegue Mark Burgess (University of Oslo, Oslo, Norway)
• File distribution 'on steroids'
• 'Robot' determines actions based upon rules.
• Very efficient and fast.

Cfengine attributes

• 'client pull' - clients request files from master servers.
• multiple servers easily possible.
• clients manage their own configuration decisions.
• clients decide when to upgrade files (typically at boot time).
• exceptions to normal distribution policy can be configured on each client.
• advanced, extensible state machine with many more options than rdist.

Incremental compliance (rdist and cfengine)

• if something's alright, don't change it.
• if something's amiss, change only what you must.
• checking twice does no harm.

cfengine.conf:

• holds rules for various kinds of machines.
• machines figure out themselves what kind they are.
• no centralized machine database required.

Examples of configuration:

• 'control' section controls behavior:

   control:
         netmask = ( 255.255.255.0 ) 
  

  what's our network config?

         domain = ( eecs.tufts.edu ) 
  

  what's our network name?

         moduledirectory = ( /cf/modules ) 
  

  where are extensions?

         sysadm = ( couch@eecs.tufts.edu ) 
  

  who should know about changes?

         Repository = ( /var/local/cfengine/backup ) 
  

  where are backup files kept?

         AddInstallable = ( has_local_usr ) 
  

  what are modules?

         actionsequence = ( 
             module:has_local_usr
             directories
             copy
             links
             shellcommands
             processes
             tidy
         ) 
  

  what should I do for each machine?
• 'groups' section defines machine groups

   groups:
         server = ( allegro presto apex forte agony largo conmoto conbrio andante
    )
  

  These machines are servers

   
         # real-time tests of machine state 
         ssh_keygen = ( "/bin/test -f /etc/ssh_host_key" ) 
         ssh2_keygen = ( "/bin/test -f /etc/ssh2/hostkey" ) 
         cshpeople = ( "/bin/test -f /etc/csh.people" ) 
  

  Each one of these tests whether a particular file exists.
• 'editfiles': dynamic convergent editing of system files!

   editfiles:
         solaris::
           { /etc/inet/hosts
             AppendIfNoSuchLine "130.64.23.33 largo mailhost loghost timehost"
           } 
        solaris.jumpstart::
  

  conjunction: solaris AND jumpstart

           { /etc/defaultdomain
               AppendIfNoSuchLine "eecs.tufts.edu"
           } 
  

• 'directories': assert mode and owner of directories

   directories:
         # needed for proper function of sendmail
         /etc                        o=root m=go-w
         /etc/mail                   o=root m=go-w
         /usr                        o=root m=go-w
         /var                        o=root m=go-w
         /var/spool                  o=root m=go-w
         /var/spool/mqueue           o=root m=go-w
  

  These change the owner and protection of all these directories as noted.

       largo::
  

  Only on largo:

         /export/5/loc               o=root g=staff m=2755
         /export/6/loc               o=root g=staff m=2755
         /export/7/loc               o=root g=staff m=2755
  

  fix protections on exported filesystems.
• 'copy': copy files to specific places:

   copy:
       #   source file              target location      master copy  check type
       solaris::
           /admin/cf/.rhosts        dest=/.rhosts        server=largo type=sum
  

  • /admin/cf/.rhosts: location of master file
  • dest=/.rhosts: copy file to /.rhosts
  • server=largo: this is where the master file resides.
  • type=sum: copy file if checksum of master and target file differ.

         solaris.!agony.!largo::
             /admin/cf/etc/bootparams dest=/etc/bootparams server=largo type=sum
    

  • solaris.!agony.!largo:: all solaris machines except agony and largo (literally, solaris and (not agony) and (not largo)).
  • all three of these are determined at runtime, by an agent on the machine.
• 'links': make symbolic links if not present

   links:
       solaris::
           /etc/sendmail.cf        ->! mail/sendmail.cf
           /etc/csh.cshrc          ->! ./.cshrc
           /etc/csh.login          ->! ./.login
           /etc/csh.logout         ->! ./.logout
           /etc/rc2.d/S95cfd       ->! ../init.d/cfd
           /etc/rc2.d/K95cfd       ->! ../init.d/cfd
           /etc/motd               ->! /var/mail/Motd
  

  These links made for all solaris boxes.

       solaris.has_ssh2::
           /etc/rc2.d/S95ssh2       ->! ../init.d/ssh2
           /etc/rc2.d/K95ssh2       ->! ../init.d/ssh2
  

  These links made for all boxes serving ssh2.
• 'shellcommands': these are commands to execute in particular cases.

   shellcommands:
       solaris.Saturday.Hr00::
  

  Every saturday, on Solaris machines, if it's midnight(!)

         `/usr/bin/catman  -M /usr/openwin/share/man`
         `/usr/bin/catman  -M /usr/share/man`
         `/usr/bin/catman  -w`
  

  build manual page indexes.

       sunos_5_7.!largo.!sendmail_891::
  

  If we're running sunos 5.7 (Solaris 7) and we're not largo, and sendmail 8.9.1 isn't installed (as per above test)

         `/loc/mail/sendmail-8.9.1/INSTALL`
  

  install sendmail here!

         `/bin/domainname eecs.tufts.edu`
  

  Set our domain name.

         `/bin/echo "eecs.tufts.edu" > /etc/defaultdomain`
  

  Put into the default domain file.

Pros

• This is much easier.
• Each machine figures out what it needs.
• Only appropriate scripts are run.
• Can test on one machine at a time.
• Much easier to see if it works.
• configuration occurs very quickly.
• no need to give root access from master server to client.

Critique

• nonstandard boolean operator '.' means 'and'. Can be confusing, especially when standard meaning is substructure reference!
• very fast, but much must be done with extensions.

Where no moron^H^H^H^H^Hsysadmin has gone before...

• dream: configure by service rather than by file.
• already possible for application programs: rpm, etc.
• not currently reasonable for system configuration.

Why does rpm work for application programs?

• installation: splatter files around an OS.
• dependencies: certain files must exist (or perhaps not exist)
• outcome: 'package' gets 'installed' if dependencies are satisfied.
• uninstall: simply remove the files you installed, and move backup copies back in.
• reality: little customization is necessary during installation; users do that afterward.

Why won't rpm work for system services?

• much more customization required.
• dependencies require changes in existing files and databases (containing other important information)
• uninstall is difficult (because you have to undo PARTICULAR edits to system files).

What I in the summer of 1999...

• analyze weaknesses of Cfengine (particularly, studying which rules cannot be implemented inside its framework)
• write a system administration interface in Prolog to address those weaknesses.
• write Prolog scripts that emulate Cfengine features.
• study how this language can be used to replace Cfengine.

Power of Prolog

• declarative programming language.
• say what you want.
• don't say how to accomplish it.
• Prolog unification == Cfengine convergence (if it ain't broke don't fix it)

Initial Results

• cfengine equivalence.
• variables drive system variants
• service-level declarations.

Cfengine equivalence:

• cfengine code:

   links:
     solaris.victim::
       /etc/sendmail.cf ->!  mail/sendmail.cf
       /etc/services    ->!  inet/services
  

• translates into Prolog as:

   links:-solaris,victim,link('mail/sendmail.cf','/etc/sendmail.cf').
   links:-solaris,victim,link('inet/services','/etc/services').
  

  Literally:
  • If you're solaris,
  • and you're a victim,
  • then make sure the link exists.
• Similar translations for every kind of service:

   editfiles:
    ftp.solaris::
     { /etc/inet/inetd.conf
       AppendIfNoSuchLine "ftp stream tcp nowait root /usr/sbin/in.ftpd in.ftpd" 
     } 
  

• translates into Prolog as

   editfiles:-
     os(Os), 
     config_path('inetd.conf',Os,Path), 
     config_path('ftpd',Os,Ftpd),
     file_base_name(Ftpd,FBase),
     appendIfNoSuchLine(Path,
       [ftp,stream,tcp,nowait,root,Ftpd,Fbase]).
  

  Whoa there! What are the extra terms?
  • os(Os): figure out operating system.
  • config_path('inetd.conf',Os,Path): where's inetd.conf in this OS?
  • config_path('ftpd',Os,Ftpd): where's ftpd in this OS?

Principle of structural invariance

• Details of operating system structure can be separated from policy descriptions of what should happen.
• So we discover variables that relate to operating system structure dynamically, rather than hard-coding them.
• This simplifies the policy file, as it need only contain our demands, not details.

The wild frontier

• static configuration: those things that are invariant about a system.
  • shouldn't change.
  • convergence is easy to accomplish: change files back if they drift from what you want.
• static convergence easy to define and accomplish.
  • cfengine,rdist,make: all do this
  • need for customizations is limited.
• dynamic configuration: things that change over time as people use a system.
  • file sizes.
  • sizes of filesystems.
  • processes.
• dynamic convergence poorly defined.
  • most people write custom scripts to monitor and fix things.
  • difficult to write a general-purpose tool to handle these problems.

Dynamic policy in Prolog:

• write rules categorizing peoples' behavior:

   pig(Login):-
    passwd(Login,_,_,_,_,Home,_),
    du(Home,Usage),
    Usage>20000.
  

  Literally, "a person is a pig if that person has more than 20000 bytes in her or his home."
• specify actions for people matching a rule:

   ?- pig(Login),
      email(Login,
            'you are a pig!',
            'oink!'),
      fail.
  

  Literally, for every pig (caused by 'special' Prolog goal 'fail') email that pig a message (with subject 'oink!').

The not-so-final frontier

• Specify what, not how.
• Prolog determines how.
• Simpler than writing a script.
• Allows description of both static and dynamic policies.

What Burgess is up to now:

• Expand cfengine to handle dynamic policy enforcement.
• Game theory: select a `strategy' for updates.

Very fundamental observation:

• Resource management via quotas doesn't work.
• Problem is that there are many counter-strategies for getting around quotas.
• Burgess suggests that an active strategy is better: clean up peoples' directories for them when they store too much stuff.