System and Network Administration

lecture in color

Managing network integrity

Policy: limit what admins can do.
Enforcement: force systems into compliance with tools
Monitoring: watch systems carefully for changes.

Enforcing integrity

rdist: the remote distribution utility.

distributes files from a master server to local hosts.

 +---------------+  distribution +----------------+
 |               | ------------> |                |
 | master server |   requests    | client machine |
 |               |               |                |
 |  /etc/motd    |    status     |  /etc/motd     |
 +---------------+ <-----------  +----------------+
                      reports

 #[key]  [source] -> [ hosts  ]
 rhosts: /.rhosts -> ( presto )
       install /.rhosts;
 #[tab] [command][destination]

Rdist directives:

MACRO=value: set a macro
install /etc/motd: put the file in question into /etc/motd
notify couch@eecs.tufts.edu: mail a report of actions to this person
except foo: omit copying foo if copying a directory.
special "/usr/lib/newaliases" : run a command on the remote machine.

Example:

 SUN= allegro presto apex forte
 aliases: /etc/mail/aliases -> (${SUN}) 
       install /etc/mail/aliases
       special "/usr/sbin/newaliases"
       notify couch@eecs.tufts.edu

Rdist strengths:

won't update files unless needed (out of date or wrong protection)
easy to invoke.
very mature software (developed over many years)

Bugs (features):

hangs forever if a target host is down.
requires by default that the distributing host have root privilege to log into the target host. Can fix this by customizing rdist (kerberos)
there are two versions of rdist that don't talk to one another very well!

Usage problems:

must maintain the database
must keep a centralized list of hosts (manually)
must remember a rather large set of commands for its configuration file.
must remember a rather large set of keys describing distribution actions.

Integrity problems:

you have to remember what you modified and copy each modification to the server.
if you modify files as root, rdist can walk on your changes unless you keep master files 'in sync'.
you can't necessarily identify the files you need to propogate if you type a command.

Solution: wrappers (front-ends)

very difficult to implement another rdist.
to add features to a command, 'wrap' it with a command that calls it.
This wrapper can have its own databases and builds the Distfile used by rdist automatically.

Case study: wrapping rdist

rdist didn't cut it.
needed simpler interface.
needed revision control
result: distr1.0 (couch)

The fascist approach:

make everything conform to a specific format
worked (for awhile)

Simple example of wrapping:

interpret a database of files to distribute
generate an rdist control file 'Distfile' by what amounts to macro processing.
invoke rdist
largo:/admin/rdist/Dist.make: generate a Distfile for rdist from a database. This reads:
/admin/rdist/Dist.cats: categories for distribution actions:
```
 _allegro
```
shorthand for __allegro allegro: creates a one-machine category.
```
 __sol2x allegro presto apex forte vitesse 
```
all solaris 2.x boxes (for some x=4,5,6,7)
```
 __sol2x_no_allegro presto apex forte vitesse 
```
all of these except allegro, which must be treated specially.
/admin/rdist/Dist.keys: actual file distribution commands:
```
 # file                                        keyword
 __sol2x_no_allegro/etc/mail/aliases           aliases
```
- largo:/admin/rdist/__sol2x_no_allegro/etc/mail/aliases: source file
- presto:/etc/mail/aliases: one target file

Other distribution examples:

 # file                                        keyword
 __sol2x/etc/motd                              motd

On every sol2x box, place this /etc/motd.

 __sol2x_no_allegro/etc/netgroup               netgroup

Likewise on every sol2x box except allegro, put this /etc/netgroup.

Based upon these it creates Distfile:

 _allegro = ( allegro )
 __sol2x = ( allegro presto apex forte vitesse )
 __sol2x_no_allegro = ( presto apex forte vitesse )
 motd: __sol2x/etc/motd -> ( ${__sol2x} )
       install /etc/motd;
 aliases: __sol2x_no_allegro/etc/mail/aliases -> ( ${__sol2x_no_allegro} )
       install /etc/mail/aliases;
 netgroup: __sol2x_no_allegro/etc/netgroup -> ( ${__sol2x_no_allegro} )
       install /etc/netgroup;

Experience

worked for awhile.
better than rdist.
limited capabilities.
- couldn't archive.
- rdist still hangs forever if machines are down.
exponential category explosion!
- need one directory HIERARCHY per machine type.
- difficult to reconcile differences between files for machines in complex heterogeneous networks.
rdist configures whole network at once.
- blew away whole network with one mistake.
- There has to be something better!

The flexible approach:

hide features we don't need, add syntactic sugar, and actually call rdist: distr
better syntax for host lists:
```
 sol2x= allegro presto apex forte vitesse
```
These hosts run solaris 2.x for some x.
```
 sol2x_tftp= presto
```
Presto supports the trivial file transfer protocol.
```
 sol2x_no_tftp= $sol2x - $sol2x_tftp
```
The other machines don't.
better syntax for distribution commands:
```
 #cat master file     target           host list (with additions and deletions)
 auto etc/auto_master /etc/auto_master $sol26_nocache $sol27 - largo 
```
In category auto, distribute Master/etc/auto_master as /etc/auto_master on all solaris 2.6 machines without cache space and all solaris 2.7 machines except for largo.
```
 inetd etc/inet/inetd.conf+2.5 /etc/inet/inetd.conf $sol25 - allegro presto agony
```
In category inetd, distribute Master/etc/inet/inetd.conf+2.5 as /etc/inet/inetd.conf on all solaris 2.5 machines except for allegro, presto, and agony.

Distr behavior

pings hosts to figure out whether they're up, then calls limited copies of rdist.
keeps archives locally of distributed files.
capable of copying backward from slave to master for archiving purposes.
slow but safe.

Prototypes, archives, masters and slaves

Distr operates on four kinds of data.
PROTOTYPES are files that we plan to rdist.
ARCHIVES are numbered 0000-9999 and hold revisions of prototypes. (as close as I ever got to a network version of CVS)
MASTERS are seperate working files which are copied to prototypes when it's time to make changes.
SLAVES are files on remote hosts that rdist updates.

Data movement

Distr commands allow one to move data between these types as follows:

 (hand-edit)  :     (sent)==rdist=>(remote)
   MASTERS===m2p===>PROTOS<---s2p---SLAVES
      |       :      | ^       :      |        
      |       : -p2a | | -a2p  :      |        
      |       :      V |       :      |        
      +------m2a-->ARCHIVES<--s2a-----+   
              :   (0000-9999)  :
              :                :

where : denotes possible filesystem boundaries, (PROTOS and ARCHIVES are always stored in the same filesystem).

-m2p : master to prototype
-p2a : prototype to archive
-a2p : archive to prototype
-p2s : prototype to slave
-s2p : slave to prototype

Typical usage:

edit masters
archive current versions with -s2a.
force download masters into slaves with -m2s.
recover from stupid errors with -a2s.

Realities

we did this for four years.
Everyone here loved it.
I threw it away! Scrapped the whole project!
Several opposing forces caused its demise:
- cost of configuration.
- service level administration.
- client-pull distribution strategy.

Problems with rdist/distr

master/slave relationship: all configuration is on the master
- inflexible
- difficult to make special cases.
- can't get slave contents up to master for archiving.
server push is unscaleable.
- configuring n machines takes n steps.
- machines must be up first.
limited command availability.
- limited to commands AFTER distribution.
- no way to make backups of current contents
certain files are always unique on every host and rdist works poorly.
- /etc/passwd
- /etc/services
hosts know who they are and can decide which files they need.
- don't need to build databases of host capabilities.
- much less work to configure.

Service-level administration

file contents are the wrong level at which to enforce system integrity.
know which services the system should provide.
each service modifies several files.

Cfengine

I switched management of the network from rdist to cfengine
written by collegue Mark Burgess (University of Oslo, Oslo, Norway)
File distribution 'on steroids'
'Robot' determines actions based upon rules.
Very efficient and fast.

Cfengine attributes

'client pull' - clients request files from master servers.
multiple servers easily possible.
clients manage their own configuration decisions.
clients decide when to upgrade files (typically at boot time).
exceptions to normal distribution policy can be configured on each client.
advanced, extensible state machine with many more options than rdist.

Incremental compliance (rdist and cfengine)

if something's alright, don't change it.
if something's amiss, change only what you must.
checking twice does no harm.

cfengine.conf:

holds rules for various KINDS of machines.
machines figure out THEMSELVES what kind they are.
much simpler configuration.

Examples of configuration:

'control' section controls behavior:

 control:
       netmask = ( 255.255.255.0 )

what's our network config?

       domain = ( eecs.tufts.edu )

what's our network name?

       moduledirectory = ( /cf/modules )

where are extensions?

       sysadm = ( couch@eecs.tufts.edu )

who should know about changes?

       Repository = ( /var/local/cfengine/backup )

where are backup files kept?

       AddInstallable = ( has_local_usr has_local_cache get_subnet )

what are modules?

       actionsequence = ( 
           module:has_local 
           module:get_subnet
           directories
           copy
           links
           shellcommands
           processes
           tidy
       )

what should I do for each machine?

'groups' section defines machine groups

 groups:
       server = ( allegro presto apex forte agony largo conmoto conbrio andante
  )

These machines are servers

 
       # real-time tests of machine state 
       ssh_keygen = ( "/bin/test -f /etc/ssh_host_key" ) 
       ssh2_keygen = ( "/bin/test -f /etc/ssh2/hostkey" ) 
       had_ssh2 = ( "/bin/test -f /etc/init.d/ssh2" )
       had_ssh = ( "/bin/test -f /etc/init.d/ssh" )
       ntpd = ( "/bin/test -f /etc/init.d/ntpd" )
       sendmail_891 = ( "/bin/test -f /etc/mail/sendmail.cf-pre8.9.1" ) 
       nisplus = ( "/bin/test -f /var/nis/NIS_COLD_START" ) 
       cshpeople = ( "/bin/test -f /etc/csh.people" )  # dert 1999-09-05

Each one of these tests whether a particular file exists.

'editfiles': dynamic convergent editing of system files!

 editfiles:
       solaris::
         { /etc/inet/hosts
           AppendIfNoSuchLine "130.64.23.33 largo mailhost loghost timehost"
         } 
      solaris.jumpstart::

conjunction: solaris AND jumpstart

         { /etc/defaultdomain
             AppendIfNoSuchLine "eecs.tufts.edu"
         }

'directories': assert mode and owner of directories

 directories:
       # needed for proper function of sendmail
       /etc                        o=root m=go-w
       /etc/mail                   o=root m=go-w
       /usr                        o=root m=go-w
       /var                        o=root m=go-w
       /var/spool                  o=root m=go-w
       /var/spool/mqueue           o=root m=go-w

These change the owner and protection of all these directories as noted.

     largo::

Only on largo:

       /export/5/loc               o=root g=staff m=2755
       /export/6/loc               o=root g=staff m=2755
       /export/7/loc               o=root g=staff m=2755

fix protections on exported filesystems.

'copy': copy files to specific places:
```
 copy:
     #   source file              target location      master copy  check type
     solaris::
         /admin/cf/.rhosts        dest=/.rhosts        server=largo type=sum
```
- /admin/cf/.rhosts: location of master file
- dest=/.rhosts: copy file to /.rhosts
- server=largo: this is where the master file resides.
- type=sum: copy file if checksum of master and target file differ.
```
     solaris.!agony.!largo::
         /admin/cf/etc/bootparams dest=/etc/bootparams server=largo type=sum
```
- solaris.!agony.!largo:: all solaris machines except agony and largo (literally, solaris and (not agony) and (not largo)).
- all three of these are determined at RUNTIME, by an agent on the machine.

'links': make symbolic links if not present

 links:
     solaris::
         /etc/sendmail.cf        ->! mail/sendmail.cf
         /loc1                   ->! /loc
         /loc2                   ->! /loc
         /loc3                   ->! /loc
         /etc/csh.cshrc          ->! ./.cshrc
         /etc/csh.login          ->! ./.login
         /etc/csh.logout         ->! ./.logout
         /etc/rc2.d/S95cfd       ->! ../init.d/cfd
         /etc/rc2.d/K95cfd       ->! ../init.d/cfd
         /etc/motd               ->! /var/mail/Motd

These links made for all solaris boxes.

     solaris.(had_ssh|has_ssh).!had_ssh2.!has_ssh2::
         $(a)/etc/rc2.d/S95ssh       ->! ../init.d/ssh
         $(a)/etc/rc2.d/K95ssh       ->! ../init.d/ssh

These links made for all boxes serving ssh1 but not ssh2 (forte).

     solaris.(had_ssh2|has_ssh2)::
         $(a)/etc/rc2.d/S95ssh2       ->! ../init.d/ssh2
         $(a)/etc/rc2.d/K95ssh2       ->! ../init.d/ssh2

These links made for all boxes serving ssh2.

'shellcommands': these are commands to execute in particular cases.
```
 shellcommands:
     solaris.Saturday.Hr00::
```
Every saturday, on Solaris machines, if it's midnight(!)
```
       `/usr/bin/catman  -M /usr/openwin/share/man`
       `/usr/bin/catman  -M /usr/share/man`
       `/usr/bin/catman  -w`
```
build manual page indexes.
```
     sunos_5_7.!largo.!sendmail_891::
```
If we're running sunos 5.7 (Solaris 7) and we're not largo, and sendmail 8.9.1 isn't installed (as per above test)
```
       `/loc/mail/sendmail-8.9.1/INSTALL`
```
install sendmail here!
```
     solaris.!largo.!nisplus::
```
If we're running solaris and we're not largo, and don't have nisplus installed, INSTALL IT!
```
       `/bin/domainname eecs.tufts.edu`
```
Set our domain name.
```
       `/bin/echo "eecs.tufts.edu" > /etc/defaultdomain`
```
Put into the default domain file.
```
       `/usr/sbin/nisinit -c -H largo`
```
Run nisinit to bind us into eecs.tufts.edu!

Pros

This is MUCH EASIER.
Each machine figures out WHAT IT NEEDS.
Only appropriate scripts are run.
Can test on ONE MACHINE at a time.
Much easier to see if it works.
configuration occurs very quickly.
no need to give root access from master server to client.

Critique

nonstandard boolean operator '.' means 'and'. Can be confusing, especially when standard meaning is substructure reference!
very fast, but
much must be done with extensions.

Where no moron^H^H^H^H^Hsysadmin has gone before...

dream: configure by SERVICE rather than by FILE.
already possible for application programs: rpm, etc.
not currently reasonable for system configuration.

Why does rpm work for application programs?

installation: splatter files around an OS.
dependencies: certain files must exist (or perhaps NOT exist)
outcome: 'package' gets 'installed' if dependencies are satisfied.
uninstall: simply remove the files you installed, and move backup copies back in.
reality: little customization is necessary during installation; users do that afterward.

Why won't rpm work for system services?

much more customization required.
dependencies require changes in EXISTING FILES and databases (containing other important information)
uninstall is difficult (because you have to undo PARTICULAR edits to system files).

What I did last summer....

analyze weaknesses of Cfengine (particularly, studying which rules cannot be implemented inside its framework)
write a system administration interface in Prolog to address those weaknesses.
write Prolog scripts that emulate Cfengine features.
study how this language can be used to replace Cfengine.

Power of Prolog

declarative programming language.
say what you want.
don't say how to accomplish it.
Prolog unification == Cfengine convergence (if it ain't broke don't fix it)

Initial Results

cfengine equivalence.
variables drive system variants
service-level declarations.

Cfengine equivalence:

cfengine code:

 links:
   solaris.victim::
     /etc/sendmail.cf ->!  mail/sendmail.cf
     /etc/services    ->!  inet/services

translates into Prolog as:

 links:-solaris,victim,link('mail/sendmail.cf','/etc/sendmail.cf').
 links:-solaris,victim,link('inet/services','/etc/services').

Literally:

If you're solaris,
and you're a victim,
then make sure the link exists.

Similar translations for every kind of service:

 editfiles:
  ftp.solaris::
   { /etc/inet/inetd.conf
     AppendIfNoSuchLine "ftp stream tcp nowait root /usr/sbin/in.ftpd in.ftpd" 
   }

translates into Prolog as
```
 editfiles:-
   os(Os), 
   config_path('inetd.conf',Os,Path), 
   config_path('ftpd',Os,Ftpd),
   file_base_name(Ftpd,FBase),
   appendIfNoSuchLine(Path,
     [ftp,stream,tcp,nowait,root,Ftpd,Fbase]).
```
Whoa there! What are the extra terms?
- os(Os): figure out operating system.
- config_path('inetd.conf',Os,Path): where's inetd.conf in this OS?
- config_path('ftpd',Os,Ftpd): where's ftpd in this OS?

Principle of configuration invariance

Details of OPERATING SYSTEM structure can be separated from POLICY descriptions of what should happen.
So we discover variables that relate to operating system structure dynamically, rather than hard-coding them.

The wild frontier

static configuration: those things that are INVARIANT about a system.
- shouldn't change.
- convergence is easy to accomplish: change files BACK if they drift from what you want.
static convergence easy to define and accomplish.
- cfengine/rdist/make: all do this
- need for customizations is limited.
dynamic configuration: things that change over time as people use a system.
- file sizes.
- sizes of filesystems.
- processes.
dynamic convergence poorly defined.
- most people write custom scripts to monitor and fix things.
- difficult to write a general-purpose tool to handle these problems.

Dynamic policy in Prolog:

write rules categorizing peoples' behavior:
```
 pig(Login):-
  passwd(Login,_,_,_,_,Home,_),
  du(Home,Usage),
  Usage>20000.
```
Literally, "a person is a pig if that person has more than 20000 bytes in her or his home."
specify actions for people matching a rule:
```
 ?- pig(Login),
    email(Login,
          'you are a pig!',
          'oink!'),
    fail.
```
Literally, FOR EVERY PIG (caused by 'special' Prolog goal 'fail') email that pig a message (with subject 'oink!').

The not-so-final frontier

Specify WHAT, not HOW.
Prolog determines HOW.
Simpler than writing a script.
Allows description of both static and dynamic policies.

Imaging:

Decide what a good system 'should look like'.
By hand, build an 'image' of what files should contain.
'Distribute' files to all your machines.
So all machines 'look alike' and function similarly.

Imaging systems:

By file: distribute certain files to all systems.
By software package: incorporate software packages into a system.
By service: enable or disable network services.

lecture in color

/comp/150NET/notes/rdist-old.php
downloaded on Nov-23-2009 03:29:48 PM,
was last modified on Feb-17-2004 10:49:26 PM.
All lecture note content is copyright 2004 by
Alva L. Couch, Computer Science, Tufts University