Integrity

lecture in color

So far, we know:

How to build a system
How to compile and install software
How to set up a user environment.
How to configure a network.
How to set up services.
But will it stay the way we set it up?

Systems Administration Maturity Model

'How well' the process of managing systems works.
Level 1: ad-hoc: randomly follow scripts.
Level 2: document changes.
Level 3: document 'best practices' for others to follow.
Level 4: assure enterprise-wide compliance with 'best practices'

The goal: "System Integrity"

System (or network) does what you told it to.
It doesn't stop doing it.
No matter what.
Even with you around...

Maintaining integrity:

Revision control: how do you remember changes made to a system?
Change control: how do you assure that changes take place in an orderly manner, with the ability to undo them?
Cookie cutters/cloning: replicating successful approaches.
Change detection: how do assure that others do not compromise your systems?

Revision control:

you're changing a lot of files.
you can't necessarily remember the changes.

Types of revision control:

by file
by directory hierarchy
by computer
by network

Revisions of files

Old: Source Code Control System (SCCS).
Current: Revision Control System (RCS): Walter F. Tichy, Purdue.
keeps a `revision history' of changes made to a file.
maintains that history in minimal space.

RCS Commands:

ci
- check in, save a copy of a file.
co
- check out, get a saved file.
rcs
- revision control system, changes the state of a file.
rlog
- revision log, gives the history of revisions made to a file.
rcsdiff
- revision diff, allows you to see the differences between revisions.

How to use RCS

Make an RCS directory
```
 % mkdir RCS   
```
This is a directory to hold revisions. This should be at the level of the file you are working on, say foo.
When starting up perform this command:
```
 % ci foo
```
"Check in" the current contents of foo as a revision. You will be asked to comment on the revision. Revisions of foo are stored in the file RCS/foo,v. foo is deleted by this command. Don't feel bad, just check it out as noted below.
To get a read-only copy, type:
```
 % co foo
```
"Check out" the most current copy of foo from RCS/foo,v. This is a read-only version that you can compile but can't edit.
To edit changes into the file, use
```
 % co -l foo
```
This command locks the file so that only you can check it back in. You should edit it and then check it in so that others may edit it as well. To check it back in use the ci command.
If you want to continue editing after posting changes, use
```
 % ci -l foo
```
This checks in the file and keeps it locked to you.
To break a lock set by another person:
```
 % rcs -u foo
```
This unlocks the file, sending mail to the offender who locked it.
```
 % co -l foo
```
This checks the file out so you can edit it.
To revert a file to a previous revision:
```
 % rlog foo
```
Make a listing of all revisions in RCS/foo,v. This includes all comments given during updating.
```
 % co -r2.3 foo
```
Check out revision 2.3 of foo from RCS/foo,v. Revision numbers are given in the output of rlog.
```
 % rcs -l foo
```
Lock the current revision of foo.
```
 % chmod u+w foo
```
Make foo editable.
```
 % ci foo
```
Make the former revision the current one.
To see what's different about the current file and revision 1.2,
```
 % rcsdiff -r1.2 foo | more
```

RCS comments

saving revisions is very efficient: typically one can store 40-50 revisions in the space of two copies of a file.
revisions are stored in reverse differential form.
- start with current file
- save changes needed to make previous file
- save changes needed to make file before that.
- ...
utilizes command diff to compute differences.

Using RCS in System Administration

back up copies of system files as you change them
- /etc/hosts
- /etc/inetd.conf
- /etc/passwd
back up original versions of software that you modify

RCS design

based upon diff -e and sed
Basic game: store differences between files rather than the files themselves.
Line-oriented differences only (store lines that differ).

Diff

command reports differences in files.
several options useful for software maintenance.

-c: create a "context diff" showing how changes fit in context.

then one can use "patch" to update a file like the first to a file like the second!

Example, if file1 contains

 This is a test.
 This is another test. 
 This is a third test.  
 This is a fourth test.

and file2 contains

 This is a test.
 This is another test. 
 This is a dumb test.  
 This is a fourth test. 
 This is a fifth test.

then

 diff -c file1 file2

produces

 *** file1       Tue Feb 29 14:28:06 2000
 --- file2       Tue Feb 29 14:28:19 2000
 ***************
 *** 1,4 ****
   This is a test.
   This is another test. 
 ! This is a third test.  
   This is a fourth test. 
 --- 1,5 ----
   This is a test.
   This is another test. 
 ! This is a dumb test.  
   This is a fourth test. 
 + This is a fifth test.

-e: create a script of sed commands that'll change one file into another!
- then one can use sed to change one file+ the diffs into the other!
- used by RCS to compress revision histories!
- Example: for the same two files above,
```
 diff -e file1 file2 
```
  produces:
```
 4a
 This is a fifth test. 
 .
 3c
 This is a dumb test.  
 .
```
- So if we type
```
 diff -e file1 file2 >diffs
 sed -f diffs <file1 >file3
```
  then file2 and file3 will be the same.

Building RCS from Diff and Sed

Always store the most current revision and the sed commands to go backward.
Suppose v1, v2, v3, v4 are versions of a file and dij is the output diff -e vi vj
Then:
- the RCS file for all the versions contains v4, d43, d32, and d21
- one can get to any version by a chain of sed commands.

Example:

 sed -f d43 <v4 | sed -f d32 | sed -f d21

outputs the original version v1!

Patch

Problem: have to reload whole software tree even to incorporate minor changes.
Approach: send only the changes.
patch: applies changes sent in an email message to a large hierarchy of files.
Written by Larry Wall (Perl inventor)

Using Patch

implementor makes two trees of files, 'old' and 'new'
you have 'old'.
'fixed' version is 'new'
implementor executes
```
 diff -r -c old new >patch01
```
implementer mails patch01 to you (or places it on the Internet)
you execute:
```
 cd old
 patch -p 1 <patch01
```
and all files change from the old to new versions!

Problems with single file version control

'version skew': trying to use incompatible versions of several files together.
Files in a hierarchy often must change together.
Need a tool that keeps changes 'in sync' for a whole tree of files.

Tree Revision control: CVS

Concurrent Version System
On top of RCS
Stores revisions for a hierarchy of files.

Using CVS

Many commands starting with 'cvs'
```
 cvs checkout <module>
```
makes a private copy of the module for you to edit. cd there and then:
```
 cvs update
```
gets new changes from others.
```
 cvs add file
```
adds a new file to the module in your private working tree.
```
 cvs remove file
```
removes a file from the module in your private working tree.
```
 cvs commit file
```
copies a file from your private working tree to the master tree.
Involved startup to create modules.
Once started, many people can cooperate on creating modules.
Network version: "cvs checkout" now works over the Internet.

Limitations of RCS and CVS

very inefficient on binary files (RCS algorithm depends upon text file structure)
only works when files are located together in the filesystem.
Reality: UNIX changes are distributed throughout the filesystem.
Dream: RCS/CVS for systems, not just software.
Reality: must settle for discipline, automated checking, and automated synchronization rather than revision control.

The big compromise:

If you can't beat "them", control yourself, detect them, and change stuff back.
discipline: control what goes into your system
automated checking: find out what changed.
automated synchronization: change things back if they change inadvertently.

Integrity maintenance:

of a machine
of a network

Approaches:

policies that control what admins do. (root can work around any tool's limits)
tools that enforce state (such as rdist, cfengine)
tools that check integrity (such as tripwire and aide)

Example: integrity of a machine (by policy)

Thou shalt not install files unnecessarily in / or /usr.
Thou shalt keep records of files actually installed therein.
Thou shalt maintain /local/bin as list of commands to be put in users' paths.
Thou shalt never install any file in /local/bin, always use symlinks to disjoint packages.
Thou shalt keep records of the links thou installest in /local, in /loc/slink/slink.conf.
Thou shalt compile and install packages as yourself, in disjoint filesystem subtrees (of /loc).
Thou shalt not upgrade or delete existing packages without checking with users.

SLINK: my way of maintaining links (there are several others)

link target source: makes source appear within target.
copy target source: copies files.
Examine allegro:/local/bin, /loc/slink/slink.conf for details.

Tools and policy:

root can do anything.
policy must control what root may do.
one approach: make tools sensitive to policies:
Examine allegro:/loc/slink/slink.mod for the modification policies for EECS.

My contribution

(what I learned through years of writing tools to maintain software repositories):
Tools should not enforce policies (which leads to disgruntled admins fighting tools) but should reinforce them (by making it easier to comply than to dissent).

Case history of SLINK approaches:

no slink: put everything in /usr/local
- ran for 5 years.
- massive corruption in /usr/local
- had to start over.
Slink1.0 (also CMU depot) the fascist approach:
- you have to name your package subdirs in a particular way.
- all subdirs with specific names get linked into /local
- result: CHAOS. Took Schmolze a year to figure out how to install Emacs under the scheme.
- Software installation of all but the simplest packages came to grinding abrupt halt. (while corruption continued when we had to install things)
Slink2.0: the anarchist's approach:
- you get to name software the way you want.
- you have to tell slink how.
- result: works, but happily corrupts things you don't intend if you mistype. e.g., no cd command, then
```
 link /loc/fool/bin bin
```
  links stuff into /bin in the root partition!
Slink5.0: incorporating policy decisions
- you can tell slink your policy on modifications (2000 lines of perl to figure this out. Sigh)
- it won't do things against your wishes.
- no possibility of corruption even as root (unless you intend it)

EECS Policy advantages and disadvantages:

can more easily rebuild systems and upgrade system software.
must fight with vendor software over where it gets installed. (sometimes we lose. see /interleaf on every host)
must keep detailed lists of vendor filesystem modifications.
more short-term work to modify Makefiles to install in local hierarchies.
less long-term work in deleting packages that aren't useful.
can create multiple user environments from the same packages.

Multiple environments:

/local/bin: normal user commands
/local/old/bin: old (deprecated) commands
/local/new/bin: new (untested) commands

Software revision control:

what software is available to users?
are all files associated with software the same version?
revision skew: some files in some version, some files in another.

EECS software revision policy (old)

put new stuff into /local/new/bin,
when tested, put into /local/bin,
while moving the old version to /local/old/bin,
and removing links to any version in /local/old/bin.
if no one complains, move unlinked source files to Trash (hidden directory on largo).
if no one complains about this, delete the Trash.

try unlinking the version in /local/old/bin. if no one complains, delete it.

                         initial
 /loc/adm/foo1.0/bin/foo -------> /local/new/bin 
                         tested
                         -------> /local/bin 
                         replaced
                         -------> /local/old/bin

Beyond revision control: detection

Theory: you're in control. Nothing changes till you say so.
*Practice*: not!
- multiple admins
- hackers.
- too little sleep.

Change detection

figure out who changed systems
be able to change them back

Monitoring static state

static state: what files look like
procedure: snapshot and compare
- snapshot: make a picture of what filesystems look like.
- compare: check that against real state, flag problems.

Tripwire

scan the whole filesystem
compute a 'signature'
scan again, compare new signature with old.
If signatures don't match, you have a problem!

Realities of scanning:

some files should change drastically.
some files should become larger, e.g., accounting logs.
some files should appear and disappear, or change state.

Configuring Tripwire

determine files to scan.
determine what to look for.
hide scan files from hackers so they can't change them.

Mode options: (from andante:/loc/adm/tripwire-1.2/data/tw.config)

these specify what to watch:

 #     - :  ignore the following atributes
 #     + :  do not ignore the following attributes
 #
 #     p :  permission and file mode bits      a: access timestamp
 #     i :  inode number                       m: modification timestamp
 #     n :  number of links (ref count)        c: inode creation timestamp
 #     u :  user id of owner                   1: signature 1
 #     g :  group id of owner                  2: signature 2
 #     s :  size of file

Most of these correspond to inode contents (pinugsamc).
A couple are signatures of the file's contents.

example:
```
 /etc    +pinugsm12-a
```
means to pay attention to everything but access timestamp for files in /etc.
! in front of a file means ignore, = means don't descend
```
 !/tmp
```
don't do anything with /tmp
```
 =/usr         R
```
check the directory /usr, but not subdirectories.
access timestamp: used to build 'jails' for hackers! Detects reading of a file.

shorthands:

 # Templates:  (default)       R :  [R]ead-only (+pinugsm12-a)
 #                             L :  [L]og file (+pinug-sam12)
 #                             N :  ignore [N]othing (+pinusgsamc12)
 #                             E :  ignore [E]verything (-pinusgsamc12)

Example file (for Solaris)

 # root's files
 =/                            L
 /.rhosts                      R       # may not exist
 /.profile                     R       # may not exist
 /.cshrc                       R       # may not exist
 /.login                       R       # may not exist
 /.logout                      R       # may not exist
 /.forward                     R       # may not exist
 /.netrc                       R       # may not exist
 
 # devices
 /dev                          L
 /devices                      L
 =/devices/pseudo              L
 
 # system databases
 /etc                          R
 /etc/default                  R
 /etc/dfs/dfstab               R
 /etc/dfs/sharetab             R
 /etc/dumpdates                L
 /etc/group                    R       # changes should be infrequent
 /etc/hosts.equiv              R
 /etc/inet/inetd.conf          R
 /etc/inet/protocols           R
 /etc/inet/services            R
 /etc/init.d                   R
 /etc/motd                     L
 /etc/opt                      R
 /etc/passwd                   L
 /etc/profile                  R
 /etc/remote                   R
 /etc/rmtab                    L
 /etc/rpc                      R
 =/etc/saf                     L
 /etc/shadow                   L
 /etc/system                   R
 /etc/ttydefs                  L
 /etc/ttysrch                  R
 
 # other sensitive directories 
 /hsfsboot                     R
 /kernel                       R
 /opt                          R
 /sbin                         R
 /ufsboot                      R
 /usr/sbin                     R
 =/var                         L
 =/var/adm                     L
 /var/adm/utmp                 L
 /var/adm/wtmp                 L
 /var/adm/wtmpx                L
 /var/adm/sulog                L
 =/var/adm/sa                  L
 =/var/spool                   L
 
 # Checksumming the following is not so critical.  However,
 #  setuid/setgid files are special-cased further down.
 =/usr                         L
 /usr/aset                     R-2
 /usr/bin                      R-2
 /usr/ccs                      R-2
 /usr/kernel                   R-2
 /usr/lib                      R-2
 /usr/ucb                      R-2
 /usr/openwin/bin              R-2
 
 # temporary filesystems
 =/tmp                         L
 =/var/tmp                     L
 =/proc                        L
 
 # set-user-id files must be checksummed 
 /usr/bin/at                   R
 /usr/bin/atq                  R
 /usr/bin/atrm                 R
 /usr/bin/chkey                R
 /usr/bin/crontab              R
 /usr/bin/ct                   R
 /usr/bin/cu                   R
 /usr/bin/eject                R
 /usr/bin/login                R
 /usr/bin/mail                 R
 /usr/bin/mailx                R
 /usr/bin/netstat              R
 /usr/bin/newgrp               R
 /usr/bin/nfsstat              R
 /usr/bin/passwd               R
 /usr/bin/ps                   R
 /usr/bin/rcp                  R
 /usr/bin/rsh                  R
 /usr/bin/rdist                R
 /usr/bin/rlogin               R
 /usr/bin/su                   R
 /usr/bin/tip                  R
 /usr/bin/uucp                 R
 /usr/bin/uuglist              R
 /usr/bin/uuname               R
 /usr/bin/uustat               R
 /usr/bin/uux                  R
 /usr/bin/volcheck             R
 /usr/bin/w                    R
 /usr/bin/write                R
 /usr/bin/yppasswd             R
 /usr/ucb/ps                   R

What tripwire does with this:

make snapshots of system status
```
 tripwire -initialize
```
compare snapshots with current state
```
 tripwire
```

what snapshots look like:

 /dev/sad/user 9 001..W 120777 16030 1 0 1 31 0unNQn 0qY.u9 0qY.u9 0 0 0 0 0 0 0

0 0 0

 /dev/sad/admin 9 001..W 120777 16031 1 0 1 32 0unNQn 0qY.u9 0qY.u9 0 0 0 0 0 0 0