SLINK: create virtual user images of directory hierarchies. -by- Alva L. Couch, Greg Owen Associate Prof. of EE/CS Xerox Information Systems Department of EE/CS Peabody, MA 02146 Tufts University, gowen@xis.xerox.com Medford, Massachusetts, 02155. couch@cs.tufts.edu Copyright (C) 1994 by Alva L. Couch and Greg Owen This file is part of SLINK SLINK is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2, or (at your option) any later version. SLINK is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with GNU CC; see the file COPYING. If not, write to the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA. ______________________ Slink the Link Manager SLINK is a program that implements a generalization of a symbolic link, called a `slink'. Unlike a symbolic link, which asserts the equivalence of single files, a slink asserts containment relationships for whole trees of files. We write A=>B to mean there's a slink from directory A to directory B, which means that A contains links into B that make B appear to be a substructure of A. If also A=>C, then A embodies the union of the directory hierarchies B and C. Slink-2.0 is written in perl-4 and works on 4.2 BSD, System VR4, and later UNIX systems supporting the concept of a symbolic link. It has been tested extensively under perl-4.036 and perl-5.001m on SunOs(SUN), Solaris(SUN), IRIX(SGI), Linux(PC),and OSF(dec) UNIXes. Slink was developed in order to construct virtual working trees of installed software from several disparate, private software trees maintained by several users. If we have trees /alfred/bin/emacs lib/emacs man/man1/emacs.1 /baker/bin/ci co rcs lib/rdiff3 man/man1/ci.1 co.1 rcs.1 /charlie/bin/xarchie man/man1/xarchie.1 and another empty directory /local, and we slink: /alfred <= /local /baker <= /local /charlie <= /local then /local is a merge of all of the structure of the three disparate trees: /local/bin/emacs ci co rcs xarchie lib/emacs rdiff3 man/man1/emacs.1 ci.1 co.1 rcs.1 xarchie.1 Thus users can refer to the combined tree /local, while installers work autonomously in their private trees that /local references. In this way we completely eliminate conflict between installers, while maintaining the illusion that they work in the same space. ____________________ Properties of Slinks A `slink' is a directive to symbolically map one subdirectory onto another without telling how that mapping is to be done or which particular symbolic link should be used. For example, the slink /loc/adm/slink-1.0/sbin/slink <= /local/sbin/slink (which installs slink itself on our system) says that by some mechanism to be determined, the file /loc/adm/slink-1.0/sbin/slink should be accessible as /local/sbin/slink. The way this is accomplished is left up to the SLINK program itself. For example, it could form the physical symbolic link /loc/adm/slink-1.0 <- /local or /loc/adm/slink-1.0/sbin <- /local/sbin or /loc/adm/slink-1.0/sbin/slink <- /local/sbin/slink at its own discretion. For now, slink's only mechanism for creating slinks is to use normal symbolic links, but one should not limit it to this mechanism; it eventually might be able to use other mapping mechanisms, such as filesystem mounts. In this discussion, we will always write slinks from B to A as A <= B in accordance with the syntax of slink, which will create A <= B if you type slink -f A B As another example, the slink /loc/adm/slink-1.0/sbin <= /local/sbin says that the things in the directory /loc/adm/slink-1.0/sbin (e.g. the command truename) should be accessible from the directory /local/sbin, e.g., /loc/adm/slink-1.0/bin/truename <= /local/sbin/truename This relationship holds for each file or directory within /loc/adm/slink-1.0/bin to arbitrary recursive depth. In other words, a slink at a particular level of the directory hierarchy implies slinks at all subdirectories, i.e., the slink source <= target implies all slinks of the form source/path <= target/path for each path existing in source. The converse is not true, but if we say source/path <= target/path slink reserves the right to slink source <= target at its convenience. In other words, relationships between subdirectories of slinked directories are guaranteed, while relationships between parents of the same name are possible but not guaranteed. Given a slink directive, the program slink chooses the appropriate method of accomplishing the slink, so that a) access to the source file involves traversing exactly one symbolic link. Paths involving multiple links are resolved into a single one. b) all slinks to the same target path are merged into a coherent target tree, barring naming conflicts, so that the target is a symbolic union of all sources. c) a minimum number of links are used to form this union. ______________________________ Promotions and optimal linking Slink uses a `promotion' algorithm to minimize the links it uses. a) If trying to make a link to a target directory, and no current link exists, make a single link to the target from the source tree. b) If a current link exists, and points to the correct target directory, leave the link alone. c) If a current link exists and points to a different place, then i) rm the link. ii) replace it with a directory of the same name iii) make links in that directory to the members of the original target directory. iv) add links in that directory to the members of the new target directory. Obviously there are a lot of nonsensical things that slink won't do, including `promoting' a file to be a directory. etc. If you ask it to do something nonsensical, it will simply refuse to do so and continue processing without doing so. ___________ Using slink Synopsis slink [options] [ [ []]] makes links from configuration file matching the given patterns slink -condense [options] [ [ []]] Condense image tree by removing redundant directories slink -link [options] [] Forces creation of a slink not in the configuration file slink -unlink [options] [] Forces deletion of a slink not in the configuration file slink -clean [options] [...] Clean image tree by removing images to nonexistent nodes slink -check [options] [...] check for possible image tree corruption slink -help print this message only OPTIONS: -verbose: print verbose description of actions. -quiet: don't echo errors to controlling terminal. -debug: print debugging trace -nolock: don't lock out multiple instances -nomap: don't read or write mapfile -remap: don't read mapfile initially -root : specify a prefix for file names -confile : specify a new configuration file. -errfile : specify a new error file. -logfile : specify a new log file. -lokfile : specify a new lock file. -newfile : specify a new `new link' file. The program slink can be used in two ways. a) standalone, to make a specific slink or group of slinks. b) configuration mode, to maintain a complex hierarchy of slinks via a configuration file. In the first case, the command slink -link sourcename slinkname echos the function of `ln -s', except that it is permissible to specify two directories and the second will thereafter contain the contents of the first as well as perhaps other nodes. In the second case, a configuration file is edited that contains all the slinks to be made. In our case this file is /loc/slink.conf. The identity of this file may be changed by editing the source to the slink command itself. To use the configuration file, type simply slink to assure that all slinks in the file are up to date, or slink [source-pattern [slink-pattern]] to update links that match particular source patterns or target patterns in the configuration file. At this point the only patterns supported are literal substring matches. For example, if the configuration file contains /loc/adm/slink-1.0/sbin /local/sbin /loc/adm/foo-1.0 /local then the command slink foo will update the links that assure that /loc/adm/foo-1.0 <= /local but ignore the links that assure that /loc/adm/slink-1.0/sbin <= /local/sbin ____________________ Maintenance commands Slink can also perform several maintenance functions to clean up after itself and respond to changes in the system. These functions are not done automatically because of the danger of affecting people using the slink tree. The syntax slink -check {slink-tree} will inform you of possible problems in the link tree, including dangling links, links not created by slink, and files that should perhaps be links. The command slink -clean {slink-tree} will clean up the slink tree by removing dangling links to files and directories that have been deleted or moved. This function does not occur automatically because dangling links are a symptom of more serious configuration problems. Thus one should pay attention to the error output when running this command. The command: slink -condense {slink-tree} condenses unneeded directories into single links. For example, if we have the files /loc/adm/slink-1.0/sbin/slink /loc/adm/slink-1.0/sbin/truename in /loc/adm/slink-1.0/sbin and the physical symbolic links /loc/adm/slink-1.0/sbin/slink <- /local/sbin/slink /loc/adm/slink-1.0/sbin/truename <- /local/sbin/truename and we execute slink -condense /local then these two links will be condensed, by removing them, removing the containing directory, and reforming the single link /loc/adm/slink-1.0/sbin <- /local/sbin Since this link embodies the effects of both of the other links, and the resulting apparent contents of /local/sbin are unchanged by the transformation, slink replaces the redundant directory /local/sbin with a single link. ___________________ Installing Software Slink is not just a tool, but also a philosophy for configuration management of large software repositories. The remainder of this document describes how to use slink to ease software configuration maintenance in large program repositories. The purpose of slink is fourfold: a) to allow new packages to be made available to users without changing their command search paths. b) to allow old packages to be removed from users' search paths without changing their search paths. c) to provide a self-documenting mechanism by which files needed for a particular package can be earmarked for that package, to avoid `tree rot' where files that are long unused and forgotten stay in an install tree forever, taking up needed space and confusing future installers, who cannot afford to delete them because the consequences of deleting them are unclear. d) to provide an easy mechanism for specifying replicated and non-replicated files in a master/slaves service environment, where parts of a single master filesystem are copied into local space on slave servers to reduce network overhead in running file-io-intensive programs. ______________ Filesystem Rot The goal of our software administration strategy is to reduce or eliminate the age-old software installation problem called `filesystem rot'. Filesystem rot occurs when many people take a lot of time to install a lot of local software on a system with a long lifetime, without appropriate documentation of what files are used by which program. Over time, parts of this software age at different rates, some becoming obsolete faster than others. As each program becomes obsolete, it is deleted along with all associated files one can _find_ or perhaps _remember_. However, since the sources have long been deleted, and in particular because software distributions typically don't provide an `uninstall' option in their installation scripts, eventually over time the system fills with dead wood files that are unused by any program, mixed freely together with essential files without which critical programs will die. So little by little, the filesystem on which software is installed `rots'; it wastes more and more space with files of unclear origin and function. There is no easy procedure for determining which programs use or used them, or what the effect of changing them or deleting them will be. Thus these files cannot be modified or deleted without unacceptable risk of disabling some unknown installed program. And the only way to find out that we broke something is to try to use it, which may happen months in the future. Then the modifications we make now will be long forgotten, and the only way to fix the broken program will be to reinstall it, creating perhaps more orphan files, etc. If we try to fix the broken program without reinstalling it, rot complicates this by expanding the search space in which a problem may hide. The only reasonable way to repair filesystem rot is to start from scratch and reinstall every program on a clean filesystem. This typically represents person-years of work for systems like our own. We in the Computer Science department have been through this process once with SunOs 3.X to SunOs 4.X, and currently are still using a 4.1.1 /usr/local partition in the throes of terminal software rot. If we had not moved to Solaris from SunOs recently, we would have had to completely reinstall /usr/local from scratch to clean out the rot that accumulated there during the last five years of installations. From the accelerated obsolescence rate of software today, we guess that if we had done nothing to address the problem of rot, we would have to rebuild /usr/local about every two years. The lifetime of /usr/local under the pre-solaris version of SunOs turned out to be about 5 years, so we designed our Solaris software installation strategies to allow us to function completely rot-free until Solaris is obsolete about 5 years from now(we hope!). ________ Packages To avoid software rot, software installed by Sun has a `package' structure that we mimic to some degree in our own installation work. A `package' is a complete installation of a program or system of programs and files that works in isolation from other packages and can be installed, used, maintained, updated, or deleted without affecting any other packages. The reason behind the use of packages is severalfold: a) Several people can concurrently work on installing or maintaining different packages without communicating with one another or having any chance to interfere with the work of another person. Formerly, we had plenty of problems with people breaking other people's installations inadvertantly by writing new files over the ones in use. b) An installer can completely test a new program before making it available to other users, without danger of breaking the program when making it available. Formerly, testing required test installation, which both made the program available before it was working and made it impossible to retract a test installation of a program that will never work. c) An ordinary user can install and maintain a package with no special privileges except ownership of a directory tree in an appropriate place. Formerly an installer had to have root privileges. This caused plenty of problems as installers `collided' while doing complex things. d) Each package can be deleted or relocated within the filesystem with absolute surety that we moved or deleted every file the package installed or referenced. This was impossible without packages. e) The size of a package shows how much space will be saved by moving or deleting a program. Formerly this information was impossible to obtain. f) The packages a user has access to can be tailored to the user's sophistication and goals. For example, we can make life simpler for naive students by putting less programs in their paths. g) It is impossible for two packages to install and refer to the same file, expecting different contents. This error is automatically detectable. This kind of error was a big problem in the old system. Quite often two people working at the same time to install two different programs would get into an `installation war' over the same file without knowing it. Since both installers needed root privileges to work, each could walk on the results of the other's changes without the other knowing it. The result would be two programs whose functions were mutually exclusive in a completely mysterious way; one would work exactly when the other didn't. Our former strategy for avoiding this was to designate exactly one installer to work at a time. This is an unacceptable constraint in modern systems. _____________________ What is in a package? Typically, a package consists of the installed files from a single tar'd software distribution obtained from anonymous ftp or a vendor, though interdependencies between related distributions often require that all the distributions be placed in the same package. For example, gcc and g++ are essentially the same program with different arguments, and libg++ is needed for anyone to do anything with g++, so all three are inseparable and should probably occupy the same package. As the main purpose of packages is to document relationships between files, the minimal package consists of two related files, typically a compiled program and its source code. Standalone shell scripts do not need to have their own packages as they consist of exactly one file, the script itself, and the consequences of deleting that script are clear: that one program will not work. This is also true for software that is distributed as a single binary image, though we usually do not, as a rule, install free software off the net unless it is available in source form and compiled by us (binary images can contain viral infections, no matter how reputable the source). A special package `alone' is provided for standalone software of this kind. Usually a package consists of one or more software distributions and splitting a single distribution among multiple packages is discouraged. There is one exception to this rule, however. A common directory of data files that is the union of data for a lot of packages and used by one or more of them should be in a common package all its own, e.g., the emacs info tree. ___________________________ Kinds of Files in a Package A typical package contains three kinds of files: a) files the user must access directly. These include: i) programs the user must execute, including compiled object-code as well as interpreted scripts. ii) libraries the user must load to run a a program, either with ld, with ld.so, or an interpreter such as perl. iii) databases that user-written programs must read, such as fonts, etc. iv) documentation the user must be able to read. b) files that the package must access, but which do not need to be available to the user. These include: i) programs and libraries that only the programs in the package should use. ii) databases used only by the programs and/or libraries in the package. c) files that document the package but are unneeded by the user or package components during normal use. i) Source code that is compiled and installed before use, such as C code. ii) Installation instructions, READMEs, and administrative documentation. The three kinds of package files must be treated differently during installation: 1) Files the user needs (type a) are made available to the user via symbolic links from a common area shared by all packages. The installer does not make these links; instead they are made by slink. 2) Files the package needs to work (types a and b) are stored in the package tree, but type b files are invisible to the user. 3) Files the package doesn't need to work (type c) are deleted when the package is working properly. The eventual goal is to place references to your installed files into the tree /local, so that users can get to the files. In /local, files are organized in subdirectories by type. The subdirectories have the same names and meanings as subdirectories of /, /usr (and /usr/local), in normal UNIX systems. bin - executable binaries invoked by users, such as games, etc. If you compile something everyone should be able to execute, put it here. These would traditionally be put in /usr/bin or /usr/local/bin. doc - documentation on use of programs, general internet use, ftp use. In general, any text not in a man page format. etc - control files for applications. These would traditionally be installed in /etc, /usr/etc, or /usr/local/etc. help - help with using programs, filed by program name. Installers should write these as small text files with the same name as the program. include - files to be included in source during compilation. `.h' files for libraries should be put here so that the types of routines are correctly declared in applications. These would traditionally be installed in /usr/include or /usr/local/include. lib - software libraries. To be included in users' compilations or at run time. These would traditionally be installed in /usr/lib or /usr/local/lib. lib/fonts fonts for postscript, etc. man general manual page subdirectory for user documentation. The subdirectories of this directory contain files read by the man command. These subdirectories would normally be found in /usr/man or /usr/local/man. These are broken down as follows: man/man1 manual pages for programs installed in bin. man/man3 manual pages for libraries installed in lib. man/man5 manual pages for formats of files and databases in etc. man/man6 manual pages for games in particular. man/man8 manual pages for administrative programs installed in bin. sbin - executable binaries intended for use by root or system administration. These would traditionally be put in /usr/sbin or /usr/local/sbin. _______________________ Building a package tree To install a piece of software, first make a directory $package for it in the appropriate category subdirectory of your package hierarchy, obtain the source code, and place it in $package/src or some similar directory. Compile the source code and install the compiled version somewhere in the package subdirectory separate from the source, so that the source can be deleted when no longer needed. It's best if you install the working code in a directory hierarchy rooted at $package and having a parallel structure to the user's image you wish to construct. _______________________________________________ Using slink to make packages available to users After making your package work in the package tree and testing it, it's time to make it work for all users. This is a matter of mapping your files into the /local tree visible to users, and is a job for slink. To do this, you must a) make a permanent record of the files that should be visible to users, and b) execute slink to make them visible. _______________________ Making packages visible In order to make your links permanent, you must add them to the master control file slink.conf. This file consists of lines of the form: just as they would appear in the direct slink command slink -link and has the effect of making and its descendents appear as and its descendents. The opposite of this is ! which is equivalent to the direct slink command slink -unlink and has the effect of removing ONLY those links that point from and its descendents to and its corresponding descendents. If appropriately links exist and point elsewhere, they are left alone. A line of the form: + sets current directories for relative addressing in the file. The first dirname is a path prefix for relative filenames, while the second is a path prefix for relative dirnames. A line of the form: + includes an auxiliary file into the current configuration file as if it had been typed at that point, and optionally changes the current directories for filename and linkname for the included file only. For example, /loc/case/rcs-5.6/bin could be installed as a subset of /local/bin by the single line: /loc/case/rcs-5.6/bin /local/bin or by the pair of lines: + /loc /local case/rcs-5.6/bin bin As a more complex example, the emacs subtree could be described in the file /loc/emacs/slink.conf, that could be referenced as: +/loc/emacs/slink.conf /loc/emacs /local and contain lines of the form: bin bin lib lib to merge /loc/emacs/bin into /local/bin and /loc/emacs/lib into /local/lib. _____________ Running slink To update the tree, simply type slink to update the whole tree. This takes a long time. If you don't want to wait, you can use the alternate form: slink that will only install subtrees matching the patterns, which are simple substrings of the fully expanded pathnames for source and link names as designated in /loc/slink.conf and its subfiles. For example, if the contents of /loc/slink.conf are as above, then slink /loc/case/rcs-5.6 will only install rcs-5.6, while slink emacs /local/lib will install only those things in /loc subtrees containing the substring `emacs' that belong in /local/lib. ______________________ Making slinking easier Obviously, you can always put a line into /loc/slink.conf for every single file users will see. But you can make life a lot easier if the directories in your package have parallel structure to directories in /local, because slink will duplicate your package directories in /local to arbitrary recursive depth, using a minimal number of symbolic links to do so. The only catch is that EVERY file in any directory you specify will be duplicated, including such annoyances as emacs temporaries `file~' and any other garbage you see fit to put in the directories. So it pays to keep your directories clean. _______________________________________ Replicating files on slave file servers Due to the network overhead of retrieving files from huge packages such as emacs, gcc, etc., it is often desirable to keep copies of selected packages on servers other than allegro. But since space is limited, we prefer to replicate only those files that improve general system performance. The rest are obtained via nfs from a master fileserver. Replication is simple indeed: first replicate the structure of the package hierarchy itself and copy the files you want to replicate. Then arrange for unreplicated subtrees to be automounted when referenced. Finally, run slink on the LOCAL machine to reconstruct the image tree pointing to both local and automounted trees. ______________________________________________ Special case: packages with a shared directory In special cases we must violate the package abstraction by making a package install files OUTSIDE its own tree. This happens when several packages need to install files in a shared location accessed as a unit. In this case, it pays to either: a) install the shared directory in the package tree of the package that must access the files, e.g., emacs info. b) install the shared directory in a new package that will outlive all contributing packages. The second option is far preferable to the first. If we put info from many applications in the emacs info tree, and change revisions of emacs, we want to reuse the info tree rather than destroying it. The same observation is true of packages that share a large dataset, except in this case, you want a shared location that contains one copy of a thing instead of several, as in the case of the emacs lisp documentation tree. This should also be put in a separate package that will outlive all versions of emacs currently loaded.