Installing new software:

• compile it (if necessary)
• install it in a public place.
• modify user's environment so it'll run there.

Software installation concepts:

• packages
• installation
• dependencies
• backout

Package:

• a group of programs that make sense together
• usually distributed as 'one distribution hierarchy'.

Installation:

• making a package available to the user.
• NOT typically just a matter of setting paths.
• usually, you have to locate (copy) files to places all over the filesystem.
  • normal program:
    • /usr/local/bin/whatever - actual program
    • /usr/local/lib/libwhatever.a - needed library
  • daemon:
    • /usr/local/sbin/whatever - actual program
    • /sbin/init.d/whatever - starts daemon
    • /sbin/rc2.d/S01whatever - makes sure daemon is started on reboot

Why is installation so difficult?

• Dependencies
• Portability
• Backout

Dependencies:

• the need to have one package installed in order to use another
• need perl to run perl scripts!
• some packages use other packages' header files and/or libraries.

Portability

• needed things are in DIFFERENT PLACES.
  • header files
  • libraries
  • data files
• needed things can be missing.
  • header files
  • library functions
  • data files
• substitutes can have DIFFERENT SYNTAX or structure.
  • example: strings.h versus string.h

Backout:

• reversing an installation, we have to remember:
• what got modified.
• original state.
• what other things depend upon this package

Backout strategies:

• transaction log + backout storage:
  • list what got modified.
  • store original copies
  • play back in reverse to back something out.
• symlinks: don't really copy files into /usr. link to them inside their package trees.
  • original files stay in place
  • links point to a new place.
  • can always point them back to the previous place.
• custom backout script: provided by software creator.

Vendor backout approaches

• /opt: contains all Sun Solaris packages.
  • symlinks from /usr/bin as appropriate.
• Environment variables:
  • tell where packages are actually installed.

     setenv MGC_HOME /loc/mentor
     setenv MGLS_LICENSE_FILE $MGC_HOME/etc/cust/mgls/mgc.licenses
     setenv MGC_LOCATION_MAP /loc/mentor/etc/mgc_location_map
    

Unix freeware packages:

• found on network as compressed image.
• 'extension' of file tells how to unwrap it:

Extension conventions:

• .Z: compressed file

   uncompress file.Z  => file
   compress file => file.Z
  

• .gz: compressed using gnu-zip

   gunzip file.gz => file
   gzip file => file.gz
  

• .zip: compressed using PKZip and equivalents.

   unzip file.zip => expands file.zip into a directory of files. 
  

• .tar: multiple files in an archive, using tar and gnu-tar

   tar tf file.tar => contents of file.tar
   tar xf file.tar => extracts contents
   tar cf file.tar something => creates file.tar from various others
  

• tar : tape archiver : tar t lists the 'tape device /dev/rmt0'

Must bootstrap this process!

• Typically, UNIX's contain tar and compress, no zip,gzip,etc.
• It behooves one to compile those FIRST!
• Unfortunately for me, sometimes one needs to COMPILE GCC in order to be able to compile THEM. (!)

Kinds of packages:

• binary: contains only executable programs.
  • non-portable: only work on one architecture of machine and OS
  • opaque: you can't tell what they're doing.
  • unsafe to install in general, esp. from untrusted sources.
  • most common way hackers compromise system security.
• source: contains c programs you can compile.
  • relatively transparent: you can read the code.
  • CAN be written to be portable (or not!)

Compiling software:

• on PC's, MAC's, almost unheard of
• less compilation needed for Linux
• On Vendor UNIX's (Solaris, OSF4, etc) it's the rule rather than the exception
• GOLDEN RULE: NEVER TRUST binary distributions of UNIX programs.
  • They can do ANYTHING to your UNIX system.
  • Even if you can GET binaries, utilize source code if you can.

Kinds of executable images:

• Scripts: first line is
  1. ! /bin/csh -fb or something like this.
  • #! : this is a script (first two characters determine file type)
  • /bin/csh: what shell will interpret this.

     #! /bin/tcsh
     #! /usr/local/bin/perl
    

  • the command

     chmod +x script
    

    makes a script executable.
  • scripts cause the program chosen to be run.
  • the default program is /bin/sh.
• a.out format (man a.out)
  • first two bytes determine ARCHITECTURE of machine (/etc/filetype)
  • rather complex file format follows
    • executable image (program that can be run)
    • optional trailer containing debugging information.
    • different for every processor type and version of unix!

Inside a.out:

• code: machine code to be executed.
• symbols: definitions of addresses of things:
  • subroutines.
  • global variables.

Making an a.out: gcc

           file.c                        source code (human-readable) 
             |
 [/usr/local/lib/gcc/.../cc1]            first-stage C compiler. 
             |
           file.o  /usr/lib/libc.a       c compile-time library
                \   /                    (reusable functions) 
            [/usr/bin/ld]                STATIC LINKING: add needed 
                  |                        library functions
                a.out  /usr/shlib/libc.so  c runtime library
                     \  /                  (reusable, shareable functions)
             [/usr/lib/ld.so]            DYNAMIC LINKING
                      |
           running IMAGE of a.out        never stored in a file. 

The players:

• file.c: SOURCE CODE, before compilation.
• file.o: OBJECT CODE, after compilation
  • code to execute
  • SYMBOL TABLE
    • where are functions?
    • where is data?
  • external symbol REFERENCES:
    • call function 'BLAH'.
    • write into array 'BAR'.
    • BAR, BLAH in ANOTHER .o file!
• libc.a: LIBRARY ARCHIVE
  • lots of .o's.
  • 'ar' format (archive) (man ar)
  • each .o contains a SYMBOL TABLE
  • game of ld: select those .o's that define symbols that file.o needs and doesn't have. This is called LINKING.
• libc.so: SHARED OBJECT
  • same as .o, but
  • allows TWO running programs to share the SAME IMAGE in memory.
• libz.sa: SHARED ARCHIVE:
  • 'ar' format archive of .so's.

Naming conventions:

• in compile and link command lines:
• libfoo.a -> -lfoo
• /usr/lib -> -L/usr/lib
• -L/usr/lib -lfoo means /usr/lib/libfoo.a if libfoo.a is in /usr/lib.....

Taking things apart

• Can take apart .o's and .a's!
• taking apart .o's and .so's: nm
• taking apart .a's and .sa's: ar

ar: taking apart .a's

• ar t /usr/lib/libc.a - lists contents of libc.a
• ar x /usr/lib/libc.a - extracts actual .o files from archive!
• ar c /usr/lib/libfoo.a file1.o file2.o .... - creates an archive.
• one subtlety: on some UNIX's you must 'index' the archive, using the program 'ranlib', after creating it.

nm: taking apart .o's

• nm file.o | more - list the 'symbol table' of a .o file.
• tells what externals are needed by file.o

Process execution environment:

• PATH: list of DIRECTORIES where programs can be found.
• LD_LIBRARY_PATH: list of DIRECTORIES in which process LIBRARIES can be found (both compile-time and run-time)
• LD_RUN_PATH: list of DIRECTORIES in which process RUN-TIME LIBRARIES can be found. Only on some UNIX's, e.g., Solaris.

Static linking: (ld)

• build a program image INCREMENTALLY.
• scan each input file in turn.
• for files ending in .o, simply load the code from them and make links between undefined externals in one file and their references in another.
• for files ending in .a, link their .o's CONDITIONALLY whenever a DEFINITION of a symbol in one file matches a REFERENCE in the linked image you have so far.

   foo.c:                    bar.c:                
    int x;                    extern int x; 
    extern int y;             int y; 
   main() {                  main() { 
    x=2; /* local */          x=4; /* extern */ 
    y=9; /* extern */         y=2; /* local */ 
   }                         }
   
    foo.o:                  bar.o:                       libc.a: 
      DEFINES x ------------> EXTERNAL x              scanf.o
      EXTERNAL y <----------- DEFINES y               bcopy.o
                              EXTERNAL printf <--+    printf.o 
                                                 +--- DEFINES printf
                                                      EXTERNAL __printf 
  

• __printf ficticious name, name differs, '|printf' on suns.

Very important notes on linking:

• compilation of objects gives them TYPES:
  • function
  • variable (int, double, struct, pointer)
• linking operation is TYPELESS (just address)
• one can REALLY MESS UP THINGS by doing:

   foo.c: int printf; 
   bar.c: printf("ho ho ho\n"); 
  

• THIS TRIES TO EXECUTE AN INT! BLAM! (Illegal Instruction - Core dumped)

Result of static linking:

• At the end of this process, we get a.out
• all normal static external symbols resolved.
• yet to be resolved: dynamic externals.
• a.out consists of all the code for the program that's private and unshareable.
• yet to be determined: how to link to 'shared' code.

Execution model:

• programs (a.out) are organized in SEGMENTS:
  • TEXT: program code, read-only.
  • STACK: space for variables
  • BSS: space for global variables.
• each SEGMENT gets PAGES of memory to use.
  • 4096 bytes or otherwise.
  • read-only or read-write
  • shared or private

Files differ in what kind of pages they're loaded into

• .o: private text
• .so: shared text (multiple processes can get to these pages BY NAME)
• to share a page, must link to it DYNAMICALLY (at the time a process gets executed)

Dynamic linking process:

• ld.so: dynamic linker.

   a.out:                libc.so: 
    EXTERNAL __printf <----- DEFINES __printf  (ficticious name __printf)
  

• goes through a TABLE in a.out listing references to dynamic links.
• loads .so and .sa data and links to it as appropriate.
• once there are NO UNDEFINED EXTERNALS, a.out is ready to run (in memory)
• so it gets executed.

This is relatively easy to break:

• ld: undefined externals
  • no library specified.
  • LD_LIBRARY_PATH wrong.
  • -L compiler option wrong.
• ld.so: undefined externals
  • moved libc.so
  • LD_LIBRARY_PATH or LD_RUN_PATH wrong.
  • ld.so missing(!)

Static and dynamic directory binding:

• LD_LIBRARY_PATH:
  • can break by setting LD_LIBRARY_PATH wrong.
  • can't break by moving libraries around if you change LD_LIBRARY_PATH
• -L/usr/lib (compile time option) (or -Rsomething)
  • can't break by changing LD_LIBRARY_PATH
  • can break by moving libraries elsewhere.

Compiler options:

• -L : define compile-time LD_LIBRARY_PATH component, remember it at run-time
• -lfoo : scan library libfoo.a in the first library directory in which you find it
• -O2 : optimize compilation to make programs run faster (default)
• -c : generate only a .o file
• man gcc for infinitely more options.

Compiler caveat:

• But you'll almost _never_ execute a compiler directly
• Instead, you'll execute compilation META-TOOLS that call the compiler for you!