Installing software

lecture in color

Software installation

adding programs to the system that users can execute.
can be accomplished in several ways
- download binary file and install it somewhere public.
- download source code and compile and install it.

ASCII and binary

ASCII file: American Standard Code for Information Interchange.
- modernized name: iso-latin-1
- made up of 8-bit bytes.
- `regular text'
- Irregular-length lines ended with \n (ASCII 10 (base 10))
- content limited to printable characters in ASCII.
binary files:
- made up of 8-bit bytes.
- any characters allowed, including unprintable ones.
- no line delimiter.
only difference between ASCII and binary: limits on contents.
- binary: anything
- ASCII: printable characters and \n

Understanding the difference between ASCII and binary:

od: `octal dump' : print contents of any file in human-readable form.

Here's a little ASCII file:

 extern int printf(); 
 main() { printf("hello world\n"); }

let's print this ASCII as octal: od -c -b hello.c

 conbrio{couch}192: od -c -b hello.c
 0000000   e   x   t   e   r   n       i   n   t       p   r   i   n   t
          145 170 164 145 162 156 040 151 156 164 040 160 162 151 156 164
 0000020   f   (   )   ;      \n   m   a   i   n   (   )       {       p
          146 050 051 073 040 012 155 141 151 156 050 051 040 173 040 160
 0000040   r   i   n   t   f   (   "   h   e   l   l   o       w   o   r
          162 151 156 164 146 050 042 150 145 154 154 157 040 167 157 162
 0000060   l   d   \   n   "   )   ;       }      \n
          154 144 134 156 042 051 073 040 175 040 012

The characters have octal equivalents in ASCII. E.g., d is octal 144.

od -b print bytes (not words)
od -c print the ASCII equivalents as well.

Now let's understand binary:

gcc -o hello hello.c: compiles that program
output program hello is binary.

Let's look at that with od:

 conbrio{couch}197: od -c -b hello | head
 0000000 177   E   L   F 001 002 001  \0  \0  \0  \0  \0  \0  \0  \0  \0
          177 105 114 106 001 002 001 000 000 000 000 000 000 000 000 000
 0000020  \0 002  \0 002  \0  \0  \0 001  \0 001 007 350  \0  \0  \0   4
          000 002 000 002 000 000 000 001 000 001 007 350 000 000 000 064
 0000040  \0  \0   Y 304  \0  \0  \0  \0  \0   4  \0      \0 005  \0   (
          000 000 131 304 000 000 000 000 000 064 000 040 000 005 000 050
 0000060  \0 034  \0 031  \0  \0  \0 006  \0  \0  \0   4  \0 001  \0   4
          000 034 000 031 000 000 000 006 000 000 000 064 000 001 000 064
 0000100  \0  \0  \0  \0  \0  \0  \0 240  \0  \0  \0 240  \0  \0  \0 005
          000 000 000 000 000 000 000 240 000 000 000 240 000 000 000 005

First character is octal 177: binary file.

magic code '177-ELF': Extensible Link Format.
The first few characters determine what kind of file it is.
Just like for scripts.

Investigating kinds of files

the file command will look at the first few bytes of a file and tell you what it is.

 conbrio{couch}200: file hello.c
 hello.c:        c program text
 conbrio{couch}201: file hello.o
 hello.o:        ELF 32-bit MSB relocatable SPARC Version 1
 conbrio{couch}202: file hello
 hello:          ELF 32-bit MSB executable SPARC Version 1, dynamically linked, not stripped

Very important: binary files are not portable

executable binary files are not portable.
architecture differences
- SPARC : executes on suns.
- i386: executes on PC's
- no way to translate between the formats (except by running 'soft-pc' interpreters that pretend to be PC's).
format differences
- ELF: Extensible Link Format
- COFF: Common Object File Format
script files may or may not be portable
- is script interpreter present?
- does it work the same way here?
- are the commands the script uses present?
It takes a lot of work to make C source code portable
- differences in UNIX's.
- differences in libraries.
- a lot more about this later.

Typical war story

Linux 1.x used COFF.
Linux 2.x used ELF.
So we had to recompile all programs from source when switching from 1.x to 2.x

Software basics

three forms of programs
scripts:
- ASCII content
- program 'compiled' when run
binaries:
- content is `machine language'.
- creating by compiling source code.
- once created, source code isn't required anymore.
sources:
- human readable versions of programs, e.g., file.c.
- must be compiled before they're usable.

The short story

get it (file something.tar.gz)
unpack it. (directory something/)
compile it (if necessary) (program something).
install it in a public place (/usr/local/bin/something)
modify user's environment so it'll run there.

At the risk of trademark infringement

most of the instructions we read for installing software are intended for dummies.
let's summarize.

Obtaining software from the Internet

set your umask as appropriate for an administrator
```
 % umask 022
```
This means that all files you create will be world-readable.

get something from outside.

 % cd /where/to/put/source
 % ftp somewhere.com
 Username: anonymous
 Password: couch@eecs.tufts.edu
 User couch logged in
 ftp> cd /pub/foo
 ftp> ls
 foo.tar.gz
 ftp> bin
 Type set to I
 ftp> get foo.tar.gz

must transfer most software in binary mode (bin)
if you use ASCII mode, all the steps below will fail.
because any binary file transferred in ASCII mode will be corrupted at this end.

Unpacking software

Goal is to transform files from particular compression formats (binary) to uncompressed format.
Starting file is binary.
End result may be either binary or ASCII files.
File extension determines what to do:
from file thing.gz to file thing: gunzip thing.gz
from file thing.Z to file thing: uncompress thing.Z
from file thing.bz2 to file thing: bunzip2 thing.bz2
from file thing.zip to file thing: unzip thing.zip
from file thing.tar to directory thing: tar xf thing.tar
from file thing.a to contents of thing: ar x thing.a
from file thing.shar to contents of thing: sh thing.shar

Installing binaries for dummies.

unpack it as above
make it executable
```
 chmod +x foo
```
move it somewhere in your execution path.
Done! OK?

Software compiling for dummies

read the documentation
```
 % more README
 % more INSTALL
```

configure for this kind of UNIX and personal preferences

 % ./configure --prefix=/where/to/install --other-configure-options...

compile the software
```
 % make 
```
install the software for general use
```
 % make install
```
Set your path so it contains the executable you installed.
Share and enjoy!

Modifying users' paths for dummies:

Add the command directory to the user's path with a command in /etc/.cshrc (for the t-shell).
```
 set path = ($path /some/where/new) 
```
Login again or start a new shell to see the results.

An installation example: hello.c

Get the software:

contents of install/hello-get...
blackhole{couch}52: ftp ftp.gnu.org
Connected to ftp.gnu.org.
220 ProFTPD 1.2.0pre9 Server (ProFTPD) [gnudist.gnu.org]
Name (ftp.gnu.org:couch): anonymous
331 Anonymous login ok, send your complete e-mail address as password.
Password: couch@eecs.tufts.edu
230-If you have any problems with the GNU software or its downloading, please
 refer your questions to <gnu@gnu.org>.
 
 There are several mirrors of this archive, a complete list can be found on
 http://www.gnu.org/order/ftp.html. Please use one of the mirrors if possible.
 
 Archives of the GNU mailing lists can be found in
 ftp://ftp-mailing-list-archives.gnu.org/.
 
 Please note that the directory structure on ftp.gnu.org was redisorganzied
 fairly recently, such that there is a directory for each program.  One side
 effect of this is that if you cd into the gnu directory, and do
 > ls emacs*
 you will get a list of all the files in the emacs directory, but it will not
 be obvious from the ls output that you have to `cd emacs' before you can
 download those files.
 
 Note further the non-GNU programs that were formerly in gnu/ have moved to
 gnu/non-gnu/.  Most of them were just pointers in the format program.README.
 If you are looking for such a file, be sure to check gnu/non-gnu/.
230 Anonymous access granted, restrictions apply.
ftp> cd pub
250 CWD command successful.
ftp> ls
200 PORT command successful.
150 Opening ASCII mode data connection for file list.
pub
welcome.msg
bin
lib
gnu
ls-lrR.txt
ls-lrR.txt.gz
non-gnu
README
lpf.README
old-gnu
find-ls.txt
find-ls.txt.gz
md5sums.txt
md5sums.txt.gz
226 Transfer complete.
156 bytes received in 0.024 seconds (6.45 Kbytes/s)
ftp> cd gnu
250-If you have problems downloading and are seeing "Access denied" or "Permission
 denied", please make sure that you started your FTP client in a directory to
 which you have write permission.
 
 Please note that all files ending in `.gz' are compressed with `gzip', not
 with the unix `compress' program. Get the file below for more info.
 
 For a list of mirrors and other ways of getting GNU software, FTP the file
 /pub/gnu/GNUinfo/FTP from ftp.gnu.org or one of its mirror sites.
 
 Finally, please note that the non-GNU programs that were formerly in
 this directory have moved to non-gnu/.  Most of them were just
 pointers in the format program.README.  If you are looking for such a
 file, be sure to check non-gnu/.
 
 This directory contains programs (or pointers thereto) that are part
 of the GNU system, as well as GNU Manuals, and general information on
 project GNU.
 
 Programs that are directly in this directory are actually GNU
 programs, developed under the auspices of GNU.
 
 We do, however, distribute some non-GNU programs through our FTP
 server, or provide pointers to where they are.  We put these
 programs/pointers in the subdirectory non-gnu since they are not
 developed by the GNU project. They are, of course, part of the GNU
 system. See:
 http://www.gnu.org/philosophy/categories.html#TheGNUsystem
250 CWD command successful.
ftp> ls
200 PORT command successful.
150 Opening ASCII mode data connection for file list.
COPYING
COPYING-1.0
GPL
LGPL
README
README-about-.diff-files
README-about-.gz-files
... lots of stuff ...
guile
gzip
hello
hp2xx
hurd
... lots of stuff ...
xshogi
zlibc
226 Transfer complete.
2742 bytes received in 1.3 seconds (2.12 Kbytes/s)
ftp> cd hello
250 CWD command successful.
ftp> ls
200 PORT command successful.
150 Opening ASCII mode data connection for file list.
hello-1.0-1.1.diff.gz
hello-1.1-1.2.diff.gz
hello-1.2-1.3.diff.gz
hello-1.3.tar.gz
226 Transfer complete.
87 bytes received in 0.028 seconds (3.09 Kbytes/s)
ftp> bin
200 Type set to I.
ftp> get hello-1.3.tar.gz
200 PORT command successful.
150 Opening BINARY mode data connection for hello-1.3.tar.gz (87942 bytes).
226 Transfer complete.
local: hello-1.3.tar.gz remote: hello-1.3.tar.gz
87942 bytes received in 0.81 seconds (105.89 Kbytes/s)
ftp> quit
221 Goodbye.
blackhole{couch}53: 
...end of install/hello-get

Unpack the software:

contents of install/hello-unpack...
blackhole{couch}51: ls
hello-1.2-1.3.diff  hello-get           hello.c             test
hello-1.3.tar.gz    hello-unpack        install.txt
blackhole{couch}52: gunzip hello-1.3.tar.gz
blackhole{couch}53: ls
hello-1.2-1.3.diff  hello-get           hello.c             test
hello-1.3.tar       hello-unpack        install.txt
blackhole{couch}54: tar xf hello-1.3.tar
blackhole{couch}55: ls
hello-1.2-1.3.diff  hello-1.3.tar       hello-unpack        install.txt
hello-1.3           hello-get           hello.c             test
blackhole{couch}56: cd hello-1.3
blackhole{couch}57: ls
COPYING        TAGS           getopt1.c      hello.info     mkinstalldirs
ChangeLog      alloca.c       gpl.texinfo    hello.ky       testdata
INSTALL        configure      hello.c        hello.pg       texinfo.tex
Makefile.in    configure.in   hello.cp       hello.texi     version.c
NEWS           getopt.c       hello.cps      hello.tp
README         getopt.h       hello.fn       hello.vr
blackhole{couch}58: 
...end of install/hello-unpack

Compile and install:

contents of install/hello-config...
blackhole{couch}51: cd hello-1.3
blackhole{couch}52: ls
COPYING        TAGS           getopt1.c      hello.info     mkinstalldirs
ChangeLog      alloca.c       gpl.texinfo    hello.ky       testdata
INSTALL        configure      hello.c        hello.pg       texinfo.tex
Makefile.in    configure.in   hello.cp       hello.texi     version.c
NEWS           getopt.c       hello.cps      hello.tp
README         getopt.h       hello.fn       hello.vr
blackhole{couch}53: ./configure --prefix=/g/150NET/public_html/notes/install
checking for gcc
checking how to run the C preprocessor
checking for install
checking for ANSI C header files
checking for string.h
checking for fcntl.h
checking for sys/file.h
checking for working alloca.h
checking for alloca
creating config.status
creating Makefile
blackhole{couch}54: make
gcc -c  -DSTDC_HEADERS=1 -DHAVE_STRING_H=1 -DHAVE_FCNTL_H=1 -DHAVE_SYS_FILE_H=1 -DHAVE_ALLOCA_H=1 -g hello.c
gcc -c  -DSTDC_HEADERS=1 -DHAVE_STRING_H=1 -DHAVE_FCNTL_H=1 -DHAVE_SYS_FILE_H=1 -DHAVE_ALLOCA_H=1 -g version.c
gcc -c  -DSTDC_HEADERS=1 -DHAVE_STRING_H=1 -DHAVE_FCNTL_H=1 -DHAVE_SYS_FILE_H=1 -DHAVE_ALLOCA_H=1 -g getopt.c
gcc -c  -DSTDC_HEADERS=1 -DHAVE_STRING_H=1 -DHAVE_FCNTL_H=1 -DHAVE_SYS_FILE_H=1 -DHAVE_ALLOCA_H=1 -g getopt1.c
gcc -g -o hello hello.o version.o getopt.o getopt1.o  
blackhole{couch}55: make install
./mkinstalldirs /g/150NET/public_html/notes/install/bin /g/150NET/public_html/notes/install/info
mkdir /g/150NET/public_html/notes/install/bin
mkdir /g/150NET/public_html/notes/install/info
cp hello /g/150NET/public_html/notes/install/bin/hello
cp ./hello.info /g/150NET/public_html/notes/install/info/hello.info
blackhole{couch}56: cd ..
blackhole{couch}57: ls
bin                 hello-1.3.tar       hello-unpack        install.txt
hello-1.2-1.3.diff  hello-config        hello.c             test
hello-1.3           hello-get           info
blackhole{couch}58: ls bin
hello
blackhole{couch}59: ls info
hello.info
blackhole{couch}60: 
...end of install/hello-config

But:

Any fool can install software!
most fools do!
really knowing what these commands do separates a `novice' from an `expert'.
and if you don't know what you're actually doing, you have no hope of coping when things go wrong!

First steps into a larger world:

man ftp- file transfer protocol.
man compress- file compression/decompression.
man tar- tape archiver.
man gcc- special compiler options used during the make
man ld- how the static linker works.
man ld.so- how the dynamic linker works.
man make- dependency manager.

Grabbing files

ftp- file transfer protocol
http- hypertext transfer protocol
Watch out: file protections may well be incorrect when you first grab a file from outside!

Compression

downloading files takes a long time.
to make that time shorter, people compress the file before you download it.
to access the true contents of the file you must uncompress (expand) the file.
Many different compression formats.

Extension conventions:

.Z: compressed file

 uncompress file.Z  => file
 compress file => file.Z

.gz: compressed using gnu-zip

 gunzip file.gz => file
 gzip file => file.gz

.zip: compressed using PKZip and equivalents.

 unzip file.zip => expands file.zip into a directory of files.

.tar: multiple files in an archive, using tar and gnu-tar

 tar tf file.tar => contents of file.tar
 tar xf file.tar => extracts contents
 tar cf file.tar something => creates file.tar from various others

tar : tape archiver : tar t lists the 'tape device /dev/rmt0'

Must bootstrap this process!

Typical UNIX's, when installed, contain tar and compress, no zip,gzip,etc.
It behooves one to compile those first!
Unfortunately for me, sometimes one needs to compile gcc in order to be able to compile them. (!)

Packages

Software is usually distributed in packages
package: a collection of related files.
usually unpacks into one directory.
usual convention: package-version.revision
hello-1.3 means
- package hello
- version 1
- revision 3

Kinds of packages:

binary: contains only executable programs.
- non-portable: only work on one architecture of machine and OS
- opaque: you can't tell what they're doing.
- unsafe to install in general, esp. from untrusted sources.
- common way hackers compromise system security.
source: contains c programs you can compile.
- Relatively transparent: you can read the code.
- Can be written to be portable (or not!)

Compiling software:

on PC's, MAC's, almost unheard of
less compilation needed for Linux
On Vendor UNIX's (Solaris, OSF4, etc) it's the rule rather than the exception
Common practice: never trust binary distributions of UNIX programs.
- They can do anything to your UNIX system.
- Even if you can get binaries, utilize source code if you can.

Kinds of executable images:

Scripts: first line is
1. ! /bin/csh -fb or something like this.
- #! : this is a script (first two characters determine file type)
- /bin/csh: what shell will interpret this.
```
 #! /bin/tcsh
 #! /usr/local/bin/perl
```
- the command
```
 chmod +x script
```
  makes a script executable.
- scripts cause the program chosen to be run.
- the default program is /bin/sh.
a.out format (man a.out)
- first two bytes determine architecture of machine (/etc/filetype)
- rather complex file format follows
  - executable image (program that can be run)
  - optional trailer containing debugging information.
  - different for every processor type and version of unix!
- man a.out for details!

Inside a.out:

code: machine code to be executed.
symbols: definitions of addresses of things:
- subroutines.
- global variables.

Making an a.out: gcc (solaris)

Suppose we have a file hello.c:

 #include <stdio.h> 
 main() { printf("hello world\n"); }

Compilation of this file via gcc hello.c -o hello works as follows:

           hello.c /usr/include/stdio.h  source code (human-readable) 
             |      /                    stdio.h is header file
 [/usr/local/lib/gcc/.../cpp]            c preprocessor includes headers,
             |                           does substitutions on #defines 
           hello.cpp                     preprocessed code (not stored) 
             |
 [/usr/local/lib/gcc/.../cc1]            first-stage C compiler. 
             |
           file.o  /usr/lib/libc.a       c compile-time library
                \   /                    (reusable functions) 
            [/usr/ccs/bin/ld]            STATIC LINKING: add needed 
                  |                        library functions
                hello  /usr/lib/libc.so    c runtime library
                     \  /                  (reusable, shareable functions)
             [/usr/lib/ld.so]            DYNAMIC LINKING: add needed
                      |                    shared library functions
           running IMAGE of hello        never stored in a file.

From source to executing image

Example: compile a program hello.c
cpp transforms hello.c by including /usr/include/stdio.h and produces preprocessed file hello.cpp ('preprocessing')
cc1 transforms hello.ccp into hello.o ('compilation')
ld turns hello.o into hello ('linking')
when you execute the command, ld.so transforms hello into a running image. ('dynamic linking')

Why so much trouble?

a program is not just determined by its source code.
it also calls libraries that perform most of its functions.
a hello-world program is not small as an executable
It calls the `standard IO library' to print its message.
This library is large.
It behooves us to `share' its image in real memory.
Hence we place it in libc.so so it can be `shared' among several programs at the same time.

The players:

hello.c: `source code', before compilation.
hello.o: `relocatable object code', after compilation
- code to execute
- SYMBOL TABLE
  - where are functions?
  - where is data?
- external symbol REFERENCES:
  - call function 'BLAH'.
  - write into array 'BAR'.
  - BAR, BLAH in ANOTHER .o file!
libc.a: LIBRARY ARCHIVE
- lots of .o's.
- 'ar' format (archive) (man ar)
- each .o contains a SYMBOL TABLE
- game of ld: select those .o's that define symbols that file.o needs and doesn't have. This is called LINKING.
libc.so: SHARED OBJECT
- same as .o, but
- allows TWO running programs to share the SAME IMAGE in memory.
libc.sa: SHARED ARCHIVE:
- 'ar' format archive of .so's.

A really simple example

The simplest program of all:

contents of install/hello.c...
extern int printf(); 
main() { printf("hello world\n"); } 
...end of install/hello.c

This doesn't actually define the function printf; it just declares it.

hello.o has the following symbol table:

contents of install/hello.nm...
conbrio{couch}51: nm hello.o
hello.o:

[Index]   Value      Size    Type  Bind  Other Shndx   Name

[3]     |         0|       0|SECT |LOCL |0    |2      |
[4]     |         0|       0|SECT |LOCL |0    |3      |
[2]     |         0|       0|NOTY |LOCL |0    |2      |gcc2_compiled.
[1]     |         0|       0|FILE |LOCL |0    |ABS    |hello.c
[6]     |         0|      28|FUNC |GLOB |0    |2      |main
[5]     |         0|       0|NOTY |GLOB |0    |UNDEF  |printf
conbrio{couch}52: 
...end of install/hello.nm

The command nm prints the symbol table of a .o file.
In this case, we see that hello has an undefined external symbol printf.
That is another function that has to come from somewhere!

Let's explore this issue further by disassembling libc.a:

contents of install/hello-libc...
blackhole{couch}51: mkdir libc
blackhole{couch}52: cd libc
blackhole{couch}53: cp /usr/lib/libc.a .
blackhole{couch}54: ar x libc.a # extract all files from libc.a
blackhole{couch}55: ls
_Q_add.o                   iswdigit.o
_Q_cmp.o                   iswgraph.o
_Q_cmpe.o                  iswlower.o
... omitted lots of stuff ... 
_ftou.o                    pread.o
_ftoull.o                  pread64.o
_getsp.o                   printf.o
_lwp_mutex_unlock.o        priocntl.o
_nfssys.o                  priocntlset.o
_pset.o                    processor_bind.o
... omitted lots of stuff ... 
iswcntrl.o                 yp_stubs.o
iswctype.o
blackhole{couch}56: ls -l printf.o
-rw-r--r--   1 couch    faculty     1852 Feb  1 12:50 printf.o
blackhole{couch}57: nm printf.o

printf.o:

[Index]   Value      Size    Type  Bind  Other Shndx   Name

[2]     |         0|       0|SECT |LOCL |0    |2      |
[3]     |         0|       0|SECT |LOCL |0    |3      |
[4]     |         0|       0|SECT |LOCL |0    |4      |
[11]    |         0|       0|NOTY |GLOB |0    |UNDEF  |___errno
[6]     |         0|       0|NOTY |GLOB |0    |UNDEF  |__iob
[5]     |         0|       0|NOTY |GLOB |0    |UNDEF  |__threaded
[12]    |         0|       0|NOTY |GLOB |0    |UNDEF  |_doprnt
[13]    |         0|       0|NOTY |GLOB |0    |UNDEF  |_ferror_unlocked
[8]     |         0|       0|NOTY |GLOB |0    |UNDEF  |_flockget
[10]    |         0|       0|NOTY |GLOB |0    |UNDEF  |_flockrel
[9]     |         0|       0|NOTY |GLOB |0    |UNDEF  |_set_orientation_byte
[7]     |         0|     352|FUNC |GLOB |0    |2      |printf
[1]     |         0|       0|FILE |LOCL |0    |ABS    |printf.c
blackhole{couch}58: cp /usr/lib/libc.so .
blackhole{couch}59: nm libc.so

libc.so:

[Index]   Value      Size    Type  Bind  Other Shndx   Name

[32]    |         0|       0|SECT |LOCL |0    |31     |
[2]     |       148|       0|SECT |LOCL |0    |1      |
[3]     |     16968|       0|SECT |LOCL |0    |2      |
... omitted lots of stuff ... 
[3341]  |    610280|      60|FUNC |WEAK |0    |12     |_door_revoke
[2947]  |    610520|      60|FUNC |WEAK |0    |12     |_door_unbind
[4143]  |    516516|   11344|FUNC |GLOB |0    |12     |_doprnt
[4360]  |    558664|     152|FUNC |GLOB |0    |12     |_doscan
[4353]  |    191860|     132|FUNC |GLOB |0    |12     |_double_to_decimal
... omitted lots of stuff ... 
[1644]  |    358196|      88|FUNC |LOCL |0    |12     |ypstub_perror
[1646]  |    357816|     380|FUNC |LOCL |0    |12     |ypstub_sperror
[541]   |    623808|       8|OBJT |LOCL |0    |16     |zero
blackhole{couch}60: 
...end of install/hello-libc

libc.a contains printf.o
printf.o defines printf
printf.o references the undefined symbol _doprnt
libc.so defines _doprnt.
So if we load hello, printf.o, and libc.so.
- There are no undefined symbols.
- So the program runs!

Taking things apart

Can take apart .o's and .a's!
taking apart .o's and .so's: nm
taking apart .a's and .sa's: ar

ar: taking apart .a's

ar t /usr/lib/libc.a - lists contents of libc.a
ar x /usr/lib/libc.a - extracts actual .o files from archive!
ar c /usr/lib/libfoo.a file1.o file2.o .... - creates an archive.
one subtlety: on some UNIX's you must 'index' the archive, using the program ranlib, after creating it.

nm: taking apart .o's

nm file.o | more - list the 'symbol table' of a .o file.
tells what externals are needed by file.o

Header files.

stdio.h is a header file
purpose: declare things that will be utilized during a program
- names and usage for external functions a program might need to call.
- data types a program might need to use.
- macros a program might want to use (e.g., MAXINT, NULL)
- external global variables a program might need to access.
Common illusions
- header files typically don't contain *definitions* of things.
- they typically contain declarations of things that are defined elsewhere.
- reason: if two .c files include the same .h, and the .h file defines something, then the result is that the thing is multiply defined (a linker error).

Let's look inside a typical header file usr/include/stdio.h

 /*    Copyright (c) 1988 AT&T */
 /*      All Rights Reserved   */
 
 /*    THIS IS UNPUBLISHED PROPRIETARY SOURCE CODE OF AT&T     */
 /*    The copyright notice above does not evidence any        */
 /*    actual or intended publication of such source code.     */
 
 /*
  * Copyright (c) 1993-1998, by Sun Microsystems, Inc.
  * All rights reserved.
  */
 
 /*
  * User-visible pieces of the ANSI C standard I/O package.
  */

The following is a trick to make sure the file isn't included twice.

 #ifndef _STDIO_H
 #define       _STDIO_H
 
 #pragma ident "@(#)stdio.h    1.69    98/07/13 SMI"   /* SVr4.0 2.34.1.2 */

Include lots of other files that contain info about specific features of stdio.

 #include <sys/feature_tests.h>
 #include <sys/va_list.h>
 #include <stdio_tag.h>
 #include <stdio_impl.h>

.. lots of details omitted here for clarity ...

Define the symbol NULL used in pointer manipulation.

 #ifndef       NULL
 #if defined(_LP64) && !defined(__cplusplus)
 #define       NULL    0L
 #else
 #define    NULL    0
 #endif
 #endif

Size of the file buffer: how many bytes to read at one time.

 #define       BUFSIZ  1024

.. lots of detail omitted for clarity here ...

Define the standard input, output, error: they're elements of an external array!

 #if defined(__STDC__)
 extern FILE        __iob[_NFILE];
 #define    stdin   (&__iob[0])
 #define    stdout  (&__iob[1])
 #define    stderr  (&__iob[2])
 #else
 extern FILE   _iob[_NFILE];
 #define       stdin   (&_iob[0])
 #define       stdout  (&_iob[1])
 #define       stderr  (&_iob[2])
 #endif        /* __STDC__ */

.. lots of detail omitted here for clarity ...

Declare types of all functions needed (don't define them!):

 #if defined(__STDC__)
 
 extern int    remove(const char *);
 extern int    rename(const char *, const char *);
 extern FILE   *tmpfile(void);
 extern char   *tmpnam(char *);
 #if   defined(__EXTENSIONS__) || defined(_REENTRANT)
 extern char   *tmpnam_r(char *);
 #endif /* defined(__EXTENSIONS__) || defined(_REENTRANT) */
 extern int fclose(FILE *);
 extern int    fflush(FILE *);
 extern FILE        *fopen(const char *, const char *);
 extern FILE   *freopen(const char *, const char *, FILE *);
 extern void   setbuf(FILE *, char *);
 #if defined(__EXTENSIONS__) || (__STDC__ == 0 && \
       !defined(_XOPEN_SOURCE) && !defined(_POSIX_C_SOURCE))
 extern void setbuffer(FILE *, char *, size_t);
 extern int setlinebuf(FILE *);
 #endif
 extern int    setvbuf(FILE *, char *, int, size_t);
 /* PRINTFLIKE2 */
 extern int fprintf(FILE *, const char *, ...);
 /* SCANFLIKE2 */
 extern int    fscanf(FILE *, const char *, ...);
 /* PRINTFLIKE1 */
 extern int printf(const char *, ...);
 /* SCANFLIKE1 */
 extern int scanf(const char *, ...);
 #if defined(__EXTENSIONS__) || (__STDC__ == 0 && \
       !defined(_XOPEN_SOURCE) && !defined(_POSIX_C_SOURCE)) || \
       defined(_XPG5)
 /* PRINTFLIKE3 */
 extern int    snprintf(char *, size_t, const char *, ...);
 #endif
 /* PRINTFLIKE2 */
 extern int    sprintf(char *, const char *, ...);
 /* SCANFLIKE2 */
 extern int    sscanf(const char *, const char *, ...);
 extern int    vfprintf(FILE *, const char *, __va_list);
 extern int    vprintf(const char *, __va_list);
 #if defined(__EXTENSIONS__) || (__STDC__ == 0 && \
       !defined(_XOPEN_SOURCE) && !defined(_POSIX_C_SOURCE)) || \
       defined(_XPG5)
 extern int    vsnprintf(char *, size_t, const char *, __va_list);
 #endif        /* defined(__EXTENSIONS__) ||  __STDC__ == 0 ... */
 extern int    vsprintf(char *, const char *, __va_list);
 extern int    fgetc(FILE *);
 extern char   *fgets(char *, int, FILE *);
 extern int    fputc(int, FILE *);
 extern int    fputs(const char *, FILE *);
 extern int    getc(FILE *);
 extern int    getchar(void);
 extern char   *gets(char *);
 extern int    putc(int, FILE *);
 extern int    putchar(int);
 extern int    puts(const char *);
 extern int    ungetc(int, FILE *);
 extern size_t fread(void *, size_t, size_t, FILE *);
 extern size_t fwrite(const void *, size_t, size_t, FILE *);
 extern int    fgetpos(FILE *, fpos_t *);
 extern int    fseek(FILE *, long, int);
 extern int    fsetpos(FILE *, const fpos_t *);
 extern long   ftell(FILE *);
 extern void   rewind(FILE *);
 extern void   clearerr(FILE *);
 extern int    feof(FILE *);
 extern int    ferror(FILE *);
 extern void   perror(const char *);
 
 #ifndef       _LP64
 extern int    __filbuf(FILE *);
 extern int    __flsbuf(int, FILE *);
 #endif        /*      _LP64   */

.. lots of detail omitted for clarity ...

getc, putc, getchar, putchar aren't functions; they're macros.

 #ifdef        __STDC__
 #define    getc(p)         (--(p)->_cnt < 0 ? __filbuf(p) : (int)*(p)->_ptr++)
 #define    putc(x, p)      (--(p)->_cnt < 0 ? __flsbuf((x), (p)) \
                               : (int)(*(p)->_ptr++ = (unsigned char) (x)))
 #else
 #define       getc(p)         (--(p)->_cnt < 0 ? _filbuf(p) : (int) *(p)->_ptr++)
 #define       putc(x, p)      (--(p)->_cnt < 0 ? \
                       _flsbuf((x), (p)) : \
                       (int) (*(p)->_ptr++ = (unsigned char) (x)))
 #endif        /* __STDC__ */
 #endif        /* _LP64 */
 
 #define    getchar()       getc(stdin)
 #define    putchar(x)      putc((x), stdout)

etc.

Understanding header files.

The C preprocessor (cccp):
- includes header files as if they were typed in.
- does macro substitutions.
- conditionally omits or includes code according to compile-time flags.
- removes comments.

Looking at what preprocessing does: If hello2.c is:

 #include <stdio.h> 
 main() { putchar('h'); printf("ello world\n"); }

then the command:

 gcc -E hello2.c

produces the preprocessed file.

 # 1 "hello2.c"
 # 1 "/usr/include/stdio.h" 1 3
 #pragma ident "@(#)stdio.h    1.69    98/07/13 SMI"   
 # 1 "/usr/include/sys/feature_tests.h" 1 3
 #pragma ident "@(#)feature_tests.h    1.17    97/12/04 SMI"
 # 1 "/usr/include/sys/isa_defs.h" 1 3
 #pragma ident "@(#)isa_defs.h 1.16    99/05/25 SMI"
 # 218 "/usr/include/sys/isa_defs.h" 3
 # 316 "/usr/include/sys/isa_defs.h" 3
 # 15 "/usr/include/sys/feature_tests.h" 2 3
 # 22 "/usr/include/stdio.h" 2 3
 # 1 "/usr/include/sys/va_list.h" 1 3
 #pragma ident "@(#)va_list.h  1.11    97/11/22 SMI"
 typedef void *__va_list;
 # 23 "/usr/include/stdio.h" 2 3
 # 1 "/usr/include/stdio_tag.h" 1 3
 #pragma ident "@(#)stdio_tag.h        1.3     98/04/20 SMI"
 typedef struct __FILE  __FILE;
 # 24 "/usr/include/stdio.h" 2 3
 # 1 "/usr/include/stdio_impl.h" 1 3
 #pragma ident "@(#)stdio_impl.h       1.7     98/04/17 SMI"
 typedef int   ssize_t;                 
 # 36 "/usr/include/stdio_impl.h" 3
 struct __FILE          
 {
       ssize_t         _cnt;    
       unsigned char   *_ptr;   
       unsigned char   *_base;  
       unsigned char   _flag;   
       unsigned char   _file;   
       unsigned        __orientation:2;  
       unsigned        __filler:6;
 };
 # 25 "/usr/include/stdio.h" 2 3
 typedef       __FILE FILE;
 typedef unsigned int  size_t;          
 typedef       long long       __longlong_t;
 typedef long          off_t;
 typedef __longlong_t  off64_t;
 typedef long          fpos_t;
 typedef __longlong_t  fpos64_t;
 extern FILE   __iob[20 ];
 extern unsigned char   _sibuf[], _sobuf[];
 # 227 "/usr/include/stdio.h" 3
 extern unsigned char  *_bufendtab[];
 extern FILE           *_lastbuf;
 # 257 "/usr/include/stdio.h" 3
 extern int    remove(const char *);
 extern int    rename(const char *, const char *);
 extern FILE   *tmpfile(void);
 extern char   *tmpnam(char *);
 extern int    fclose(FILE *);
 extern int    fflush(FILE *);
 extern FILE   *fopen(const char *, const char *);
 extern FILE   *freopen(const char *, const char *, FILE *);
 extern void   setbuf(FILE *, char *);
 extern void setbuffer(FILE *, char *, size_t);
 extern int setlinebuf(FILE *);
 extern int    setvbuf(FILE *, char *, int, size_t);
 extern int    fprintf(FILE *, const char *, ...);
 extern int    fscanf(FILE *, const char *, ...);
 extern int printf(const char *, ...);
 extern int    scanf(const char *, ...);
 extern int    snprintf(char *, size_t, const char *, ...);
 extern int    sprintf(char *, const char *, ...);
 extern int    sscanf(const char *, const char *, ...);
 extern int    vfprintf(FILE *, const char *, __va_list);
 extern int    vprintf(const char *, __va_list);
 extern int    vsnprintf(char *, size_t, const char *, __va_list);
 extern int    vsprintf(char *, const char *, __va_list);
 extern int    fgetc(FILE *);
 extern char   *fgets(char *, int, FILE *);
 extern int    fputc(int, FILE *);
 extern int    fputs(const char *, FILE *);
 extern int    getc(FILE *);
 extern int    getchar(void);
 extern char   *gets(char *);
 extern int    putc(int, FILE *);
 extern int    putchar(int);
 extern int    puts(const char *);
 extern int    ungetc(int, FILE *);
 extern size_t fread(void *, size_t, size_t, FILE *);
 extern size_t fwrite(const void *, size_t, size_t, FILE *);
 extern int    fgetpos(FILE *, fpos_t *);
 extern int    fseek(FILE *, long, int);
 extern int    fsetpos(FILE *, const fpos_t *);
 extern long   ftell(FILE *);
 extern void   rewind(FILE *);
 extern void   clearerr(FILE *);
 extern int    feof(FILE *);
 extern int    ferror(FILE *);
 extern void   perror(const char *);
 extern int    __filbuf(FILE *);
 extern int    __flsbuf(int, FILE *);
 extern FILE   *fdopen(int, const char *);
 extern char   *ctermid(char *);
 extern int    fileno(FILE *);
 # 358 "/usr/include/stdio.h" 3
 extern FILE   *popen(const char *, const char *);
 extern char   *cuserid(char *);
 extern char   *tempnam(const char *, const char *);
 extern int    getopt(int, char *const *, const char *);
 extern int    getsubopt(char **, char *const *, char **);
 extern char   *optarg;
 extern int    optind, opterr, optopt;
 extern int    getw(FILE *);
 extern int    putw(int, FILE *);
 extern int    pclose(FILE *);
 extern int    fseeko(FILE *, off_t, int);
 extern off_t  ftello(FILE *);
 extern FILE   *fopen64(const char *, const char *);
 extern FILE   *freopen64(const char *, const char *, FILE *);
 extern FILE   *tmpfile64(void);
 extern int    fgetpos64(FILE *, fpos64_t *);
 extern int    fsetpos64(FILE *, const fpos64_t *);
 extern int    fseeko64(FILE *, off64_t, int);
 extern off64_t        ftello64(FILE *);
 # 515 "/usr/include/stdio.h" 3
 # 567 "/usr/include/stdio.h" 3
 # 1 "hello2.c" 2
 main() { 
 (--(  (&__iob[1])  )->_cnt < 0 
   ? __flsbuf(( ( 'h' ) ), (  (&__iob[1])  )) 
   : (int)(*(  (&__iob[1])  )->_ptr++ = (unsigned char) ( ( 'h' ) )))  ; 
  printf("ello world\n"); 
 }

The putchar was expanded into a complex piece of code. This is called inlining.

Preprocessing, compiling, linking

once the program is preprocessed, it gets compiled.
after compilation: it gets linked.
then it's ready to run.

Recall: Process execution environment:

PATH: list of directories where programs can be found.
new paths specifically for linking!
LD_LIBRARY_PATH: list of directories in which process libraries can be found (both compile-time and run-time)
LD_RUN_PATH: list of *directories* in which process run-time libraries can be found. Only on some UNIX's, e.g., Solaris.

Static linking: (ld)

called implicitly by gcc.
build a program image incrementally.
scan each input file in turn.
for files ending in .o, simply load the code from them and make links between undefined externals in one file and their references in another.

for files ending in .a, link their .o's conditionally whenever a definition of a symbol in one file matches a reference in the linked image you have so far.

 foo.c:                    bar.c:                
  int x;                    extern int x; 
  extern int y;             int y; 
  extern int foo(); 
 main() {                  foo() { 
  x=2; /* local */          x=4; /* extern */ 
  y=9; /* extern */         y=2; /* local */ 
  foo(); /* extern */ 
 }                         }
  foo.o:                  bar.o:               
    DEFINES x ------------> EXTERNAL x              
    EXTERNAL y <----------- DEFINES y             
    EXTERNAL foo <----------DEFINES foo

Very important notes on linking:

compilation of objects gives them types:
- function
- variable (int, double, struct, pointer)
linking operation is typeless (just address)

one can really mess up things by doing:

 foo.c: int printf; 
 bar.c: printf("ho ho ho\n");

This tries to execute an int! (Illegal Instruction - Core dumped)

Result of static linking:

At the end of this process, we get a.out
all normal static external symbols resolved.
yet to be resolved: dynamic externals.
a.out consists of all the code for the program that's private and unshareable.
yet to be determined: how to link to 'shared' code.

Execution model:

programs (a.out) are organized in segments:
- TEXT: program code, read-only.
- STACK: space for variables
- BSS: space for global variables.
each segment gets pages of memory to use.
- 4096 bytes or otherwise.
- read-only or read-write
- shared or private

Files differ in what kind of pages they're loaded into

.o: private text
.so: shared text (multiple processes can get to these pages by name)
to share a page, must link to it dynamically (at the time a process gets executed)

Dynamic linking process:

ld.so: dynamic linker.

 a.out:                libc.so: 
  external _doprnt <----- defines _doprnt

goes through a TABLE in a.out listing references to dynamic links.
loads .so and .sa data and links to it as appropriate.
once there are no undefined externals, a.out is ready to run (in memory)
so it gets executed.

This is relatively easy to break:

ld: undefined externals
- no library specified.
- LD_LIBRARY_PATH wrong.
- -L compiler option wrong.
ld.so: undefined externals
- moved libc.so
- LD_LIBRARY_PATH or LD_RUN_PATH wrong.
- ld.so missing(!)

Compiler defaults:

implicitly scan /usr/lib/libc.a.
implicitly scan /usr/lib for all other libraries.
implicitly scan everything in LD_LIBRARY_PATH for other libraries.

Compiler options:

-c: don't call ld; just create .o file.
-lfoo: scan libfoo.a for other externals. Search all appropriate directories.
-L/local/lib: scan /local/lib for libraries as well as /usr/lib.
-L/local/lib -lfoo means to check for and scan /local/lib/libfoo.a if libfoo.a is in /local/lib.

Static and dynamic directory binding:

to run, a compiled program has to find its libraries!
two ways to compile a program
- static directory scanning: always look in the same places for .so's.
- dynamic directory scanning: look wherever LD_LIBRARY_PATH or LD_RUN_PATH says to look.
LD_LIBRARY_PATH:
- can break by setting LD_LIBRARY_PATH wrong.
- can't break by moving libraries around if you change LD_LIBRARY_PATH
-L/usr/lib (compile time option) (or -R/usr/lib)
- can't break by changing LD_LIBRARY_PATH
- can break by moving libraries elsewhere.

Compiler caveat:

But you'll almost never execute a compiler directly
Instead, you'll execute compilation meta-tools that call the compiler for you!

Epilogue: what's the difference between gcc and g++

Answer: one command.
- gcc: real compiler is cc1
- g++: real compiler is cc1plus

Compiling with gcc:

contents of install/hello-gcc...
blackhole{couch}51: gcc --verbose hello.c
Reading specs from /loc/gnu/gcc-2.8.1a/sol/5.7/lib/gcc-lib/sparc-sun-solaris2.7/2.8.1/specs
gcc version 2.8.1
 /loc/gnu/gcc-2.8.1a/sol/5.7/lib/gcc-lib/sparc-sun-solaris2.7/2.8.1/cpp -lang-c -v -undef -D__GNUC__=2 -D__GNUC_MINOR__=8 -Dsparc -Dsun -Dunix -D__svr4__ -D__SVR4 -D__sparc__ -D__sun__ -D__unix__ -D__svr4__ -D__SVR4 -D__sparc -D__sun -D__unix -Asystem(unix) -Asystem(svr4) -D__GCC_NEW_VARARGS__ -Acpu(sparc) -Amachine(sparc) hello.c /var/tmp/ccx3aa1W.i
GNU CPP version 2.8.1 (sparc)
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /loc/gnu/gcc-2.8.1a/sol/5.7/sparc-sun-solaris2.7/include
 /loc/gnu/gcc-2.8.1a/sol/5.7/lib/gcc-lib/sparc-sun-solaris2.7/2.8.1/include
 /usr/include
End of search list.
 /loc/gnu/gcc-2.8.1a/sol/5.7/lib/gcc-lib/sparc-sun-solaris2.7/2.8.1/cc1 /var/tmp/ccx3aa1W.i -quiet -dumpbase hello.c -version -o /var/tmp/ccx3aa1W.s
GNU C version 2.8.1 (sparc-sun-solaris2.7) compiled by GNU C version 2.8.1.
 /usr/ccs/bin/as -V -Qy -s -o /var/tmp/ccx3aa1W1.o /var/tmp/ccx3aa1W.s
/usr/ccs/bin/as: WorkShop Compilers 5.0 Alpha 03/27/98 Build
 /usr/ccs/bin/ld -V -Y P,/usr/ccs/lib:/usr/lib -Qy /loc/gnu/gcc-2.8.1a/sol/5.7/lib/gcc-lib/sparc-sun-solaris2.7/2.8.1/crt1.o /loc/gnu/gcc-2.8.1a/sol/5.7/lib/gcc-lib/sparc-sun-solaris2.7/2.8.1/crti.o /usr/ccs/lib/values-Xa.o /loc/gnu/gcc-2.8.1a/sol/5.7/lib/gcc-lib/sparc-sun-solaris2.7/2.8.1/crtbegin.o -L/loc/gnu/gcc-2.8.1a/sol/5.7/lib/gcc-lib/sparc-sun-solaris2.7/2.8.1 -L/loc/gnu/gcc-2.8.1a/sol/5.7/sparc-sun-solaris2.7/lib -L/usr/ccs/bin -L/usr/ccs/lib -L/loc/gnu/gcc-2.8.1a/sol/5.7/lib /var/tmp/ccx3aa1W1.o -lgcc -lc -lgcc /loc/gnu/gcc-2.8.1a/sol/5.7/lib/gcc-lib/sparc-sun-solaris2.7/2.8.1/crtend.o /loc/gnu/gcc-2.8.1a/sol/5.7/lib/gcc-lib/sparc-sun-solaris2.7/2.8.1/crtn.o
ld: Software Generation Utilities - Solaris/ELF (3.0)
blackhole{couch}52: 
...end of install/hello-gcc

Compiling with g++:

contents of install/hello-g++...
blackhole{couch}52: g++ --verbose hello.cc
Reading specs from /loc/gnu/gcc-2.8.1a/sol/5.7/lib/gcc-lib/sparc-sun-solaris2.7/2.8.1/specs
gcc version 2.8.1
 /loc/gnu/gcc-2.8.1a/sol/5.7/lib/gcc-lib/sparc-sun-solaris2.7/2.8.1/cpp -lang-c++ -v -undef -D__GNUC__=2 -D__GNUG__=2 -D__cplusplus -D__GNUC_MINOR__=8 -Dsparc -Dsun -Dunix -D__svr4__ -D__SVR4 -D__sparc__ -D__sun__ -D__unix__ -D__svr4__ -D__SVR4 -D__sparc -D__sun -D__unix -Asystem(unix) -Asystem(svr4) -D__EXCEPTIONS -D__GCC_NEW_VARARGS__ -Acpu(sparc) -Amachine(sparc) hello.cc /var/tmp/ccjaaO1W.ii
GNU CPP version 2.8.1 (sparc)
#include "..." search starts here:
#include <...> search starts here:
 /loc/gnu/gcc-2.8.1a/sol/5.7/include/g++
 /usr/local/include
 /loc/gnu/gcc-2.8.1a/sol/5.7/sparc-sun-solaris2.7/include
 /loc/gnu/gcc-2.8.1a/sol/5.7/lib/gcc-lib/sparc-sun-solaris2.7/2.8.1/include
 /usr/include
End of search list.
 /loc/gnu/gcc-2.8.1a/sol/5.7/lib/gcc-lib/sparc-sun-solaris2.7/2.8.1/cc1plus /var/tmp/ccjaaO1W.ii -quiet -dumpbase hello.cc -version -o /var/tmp/ccjaaO1W.s
GNU C++ version 2.8.1 (sparc-sun-solaris2.7) compiled by GNU C version 2.8.1.
 /usr/ccs/bin/as -V -Qy -s -o /var/tmp/ccjaaO1W1.o /var/tmp/ccjaaO1W.s
/usr/ccs/bin/as: WorkShop Compilers 5.0 Alpha 03/27/98 Build
 /usr/ccs/bin/ld -V -Y P,/usr/ccs/lib:/usr/lib -Qy /loc/gnu/gcc-2.8.1a/sol/5.7/lib/gcc-lib/sparc-sun-solaris2.7/2.8.1/crt1.o /loc/gnu/gcc-2.8.1a/sol/5.7/lib/gcc-lib/sparc-sun-solaris2.7/2.8.1/crti.o /usr/ccs/lib/values-Xa.o /loc/gnu/gcc-2.8.1a/sol/5.7/lib/gcc-lib/sparc-sun-solaris2.7/2.8.1/crtbegin.o -L/loc/gnu/gcc-2.8.1a/sol/5.7/lib/gcc-lib/sparc-sun-solaris2.7/2.8.1 -L/loc/gnu/gcc-2.8.1a/sol/5.7/sparc-sun-solaris2.7/lib -L/usr/ccs/bin -L/usr/ccs/lib -L/loc/gnu/gcc-2.8.1a/sol/5.7/lib /var/tmp/ccjaaO1W1.o -lstdc++ -lm -lgcc -lc -lgcc /loc/gnu/gcc-2.8.1a/sol/5.7/lib/gcc-lib/sparc-sun-solaris2.7/2.8.1/crtend.o /loc/gnu/gcc-2.8.1a/sol/5.7/lib/gcc-lib/sparc-sun-solaris2.7/2.8.1/crtn.o
ld: Software Generation Utilities - Solaris/ELF (3.0)
blackhole{couch}53: 
...end of install/hello-g++

lecture in color

/comp/150NET/notes/install.php
downloaded on Nov-23-2009 02:57:47 PM,
was last modified on Feb-17-2004 10:48:31 PM.
All lecture note content is copyright 2004 by
Alva L. Couch, Computer Science, Tufts University