System and Network Administration

lecture in color

Troubleshooting

Of programs.
Of networks.
Too-common task.
90% of network administration.

Why is troubleshooting necessary?

can't control environment.
can't predict effects of changes.

Real life story (today)

came in this morning.
workstation didn't know my home directory.
web server down.

What I did:

scream in agony (mandatory).
reboot my workstation -> works
- conclusion: problem was move of filesystems accomplished last night.
reboot web server -> fails.
- conclusion: file service is not the problem for the web server.
manually started web server
```
 /etc/init.d/apachectl start
```
- complaint: can't find libz.so.1
- conclusion: LD_LIBRARY_PATH or contents got messed up.

checked /local/lib: required file there.

 % ls -l /local/lib/libz*
 lrwxrwxrwx   1 dert     staff         48 Apr  5 15:52 /local/lib/libz.a -> /loc/utils/zlib-1.1.3/sunos-sun4u-5.7/lib/libz.a
 lrwxrwxrwx   1 dert     staff         55 Apr  5 15:52 /local/lib/libz.so -> /loc/utils/zlib-1.1.3/sunos-sun4u-5.7/lib/libz.so.1.1.3
 lrwxrwxrwx   1 dert     staff         55 Apr  5 15:52 /local/lib/libz.so.1 -> /loc/utils/zlib-1.1.3/sunos-sun4u-5.7/lib/libz.so.1.1.3
 lrwxrwxrwx   1 dert     staff         55 Apr  5 15:52 /local/lib/libz.so.1.1.3 -> /loc/utils/zlib-1.1.3/sunos-sun4u-5.7/lib/libz.so.1.1.3

Oops! This is a new version!

checked the file itself:

 % cd /loc/utils/zlib-1.1.3/sunos-sun4u-5.7/lib
 % ls -l 
 total 286
 -rwxr-xr-x   1 dert     staff      73768 Apr  5 15:52 libz.a
 lrwxrwxrwx   1 dert     staff         13 Apr  5 15:52 libz.so -> libz.so.1.1.3
 lrwxrwxrwx   1 dert     staff         13 Apr  5 15:52 libz.so.1 -> libz.so.1.1.3
 -rwxr-xr-x   1 dert     staff      68932 Apr  5 15:52 libz.so.1.1.3

checked /loc/utils and found /loc/utils/zlib-1.1.3-old.
- Conclusion: new version has problems.
changed script conbrio:/etc/init.d/apachectl so that LD_LIBRARY_PATH points to old version:
```
 LD_LIBRARY_PATH=/loc/utils/zlib-1.1.3-old/lib:/local/lib; export LD_LIBRARY_PATH
```
started apache again => worked.
- Conclusion: zlib needs help, apache now OK.
Full problem description:
- ran web server
- changed zlib.
- didn't reboot => worked fine.
- reboot => failure.

What was hard?

Problem was a hidden dependency.
First: must acknowledge that one might exist.
Then: must make a test to see whether it does.
As one makes tests, keep a history of all data from which to draw conclusions.
Reconcile inconsistencies with further tests.
Be prepared to learn something about the true dependencies that you never suspected.
Really fast (this whole process took me 10 minutes).

Dependencies

NFS client -> NFS server.
named client -> named server.
program -> program files.
network node -> gateway.
printer -> print server.

Program troubleshooting

There are a lot of variables involved in installing any program successfully:
executables(binaries)
compile-time libraries needed.
- lm: /usr/lib/libm.a
- lcurses: /usr/lib/libcurses.a
- ltermcap: /usr/lib/libtermcap.a implicit /loc/gnu/gcc-2.6.2/lib/gcc-lib/sparc-sun-solaris2.3/2.6.2/libgcc.a
include files needed. /usr/include/curses.h /usr/include/termcap.h /usr/include/stdlib.h
run-time libraries needed. /usr/lib/libc.so
documentation (man pages, emacs info, other) /usr/man/man3/curses.3 /usr/man/man3/termcap.3

My program troubleshooting chart:

program doesn't work
- program not found?
  - check path for appropriate bin
    - fix path if wrong, rehash
  - check bins in path for program
    - fix bins if wrong
- dynamic library not found
  - check LD_LIBRARY_PATH
    - fix if wrong
  - check library locations for library: /lib, /usr/lib, /usr/dt/lib, etc.
  - check whether program is statically bound.
    - If so, you may have to make a symlink or mount to help it find the appropriate library.
      - symlink if easy fix or file.
      - use mount if major directory.
- program doesn't find info it needs once running and linked
  - environment wrong
    - check environment
      - if wrong, either fix .cshrc or tell user how to fix.
    - check .cshrc, /etc/csh.cshrc for problems
      - fix if found
    - tell user how to set variables manually
      - repeat 1000 times.
    - write source scripts that set the @#$^&* environment variables.
    - envelope programs in scripts that set the @#$^&* variables correctly and then invoke the program.
  - Comment: often two programs will CONFLICT in their needs for environment settings. Esp. large CAD packages.
  - program can't find its files.
    - check what files it's looking for:
```
 strings program | more
```
      gets a list of strings from the program.
    - if files there, suspect contents
      - contents bad -> reinstall program
    - all else fails, reinstall from scratch.
  - program has license failure
    - check FM_FLS_HOST and FM_LICENSE_FILE environment variables.
    - check that daemon running on FM_FLS_HOST (lmgrd)
    - restart daemon
    - reboot license server
    - fend off 1000 crass comments about the @#$^&* who rebooted the server.
  - program halts and does nothing for awhile
    - too many processes:
```
 ps -ef
```
      - reach out and crush someone's processes (typically your instances of nethack)
    - out of memory or disk
```
 df -k /var/tmp
 df -k /tmp
```
      - clean up if possible
      - reboot to clear /tmp (except directories)
    - network services unavailable
      - netstat -a to see whether program is bound to an ip port.
      - check program documentation for port use.
      - port listed in /etc/services?
        
        try listing it, try again
      - other host providing services is up?
        
        no, call someone; tell them to fix it.
      - can ping other host?
        
        no, check routing table (netstat -r)
        
        wrong, use static route as temporary fix.
        
        doesn't work: use traceroute to figure out WHO messed it up.
      - can get to other host for other services?
      - daemon running on other end?
        
        check /etc/services for port
        
        check /etc/inetd.conf for configuration.
        
        check that /etc/init.d contains script to start daemon.
        
        check whether /etc/rc.local starts it.
        
        check whether /etc/rc2.d invokes the script.
program doesn't compile
- compiler not found
  - check path
    - put compiler bin into path
  - check compiler bin
    - put compiler there if absent
- include file not found
  - check system include files (may be hidden for gcc)
  - check spelling, revision
  - refer to man page for function to find different include file that applies. strcat is declared in
    1. /usr/include/string.h
    2. /usr/include/strings.h
    3. /usr/include/stdlib.h depending upon your system. (check /usr/include, /usr/include/sys)
  - write in declarations manually based upon all available info.
    - into one of your header files for the software.
    - into c code as appropriate.
  - slap yourself for not backing up your sources first.
- undeclared type or function: problem with include files.
  - check reference for validity in source code
    - if valid, look for declaration in /usr/include/* and /usr/include/sys/*
    - if there, include the appropriate file.
- converts pointer to integer or integer to pointer without a cast.
  - usually NOT a problem.
  - artifact of C: pointers and integers used interchangeably by some old-school programmers (me and dwk).
- compile-time library not found
  - check system library directories for the offending library, possibly a different revision. /lib, /usr/lib, /local/lib.
    - If it's there, check LD_LIBRARY_PATH (dynamic binding) or use the -L/local/lib linker option (static) to hard-code its location into the compile line (man ld)
    - If it's not there, look in the manual to see if its documented as being elsewhere. SYNOPSIS section of section 3 commands. Change compile line -lfoo.a -> -lbar.a as appropriate.
- undefined external 'foo'
  - library not included
    - man foo, man -k foo to figure out whether there's a foo anywhere in your system.
      - if yes, use -lfooloc instead of -lfoo as ordered by manual pages.
        
        local winners of the mistaken identity contest:
        
        lsocket: some systems put this in the default library.
        
        lnsl: network socket library, ditto
    - search network for references to library foo, compile it here.
    - rewrite source not to use library.

Libraries and includes:

biggest problem in installing programs: "include file not found" "library not found"
In general
- /usr/include/foo.h gives DECLARATIONS of the definitions in /usr/lib/libfoo.a
- /usr/lib/libfoo.a may optionally refer to SHARED definitions in /usr/lib/libfoo.so.1.2 or /usr/lib/libfoo.sa.1.2
- /usr/include/stdio.h:
```
 extern void printf(const char *, ...) ; 
```
  /usr/lib/libc.a:
```
 printf.o
```
  printf.c:
```
 void printf(const char *, ...) { 
  ...
 }
```

Dynamic and static linking and binding:

static linking: libraries pre-linked at compile-time.
dynamic linking: libraries found and linked at execution time. Can have
- dynamic binding: library names determined by looking at LD_LIBRARY_PATH
  - the environment for the runtime program must include these!
- static binding: library names compiled in.
  - decoupled from environment.

A dynamically linked executable

looks for .so's and .sa's when you run it.
in a predetermined place (static binding)
in a flexible place (dynamic binding)

A statically linked executable

has contents of .so's and .sa's hard-coded (and unshareable)

Compile options:

-R/usr/openwin/lib: says to use /usr/openwin/lib as a runtime library and override all settings of LD_RUN_PATH and LD_LIBRARY_PATH
-L/usr/openwin/lib: same for compile-time library, both together.
Also, must be able to find included header files:
-I/usr/openwin/include: look for #include <openwin.h> in /usr/openwin/include/openwin.h as well as in standard places.
```
 #include <X11/X11.h>
```
in sources means to search all includes in the current search path as determined by predefined locations, -I commands
Sometimes INCLUDE_PATH works.

Compiling errors and how to deal with them:

incompatible headers:
- "undefined type" or "no size for type"
- Procedure:
```
 man function   # to tell what the relevant libraries and include files are
 more /usr/include/sys/blah.h
```
  Edit the code to conform to function prototypes

undefined externals

"symbol not found"

Procedure:

 man function   # to tell you, in this case, where the LIBRARY is
 man 3 function # makes sure you're looking for a FUNCTION, not a command

put -lnsl, -lucb (man page tells you to use /usr/ccs/bin/cc!) into compile line

Navigating the manual space:

section 1: user commands (things you type)
section 2: very low level c subroutines (calls directly to the UNIX kernel).
section 3: application subroutines: these are what are typically used by normal programmers.
section 4: file formats: how to edit hand-editable files; how to parse binary files with programs
section 5: user libraries for normal programmers.
section 6: games and demos
section 7: devices: hardware and software devices; instruction for use, control, configuration, etc.
section 8: commands appropriate for root
section 9: device drivers: none of these

Manual tricks:

how to read an uninstalled man page
```
     nroff -man foo.1 | more     
```
(-m=macro, an=the UNIX manual macro set)

how to find manual pages: SOURCES: before nroff

     /usr/man/man[1-9nl]
     /usr/share/man/man[1-9nl][a-zA-Z]?
     /usr/local/man/man[1-9nl][a-zA-Z]?
     /local/man/man[1-9nl][a-zA-Z]?
     /usr/openwin/man/man[1-5]
     /loc/X11/man/man[1-5]

COMPILED man pages: s/man/cat in all the above.
MANPATH: how to put a new manual hierarchy into your own
```
     setenv MANPATH /local/new/man:${MANPATH}
```

indexing:

     man -k think | more    # lists every man page ABOUT a thing

joke in sun default .cshrc:
```
     alias fem man
```

Comments on maintaining man pages:

man will save formatted copies of pages if there's a cat1, cat2, cat3 directory in manual space.
```
 /usr/man/man1/man.1 -> /usr/man/cat1/man.1
```

two separate indexes for man -k:

 /usr/man/whatis: to make: /usr/lib/makewhatis. 
 /usr/man/windex: to make /usr/sbin/catman -w

format all man pages into text:
```
 /usr/sbin/catman
```

In practice, software installation proceeds as

failure
quick fix
reverse-engineer the correct fix.

Typical:

configure generates Makefile
you hack it into working.
works
reverse engineer arguments to configure so it works correctly next time.

Network troubleshooting

big problem: localize the difficulty.
same pattern: perform experiments, keep notes, infer causes from notes.

Real life example

Installed new router.
Rebooted stations.
Small number of stations didn't reboot.
Conclusion: something about router is funny.
Edited startup script /etc/init.d on affected machines to print what started.
Rebooted again: failure in NIS+ startup.
Scratched head, had dinner.
Snooped on network of affected host to watch NIS+ packets.
- Server isn't answering.
Went to server and snooped the same way.
- Server is answering.
Conclusion: something about router or answer is funny (but not that funny on a fried network at 10 pm on a Fri!)
made 3 hours of further tests to no avail.
described problem to TCCS expert.
he suggested that multihoming might be a problem
checked this theory by looking at addressing of packets
- Packets were being answered on the wrong interface.
- Conclusion: multi-homing is broken on NIS+ master servers.
action: single-home all servers.
(12 hours later): everything works.
Conclusion: multi-homing theory correct.
It's not over.
Hit manuals.
- router manual: can avoid this via operating in "closed VLAN mode".
- sun manual: can avoid this by changing instructions to the kernel regarding networking.
Applied both fixes => worked => original problem solved.

Network Troubleshooting Chart.

host can't contact network
- is our interface configured correctly?
```
 % ifconfig 
```
  - no: check /etc/sysconfig/network (on lunix) or /etc/hostname.hme0, /etc/netmasks (on suns)
    - wrong => correct, reboot to test.
    - right => /etc/rc.d scripts corrupt, check and perhaps copy from elsewhere.
- can we ping ourselves numerically?
```
 % ping 127.0.0.1
 % ping 130.64.24.98
```
  - no: ifconfig actually wrong. Go back>
- is output of netstat -rn OK?
```
 % netstat -rn
```
- can we ping the nearest router in that display?
```
 % ping 130.64.24.127
```
  - if not: check whether routing table has been correctly set up.
    - manual route commands.
    - /etc/sysconfig/network (linux).
    - /etc/defaultrouter (suns).
- can we ping other hosts farther away, one hop at a time.
  - no: some router or firewall is mis-configured.
    - use traceroute to find it.
    - call that admin to fix it.
- (At this point, we can ping anyone by number.)
- can we look up our own IP address?
```
 % nslookup
 > foo.eecs.tufts.edu
```
  - if not: name mapping is messed up
    - check /etc/hosts
    - check /etc/resolv.conf
- can we ping ourselves by name?
```
 % ping foo.eecs.tufts.edu
```
  - if not found, check /etc/nsswitch.conf to make sure you're using the right name service.
- can we ping other hosts by name?
  - no: their name service is somehow broken.
  - call the appropriate administrator.
- (at this point, we can ping everyone by name)
- can we use higher-level services to other host?
  - no: suspect host problem or firewall limit.
    - check other host for proper function
    - check whether firewall blocks proper connection.
- if we get this far, we're able to work.
nfs mount doesn't work
- does client have network connectivity?
  - no: continue from above.
- does client have right mount request?
  - check remote host name.
  - check that remote host name is known in /etc/hosts or nslookup.
  - check that remote directory exists.
  - check that remote directory is exported from server host.
- does server permit operation?
  - is /etc/exports (or /etc/dfs/dfstab on suns) correct?
  - is client host known (in /etc/hosts)?
printing doesn't work.
- user typed command correctly?
  - no: instruct user.
- printing from software package?
  - yes: check package printing configuration.
- /var/spool full?
  - yes: clear up space by deleting or storing logfiles elsewhere.
- user trying to print supported file type
  - no: supply converter (e.g., pdf to pcl).
    - ps: postscript.
    - pcl: HP printer control language
    - impress: imagen control language.
    - troff: text runoff (old newspaper typesetters).
    - a.out: spank user!
- local print daemon running?
```
 ps -ef | grep lpd
```
  or
```
 ps -ef | grep lpsched
```
  on suns.
  - no: try starting print daemon.
    - starting daemon works?
      - no: check /var/spool/lpd (or equivalent) for bad print job, delete.
      - no: check for misconfiguration in configuration file.
- (at this point, the daemon is running)
- printer configured in /etc/printers.conf or /etc/printcap
  - remote printer: can we ping printer? Go to network debugging above.
  - local printer: is it connected and powered up?

lecture in color

/comp/150NET/notes/trouble-old.php
downloaded on Nov-23-2009 04:40:06 PM,
was last modified on Feb-17-2004 10:50:03 PM.
All lecture note content is copyright 2004 by
Alva L. Couch, Computer Science, Tufts University