Being a generalist

• The next level: becoming a *generalist*.
• Knowing what could have caused a specific behavior.
• Not just knowing how to do something, but all ways of doing it.

Example: daemons

• know one way to start a daemon.
• Turns out that there are at least four ways.

Four ways to start a daemon:

• in /etc/rc.d/rc and sub-scripts in /etc/rc.d/rc?.d/S*
• as a child of init: /etc/inittab.
• as a child of inetd: /etc/inetd.conf (or /etc/inet/inetd.conf).
• as a child of cron: /var/spool/cron/, /etc/cron.d/.

init and /etc/inittab

• init is process 1;
• everything is a descendant.
• specifies daemons that are absolutely essential for any kind of multiuser operation.
  • getty: accept login.

     S0:2345:respawn:/sbin/getty ttyS0 DT9600 vt100
     || |  | |     | |         | ^^^^^^^^^^^^^^^^^^ command line arguments
     || |  | |     | ^^^^^^^^^^^ name of daemon
     || |  | ^^^^^^^ action if it dies 
     || ^^^^ run levels that invoke it
     ^^ name
    

    • run levels 2-5
    • name of service: S0
    • restart if it dies.
    • make a getty on ttyS0, 9600 baud, emulating a vt100.
    • see /etc/gettytab for details on DT9600.
  • prefdm: X display manager

     x:5:respawn:/etc/X11/prefdm -nodaemon
     | | |     | |             | ^^^^^^^^^ arguments
     | | |     | ^^^^^^^^^^^^^^^ command
     | | ^^^^^^^ restart if dies 
     | ^ run level 5 only
     ^ name
    

    • run level 5 only
    • restart if dies.
    • -nodaemon: don't detach console (helps init know when to restart it)

Common pattern: restarting a meta-daemon

• To restart a daemon, so that it re-reads its databases, send it the HANGUP signal (-1).
• kill -HUP 1 (or kill -1 1: instructs init to re-read its databases and change the state of its daemons (either stopping or starting them, as appropriate to your run-level).
  • -HUP: send a hangup signal.
  • 1: process ID of init (always).
• This is a common pattern for many daemons.
• but it's daemon-specific; a daemon can choose to interpret or not interpret HUP signals this way.

inetd and inetd.conf

• inetd is the internet meta-daemon.
• role: start other daemons as needed.
• List of daemons to start stored in inetd.conf

   telnet stream tcp nowait root /usr/sbin/in.telnetd in.telnetd
   |    | |    | | | |    | |  | |                  | ^^^^^^^^^^ name of daemon
   |    | |    | | | |    | |  | ^^^^^^^^^^^^^^^^^^^^ true path of daemon
   |    | |    | | | |    | ^^^^ what user
   |    | |    | | | ^^^^^^ what to do after starting daemon
   |    | |    | ^^^ network layer to use
   |    | ^^^^^^ type of network connection 
   ^^^^^^ name of service (see /etc/services)
  

• Compare this with another service:

   talk   dgram  udp wait   root /usr/sbin/tcpd in.talkd
   |  |   |   |  | | |  |   |  | |            | ^^^^^^^^ name of daemon
   |  |   |   |  | | |  |   |  | ^^^^^^^^^^^^^^ command to run 
   |  |   |   |  | | |  |   ^^^^ what user to run as
   |  |   |   |  | | ^^^^   wait for completion before starting another
   |  |   |   |  ^^^ network layer to use
   |  |   ^^^^^ type of network connection 
   ^^^^ name of service
  

• tcp and udp: types of network connections.
  • roughly correspond to stream and dgram (redundant)
  • same meaning as for network layers.
• wait and nowait: whether to wait to start another instance
  • wait: only one instance should run at a time.
  • nowait: run one instance per request. This behavior must be programmed into the daemon.
• daemon command versus daemon name.
  • command: what command to run
  • name: what name to identify it as.
  • Note the last example: /usr/sbin/tcpd aliased to in.talkd.
  • This is an important variant.

services and /etc/services

• /etc/services: describes what services are available by name
  • for tcp:

     telnet  23/tcp  
                ^^^ protocol (tcp or udp) (see /etc/protocols) 
             ^^ port number between 0-65535
     ^^^^^^ service name (same as first field of /etc/inetd.conf)
    

  • for udp:

     talk    517/udp
                 ^^^ protocol (see /etc/protocols) 
             ^^^ port number
     ^^^^ service name (same as first field of /etc/inetd.conf) 
    

protocols and /etc/protocols

• /etc/protocols changes the numbers in a protocol headers to words we can read.

   tcp  6 TCP  # transmission control protocol
   udp 17 UDP  # user datagram protocol
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ comments
          ^^^ aliases
       ^^ protocol number
   ^^^ protocol name
  

The numbers game:

• Computers describe services by protocol number, ip number and port number.
• We want to describe them by name instead.
• /etc/protocols, /etc/services, /etc/inetd.conf are simply ways of naming things so we can understand them.

Structure of a network connection:

• A connection between two machines is determined by a quintuple:
  • base protocol (tcp or udp)
  • client ip address.
  • client port (usually randomly chosen by client software)
  • server ip address.
  • server port (usually set by /etc/inetd.conf, /etc/services)
• To see what's going on on your machine, use netstat -a:

   UDP
      Local Address         Remote Address     State
   -------------------- -------------------- -------
         *.sunrpc                              Idle 
         *.*                                   Unbound
   ...etc...
         *.talk                                Idle
   ...etc...
   
   TCP
      Local Address        Remote Address    Swind Send-Q Rwind Recv-Q  State
   -------------------- -------------------- ----- ------ ----- ------ -------
         *.*                  *.*                0      0     0      0 IDLE
   ...etc...
         *.telnet             *.*                0      0     0      0 LISTEN
   ...etc...
   Conbrio.EECS.Tufts.EDU.689 Conmoto.EECS.Tufts.EDU.nfsd  8760      0  8760    120 ESTABLISHED
   ...
   Conbrio.EECS.Tufts.EDU.22 Louvre.EECS.Tufts.EDU.2571  7908      0  8760      0 ESTABLISHED
   ...
  

  • each line describes one daemon.
  • it reports what port the daemon is listening upon.
  • but not that the proper daemon is listening to the proper port!
• Whenever possible, netstat translates between numbers and names.
• Example: NFS
  • Conbrio.EECS.Tufts.EDU.689 means ip 130.64.23.39 port 689
  • Conmoto.EECS.Tufts.EDU.nfsd means ip 130.64.23.38 port 2049
  • ip lookups from DNS, port lookups from /etc/services!
  • This line describes one NFS mount (there are plenty).
• Example: ssh
  • Conbrio.EECS.Tufts.EDU.22 means ip 130.64.23.39 port 22 (ssh)
  • Louvre.EECS.Tufts.EDU.2571 means ip 130.64.24.90 port 2571
  • Why no ssh lookup: not in /etc/services!

Watch out:

• if all is well-behaved, telnetd is listening for telnet.
• This need not be the case!
• Common attack: replace telnetd with something else.
• reads username and password (and stores)
• denies access, looking like a system failure.
• result: hacker gets access to all usernames and passwords.
• This may seem bad, but it's the least of our worries when telnet is concerned.

Some ports are special

• In a typical UNIX operating system, port numbers below 1000 are reserved for root.
• So you know, if you're getting a connection from such a port, that it's coming from a privileged account.
• No such convention on Windoze.

TCP wrappers

• way of maintaining host security.
• basic idea: intercept daemon requests before starting the daemon.
  • If a request is valid, allow it through to the real daemon.
  • If a request is invalid, blackhole it and perhaps do other things.

Installing TCP wrappers

1. put the tcpd program into the same directory as the daemon.
2. edit /etc/inetd.conf to call tcpd instead of the real daemon.
3. edit /etc/hosts.allow and /etc/hosts.deny to tell which daemons to allow or deny access to. Most people just use hosts.allow.

Example: andante telnet:

• /etc/inetd.conf:

   telnet stream tcp nowait root /usr/sbin/tcpd in.telnetd
  

  (real telnet is /usr/sbin/in.telnetd)
• /etc/hosts.allow:

   in.telnetd : emerald.tufts.edu : banners /var/local/tcpd/banners/has-ssh : \
     spawn (/usr/bin/logger -i -p daemon.notice denied %s to %u@%h) & : DENY
   in.telnetd : .eecs.tufts.edu : banners /var/local/tcpd/banners/warn : \
     spawn (/usr/bin/logger -i -p daemon.notice warned %s to %u@%h) & : ALLOW
   in.telnetd : ALL : banners /var/local/tcpd/banners/warn : \
     spawn (/usr/bin/logger -i -p daemon.notice warned %s to %u@%h) & : ALLOW
  

  To wit:
  • deny access to emerald and tell them they're idiots for not using SSH!
  • allow access within eecs.tufts.edu but warn everyone about the dangers.
  • allow access from the internet but warn everyone about the dangers.

Starting inetd:

• To start inetd, use /etc/rc.d/init.d/network
  • /etc/rc.d/init.d/network start
  • /etc/rc.d/init.d/network stop
• To force inetd to re-read /etc/inetd.conf, send it a HUP signal.
  • E.g., if its PID is 357, kill -HUP 357.
  • This does not stop launched daemons.
  • It does apply changes to new invocations.

Third way of starting a daemon: cron

• some things must be done at certain times.
• some things must be done hourly, weekly, etc.
• Answer: the cron daemon handles repetitive scheduling.
• Configuration files: /var/spool/cron/crontabs/*
• conbrio:/var/spool/cron/crontabs/root:

   10 3 * * 0   /usr/lib/newsyslog
   |  | | |     ^^^^^^^^^^^^^^^^^^ command to run 
   |  | | | ^ day of week: sun=0, sat=6
   |  | | * month number; *=all
   |  | ^ day of month: *=all
   |  ^ hour
   ^^ minute
  

  This says to run the script newsyslog at 3:10 am every Sunday morning (the script copies /var/adm/messages to /var/adm/messages.1 and starts over with a blank messages file).
• day indicators are or'd together, and commas separate options.

   0 0 1,10 * 1,2
  

  means to do something at midnight on the 1st and 10th of each month, as well as every monday and tuesday.

User interfaces to cron

• Not just for system work.
• crontab: allows editing of your personal crontab.
  • crontab -e root: edit root's crontab.
    • setenv EDITOR emacs first or you'll end up in vi.
  • handles HUP'ing cron for you!
• at: allows scheduling a job in the future of now.

   andante% at 1am next week <stuff_to_do
  

  where stuff_to_do consists of a list of the commands you want to execute. This allows you to do things when the system is less busy (and you're asleep).
  • at -l: describes the jobs you're waiting to do.
  • at -r <jobid>: removes a job from the queue.

The point

• There is no one source for daemons and inexplicable events.
• To diagnose a problem, you must know about all sources.
• This is the real difficulty of system administration.