Swiss Army Chainsaws

• Perl is often referred to as the Swiss Army Chainsaw.
• But that description applies to many of the UNIX commands upon which Perl is based.
• Lots of commands available; lots of power; lots of options.
• Some single commands encompass entire programming languages.

UNIX Commands

• Typed as first field of a shell command.
• All else are arguments.
• Example: ps -ef
  • Command: ps
  • Arguments: -ef
  • Documentation: man ps

Informal UNIX Command Conventions

• Most UNIX commands follow the same common set of conventions.
• Conformance is optional: a command can interpret your arguments any way it wishes.
• Options to commands start with -.
  • ls -l
• - by itself means use standard input or output as a file.
  • tar -cf - sends output to standard output.
• Startup files for a command are stored in ~/.namerc where name is the name of the command.
  • .cshrc - startup for csh and tcsh
  • .mailrc - startup commands for mail and mailx.
  • .pinerc _ startup commands for pine.
  • .profile - startup for sh.
  • .emacs - non-conformist.

From whence come arguments

• After shell processing, the result of shell substitutions is provided to the command that you execute.
• If the command is written in C, there are two special arrays argv and envp that contain the arguments (as strings). You can grab them and use them in your program!
• Here's a C program that prints all its arguments and everything that it knows about from its execution environment:

  contents of chainsaw/args.c...
  /* simple C program does nothing but print its arguments and environment */ 
  #include <stdio.h> 
  int main (int argc, char **argv, char **envp) { 
  /*                               ^^^^^^^^^^^ array of environment strings
                      ^^^^^^^^^^^ array of argc strings
            ^^^^^^^^  argument count 
   */ 
    int i; 
    printf("I have %d arguments\n",argc); 
    for (i=0; i<argc; i++) /* there are argc of these */ 
      printf("argv[%d] is '%s'\n", i, argv[i]); 
    printf("Here's my environment:\n"); 
    for (i=0; envp[i]!=0; i++) { /* last string is NULL (0) */ 
      printf("envp[%d] is '%s'\n",i,envp[i]); 
    } 
  } 
  ...end of chainsaw/args.c
  

• If one compiles this with:

   
  gcc -g args.c -o args
  

  to create a command args, and then invokes the command with

   
  args foo bar cat
  

  then the result is:

  contents of chainsaw/args.out...
  I have 4 arguments
  argv[0] is 'args'
  argv[1] is 'foo'
  argv[2] is 'bar'
  argv[3] is 'cat'
  Here's my environment:
  envp[0] is 'MANPATH=/local/sol/5.7/man:/local/man:/usr/share/man:/usr/openwin/share/man'
  envp[1] is 'DTDATABASESEARCHPATH=/u/f/couch/.dt/types,/etc/dt/appconfig/types/%L,/etc/dt/appconfig/types/C,/usr/dt/appconfig/types/%L,/usr/dt/appconfig/types/C'
  envp[2] is 'DTXSERVERLOCATION=local'
  envp[3] is 'LANG=C'
  envp[4] is 'HELPPATH=/usr/openwin/lib/help'
  envp[5] is 'PATH=.:/u/f/couch/bin:/u/f/couch:/local/sol/5.7/bin:/local/bin:/local/games:/usr/openwin/bin:/usr/bin:/usr/ucb:/usr/ccs/bin:/local/hosts'
  envp[6] is 'AB_CARDCATALOG=/usr/dt/share/answerbooks/C/ab_cardcatalog:/usr/dt/share/answerbooks/C/ab_cardcatalog'
  envp[7] is 'DTUSERSESSION=couch-blackhole-0'
  envp[8] is 'XMICONBMSEARCHPATH=/u/f/couch/.dt/icons/%B%M.bm:/u/f/couch/.dt/icons/%B%M.pm:/u/f/couch/.dt/icons/%B:/etc/dt/appconfig/icons/%L/%B%M.bm:/etc/dt/appconfig/icons/%L/%B%M.pm:/etc/dt/appconfig/icons/%L/%B:/etc/dt/appconfig/icons/C/%B%M.bm:/etc/dt/appconfig/icons/C/%B%M.pm:/etc/dt/appconfig/icons/C/%B:/usr/dt/appconfig/icons/%L/%B%M.bm:/usr/dt/appconfig/icons/%L/%B%M.pm:/usr/dt/appconfig/icons/%L/%B:/usr/dt/appconfig/icons/C/%B%M.bm:/usr/dt/appconfig/icons/C/%B%M.pm:/usr/dt/appconfig/icons/C/%B'
  envp[9] is 'SESSION_SVR=blackhole'
  envp[10] is 'OPENWINHOME=/usr/openwin'
  envp[11] is 'EDITOR=vi'
  envp[12] is 'LOGNAME=couch'
  envp[13] is 'DTSCREENSAVERLIST=StartDtscreenSwarm StartDtscreenQix     StartDtscreenFlame StartDtscreenHop StartDtscreenImage StartDtscreenLife     StartDtscreenRotor StartDtscreenPyro StartDtscreenWorm StartDtscreenBlank'
  envp[14] is 'MAIL=/var/mail/couch'
  envp[15] is 'USER=couch'
  envp[16] is 'DISPLAY=:0.0'
  envp[17] is 'SHELL=/usr/bin/tcsh'
  envp[18] is 'DTAPPSEARCHPATH=/u/f/couch/.dt/appmanager:/etc/dt/appconfig/appmanager/%L:/etc/dt/appconfig/appmanager/C:/usr/dt/appconfig/appmanager/%L:/usr/dt/appconfig/appmanager/C'
  envp[19] is 'HOME=/u/f/couch'
  envp[20] is 'XFILESEARCHPATH=/usr/openwin/lib/locale/%L/%T/%N%S:/usr/openwin/lib/%T/%N%S:/usr/openwin/lib/locale/%L/%T/%N%S:/usr/openwin/lib/%T/%N%S'
  envp[21] is 'XMICONSEARCHPATH=/u/f/couch/.dt/icons/%B%M.pm:/u/f/couch/.dt/icons/%B%M.bm:/u/f/couch/.dt/icons/%B:/etc/dt/appconfig/icons/%L/%B%M.pm:/etc/dt/appconfig/icons/%L/%B%M.bm:/etc/dt/appconfig/icons/%L/%B:/etc/dt/appconfig/icons/C/%B%M.pm:/etc/dt/appconfig/icons/C/%B%M.bm:/etc/dt/appconfig/icons/C/%B:/usr/dt/appconfig/icons/%L/%B%M.pm:/usr/dt/appconfig/icons/%L/%B%M.bm:/usr/dt/appconfig/icons/%L/%B:/usr/dt/appconfig/icons/C/%B%M.pm:/usr/dt/appconfig/icons/C/%B%M.bm:/usr/dt/appconfig/icons/C/%B'
  envp[22] is 'TERM=dtterm'
  envp[23] is 'dtstart_sessionlogfile=/dev/null'
  envp[24] is 'TZ=US/Eastern'
  envp[25] is 'DTHELPSEARCHPATH=/u/f/couch/.dt/help/couch-blackhole-0/%H:/u/f/couch/.dt/help/couch-blackhole-0/%H.sdl:/u/f/couch/.dt/help/couch-blackhole-0/%H.hv:/u/f/couch/.dt/help/%H:/u/f/couch/.dt/help/%H.sdl:/u/f/couch/.dt/help/%H.hv:/usr/dt/appconfig/help/%L/%H:/usr/dt/appconfig/help/%L/%H.sdl:/usr/dt/appconfig/help/%L/%H.hv:/usr/dt/appconfig/help/C/%H:/usr/dt/appconfig/help/C/%H.sdl:/usr/dt/appconfig/help/C/%H.hv'
  envp[26] is 'XMBINDDIR=/usr/dt/lib/bindings'
  envp[27] is 'SHLVL=2'
  envp[28] is 'HOST=blackhole'
  envp[29] is 'HOSTTYPE=sun4'
  envp[30] is 'CLASSES='
  envp[31] is 'LD_LIBRARY_PATH=/local/sol/5.7/lib:/usr/openwin/lib:/usr/dt/lib:/loc/X11/lib:/local/lib:/usr/ccs/lib'
  envp[32] is 'FONTPATH=/usr/openwin/lib/fonts'
  envp[33] is 'XAPPLRESDIR=/local/X11/lib/X11/app-defaults'
  envp[34] is 'MOZILLA_HOME=/loc/xclients/netscape-v461'
  envp[35] is 'MORE=-c'
  envp[36] is 'WINDOWID=79691829'
  envp[37] is 'TERMINAL_EMULATOR=dtterm'
  envp[38] is 'PWD=/g/150NET/public_html/n/chainsaw'
  envp[39] is 'PGLIB=/loc/postgres/lib'
  envp[40] is 'PGDATA=/loc/postgres/data'
  envp[41] is 'SEEPLEXPATH=/u/f/couch/current/seeplex/environ:.:../Defaults'
  envp[42] is 'CLASS=15'
  ...end of chainsaw/args.out
  

• Here's a csh script that tells you about its arguments.

  contents of chainsaw/args.csh...
  #! /bin/csh 
  # demonstration of argument variables inside scripts
  echo I was invoked by the command \'$0\'
  echo All my arguments are \'$argv[*]\'
  echo My first argument is \'$1\' or \'$argv[1]\'
  echo My second argument is \'$2\' or \'$argv[2]\'
  set i = 1
  while ($i <= $#argv) 
     echo argument $i is \'$argv[$i]\'
     @ i = $i + 1
  end
  ...end of chainsaw/args.csh
  

• If we

  chmod +x args.csh
  

  and then execute

   
  args.csh foo bar cat
  

  then the result is

  contents of chainsaw/args.csh.out...
  I was invoked by the command 'args.csh'
  All my arguments are 'foo bar cat'
  My first argument is 'foo' or 'foo'
  My second argument is 'bar' or 'bar'
  argument 1 is 'foo'
  argument 2 is 'bar'
  argument 3 is 'cat'
  ...end of chainsaw/args.csh.out
  

• Inside a shell script:
  • $0 is the command name for the script.
  • $argv[*] (or $*) is an array of all arguments.
  • $argv[1] (or $1) is the first argument.
  • $argv[2] (or $2) is the first argument.
  • The number of arguments is $#argv.

Some "power tool" commands every administrator must understand

• find: search directories for files matching particular criteria.
• awk: a language for processing tables.
• sed: the stream editor: a text editor that works on pipes.
• Each of these commands defines its own programming language.
• One can write a program for each one within its arguments.

Find

• Purpose: find files matching specific criteria.
• Criteria can be anything except content.
• man find for details.

Using find

• Example:

   
  find . -name foo -print 
  

• find: the command name.
• .: the directory in which to start looking for files matching criteria. . means your current directory.
• -name foo: the name of the file should be foo.
• -print: what to do with that file: print its name.

Interpreting find commands

• Find checks every file or directory contained in the whole hierarchy starting at your current directory.
• For each one, it evaluates its argument list from left to right.
• If a test fails, find skips the current file or directory.
• Otherwise it processes the command line from left to right all the way to the end.
• Thus directives on the line are separated by implicit logical and's.

  find /bin -type f -owner root -ls
  

  says the following:
  • For every file and directory in the hierarchy under /bin
  • If it is a normal file (-type f)
  • and it's owned by the superuser (-owner root)
  • then ls it (-ls).

Shell games

• You can use parentheses and regular expressions inside find.
• But these have another meaning in the shell.
• To get find to receive them, one has to stop the shell from interpreting them.
• Compare

   
  find /bin -name *b -print
  

  and

   
  find /bin -name \*b -print
  

• The first one replaces *b with a list of every file in the current directory ending with b, before calling find.
• The second form passes the *b to find itself. The find command checks every file in the whole hierarchy against this pattern.
• These behaviors are very different.

Problem of the day #1

• Suppose /bin contains the hierarchy of files:

   
  ./bar
  ./foo
  ./joe/dab       
  ./joe/nab 
  

  and that your current directory contains the hierarchy of files:

   
  ./cat/dog
  ./fob
  ./junk
  ./hab
  

  Compare what happens when the two find commands above are executed.

Shell parens and find parens

• In the shell, parens delimit an array.
• In find, parens group sections of a program.

   
  find /usr \( -name cat -o -name dog \) -ls
  

  finds all files or directories in /usr that are named either cat or dog.
  • -o: "or", allows more than one possibility.
  • \( .. \): the scope of the possibilities.
• Watch out: scope is by default, the whole line:

  find /usr -name cat -o -name dog -ls
  

  means

  find /usr \( -name cat -o -name dog -ls \)
  

  which does something truely weird.
• All files named cat are found but not printed.
• All files named dog are listed with ls!

Problem of the day #2

• What happens when one executes

   
  find /usr (-name cat -o -name dog) -ls
  

  without escaping ()?

Find hacks

• executing custom tests within find
• using find to scan content of files.

Exit codes

• Every command you issue returns an exit code between 0 and 255.
• 0 means it worked.

   
  % ls 
  % echo $status
  0
  % # means everything was OK
  

• nonzero means something went wrong.

   
  % grep foo file
  % echo $status
  1
  % # means that the word foo wasn't found in file
  

• Every command issues exit codes according to its own wishes. See the man pages for what they mean.
• For grep, 0 means it worked, 1 means "no match".

Test/Command duality

• In find, each step can be either a test or an action (or both).
• -ls, -print always succeed.
• If you utilize -exec to run a command, it succeeds if the exit code of the command is 0.

Stupid find trick

• List every file containing the word foo
• Consider the code:

   
  find . -type f -exec grep -l foo {} \; -print 
  

• -type f makes sure we have a file, not a directory!
• -exec grep -l foo {} \; executes an embedded command within the find.
• -exec starts an embedded command.
• \; ends an embedded command. ; without \ would end the whole find command.
• Within the command, {} represents the pathname to be tested. If that file was ./cat/dog the command would be

  grep -l foo ./cat/dog

• If the command succeeds, processing continues.
• If the command fails, processing stops.

awk

• Purpose: scanning and grabbing data from tables.
• A table is any text containing easily recognised delimiters.
• E.g., an ls -l output contains fields separated by spaces.
• Allows one to easily scan tables for patterns and summarize them.

Basic awk syntax

• two parts to an awk program: patterns and actions.
• Patterns outside {} describe criteria that determine whether a a line of input will be processed.
• Inside {} actions say what to do (when a line matches).
• $1,$2,$3,... are fields in the line.
• Can also make up other variable names as needed (no preceeding $ needed).

Simple awk examples

•  
  awk '$2 == "couch" { print }' <joe
  

  print every line of the file joe in which couch is the second word in the line. Note use of shell quote '' to avoid shell evaluation of the awk program!

•  
  ps -ef | awk '{ print $2 }' 
  

  Print the process identifier (PID : field 2) of every executing process.

•  
  ps -ef | awk '$1 == "root" { print $2 }' 
  

  Print the process number of all processes owned by the superuser (root).

Pattern matching

• Awk allows one to match patterns against fields of files.
• Awk's regular expression patterns are different than those used for matching against filenames.
• (but all other languages except shells use variants of awk's syntax).

Awk Regular expressions

• Contained in /../.
• As in shell regular expressions:
  • [abc]: one of the characters a, b, or c
  • [^abc]: any character that's not a, b, or c
  • [a-z]: any lowercase character.
• Different from shell regular expressions:
  • .: one character of any kind, except newline. (similar to ? in the shell)
  • \(ab|bz\): either 'ab' or 'bz' (similar to {ab,bz} in the shell)
• New way to specify repeats:
  • X?: 0 or 1 appearances of X.
  • X*: 0 or more appearances of X.
  • X+: 1 or more appearances of X.
  • In the above, X can be replaced by a bracket expression, a parenthetic expression, etc. Each of ?, * operates on the single thing immediately preceding it.
• New patterns with no analog so far:
  • ^ represents the beginning of a line.
  • $ represents the end of a line.
  • \t: matches a tab character.
  • \n: matches a newline character.
• Any special characters in the above are accessible by placing a \ in front.
  • \[ and \] represent the literal characters open and close bracket (instead of the pattern operators).
  • \* represents a literal asterisk.
  • \. represents a literal period.
  • \/ represents a literal / (which would otherwise end the pattern).

Simple Regular Expression Examples

• Main difference is that awk patterns need not represent the complete pattern unless you want them to.
• /./: matches any text at all (not containing a \n).
• /a/: matches any text containing an a anywhere within it.
• /b$/: matches any text ending in b.
• /^c/: matches any text beginning with c.
• /[^x]/: matches any text containing any character that isn't x (including, e.g., xy).

Why all this trouble?

• RE's provide a very efficient way of specifying patterns in which you are interested.
• Lots less trouble than listing all alternatives.

Using RE's in awk

• You have forgotten your friend's username but you know it starts with a c and that she's logged on now.

   
  users | awk '$1 ~ /^c/ { print }' 
  

  will print all people satisfying those criteria!

Problem of the day #3

• If ls -l prints

   
  drwx------   2 couch    student      512 Jun 20  2000 Archives
  drwxr-xr-x   2 couch    student      512 May 17  2000 Fall99
  drwxr-xr-x   2 couch    student      512 May 17  2000 NEWnotes
  -rwxr-xr-x   1 couch    student     2355 Jan 20  1999 NOTES
  drwxr-xr-x   2 couch    faculty     2048 Sep  7 14:14 OldHtml
  -rwxr-xr-x   1 couch    student      726 Jan 19  1999 SCHEDULE
  -rwxr-xr-x   1 couch    student      657 Jan 19  1999 TODO
  

  then what does this command do?

   
  ls -l  | awk '$1 ~ /^-/ && $5 > 100 { print $9 "(" $6 ")" }'
  

Tip of the iceberg

• So far we've done really simple things with awk.
• In actuality it's a full programming language with variables and arrays.
• Most difficult thing to get used to is that in awk, arrays are indexed by strings.

Silly but powerful example

• Compile a summary of all commands currently running.
• Step 1: figure out a command that'll list all commands.

   
  % ps -e
     PID TTY      TIME CMD
       0 ?        0:01 sched
       1 ?        0:47 init
       2 ?        0:37 pageout
       3 ?       52:56 fsflush
     203 ?        1:10 utmpd
     101 ?        0:04 keyserv
     295 ?        0:00 sac
      99 ?        0:07 rpcbind
  

• Step 2: note that we don't want the header (line 1)

   
  ps -e | awk 'NR != 1 {print}'
  

• Step 3: accumulate number of times each command is running.

   
  ps -e | \
  awk 'NR != 1 { if (run[$4]=="") cmd[++len]=$4; run[$4]++ }'
  

  Really subtle things going on here!
  • cmd[1] to cmd[len] are a list of commands seen so far.
  • run["foo"] is the number of times the command "foo" is running.
  • if (run[$4]=="") Check if this command has been encountered before. If not, we won't have a record of it.
  • cmd[++len]=$4 make a list of ones you haven't seen. len is how many things you have in it.
  • run[$4]++ keep track of how many times this command is running. This array is indexed by command name.

Why does this work at all?

• In awk, things are arranged so that the simplest depiction of a program usually works.
• Variables spring into existence when named.
• The default value of a variable is "".
• The numeric value of "" is 0.
• so if len is undefined, ++len sets len to 1.
• Arrays are indexed by strings, not numbers.

Last step: print out summary

• Special pattern BEGIN: specifies actions to occur before reading input.
• Special pattern END: specifies actions to occur after reading all input.
• We need to do something after reading everything:

   
  ps -e | 
  awk 'NR != 1 { if (run[$4]=="") cmd[++len]=$4; run[$4]++ } 
       END     { for (i=1; i<=len; i++) print run[cmd[i]] "\t" cmd[i] }'
  

  (indented here for readability).
• Here's how this works:
  • If len is the number of commands seen,
  • and cmd contains those commands,
  • then cmd[i] is a command name (1<=i<=len)
  • and if run["foo"] is the number of times that foo is running then:
  • cmd[i] is running run[cmd[i]] times!

Storing awk scripts in files

• from this example, you can see that awk scripts can get long.
• If you store the script you want into a file:

  contents of chainsaw/runner.awk...
  BEGIN   { len=0; } 
  NR != 1 { if (run[$4]=="") cmd[len++]=$4; 
            run[$4]++; 
          }
  END     { for (i=0; i<len; i++) 
              print run[cmd[i]] "\t" cmd[i]; 
          }
  ...end of chainsaw/runner.awk
  

  and you invoke awk by giving the name of the file:

   
  awk -f runner.awk
  

  then all will work as before.
• You don't have to worry about shell/awk conflicts this way!
• Some people prefer this version as it initializes variables in BEGIN>

sed

• awk works on structured input (such as tables and reports).
• It isn't very good at doing things with unstructured input (such as documents and printouts).
• Special power tool for non-interactive editing of documents: sed

sed concepts

• Line-oriented: reads one line at a time.
• Can print, edit, or omit each line.
• A sed program, like an awk program, is a list of pattern/action pairs.
• The first pattern match to the line determines what happens.
• Sed syntax is so complex that we'll start by putting the sed command streams into files (-f option just as in awk)

Simple sed examples

• delete the 5th line of the input; leave all else

   
  sed -e '5d'
  

• Add a line 'kilroy was here' after the 3rd line.

  contents of chainsaw/append.sed...
  3a\
  kilroy was here.
  ...end of chainsaw/append.sed
  

  Call this with

  sed -f append.sed
  

• Add a disclaimer line before any line containing couch:

  contents of chainsaw/insert.sed...
  /couch/i\
  (All opinions expressed herein are the sole responsibility of the author!)
  ...end of chainsaw/insert.sed
  

• /couch/ is a pattern match. This one selects all lines containing couch.

Search and replace

• Most powerful feature of sed: pattern search and replacement.
• Can use this to change variable names in programs, or even transform one programming language into another.
• s/pattern/replacement/: replace one pattern with another.
• Example: replace the word couch with the Grinch everywhere it appears in the input:

   
  sed -e 's/\<couch\>/the Grinch/'
  

• Special symbols \<, \> match only word boundaries.

Practical use of sed: reformat input for awk processing

• It'd be nice if awk could always do what we want.
• However, output isn't always formatted so that awk can handle it.
• Unless everything always falls in the same place in fields, awk can't make much sense of it.
• But we can write a sed command to make everything line up correctly so awk will work.

Example: 'fixing' ps -ef timestamps

• Standard listing for ps -ef isn't perfectly columnar.
• There are inconsistencies in which column falls where:

   
     $1    $2    $3     $4  $5  $6 $7       $8   $9
     couch 11313 11292  0   Jan 24 pts/4    0:12 /usr/bin/tcsh
     $1    $2    $3     $4  $5     $6       $7   $8
     couch 13721 13642  0 12:12:04 pts/3    9:46 netscape http://www.eecs.tufts.edu/g/150NET/n
  

• Two ways of handling this: split up times or join dates.
• Let's convert Jan 24 to Jan:24.

   
  ps -ef | sed -e 's/Jan /Jan:/'
  

The story you have just seen is true...

• Names have been changed simply to confuse you
• Names of commands are variant and change from UNIX to UNIX.
• Principles are invariant.

Invariants of User Environment

• The shell (syntax varies with the flavor).
• Inherited variables ("Environment Variables").
• Shell variables (not inherited).
• Programming (scripting) features

Variants and other nuisances

• options to commands.
  • ps -aux: BSD
  • ps -efl: SysV
  • ps aux: linux
• paths to commands, manual pages, etc.
  • /usr/local/bin: locally installed commands for most people.
  • /local/bin: our old convention inside EECS.

Some commands to know