# NAME

Babble - an automation tool for crafting interactive scripts.

# DESCRIPTION

Babble is a language for creating very complex interactive scripts to be used in managing a network of standalone appliance-style devices. Unlike 'expect' and its relatives, a Babble script does not contain any code from another language such as tcl or perl (unless desired). A Babble 'script' is instead documentation of the structure of an interactive conversation, described using an XML-like syntax.

The Babble interpreter utilizes this documentation to interact with devices, thus avoiding traditional scripting entirely. A powerful variant mechanism allows one to reuse this documentation for accomplishing several related effects, while a type-checking system avoids common documentation errors. Although Babble was originally intended to interact with external devices, it can also be used to document and replay shell interactions, thus automating many common tasks without scripting. In this way we also avoid documenting the scripts, and avoid all the maintenance problems inherent for hastily-written code written in rapid prototyping languages.

## Documentation as script

One advantage of using documentation rather than scripting is that one can craft the documentation entirely by annotating a stored script of user-machine interactions. One captures this 'script' while manually performing the operation desired, and then annotates the stored script so that Babble knows exactly how to repeat it. This only requires telling Babble things that are not intuitively obvious from the script, e.g., which output was typed by the user and which by the machine. The creator of Babble documentation need not know how to program; just how to accomplish desired ends with appropriate user commands.

## Avoiding Scripting

While writing a script to automate a complex process is empowering for the administrator, custom scripts create maintenance problems. These scripts are typically poorly documented, and thus only usable by the creator. This renders the creator indispensible, while rendering the overall organization in which the creator works unstable.

Crafting documentation is different from crafting scripts in several important ways. First, one need not abstract behavior from examples, but can instead directly annotate examples to create things that work like scripts. This means in particular that the creator of Babble documentation need not know how to program in the traditional sense, though it is necessary to know how to manually perform all the actions you would like to accomplish.

• There is no need to document a 'babble script'; its function is self-evident from the fact that it documents a process.

• Common mistakes can be caught in Babble scripts by several automatic methods. Documentation is subject to validation in ways that a normal script is not.

• Changes in equipment are caught instantly by Babble's error catching mechanism and one need not write excruciatingly complex error handlers in custom scripts.

This method of achieving automation is non-intuitive at first because we must stop writing commands that accomplish actions and instead document the effects of commands we could type. Normal scripting requires imperative thinking, while Babble requires declarative thinking. If we take the extra effort, however, Babble then takes responsibility for many quality checks that we would have to craft ourselves when writing custom scripts.

# INSTALLATION

## Limitations

This file is part of Babble. All files in the distribution of Babble are Copyright 2000 by

    Alva L.  Couch,
Associate Professor of Computer Science,
Tufts University, Medford MA 02155
email: couch@eecs.tufts.edu
phone: 617-627-3674


Redistribution and use are permitted provided that this entire
copyright notice is duplicated in all such copies, and that any
documentation, announcements, and other materials related to such
distribution and use acknowledge that the software was developed by
Alva L.  Couch at Tufts University in Medford, Massachusetts, USA.
No charge, other than an "at-cost" distribution fee, may be charged
for copies, derivations, or distributions of this material without
the express written consent of the copyright holder.  Neither the
name of the University nor the names of the authors may be used to
endorse or promote products derived from this material without
specific prior written permission.  THIS SOFTWARE IS PROVIDED AS
IS'' AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING,
WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR ANY PARTICULAR PURPOSE.
=head2 Requirements

Babble-0.1Alpha is written in perl-5.006 and works on virtually all UN*X-compatible systems supporting Perl. While it should in principle run on any Perl >= 5.000, some of the code for embedded Perl exposes bugs in Perl 5.005_02. causing core dumps at the end of execution of perl segments. For best results utilize Perl 5.006 or later.

Babble-0.1Alpha has been tested only on RedHat Linux and Solaris 2.7 systems. It should execute as documented herein on any UNIX system where shell execution semantics is similar to these. But there are no guarantees.

## Required Perl Modules

Babble requires three Perl modules that you can load from the Comprehensive Perl Archive Network (CPAN).

Data::Dumper
This allows Babble to dump its state during error messages.

XML::Parser
This allows Babble to parse XML specifications of interactive user sessions.

IO::Tty
This allows Babble to simulate typing and fool most applications into thinking that they are interacting with a user when in fact they're interacting with Babble. Even shells can be scripted although they generally complain about not being able to call enough Ioctl's!

## Installation steps

Babble installation is very simple because the Babble interpreter is packaged as one very large Perl script.

• Pick an instance of Perl in which to run Babble.

• For this instance, make sure required Perl modules Data::Dumper, XML::Parser, and IO::Tty are installed. You can most easily do this from CPAN. See your Perl documentation for details on how to add modules. For most modern perls, you can do this via:
        root# perl -MCPAN -e shell
cpan> install XML::Parser
cpan> install IO::Tty
cpan> install Data::Dumper

If XML::Parser complains that Expat is not yet installed, fetch it from the source that the CPAN script recommends and compile it before continuing.

• Edit the first line of the included babble script to point to the Perl interpreter that you upgraded in the last step. E.g., if you used /usr/bin/perl, make the first line of babble
      #! /usr/bin/perl

• Copy the included babble script somewhere permanent and remember the location. A good location would be /usr/local/bin/babble.

• When creating Babble documentation, edit the first line of each Babble document so that it contains:
      #! <location of perl> <location of babble>

If your perl is /usr/bin/perl, and babble is located in /usr/local/bin/babble, edit each script so that the first line is:

      #! /usr/bin/perl /usr/local/bin/babble

# TUTORIAL

The next step in using Babble is to create some documentation that it can utilize to perform a task. This documentation replaces a traditional script. Simple documentation involves transforming a script of user interactions so that they instead document the interactions. This documentation process requires embedding markups that describe the function of each component of the script in a way that allows it to be reused.

The simplest way to document script elements is by labelling them with <get> and <put> tags. Content between <get> and </get> tags is something we expect from the device with which we are conversing. Content between <put> and </put> tags is something we type in accomplishing the task.

For example, suppose we want to add a user to a LightWave ConsoleServer 3200. The dialog we must undertake looks like this:

        sys admin> adduser
Number of available user records: 196
Number of users defined: 4
Enter user id | USER ID > foo
0-17 | MAX CONCURRENT LOGINS: 1>1
Allowed devices example: 1-5,10 | DEVICES 0 > 1-11
Allowed listen devices example: 1-5,10 | DEVICES 0 > 0
Allow user to clear device buffer (Y/N) | YES > Y
Clear screen after a command (Y/N) | YES > Y
        sys admin>

The first step is to translate this script into Babble's language. This requires replacing whitespace and XML special characters with new equivalents:

        sys\sadmin&gt;\sadduser\n
Number\sof\savailable\suser\srecords:\s196\n
Number\sof\susers\sdefined:\s4\n
Enter\suser\sid\s|\sUSER\sID\s&gt;\sfoo\n
Allowed\sdevices\sexample:\s1-5,10\s|\sDEVICES\s0\s&gt;\s1-11\n
Allowed\slisten\sdevices\sexample:\s1-5,10\s|\sDEVICES\s0\s&gt;\s0\n
Allow\suser\sto\sclear\sdevice\sbuffer\s(Y/N)\s|\sYES\s&gt;\sY\n
Clear\sscreen\safter\sa\scommand\s(Y/N)\s|\sYES\s&gt;\sY\n
\n
sys\sadmin&gt;\s\n

In each Babble script, literal spaces, tabs, and linefeeds are ignored. The following substitutions are utilized to represent literal characters in the input and output streams:

\s
represents a literal space.

\t
represents a literal tab.

\r
represents a literal carriage return.

\n
represents a literal linefeed.

\003
represents the character with ASCII code 003 (octal), i.e., Control-C. In general, \ followed by three digits represents a character by code.

In addition, Babble scripts are a variant of the Extensible Markup Language (XML). This means that one must also utilize special codes for special XML markup characters:

&lt;
means <

&gt;
means >

&amp;
means &

&quot;
means 

The next step is to annotate things typed by and read by you. I put things I type into the put manpage brackets, while things the machine types are put into the get manpage brackets. The following example documents this script, showing what Babble should see and what Babble should type:

        <get>sys\sadmin&gt;\s</get><put>adduser</put>\n
Number\sof\savailable\suser\srecords:\s196\n
Number\sof\susers\sdefined:\s4\n
<get>Enter\suser\sid\s|\sUSER\sID\s&gt;\s</get>
<put>foo<noecho>\n</noecho></put>
<put>bar<noecho>\n</noecho></put>
<put>bar<noecho>\n</noecho></put>
<put>1<noecho>\n</noecho></put>
<get>Allowed\sdevices\sexample:\s1-5,10\s|\sDEVICES\s0\s&gt;\s</get>
<put>1-11<noecho>\n</noecho></put>
<get>Allowed\slisten\sdevices\sexample:\s1-5,10\s|\sDEVICES\s0\s&gt;\s</get>
<put>1-11<noecho>\n</noecho></put>
<get>Allow\suser\sto\sclear\sdevice\sbuffer\s(Y/N)\s|\sYES\s&gt;\s<get>
<put>Y<noecho>\n</noecho></put>
<get>Clear\sscreen\safter\sa\scommand\s(Y/N)\s|\sYES\s&gt;\s</get>
<put>Y<noecho>\n</noecho></put>

To actually execute this script, we have to put it into an execution context, giving it a name and a device to which to talk. We do this by surrounding the script with the directives

        #! /usr/bin/perl /usr/local/bin/babble
<stream>
<brook name='main'>
<spawn command="cu -l cua/b -b 8">
.... script here ....
</spawn>
</brook>
</stream>

These lines satisfy several requirements of a complete Babble script.

• Each Babble script must be enclosed in <stream> .. </stream> directives.

• A script is made up of the brook manpages that have names. Unless you say otherwise, the main brook is what Babble executes.

• To interact with a device or other command, one must spawn it inside a <spawn> directive. The scope of the directive is until the </spawn>. Spawns nest to any depth, so one can suspend talking with one device with a new spawn, and continue after it.

• Within a <spawn> .. </spawn> range, <get> .. </get> and <put> .. </put> ranges describe what to expect and how to react.

• A <noecho> .. </noecho> range inside a <put> .. </put> range means that input may not be echoed as typed. It may be echoed some other way! It is used here to avoid looking for \n when the shell actually echoes \r\n.

A Babble script must be compliant with XML syntax (except for embedded Perl, which is handled specially). This means that within a script, markups must all have closing tags. Leaving any <get> or <put> tag unclosed comprises a syntax error.

The whole script, ready to run, looks like this:

        #! /usr/bin/perl /usr/local/bin/babble
<stream>
<brook name='main'>
<spawn command="cu -l cua/b -b 8">
Number\sof\savailable\suser\srecords:\s196\n
Number\sof\susers\sdefined:\s4\n
<get>Enter\suser\sid\s|\sUSER\sID\s&gt;\s</get>
<put>foo<noecho>\n</noecho></put>
<put>bar<noecho>\n</noecho></put>
<put>bar<noecho>\n</noecho></put>
<put>1<noecho>\n</noecho></put>
<get>Allowed\sdevices\sexample:\s1-5,10\s|\sDEVICES\s0\s&gt;\s</get>
<put>1-11<noecho>\n</noecho></put>
<get>Allowed\slisten\sdevices\sexample:\s1-5,10\s|\sDEVICES\s0\s&gt;\s</get>
<put>1-11<noecho>\n</noecho></put>
<get>Allow\suser\sto\sclear\sdevice\sbuffer\s(Y/N)\s|\sYES\s&gt;\s<get>
<put>Y<noecho>\n</noecho></put>
<get>Clear\sscreen\safter\sa\scommand\s(Y/N)\s|\sYES\s&gt;\s</get>
<put>Y<noecho>\n</noecho></put>
</spawn>
</brook>
</stream>

Whitespace is ignored. You may utilize this to make things print more neatly ,e.g., folding lines like:

           <get>Allowed\slisten\sdevices\sexample:\s1-5,10\s|\sDEVICES\s0\s&gt;\s</get>

anywhere within the line, say, at

           <get>Allowed\slisten\sdevices\sexample:
\s1-5,10\s|\s DEVICES\s0\s&gt;\s
</get>

To run this script, make this file executable, and simply type it as a command. Babble will take over and execute the interactions for you.

## Command line options

If you want to see what Babble does, the command that you just created takes command line options that Babble interprets to change its behavior. If the command you created above is 'uadd', then

• uadd --trace=3 turns on all tracing so that you can watch Babble work. Tracing tells you what babble is doing at each point in its execution.

• uadd --tracefile=/tmp/trace redirects the trace to a file rather than the screen so you can edit it later.

• uadd --rawout --rawoutfile=/tmp/raw

creates a file containing raw output from the device, as received by Babble.

• uadd --xmlout --xmloutfile=/tmp/xml
creates an output file annotated in XML to show the correspondence between Babble's input and device output.

## Variants

Literal scripts that simply repeat actions aren't very useful. We need to be able to generalize them so they can deal with both variations in tasks and behavior. The former requires being able to define variables, while the latter requires defining logical branches.

A variant is the Babble version of a variable. I call it something different because it behaves somewhat differently than a normal variable. There is no way to set a variant in a Babble script. Instead, one defines a configuration that determines variant values for a specific run of Babble.

Suppose that we want to annotate the <code>uadd</code> command we created above so that we can create different users. The first step is to figure out what varies in the above stream. We mark things that vary from instance to instance with var tags:

        #! /usr/bin/perl /usr/local/bin/babble
<stream>
<brook name='main'>
<spawn command="cu -l cua/b -b 8">
<get>Number\sof\savailable\suser\srecords:\s
<var>196</var>\n</get>
<get>Number\sof\susers\sdefined:\s
<var>4</var>\n</get>
<get>Enter\suser\sid\s|\sUSER\sID\s&gt;\s</get>
<put>
<var>foo</var>
<noecho>\n</noecho>
</put>
<put>
<var>bar</var>
<noecho>\n</noecho>
</put>
<put>
<var>bar</var>
<noecho>\n</noecho>
</put>
<put>
<var>1</var>
<noecho>\n</noecho>
</put>
<get>Allowed\sdevices\sexample:\s1-5,10\s|\sDEVICES\s0\s&gt;\s</get>
<put>
<var>1-11</var>
<noecho>\n</noecho>
</put>
<get>Allowed\slisten\sdevices\sexample:\s1-5,10\s|\sDEVICES\s0\s&gt;\s</get>
<put>
<var>1-11</var>
<noecho>\n</noecho>
</put>
<get>Allow\suser\sto\sclear\sdevice\sbuffer\s(Y/N)\s|\sYES\s&gt;\s<get>
<put>
<var>Y</var>
<noecho>\n</noecho>
</put>
<get>Clear\sscreen\safter\sa\scommand\s(Y/N)\s|\sYES\s&gt;\s</get>
<put>
<var>Y</var>
<noecho>\n</noecho>
</put>
</spawn>
</brook>
</stream>

Since var markups have to occur inside get or put markups, I added two get markups at the beginning. This tells Babble that it can make a sanity check by listening for these two extra lines.

This markup by itself isn't enough to instruct Babble on how to run the script. We also have to give each variant attributes that describe it so Babble can perform substitutions. Variants inside put need names; variants inside <get> need regular expression patterns that identify them:

        #! /usr/bin/perl /usr/local/bin/babble
<stream>
<brook name='main'>
<spawn command="cu -l cua/b -b 8">
<get>Number\sof\savailable\suser\srecords:\s
<var pattern='[0-9]+'>196</var>\n</get>
<get>Number\sof\susers\sdefined:\s
<var pattern='[0-9]+'>4</var>\n</get>
<get>Enter\suser\sid\s|\sUSER\sID\s&gt;\s</get>
<put>
<noecho>\n</noecho>
</put>
<put>
<noecho>\n</noecho>
</put>
<put>
<noecho>\n</noecho>
</put>
<put>
<noecho>\n</noecho>
</put>
<get>Allowed\sdevices\sexample:\s1-5,10\s|\sDEVICES\s0\s&gt;\s</get>
<put>
<var name='direct'>1-11</var>
<noecho>\n</noecho>
</put>
<get>Allowed\slisten\sdevices\sexample:\s1-5,10\s|\sDEVICES\s0\s&gt;\s</get>
<put>
<var name='listen'>1-11</var>
<noecho>\n</noecho>
</put>
<get>Allow\suser\sto\sclear\sdevice\sbuffer\s(Y/N)\s|\sYES\s&gt;\s<get>
<put>
<var name='buffer'>Y</var>
<noecho>\n</noecho>
</put>
<get>Clear\sscreen\safter\sa\scommand\s(Y/N)\s|\sYES\s&gt;\s</get>
<put>
<var name='clear'>Y</var>
<noecho>\n</noecho>
</put>
</spawn>
</brook>
</stream>

The final step is to separate values of variants from the script. We do this by creating another file uadd.cfg that contains the values we wish to use for variants:

        <config>
<var name='direct'>1-11</var>
<var name='listen'>1-11</var>
<var name='buffer'>Y</var>
<var name='clear'>Y</var>
</config>

The final step is to utilize this set of variants in the Babble invocation

        uadd --bind=uadd.cfg

If we change anything in uadd.cfg, the change will be immediately reflected in the script.

## Cases

One immediate thing we might want to do is to reuse this script for creating multiple users. We can do this by using cases. A case is a specific instance of a problem. We can feed cases to the script by renaming it and calling it from another.

Let's suppose that we rename our user creating brook as adduser. Then we can make up a couple of cases in the configuration file as follows:

        <config>
<case name='c1'>
<var name='direct'>1-11</var>
<var name='listen'>1-11</var>
<var name='buffer'>Y</var>
<var name='clear'>Y</var>
</case>
<case name='c2'>
<var name='direct'>1-11</var>
<var name='listen'>1-11</var>
<var name='buffer'>Y</var>
<var name='clear'>Y</var>
</case>
</config>

Then we can call each case in a new main brook as follows:

        <brook name='main'>
<case instance='c1'>
</case>
<case instance='c2'>
</case>
</brook>

The insert clause simulates typing the code for the uadd brook twice. We again invoke this with

        uadd --bind=uadd.cfg

For each case, variants are bound as listed in the configuration file, and the brook uadd is invoked with those bindings in place. The result is to add two users.

## Repeats

What if we want to create multiple users? It gets tedious to have to name all the instances. We could create a uadd.cfg for each one, or a the case manpage for each one, but it'd be better to be able to repeat the whole script for a set of people. We can do this by using repeat markups in both script and configuration.

For example, suppose we want to create the users foo and cat. We can make these the instance manpages of a the repeat manpage clause in the configuration as follows:


<config>
<repeat name='people'>
<instance>
<var name='direct'>1-11</var>
<var name='listen'>1-11</var>
<var name='buffer'>Y</var>
<var name='clear'>Y</var>
</instance>
<instance>
<var name='direct'>1-11</var>
<var name='listen'>1-11</var>
<var name='buffer'>Y</var>
<var name='clear'>Y</var>
</instance>
</repeat>
</config>

In the code, if the name of our user adding brook is uadd, we add a main brook:

        <brook name='main'>
<repeat instances='people'>
</repeat>
</bind>
</brook>
</brook>

The the bind manpage markup binds the names in the given file into our namespace (temporarily), the the repeat manpage markup repeats its contents for each instance of the given the repeat manpage configuration collection (people), and the the insert manpage markup invokes our brook for each instance.

## Branches

Unfortunately, most interactive processes aren't this smooth. Sometimes things vary at runtime and one must react. To account for this, Babble allows a stream to follow the branch manpagees depending upon what was received from the device.

As an example of the use of branches, consider the task of logging in as an administrator into the LightWave 3200. The console can actually be in any state, and we need to be able to detect its state and react appropriately when login isn't required:

        <brook name="enable">
<put><noecho>\003</noecho></put>
<get>Operation\scancelled\sby\suser</get>
<get>&gt;&gt;</get>
<junction>
<branch>
</branch>
<branch>
</branch>
</junction>
</brook>

If we try to login and we're already logged in, we skip typing a password, else we do.

Branches aren't really fully compliant XML because they have an order constraint. The first tag inside any the branch manpage must be a the get manpage so we can check whether the branch should be taken. After that initial the get manpage, branches can contain any free mix of the put manpage's and the get manpage's.

Junctions are executed by collecting all the initial gets of all branches, and checking each of them against input from the device until there is a match or timeout. The first match wins. This can be a bit subtle in practice because it is the opposite of normal pattern matching behavior. In Babble the shortest pattern match always wins. If one pattern in a junction is a prefix of another, the latter will never be matched.

## Discovering variants

So far we have only used predefined variants in running a script. Babble can also discover the values of variants at runtime. The mechanism for this is to place a named the var manpage markup inside a the get manpage rather than a the put manpage.

Suppose first that we want to be able to read user data for users we have already defined. The following the brook manpage will do this:

        <brook name="readuser">
<get>
User\sId\s>\s
<var pattern="[A-Z]+">FOO</var>
<var pattern="\s+">\s</var>\r\n
</get>
<get>
Allowed\sdevices\s>\s
<var name="direct" pattern="[-0-9,]+">1-32</var>
\r\n
</get>
<get>
Allowed\slisten\sdevices\s>\s
<var name="listen" pattern="[-0-9,]+">1-32</var>
\r\n
</get>
<get>
</get>
<get>Allow\suser\sto\sclear\sdevice\sbuffer\s>\s
<var name="buffer" pattern="Y|N">N</var>
<var pattern="ES|O">0</var>\r\n
</get>
<get>Clear\sscreen\safter\sa\scommand\s>\s
<var name="clear" pattern="Y|N">Y</var>
<var pattern="ES|O">ES</var>\r\n
</get>
<get>\r\n</get>
</brook>

In this example, there are two kinds of the var manpage's inside the get manpage's. A named variant is one with a name attribute. This will cause variant space to be updated with the new value. An unnamed variant is one without a name. This allows one to skip over input that varies in a predictable way, without binding any variants to values. In this version of Babble, naming a variant always causes binding of that variant. This behavior may be alterable in future versions of Babble via a bind attribute.

## Whiles

Very often, a device one wishes to control has more than one instance of a thing defined that we must discover. In this case we may utilize a the while manpage loop to discover all of them. For example, let's discover all users by reading a listing of their names:

        <brook name="usernames">
<put>listusers\r\n</put>
<while instances="people">
<junction>
<branch>
<get>User\sId\s>\s
<var pattern="\s*"/>\r\n
</get>
</branch>
<branch>
<break/>
</branch>
<branch>
<get>More</get>
<put><noecho>\s</noecho></put>
</branch>
</junction>
</while>
</brook>

This code is the first really powerful use of Babble. The the while manpage phrase collects instances of people into a new the repeat manpage configuration variable that can be used with a future the repeat manpage form. Inside the the while manpage there are three branches. The first branch grabs a username. The second branch breaks out of the the while manpage loop if the device gives a prompt, using a the break manpage tag just as in other programming languages. The third presses 'spacebar' to continue the builtin pager. If there were no the break manpage tag, the the while manpage would have ended upon the timeout manpage after no further input lines matched the first phrase.

The real power of the while manpage is that the output of a the while manpage can be the input to a the repeat manpage. Here we take the process further by reading the attributes of each user with another script readuser:

        <brook name="readuser">
<get>
User\sId\s>\s
<var pattern="[A-Z]+">COUCH</var>
<var pattern="\s+">\s</var>\r\n
</get>
<get>
Allowed\sdevices\s>\s
<var name="direct" pattern="[-0-9,]+">1-32</var>
\r\n
</get>
<get>
Allowed\slisten\sdevices\s>\s
<var name="listen" pattern="[-0-9,]+">1-32</var>
\r\n
</get>
<get>
</get>
<get>Allow\suser\sto\sclear\sdevice\sbuffer\s>\s
<var name="buffer" pattern="Y|N">N</var>
<var pattern="ES|O">0</var>\r\n
</get>
<get>Clear\sscreen\safter\sa\scommand\s>\s
<var name="clear" pattern="Y|N">Y</var>
<var pattern="ES|O">ES</var>\r\n
</get>
<get>\r\n</get>
</brook>
        <brook name="readusers">
<repeat instances="people">
</repeat>
</brook>
        <brook name='main'>
<dump file="/tmp/users"/>
</brook>

The brook readusers uses a repeat on the discovered instances people to complete each instance with all user information. At the end of this process it writes all user info into a file /tmp/users that can be utilized as a configuration file itself. This allows one to read and archive a running configuration that can be reused in order to create a configuration from scratch.

Something really subtle is happening here. If you use a the repeat manpage after a the while manpage and then discover values within the the repeat manpage, the process of discovery is inductive. This means that after both clauses, the people instances contain information from both. This process can be repeated interminably to update the instance information in as many ways as desired.

## Convergence

The most powerful feature of the current version of Babble is convergence. This feature allows you to script automatic changes that incrementally modify an existing set of instances to be another. For example, consider the task of creating users. If there's an extra user we'd like to delete it, while if there's a user not yet created we'd like to create it. This is difficult to program just using the repeat manpage and the while manpage so there is another tag the converge manpage to handle this.

For example, suppose we have several brooks already defined that do the following things:

• usernames lists current user names.

• adduser adds a user.

• removeuser adds a user.

• modifyuser modifies a user.

• readuser reads one user's data (by username).

Then the code:

        <converge key="username" instances="people">
<remove><insert brook="removeuser"/></remove>
<modify><insert brook="modifyuser"/></modify>
</converge>

invokes a really complex process that matches user data in the configuration file with user data in the device, making minimal changes in order to do so. The process proceeds as follows.

• First Babble invokes usernames to read a list of user names from the device.

• For any users that are in the device but not in the configuration file, Babble invokes removeuser to remove them from the device.

• For any users that are in the configuration file but not the device, Babble invokes adduser to add them to the device.

• For any users that are in both the configuration file and the device, Babble invokes readuser to read their data and compare it with the configuration file.

• If desired and running configurations differ for a user, Babble calls modifyuser to modify its records.

In this way, we 'converge' to the desired configuration with minimal changes to the running configuration in the process.

## Embedding Perl code

Ideally, the Babble user won't have to write any Perl, but there are specific situations in which the ability to embed Perl code comes in handy. The Perl embedding interface allows one to operate on Babble configuration parameters directly and thus modify Babble's knowledge of the world. This comes in handy if, e.g., your configuration data is not already in a configuration file, but lives instead in some external Perl-accessible database. You can then write Perl code to import data into Babble's configuration hierarchy directly. For more information, see the entry for the the perl manpage markup in the next section.

# STREAM MARKUPS

The following markups can be used in the stream file that Babble interprets as a script. These are different from, but related to, the CONFIG MARKUPS that comprise Babble's configuration files.

## stream

The the stream manpage markup must begin and end the stream file, occurring directly after the Babble invocation comment (the first line of the file, which must start with #!). A stream should only contain the brook manpage's to be invoked. Within a the stream manpage (as well as within all other tags except the get manpage and the put manpage), text content is ignored.

## brook

The the brook manpage tag contains a sequence of instructions to be played. It must have a name but all other attributes are ignored. All the brook manpages should be immediate children of the the stream manpage tag. One can invoke a brook either directly from the command line or with the the insert manpage directive. E.g., the the brook manpage:

        <brook name='foo'>
...
</brook>

may be invoked with the argument --brook=foo or with the XML markup:

        <insert brook='foo'/>

inside another brook or its content.

## insert

The the insert manpage tag allows one to invoke one the brook manpage inside another. Its purpose is to implement a primitive 'subroutine call' for the brook manpages. At time of invocation, Babble simulates the physical presence of the the brook manpage at the insert point, as if one manually typed it in at that point. For an example see the brook manpage.

## bind

The the bind manpage tag allows one to control variant values within a stream execution by reading and utilizing a new configuration file. It has one required attribute file that denotes the name of a configuration file to be loaded. For example:

        <bind file='foo.cfg'>
...
</bind>

reads the file foo.cfg and makes its variants available inside the the bind manpage.

Configuration files nest. If one invokes more than one the bind manpage, all values are available, subject to shadowing. For example, if foo.cfg is:

        <config>
<var name='foo'>hi there</var>
</config>

and bar.cfg is

        <config>
<var name='foo'>yo</var>
<var name='bar'>bo</var>
</config>

then inside the context created by

        <bind file='foo.cfg'>
<bind file='bar.cfg'>
...
</bind>
</bind>

foo is yo and bar is bo. The outer value of foo (hi there) is shadowed (hidden) by the inner one.

Each binding creates a scope in which inductive discovery occurs. If you utilize a the get manpage and/or the while manpage to discover variant values within a binding scope, those values are lost when the binding scope closes. Closing a the bind manpage does not save discovered values; these must be saved via a the dump manpage command before closing the the bind manpage.

## spawn

The the spawn manpage markup creates an interaction scope in which device interaction occurs. It requires one attribute command which is the text of a command to execute in a pseudo-tty.

Nested the spawn manpage markups completely shadow one another so that only the innermost is active at any one time. This allows one to craft streams in which interactions roam freely between several devices or processes. For example,

        <spawn command='cu -l cua/b -b 8'>
<!-- talking to cua/b ... -->
<spawn command='cu -l cua/a -b 8'>
<!-- now talking to cua/a ... -->
</spawn>
<!-- again talking to cua/b ... -->
</spawn>

The pseudo-tty interface is imperfect; in particular shells complain about not being able to complete certain ioctl calls. A the spawn manpage connection cannot be forced to type a password to a shell, because most password routines try to invoke /dev/tty directly. Since Babble's tty interface dissociates from the user's controlling terminal, this will hang.

The solution (a.k.a. dirty hack) to interacting with real shells and typing passwords in Babble scripts is to launder' the session so that passwords are being typed to a remote shell rather than a local one, e.g., by invoking a remote ssh session and then interacting with it instead of a local one. This outsmarts the shell's builtin protection against such dirty hacks (unfortunately).

## put

The the put manpage tag instructs Babble to interact with the 'current' device by sending data to the device. This corresponds to the innermost the spawn manpage containing the the put manpage, or any device specified on the command line if there is no enveloping the spawn manpage. A the put manpage tag may itself contain only text, named the var manpage's, and the special tags the noecho manpage and the stars manpage. Specifing any other markup inside a the put manpage constitutes a markup error.

During the the put manpage all the var manpage tags are expanded into the values of the corresponding named variants. Unless otherwise documented, a the put manpage tag implies a matching the get manpage for the same data. Exceptions are noted with the the noecho manpage and the stars manpage tags, which can only appear inside a the put manpage.

## noecho

The the noecho manpage tag occurs only inside the put manpage tags and denotes that the text to be the put manpage should not be matched in full duplex. This is useful when the text being typed won't be echoed, e.g., when typing passwords. For example, in the fragment:

        <get>Username:\s</get>
<put>foo<noecho>\n</noecho></put>
<put><noecho>bar\n</noecho></put>

\n expands to \r\n so we don't match it to avoid confusing the parser. The password bar is not echoed so we place the whole password inside the the noecho manpage tag.

## stars

The the stars manpage tag occurs only inside the put manpage tags and is a very special case. Some devices echo stars ('*') for each character of a password. Placing a password inside a the stars manpage tag causes Babble to match the appropriate number of stars instead.

## get

The the get manpage tag instructs Babble to interact with the 'current' device by receiving data from the device. the get manpage can contain only text and the var manpage elements, either named or unnamed. All the var manpage's contained in the get manpage's must have a pattern attribute defined that matches appropriate values for the the var manpage. the get manpage can fail if the pattern it seeks does not occur in the the timeout manpage value currently in effect. The failure of a the get manpage outside special contexts such as the branch manpage and the while manpage causes an immediate execution halt. Inside these contexts, failure instead modifies context behavior.

If an the var manpage inside a the get manpage has a name, the value matched is bound into the current the spawn manpage context as created by the repeat manpage, the while manpage, or the case manpage subscoping. If there is no the repeat manpage, the while manpage, or the case manpage in effect at the time of a the get manpage, all bindings occur within the immediately enclosing the bind manpage. Within a the repeat manpage or the while manpage, a the get manpage binds and named the var manpage's into the current instance rather than the global spawn scope. Any the case manpage in effect likewise causes a the get manpage to bind into the case instance.

## var

The the var manpage tag indicates variational structure within both input and output of a stream. It is interpreted differently when it occurs in input than when it occurs in a output.

Inside a the put manpage tag, a the var manpage must have a name that corresponds to that of a configuration parameter in the current binding scope. The corresponding configuration parameter is substituted for the the var manpage in place, in the output.

Inside a the get manpage tag, a the var manpage must have a pattern. This is a regular expression in the Perl sense, except that the magic characters ( and ) have been disabled. Any the var manpage that also has a name will cause binding of the matched value to that name in the appropriate binding scope, which is the innermost the case manpage, the repeat manpage, the while manpage, or the bind manpage scope containing the the var manpage. In executing the regular expression, each the var manpage pattern is placed inside ( and ) in order to parse values for named the var manpage's.

the var manpage matching is different from Perl matching in one very important sense. It is opportunistic in the sense that the first substring that matches will cause a return from the the get manpage. This means that it is very important to delimit the get manpage the var manpage's with characters that indicate the end of a match. The wrong way to parse a name out of a file is this;

        <get>
<var name='joe' pattern='[a-zA-Z]'>
</get>

This will match and bind the first character it sees. One must in some way delimit the name, e.g., with a trailing space:

        <get>
<var name='joe' pattern='[a-zA-Z]'>
\s
</get>

This matches the first word with a space at the end that is not part of the word.

A (silly) current implementation restriction is that there can be at most 16 the var manpage's inside any one the get manpage. This will be fixed when I get time.


=head2 junction

A junction allows one to specify different possible execution paths within a the brook manpage. A junction can contain only the branch manpage's representing possible execution paths.

## branch

A the branch manpage represents one execution path for a junction and is only meaningful in that context. the branch manpage content must include a the get manpage as the first tag, after which the branch can be a free mix of any sequence of tags. A branch is only invoked if its the get manpage is the first match to the input from the device in context.

## repeat

A the repeat manpage allows one to iterate over a group of instances that require similar handling. It requires one parameter, instances, which should be the name of a variant that in the current binding context is a the repeat manpage variant. For example, given the configuration (binding context):

        <config>
<repeat name='cool'>
<instance>
<var name='joe'>Snoopy</var>
</instance>
<instance>
<var name='joe'>Kennedy</var>
</instance>
</repeat>
</config>

and the repeat markup

        <repeat instances='cool'>
...
</repeat>

the statements inside the repeat markup will be repeated twice, once with joe bound to Snoopy and once with joe bound to Kennedy. A the repeat manpage invokes a new binding context within which the get manpage statements bind values to the repeat context instead of any already open outer binding contexts. Within each of these repeats, any discovered variants will be filed within the appropriate instance in effect at the time of the the get manpage.

## while

The the while manpage statement allows one to inductively discover a set of instances of a thing. It can be used, e.g., to capture information on individual entities described in a listing command. It has one required attribute instances, which is the name of a set of instances to discover. Any current value of that set of instances will be lost after the the while manpage.

The the while manpage executes by repeatedly running its content until a contained the get manpage fails or a the break manpage is encountered. At this point execution continues after the the while manpage. Thus failure of a the get manpage inside a the while manpage is not an execution error. See the TUTORIAL for a realistic example of the the while manpage statement.

## break

The the break manpage statement is used as a controlled failure. When it is encountered inside a the while manpage it causes the the while manpage to end. At any other time, it causes the whole the brook manpage to end.

## converge

The the converge manpage statement allows one to craft convergent processes that edit instances to match a master list. the converge manpage must have two attributes key and instances. instances must be a set of instances stored in the configuration as a the repeat manpage element. key is the name of a variant that is a unique identifier of each instance.

A the converge manpage markup must contain exactly five other markups the list manpage, the read manpage, the add manpage, the remove manpage, and the modify manpage (in any order). Each of these describes how to perform one part of the convergent process. the list manpage creates a the repeat manpage block of instances having unique key's, while the others act on particular instances in the instances collection with names in instance scope. For example:

        <converge key='user' instances='people'>
<list> ... </list>
<remove> ... </remove>
<modify> ... </modify>
</converge>

## list

The the list manpage markup must be contained within a the converge manpage markup and is not meaningful in any other scope. It should contain a the while manpage markup that generates a list of instances with unique keys as documented in the key tag of the containing the converge manpage.

The the read manpage markup must be contained within a the converge manpage markup and is not meaningful in any other scope. It should contain code that, given a key named as described in the containing the converge manpage, reads all configuration data for that instance. In the the converge manpage example above, the read manpage should read all data corresponding to one user, where user is predefined to be one real user.

The the add manpage markup must be contained within a the converge manpage markup and is not meaningful in any other scope. It should contain code that, given configuration data for an instances described in the instances parameter of the containing the converge manpage, creates a new real corresponding instance on the device. In the the converge manpage example above, the add manpage should add all data corresponding to one user from an instance within the the repeat manpage context named people. where user is predefined to be one real user.

## remove

The the remove manpage markup must be contained within a the converge manpage markup and is not meaningful in any other scope. It should contain code that, given a key named as described in the containing the converge manpage, removes that instance from the device. In the the converge manpage example above, the read manpage should remove one user, where user is predefined to be one real user.

## modify

The the modify manpage markup must be contained within a the converge manpage markup and is not meaningful in any other scope. It should contain code that, given configuration data for an instances described in the instances parameter of the containing the converge manpage, modifies the real pre-existing instance on the device to match data in the instance. In the the converge manpage example above, the modify manpage should modify all data corresponding to one user from an instance within the the repeat manpage context named people. where user is predefined to be one real user.

## perl

The the perl manpage markup allows one to embed simple Perl code that acts on variant data and whose effects can be imported back into Babble for later use. It requires two attributes import and export with the same format. These are lists of variants to import and export, together with the structure of each variant.

Each import or export list has the same structure. It is a comma-separated list of variant names with structural declarations following each name. The default structure is 'string'. A the case manpage structure is indicated via { ... } where the { ... } can optionally contain structures itself to describe structures of subcases. A the repeat manpage structure is indicated via [ ... ] where the [ ... ] can optionally contain the structure accorded to each element of the repeating structure. Thus the structure declaration:

        import='foo,bar[],cat{},dog[a,b],horse{a,b[hoodoo]}'

grabs a string variant named foo, the repeat manpage's named bar and dog, cases named cat, and horse, where the substructure of dog and horse is recursively defined. In particular, the case horse has a variant that is a repeat containing a string variant named hoodoo.

This declaration is utilized in two ways. Before the contained Perl is invoked, the import list is checked against actual variant types in the active binding scope. If types do not match for declared subparts, the run is aborted. The types of undeclared subparts aren't checked; e.g., the the repeat manpage variant dog above could have another attribute george whose type could be anything at all.

If types match the declarations, then variables in the declaration are imported directly into a Perl module Sandbox that will contain the execution. Variants are mapped to scalars whose structure reflects type. Strings are simple scalars; the case manpages map to references to associative arrays, and the repeat manpage's map to references to regular arrays.

For example the import type above creates

• a string $foo • an array reference$bar (with length scalar(@$bar)), where every element of the array is an associative array describing an instance. • an associative reference$cat.

• an array reference $dog where every element of the array is an associative array reference with (at least) string values for 'a' and 'b'. • an associative array reference$horse with substructure as declared.

In particular, we know that $horse->{'b'} is an array reference with length scalar(@{$horse->{'b'}}) and, for each element $i of that array,$horse->{'b'}->[\$i]->{'hoodoo'} should be a string!

## timeout

The the timeout manpage markup controls the length of timeout before a the get manpage fails. It must have one attribute seconds representing the number of seconds to wait. For example, <timeout seconds=10/> (the default) says to wait 10 seconds after the last received input before declaring a the get manpage failure.

## dump

The dump markup causes a dump of a snapshot of variant space. All binding contexts are flattened into one inclusive context. It requires a file attribute describing where to dump the information. The markup

        <dump file='/tmp/foo'/>

dumps the whole binding context to the given file.

# CONFIG MARKUPS

The following markups can be used in configuration files to declare kinds of variants. These are different from, but related to, the STREAM MARKUPS that comprise Babble's configuration files.

## config

The the config manpage markup must contain all config tags. This spans each configuration file.

## var

The the var manpage markup declares a string value. It must have a name attribute. The value is contained within the content of the the var manpage. Only string content may appear there. For example, the markup

        <var name='mood'>Happy</var>

declares a variant mood with value Happy.

## case

The the case manpage markup declares a special case. It can contain the var manpage, the case manpage, or the repeat manpage markups as content. It cannot contain unmarked string content. For example, the markup

        <case name='joe'>
<var name='material'>silicon</var>
<var name='pseudonym'>D'Artignan</var>
</case>

creates a special case with two contained variants, material and pseudonym.

## repeat

The the repeat manpage tag indicates that its content is a set of instances of a single entity to be processed as a batch. It must have a name. All other attributes are ignored. Its contents must be the instance manpages that describe different versions of the entity. For example:

        <repeat name='tigers'>
<instance>
<var name='first'>Hobbes</var>
</instance>
<instance>
<var name='first'>Leo</var>
</instance>
</repeat>

describes a variant tigers that contains two entities, whose first names are Hobbes and Leo, respectively.

## instance

The the instance manpage tag only occurs inside a the repeat manpage and must comprise its only content. It has no name but can contain a free mix of named forms such as the var manpage's, the case manpage's, and the repeat manpage's.

# VALUE INHERITANCE

Babble is a dynamically scoped environment. This means that in a given the put manpage context, variants are searched for from inner to outer scopes, where the first match wins. This may seem confusing until one realizes that it allows very terse variant specifications involving defaults.

For example, one might note that there is a lot of repetition in the configuration:

        <config>
<repeat name='people'>
<instance>
<var name='direct'>1-11</var>
<var name='listen'>1-11</var>
<var name='buffer'>Y</var>
<var name='clear'>Y</var>
</instance>
<instance>
<var name='direct'>1-11</var>
<var name='listen'>1-11</var>
<var name='buffer'>Y</var>
<var name='clear'>Y</var>
</instance>
</repeat>
</config>

We can exploit dynamic scoping to define defaults for repeated the var manpage's:

        <config>
<var name='direct'>1-11</var>
<var name='listen'>1-11</var>
<var name='buffer'>Y</var>
<var name='clear'>Y</var>
<repeat name='people'>
<instance>
</instance>
<instance>
</instance>
</repeat>
</config>

In the inner context of the repeat, username is defined for each variant. However, logins is only defined in an upper context containing the the repeat manpage. The effect is that the initial the var manpage's become defaults for the lower context. If one wishes, one can still override a default:

        <config>
<var name='direct'>1-11</var>
<var name='listen'>1-11</var>
<var name='buffer'>Y</var>
<var name='clear'>Y</var>
<repeat name='people'>
<instance>
</instance>
<instance>
</instance>
</repeat>
</config>

This says that user foo gets 2 logins while everyone else gets 1.

## Contravariance

In Babble it is very poor style to change the type of a variant from one type to another, e.g., change a string to a the case manpage or the repeat manpage. Future versions of Babble will detect and prohibit such acts. But there is a much more subtle version of contravariance of which one should be wary.

Babble scripts depend upon a homogeneity property in the repeat manpage contexts, that a named variant that is available in one instance is available in all instances. This can be assured either by specific data or defaults. It is particularly bad style to create instances with parameters that aren't available in others, e.g.,

        <repeat name='hosed'>
<instance>
<var name='crash'>me</var>
</instance>
<instance>
<var name='with'>this</var>
</instance>
</repeat>

Future versions of Babble will detect and prohibit this more subtle contravariance. Of course, one can always repair the damage by declaring defaults:

        <var name='crash'>averted</var>
<var name='with'>defaults</var>
<repeat name='hosed'>
<instance>
<var name='crash'>me</var>
</instance>
<instance>
<var name='with'>this</var>
</instance>
</repeat>

# INVOKING BABBLE

Babble has a number of command-line options that become available to any script created with the #! convention described above. These options control logging, tracing, debugging, and other behavior.

Babble executes in two modes. In its primary mode, Babble implements a documented stream of interactions. In another mode, it records a stream for future execution. The recording mode is not yet fully functional; use at your own risk.

In the first mode, Babble is executed either from within #! or from the command line with an explicit input file. If, e.g., the first line of your script 'boo' is

        #! /usr/bin/perl /usr/local/bin/babble

then the invocation

        boo --trace=9

is equivalent with the invocation

        /usr/local/bin/babble boo --trace=9

so that the babble parameters are

        boo --trace=9

In most UNIX's, one cannot place options into the #! line; these must be placed on subsequent command lines.

In the first mode, Babble is executed with an immediate file name of a script to parse and run. If this script is invoked with #!, the options are as follows:

--help
Print a help message.

--errorfile=/path/to/file
Redirect error output to the given file (default STDERR). =item --debug=9

Set debug level (0-9) (default 0). This is only really useful when trying to debug or extend Babble itself.

--debugfile=/path/to/file
Redirect debug trace to the given file (default STDERR). This only redirects debugging and has no effect if debugging is off (0).

--trace=3
Set execution trace level (0-9) (default 0). There are four trace levels of interest to the casual user. 0 means tracing is off. 1 means to trace entry and exit from markups only. 2 means to show some internal details. 3 means to show all reasonable internal details that could aid in debugging.

--tracefile=/path/to/file
Redirect execution trace to file (default STDERR). This only redirects tracing and has no effect if tracing is off (0).

--xmlout
Generate XML rendition of execution (default OFF). This option allows one to generate an XML markup file for execution itself, showing how tags correspond with output. At this point it is a debugging aid. Future versions of Babble will be able to utilize this information to construct a graphical depiction of execution that can be browsed.

--xmloutfile=/path/to/file
Redirect XML to the given file (default xml.out). This only affects XML rendition and has no effect if XML generation is off.

--rawout
Record raw rendition of execution (default OFF). This flag tells Babble to record a raw version of all device output for use in debugging.

--rawoutfile=/path/to/file
Redirect raw rendition to a given file (default raw.out). This only affects raw rendition and has no effect if raw rendition is off.

--invoke=brookname
Start invocation with the given the brook manpage (default 'main'). This forces execution to 'jump' to the named the brook manpage instead of main when beginning execution. This can be used to create one file with several functions.

--bind=/path/to/file
Read variants from specified the config manpage file before execution (default none). The file is parsed and treated as configuration information. The scope of this information is the whole Babble run.

--spawn=command to script''
Make the given command active before execution so that all of execution may interact with it (default none). The given command is the spawn manpageed just as if it appeared in a the spawn manpage command. Scope is all of execution.

Stream recording allows one to use Babble to record interactions between a human and a device for future playback. To do this one invokes Babble as:

        babble -converse="command to script" <options>

In this mode, babble tries to capture an interactive session in a runnable form. Options may include:

--conversefile=/path/to/file
Redirect interactive capture to given file (default converse.out).

In conversational mode, Babble tries to open the device and allow you to type its responses. It records these in XML format so that the result is runnable as a Babble script.

Warning: this has plenty of bugs. First, Babble keeps your terminal in line-buffered mode, so that it won't have to cope with line-editing characters in its logs. This breaks some device contacts.

# BUGS

Babble is an unbelievably complex Perl script and the list of bugs and deficiencies (todo's) for Babble is almost as long as its documentation. Here's a short list of the major problems upon which I am working at this time:

• Babble is too lenient for many errors and simply warns and ignores extreme indiscretions instead of bombing outright. Examples include markups with missing required attributes.

• Type checking in Babble does include flow analysis of the effects of bindings by the get manpage, the repeat manpage, the while manpage, and the perl manpage markups, but the type checking is still rather naive as documented above. One major omission is contravariance checking on scope chains as mentioned above in the section on the repeat manpage and the instance manpage. Another problem concerns flow analysis and contravariance checking for junctions contained within the repeat manpage and the while manpage. Currently the union of all variants instantiated in the junction is passed into the flow analysis, making for a rather naive result.

• XML output from Babble doesn't contextualize the whole raw output, just what got matched. This needs to be fixed before we can use that output to visualize execution.

• Each the get manpage markup can contain at most 16 the get manpage markups. This is due to a silly technique for binding variants as a result of the the get manpage.

# TODO

There are an almost unlimited number of things to do to make Babble better.

• First, the --converse option should be fixed so that it really allows one to converse with a remote device with Babble recording the results. This in turn will allow one to run Babble in a mixed mode in which it prompts a user whenever a the get manpage times out.

• There is no reason that users have to write their own the branch manpage statements. This sort of thing can be inferred automatically from knowledge the user can give. The program should be able to fuse variant executions into one brook with appropriate variant structure.

• One major omission is the ability to split apart fixed-format fields. This will be provided by a tab markup that skips to a particular column of input.

# AUTHOR

Alva L. Couch, couch@eecs.tufts.edu, http://www.eecs.tufts.edu/~couch`, Copyright 2000 by Alva L. Couch.