Babble - an automation tool for crafting interactive scripts.
Babble is a language for creating very complex interactive scripts to be used in managing a network of standalone appliance-style devices. Unlike 'expect' and its relatives, a Babble script does not contain any code from another language such as tcl or perl (unless desired). A Babble 'script' is instead documentation of the structure of an interactive conversation, described using an XML-like syntax.
The Babble interpreter utilizes this documentation to interact with devices, thus avoiding traditional scripting entirely. A powerful variant mechanism allows one to reuse this documentation for accomplishing several related effects, while a type-checking system avoids common documentation errors. Although Babble was originally intended to interact with external devices, it can also be used to document and replay shell interactions, thus automating many common tasks without scripting. In this way we also avoid documenting the scripts, and avoid all the maintenance problems inherent for hastily-written code written in rapid prototyping languages.
One advantage of using documentation rather than scripting is that one can craft the documentation entirely by annotating a stored script of user-machine interactions. One captures this 'script' while manually performing the operation desired, and then annotates the stored script so that Babble knows exactly how to repeat it. This only requires telling Babble things that are not intuitively obvious from the script, e.g., which output was typed by the user and which by the machine. The creator of Babble documentation need not know how to program; just how to accomplish desired ends with appropriate user commands.
While writing a script to automate a complex process is empowering for the administrator, custom scripts create maintenance problems. These scripts are typically poorly documented, and thus only usable by the creator. This renders the creator indispensible, while rendering the overall organization in which the creator works unstable.
Crafting documentation is different from crafting scripts in several important ways. First, one need not abstract behavior from examples, but can instead directly annotate examples to create things that work like scripts. This means in particular that the creator of Babble documentation need not know how to program in the traditional sense, though it is necessary to know how to manually perform all the actions you would like to accomplish.
This approach has several advantages:
This method of achieving automation is non-intuitive at first because we must stop writing commands that accomplish actions and instead document the effects of commands we could type. Normal scripting requires imperative thinking, while Babble requires declarative thinking. If we take the extra effort, however, Babble then takes responsibility for many quality checks that we would have to craft ourselves when writing custom scripts.
This file is part of Babble. All files in the distribution of Babble are Copyright 2000 by
Alva L. Couch,
Associate Professor of Computer Science,
Tufts University, Medford MA 02155
email: couch@eecs.tufts.edu
phone: 617-627-3674
All rights reserved.
Redistribution and use are permitted provided that this entire copyright notice is duplicated in all such copies, and that any documentation, announcements, and other materials related to such distribution and use acknowledge that the software was developed by Alva L. Couch at Tufts University in Medford, Massachusetts, USA. No charge, other than an "at-cost" distribution fee, may be charged for copies, derivations, or distributions of this material without the express written consent of the copyright holder. Neither the name of the University nor the names of the authors may be used to endorse or promote products derived from this material without specific prior written permission. THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR ANY PARTICULAR PURPOSE. =head2 Requirements
Babble-0.1Alpha is written in perl-5.006 and works on virtually all UN*X-compatible systems supporting Perl. While it should in principle run on any Perl >= 5.000, some of the code for embedded Perl exposes bugs in Perl 5.005_02. causing core dumps at the end of execution of perl segments. For best results utilize Perl 5.006 or later.
Babble-0.1Alpha has been tested only on RedHat Linux and Solaris 2.7 systems. It should execute as documented herein on any UNIX system where shell execution semantics is similar to these. But there are no guarantees.
Babble requires three Perl modules that you can load from the Comprehensive Perl Archive Network (CPAN).
Babble installation is very simple because the Babble interpreter is packaged as one very large Perl script.
root# perl -MCPAN -e shell
cpan> install XML::Parser
cpan> install IO::Tty
cpan> install Data::Dumper
If XML::Parser complains that Expat is not yet installed, fetch it from the source that the CPAN script recommends and compile it before continuing.
#! /usr/bin/perl
#! <location of perl> <location of babble>
If your perl is /usr/bin/perl, and babble is located in /usr/local/bin/babble, edit each script so that the first line is:
#! /usr/bin/perl /usr/local/bin/babble
The next step in using Babble is to create some documentation that it can utilize to perform a task. This documentation replaces a traditional script. Simple documentation involves transforming a script of user interactions so that they instead document the interactions. This documentation process requires embedding markups that describe the function of each component of the script in a way that allows it to be reused.
The simplest way to document script elements is by labelling them with <get> and <put> tags. Content between <get> and </get> tags is something we expect from the device with which we are conversing. Content between <put> and </put> tags is something we type in accomplishing the task.
For example, suppose we want to add a user to a LightWave ConsoleServer 3200. The dialog we must undertake looks like this:
sys admin> adduser
Number of available user records: 196
Number of users defined: 4
Enter user id | USER ID > foo
Enter case sensitive password | PASSWORD > ***
Re-enter case sensitive password | PASSWORD > ***
0-17 | MAX CONCURRENT LOGINS: 1>1
Allowed devices example: 1-5,10 | DEVICES 0 > 1-11
Allowed listen devices example: 1-5,10 | DEVICES 0 > 0
Allow user to clear device buffer (Y/N) | YES > Y
Clear screen after a command (Y/N) | YES > Y
sys admin>
The first step is to translate this script into Babble's language. This requires replacing whitespace and XML special characters with new equivalents:
sys\sadmin>\sadduser\n
Number\sof\savailable\suser\srecords:\s196\n
Number\sof\susers\sdefined:\s4\n
Enter\suser\sid\s|\sUSER\sID\s>\sfoo\n
Enter\scase\ssensitive\spassword\s|\sPASSWORD\s>\s***\n
Re-enter\scase\ssensitive\spassword\s|\sPASSWORD\s>\s***\n
0-17\s|\sMAX\sCONCURRENT\sLOGINS:\s1>1\n
Allowed\sdevices\sexample:\s1-5,10\s|\sDEVICES\s0\s>\s1-11\n
Allowed\slisten\sdevices\sexample:\s1-5,10\s|\sDEVICES\s0\s>\s0\n
Allow\suser\sto\sclear\sdevice\sbuffer\s(Y/N)\s|\sYES\s>\sY\n
Clear\sscreen\safter\sa\scommand\s(Y/N)\s|\sYES\s>\sY\n
\n
sys\sadmin>\s\n
In each Babble script, literal spaces, tabs, and linefeeds are ignored. The following substitutions are utilized to represent literal characters in the input and output streams:
In addition, Babble scripts are a variant of the Extensible Markup Language (XML). This means that one must also utilize special codes for special XML markup characters:
The next step is to annotate things typed by and read by you. I put things I type into the put manpage brackets, while things the machine types are put into the get manpage brackets. The following example documents this script, showing what Babble should see and what Babble should type:
<get>sys\sadmin>\s</get><put>adduser</put>\n
Number\sof\savailable\suser\srecords:\s196\n
Number\sof\susers\sdefined:\s4\n
<get>Enter\suser\sid\s|\sUSER\sID\s>\s</get>
<put>foo<noecho>\n</noecho></put>
<get>Enter\scase\ssensitive\spassword\s|\sPASSWORD\s>\s</get>
<put>bar<noecho>\n</noecho></put>
<get>Re-enter\scase\ssensitive\spassword\s|\sPASSWORD\s>\s</get>
<put>bar<noecho>\n</noecho></put>
<get>0-17\s|\sMAX\sCONCURRENT\sLOGINS:\s1></get>
<put>1<noecho>\n</noecho></put>
<get>Allowed\sdevices\sexample:\s1-5,10\s|\sDEVICES\s0\s>\s</get>
<put>1-11<noecho>\n</noecho></put>
<get>Allowed\slisten\sdevices\sexample:\s1-5,10\s|\sDEVICES\s0\s>\s</get>
<put>1-11<noecho>\n</noecho></put>
<get>Allow\suser\sto\sclear\sdevice\sbuffer\s(Y/N)\s|\sYES\s>\s<get>
<put>Y<noecho>\n</noecho></put>
<get>Clear\sscreen\safter\sa\scommand\s(Y/N)\s|\sYES\s>\s</get>
<put>Y<noecho>\n</noecho></put>
To actually execute this script, we have to put it into an execution context, giving it a name and a device to which to talk. We do this by surrounding the script with the directives
#! /usr/bin/perl /usr/local/bin/babble
<stream>
<brook name='main'>
<spawn command="cu -l cua/b -b 8">
.... script here ....
</spawn>
</brook>
</stream>
These lines satisfy several requirements of a complete Babble script.
main brook is what Babble executes.
A Babble script must be compliant with XML syntax (except for embedded Perl, which is handled specially). This means that within a script, markups must all have closing tags. Leaving any <get> or <put> tag unclosed comprises a syntax error.
The whole script, ready to run, looks like this:
#! /usr/bin/perl /usr/local/bin/babble
<stream>
<brook name='main'>
<spawn command="cu -l cua/b -b 8">
<get>sys\sadmin>\s</get><put>adduser</put>\n
Number\sof\savailable\suser\srecords:\s196\n
Number\sof\susers\sdefined:\s4\n
<get>Enter\suser\sid\s|\sUSER\sID\s>\s</get>
<put>foo<noecho>\n</noecho></put>
<get>Enter\scase\ssensitive\spassword\s|\sPASSWORD\s>\s</get>
<put>bar<noecho>\n</noecho></put>
<get>Re-enter\scase\ssensitive\spassword\s|\sPASSWORD\s>\s</get>
<put>bar<noecho>\n</noecho></put>
<get>0-17\s|\sMAX\sCONCURRENT\sLOGINS:\s1></get>
<put>1<noecho>\n</noecho></put>
<get>Allowed\sdevices\sexample:\s1-5,10\s|\sDEVICES\s0\s>\s</get>
<put>1-11<noecho>\n</noecho></put>
<get>Allowed\slisten\sdevices\sexample:\s1-5,10\s|\sDEVICES\s0\s>\s</get>
<put>1-11<noecho>\n</noecho></put>
<get>Allow\suser\sto\sclear\sdevice\sbuffer\s(Y/N)\s|\sYES\s>\s<get>
<put>Y<noecho>\n</noecho></put>
<get>Clear\sscreen\safter\sa\scommand\s(Y/N)\s|\sYES\s>\s</get>
<put>Y<noecho>\n</noecho></put>
</spawn>
</brook>
</stream>
Whitespace is ignored. You may utilize this to make things print more neatly ,e.g., folding lines like:
<get>Allowed\slisten\sdevices\sexample:\s1-5,10\s|\sDEVICES\s0\s>\s</get>
anywhere within the line, say, at
<get>Allowed\slisten\sdevices\sexample:
\s1-5,10\s|\s DEVICES\s0\s>\s
</get>
To run this script, make this file executable, and simply type it as a command. Babble will take over and execute the interactions for you.
If you want to see what Babble does, the command that you just created takes command line options that Babble interprets to change its behavior. If the command you created above is 'uadd', then
uadd --trace=3 turns on all tracing so that you can watch Babble work.
Tracing tells you what babble is doing at each point in
its execution.
uadd --tracefile=/tmp/trace
redirects the trace to a file rather than the screen so you can edit
it later.
uadd --rawout --rawoutfile=/tmp/raw
creates a file containing raw output from the device, as received by Babble.
uadd --xmlout --xmloutfile=/tmp/xml
Literal scripts that simply repeat actions aren't very useful. We need to be able to generalize them so they can deal with both variations in tasks and behavior. The former requires being able to define variables, while the latter requires defining logical branches.
A variant is the Babble version of a variable. I call it something different because it behaves somewhat differently than a normal variable. There is no way to set a variant in a Babble script. Instead, one defines a configuration that determines variant values for a specific run of Babble.
Suppose that we want to annotate the <code>uadd</code> command we created above so that we can create different users. The first step is to figure out what varies in the above stream. We mark things that vary from instance to instance with var tags:
#! /usr/bin/perl /usr/local/bin/babble
<stream>
<brook name='main'>
<spawn command="cu -l cua/b -b 8">
<get>sys\sadmin>\s</get><put>adduser</put>\n
<get>Number\sof\savailable\suser\srecords:\s
<var>196</var>\n</get>
<get>Number\sof\susers\sdefined:\s
<var>4</var>\n</get>
<get>Enter\suser\sid\s|\sUSER\sID\s>\s</get>
<put>
<var>foo</var>
<noecho>\n</noecho>
</put>
<get>Enter\scase\ssensitive\spassword\s|\sPASSWORD\s>\s</get>
<put>
<var>bar</var>
<noecho>\n</noecho>
</put>
<get>Re-enter\scase\ssensitive\spassword\s|\sPASSWORD\s>\s</get>
<put>
<var>bar</var>
<noecho>\n</noecho>
</put>
<get>0-17\s|\sMAX\sCONCURRENT\sLOGINS:\s1></get>
<put>
<var>1</var>
<noecho>\n</noecho>
</put>
<get>Allowed\sdevices\sexample:\s1-5,10\s|\sDEVICES\s0\s>\s</get>
<put>
<var>1-11</var>
<noecho>\n</noecho>
</put>
<get>Allowed\slisten\sdevices\sexample:\s1-5,10\s|\sDEVICES\s0\s>\s</get>
<put>
<var>1-11</var>
<noecho>\n</noecho>
</put>
<get>Allow\suser\sto\sclear\sdevice\sbuffer\s(Y/N)\s|\sYES\s>\s<get>
<put>
<var>Y</var>
<noecho>\n</noecho>
</put>
<get>Clear\sscreen\safter\sa\scommand\s(Y/N)\s|\sYES\s>\s</get>
<put>
<var>Y</var>
<noecho>\n</noecho>
</put>
</spawn>
</brook>
</stream>
Since var markups have to occur inside get or put markups, I added two get markups at the beginning. This tells Babble that it can make a sanity check by listening for these two extra lines.
This markup by itself isn't enough to instruct Babble on how to run the script. We also have to give each variant attributes that describe it so Babble can perform substitutions. Variants inside put need names; variants inside <get> need regular expression patterns that identify them:
#! /usr/bin/perl /usr/local/bin/babble
<stream>
<brook name='main'>
<spawn command="cu -l cua/b -b 8">
<get>sys\sadmin>\s</get><put>adduser</put>\n
<get>Number\sof\savailable\suser\srecords:\s
<var pattern='[0-9]+'>196</var>\n</get>
<get>Number\sof\susers\sdefined:\s
<var pattern='[0-9]+'>4</var>\n</get>
<get>Enter\suser\sid\s|\sUSER\sID\s>\s</get>
<put>
<var name='username'>foo</var>
<noecho>\n</noecho>
</put>
<get>Enter\scase\ssensitive\spassword\s|\sPASSWORD\s>\s</get>
<put>
<var name='password'>bar</var>
<noecho>\n</noecho>
</put>
<get>Re-enter\scase\ssensitive\spassword\s|\sPASSWORD\s>\s</get>
<put>
<var name='password'>bar</var>
<noecho>\n</noecho>
</put>
<get>0-17\s|\sMAX\sCONCURRENT\sLOGINS:\s1></get>
<put>
<var name='logins'>1</var>
<noecho>\n</noecho>
</put>
<get>Allowed\sdevices\sexample:\s1-5,10\s|\sDEVICES\s0\s>\s</get>
<put>
<var name='direct'>1-11</var>
<noecho>\n</noecho>
</put>
<get>Allowed\slisten\sdevices\sexample:\s1-5,10\s|\sDEVICES\s0\s>\s</get>
<put>
<var name='listen'>1-11</var>
<noecho>\n</noecho>
</put>
<get>Allow\suser\sto\sclear\sdevice\sbuffer\s(Y/N)\s|\sYES\s>\s<get>
<put>
<var name='buffer'>Y</var>
<noecho>\n</noecho>
</put>
<get>Clear\sscreen\safter\sa\scommand\s(Y/N)\s|\sYES\s>\s</get>
<put>
<var name='clear'>Y</var>
<noecho>\n</noecho>
</put>
</spawn>
</brook>
</stream>
The final step is to separate values of variants from the script. We do this by creating another file uadd.cfg that contains the values we wish to use for variants:
<config>
<var name='username'>foo</var>
<var name='password'>bar</var>
<var name='logins'>1</var>
<var name='direct'>1-11</var>
<var name='listen'>1-11</var>
<var name='buffer'>Y</var>
<var name='clear'>Y</var>
</config>
The final step is to utilize this set of variants in the Babble invocation
uadd --bind=uadd.cfg
If we change anything in uadd.cfg, the change will be immediately reflected in the script.
One immediate thing we might want to do is to reuse this script for creating multiple users. We can do this by using cases. A case is a specific instance of a problem. We can feed cases to the script by renaming it and calling it from another.
Let's suppose that we rename our user creating brook as
adduser. Then we can make up a couple of cases
in the configuration file as follows:
<config>
<case name='c1'>
<var name='username'>cat</var>
<var name='password'>dog</var>
<var name='logins'>1</var>
<var name='direct'>1-11</var>
<var name='listen'>1-11</var>
<var name='buffer'>Y</var>
<var name='clear'>Y</var>
</case>
<case name='c2'>
<var name='username'>foo</var>
<var name='password'>bar</var>
<var name='logins'>1</var>
<var name='direct'>1-11</var>
<var name='listen'>1-11</var>
<var name='buffer'>Y</var>
<var name='clear'>Y</var>
</case>
</config>
Then we can call each case in a new main brook as follows:
<brook name='main'>
<case instance='c1'>
<insert brook="adduser"/>
</case>
<case instance='c2'>
<insert brook="adduser"/>
</case>
</brook>
The insert clause simulates typing the code for the uadd brook
twice. We again invoke this with
uadd --bind=uadd.cfg
For each case, variants are bound as listed in the configuration
file, and the brook uadd is invoked with those bindings in place.
The result is to add two users.
What if we want to create multiple users?
It gets tedious to have to name all the instances.
We could create a uadd.cfg for each one,
or a the case manpage for each one,
but it'd be better to be able to repeat the whole script
for a set of people. We can do this by
using repeat markups in both script and configuration.
For example, suppose we want to create the users foo and
cat. We can make these the instance manpages of a the repeat manpage clause
in the configuration as follows:
<config>
<repeat name='people'>
<instance>
<var name='username'>cat</var>
<var name='password'>dog</var>
<var name='logins'>1</var>
<var name='direct'>1-11</var>
<var name='listen'>1-11</var>
<var name='buffer'>Y</var>
<var name='clear'>Y</var>
</instance>
<instance>
<var name='username'>foo</var>
<var name='password'>bar</var>
<var name='logins'>1</var>
<var name='direct'>1-11</var>
<var name='listen'>1-11</var>
<var name='buffer'>Y</var>
<var name='clear'>Y</var>
</instance>
</repeat>
</config>
In the code, if the name of our user adding brook is
uadd, we add a main brook:
<brook name='main'>
<bind file="uadd.cfg">
<repeat instances='people'>
<insert brook="adduser"/>
</repeat>
</bind>
</brook>
</brook>
The the bind manpage markup binds the names in the given file into our namespace
(temporarily), the the repeat manpage markup repeats its contents for each instance
of the given the repeat manpage configuration collection (people),
and the the insert manpage markup invokes our brook for each instance.
Unfortunately, most interactive processes aren't this smooth. Sometimes things vary at runtime and one must react. To account for this, Babble allows a stream to follow the branch manpagees depending upon what was received from the device.
As an example of the use of branches, consider the task of logging in as an administrator into the LightWave 3200. The console can actually be in any state, and we need to be able to detect its state and react appropriately when login isn't required:
<brook name="enable">
<put><noecho>\003</noecho></put>
<get>Operation\scancelled\sby\suser</get>
<get>>></get>
<put>login\sadmin\r</put>
<junction>
<branch>
<get>PLEASE\sENTER\sPASSWORD\s</get>
<put><stars><var name="adminpass">PASS</var></stars>\r</put>
</branch>
<branch>
<get>Already\slogged\sin.\r</get>\n
</branch>
</junction>
<get>sys\sadmin>></get>
</brook>
If we try to login and we're already logged in, we skip typing a password, else we do.
Branches aren't really fully compliant XML because they have an order constraint. The first tag inside any the branch manpage must be a the get manpage so we can check whether the branch should be taken. After that initial the get manpage, branches can contain any free mix of the put manpage's and the get manpage's.
Junctions are executed by collecting all the initial gets of
all branches, and checking each of them against input from the
device until there is a match or timeout. The first match wins.
This can be a bit subtle in practice because it is the opposite
of normal pattern matching behavior. In Babble the shortest
pattern match always wins. If one pattern in a junction is
a prefix of another, the latter will never be matched.
So far we have only used predefined variants in running a script. Babble can also discover the values of variants at runtime. The mechanism for this is to place a named the var manpage markup inside a the get manpage rather than a the put manpage.
Suppose first that we want to be able to read user data for users we have already defined. The following the brook manpage will do this:
<brook name="readuser">
<put>listuser\s<var name="username">foo</var>\r</put>\n
<get>
User\sId\s>\s
<var pattern="[A-Z]+">FOO</var>
<var pattern="\s+">\s</var>\r\n
</get>
<get>
Allowed\sdevices\s>\s
<var name="direct" pattern="[-0-9,]+">1-32</var>
\r\n
</get>
<get>
Allowed\slisten\sdevices\s>\s
<var name="listen" pattern="[-0-9,]+">1-32</var>
\r\n
</get>
<get>
Max\slogins\s>\s<var name="logins" pattern="[0-9]+">1</var>\r\n
</get>
<get>Allow\suser\sto\sclear\sdevice\sbuffer\s>\s
<var name="buffer" pattern="Y|N">N</var>
<var pattern="ES|O">0</var>\r\n
</get>
<get>Clear\sscreen\safter\sa\scommand\s>\s
<var name="clear" pattern="Y|N">Y</var>
<var pattern="ES|O">ES</var>\r\n
</get>
<get>\r\n</get>
<get>sys\sadmin></get>
</brook>
In this example, there are two kinds of the var manpage's inside the get manpage's.
A named variant is one with a name attribute. This will
cause variant space to be updated with the new value.
An unnamed variant is one without a name. This
allows one to skip over input that varies in
a predictable way, without binding any variants to values.
In this version of Babble, naming a variant always
causes binding of that variant. This behavior may be alterable in
future versions of Babble via a bind attribute.
Very often, a device one wishes to control has more than one instance of a thing defined that we must discover. In this case we may utilize a the while manpage loop to discover all of them. For example, let's discover all users by reading a listing of their names:
<brook name="usernames">
<put>listusers\r\n</put>
<while instances="people">
<junction>
<branch>
<get>User\sId\s>\s
<var name="username" pattern="[A-Z0-9]+">COUCH</var>
<var pattern="\s*"/>\r\n
</get>
</branch>
<branch>
<get>sys\sadmin></get>
<break/>
</branch>
<branch>
<get>More</get>
<put><noecho>\s</noecho></put>
</branch>
</junction>
</while>
</brook>
This code is the first really powerful use of Babble.
The the while manpage phrase collects instances of people
into a new the repeat manpage configuration variable that can be used with a future
the repeat manpage form. Inside the the while manpage there are three
branches. The first branch grabs a username.
The second branch breaks out of the the while manpage loop
if the device gives a prompt, using a the break manpage
tag just as in other programming languages. The third
presses 'spacebar' to continue the builtin
pager. If there were no the break manpage tag, the the while manpage would
have ended upon the timeout manpage after no further input lines
matched the first phrase.
The real power of the while manpage is that the output of a the while manpage
can be the input to a the repeat manpage. Here we take the process
further by reading the attributes of each user with another
script readuser:
<brook name="readuser">
<put>listuser\s<var name="username">couch</var>\r</put>\n
<get>
User\sId\s>\s
<var pattern="[A-Z]+">COUCH</var>
<var pattern="\s+">\s</var>\r\n
</get>
<get>
Allowed\sdevices\s>\s
<var name="direct" pattern="[-0-9,]+">1-32</var>
\r\n
</get>
<get>
Allowed\slisten\sdevices\s>\s
<var name="listen" pattern="[-0-9,]+">1-32</var>
\r\n
</get>
<get>
Max\slogins\s>\s<var name="logins" pattern="[0-9]+">1</var>\r\n
</get>
<get>Allow\suser\sto\sclear\sdevice\sbuffer\s>\s
<var name="buffer" pattern="Y|N">N</var>
<var pattern="ES|O">0</var>\r\n
</get>
<get>Clear\sscreen\safter\sa\scommand\s>\s
<var name="clear" pattern="Y|N">Y</var>
<var pattern="ES|O">ES</var>\r\n
</get>
<get>\r\n</get>
<get>sys\sadmin></get>
</brook>
<brook name="readusers">
<repeat instances="people">
<insert brook="readuser"/>
</repeat>
</brook>
<brook name='main'>
<insert brook="usernames"/>
<insert brook="readusers"/>
<dump file="/tmp/users"/>
</brook>
The brook readusers uses a repeat on the discovered instances
people to complete each instance with all user information.
At the end of this process it writes all user info into a
file /tmp/users that can be utilized as a configuration file
itself. This allows one to read and archive a running configuration
that can be reused in order to create a configuration from scratch.
Something really subtle is happening here. If you use a the repeat manpage
after a the while manpage and then discover values within the the repeat manpage,
the process of discovery is inductive. This means that after
both clauses, the people instances contain information
from both. This process can be repeated interminably to update
the instance information in as many ways as desired.
The most powerful feature of the current version of Babble is convergence. This feature allows you to script automatic changes that incrementally modify an existing set of instances to be another. For example, consider the task of creating users. If there's an extra user we'd like to delete it, while if there's a user not yet created we'd like to create it. This is difficult to program just using the repeat manpage and the while manpage so there is another tag the converge manpage to handle this.
For example, suppose we have several brooks already defined that do the following things:
usernames lists current user names.
adduser adds a user.
removeuser adds a user.
modifyuser modifies a user.
readuser reads one user's data (by username).
Then the code:
<converge key="username" instances="people">
<list><insert brook="usernames"/></list>
<add><insert brook="adduser"/></add>
<remove><insert brook="removeuser"/></remove>
<modify><insert brook="modifyuser"/></modify>
<read><insert brook="readuser"/></read>
</converge>
invokes a really complex process that matches user data in the configuration file with user data in the device, making minimal changes in order to do so. The process proceeds as follows.
usernames to read a list
of user names from the device.
removeuser to remove them from the device.
adduser to add them to the device.
readuser
to read their data and compare it with
the configuration file.
modifyuser to modify its records.
In this way, we 'converge' to the desired configuration with minimal changes to the running configuration in the process.
Ideally, the Babble user won't have to write any Perl, but there are specific situations in which the ability to embed Perl code comes in handy. The Perl embedding interface allows one to operate on Babble configuration parameters directly and thus modify Babble's knowledge of the world. This comes in handy if, e.g., your configuration data is not already in a configuration file, but lives instead in some external Perl-accessible database. You can then write Perl code to import data into Babble's configuration hierarchy directly. For more information, see the entry for the the perl manpage markup in the next section.
The following markups can be used in the stream file that Babble interprets as a script. These are different from, but related to, the CONFIG MARKUPS that comprise Babble's configuration files.
The the stream manpage markup must begin and end the stream file, occurring directly after the Babble invocation comment (the first line of the file, which must start with #!). A stream should only contain the brook manpage's to be invoked. Within a the stream manpage (as well as within all other tags except the get manpage and the put manpage), text content is ignored.
The the brook manpage tag contains a sequence of instructions to be played.
It must have a name but all other attributes are ignored.
All the brook manpages should be immediate children of the the stream manpage tag.
One can invoke a brook either directly from the command line
or with the the insert manpage directive. E.g., the the brook manpage:
<brook name='foo'>
...
</brook>
may be invoked with the argument --brook=foo or with the XML markup:
<insert brook='foo'/>
inside another brook or its content.
The the insert manpage tag allows one to invoke one the brook manpage inside another. Its purpose is to implement a primitive 'subroutine call' for the brook manpages. At time of invocation, Babble simulates the physical presence of the the brook manpage at the insert point, as if one manually typed it in at that point. For an example see the brook manpage.
The the bind manpage tag allows one to control variant values within
a stream execution by reading and utilizing a new configuration
file. It has one required attribute file that denotes
the name of a configuration file to be loaded. For example:
<bind file='foo.cfg'>
...
</bind>
reads the file foo.cfg and makes its variants available inside the the bind manpage.
Configuration files nest. If one invokes more than one the bind manpage, all values are available, subject to shadowing. For example, if foo.cfg is:
<config>
<var name='foo'>hi there</var>
</config>
and bar.cfg is
<config>
<var name='foo'>yo</var>
<var name='bar'>bo</var>
</config>
then inside the context created by
<bind file='foo.cfg'>
<bind file='bar.cfg'>
...
</bind>
</bind>
foo is yo and bar is bo.
The outer value of foo (hi there) is shadowed (hidden)
by the inner one.
Each binding creates a scope in which inductive discovery occurs. If you utilize a the get manpage and/or the while manpage to discover variant values within a binding scope, those values are lost when the binding scope closes. Closing a the bind manpage does not save discovered values; these must be saved via a the dump manpage command before closing the the bind manpage.
The the spawn manpage markup creates an interaction scope in which
device interaction occurs. It requires one attribute command
which is the text of a command to execute in a pseudo-tty.
Nested the spawn manpage markups completely shadow one another so that only the innermost is active at any one time. This allows one to craft streams in which interactions roam freely between several devices or processes. For example,
<spawn command='cu -l cua/b -b 8'>
<!-- talking to cua/b ... -->
<spawn command='cu -l cua/a -b 8'>
<!-- now talking to cua/a ... -->
</spawn>
<!-- again talking to cua/b ... -->
</spawn>
The pseudo-tty interface is imperfect; in particular
shells complain about not being able to complete certain
ioctl calls. A the spawn manpage connection cannot be
forced to type a password to a shell, because
most password routines try to invoke /dev/tty
directly. Since Babble's tty interface dissociates
from the user's controlling terminal, this will hang.
The solution (a.k.a. dirty hack) to interacting with real shells and typing passwords in Babble scripts is to `launder' the session so that passwords are being typed to a remote shell rather than a local one, e.g., by invoking a remote ssh session and then interacting with it instead of a local one. This outsmarts the shell's builtin protection against such dirty hacks (unfortunately).
The the put manpage tag instructs Babble to interact with the 'current' device by sending data to the device. This corresponds to the innermost the spawn manpage containing the the put manpage, or any device specified on the command line if there is no enveloping the spawn manpage. A the put manpage tag may itself contain only text, named the var manpage's, and the special tags the noecho manpage and the stars manpage. Specifing any other markup inside a the put manpage constitutes a markup error.
During the the put manpage all the var manpage tags are expanded into the values of the corresponding named variants. Unless otherwise documented, a the put manpage tag implies a matching the get manpage for the same data. Exceptions are noted with the the noecho manpage and the stars manpage tags, which can only appear inside a the put manpage.
The the noecho manpage tag occurs only inside the put manpage tags and denotes that the text to be the put manpage should not be matched in full duplex. This is useful when the text being typed won't be echoed, e.g., when typing passwords. For example, in the fragment:
<get>Username:\s</get>
<put>foo<noecho>\n</noecho></put>
<get>Password:\s</get>
<put><noecho>bar\n</noecho></put>
\n expands to \r\n so we don't match it to avoid confusing the parser.
The password bar is not echoed so we place the whole password
inside the the noecho manpage tag.
The the stars manpage tag occurs only inside the put manpage tags and is a very special case. Some devices echo stars ('*') for each character of a password. Placing a password inside a the stars manpage tag causes Babble to match the appropriate number of stars instead.
The the get manpage tag instructs Babble to interact with the 'current'
device by receiving data from the device. the get manpage can contain
only text and the var manpage elements, either named or unnamed.
All the var manpage's contained in the get manpage's must have a pattern
attribute defined that matches appropriate values for the the var manpage.
the get manpage can fail if the pattern it seeks does not occur in
the the timeout manpage value currently in effect. The failure of a the get manpage
outside special contexts such as the branch manpage and the while manpage causes an
immediate execution halt. Inside these contexts, failure instead
modifies context behavior.
If an the var manpage inside a the get manpage has a name, the value matched is bound into the current the spawn manpage context as created by the repeat manpage, the while manpage, or the case manpage subscoping. If there is no the repeat manpage, the while manpage, or the case manpage in effect at the time of a the get manpage, all bindings occur within the immediately enclosing the bind manpage. Within a the repeat manpage or the while manpage, a the get manpage binds and named the var manpage's into the current instance rather than the global spawn scope. Any the case manpage in effect likewise causes a the get manpage to bind into the case instance.
The the var manpage tag indicates variational structure within both input and output of a stream. It is interpreted differently when it occurs in input than when it occurs in a output.
Inside a the put manpage tag, a the var manpage must have a name that corresponds
to that of a configuration parameter in the current binding scope.
The corresponding configuration parameter is substituted for the
the var manpage in place, in the output.
Inside a the get manpage tag, a the var manpage must have a pattern.
This is a regular expression in the Perl sense, except that
the magic characters ( and ) have been disabled.
Any the var manpage that also has a name will cause binding
of the matched value to that name in the appropriate binding
scope, which is the innermost the case manpage, the repeat manpage, the while manpage,
or the bind manpage scope containing the the var manpage.
In executing the regular expression, each the var manpage
pattern is placed inside ( and ) in order to
parse values for named the var manpage's.
the var manpage matching is different from Perl matching in one very important sense. It is opportunistic in the sense that the first substring that matches will cause a return from the the get manpage. This means that it is very important to delimit the get manpage the var manpage's with characters that indicate the end of a match. The wrong way to parse a name out of a file is this;
<get>
<var name='joe' pattern='[a-zA-Z]'>
</get>
This will match and bind the first character it sees. One must in some way delimit the name, e.g., with a trailing space:
<get>
<var name='joe' pattern='[a-zA-Z]'>
\s
</get>
This matches the first word with a space at the end that is not part of the word.
A (silly) current implementation restriction is that there can be at most 16 the var manpage's inside any one the get manpage. This will be fixed when I get time.
=head2 junction
A junction allows one to specify different possible
execution paths within a the brook manpage. A junction
can contain only the branch manpage's representing possible
execution paths.
A the branch manpage represents one execution path for a junction
and is only meaningful in that context. the branch manpage content
must include a the get manpage as the first tag, after which
the branch can be a free mix of any sequence of tags.
A branch is only invoked if its the get manpage is the first match
to the input from the device in context.
A the repeat manpage allows one to iterate over a group of instances
that require similar handling. It requires one parameter,
instances, which should be the name of a variant that
in the current binding context is a the repeat manpage variant.
For example, given the configuration (binding context):
<config>
<repeat name='cool'>
<instance>
<var name='joe'>Snoopy</var>
</instance>
<instance>
<var name='joe'>Kennedy</var>
</instance>
</repeat>
</config>
and the repeat markup
<repeat instances='cool'>
...
</repeat>
the statements inside the repeat markup will be repeated twice,
once with joe bound to Snoopy and once with joe bound
to Kennedy. A the repeat manpage invokes a new binding context
within which the get manpage statements bind values to the repeat
context instead of any already open outer binding contexts.
Within each of these repeats, any discovered variants will be
filed within the appropriate instance in effect at the time
of the the get manpage.
The the while manpage statement allows one to inductively discover a
set of instances of a thing. It can be used, e.g., to capture
information on individual entities described in a listing command.
It has one required attribute instances, which is the name
of a set of instances to discover. Any current value of that
set of instances will be lost after the the while manpage.
The the while manpage executes by repeatedly running its content
until a contained the get manpage fails or a the break manpage is encountered.
At this point execution continues after the the while manpage.
Thus failure of a the get manpage inside a the while manpage is not an
execution error. See the TUTORIAL for a realistic
example of the the while manpage statement.
The the break manpage statement is used as a controlled failure. When it is encountered inside a the while manpage it causes the the while manpage to end. At any other time, it causes the whole the brook manpage to end.
The the converge manpage statement allows one to craft convergent
processes that edit instances to match a master list.
the converge manpage must have two attributes key and instances.
instances must be a set of instances stored in the configuration
as a the repeat manpage element. key is the name of a variant that is
a unique identifier of each instance.
A the converge manpage markup must contain exactly five other
markups the list manpage, the read manpage, the add manpage, the remove manpage, and the modify manpage (in any order).
Each of these describes how to perform one part of the convergent process.
the list manpage creates a the repeat manpage block of instances having unique
key's, while the others act on particular instances in the
instances collection with names in instance scope.
For example:
<converge key='user' instances='people'>
<list> ... </list>
<read> ... </read>
<add> ... </add>
<remove> ... </remove>
<modify> ... </modify>
</converge>
The the list manpage markup must be contained within a the converge manpage
markup and is not meaningful in any other scope.
It should contain a the while manpage markup that generates a list of instances
with unique keys as documented in the key tag of the
containing the converge manpage.
The the read manpage markup must be contained within a the converge manpage
markup and is not meaningful in any other scope.
It should contain code that, given a key named
as described in the containing the converge manpage, reads all configuration
data for that instance. In the the converge manpage example above,
the read manpage should read all data corresponding to one user,
where user is predefined to be one real user.
The the add manpage markup must be contained within a the converge manpage
markup and is not meaningful in any other scope.
It should contain code that, given configuration data
for an instances described in the instances parameter
of the containing the converge manpage, creates a new real corresponding
instance on the device. In the the converge manpage example above,
the add manpage should add all data corresponding to one user
from an instance within the the repeat manpage context named people.
where user is predefined to be one real user.
The the remove manpage markup must be contained within a the converge manpage
markup and is not meaningful in any other scope.
It should contain code that, given a key named
as described in the containing the converge manpage, removes that
instance from the device. In the the converge manpage example above,
the read manpage should remove one user, where user is predefined
to be one real user.
The the modify manpage markup must be contained within a the converge manpage
markup and is not meaningful in any other scope.
It should contain code that, given configuration data
for an instances described in the instances parameter
of the containing the converge manpage, modifies the real pre-existing instance
on the device to match data in the instance. In the the converge manpage example above,
the modify manpage should modify all data corresponding to one user
from an instance within the the repeat manpage context named people.
where user is predefined to be one real user.
The the perl manpage markup allows one to embed simple Perl code that
acts on variant data and whose effects can be imported
back into Babble for later use. It requires two
attributes import and export with the same format.
These are lists of variants to import and export, together
with the structure of each variant.
Each import or export list has the same structure.
It is a comma-separated list of variant names with
structural declarations following each name.
The default structure is 'string'. A the case manpage structure
is indicated via { ... } where the { ... } can optionally
contain structures itself to describe structures of subcases.
A the repeat manpage structure is indicated via [ ... ] where the [ ... ]
can optionally contain the structure accorded to each
element of the repeating structure. Thus the structure declaration:
import='foo,bar[],cat{},dog[a,b],horse{a,b[hoodoo]}'
grabs a string variant named foo, the repeat manpage's named bar and dog,
cases named cat, and horse, where the substructure
of dog and horse is recursively defined.
In particular, the case horse has a variant that is a repeat
containing a string variant named hoodoo.
This declaration is utilized in two ways.
Before the contained Perl is invoked,
the import list is checked against actual variant
types in the active binding scope.
If types do not match for declared subparts, the
run is aborted. The types of undeclared subparts
aren't checked; e.g., the the repeat manpage variant dog
above could have another attribute george whose
type could be anything at all.
If types match the declarations, then variables in the
declaration are imported directly into a Perl module
Sandbox that will contain the execution. Variants
are mapped to scalars whose structure reflects type. Strings
are simple scalars; the case manpages map to references to associative arrays,
and the repeat manpage's map to references to regular arrays.
For example the import type above creates
In particular, we know that $horse->{'b'} is an array reference with length scalar(@{$horse->{'b'}}) and, for each element $i of that array, $horse->{'b'}->[$i]->{'hoodoo'} should be a string!
The the timeout manpage markup controls the length of timeout before
a the get manpage fails. It must have one attribute seconds representing
the number of seconds to wait. For example,
<timeout seconds=10/>
(the default) says to wait 10 seconds after the last
received input before declaring a the get manpage failure.
The dump markup causes a dump of a snapshot of variant space.
All binding contexts are flattened into one inclusive context.
It requires a file attribute describing where to dump
the information. The markup
<dump file='/tmp/foo'/>
dumps the whole binding context to the given file.
The following markups can be used in configuration files to declare kinds of variants. These are different from, but related to, the STREAM MARKUPS that comprise Babble's configuration files.
The the config manpage markup must contain all config tags. This spans each configuration file.
The the var manpage markup declares a string value.
It must have a name attribute. The value is contained
within the content of the the var manpage. Only string content may appear there.
For example, the markup
<var name='mood'>Happy</var>
declares a variant mood with value Happy.
The the case manpage markup declares a special case. It can contain the var manpage, the case manpage, or the repeat manpage markups as content. It cannot contain unmarked string content. For example, the markup
<case name='joe'>
<var name='material'>silicon</var>
<var name='pseudonym'>D'Artignan</var>
</case>
creates a special case with two contained variants,
material and pseudonym.
The the repeat manpage tag indicates that its content is a set of instances of a single entity to be processed as a batch. It must have a name. All other attributes are ignored. Its contents must be the instance manpages that describe different versions of the entity. For example:
<repeat name='tigers'>
<instance>
<var name='first'>Hobbes</var>
</instance>
<instance>
<var name='first'>Leo</var>
</instance>
</repeat>
describes a variant tigers that contains two entities, whose first names
are Hobbes and Leo, respectively.
The the instance manpage tag only occurs inside a the repeat manpage and must comprise its only content. It has no name but can contain a free mix of named forms such as the var manpage's, the case manpage's, and the repeat manpage's.
Babble is a dynamically scoped environment. This means that in a given the put manpage context, variants are searched for from inner to outer scopes, where the first match wins. This may seem confusing until one realizes that it allows very terse variant specifications involving defaults.
For example, one might note that there is a lot of repetition in the configuration:
<config>
<repeat name='people'>
<instance>
<var name='username'>cat</var>
<var name='password'>dog</var>
<var name='logins'>1</var>
<var name='direct'>1-11</var>
<var name='listen'>1-11</var>
<var name='buffer'>Y</var>
<var name='clear'>Y</var>
</instance>
<instance>
<var name='username'>foo</var>
<var name='password'>bar</var>
<var name='logins'>1</var>
<var name='direct'>1-11</var>
<var name='listen'>1-11</var>
<var name='buffer'>Y</var>
<var name='clear'>Y</var>
</instance>
</repeat>
</config>
We can exploit dynamic scoping to define defaults for repeated the var manpage's:
<config>
<var name='logins'>1</var>
<var name='direct'>1-11</var>
<var name='listen'>1-11</var>
<var name='buffer'>Y</var>
<var name='clear'>Y</var>
<repeat name='people'>
<instance>
<var name='username'>cat</var>
<var name='password'>dog</var>
</instance>
<instance>
<var name='username'>foo</var>
<var name='password'>bar</var>
</instance>
</repeat>
</config>
In the inner context of the repeat, username is
defined for each variant. However, logins is only
defined in an upper context containing the the repeat manpage.
The effect is that the initial the var manpage's become defaults
for the lower context. If one wishes, one can still override
a default:
<config>
<var name='logins'>1</var>
<var name='direct'>1-11</var>
<var name='listen'>1-11</var>
<var name='buffer'>Y</var>
<var name='clear'>Y</var>
<repeat name='people'>
<instance>
<var name='username'>cat</var>
<var name='password'>dog</var>
</instance>
<instance>
<var name='logins'>2</var>
<var name='username'>foo</var>
<var name='password'>bar</var>
</instance>
</repeat>
</config>
This says that user foo gets 2 logins while everyone
else gets 1.
In Babble it is very poor style to change the type of a variant from one type to another, e.g., change a string to a the case manpage or the repeat manpage. Future versions of Babble will detect and prohibit such acts. But there is a much more subtle version of contravariance of which one should be wary.
Babble scripts depend upon a homogeneity property in the repeat manpage contexts, that a named variant that is available in one instance is available in all instances. This can be assured either by specific data or defaults. It is particularly bad style to create instances with parameters that aren't available in others, e.g.,
<repeat name='hosed'>
<instance>
<var name='crash'>me</var>
</instance>
<instance>
<var name='with'>this</var>
</instance>
</repeat>
Future versions of Babble will detect and prohibit this more subtle contravariance. Of course, one can always repair the damage by declaring defaults:
<var name='crash'>averted</var>
<var name='with'>defaults</var>
<repeat name='hosed'>
<instance>
<var name='crash'>me</var>
</instance>
<instance>
<var name='with'>this</var>
</instance>
</repeat>
Babble has a number of command-line options that become available to any script created with the #! convention described above. These options control logging, tracing, debugging, and other behavior.
Babble executes in two modes. In its primary mode, Babble implements a documented stream of interactions. In another mode, it records a stream for future execution. The recording mode is not yet fully functional; use at your own risk.
In the first mode, Babble is executed either from within #! or from the command line with an explicit input file. If, e.g., the first line of your script 'boo' is
#! /usr/bin/perl /usr/local/bin/babble
then the invocation
boo --trace=9
is equivalent with the invocation
/usr/local/bin/babble boo --trace=9
so that the babble parameters are
boo --trace=9
In most UNIX's, one cannot place options into the #! line; these must be placed on subsequent command lines.
In the first mode, Babble is executed with an immediate file name of a script to parse and run. If this script is invoked with #!, the options are as follows:
Set debug level (0-9) (default 0). This is only really useful when trying to debug or extend Babble itself.
main when beginning execution.
This can be used to create one file with several
functions.
Stream recording allows one to use Babble to record interactions between a human and a device for future playback. To do this one invokes Babble as:
babble -converse="command to script" <options>
In this mode, babble tries to capture an interactive session in a runnable form. Options may include:
In conversational mode, Babble tries to open the device and allow you to type its responses. It records these in XML format so that the result is runnable as a Babble script.
Warning: this has plenty of bugs. First, Babble keeps your terminal in line-buffered mode, so that it won't have to cope with line-editing characters in its logs. This breaks some device contacts.
Babble is an unbelievably complex Perl script and the list of bugs and deficiencies (todo's) for Babble is almost as long as its documentation. Here's a short list of the major problems upon which I am working at this time:
union of all variants instantiated in the
junction is passed into the flow analysis, making for a rather
naive result.
There are an almost unlimited number of things to do to make Babble better.
tab markup that
skips to a particular column of input.
Alva L. Couch, couch@eecs.tufts.edu,
http://www.eecs.tufts.edu/~couch,
Copyright 2000 by Alva L. Couch.
Dec 1, 2000