Not the end, but the beginning

• We've described the entire language in most of its complexity.
• Other people, meanwhile, have been extending it in thousands of ways.
• Real source of Perl popularity: "it was there" when many people had problems to solve.
• Not so much a matter of elegance, as of presence and ubiquity.

Extending perl

• Write a module.
• Document it.
• Post to CPAN (Comprehensive Perl Archive Network)
• Maintain it!

Plain Old Documentation

• very simple hack for "literate programming"
• part of the Perl script itself.
• Processed to make manpages, HTML documentation, etc.
• Interface: perldoc

Perldoc usage

• perldoc ModName : print documentation on module named ModName.
• perldoc -f func : print documentation on builtin function func.
• perldoc -m ModName: print source code for the appropriate module!

Basic POD Directives

• =pod begin a POD block (don't parse as Perl)
• =cut end a POD block (start parsing Perl again)
• =head1 More about foo a heading
• =head2 Barring nothing a subheading
• =over 5 indent for itemized list
• =item foo a particular item in the list
• =back end an indented list.

POD text markups

• I<text> italicized text.
• B<text> boldface text.
• C<text> literal code.
• S<text> text with non-breaking spaces.
• L<page> link to another manpage (see book).

POD caveats

• Translators sometimes too clever for themselves; make automatic links to inappropriate things such as the numbers of lists.
• POD slows down compilation, not runtime.
• Can separate POD into .pod files corresponding to modules to completely avoid these files during compilation.

CPAN

• Comprehensive Perl Archive Network
• thousands of modules contributed by users.
• module quality varies greatly.
• millions of bugs

Kinds of modules:

• Language extensions.
• Glue between Perl and external software written in other languages.

Brief map of CPAN

• Core modules
• Development support
• Operating system interfaces
• Networking and inter-process communications
• Data types and utilities
• Database interfaces
• User interfaces
• Interfaces to or emulations of other programming languages
• File names, systems, and Locking (see also file handles)
• String processing, text processing, parsing, and searching
• Option, argument, parameter and configuration file processing
• Internationalization and locale
• Authentication, security, and encryption
• World Wide Web, HTML, HTTP, CGI, MIME
• Server and daemon utilities
• Archiving, compression, and conversion
• Images, pixmap and bitmap manipulation, drawing, and graphing
• Mail and USENET news
• Control flow utilities (callbacks and exceptions etc)
• File handle, directory handle, and input/output stream utilities
• Microsoft Windows modules
• Miscellaneous modules
• Interface modules to commercial software
• Bundles of cooperating modules.

CPAN Caveats

• Modules can change without notice, breaking your code when they're upgraded.
• Modules can fail to change without notice, breaking their own code when Perl is upgraded.
• In general, can only count on consistent behavior from a module for one version of Perl and one version of the module.
• So, in practice, one either uses old code or is constantly upgrading code to fit the modules as they are upgraded.

Brief tour of CPAN Web utilities

• how to write a CGI script (CGI.pm)
• how to read data from a web server (LWP)
• how to parse and write HTML (HTML)
• how to parse and write XML (XML)

Writing CGI's

• Consider the code:

  contents of perl_exten/debug.cgi...
  #! /var/local/couch/bin/perl
  use CGI;
  use CGI::Carp 'fatalsToBrowser';        # Send fatal errors to browser
  $cgi = new CGI; 			# read CGI parameters 
  @params = $cgi->param; 
  $params = join(',',@params); 
  print $cgi->header; 
  print $cgi->start_html; 
  print $cgi->h1("Values of CGI names:"); 
  my $txt = ''; 
  foreach $param (@params) { 
      $val = $cgi->param($param); 
      $txt .= $cgi->li("value of $param is $val"); 
  } 
  print $cgi->ul($txt); 
  print $cgi->end_html; 
  print "\n"; 
  ...end of perl_exten/debug.cgi
  

• Try it: perl_exten/debug.cgi

Creating HTML

• Consider the code:

  contents of perl_exten/html01.cgi...
  #! /var/local/couch/bin/perl
  use HTML::Element;
  print "Content-type: text/html\n\n";
  
  ## make an HTML document, that's empty
  $html = new HTML::Element('html');
  
  # make a body that's empty.
  $body = new HTML::Element('body');
  
  # make a paragraph that's empty
  $pg1 = new HTML::Element('p');
  
  # Fill the paragraph with content. 
  $pg1->push_content("This is an HTML paragraph. ");
  $anchor = new HTML::Element('a','href'=>'http://www.cs.tufts.edu');
  $anchor->push_content("Computer Science");
  $pg1->push_content("Here's a link to ");
  $pg1->push_content($anchor);
  $pg1->push_content(".");
  
  # make a second paragraph
  $pg2 = new HTML::Element('p');
  $pg2->push_content("A second paragraph. ");
  
  # put paragraphs into the body. 
  $body->push_content($pg1);
  $body->push_content($pg2);
  
  # put the body into the HTML document. 
  $html->push_content($body);
  
  # print the whole thing. 
  print $html->as_HTML;
  ...end of perl_exten/html01.cgi
  

• Try it: perl_exten/html01.cgi
• Let's look inside:

  contents of perl_exten/html02.pl...
  #! /var/local/couch/bin/perl
  use HTML::Element;
  use Data::Dumper; 
  
  ## make an HTML document, that's empty
  $html = new HTML::Element('html');
  
  # make a body that's empty.
  $body = new HTML::Element('body');
  
  # make a paragraph that's empty
  $pg1 = new HTML::Element('p');
  
  # Fill the paragraph with content. 
  $pg1->push_content("This is an HTML paragraph. ");
  $anchor = new HTML::Element('a','href'=>'http://www.cs.tufts.edu');
  $anchor->push_content("Computer Science");
  $pg1->push_content("Here's a link to ");
  $pg1->push_content($anchor);
  $pg1->push_content(".");
  
  # make a second paragraph
  $pg2 = new HTML::Element('p');
  $pg2->push_content("A second paragraph. ");
  
  # put paragraphs into the body. 
  $body->push_content($pg1);
  $body->push_content($pg2);
  
  # put the body into the HTML document. 
  $html->push_content($body);
  
  # print the whole thing. 
  my $d = new Data::Dumper([$html]); 
  $d->Indent(1); 
  print $d->Dump(); 
  ...end of perl_exten/html02.pl
  

• This prints:

  contents of perl_exten/html02.pl.out...
  $VAR1 = bless( {
    '_content' => [
      bless( {
        '_parent' => $VAR1,
        '_content' => [
          bless( {
            '_parent' => $VAR1->{'_content'}[0],
            '_content' => [
              'This is an HTML paragraph. Here\'s a link to ',
              bless( {
                '_parent' => $VAR1->{'_content'}[0]{'_content'}[0],
                '_content' => [
                  'Computer Science'
                ],
                'href' => 'http://www.cs.tufts.edu',
                '_tag' => 'a'
              }, 'HTML::Element' ),
              '.'
            ],
            '_tag' => 'p'
          }, 'HTML::Element' ),
          bless( {
            '_parent' => $VAR1->{'_content'}[0],
            '_content' => [
              'A second paragraph. '
            ],
            '_tag' => 'p'
          }, 'HTML::Element' )
        ],
        '_tag' => 'body'
      }, 'HTML::Element' )
    ],
    '_pos' => undef,
    '_tag' => 'html'
  }, 'HTML::Element' );
  ...end of perl_exten/html02.pl.out
  

• HTML::Element is a nested structure of elements, each of which is similar in structure to the parent.

Parsing HTML

• Suppose we have:

  contents of perl_exten/junk.html...
  <html>
  <head><title>all about foo</title></head> 
  <body> 
  <h1>All about foo</h1> 
  <p>There's not much I can <em>tell</em> you.</p> 
  </body>
  </html> 
  ...end of perl_exten/junk.html
  

• And we parse it with:

  contents of perl_exten/html03.pl...
  #! /var/local/couch/bin/perl
  use HTML::Element;
  use HTML::TreeBuilder; 
  use Data::Dumper; 
  
  # parse a file of HTML 
  my $html = HTML::TreeBuilder->new; 
  $html->parse_file('junk.html');
  
  # print the whole thing. 
  my $d = new Data::Dumper([$html]); 
  $d->Indent(1); 
  print $d->Dump(); 
  ...end of perl_exten/html03.pl
  

• This prints:

  contents of perl_exten/html03.pl.out...
  $VAR1 = bless( {
    '_done' => 1,
    '_implicit_tags' => 1,
    '_tighten' => 1,
    '_head' => bless( {
      '_parent' => $VAR1,
      '_content' => [
        bless( {
          '_parent' => $VAR1->{'_head'},
          '_content' => [
            'all about foo'
          ],
          '_tag' => 'title'
        }, 'HTML::Element' )
      ],
      '_tag' => 'head'
    }, 'HTML::Element' ),
    '_store_comments' => 0,
    '_content' => [
      $VAR1->{'_head'},
      bless( {
        '_parent' => $VAR1,
        '_content' => [
          bless( {
            '_parent' => $VAR1->{'_content'}[1],
            '_content' => [
              'All about foo'
            ],
            '_tag' => 'h1'
          }, 'HTML::Element' ),
          bless( {
            '_parent' => $VAR1->{'_content'}[1],
            '_content' => [
              'There\'s not much I can ',
              bless( {
                '_parent' => $VAR1->{'_content'}[1]{'_content'}[1],
                '_content' => [
                  'tell'
                ],
                '_tag' => 'em'
              }, 'HTML::Element' ),
              ' you.'
            ],
            '_tag' => 'p'
          }, 'HTML::Element' )
        ],
        '_tag' => 'body'
      }, 'HTML::Element' )
    ],
    '_body' => $VAR1->{'_content'}[1],
    '_ignore_unknown' => 1,
    '_pos' => undef,
    '_ignore_text' => 0,
    '_no_space_compacting' => 0,
    '_implicit_body_p_tag' => 0,
    '_warn' => 0,
    '_p_strict' => 0,
    '_hparser_xs_state' => \136908632,
    '_store_declarations' => 0,
    '_element_count' => 3,
    '_tag' => 'html',
    '_store_pis' => 0,
    '_element_class' => 'HTML::Element'
  }, 'HTML::TreeBuilder' );
  ...end of perl_exten/html03.pl.out
  

• This converts the HTML into a parsable tree.

Reading HTML from other sites

• Consider:

  contents of perl_exten/LWP01.cgi...
  #! /var/local/couch/bin/perl
  
  use CGI::Carp "fatalsToBrowser"; 
  
  use LWP;
  require HTTP::Request;
  require HTTP::Response;
  
  print "Content-type: text/html\n\n"; 
  
  $ua = LWP::UserAgent->new;
  $request = HTTP::Request->new(GET => 'http://www.cs.tufts.edu/');
  $response = $ua->request($request);
  if ($response->is_success) {
      print $response->content;
  } else {
      print $response->error_as_HTML;
  }
  
  ...end of perl_exten/LWP01.cgi
  

• Try it: perl_exten/LWP01.cgi
• Subtle: some images are broken. This is because the link seems to be relative to our page, but its images are specified relative to the original source.

Correcting URI's

• Consider

  contents of perl_exten/uri01.pl...
  #! /var/local/couch/bin/perl 
  
  use URI; 
  $u =  URI->new("bar/"); 
  print "u=". $u->abs("http://www.cs.tufts.edu/~couch/foo/") . "\n"; 
  ...end of perl_exten/uri01.pl
  

• This prints:

  contents of perl_exten/uri01.pl.out...
  u=http://www.cs.tufts.edu/~couch/foo/bar/
  ...end of perl_exten/uri01.pl.out
  

Putting it all together

• Consider:

  contents of perl_exten/LWP02.cgi...
  #! /var/local/couch/bin/perl
  
  use CGI; 
  use CGI::Carp "fatalsToBrowser"; 
  
  use HTML::Element;
  use HTML::TreeBuilder; 
  use URI; 
  
  use LWP;
  require HTTP::Request;
  require HTTP::Response;
  
  $cgi = new CGI; 
  $url = $cgi->param('url'); 
  $url='http://www.cs.tufts.edu/' if $url eq ''; 
  print $cgi->header; 
  
  $ua = LWP::UserAgent->new;
  $request = HTTP::Request->new(GET => $url);
  $response = $ua->request($request);
  
  if ($response->is_success) {
      my $doc = new HTML::TreeBuilder;
      $doc->parse($response->content)
      &edit($doc->content,$url); 
      print $doc->as_HTML; 
  } else {
      print $response->error_as_HTML;
  }
  
  sub edit { 
      my $content = shift; 
      my $url = shift; 
      my $i; 
      for ($i=0; $i<@$content; $i++) { 
          if (ref ($content->[$i]) eq 'HTML::Element') { 
  	   if ($content->[$i]->tag eq 'img') { 
  	        my $src = $content->[$i]->attr('src'); 
  	        my $suri = URI->new($src); 
  	        my $duri = $suri->abs($url); 
  		$content->[$i]->attr('src',$duri); 
  	   } 
  	   if ($content->[$i]->tag eq 'a') { 
  	        my $href = $content->[$i]->attr('href'); 
  	        my $suri = URI->new($href); 
  	        my $duri = $suri->abs($url); 
  		$content->[$i]->attr('href',$duri); 
  	   } 
  	   &edit($content->[$i]->content,$url); 
  	}
      } 
  } 
  ...end of perl_exten/LWP02.cgi
  

• Try it: perl_exten/LWP02.cgi
• This fixes broken links by making them absolute.
• Consider:

  contents of perl_exten/LWP04.cgi...
  #! /var/local/couch/bin/perl
  
  use CGI; 
  use CGI::Carp "fatalsToBrowser"; 
  
  use HTML::Element;
  use HTML::TreeBuilder; 
  use URI; 
  use LWP;
  require HTTP::Request;
  require HTTP::Response;
  
  $cgi = new CGI; 
  $url = $cgi->param('url'); 
  $url='http://www.cs.tufts.edu/' if $url eq ''; 
  print $cgi->header; 
  
  $ua = LWP::UserAgent->new;
  $request = HTTP::Request->new(GET => $url);
  $response = $ua->request($request);
  
  if ($response->is_success) {
      my $doc = new HTML::TreeBuilder;
      $doc->parse($response->content); 
      @urls = &mine($doc->content,$url); 
      my $content = ''; 
      foreach my $u (@urls) { $content .= $cgi->li($cgi->a({'href'=>$u},$u)); } 
      print $cgi->start_html; 
      print $cgi->h1("links"),$cgi->ol($content); 
      print $cgi->end_html; 
  } else {
      print $response->error_as_HTML;
  }
  
  # return an array of all links in a current page 
  sub mine { 
      my $content = shift; 
      my $url = shift; 
      my @out = (); 
      my $i; 
      for ($i=0; $i<@$content; $i++) { 
          if (ref ($content->[$i]) eq 'HTML::Element') { 
  	   if ($content->[$i]->tag eq 'a') { 
  	        my $href = $content->[$i]->attr('href'); 
  	        my $suri = URI->new($href); 
  	        my $duri = $suri->abs($url); 
  	        push(@out,$duri); 
  	   } 
  	   push(@out,&mine($content->[$i]->content,$url)); 
  	}
      } 
      return @out; 
  } 
  ...end of perl_exten/LWP04.cgi
  

• Try it: perl_exten/LWP04.cgi
• This mines URI's out of a page in preparation for web crawling.