In the following,
Thursday, February 24, 2011
1:36 PM
In the following, 
I am going to make significant use of schema transformation. 
Figuring out what I am doing is going to depend upon 
tracking how schemas change and 
what the changes mean.
Let's review the basic schema transformations: 
If I have a schema, e.g., 
input: {a:int, b:chararray}
then 
output = FOREACH input GENERATE a AS c, a+1 as d;
produces the schema
output: {c:int, d:int}
And the statement
g = GROUP input by a; 
produces the schema
g: {group:int, input:{a:int, b:chararray}}
If I want, I can remove a part of the inner bag tuples by subranging: 
g2 = FOREACH g GENERATE group as a, input.b as bs; 
which will give the schema
  g2: {a:int, bs:{b: chararray}}
If I have two schemas: 
  in1: {a:int, b:chararray}
  in2: {c:int, d:chararray}
then the inner join
j = JOIN in1 BY a, in2 BY c; 
produces the schema
j: {in1::a :int, in1::b :chararray, in2::c :int, in2::d :chararray}; 
Note: {} implies {()} because a bag can only contain tuples. 
Projecting those to scalars creates scalar tuples, e.g., 
{(a, {(b), (d)})}