Determining valid login sites
Thursday, February 24, 2011
3:57 PM
Now our schema is 
vulnerable3: {time:long, user:chararray, status:chararray, subnet:chararray}
Let's filter out subnets from which the user regularly logs in correctly. Let's make a whitelist.
correct = FILTER vulnerable3 BY status=='success'; 
correct2 = FOREACH correct GENERATE user, subnet; 
good = GROUP correct2 BY (user,subnet); 
Now schema is
good: {group: (user:chararray, subnet:chararray), correct2: {user:chararray, subnet:chararray}}
Let's count the correct logins:
whitelist = FOREACH good GENERATE group.user,group.subnet,COUNT(correct2) as count; 
whitelist2 = FILTER whitelist by counter>100;
And throw them out of the log!