108.bz » Perl http://www.108.bz Wandering futilities... Wed, 08 Sep 2010 13:45:53 +0000 en hourly 1 http://wordpress.org/?v=3.0 Dumping streaming media in 25 lines of Perl http://www.108.bz/posts/it/dumping-streaming-media-in-25-lines-of-perl/ http://www.108.bz/posts/it/dumping-streaming-media-in-25-lines-of-perl/#comments Thu, 13 May 2010 10:11:23 +0000 Giuliano http://www.108.bz/?p=459 Analysing TCP based protocols often means dealing with TCP sessions (also called streams or flows).
A TCP connection, from an application point of view, is much like a bidirectional file descriptor through which ordered data can be read or written. “On the wire” though, data is not ordered at all. It is split into packets, possibly shuffled and mixed with other traffic. You can capture packets using a sniffer, but to make any sense of them you also need an analyzer tool able to do the reordering/reassembling job. Wireshark, for instance, doubles as a sniffer and an analyzer, backed up by the ubiquitous libpcap.

Imagine having dumped/sniffed 1GB worth of traffic. We would like to pinpoint a single TCP session, isolating it from the rest. Here’s how we could proceed:

  • Identify the source/destination addresses and source/destination ports we’re interested in. Then throw away any packet that doesn’t match this tuple. That’s what Wireshark basically does when you select a packet, right click and hit “Follow TCP Stream”. If the same tuple doesn’t get reused for another, unrelated, session, this method works just fine1.
  • Reorder/reassemble packets.
  • Extract packets’ payload.
  • Present the payload in a way that makes sense. That depends on the L7 protocol. HTTP without keep-alive is strictly request/response: print what the client sent to the server (outbound traffic) before and then what the server answered (inbound traffic). Other protocols may behave differently and you may choose to separate inbound traffic from outbound, or rely on timing to correctly present the dialogue between peers.

Besides Wireshark, there are tools that do just that and can also be automated. See TShark or tcpflow.

What if you want to script everything and build your own TCP analyzer? Perl’s module Net::Analysis is surprisingly convenient for the task. It does the dirty job I described above and presents your code with ready to be processed TCP sessions.

Practical goal: saving MP3 files streamed by Grooveshark. Disclaimer: I’m by no means pushing anyone to illegally download stuff, this is just a working, sensible, instructional example that uses a song freely available anyway (by Revolution Void, check them out here, they’re great).

GroovesharkListener.pm extends Net::Analysis::Listener::HTTP. It sniffs all the traffic from/to port 80 and, as soon as he sees an HTTP response with a content-type of “audio”, dumps its content to file and quits. Simple as that.

Put the module some place where Perl can find it and then launch (as root):

# perl -MNet::Analysis -e main GroovesharkListener 'port 80'
(starting live capture)
/crossdomain.xml
text/xml
/service.php?addSongsToQueueExt
text/html; charset=UTF-8
/static/amazonart/m8c8c9f4291508bca130c1caac2bda75b.png
image/png
[...some more cruft...]
/stream.php
audio/mpeg
Dumping 8481224 bytes to groovesharkgyzBy.mp3 be patient...

# id3v2 -l groovesharkgyzBy.mp3
id3v1 tag info for groovesharkgyzBy.mp3:
Title  : Invisible Walls                 Artist: Revolution Void              
Album  : Increase the Dosage             Year: 2004, Genre: Other (12)
Comment: http://www.jamendo.com/         Track: 1

That’s it, just one more thing. Net::Analysis doesn’t allow you to select a specific network interface, it just picks up the first available one. I wrote a small patch to address this shortcoming, it adds a “device=” parameter that you can use this way:

# perl -MNet::Analysis -e main GroovesharkListener,device=wlan1 'port 80'

And here’s what GroovesharkListener.pm looks like:

# choose a song
# run (as root or via sudo):
#   perl -MNet::Analysis -e main GroovesharkListener 'port 80'
# click "play" and wait for the file to be dumped...
#                             -- Giuliano - http://www.108.bz
package Net::Analysis::Listener::GroovesharkListener;
use strict;
use base qw(Net::Analysis::Listener::HTTP);
use File::Temp;

sub http_transaction {
    my ($self, $args) = @_;
    my ($http_req) = $args->{req};
    my ($http_resp) = $args->{resp};

    print $http_req->uri(), "\n";
    my $content_type = $http_resp->header('Content-Type');
    print "$content_type\n";
    if ($content_type =~ /audio/i) {
        my $fh = new File::Temp(TEMPLATE => 'groovesharkXXXXX',
            SUFFIX   => '.mp3',
            UNLINK   => 0);
        print "Dumping ".length($http_resp->content)." bytes to ".$fh->filename." be patient...\n";
        print $fh $http_resp->content;
        exit;
    }
}

1;
  1. newer Wireshark(s) use the “tcp.stream eq x” primitive ↩
]]> http://www.108.bz/posts/it/dumping-streaming-media-in-25-lines-of-perl/feed/ 2 Abusing Backup Exec’s internal database http://www.108.bz/posts/it/abusing-backup-exec-internal-database/ http://www.108.bz/posts/it/abusing-backup-exec-internal-database/#comments Mon, 15 Mar 2010 23:18:35 +0000 Giuliano http://www.108.bz/?p=348 What’s the point of having data stored somewhere if you can’t access it and turn it into useful information? Of course the means to do so should be safe, supported, non destructive and flexible if not easy. But usually all you’re left with is some kind of “reporting” feature that necessarily doesn’t do exactly what you need, doesn’t output in a convenient format and so on.

But enough squabbling: in this article I’ll deal with Backup Exec’s internal database.
Here’s what I’m trying to do:

  • Look up all the “Duplicate” Jobs. Show when they started, how long they took to complete, the rate, …
  • For each one of them, try and find the relevant tapes.

I will use the generated report to know which media I should eject out of the library for safe storage. The report will also allow me to quickly and easily update the Excel worksheets where we keep track of how backup’s going.

Our BE database runs on Microsoft SQL Server Express. First thing to do is configure the instance to allow remote TCP/IP connections. Refer to this post, and KB914277.

Then I’m able to point SQL Server Management Studio at it1, and see how the BEDB database is organized.

The view named vwJobHistorySummary is the equivalent of what is seen in BE’s GUI, under Job Monitor → Job List → Job History. Easy enough to find out.

What’s not that immediate to guess is how Media IDs relate to Job IDs: skimming through the database tables doesn’t help… How could you reverse engineer BE GUI and discover what SQL queries it’s doing to carry out its job? In fact, there’s a way to “sniff” SQL queries while they’re running:

  • open up BE GUI and select (but don’t open) a completed Job in Job History.
  • run SQL Server Profiler.
  • create a New Trace.
  • Under the Event Selection tab, deselect everything except SQL:BatchStarting. This is not a particularly crowded database, hence no need for filters.
  • Double click on the previously selected Job; SQL Profiler should capture a query similar to:
SELECT * FROM dbo.vwJobHistory WHERE
JobHistoryID='8507cfa9-8417-44ae-88e6-9ac19a0333a9' ORDER BY [JHD.StartTime]
FOR XML AUTO

Looks like Job details are fetched as globs of XML data, perfect to throw our beloved Regular Expressions at.

You can find the script I made at the end of the post. The obligatory notes are:

  • By convention, in our scenario, Policies used to create Duplicate jobs bear a name ending with “-D”. I’m SELECting the last Job IDs with a similar pattern; change it according to your needs, for instance if you’re interested in all the tape directed Jobs (and not just the Duplicate ones).
  • Columns are as follow:
    • Job name. In case you wonder, “FSIWDTH” means: Full Saturday, Incremental Weekdays, Duplicate on Thursday.
    • Actual start timestamp.
    • End timestamp.
    • Elapsed time (seconds).
    • Total bytes written. No bytes written? I skip this Job.
    • Rate (MBytes/minute). Oddly, BE doesn’t seem to always get this value right.
  • The “convert( varchar(” stuff in the main query is needed to fetch dates in a non driver-dependent format (see FreeTDS FAQ).
  • Dates are stored in UTC timezone. I make sure of adding the local TZ offset before printing them out.

Example output:

./be_job_media_report.pl
Sel SERVER03-FSIWDTH-D;20100311 09:15;20100312 11:26;94239;602103130444;405
AK8008L3
AK8011L3
Sel SERVER07-FSIWDTH-D;20100311 09:00;20100311 09:15;921;5133161638;452
AK8011L3
Sel SERVER16-FSIWDWE-D;20100310 10:35;20100310 17:34;25155;5352;0
AK8011L3
Sel SERVER13-FSIWDWE-D;20100310 09:00;20100310 17:29;30572;230425324573;515
AK7140L1
AK8011L3

And the script itself:

#!/usr/bin/perl
# Giuliano - http://www.108.bz
use strict;
use DBI;
use List::Uniq qw(uniq);
use Time::Piece;

sub pretty_time($) {
    my $time;
    $time = Time::Piece->strptime(shift, "%Y-%m-%d %H:%M:%S"); # 2010-03-10 16:34:19
    $time += $time->localtime->tzoffset;
    return $time->strftime('%Y%m%d %H:%M');
}

sub print_last_jobids($$;$) {
    my ($dbh, $number, $jobname_like) = @_;

    my $q = <<EOQ;
SELECT TOP 20 JobHistoryID, JobName,
              convert( varchar(30), OriginalStartTime, 120),
              convert( varchar(30), ActualStartTime, 120),
              convert( varchar(30), EndTime, 120),
              ElapsedTimeSeconds,
              FinalJobStatus, FinalErrorCode, TotalDataSizeBytes, TotalRateMBMin
FROM vwJobHistorySummary
-- WHERE
-- Jobname LIKE '$jobname_like'
ORDER BY ActualStartTime DESC
EOQ
    $q =~ s/-- (WHERE)/$1/ if $jobname_like;
    $q =~ s/-- (Jobname LIKE)/$1/ if $jobname_like;
    my $sth = $dbh->prepare($q);
    $sth->execute();
    my $row;
    while ( $row = $sth->fetchrow_arrayref ) {
        if ($row->[8]) { # TotalDataSizeBytes > 0
            $row->[3] = pretty_time($row->[3]);
            $row->[4] = pretty_time($row->[4]);
            printf "%s;%.f\n", (join ';', @{$row}[1,3,4,5,8]), $row->[9];
        }
        print_media_by_jobhistoryid($dbh, $row->[0]);
    }
}

sub print_media_by_jobhistoryid($$) {
    my ($dbh, $jobid) = @_;

    my $sth = $dbh->prepare(<<EOQ
SELECT * FROM dbo.vwJobHistory where
JobHistoryID='$jobid' ORDER BY [JHD.StartTime]
FOR XML AUTO
EOQ
    );
    $sth->execute();
    my @media;
    my $row;
    while ( $row = $sth->fetchrow_arrayref ) {
        my $record = join ';', @$row;
        push @media, $1 if $record =~ /Data="(.*?)"/;
    }
    print +(join "\n", uniq({sort => 1},@media)), "\n" if @media;
}

### Main

my $dbh = DBI->connect('dbi:Sybase:server=bedbdatasource;database=BEDB','DOMAIN\username','password') or die;
print_last_jobids($dbh, 10, '%-D');

exit;
  1. No need to enable TCP/IP on the instance, if Management Studio is installed on BE server itself ↩
]]> http://www.108.bz/posts/it/abusing-backup-exec-internal-database/feed/ 1 Detecting malware using Windows Auditing events http://www.108.bz/posts/it/detecting-malware-using-windows-auditing-events/ http://www.108.bz/posts/it/detecting-malware-using-windows-auditing-events/#comments Sun, 28 Feb 2010 23:58:18 +0000 Giuliano http://www.108.bz/?p=314 This post1 explains how to use nmap and smb-check-vulns to scan a network in search of Conficker infected hosts. I thought that the whole Conficker case was over, but hopefully some of the measures I took to deal with it almost an year ago, will still be relevant to other kinds of malware. And, also, the method I’ll show you here differs from the nmap one in that the latter is active, whereas mine is passive. Actively probing an host for vulnerabilities could be very very much alike “exploiting” it as malware does, and have similar effects. For instance, a service/process could crash, making it not always advisable to run active scans on your servers subnet. Passive analysis, on the other hand, unobtrusively collects clues about who’s misbehaving.

During the Conficker/Downadup outburst, we observed that:

  • Antivirus wasn’t always able to detect/stop it.
  • The virus was copying files in known directories (C:\WINDOWS\SYSTEM32) on about to be infected machines.
  • Security patched hosts were still subject to the remote malicious file copying routine. The copy could either succeed or fail, depending on which permissions had the user that “runs” the virus. The copy in itself doesn’t pose any security concern. Even if no A/V is active on the destination host, but virus exploitable flaws have been patched, malware won’t be able to activate itself. Otherwise, the A/V would remove suspect files as soon as they are caught, without interfering with our detection purposes.

This behaviour makes it possible to use a “honeypot” approach. The detecting server can be any production host provided that it is security patched and A/V protected. You could, as we did, choose a Domain Controller and:

  • Run Administrative Tools → Domain Controller Security Policy

  • Modify the Audit Policy, enabling tracking of successful logon events and object access. By default the OS will only log failures, but that’s not enough.

  • Object Access is activated at a file/directory level. Open up the Properties of a directory you know is accessed by the virus, click on Security, then Advanced. The Auditing tab is what you’re interested in. Set things up so that any “Create File/Write Data” attempt of Type “Success” will be logged. The semantics about how auditing settings are propagated from parent to child works in the same way as NFTS permissions.

  • From this point on, you should monitor the honeypot server’s Security Event Log. I wrote a Perl script to do it for me. It works by selecting events with ID 560 and 540, extracting their text and printing just the needed info.

Let’s look at how it’s used (the only parameter is the hostname/address of the honeypot server):

C:\loganalysis>perl ddloganalysis.pl honeypot-srv.domain.lan > ddlog.txt

Skimming through the generated log, you’ll notice the files being dropped into C:\WINDOWS\system32 (or any directory you set up for auditing), the user that actually created them and, before (time-wise), from which address the user is coming.

17/03/2009 16.26.19   560 : C:\WINDOWS\system32\onevthx.vr (Administrator)
17/03/2009 16.26.18   540 :  (10.1.1.94 - Administrator)
17/03/2009 15.35.24   560 : C:\WINDOWS\system32\onevthx.vr (SpectrumLT)
17/03/2009 15.35.24   540 :  (10.6.3.6 - SpectrumLT)

We successfully used the script to pinpoint the rogue hosts. Deeming it useful, here it is:

#!perl

use strict;
use Win32::EventLog;
use POSIX qw ( strftime );

my @matches = (
    #'job$',   # useless, since scheduled tasks are always created by SYSTEM
    'system32',
    'eicar.com'
);

die "Usage:\n$0 servername" unless $ARGV[0];

my $ev=Win32::EventLog->new('Security', $ARGV[0])
    or die "Can't open EventLog\n";
my $recs;
$ev->GetNumber($recs)
    or die "Can't get number of EventLog records\n";
my $base;
$ev->GetOldest($base)
    or die "Can't get number of oldest EventLog record\n";

sub getts($) {
    return strftime '%d/%m/%Y %H.%M.%S', (localtime shift);
}

my @progress = ('-','\','|','/','-','\','|','/');

my $x = $recs-1;
my $h;
while ($x >= 0) {
    $ev->Read(EVENTLOG_FORWARDS_READ|EVENTLOG_SEEK_READ,
        $base + $x,
        $h)
        or die "Can't read EventLog entry #$x\n";
    print STDERR $progress[$#progress - ($x % @progress)] . "\r";
    if ($h->{Source} eq 'Security' and ($h->{EventID} == 560 or $h->{EventID} == 540)) {
        Win32::EventLog::GetMessageText($h);
        if ($h->{EventID} == 560) {
            $h->{Message} =~ /Object Name:[\t ]*(.*?)\r/gis;
            my $filename = $1;
            $h->{Message} =~ /Client User Name:[\t ]*(.*?)\r/gis;
            my $clientusername = $1;
            if ($filename) {
                if (grep { my $m = $_; $filename =~ /$m/i} @matches) {
                    printf "%s %5d : %s (%s)\n", getts($h->{TimeGenerated}), $h->{EventID}, $filename, $clientusername;
                }
            }
        } elsif ($h->{EventID} == 540) {
            $h->{Message} =~ /User Name:[\t ]*(.*?)\r/gis;
            my $username = $1;
            $h->{Message} =~ /Workstation Name:[\t ]*(.*?)\r/gis;
            my $workstation = $1;
            $h->{Message} =~ /Source Network Address:[\t ]*(.*?)\r/gis;
            my $addr = $1;
            printf "%s %5d : %s (%s - %s)\n", getts($h->{TimeGenerated}), $h->{EventID}, $workstation, $addr, $username
              if $workstation or $addr;
        }
    }
    $x--;
}

exit;
  1. In italian, sorry. Look here for an english equivalent and here for more info. ↩
]]> http://www.108.bz/posts/it/detecting-malware-using-windows-auditing-events/feed/ 0 From SQL to Excel, with Perl http://www.108.bz/posts/it/from-sql-to-excel-with-perl/ http://www.108.bz/posts/it/from-sql-to-excel-with-perl/#comments Wed, 20 Jan 2010 12:24:39 +0000 Giuliano http://www.108.bz/?p=239 Quite often I’m asked to pull out some information from a database, process it and produce an Excel report.
Here is a minimal Perl script that carries out the task.

  • Define the column headings and their widths. @columns array.
  • Handle the command line parameters. There are 5 in the example, assigned to the $p_* variables.
  • Prepare the Excel worksheet, defining cell formatting, …
  • Connect to the database.
  • Prepare the query, substituting the command line parameters.
  • Fetch rows, populate the sheet.
#!/usr/bin/perl
# Giuliano - http://www.108.bz
use strict;
use DBI;
use Spreadsheet::WriteExcel;

use constant C_HEADING => 0;
use constant C_WIDTH   => 1;
my @columns = (
    ['Date',      22 ],
    ['Caller',    20 ],
    ['Called',    20 ],
    ['Connected', 11 ],
    ['Duration',  11 ],
    ['Reason',    24 ],
    ['XferExt',   11 ],
    ['XferName',  22 ]
);

die <<EOM unless @ARGV == 5;
usage:
$0 year month day phonenumber file.xls
EOM
my ($p_year, $p_month, $p_day, $p_phnumber, $p_filename) = @ARGV;

my $workbook        = Spreadsheet::WriteExcel->new($p_filename);
my $sheet           = $workbook->add_worksheet("Data");
my $default_format  = $workbook->add_format(num_format => '@'); $default_format->set_font('Verdana'); $default_format->set_border(1);
my $bold_format     = $workbook->add_format(); $bold_format->set_font('Verdana'); $bold_format->set_bold(); $bold_format->set_border(1);

$sheet->write(0,$_,$columns[$_]->[C_HEADING], $bold_format) for (0..$#columns);
$sheet->set_column($_, $_, $columns[$_]->[C_WIDTH]) for (0..$#columns);

my $dbh = DBI->connect('dbi:Sybase:server=dsnname;database=dnbame','username','password') or die;

my $sth = $dbh->prepare(<<EOQ
SELECT IpPbxCDR.StartTime, IpPbxCDR.OriginationNumber, IpPbxCDR.CalledNumber, IpPbxCDR.DestinationNumber, DATEDIFF(ss, IpPbxCDR.StartTime, IpPbxCDR.EndTime) AS Duration, IpPbxCDR.DisconnectReason, IpPbxCDR_1.CalledNumber AS XferExt,
IpPbxCDR_1.CalledName AS XferName
FROM IpPbxCDR LEFT OUTER JOIN
IpPbxCDR AS IpPbxCDR_1 ON IpPbxCDR.TransferredToCallId = IpPbxCDR_1.CallId
WHERE (IpPbxCDR.CalledNumber LIKE '$p_phnumber') AND
(MONTH(IpPbxCDR.StartTime) = $p_month) AND
(YEAR(IpPbxCDR.StartTime) = $p_year) AND
(DAY(IpPbxCDR.StartTime) = $p_day)
ORDER BY IpPbxCDR.StartTime
EOQ
);

$sth->execute();

my $i = 1;
my $row;
while ( $row = $sth->fetchrow_arrayref ) {
    $sheet->write_string($i,$_,$row->[$_], $default_format) for (0..$#$row);
    $i++;
}

$sheet->activate();

exit;

Actually, the example does something useful. It connects to a Swyx Call Detail Record database, selecting phone calls placed to a given number on a given day. The generated report also contains call duration and transfer status/destination, if any. Here’s what it looks like (some data has been obfuscated, to protect the innocent – click to see all the columns):

And here’s the command that produces it:

./callreport.pl 2010 1 19 '+39%10123123' x.xls
]]> http://www.108.bz/posts/it/from-sql-to-excel-with-perl/feed/ 0