From MS SQL to MySQL, realtime row copy

Giuliano — Mon, 06 Sep 2010 13:26:42 +0000

Or: “How to call a web server from a Microsoft SQL Server Stored Procedure”.
Customer has got a VoIP software PBX (Swyx). It logs incoming calls (the CDR) in a MS SQL Server database. The CDR structure is straightforward: a single table where each row is a call, indexed by CallId (transferred calls get, eventually, a new row and a “child CallId”).
I needed to process the CDR within these specs/restrictions:

Each row has to be processed as soon as it is INSERTed
Rows must be filtered (depending on the called number)
Filtered rows must be “mirrored” to a MySQL DB
MS SQL machine is heavily loaded and mission critical; the row-copy mechanism must be light and fast

The first and second specs imply the use of triggers/stored procedures.

I originally thought that the “DB link”-kind of functionality could be achieved natively on MS SQL. In theory it can, via Linked Servers (bound to ODBC Data Sources). There’s a catch though: you can SELECT stuff on linked servers at will, but as soon as you try to INSERT, you’ll hit error 7391¹. MS SQL, can’t really blame it, would like to be able to rollback any change made, even on the linked MySQL. It needs to start a (implicit, distributed) transaction on MySQL, but that’s not supported and the write fails. This workaround (forcibly switch off implicit transactions) didn’t work for me. Apparently, the Oracle OLEDB Provider is able to ignore/disable distributing transactions when the parameter DistribTX=0 is in the provider string. MySQL’s ODBC driver doesn’t provide a similar toggle.

The easiest way to push data “out” of MS SQL is (arguably) through HTTP. The DB GETs a full URL, passing key/value parameters to a Web Service that outputs to MySQL.

On with the code, starting with the “Web Service”. What follow is a mere Perl script, useful for testing. Depending on the expected load, you may want to use a proper application server, providing MySQL DB connection pooling. What you should really do, is serve the script through HTTPS and password protect it. Without SSL, a malicious user could sniff the cleartext requests sent by the source DB, forge similar ones and litter/DOS the MySQL instance. Of course, the Web Service could output to just any supported DB, not only to MySQL.

#!/usr/bin/perl

use DBI;
use CGI;
use strict;

my $DEBUG = 0;
my @FIELDS = qw(
CallId
OriginationNumber
CalledNumber
DestinationNumber
StartTime
ScriptConnectTime
DeliveredTime
ConnectTime
TransferTime
EndTime
DisconnectReason
TransferredToCallId
);

my $q = new CGI;
print $q->header(-type => 'text/plain', -charset => 'ISO-8859-1', -expires => '-1d');

# checks
my $checkresult = 1;
my $checkmessage = '';
sub setcheck ($$$$) {
my ($rrc, $rc, $rrs, $rs) = @_;
$$rrc = $rc;
$$rrs = $rs;
}
sub isnumber { return 1 if $_[0] =~ /^[0-9]*$/i; return 0; }
sub issane { return 1 if $_[0] =~ /^[a-z0-9%:\- ]*$/i; return 0; }
setcheck(\$checkresult,0,\$checkmessage,'NULL CallId') if $checkresult and not $q->param('CallId');
setcheck(\$checkresult,0,\$checkmessage,'CallId must be a number') if $checkresult and not isnumber($q->param('CallId'));
foreach (@FIELDS) {
setcheck(\$checkresult,0,\$checkmessage,"$_ value contains invalid characters")
if $checkresult and not issane($q->param($_));
}

if ($checkresult) {
my $dbh = DBI->connect('DBI:mysql:database=dbname','dbuser','password') or ((print "KO: Error $DBI::err - $DBI::errstr\n"), exit);
my $values = join ',', ( map { $dbh->quote( $q->param($_) ? $q->param($_) : '') } @FIELDS );
my $sth = $dbh->prepare("INSERT INTO callslog VALUES ($values)") or ((print "KO: Error $DBI::err - $DBI::errstr\n"), exit);
$sth->execute or ((print "KO: Error $DBI::err - $DBI::errstr\n"), exit);
if ($DEBUG) {
print $_.': '.$q->param($_)."\n" for @FIELDS;
}
print "OK\n";
} else {
print "KO: $checkmessage\n";
}

exit;

Next, the trigger code. It acts after each INSERT on the IpPbxCDR table. If a called number ends with the given digits, calls the Stored Procedure spLogCall, passing it the fields we’re interested in. I use the (commented) raiserror call for debugging purposes.

USE [ippbxlog]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TRIGGER [dbo].[tr_ProcessCall]
ON [dbo].[IpPbxCDR]
AFTER INSERT
AS
BEGIN
DECLARE
@RightMatch nvarchar (10),
@CallId int,
@OriginationNumber nvarchar(50),
@CalledNumber nvarchar(50),
@DestinationNumber nvarchar(50),
@StartTime datetime,
@ScriptConnectTime datetime,
@DeliveredTime datetime,
@ConnectTime datetime,
@TransferTime datetime,
@EndTime datetime,
@DisconnectReason nvarchar(50),
@TransferredToCallId int
SET @RightMatch = '12345678'
SELECT
@CallId = CallId,
@OriginationNumber = OriginationNumber,
@CalledNumber = CalledNumber,
@DestinationNumber = DestinationNumber,
@StartTime = StartTime,
@ScriptConnectTime = ScriptConnectTime,
@DeliveredTime = DeliveredTime,
@ConnectTime = ConnectTime,
@TransferTime = TransferTime,
@EndTime = EndTime,
@DisconnectReason = DisconnectReason,
@TransferredToCallId = TransferredToCallId
FROM INSERTED
IF (RIGHT(@DestinationNumber,LEN(@RightMatch)) = @RightMatch) OR (RIGHT(@CalledNumber,LEN(@RightMatch)) = @RightMatch)
BEGIN
--raiserror('%s',16,1, @DestinationNumber)
EXEC spLogCall
@CallId,
@OriginationNumber,
@CalledNumber,
@DestinationNumber,
@StartTime,
@ScriptConnectTime,
@DeliveredTime,
@ConnectTime,
@TransferTime,
@EndTime,
@DisconnectReason,
@TransferredToCallId
END
END

Lastly, the Web Service contacting Stored Procedure. I use sp_OACreate to create an OLE object of class MSXML2.ServerXMLHTTP passing it the contructed GET URL (address + parameters). Depending on MS SQL’s version, you may have to explicitly enable in-database OLE automation, this way:

exec sp_configure 'show advanced options', 1
go
reconfigure
go
exec sp_configure 'Ole Automation Procedures', 1
go
reconfigure
go

Timeouts for various operations are set to reasonably low values, we don’t want the DB to “block” for too long. And again: use HTTPS. Get your certificates right (on MS SQL’s server, install the root certificate for the CA who issued the cert you’re using on the web/application server) and use HTTPS.

USE [ippbxlog]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO

CREATE PROCEDURE [dbo].[spLogCall]
@CallId int,
@OriginationNumber nvarchar(50),
@CalledNumber nvarchar(50),
@DestinationNumber nvarchar(50),
@StartTime datetime,
@ScriptConnectTime datetime,
@DeliveredTime datetime,
@ConnectTime datetime,
@TransferTime datetime,
@EndTime datetime,
@DisconnectReason nvarchar(50),
@TransferredToCallId int
AS

BEGIN
DECLARE
@Object int,
@hr int,
@openparams nvarchar(2048),
@responsetext varchar(8000);

EXEC @hr = sp_OACreate 'MSXML2.ServerXMLHTTP', @Object out
IF @hr = 0
BEGIN
SET CONCAT_NULL_YIELDS_NULL OFF
SET @openparams = 'open("GET", "http://10.1.1.123/ws/CDR.pl?' +
'CallId=' + CAST(@CallId AS varchar) + '&' +
'OriginationNumber=' + CAST(@OriginationNumber AS varchar) + '&' +
'CalledNumber=' + CAST(@CalledNumber AS varchar) + '&' +
'DestinationNumber=' + CAST(@DestinationNumber AS varchar) + '&' +
'StartTime=' + CONVERT(varchar, @StartTime, 120) + '&' +
'ScriptConnectTime=' + CONVERT(varchar, @ScriptConnectTime, 120) + '&' +
'DeliveredTime=' + CONVERT(varchar, @DeliveredTime, 120) + '&' +
'ConnectTime=' + CONVERT(varchar, @ConnectTime, 120) + '&' +
'TransferTime=' + CONVERT(varchar, @TransferTime, 120) + '&' +
'EndTime=' + CONVERT(varchar, @EndTime, 120) + '&' +
'DisconnectReason=' + CAST(@DisconnectReason AS varchar) + '&' +
'TransferredToCallId=' + CAST(@TransferredToCallId AS varchar) +
'", False)'
EXEC @hr = sp_OAMethod @Object, 'setTimeouts(3000,3000,3000,3000)'
EXEC @hr = sp_OAMethod @Object, @openparams
EXEC @hr = sp_OAMethod @Object, 'Send'
EXEC @hr = sp_OAGetProperty @Object, 'responseText', @responseText out
END
END

That’s it, the method performs and scales quite well. I think I’ll find other uses for it soon…

The operation could not be performed because OLE DB provider “%ls” for linked server “%ls” was unable to begin a distributed transaction. ↩

From SQL to Excel, with Perl

Giuliano — Wed, 20 Jan 2010 12:24:39 +0000

Quite often I’m asked to pull out some information from a database, process it and produce an Excel report.
Here is a minimal Perl script that carries out the task.

Define the column headings and their widths. @columns array.
Handle the command line parameters. There are 5 in the example, assigned to the $p_* variables.
Prepare the Excel worksheet, defining cell formatting, …
Connect to the database.
Prepare the query, substituting the command line parameters.
Fetch rows, populate the sheet.

#!/usr/bin/perl
# Giuliano - http://www.108.bz
use strict;
use DBI;
use Spreadsheet::WriteExcel;

use constant C_HEADING => 0;
use constant C_WIDTH => 1;
my @columns = (
['Date', 22 ],
['Caller', 20 ],
['Called', 20 ],
['Connected', 11 ],
['Duration', 11 ],
['Reason', 24 ],
['XferExt', 11 ],
['XferName', 22 ]
);

die <<EOM unless @ARGV == 5;
usage:
$0 year month day phonenumber file.xls
EOM
my ($p_year, $p_month, $p_day, $p_phnumber, $p_filename) = @ARGV;

my $workbook = Spreadsheet::WriteExcel->new($p_filename);
my $sheet = $workbook->add_worksheet("Data");
my $default_format = $workbook->add_format(num_format => '@'); $default_format->set_font('Verdana'); $default_format->set_border(1);
my $bold_format = $workbook->add_format(); $bold_format->set_font('Verdana'); $bold_format->set_bold(); $bold_format->set_border(1);

$sheet->write(0,$_,$columns[$_]->[C_HEADING], $bold_format) for (0..$#columns);
$sheet->set_column($_, $_, $columns[$_]->[C_WIDTH]) for (0..$#columns);

my $dbh = DBI->connect('dbi:Sybase:server=dsnname;database=dnbame','username','password') or die;

my $sth = $dbh->prepare(<<EOQ
SELECT IpPbxCDR.StartTime, IpPbxCDR.OriginationNumber, IpPbxCDR.CalledNumber, IpPbxCDR.DestinationNumber, DATEDIFF(ss, IpPbxCDR.StartTime, IpPbxCDR.EndTime) AS Duration, IpPbxCDR.DisconnectReason, IpPbxCDR_1.CalledNumber AS XferExt,
IpPbxCDR_1.CalledName AS XferName
FROM IpPbxCDR LEFT OUTER JOIN
IpPbxCDR AS IpPbxCDR_1 ON IpPbxCDR.TransferredToCallId = IpPbxCDR_1.CallId
WHERE (IpPbxCDR.CalledNumber LIKE '$p_phnumber') AND
(MONTH(IpPbxCDR.StartTime) = $p_month) AND
(YEAR(IpPbxCDR.StartTime) = $p_year) AND
(DAY(IpPbxCDR.StartTime) = $p_day)
ORDER BY IpPbxCDR.StartTime
EOQ
);

$sth->execute();

my $i = 1;
my $row;
while ( $row = $sth->fetchrow_arrayref ) {
$sheet->write_string($i,$_,$row->[$_], $default_format) for (0..$#$row);
$i++;
}

$sheet->activate();

exit;

Actually, the example does something useful. It connects to a Swyx Call Detail Record database, selecting phone calls placed to a given number on a given day. The generated report also contains call duration and transfer status/destination, if any. Here’s what it looks like (some data has been obfuscated, to protect the innocent – click to see all the columns):

And here’s the command that produces it:

./callreport.pl 2010 1 19 '+39%10123123' x.xls

108.bz » SQL

From MS SQL to MySQL, realtime row copy

From SQL to Excel, with Perl