108.bz » IT

Oracle Reports Server font issues

Giuliano — Wed, 08 Sep 2010 13:45:53 +0000

Here’s a couple of hints you may find useful when shooting troubles around Oracle Reports Server.
It all starts when Customer wants to produce PDF reports using a custom font. Of course the font won’t be there when the file is opened on a client PC. The report server must either embed (include entirely) or subset (include just the actually used glyphs) the font into the PDF.

We’re running Oracle Application Server 10g Release 2 on Windows.

More than one Reports Server (RS from now on) can be run at the same time. Each RS is identified by name. Open a DOS window, change directory to the one where reports templates/resources are, then launch:

rwserver server=rstest

RS named “rstest” will get its own log directory and configuration file, under the Application Server “HOME”:

reports\conf\rstest.conf
reports\logs\rstest\
reports\logs\rstest\rwserver.log

Quite convenient: you’ll leave the production RS alone, be able to activate debug tracing, restart at will, …

Here’s how tu run a test report on the “rstest” RS:

rwclient SERVER=rstest REPORT=d01_skt_anag1.rdf userid=user/pwd@db DESFORMAT=pdf DESTYPE=file DESNAME=c:\temp\testoutput.pdf

Back to our issues. First step, I’d say, is to find out the exact name of the font we’d like to embed. Did you know you can convert Report Developer “source” files (.rdf) to XML, then peek in them? Use:

rwconverer STYPE=rdffile SOURCE=d01_skt_anag1.rdf DTYPE=xmlfile dest=c:\temp\rpt.xml

“Cooper Black” is the name:

C:\dev\appl\Reports>findstr /I face c:\temp\rpt.xml | findstr /I coop

Install the font in Windows. Oddly, things didn’t seem to work for me when I just copied the .ttf file in the C:\WINDOWS\Fonts directory. I had to use the “Install New Font” menu item. I thought both methods were equivalent, maybe I was wrong, maybe I’m talking junk now.

Modify the uifont.ali file (tools\common), telling the Report Server to subset “Cooper Black” into the generated PDF files. Just add a line under the PDF:Subset section, equating the font name to the TTF file name, both enclosed in double quotes.

[ PDF:Subset ]
"Cooper Black" = "COOPBL.TTF"

The fnchk.exe program (ran without arguments) displays the full path of uifont.ali and tells if everything is fine or the file contains any syntax error.

Installing the font in Windows is not enough, you should also put it in one of the REPORTS_PATH directories. The value of this variable can be found in the registry, I chose: C:\Oracle\Products\FRHome\reports\templates. FileMon is essential when trying to see which files can’t be found by a process.

Time to restart the Reports Service, generate a PDF report and check if fonts are good (in Acrobat Reader: CTRL-D, “Fonts” tab). Ours is listed as “Embedded Subset”; we’re done.

See also: Oracle Support Note.350971.1 “Troubleshooting Guide for Font Aliasing / Font Subsetting / Font Embedding Issues”.

From MS SQL to MySQL, realtime row copy

Giuliano — Mon, 06 Sep 2010 13:26:42 +0000

Or: “How to call a web server from a Microsoft SQL Server Stored Procedure”.
Customer has got a VoIP software PBX (Swyx). It logs incoming calls (the CDR) in a MS SQL Server database. The CDR structure is straightforward: a single table where each row is a call, indexed by CallId (transferred calls get, eventually, a new row and a “child CallId”).
I needed to process the CDR within these specs/restrictions:

Each row has to be processed as soon as it is INSERTed
Rows must be filtered (depending on the called number)
Filtered rows must be “mirrored” to a MySQL DB
MS SQL machine is heavily loaded and mission critical; the row-copy mechanism must be light and fast

The first and second specs imply the use of triggers/stored procedures.

I originally thought that the “DB link”-kind of functionality could be achieved natively on MS SQL. In theory it can, via Linked Servers (bound to ODBC Data Sources). There’s a catch though: you can SELECT stuff on linked servers at will, but as soon as you try to INSERT, you’ll hit error 7391¹. MS SQL, can’t really blame it, would like to be able to rollback any change made, even on the linked MySQL. It needs to start a (implicit, distributed) transaction on MySQL, but that’s not supported and the write fails. This workaround (forcibly switch off implicit transactions) didn’t work for me. Apparently, the Oracle OLEDB Provider is able to ignore/disable distributing transactions when the parameter DistribTX=0 is in the provider string. MySQL’s ODBC driver doesn’t provide a similar toggle.

The easiest way to push data “out” of MS SQL is (arguably) through HTTP. The DB GETs a full URL, passing key/value parameters to a Web Service that outputs to MySQL.

On with the code, starting with the “Web Service”. What follow is a mere Perl script, useful for testing. Depending on the expected load, you may want to use a proper application server, providing MySQL DB connection pooling. What you should really do, is serve the script through HTTPS and password protect it. Without SSL, a malicious user could sniff the cleartext requests sent by the source DB, forge similar ones and litter/DOS the MySQL instance. Of course, the Web Service could output to just any supported DB, not only to MySQL.

#!/usr/bin/perl

use DBI;
use CGI;
use strict;

my $DEBUG = 0;
my @FIELDS = qw(
CallId
OriginationNumber
CalledNumber
DestinationNumber
StartTime
ScriptConnectTime
DeliveredTime
ConnectTime
TransferTime
EndTime
DisconnectReason
TransferredToCallId
);

my $q = new CGI;
print $q->header(-type => 'text/plain', -charset => 'ISO-8859-1', -expires => '-1d');

# checks
my $checkresult = 1;
my $checkmessage = '';
sub setcheck ($$$$) {
my ($rrc, $rc, $rrs, $rs) = @_;
$$rrc = $rc;
$$rrs = $rs;
}
sub isnumber { return 1 if $_[0] =~ /^[0-9]*$/i; return 0; }
sub issane { return 1 if $_[0] =~ /^[a-z0-9%:\- ]*$/i; return 0; }
setcheck(\$checkresult,0,\$checkmessage,'NULL CallId') if $checkresult and not $q->param('CallId');
setcheck(\$checkresult,0,\$checkmessage,'CallId must be a number') if $checkresult and not isnumber($q->param('CallId'));
foreach (@FIELDS) {
setcheck(\$checkresult,0,\$checkmessage,"$_ value contains invalid characters")
if $checkresult and not issane($q->param($_));
}

if ($checkresult) {
my $dbh = DBI->connect('DBI:mysql:database=dbname','dbuser','password') or ((print "KO: Error $DBI::err - $DBI::errstr\n"), exit);
my $values = join ',', ( map { $dbh->quote( $q->param($_) ? $q->param($_) : '') } @FIELDS );
my $sth = $dbh->prepare("INSERT INTO callslog VALUES ($values)") or ((print "KO: Error $DBI::err - $DBI::errstr\n"), exit);
$sth->execute or ((print "KO: Error $DBI::err - $DBI::errstr\n"), exit);
if ($DEBUG) {
print $_.': '.$q->param($_)."\n" for @FIELDS;
}
print "OK\n";
} else {
print "KO: $checkmessage\n";
}

exit;

Next, the trigger code. It acts after each INSERT on the IpPbxCDR table. If a called number ends with the given digits, calls the Stored Procedure spLogCall, passing it the fields we’re interested in. I use the (commented) raiserror call for debugging purposes.

USE [ippbxlog]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TRIGGER [dbo].[tr_ProcessCall]
ON [dbo].[IpPbxCDR]
AFTER INSERT
AS
BEGIN
DECLARE
@RightMatch nvarchar (10),
@CallId int,
@OriginationNumber nvarchar(50),
@CalledNumber nvarchar(50),
@DestinationNumber nvarchar(50),
@StartTime datetime,
@ScriptConnectTime datetime,
@DeliveredTime datetime,
@ConnectTime datetime,
@TransferTime datetime,
@EndTime datetime,
@DisconnectReason nvarchar(50),
@TransferredToCallId int
SET @RightMatch = '12345678'
SELECT
@CallId = CallId,
@OriginationNumber = OriginationNumber,
@CalledNumber = CalledNumber,
@DestinationNumber = DestinationNumber,
@StartTime = StartTime,
@ScriptConnectTime = ScriptConnectTime,
@DeliveredTime = DeliveredTime,
@ConnectTime = ConnectTime,
@TransferTime = TransferTime,
@EndTime = EndTime,
@DisconnectReason = DisconnectReason,
@TransferredToCallId = TransferredToCallId
FROM INSERTED
IF (RIGHT(@DestinationNumber,LEN(@RightMatch)) = @RightMatch) OR (RIGHT(@CalledNumber,LEN(@RightMatch)) = @RightMatch)
BEGIN
--raiserror('%s',16,1, @DestinationNumber)
EXEC spLogCall
@CallId,
@OriginationNumber,
@CalledNumber,
@DestinationNumber,
@StartTime,
@ScriptConnectTime,
@DeliveredTime,
@ConnectTime,
@TransferTime,
@EndTime,
@DisconnectReason,
@TransferredToCallId
END
END

Lastly, the Web Service contacting Stored Procedure. I use sp_OACreate to create an OLE object of class MSXML2.ServerXMLHTTP passing it the contructed GET URL (address + parameters). Depending on MS SQL’s version, you may have to explicitly enable in-database OLE automation, this way:

exec sp_configure 'show advanced options', 1
go
reconfigure
go
exec sp_configure 'Ole Automation Procedures', 1
go
reconfigure
go

Timeouts for various operations are set to reasonably low values, we don’t want the DB to “block” for too long. And again: use HTTPS. Get your certificates right (on MS SQL’s server, install the root certificate for the CA who issued the cert you’re using on the web/application server) and use HTTPS.

USE [ippbxlog]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO

CREATE PROCEDURE [dbo].[spLogCall]
@CallId int,
@OriginationNumber nvarchar(50),
@CalledNumber nvarchar(50),
@DestinationNumber nvarchar(50),
@StartTime datetime,
@ScriptConnectTime datetime,
@DeliveredTime datetime,
@ConnectTime datetime,
@TransferTime datetime,
@EndTime datetime,
@DisconnectReason nvarchar(50),
@TransferredToCallId int
AS

BEGIN
DECLARE
@Object int,
@hr int,
@openparams nvarchar(2048),
@responsetext varchar(8000);

EXEC @hr = sp_OACreate 'MSXML2.ServerXMLHTTP', @Object out
IF @hr = 0
BEGIN
SET CONCAT_NULL_YIELDS_NULL OFF
SET @openparams = 'open("GET", "http://10.1.1.123/ws/CDR.pl?' +
'CallId=' + CAST(@CallId AS varchar) + '&' +
'OriginationNumber=' + CAST(@OriginationNumber AS varchar) + '&' +
'CalledNumber=' + CAST(@CalledNumber AS varchar) + '&' +
'DestinationNumber=' + CAST(@DestinationNumber AS varchar) + '&' +
'StartTime=' + CONVERT(varchar, @StartTime, 120) + '&' +
'ScriptConnectTime=' + CONVERT(varchar, @ScriptConnectTime, 120) + '&' +
'DeliveredTime=' + CONVERT(varchar, @DeliveredTime, 120) + '&' +
'ConnectTime=' + CONVERT(varchar, @ConnectTime, 120) + '&' +
'TransferTime=' + CONVERT(varchar, @TransferTime, 120) + '&' +
'EndTime=' + CONVERT(varchar, @EndTime, 120) + '&' +
'DisconnectReason=' + CAST(@DisconnectReason AS varchar) + '&' +
'TransferredToCallId=' + CAST(@TransferredToCallId AS varchar) +
'", False)'
EXEC @hr = sp_OAMethod @Object, 'setTimeouts(3000,3000,3000,3000)'
EXEC @hr = sp_OAMethod @Object, @openparams
EXEC @hr = sp_OAMethod @Object, 'Send'
EXEC @hr = sp_OAGetProperty @Object, 'responseText', @responseText out
END
END

That’s it, the method performs and scales quite well. I think I’ll find other uses for it soon…

The operation could not be performed because OLE DB provider “%ls” for linked server “%ls” was unable to begin a distributed transaction. ↩

Recovering NTBackup Tapes

Giuliano — Wed, 25 Aug 2010 10:36:43 +0000

This article will show you how to handle tape backups generated by NTBackup, turn them into a .BKF file (using Linux) and extract specific files (using Linux/Windows). Some of the stuff explained here may also be useful when dealing with corrupt tapes/files, and may work for any backup software that generates MTF (Microsoft Tape Format) output, such as maybe Symantec Backup Exec¹.

The scenario: an old machine (hosting a not so important app) crashes badly due to multiple disk failures. O.S. (Windows 2000 Server) won’t boot anymore. Backups were directed to a local DDS tape drive, the only one of its kind surviving in the whole Company. While reinstalling the app to another server, I need to recover some files and have access to the pre-crash registry.

And here’s the plan:

Boot the half-dead server with the invaluable SystemRescueCd.
Put the last available tape backup in the drive.
Save an image of the tape somewhere.
Extract stuff from the image.

When SystemRescueCd is running and network connected, make available a shared folder:

root@sysresccd /root % mkdir /mnt/storagespace
root@sysresccd /root % mount -t cifs //fileserver/e$ /mnt/storagespace -o username=administrator,workgroup=domain.local

Then, generate the image. NTBackup tape backups are spread across multiple “tape files”. If you read the tape from the beginning, sooner or later you will hit EOF (an end-of-file condition). Don’t rewind it: go on to the next file instead. Repeat until there are no more files to read.
On Unix, the first SCSI tape device is mapped to /dev/st0 and /dev/nst0. When a process finishes reading from /dev/st0, the tape is implicitly rewound. Viceversa, using /dev/nst0 doesn’t cause any rewind; tape will stay positioned right after the last block read.

Just in case, perform a manual rewind:

root@sysresccd /mnt/storagespace/temp % mt -f /dev/st0 rewind

Then, try to guess the right block size. It seems to be set at 16K. Should this method fail, check the “How do I find out tape block size?” method here.

root@sysresccd /mnt/storagespace/temp % mt -f /dev/st0 status
SCSI 2 tape drive:
File number=0, block number=0, partition=0.
Tape block size 16384 bytes. Density code 0x26 (DDS-4 or QIC-4GB).
Soft error count since last status=0
General status bits on (41010000):
BOT ONLINE IM_REP_EN

Start reading (by means of dd) with the specified block size. See? We’re using the non-rewinding tape device /dev/nst0.

root@sysresccd /mnt/storagespace/temp % for f in `seq 1 10`; do echo dd if=/dev/nst0 of=tapeblock`printf "%
06g" $f`.bin ibs=16384; done
dd if=/dev/nst0 of=tapeblock000001.bin ibs=16384
dd if=/dev/nst0 of=tapeblock000002.bin ibs=16384
dd if=/dev/nst0 of=tapeblock000003.bin ibs=16384
dd if=/dev/nst0 of=tapeblock000004.bin ibs=16384
dd if=/dev/nst0 of=tapeblock000005.bin ibs=16384
dd if=/dev/nst0 of=tapeblock000006.bin ibs=16384
dd if=/dev/nst0 of=tapeblock000007.bin ibs=16384
dd if=/dev/nst0 of=tapeblock000008.bin ibs=16384
dd if=/dev/nst0 of=tapeblock000009.bin ibs=16384
dd if=/dev/nst0 of=tapeblock000010.bin ibs=16384
root@sysresccd /mnt/storagespace/temp % dd if=/dev/nst0 of=tapeblock000001.bin ibs=16384
1+0 records in
32+0 records out
16384 bytes (16 kB) copied, 0.0341089 s, 480 kB/s
root@sysresccd /mnt/storagespace/temp % dd if=/dev/nst0 of=tapeblock000002.bin ibs=16384
270540+0 records in
8657280+0 records out
4432527360 bytes (4.4 GB) copied, 7666.22 s, 578 kB/s
[..]
root@sysresccd /mnt/storagespace/temp % dd if=/dev/nst0 of=tapeblock000007.bin ibs=16384
4+0 records in
128+0 records out
65536 bytes (66 kB) copied, 0.1176 s, 557 kB/s
root@sysresccd /mnt/storagespace/temp % dd if=/dev/nst0 of=tapeblock000008.bin ibs=16384
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.00461986 s, 0.0 kB/s
root@sysresccd /mnt/storagespace/temp % dd if=/dev/nst0 of=tapeblock000009.bin ibs=16384
dd: reading `/dev/nst0': Input/output error
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.00356835 s, 0.0 kB/s

Reads beyond the last file will result in an “Input/output error”.

It’s time to join the .bin files into a single one: the tape image (maybe using HJSplit for the task). You could’ve been more clever than me and appended dd‘s output to a single file, thus skipping the join step and saving space. I didn’t do it because I wanted to see if any tape file was corrupted (and be able to re-read it, if needed).

I called the tape image “backup.bkf”, even though it’s not a true .BKF file… As we said, it’s a tape image. NTBackup is not able to read it, whereas the abundance of “BKF recovery” software can. Before going on, let me polemicize a bit. There are many, almost identical, software of this kind. I’ve got the feeling that they all “borrow” from this open source BKF reader². Looks like different commercial developers grabbed the same source, embellished the GUI just a bit, and made a product to sell. How lame. But it turns out that you don’t need to pay a cent to extract files from a .BKF or tape image, on Linux or Windows as well.

On windows, get ntbkup by William T. Kranz . I use the (optional) -s3 switch to target set 3 which I know holds the System State.

C:\temp\x>..\ntbkup.exe ..\backup.bkf -s3 -l"\Registry:"

NTBKUP Ver 1.07c compiled for WIN32 with MAX_PATH = 100
compiled for 64 bit file offsets
Copyright (C) 2003 William T. Kranz
NTBKUP comes with ABSOLUTELY NO WARRANTY
Free software distributed under the terms of the GNU General Public license
See http://www.gnu.org/licenses/gpl.html for license information
Check http://www.fpns.net/willy/msbackup.htm for Updates & Documentation

resrict operations to backup set 3
device name: C:
volume name: Local disk
device name: D:
volume name: Volume
Set 3:
Name: Company lun-29-06-2009-23.04
Description:
User: DOMAIN\administrator

device name: System state data from 0x36113d956 to 0x3611ec956
length 716800 atrib 0x20 05/14/2008 03:10:38 PM
extracing: default:
data from 0x3611ecd4e to 0x3611f2d4e
length 24576 atrib 0x20 06/29/2009 09:42:50 PM
extracing: SAM:
data from 0x3611f3156 to 0x3611fe156
length 45056 atrib 0x20 06/29/2009 11:00:07 PM
extracing: SECURITY:
data from 0x3611fe556 to 0x3621cb556
length 16568320 atrib 0x20 06/30/2009 01:35:59 AM
extracing: software:
data from 0x3621cb952 to 0x362487952
length 2867200 atrib 0x20 06/29/2009 10:42:23 PM
extracing: system:
data from 0x362487d2e to 0x3624aad2e
length 143360 atrib 0x20 06/04/2003 01:24:14 PM
extracing: userdiff:

Bingo, I can load the software hive with “reg load” (see here) and extract the keys I need.

Should you prefer so, download mtftar on your Linux box, compile it and run something like:

./mtftar -f /mnt/storagespace/temp/backup.bkf | tar xvf - "Registry"

Extract the other files you need and voilà…

ex Veritas Backup Exec ↩
There’s no executable in the archive. You need Visual Studio and compile it for yourself. ↩

FortiGate firewall clusters group-id

Giuliano — Sat, 03 Jul 2010 15:53:31 +0000

A newly installed FortiGate cluster (a simple two node HA active-passive setup) and some packet loss issues…
Ping from the LAN side to the Internet (or from the firewall itself) resulted in about 20% packet loss, while the other way around (WAN to firewall’s main public IP) didn’t work at all.

I used the following command to check my MAC addresses:

FORTIGATE-PRI # diagnose hardware deviceinfo nic wan1
[..]
Current_HWaddr 00:09:0f:09:00:08
Permanent_HWaddr 00:09:0f:d1:be:ef
[..]

Then resorted to the “show mac” switches facilites (some Cisco, some ProCurve) to know on which network ports that particular MAC lied… Only to discover that the cluster’s “logical” MAC address (00:09:0f:09:00:08) wasn’t really located where I expected it to be.
Well, FortiGate’s MAC addresses aren’t randomly generated. They have predictable values that depend on the firewall’s port number. The eight port (or wan1, in my case) will always have a virtual MAC as the one above. What will happen if you have two clusters (as we had) sitting on the same L2 network segment (on the same broadcast domain, that is)? You said MAC address conflict? You’re right.
The solution is simple, use the group-id directive to tweak the logical MAC address, i.e.:

config system ha
set group-id 10
end

Changes the second right-most bytes of the MAC, from 00 to 0a:

before 00:09:0f:09:00:08
after 00:09:0f:09:0a:08

Point is that the “FortiOS High Availablity Handbook” explains the case very thoroughly! See page 192, paragraph “Diagnosing packet loss with two FortiGate HA clusters in the same broadcast domain”. We’re so used to discardable product documentation that sometimes we don’t even try to look for clues where they should normally reside.
Instead of troubleshooting, this time, I should really have Read The (unexpectedly) Fine Manual…

Unknown devices on HP servers

Giuliano — Mon, 14 Jun 2010 12:14:51 +0000

Following up on the “Unknown devices on IBM servers” post, let me talk about a similar situation with HP machines (DL180 G6, in my case).

The device that Windows fails to identify is this one:

PCI\VEN_8086&DEV_3A22&CC_0106

More info can be found by looking up the IDs in the pci.ids file (as I often do), or by means of the various “Unkown Device Identifier” type of software (e.g. this one). If you have a Linux machine at hand, such a one-liner may suit you:

# sed -n -e '/^8086/,/3a22/p' /usr/share/misc/pci.ids | sed -n -e '1p;$p'
8086 Intel Corporation
3a22 82801JI (ICH10 Family) SATA AHCI Controller

What’s missing is an Intel SATA driver; needless to say that you won’t find it anywere on HP site.
I downloaded and installed the Rapid Storage Technology Driver from Intel’s web site (here). A 280KB download named “STOR_all32_f6flpy_9.6.0.1014_PV.zip” fixed things up for me.
Maybe the proper thing to try would’ve been the latest (March 2010) Proliant Support Pack, but it’s a big download and I didn’t have the time. Also, the onboard SATA controller isn’t really used (the additional SAS RAID is, instead) and I just wanted to get rid of the yellow warning sign in Device Manager.

The offline ACU CD

Giuliano — Fri, 28 May 2010 10:03:12 +0000

Well hidden in their labyrinthian web site, you may stumble upon HP’s “Array Configuration Utility (ACU) Offline CD for Smart Array”. A plain bootable CD, useful when ACU simply can’t be installed on the server/OS.
Example: I needed to tweak SSP (Selective Storage Presentation) settings on an MSA1000, connected through Fiber Channel HBAs (QLogic) to some rather old HP DL580 G2. The servers were running VMware ESX 3i 3.5.0 build-207095 (the latest one compatibile with those kind of CPUs) with no management agents installed. Since the MSA1000 can only be managed “in-band” or via a non standard serial cable the Customer, of course, lost long ago, I rebooted an ESX host with the offline ACU CD…
Before that, I also tried a standard SmartStart CD, but it didn’t work. I had version 7.80 (way younger than the servers/HBAs), but no link lights on the FC switch, meaning no firmware loaded on the QLogic card, meaning no SmartStart supported HBA drivers. Offline ACU CD version 8.20.19 worked like a charm instead. Find its latest release by searching “array configuration utility” on hp.com, clicking on “Download software”, then “Linux GUI ACU”. Download link is somewhere in that page…

Using the CLI to manage Windows DNS servers

Giuliano — Mon, 24 May 2010 14:16:46 +0000

(This, for once, is going to be quick.)
Did you know about the Dnscmd.exe command? Read about it here and here. It’s the command-line/DOS prompt way to configure Microsoft’s DNS servers… If you need to create many zones/records at once, it saves you lots of clicks.
Here’s how to add six DNS zones (same domain name, different TLD). With the /DSPrimary option, the zone will be stored into Active Directory (rather than a file).

dnscmd /ZoneAdd domainname.bz /DSPrimary
dnscmd /ZoneAdd domainname.biz /DSPrimary
dnscmd /ZoneAdd domainname.com /DSPrimary
dnscmd /ZoneAdd domainname.eu /DSPrimary
dnscmd /ZoneAdd domainname.net /DSPrimary
dnscmd /ZoneAdd domainname.org /DSPrimary

And here’s how to add the same “A” record (named “www”) to each of the zones created above.

dnscmd dns-dc-hostname /RecordAdd domainname.bz www A 10.0.0.123
dnscmd dns-dc-hostname /RecordAdd domainname.biz www A 10.0.0.123
dnscmd dns-dc-hostname /RecordAdd domainname.com www A 10.0.0.123
dnscmd dns-dc-hostname /RecordAdd domainname.eu www A 10.0.0.123
dnscmd dns-dc-hostname /RecordAdd domainname.net www A 10.0.0.123
dnscmd dns-dc-hostname /RecordAdd domainname.org www A 10.0.0.123

As you may have guessed this is the typical scenario where you’ve got to re-create some external zones, on the internal DNS servers. That’s needed in order for the internal hosts to reach some server with the “public” DNS name, but the private IP.
For the sake of completeness, let me also mention that you could achieve the same effect by leaving DNS as it is, and configuring “loopback NAT”/”double NAT” on the router/firewall. E.g.: an internal Host wants to reach an internal Server, given it’s public hostname, mapped to a public IP address. It asks the (possibly internal) DNS to translate the name. DNS doesn’t know the zone, it forwards the query to an external DNS Server, obtaining a public IP address that it hands back to the Client. Since its address is non-local, while trying to talk with the Server, the Client sends packets to its default gateway (possibly the router/firewall). The firewall matches the server’s public IP addresses, substituting it with the right private one. It also changes the source IP, swapping the Client’s with the firewall’s LAN address. This way Client and Server are actually communicating through the firewall, even if they’re both internal hosts. And the Server can’t tell Client A from Client B since every connection to it comes from the firewall’s IP address. That’s the main reason why I prefer duplicating the public DNS zones on internal DNS servers, with private IP addresses: you avoid routing internal traffic through the firewall, and avoid NAT where there shouldn’t be any.

Bulk unlocking Active Directory user accounts

Giuliano — Sat, 15 May 2010 22:23:59 +0000

This post will show you how to generate a list of all the users’ Distinguished Name, then filter it, then do something useful with it.

Scenario: saturday morning (after having crashed into bed at 4:00 a.m., btw), Customer calls. A virus hit the Company and one of the most annoying consequences of the outburst, is that every domain user account gets locked due to brute-force login attempts (as per the “Account Lockout Threshold” policy). While they run around cleaning PCs and fixing A/V installations¹, I’m asked for a method to quickly unlock the accounts.

I tend to carry out these kind of tasks “the Unix way”, using the available DOS prompt commands and a bit of VBScript.

Start off by calling LDIFDE:

ldifde -r "(objectclass=user)" -l sAMAccountName -m -f users.ldf

LDIFDE exports/imports Active Directory data to/from properly formatted (LDIF) text files. I use it a lot. Ran as shown above, LDIFDE exports the objects of class “user” into a file named users.ldf . Of the many attributes an LDAP object bears, I tell LDIFDE to output just the “sAMAccountName” one. If I hadn’t specified any attribute, in the resulting file I’d have found duplicate DNs for the same user. That’s because of how the resulting LDIF file is described. Some A/D data is “incrementally” added to existing objects given their DN. I just picked sAMAccountName because every user has one and, also, to keep the file small.
Then:
findstr /I /b dn.*ou=service.users users.ldf > service_users.txt
findstr /I /b dn.*cn=users users.ldf > normal_users.txt

findstr is Microsoft’s “poor man version” of grep, supporting a subset of the regular expression everyone has or should’ve come to love. Here I’m using it to extract Distinguished Names from the LDIF (only the ones that lie in a given Organizational Unit), and saving them to the *_users.txt files. They will look like:

dn: CN=squidauth,OU=Service Users,DC=contoso,DC=com
dn: CN=exchangebackup,OU=Service Users,DC=contoso,DC=com
dn: CN=ldap,OU=Service Users,DC=contoso,DC=com
dn: CN=batchcopy,OU=Service Users,DC=contoso,DC=com
Here’s the VBScript function to unlock an account given its DN:
Sub unlockuser(userDN)
Set objUser = GetObject ("LDAP://" & userDN)
objUser.IsAccountLocked = False
objUser.SetInfo
End Sub

We just need to transform findstr’s output, substituting the leading “dn: ” with “unlockuser” and enclosing in double quotes what follows. At the top of the new, transformed, file, we’ll copy/paste unlockuser subroutine definition. That’ll make our final script.
How to carry out the transform? Using this VBS snippet; it processes its Standard Input line by line, and outputs the modifications on Standard Output, just like any Unix file filtering command.
Set StdIn = WScript.StdIn
Do While Not StdIn.AtEndOfStream
line = stdin.readline
line = right(line,len(line)-4)
wscript.echo "unlockuser """ & line & """"
Loop

I saved it in a “dnfilter.vbs” file and used it this way:

type service_users.txt | cscript /nologo dnfilter.vbs > unlock_service_users.vbs

To obtain something like this:

unlockuser "CN=squidauth,OU=Service Users,DC=contoso,DC=com"
unlockuser "CN=exchangebackup,OU=Service Users,DC=contoso,DC=com"
unlockuser "CN=ldap,OU=Service Users,DC=contoso,DC=com"
unlockuser "CN=batchcopy,OU=Service Users,DC=contoso,DC=com"

As I said, add the unlockuser function at the top of unlock_service_users.vbs and you’ll have your bulk unlocking script.

A/V usefulness is often questionable. At least three times a year an unfortunate Customer gets infected by a 0-day threat… ↩

Dumping streaming media in 25 lines of Perl

Giuliano — Thu, 13 May 2010 10:11:23 +0000

Analysing TCP based protocols often means dealing with TCP sessions (also called streams or flows).
A TCP connection, from an application point of view, is much like a bidirectional file descriptor through which ordered data can be read or written. “On the wire” though, data is not ordered at all. It is split into packets, possibly shuffled and mixed with other traffic. You can capture packets using a sniffer, but to make any sense of them you also need an analyzer tool able to do the reordering/reassembling job. Wireshark, for instance, doubles as a sniffer and an analyzer, backed up by the ubiquitous libpcap.

Imagine having dumped/sniffed 1GB worth of traffic. We would like to pinpoint a single TCP session, isolating it from the rest. Here’s how we could proceed:

Identify the source/destination addresses and source/destination ports we’re interested in. Then throw away any packet that doesn’t match this tuple. That’s what Wireshark basically does when you select a packet, right click and hit “Follow TCP Stream”. If the same tuple doesn’t get reused for another, unrelated, session, this method works just fine¹.
Reorder/reassemble packets.
Extract packets’ payload.
Present the payload in a way that makes sense. That depends on the L7 protocol. HTTP without keep-alive is strictly request/response: print what the client sent to the server (outbound traffic) before and then what the server answered (inbound traffic). Other protocols may behave differently and you may choose to separate inbound traffic from outbound, or rely on timing to correctly present the dialogue between peers.

Besides Wireshark, there are tools that do just that and can also be automated. See TShark or tcpflow.

What if you want to script everything and build your own TCP analyzer? Perl’s module Net::Analysis is surprisingly convenient for the task. It does the dirty job I described above and presents your code with ready to be processed TCP sessions.

Practical goal: saving MP3 files streamed by Grooveshark. Disclaimer: I’m by no means pushing anyone to illegally download stuff, this is just a working, sensible, instructional example that uses a song freely available anyway (by Revolution Void, check them out here, they’re great).

GroovesharkListener.pm extends Net::Analysis::Listener::HTTP. It sniffs all the traffic from/to port 80 and, as soon as he sees an HTTP response with a content-type of “audio”, dumps its content to file and quits. Simple as that.

Put the module some place where Perl can find it and then launch (as root):

# perl -MNet::Analysis -e main GroovesharkListener 'port 80'
(starting live capture)
/crossdomain.xml
text/xml
/service.php?addSongsToQueueExt
text/html; charset=UTF-8
/static/amazonart/m8c8c9f4291508bca130c1caac2bda75b.png
image/png
[...some more cruft...]
/stream.php
audio/mpeg
Dumping 8481224 bytes to groovesharkgyzBy.mp3 be patient...

# id3v2 -l groovesharkgyzBy.mp3
id3v1 tag info for groovesharkgyzBy.mp3:
Title : Invisible Walls Artist: Revolution Void
Album : Increase the Dosage Year: 2004, Genre: Other (12)
Comment: http://www.jamendo.com/ Track: 1

That’s it, just one more thing. Net::Analysis doesn’t allow you to select a specific network interface, it just picks up the first available one. I wrote a small patch to address this shortcoming, it adds a “device=” parameter that you can use this way:

# perl -MNet::Analysis -e main GroovesharkListener,device=wlan1 'port 80'

And here’s what GroovesharkListener.pm looks like:

# choose a song
# run (as root or via sudo):
# perl -MNet::Analysis -e main GroovesharkListener 'port 80'
# click "play" and wait for the file to be dumped...
# -- Giuliano - http://www.108.bz
package Net::Analysis::Listener::GroovesharkListener;
use strict;
use base qw(Net::Analysis::Listener::HTTP);
use File::Temp;

sub http_transaction {
my ($self, $args) = @_;
my ($http_req) = $args->{req};
my ($http_resp) = $args->{resp};

print $http_req->uri(), "\n";
my $content_type = $http_resp->header('Content-Type');
print "$content_type\n";
if ($content_type =~ /audio/i) {
my $fh = new File::Temp(TEMPLATE => 'groovesharkXXXXX',
SUFFIX => '.mp3',
UNLINK => 0);
print "Dumping ".length($http_resp->content)." bytes to ".$fh->filename." be patient...\n";
print $fh $http_resp->content;
exit;
}
}

1;

newer Wireshark(s) use the “tcp.stream eq x” primitive ↩

Active Directory Graphs

Giuliano — Wed, 28 Apr 2010 15:26:32 +0000

Domain Controllers replicate Active Directory data with each other. They do so through Connections that are partly generated by the KCC (Knowledge Consistency Checker), partly configured by you: the Sysadmin . Each connection is one-way. If you open Active Directory Sites and Services, expand a Site and then a Server node, you’ll notice that Connections listed under NTDS Settings are labeled “From Server” and “From Site”. In the image below (stolen from here), the DC named HEIDITEST will replicate AD changes by sending them to MHILLMAN2. The Connection Object is thus defined from HEIDITEST, to MHILLMAN2. You can expect a specular Connection to exist, defined under the NTDS Settings node of HEIDITEST.

See Active Directory Replication for a more in-depth explanation.
Besides Connection objects automatically created by the KCC, which does its best to build a proper replication topology, you sometimes add your own for fault/link tolerance or other reasons. If the domain is sufficiently big, things may become messy. Instead of fumbling my way through Active Directory Sites and Services I wanted to automatically generate a visual representation of such topology, with DCs and Connections: time to write yet another script.

This time I chose VBS over Perl, hoping that this post would be more “instructional”. Perl on Windows is not so common, while VBScript is the standard way to automate stuff on that O.S. (despite the language being incredibly clumsy and annoying¹).

As for the graph format, I chose to output Graphviz DOT format/language.

The script works this way:

Find the current domain.
Find all the Domain Controllers (AD objects of class nTDSDSA, see this) and the Site they’re in.
For each DC/Site, select nTDSConnection objects in NTDS Settings. Of course this is done by means of LDAP queries over ADO, but the view we get is equivalent to what we’re seeing in Active Directory Sites and Services.
Print the DOT graph on standard output: DCs, connections and sites. DCs in the same site will be clustered together.

To use it, first generate the graph’s definition:

cscript /nologo ntdsconnections_graph.vbs > AD-pre.dot

Then use Graphviz’s tools to lay out the graph and turn it into an actual image. For optimal results, I suggest something like:

ccomps -x AD-pre.dot | dot | gvpack -u | neato -Tpng -n2 > AD-pre.png

Here’s what showed up, in my test case:

And here’s the same Domain, after some treatment:

Such graphs may be useful from a Sysadmin point of view, but they’re quite ugly, honestly. I originally thought to use Graphviz to output “some” format, read it in Dia or similar diagram drawing software, and then fix the aesthetics. But Dia support (if it ever worked) has been dropped from Grapviz (December 10, 2009). Dia’s 0.97.1 tarball bears a “dot2dia.py” plugin, but I haven’t hacked it into working. Any other editable format known to Graphviz (e.g.: SVG) doesn’t support “connector” primitives meaning that arrows won’t stick to objects while you drag them around… I’ll follow up if I make some progress.

' A/D replication topology graph (Graphviz .DOT format)
' in the current Domain.
' ----------------------------
' Giuliano - http://www.108.bz

Set objRootDSE = GetObject("LDAP://RootDSE")
strConfigurationNC = objRootDSE.Get("configurationNamingContext")

Set adoCommand = CreateObject("ADODB.Command")
Set adoConnection = CreateObject("ADODB.Connection")
adoConnection.Provider = "ADsDSOObject"
adoConnection.Open "Active Directory Provider"
adoCommand.ActiveConnection = adoConnection

strBase = " & strConfigurationNC & ">"
strFilter = "(objectClass=nTDSDSA)"
strAttributes = "AdsPath"
strQuery = strBase & ";" & strFilter & ";" & strAttributes & ";subtree"

adoCommand.CommandText = strQuery
adoCommand.Properties("Page Size") = 100
adoCommand.Properties("Timeout") = 60
adoCommand.Properties("Cache Results") = False

Set adoRecordset = adoCommand.Execute

Dim dictDCtoSite
Set dictDCtoSite = CreateObject("Scripting.Dictionary")
Dim dictSites
Set dictSites = CreateObject("Scripting.Dictionary")
Dim arrLink()

Function pp(s)
pp = Replace(right(s,len(s)-3), "-", "_") ' trash the leading "CN="
End Function

Do Until adoRecordset.EOF
Set objDC = _
GetObject(GetObject(adoRecordset.Fields("AdsPath").Value).Parent)
Set objSite = _
GetObject(GetObject(objDC.Parent).Parent)
dictDCtoSite.Add objDC.name, objSite.name
if not dictSites.Exists(objSite.name) Then
dictSites.Add objSite.name, 1
End If
adoRecordset.MoveNext
Loop
adoRecordset.Close

For Each strDcRDN in dictDCtoSite.Keys
strSiteRDN = dictDCtoSite.Item(strDcRDN)

strNtdsSettingsPath = "LDAP://cn=NTDS Settings," & strDcRDN & _
",cn=Servers," & strSiteRDN & ",cn=Sites," & strConfigurationNC

Set objNtdsSettings = GetObject(strNtdsSettingsPath)

objNtdsSettings.Filter = Array("nTDSConnection")

For Each objConnection In objNtdsSettings
'WScript.Echo strSiteRDN & " : " & Split(objConnection.fromServer, ",")(1) & " -> " & strDcRDN
ReDim Preserve arrLink(2,k)
arrLink(0,k) = strSiteRDN
arrLink(1,k) = Split(objConnection.fromServer, ",")(1)
arrLink(2,k) = strDcRDN
k = k + 1
Next

Set strNtdsSettingsPath = Nothing
Next

Dim arrSubgraphs()
Redim arrSubgraphs(dictSites.Count-1)

WScript.Echo "Digraph AD {"
WScript.Echo " fontname=helvetica;"
WScript.Echo " node [fontname=helvetica];"
' Same site links
For Each strSiteRDN in dictSites
nosamesitelinks = True
headerwritten = False
For k = 0 To Ubound(arrLink, 2)
If strSiteRDN = arrLink(0,k) Then
if dictDCtoSite.Item(arrLink(1,k)) = dictDCtoSite.Item(arrLink(2,k)) Then
if nosamesitelinks Then
nosamesitelinks = False
WScript.Echo " subgraph cluster_" & pp(strSiteRDN) & " {"
headerwritten = True
End If
WScript.Echo " " & pp(arrLink(1,k)) & " -> " & pp(arrLink(2,k)) & ";"
End If
End If
Next
If headerwritten Then
WScript.Echo " label= """ & pp(strSiteRDN) & """"
WScript.Echo " }"
End If
Next
Wscript.Echo
' Inter-site links
For k = 0 To Ubound(arrLink, 2)
if dictDCtoSite.Item(arrLink(1,k)) <> dictDCtoSite.Item(arrLink(2,k)) Then
WScript.Echo " " & pp(arrLink(1,k)) & " -> " & pp(arrLink(2,k)) & ";"
End If
Next
WScript.Echo "}"

No powerful and convenient data types, no free and ready to use debugger, no public CPAN-like module repository, unnecessarily verbose syntax; I may go on for an hour… ↩