I used the following command to check my MAC addresses:

Then resorted to the “show mac” switches facilites (some Cisco, some ProCurve) to know on which network ports that particular MAC lied… Only to discover that the cluster’s “logical” MAC address (00:09:0f:09:00:08) wasn’t really located where I expected it to be.
Well, FortiGate’s MAC addresses aren’t randomly generated. They have predictable values that depend on the firewall’s port number. The eight port (or wan1, in my case) will always have a virtual MAC as the one above. What will happen if you have two clusters (as we had) sitting on the same L2 network segment (on the same broadcast domain, that is)? You said MAC address conflict? You’re right.
The solution is simple, use the group-id directive to tweak the logical MAC address, i.e.:

Changes the second right-most bytes of the MAC, from 00 to 0a:

Point is that the “FortiOS High Availablity Handbook” explains the case very thoroughly! See page 192, paragraph “Diagnosing packet loss with two FortiGate HA clusters in the same broadcast domain”. We’re so used to discardable product documentation that sometimes we don’t even try to look for clues where they should normally reside.
Instead of troubleshooting, this time, I should really have Read The (unexpectedly) Fine Manual…

The device that Windows fails to identify is this one:

More info can be found by looking up the IDs in the pci.ids file (as I often do), or by means of the various “Unkown Device Identifier” type of software (e.g. this one). If you have a Linux machine at hand, such a one-liner may suit you:

What’s missing is an Intel SATA driver; needless to say that you won’t find it anywere on HP site.
I downloaded and installed the Rapid Storage Technology Driver from Intel’s web site (here). A 280KB download named “STOR_all32_f6flpy_9.6.0.1014_PV.zip” fixed things up for me.
Maybe the proper thing to try would’ve been the latest (March 2010) Proliant Support Pack, but it’s a big download and I didn’t have the time. Also, the onboard SATA controller isn’t really used (the additional SAS RAID is, instead) and I just wanted to get rid of the yellow warning sign in Device Manager.

And here’s how to add the same “A” record (named “www”) to each of the zones created above.

As you may have guessed this is the typical scenario where you’ve got to re-create some external zones, on the internal DNS servers. That’s needed in order for the internal hosts to reach some server with the “public” DNS name, but the private IP.
For the sake of completeness, let me also mention that you could achieve the same effect by leaving DNS as it is, and configuring “loopback NAT”/”double NAT” on the router/firewall. E.g.: an internal Host wants to reach an internal Server, given it’s public hostname, mapped to a public IP address. It asks the (possibly internal) DNS to translate the name. DNS doesn’t know the zone, it forwards the query to an external DNS Server, obtaining a public IP address that it hands back to the Client. Since its address is non-local, while trying to talk with the Server, the Client sends packets to its default gateway (possibly the router/firewall). The firewall matches the server’s public IP addresses, substituting it with the right private one. It also changes the source IP, swapping the Client’s with the firewall’s LAN address. This way Client and Server are actually communicating through the firewall, even if they’re both internal hosts. And the Server can’t tell Client A from Client B since every connection to it comes from the firewall’s IP address. That’s the main reason why I prefer duplicating the public DNS zones on internal DNS servers, with private IP addresses: you avoid routing internal traffic through the firewall, and avoid NAT where there shouldn’t be any.

Scenario: saturday morning (after having crashed into bed at 4:00 a.m., btw), Customer calls. A virus hit the Company and one of the most annoying consequences of the outburst, is that every domain user account gets locked due to brute-force login attempts (as per the “Account Lockout Threshold” policy). While they run around cleaning PCs and fixing A/V installations¹, I’m asked for a method to quickly unlock the accounts.

I tend to carry out these kind of tasks “the Unix way”, using the available DOS prompt commands and a bit of VBScript.

• Start off by calling LDIFDE:
  ldifde -r "(objectclass=user)" -l sAMAccountName -m -f users.ldf

  LDIFDE exports/imports Active Directory data to/from properly formatted (LDIF) text files. I use it a lot. Ran as shown above, LDIFDE exports the objects of class “user” into a file named users.ldf . Of the many attributes an LDAP object bears, I tell LDIFDE to output just the “sAMAccountName” one. If I hadn’t specified any attribute, in the resulting file I’d have found duplicate DNs for the same user. That’s because of how the resulting LDIF file is described. Some A/D data is “incrementally” added to existing objects given their DN. I just picked sAMAccountName because every user has one and, also, to keep the file small.

• Then:
  findstr /I /b dn.*ou=service.users users.ldf > service_users.txt
  findstr /I /b dn.*cn=users users.ldf > normal_users.txt

  findstr is Microsoft’s “poor man version” of grep, supporting a subset of the regular expression everyone has or should’ve come to love. Here I’m using it to extract Distinguished Names from the LDIF (only the ones that lie in a given Organizational Unit), and saving them to the *_users.txt files. They will look like:

  dn: CN=squidauth,OU=Service Users,DC=contoso,DC=com
  dn: CN=exchangebackup,OU=Service Users,DC=contoso,DC=com
  dn: CN=ldap,OU=Service Users,DC=contoso,DC=com
  dn: CN=batchcopy,OU=Service Users,DC=contoso,DC=com
• Here’s the VBScript function to unlock an account given its DN:
  Sub unlockuser(userDN)
    Set objUser = GetObject ("LDAP://" & userDN)
    objUser.IsAccountLocked = False
    objUser.SetInfo
  End Sub

  We just need to transform findstr’s output, substituting the leading “dn: ” with “unlockuser” and enclosing in double quotes what follows. At the top of the new, transformed, file, we’ll copy/paste unlockuser subroutine definition. That’ll make our final script.

• How to carry out the transform? Using this VBS snippet; it processes its Standard Input line by line, and outputs the modifications on Standard Output, just like any Unix file filtering command.
  Set StdIn = WScript.StdIn
  Do While Not StdIn.AtEndOfStream
      line = stdin.readline
      line = right(line,len(line)-4)
      wscript.echo "unlockuser """ & line & """"
  Loop

  I saved it in a “dnfilter.vbs” file and used it this way:

  type service_users.txt | cscript /nologo dnfilter.vbs > unlock_service_users.vbs

  To obtain something like this:

  unlockuser "CN=squidauth,OU=Service Users,DC=contoso,DC=com"
  unlockuser "CN=exchangebackup,OU=Service Users,DC=contoso,DC=com"
  unlockuser "CN=ldap,OU=Service Users,DC=contoso,DC=com"
  unlockuser "CN=batchcopy,OU=Service Users,DC=contoso,DC=com"

As I said, add the unlockuser function at the top of unlock_service_users.vbs and you’ll have your bulk unlocking script.

Imagine having dumped/sniffed 1GB worth of traffic. We would like to pinpoint a single TCP session, isolating it from the rest. Here’s how we could proceed:

• Identify the source/destination addresses and source/destination ports we’re interested in. Then throw away any packet that doesn’t match this tuple. That’s what Wireshark basically does when you select a packet, right click and hit “Follow TCP Stream”. If the same tuple doesn’t get reused for another, unrelated, session, this method works just fine¹.
• Reorder/reassemble packets.
• Extract packets’ payload.
• Present the payload in a way that makes sense. That depends on the L7 protocol. HTTP without keep-alive is strictly request/response: print what the client sent to the server (outbound traffic) before and then what the server answered (inbound traffic). Other protocols may behave differently and you may choose to separate inbound traffic from outbound, or rely on timing to correctly present the dialogue between peers.

Besides Wireshark, there are tools that do just that and can also be automated. See TShark or tcpflow.

What if you want to script everything and build your own TCP analyzer? Perl’s module Net::Analysis is surprisingly convenient for the task. It does the dirty job I described above and presents your code with ready to be processed TCP sessions.

Practical goal: saving MP3 files streamed by Grooveshark. Disclaimer: I’m by no means pushing anyone to illegally download stuff, this is just a working, sensible, instructional example that uses a song freely available anyway (by Revolution Void, check them out here, they’re great).

GroovesharkListener.pm extends Net::Analysis::Listener::HTTP. It sniffs all the traffic from/to port 80 and, as soon as he sees an HTTP response with a content-type of “audio”, dumps its content to file and quits. Simple as that.

Put the module some place where Perl can find it and then launch (as root):

That’s it, just one more thing. Net::Analysis doesn’t allow you to select a specific network interface, it just picks up the first available one. I wrote a small patch to address this shortcoming, it adds a “device=” parameter that you can use this way:

And here’s what GroovesharkListener.pm looks like:

This time I chose VBS over Perl, hoping that this post would be more “instructional”. Perl on Windows is not so common, while VBScript is the standard way to automate stuff on that O.S. (despite the language being incredibly clumsy and annoying¹).

As for the graph format, I chose to output Graphviz DOT format/language.

The script works this way:

• Find the current domain.
• Find all the Domain Controllers (AD objects of class nTDSDSA, see this) and the Site they’re in.
• For each DC/Site, select nTDSConnection objects in NTDS Settings. Of course this is done by means of LDAP queries over ADO, but the view we get is equivalent to what we’re seeing in Active Directory Sites and Services.
• Print the DOT graph on standard output: DCs, connections and sites. DCs in the same site will be clustered together.

To use it, first generate the graph’s definition:

Then use Graphviz’s tools to lay out the graph and turn it into an actual image. For optimal results, I suggest something like:

Here’s what showed up, in my test case:

And here’s the same Domain, after some treatment:

Such graphs may be useful from a Sysadmin point of view, but they’re quite ugly, honestly. I originally thought to use Graphviz to output “some” format, read it in Dia or similar diagram drawing software, and then fix the aesthetics. But Dia support (if it ever worked) has been dropped from Grapviz (December 10, 2009). Dia’s 0.97.1 tarball bears a “dot2dia.py” plugin, but I haven’t hacked it into working. Any other editable format known to Graphviz (e.g.: SVG) doesn’t support “connector” primitives meaning that arrows won’t stick to objects while you drag them around… I’ll follow up if I make some progress.

I wasn’t completely satisfied with NetEnforcer reporting functions and wanted something more dependable and realtime. Well, if you turn to the device’s CLI access (SSH), you’ll notice an interesting acthruput command.
It shows the current throughput per Interface, Line, Pipe and Virtual Channel. What more could you ask for?

As you can see, acthruput identifies Pipes by number. How do you relate this number to the actual mnemonic pipe name? Use “acstat -l pipe“, which also displays the total number of live connections per pipe .

Wrap acthruput in a while loop that adds a timestamp and a delay (→ sampling frequency). Start your terminal emulator logging facilities, hit enter, wait, ctrl-c, stop logging.

Eventually, clean the log a bit and feed it to the Perl script you’ll find at the end of this post.

The script outputs CSV formatted data:

And here’s what it looks like when opened up in OpenOffice Calc (sorry, no fancy formatting).

The graph above shows that the 8Mbps link (the “Line”, in Allot’s parlance) is not saturated. Problem was that, during that timeframe, we were also trying to make Iperf “consume” all of the available bandwidth. We couldn’t make it because one of the firewalls was acting as a bottleneck if presented with certain workloads (many connections, see this) . Being able to generate these kinds of report proved very useful in troubleshooting…

• A locally generated email whose destination is inside the company, should leave Postfix with a @FQDN suffix (@hostname.localdomain.lan) in its sender addresses. Sender addresses shouldn’t be rewritten/masqueraded at all.
• A locally generated email whose destination is outside of the company, needs to be masquerated, its sender addresses rewritten as @extdomain.com .

Moreover, but that’s a routing matter rather than a rewriting one:

• Emails directed to @smsgw.localdomain.lan have to be relayed through a different mail server.

As you can see, the logic is: lookup B (rewritten sender) given A (sender) depending on C (recipient).

I found the right hint deeply buried in Postfix’s mailing list: check out Noel Jones post, kudos to him.

• First, define a new smtp transport in “master.cf”; just copy/paste the existing one and change the first column to whatever name you like. We are explicitly telling the new transport that it will use its own generic regexp map (the -o command-line option).
  [root@hostname postfix]# cd /etc/postfix
  [root@hostname postfix]# grep '^\(smtp\|toext\).*unix' master.cf
  smtp      unix  -       -       n       -       -       smtp
  toext     unix  -       -       n       -       -       smtp -o smtp_generic_maps=regexp:/etc/postfix/generic_toext
• We also need to take control over the mail routing mechanism. This is done by enabling transport maps.
  [root@hostname postfix]# grep ^transport main.cf
  transport_maps = regexp:/etc/postfix/transport
• Transport maps (remember that they’re matched against From addresses) are configured in order to:
  • Route mail that should be delivered locally through the local transport. This will preserve /etc/aliases and .forward behaviour and make everything act like you expect on Unix.
  • Route mail to @smsgw.localdomain.lan, via its dedicated gateway, using the “standard” smtp transport.
  • Route mail to @localdomain.lan, through the main SMTP gateway, using the smtp transport.
  • Route any other message through the main SMTP gateway, but use our custom transport.
  [root@hostname postfix]# tail -4 transport
  /@hostname\.localdomain\.lan$/  local:hostname.localdomain.lan
  /@smsgw\.localdomain\.lan$/     smtp:[smsgw.localdomain.lan]
  /@localdomain\.lan/             smtp:[gateway.localdomain.lan]
  /@.*$/                          toext:[gateway.localdomain.lan]
• The custom transport’s generic map rewrites recipient adresses, shortening the FQDN by preserving just the domain name, and changing the address part before the @ sign. Hostname is being stripped but I still want to be able to tell, at a glance, from where the message originates. When they leave the mail system, rewritten addresses look like username-hostname@extdomain.com .
  [root@hostname postfix]# cat generic_toext
  /^(.*)@hostname\.localdomain\.lan$/ $1-hostname@extdomain.com

How do you find out why this is happening?

The possible causes boil down to:

1. Router/Firewall¹ is not pleased by “something”. Could be an attack or a bug in the device firmware.
2. Too many connections. Maybe they’re not passing much traffic, but the internet gateway can’t keep up with their number. I’ve seen firewalls perform very badly in this respect. E.g.: 3 connections trying to download/upload as fast as they can, and a total, aggregate, b/w of 10Mbps. Those 3 plus 3000 “normal” connections and a total b/w of 6Mbps.
3. A reasonable amount of connections, effectively eating all of the available bandwidth.

I’ll skip case A, for now.
In case B you’ll likely want to know the firewall’s idea of “netstat”, meaning the complete listing of TCP/UDP/other connections. No big deal if the device has got some sort of CLI access: capture its output, import it into a spreadsheet, or use awk/sort/grep² to build your stats. Usually, computing total number of connections by source IP address and sorting accordingly, is enough to gain some insight about what’s going on.
Case C… For long-running (days) data analysis, you could use a tool like NTOP. But if, like me today, you need to act quickly (perhaps because you know that the issue will disappear soon), iftop can hardly be beaten.
Both tools require the machine they run on to be able to “sniff” all the traffic passing through the firewall. This can be accomplished by configuring monitoring/monitored port(s) on a switch. Monitored ports get their inbound/outbound traffic copied to the monitoring one. Different vendors call the thing a different way, port mirroring is also a good keyphrase. Here are a couple of resources:

• (Old) 3Com Superstack: Monitor Port on 3Com 4400
• HP ProCurve, pretty straightforward to set up using the “menu” interface: How do I attach a LAN Analyzer to a Switch 208t/224t port to monitor LAN traffic for diagnostic purposes?
• ProCurve switches are not limited to mirroring ports that belong to the same device/chassis: How to configure remote and intelligent mirroring on ProCurve switches
• Low-end HP switches (like the ProCurve 1800 one I encountered here), though, are only manageable via a web gui:
  
  

  Port Mirroring on a ProCurve 1800

• Cisco: Port Mirroring, Configuring a Cisco Catalyst Switch SPAN mirroring port

(You could as well use a hub instead of a switch and get implicit mirroring of any port, to any port of the hub. Just unplug the firewall, link the hub to the switch, plug firewall and monitoring host in the hub. Kludgy but quick and easy, if you can afford the temporary cabling changes, and the bottleneck introduced by the hub…)

So:

• Find the switch where the firewall is connected to. Which side of the firewall? It depends on where you believe the issues originates from. Let’s say the culprit is most likely to lie on the LAN → switch port A.
• Connect your laptop/monitoring machine to the same switch → port B.
• Set up monitoring: port A is monitored, port B is monitoring.
• Run iftop, maybe telling it to also show port numbers (“-P”, without this switch, you’ll only see totals by source/destination IP addresses couple), don’t display hostnames “-n”, the interface “-i eth0″ and provide a meaningful filter (here I’m selecting packets whose source is not on the LAN³. The “-p” option instructs iftop to capture packets in promiscuous mode. Without it, iftop won’t lift off the wire packets that aren’t addressed to the machine on which it is running.
  iftop -p -P -n -i eth0 -f 'not src net 192.168.200.0/23'

  Iftop will produce a realtime table of running connections, sorted by how demanding they are in terms of bandwidth (10s average, by default). See the screenshot below; the top connections are due to two running video conference streams stealing 1Mbit/second worth of bandwidth, each.
  

  

  iftop's output

  
  Once everything is set up and you’re able to read iftop’s output, spotting the “top talkers” of your net becomes kids play, enjoy!
  1. for brevity, I’ll just say “firewall” from now on. ↩
  2. Yuri is king at doing that. See his AWK weekly series. ↩
  3. iftop will still show these source addresses, since its output is always made of bidirectional “connections”. Only, counters pertaining to the LAN → outside direction, won’t increase. ↩
  ]]> http://www.108.bz/posts/it/who-ate-all-the-bandwidth/feed/ 2