108.bz » FortiGate

FortiGate firewall clusters group-id

Giuliano — Sat, 03 Jul 2010 15:53:31 +0000

A newly installed FortiGate cluster (a simple two node HA active-passive setup) and some packet loss issues…
Ping from the LAN side to the Internet (or from the firewall itself) resulted in about 20% packet loss, while the other way around (WAN to firewall’s main public IP) didn’t work at all.

I used the following command to check my MAC addresses:

FORTIGATE-PRI # diagnose hardware deviceinfo nic wan1
[..]
Current_HWaddr 00:09:0f:09:00:08
Permanent_HWaddr 00:09:0f:d1:be:ef
[..]

Then resorted to the “show mac” switches facilites (some Cisco, some ProCurve) to know on which network ports that particular MAC lied… Only to discover that the cluster’s “logical” MAC address (00:09:0f:09:00:08) wasn’t really located where I expected it to be.
Well, FortiGate’s MAC addresses aren’t randomly generated. They have predictable values that depend on the firewall’s port number. The eight port (or wan1, in my case) will always have a virtual MAC as the one above. What will happen if you have two clusters (as we had) sitting on the same L2 network segment (on the same broadcast domain, that is)? You said MAC address conflict? You’re right.
The solution is simple, use the group-id directive to tweak the logical MAC address, i.e.:

config system ha
set group-id 10
end

Changes the second right-most bytes of the MAC, from 00 to 0a:

before 00:09:0f:09:00:08
after 00:09:0f:09:0a:08

Point is that the “FortiOS High Availablity Handbook” explains the case very thoroughly! See page 192, paragraph “Diagnosing packet loss with two FortiGate HA clusters in the same broadcast domain”. We’re so used to discardable product documentation that sometimes we don’t even try to look for clues where they should normally reside.
Instead of troubleshooting, this time, I should really have Read The (unexpectedly) Fine Manual…

FortiGate High Availability interface addressing

Giuliano — Thu, 07 Jan 2010 15:01:51 +0000

When setting up High Availability on FortiGate, one thing struck me as a bit unusual. Differently from other firewall clustering solutions (correct me if I’m wrong), FortiGate devices don’t force you to assign both physical and logical IP addresses on interfaces. You are supposed to configure logical IP addresses only. This implies that you can’t directly access a specific node/firewall in your cluster. You have to SSH into the Master unit and, from there, log into the Subordinate one(s). Here are the relevant CLI commans:

FW-NODE-A # get system ha status
Model: 100
Mode: a-p
Group: 0
Debug: 0
ses_pickup: disable
Master:129 FW-NODE-A FG100C3000000000 0
Slave :128 FW-NODE-B FG100C3000000001 1
number of vcluster: 1
vcluster 1: work 169.254.0.1
Master:0 FG100C3000000000
Slave :1 FG100C3000000001

FW-NODE-A # execute ha manage ?
please input peer box index.
<1> Subsidary unit FG100C3000000001

FW-NODE-A # execute ha manage 1

FW-NODE-B $

I wonder what would happen if the Master unit were to hang. I mean: stuck itself in a state where the failover mechanism doesn’t work and neither does SSH/HTTPS access. How could you remotely force a failover to another node? In such a scenario, is a physical power cycle of the master unit the only option?

FortiGate/Cisco Layer 2 woes

Giuliano — Sun, 27 Dec 2009 13:41:17 +0000

The other day I swapped a firewall with a different one, a FortiGate 60B. After having re-created the config, everything seemed to be functional but: Internet browsing “felt” a bit sluggish (I was on a 20Mbps uplink) and, here comes the weirdness, when I did “something” the whole WAN connectivity would just hang for a couple of minutes. The issue was reproducible by trying to connect via Remote Desktop to one of the published servers (by tunneling through my Employer’s Office, and bouncing back on the Customer’s firewall) or even by opening my Flickr page (but then the cause could’ve been the poor quality of the pictures therein ).
At first, I thought about a dreadful MTU issue, maybe the firewall/router or something along the road was choking when fragmenting or reassembling packets. But, a “ping outside_host -s 1472 -M do” (or “ping -f -l 1472 outside_host“, on Windows) proved that ICMP packets 1500 bytes big (1472 bytes of payload, plus 28 bytes of ICMP header) could indeed flow out and back without being fragmented: the issue was totally random.
Besides that, even lowering the MTU on my PC wouldn’t change anything.
After much cursing, I tried to see if anything was going on at L2 level. Firewall and router (Cisco, owned by the ISP, not accessible to me) were connected together by a crossover cable.
The relevant FortiOS CLI command is the following:

FIREWALLNAME # diagnose hardware deviceinfo nic wan1
System_Device_Name wan1
Link up
Speed 100 Mbps full duplex
FlowControl Tx off, Rxoff
MTU_Size 1500

My firewall (the above example comes out from another one) was negotiating 100Mbps speed, Half Duplex. Nothing wrong with that, I tried to fix these parameters on the FortiGate but the Ethernet link would not come up. So, auto-negotiation was mandatory and I had no way to change that on the router.
At some point, when Internet connectivity was stuck, it seemed to me that unplugging and plugging back in the cable between firewall/router, would allow for a faster recovery. Definitely, something was wrong at L2.
The solution was to insert a 15€ DLink switch between firewall and router. No problems since then, it really looks like FortiGate and Cisco NICs don’t play well together, at least in that conditions. The Customer will call the ISP in order to tweak the settings Cisco side and see if they can get rid of the switch.
The proper way to diagnose the problem would’ve been to ping the router from the outside during a connectivity stop. Since the issue was “local”, the router should answer while no traffic should pass from the firewall to the router.