2009
12.27

FortiGate/Cisco Layer 2 woes

Be the first to like.
Share

The other day I swapped a firewall with a different one, a FortiGate 60B. After having re-created the config, everything seemed to be functional but: Internet browsing “felt” a bit sluggish (I was on a 20Mbps uplink) and, here comes the weirdness, when I did “something” the whole WAN connectivity would just hang for a couple of minutes. The issue was reproducible by trying to connect via Remote Desktop to one of the published servers (by tunneling through my Employer’s Office, and bouncing back on the Customer’s firewall) or even by opening my Flickr page (but then the cause could’ve been the poor quality of the pictures therein 😉 ).
At first, I thought about a dreadful MTU issue, maybe the firewall/router or something along the road was choking when fragmenting or reassembling packets. But, a “ping outside_host -s 1472 -M do” (or “ping -f -l 1472 outside_host“, on Windows) proved that ICMP packets 1500 bytes big (1472 bytes of payload, plus 28 bytes of ICMP header) could indeed flow out and back without being fragmented: the issue was totally random.
Besides that, even lowering the MTU on my PC wouldn’t change anything.
After much cursing, I tried to see if anything was going on at L2 level. Firewall and router (Cisco, owned by the ISP, not accessible to me) were connected together by a crossover cable.
The relevant FortiOS CLI command is the following:

FIREWALLNAME # diagnose hardware deviceinfo nic wan1
System_Device_Name              wan1
Link                            up
Speed                           100 Mbps full duplex
FlowControl                     Tx off, Rxoff
MTU_Size                        1500

My firewall (the above example comes out from another one) was negotiating 100Mbps speed, Half Duplex. Nothing wrong with that, I tried to fix these parameters on the FortiGate but the Ethernet link would not come up. So, auto-negotiation was mandatory and I had no way to change that on the router.
At some point, when Internet connectivity was stuck, it seemed to me that unplugging and plugging back in the cable between firewall/router, would allow for a faster recovery. Definitely, something was wrong at L2.
The solution was to insert a 15€ DLink switch between firewall and router. No problems since then, it really looks like FortiGate and Cisco NICs don’t play well together, at least in that conditions. The Customer will call the ISP in order to tweak the settings Cisco side and see if they can get rid of the switch.
The proper way to diagnose the problem would’ve been to ping the router from the outside during a connectivity stop. Since the issue was “local”, the router should answer while no traffic should pass from the firewall to the router.

Share

4 comments so far

Add Your Comment
  1. I believe we are having the same problem here…
    We got about 90 FortiGate 60 firewalls and many Cisco & Cisco/Linksys switches, once awhile the link between the FG firewall and Cisco switch would lock up, and the only fix it to restart the Cisco router. This is causing me a serious headache as I am the only network administrator that look after network devices.
    I tried changing to different speed and duplex when the switch port is locked up but it doesn’t help at all. Also when it is locked up, I turned on the sniffer on the FortiGate firewall, it can’t ‘see’ anthing as if nothing is on the other end…

    I just can’t believe I have to deal with these type of problem and it seems like a basic network 101 but can’t be solved…

    Please do let me know if you have any further information. Much appriciated!

    Cheers,
    Dan

  2. Dear Dan,

    I had the issue while router and firewall were directly attached via a crossover cable. I’m pretty sure that using fixed speed/duplex on both devices would’ve solved it but I had no access to the router. Did you do that? I’d follow these steps:
    – Turn off auto-negotiation and use fixed (say) 100Mbps/Full Duplex. On each of the router’s ethernet ports. On each of the firewall’s ethernet ports. If there are switches in between, on the switches’ ports where firewall/router are connected.
    – Make sure that the issue really lies on the firewall facing side of the router by pinging the router’s WAN interface from outside of the Company.
    – Disconnect a firewall from a router and put a switch between them (or, if there already was one, use a different brand of switch). See if that makes lock-ups go away.

    I’d be glad if you’ll keep me posted about the issue. I noticed that FortiGate auto-negotiation doesn’t play well with ProCurve switches, also (serious performance issues, but no lock-ups). But in any of the cases I faced, turning auto-negotiation off has been enough.

    ciao,

    Giuliano

  3. Hi Giuliano,

    Thanks for replying, I lost this link and couldn’t find it, but I somehow found this again from google today.
    The problem I am having is kind of solved… The problem is caused by both FortiGate 60Bs and the Cisco/Linisys switches that we are using. The Cisco/Linksys (SRW224G4P) switch have a weird bug that will sometimes lock up and as soon as it locks up, all the ports on the switch will stop working and you can’t ping or remote access the switch at all! Recently I found that this might be caused by mismatch of duplex setting between the switch and the device connecting to it (PS. I am already using latest version of firmeware – v1.3.1 for the switch). The problem on the FortiGate is that it just won’t able to connect to the switch with 100full, the switch would connect to FortiGate with 100half and FortiGate would connect to the switch with 100full. After a while of running, the switch would lock up. This happened to Cisco 1800 router as well. The setting on the router is 100full and switch is running with 100half, when this happens, the switch will lock up at least twice a day!
    One last thing, if you have a chance, try run the following two commands on the FortiGate:

    Global mode:
    diagnose hardware deviceinfo nic dmz

    Vdom mode:
    get system interface physical

    I get different duplex results from these 2 commands!! One says half duplex and another says full duplex!!

    To be honest, I had so many of these type of bugs with FortiGate, I am really getting sick and tired of them… And not to mention the speed and ability of their TAC support, it’s slow and unhelpful most of the time.

    So my solution is to set the connection between the FortiGate and Cisco/Linksys switch to 100half, that seems to fix the problem.

    Cheers,
    Dan

  4. Thank you Dan for yet another update. Never experienced such a behaviour though. The issues I had have always been solved by turning off auto negotiation.
    As for Fortigate TAC, I share your feelings. It’s a pity because the boxes themselves are quite nice (compared to other UTM devices in the same market/price segment).

    ciao,

    Giuliano