The USB disk (shown below as /dev/sdb*) “feels” fast when reading and awfully slow when writing. The most simple way to do an HDD benchmark is, of course dd. Use it along with dstat (an essential tool when pinpointing performance issues, whatever they may be) and you’ll quickly gather some useful figures. Beware! dd can ruin all your data just by mistaking a “b” for an “a”: triple-check and make sure that you’re running it on the right devices!

A sequential write test:

3.7 MB/s only, definitely slow. Note that you shouldn’t use /dev/{random,urandom} as input file, they’re a bottleneck by themselves. /dev/zero, on the other hand, is super-fast. “dd if=/dev/zero of=/dev/null bs=16384 count=$((10000*1024))” (shove zeros to /dev/null) is bound only by the CPU, running at about 9.3 GB/s here.

A sequential read test:

30 MB/s, that’s the order of magnitude I was expecting (confirmed here).

If I repeat the write test and, at the same time, run dstat, I notice that there are no burst or drops: speed is constant.

Since reading works, kernel and USB Host Controller seem to go along well. Issue should lie on the disk’s side. I had no clue of what was happening until I tried writing straight to the disk instead of the first primary partition (i.e.: /dev/sdb instead of /dev/sdb1), thus trashing the filesystem (I’ve got no data to lose on that disk: no worries).

Even though the difference between read and write throughput seems to be too much (almost one order of magnitude), this is starting to look like a FS blocksize/partition aligment issue. Well, some disks use a physical sector size (PSS) of 512 bytes. Others use 4096 bytes (4 KiB). Others use the latter but tell the OS that they’re using 512 bytes or more simply the OS can’t figure out the right physical sector size… And USB mass storage devices tell the OS almost nothing (hdparm won’t help this time)…

Filesystem (or other “structured storage” systems like, for instance, datafiles in databases) organize their data in blocks. The block size can sometimes be adjusted. 4096 bytes is a quite common value:

A sector represents the smallest chunk of data that can be read/written from/to a disk. If its size is 512, and the filesystem block size is 4096, the filesystem driver will read/write batches of 8 sectors. Better said: the FS thinks to deal with 4k blocks, not knowing that lower level functions will further split them in eight (if only logically).
Consider another example: the PSS is 4096, but the drive acts as if it was 512. Physical sectors can be found at absolute offset sector_number*512*8 (0, 4096, 8192, …). What if a 4k write operation happens at offset 1*512*8-512? (3584, it doesn’t look like a “bad” offset: as far as the OS is concerned, any multiple of 512 is fine). The drive, being unable to write less than 4k and at proper locations, will: read sector 0, read sector 1, modify the last 512 bytes chunk of sector 0, modify the first three chunks of sector 1 then write both sectors (or something similar). If things were properly aligned, a single write operation would’ve sufficed. Read operations speed, on the other hand, may be almost unaffected. Think about it: unless you’re dealing with tons of 4k files spread across sector couples (i.e.: two sectors read instead of one), large chunks of data are (hopefully) laid out sequentially of the disc. Reading 1GB plus 512 bytes, instead of 1GB alone, won’t change anything benchmarks.

What’s up with my partition?

sfdisk (one of fdisk‘s cousins) shows that the partition starts at byte 63*512=32256. This value isn’t divisible by 4096, yielding a non integer result. Sector 64, instead, is a good place to start an aligned partition:

63*512/4096 = 7.87
64*512/4096 = 8.00

Similarly, other partitions should start at sectors that are multiples of 8 (because 512*8=4096).
This is the corrected partition table. Moving the partition forward (by a mere 512 bytes) causes a 10x write speed increase.

You may still have a question though. Does aligning a partition mean that the contained filesystem is aligned too? You’re right, that assumption should not be taken for a fact.

A filesystem is made up of “your” data and “its” data (the latter being internal structures necessary to organize the former). In any case, a FS will try to pad/align stuff to the block size. That was to say that, if you start a partition and the partition is aligned to a given boundary, the filesystem (all of its composing blocks) will be aligned too.

You can find a description of the ext2 layout here. Partition starts with two sectors reserved for the boot loader. Then comes a 1k chunk holding ext2 “superblock”. At offset 56 within the superblock, we should find the ext2 magic number (0x53EF), and here it is:

The next byte after the superblock, is byte 4096. From then on, everything happens (from the FS point of view) in chunks as big as the configured block size. My disk is a 4k sector disk, formatted with a single partition aligned (as the FS) to a 4K block. FS block size is 4K too. Can’t do really do any better than that besides choosing a filesystem that manages to handle the given workload with fewer read/write operations, but I digress…

• If anything, the flipping shows that failover works as expected (VMs don’t crash despite the chaos).
• That said, I could just disconnect a controller. Not really because the same storage system hosts an Oracle RAC cluster, humming along happily, unaffected by the issue.
• I need a way to selectively “hide” a controller from one or more hosts. I can do it easily by tweaking the SAN zoning configuration.

A Zone (much like a VLAN) is basically a group of WWNs (or ports). Objects in the Zone can only talk to each other. While creating Zones, it is common practice to “go minimal”: they should contain as few stuff as possible. I usually name them like this:
    Z_HOSTNAME_P1_DS4800_CA1_CB1
HBA Port 1 of HOSTNAME can see Controller A/Port 1 and Controller B/Port 1 of the DS4800.
Thus, going through each ESX server’s Zone, I just remove the Controller that the host shouldn’t see. Path thrashing is temporarily stopped.
The above rant serves mainly as a pro-zoning argument. “If every HBA port has to access every Controller’s port, why implement zoning?”. As you just read, zoning saved me from serious trouble, today.
About the “real” issue, it was ultimately caused by a thing called “Auto Volume Transfer” (AVT)¹. Let’s say that a LUN is assigned to controller A, but I/O for the LUN is issued to controller B. With AVT switched on the storage system will automatically transfer the LUN from A to B.
The Customer ESX servers are all (correctly) configured to use the “Most Recently Used” (MRU) path to a LUN. It seems that ESX, from a certain version on, issues I/O on the standby path, causing havoc if AVT is on. I can’t tell if that’s because it is fooled into thinking that the storage is an Active/Active one or if it just sort of periodically “probes” standby paths.
How do you switch AVT off? By using the DS “Storage Manager” and changing the ESX Hosts’ type from “Linux” (or whatever) to “LNXCLVMWARE”. This applies to all of the LSI derived Storage Systems (IBM, SUN StorageTek, Engenio, …). The latter host type is the right one to use when hooking an ESX cluster to an IBM DS Storage System. But “Linux” seems to do just fine on not so new ESX hosts version 4.1.x … When AVT is off, the Storage will decide to trespass LUNs only in the event of an internal hardware failure while, normally, LUN ownership will be handled by the multipathing software on the Host.

More reading on the subject:

[1] Differences between the “Linux” and “LNXCLVMWARE” host types.
[2] How does Auto Volume Transfer (AVT) work? Courtesy of Google’s cache. Lists which SCSI commands trigger AVT.
[3] A really nice blog post about the same issue described here. (Found, of course, when I was writing mine)

Here are the relevant details:

• OS: Red Hat Enterprise Linux Server 5.3 x86-64
• GC Agent: Oracle Enterprise Manager 10g Grid Control Release 3 (10.2.0.3) for Linux x86-64
• GC Console: Oracle Enterprise Manager 10g Release 5 (10.2.0.5) Grid Control for Microsoft Windows 32-bit

And here’s the error message (the most interesting portions):

To go past this show-stopper I tried a few things…

The Heap report produced by java at crash time, seemed to indicate a memory shortage. By editing the “install/oraparam.ini” file, you can tweak how much RAM is available for OUI’s JVM. Just alter “JRE_MEMORY_OPTIONS” value.

This is also a safe place to put additional command line parameters: they’ll mostly be passed to java’s command line. I said “mostly” because OUI wrapper/launcher seems to check some sort of allowed parameters list and may refuse to go on if somethings doesn’t look right.

The “-XX:MaxPermSize=32m” is one of the knobs that doesn’t pass the sanity check. In order to run OUI’s JVM by hand, with the right parameters, just keep the first lines of runInstaller (the ones starting with ‘Arg:‘):

Strip “^Arg:“, “^\d*:“, “:$“, add a trailing “ \” and you’ll have an OUI launching shell script you can alter at will.

Increasing JVM’s memory led to no effect. Heap report looked fine (usage percentages went down) but crash was still there.

Another useful switch is “-XX:+ShowMessageBoxOnError“. It makes java halt on error, allowing us to attach a debugger and perform a stack backtrace, e.g.:

I also tried to “inject” a couple of newer JVM’s into the stage directory. The quickest way is to borrow it from another installer.

The server’s has a “working” directory were Oracle patches/products are stored before use. In my case, changing OUI’s JVM from 1.4.2.8 to 1.4.2.14 is a matter of copying:

to:

Then modifing the same “oraparam.ini” file mentioned before.

You could as well download a specific JRE from http://java.sun.com (sorry: from Oracle) and:

• install the new JRE somewhere
• unzip (-t) the “filegroup1.jar” file that corresponds to OUI’s “factory” JRE. Note how the directories are laid out (something like: “jre/1.4.2″). Modify the new JRE accordingly.
• zip the new JRE, rename the resulting file to “filegroup1.jar”, copy it in the right place.
• modify oraparam.ini and choose the JVM version you’ll boot OUI into.

Three different JREs, each of them segfaulting in the same spot, as we saw in the backtrace:

Who’s the owner of libXt?

After making sure that none of the running processes was using that package contents, I decided to remove it (rpm -e –nodeps libXt-1.0.2-3.1.i386) and reinstall it. Surprisingly, OUI worked flawlessy after this last action. Too bad I can’t really explain why. libXt version didn’t change before/after reinstall. I should diff it anyway with what’s left untouched on other RAC cluster members. I’ll update the post when I have a stricter explanation…

There are several posts mentioning the issue, this one pointed me in the right direction. Basically, because of how SEP components communicate, Windows is triggered into updating the list of trusted root Certification Authorities. It tries to do so through the Internet using the Computer account. The latter may not have any proxy configured. Being unable to reach outside, the host gets flooded by crypt32 errors.

In order to solve the issue, I decided to deploy a valid proxy configuration, for the Computer account (SYSTEM user), on a subset of the Domain’s hosts.
One of the ways to script that is the “proxycfg -u” command¹ that works by copying the current user proxy settings to the SYSTEM’s registry. Sounds cool but if the current user is not a member of the local Administrators group, he won’t have the necessary rights. The following script instead, can be launched via Group Policy² during operating system startup, and since it’s a startup script rather than a login one, it will run with administrative privileges.

Nothing fancy in the below source. It creates the registry key if it doesn’t exist, then sets the right value for WinHttpSettings which I obtained this way:

• use “proxycfg -u” on a test host
• use the Registry editor to export the contents of HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\Connections

The value is of type REG_BINARY. Since the RegWrite API (method of class WScript.Shell) cannot deal with binary values, WMI (StdRegProv registry provider) needs to be used. Also, SetBinaryValue expects an array of decimal values, while Regedit exports them as hexadecimal digits (you’ll have to take care of the conversion yourself).

If the scripts works as it should, you’ll be greeted by these events:

And, hopefully, crypt32 errors will be gone for good.

We’re running Oracle Application Server 10g Release 2 on Windows.

More than one Reports Server (RS from now on) can be run at the same time. Each RS is identified by name. Open a DOS window, change directory to the one where reports templates/resources are, then launch:

RS named “rstest” will get its own log directory and configuration file, under the Application Server “HOME”:

Quite convenient: you’ll leave the production RS alone, be able to activate debug tracing, restart at will, …

Here’s how tu run a test report on the “rstest” RS:

Back to our issues. First step, I’d say, is to find out the exact name of the font we’d like to embed. Did you know you can convert Report Developer “source” files (.rdf) to XML, then peek in them? Use:

“Cooper Black” is the name:

Install the font in Windows. Oddly, things didn’t seem to work for me when I just copied the .ttf file in the C:\WINDOWS\Fonts directory. I had to use the “Install New Font” menu item. I thought both methods were equivalent, maybe I was wrong, maybe I’m talking junk now.

Modify the uifont.ali file (tools\common), telling the Report Server to subset “Cooper Black” into the generated PDF files. Just add a line under the PDF:Subset section, equating the font name to the TTF file name, both enclosed in double quotes.

The fnchk.exe program (ran without arguments) displays the full path of uifont.ali and tells if everything is fine or the file contains any syntax error.

Installing the font in Windows is not enough, you should also put it in one of the REPORTS_PATH directories. The value of this variable can be found in the registry, I chose: C:\Oracle\Products\FRHome\reports\templates. FileMon is essential when trying to see which files can’t be found by a process.

Time to restart the Reports Service, generate a PDF report and check if fonts are good (in Acrobat Reader: CTRL-D, “Fonts” tab). Ours is listed as “Embedded Subset”; we’re done.

See also: Oracle Support Note.350971.1 “Troubleshooting Guide for Font Aliasing / Font Subsetting / Font Embedding Issues”.

I wasn’t completely satisfied with NetEnforcer reporting functions and wanted something more dependable and realtime. Well, if you turn to the device’s CLI access (SSH), you’ll notice an interesting acthruput command.
It shows the current throughput per Interface, Line, Pipe and Virtual Channel. What more could you ask for?

As you can see, acthruput identifies Pipes by number. How do you relate this number to the actual mnemonic pipe name? Use “acstat -l pipe“, which also displays the total number of live connections per pipe .

Wrap acthruput in a while loop that adds a timestamp and a delay (→ sampling frequency). Start your terminal emulator logging facilities, hit enter, wait, ctrl-c, stop logging.

Eventually, clean the log a bit and feed it to the Perl script you’ll find at the end of this post.

The script outputs CSV formatted data:

And here’s what it looks like when opened up in OpenOffice Calc (sorry, no fancy formatting).

The graph above shows that the 8Mbps link (the “Line”, in Allot’s parlance) is not saturated. Problem was that, during that timeframe, we were also trying to make Iperf “consume” all of the available bandwidth. We couldn’t make it because one of the firewalls was acting as a bottleneck if presented with certain workloads (many connections, see this) . Being able to generate these kinds of report proved very useful in troubleshooting…

How do you find out why this is happening?

The possible causes boil down to:

1. Router/Firewall¹ is not pleased by “something”. Could be an attack or a bug in the device firmware.
2. Too many connections. Maybe they’re not passing much traffic, but the internet gateway can’t keep up with their number. I’ve seen firewalls perform very badly in this respect. E.g.: 3 connections trying to download/upload as fast as they can, and a total, aggregate, b/w of 10Mbps. Those 3 plus 3000 “normal” connections and a total b/w of 6Mbps.
3. A reasonable amount of connections, effectively eating all of the available bandwidth.

I’ll skip case A, for now.
In case B you’ll likely want to know the firewall’s idea of “netstat”, meaning the complete listing of TCP/UDP/other connections. No big deal if the device has got some sort of CLI access: capture its output, import it into a spreadsheet, or use awk/sort/grep² to build your stats. Usually, computing total number of connections by source IP address and sorting accordingly, is enough to gain some insight about what’s going on.
Case C… For long-running (days) data analysis, you could use a tool like NTOP. But if, like me today, you need to act quickly (perhaps because you know that the issue will disappear soon), iftop can hardly be beaten.
Both tools require the machine they run on to be able to “sniff” all the traffic passing through the firewall. This can be accomplished by configuring monitoring/monitored port(s) on a switch. Monitored ports get their inbound/outbound traffic copied to the monitoring one. Different vendors call the thing a different way, port mirroring is also a good keyphrase. Here are a couple of resources:

• (Old) 3Com Superstack: Monitor Port on 3Com 4400
• HP ProCurve, pretty straightforward to set up using the “menu” interface: How do I attach a LAN Analyzer to a Switch 208t/224t port to monitor LAN traffic for diagnostic purposes?
• ProCurve switches are not limited to mirroring ports that belong to the same device/chassis: How to configure remote and intelligent mirroring on ProCurve switches
• Low-end HP switches (like the ProCurve 1800 one I encountered here), though, are only manageable via a web gui:
  
  

  Port Mirroring on a ProCurve 1800

• Cisco: Port Mirroring, Configuring a Cisco Catalyst Switch SPAN mirroring port

(You could as well use a hub instead of a switch and get implicit mirroring of any port, to any port of the hub. Just unplug the firewall, link the hub to the switch, plug firewall and monitoring host in the hub. Kludgy but quick and easy, if you can afford the temporary cabling changes, and the bottleneck introduced by the hub…)

So:

• Find the switch where the firewall is connected to. Which side of the firewall? It depends on where you believe the issues originates from. Let’s say the culprit is most likely to lie on the LAN → switch port A.
• Connect your laptop/monitoring machine to the same switch → port B.
• Set up monitoring: port A is monitored, port B is monitoring.
• Run iftop, maybe telling it to also show port numbers (“-P”, without this switch, you’ll only see totals by source/destination IP addresses couple), don’t display hostnames “-n”, the interface “-i eth0″ and provide a meaningful filter (here I’m selecting packets whose source is not on the LAN³. The “-p” option instructs iftop to capture packets in promiscuous mode. Without it, iftop won’t lift off the wire packets that aren’t addressed to the machine on which it is running.
  iftop -p -P -n -i eth0 -f 'not src net 192.168.200.0/23'

  Iftop will produce a realtime table of running connections, sorted by how demanding they are in terms of bandwidth (10s average, by default). See the screenshot below; the top connections are due to two running video conference streams stealing 1Mbit/second worth of bandwidth, each.
  

  

  iftop's output

  
  Once everything is set up and you’re able to read iftop’s output, spotting the “top talkers” of your net becomes kids play, enjoy!
  1. for brevity, I’ll just say “firewall” from now on. ↩
  2. Yuri is king at doing that. See his AWK weekly series. ↩
  3. iftop will still show these source addresses, since its output is always made of bidirectional “connections”. Only, counters pertaining to the LAN → outside direction, won’t increase. ↩
  ]]> http://www.108.bz/posts/it/who-ate-all-the-bandwidth/feed/ 2 http://www.108.bz/posts/it/executing-processes-as-the-system-user/ http://www.108.bz/posts/it/executing-processes-as-the-system-user/#comments Fri, 08 Jan 2010 22:05:08 +0000 Giuliano http://www.108.bz/?p=179 On MS Windows operating systems, many processes run under the NT AUTHORITY\SYSTEM account, be them scheduled tasks or services.
  Sometimes it’s useful to run cmd.exe as the SYSTEM user and see what’s going on. Here’s a nifty trick to do it.
  C:\Documents and Settings\giuliano>time /t
  17:10
  
  C:\Documents and Settings\giuliano>at 17:11 /interactive cmd.exe
  Added a new job with job ID = 1
  
  C:\Documents and Settings\giuliano>

  Basically you check what time it is and schedule cmd.exe to run on the next minute. You do that by means of the at.exe OS command.

  When the time comes, a Command Prompt window should pop-up. It runs under the SYSTEM account:

  Microsoft Windows [Version 5.2.3790]
  (C) Copyright 1985-2003 Microsoft Corp.
  
  C:\WINDOWS\system32>whoami
  nt authority\system
  
  C:\WINDOWS\system32>

  Each process you run from there, also runs as SYSTEM. If you run regedit.exe, for instance, you can import registry data into the SYSTEM user’s hive. Today I used this tecnique to export/import Putty’s settings (they are stored in the registry) in order to make plink.exe, as run from a UPS monitoring Agent, see a pre-configured SSH “Session” (hostname, login username, private key, …). I needed the Agent to shut down a bunch of Linux servers when the battery charge was running low: plink.exe on Windows side and sudo on the Linux one, did the job.

  For completness sake, here‘s a post on the same subject. It also deals about Vista/Windows Server 2008 and how to achieve our goal using PsExec.

  ]]> http://www.108.bz/posts/it/executing-processes-as-the-system-user/feed/ 0 http://www.108.bz/posts/it/true-shell-access-on-symantec-brightmail/ http://www.108.bz/posts/it/true-shell-access-on-symantec-brightmail/#comments Tue, 29 Dec 2009 10:27:54 +0000 Giuliano http://www.108.bz/?p=118 By connecting (SSH, “admin” user) to a Symantec Brightmail Gateway appliance ¹, you are left in a restricted shell where only a limited set of commands is available. The undocumented “set-support” command may come in handy: it assigns a temporary password to the “support” user, a normal unix account with a standard shell.
  giuliano@balrog ~ $ ssh admin@192.1.2.3
  admin@192.1.2.3's password:
  
  bmail> set-support
  Warning: Do NOT execute this script without explicit direction from a Symantec
  Customer Support person.
  Changing password for user support.
  New password:
  BAD PASSWORD: it is based on a dictionary word
  Retype new password:
  passwd: all authentication tokens updated successfully.
  User support enabled until 01/04/2010.
  bmail> logout
  Connection to 192.1.2.3 closed.
  
  giuliano@balrog ~ $ ssh support@192.1.2.3
  support@192.1.2.3's password:
  
  [support@bmail support]$ echo $SHELL
  /bin/bash

  What’s nice about the “support” user is that he can run tcpdump and access useful logfiles, e.g.:

  support@bmail support]$ tail -f /data/logs/stats.maillog
  2007 Mar 30 11:36:17 (info) delivery-mta/smtp[2008]: 45A689A9: to=, relay=192.1.2.3[192.1.2.3], delay=0, status=sent (250 OK)

  A note about the restricted shell command “watch maillog” and the “/data/logs/stats.maillog” file.
  The latter is the truly useful MTA log file (holding a realtime record of which messages are relayed through the appliance), while the “watch maillog” command shows entirely different stuff. There used to be a proper “watch stats.maillog” command, but at some point Symantec decided to remove it, can’t really tell why.

  I originally learnt about the “set-support” command here (Symantec Support Forums).

  If you need full root access, you can restart the appliance, break into GRUB’s command line interface, append a “1″ to the kernel parameters in order to boot to runlevel 1 (single user mode). There you can change the root password to whatever you like and make Symantec’s Tech Support upset. I had to do it a couple of times to replace a failed disk, though…

  1. Or Symantec Mail Security, like it was previously called. It’s an antispam device, coming in either hardware or virtual (VMware) appliance versions. Models I’ve seen: 8240, 8260. Almost “install and forget”, if you ask me. That means it works quite well! ↩
  ]]> http://www.108.bz/posts/it/true-shell-access-on-symantec-brightmail/feed/ 11