The following info is blatantly stolen from this precious post, but the issue I faced is so odd that I wanted to stress about it myself. All credits go to Philip Hofstetter and his blog.

[Edit: the following still applies even when, as described here, the Provisioning API is enabled.]
[Edit: the script below doesn’t handle Results Pagination. That means that it will just return the first 200 or so queried objects. I’ve yet to complete it… Depending on your needs, you may just use Google Apps Manager instead.

I was trying to fetch some info using Google Data API and Python. At some point, I decided to move from simple authentication with user supplied credentials to two-legged OAuth. The Contact feed remained accessible while trying to read Groups, Users or Nicknames (by means of the Provisioning API) failed with “Internal Error” 500 or “Authentication Failure”.

As Philip discovered, some feeds just won’t work unless explicitly permitted access to.
Inconsistency 1: You API Client name (the one you’ll use as “customer_key” in OAuth and the one whose name will most likely match your Google Apps domain name), is already listed under “Manage this domain”, “Advanced tools”, “Manage third party OAuth Client access”. The wording “This client has access to all APIs” is clearly a lie.
Inconsistency 2: I followed Philip advice and manually added the (readonly) feeds/scopes, except that they don’t show up under “Manage API client access”. But they’re somewhat being honored (i.e.: without the tweak, my script won’t work). Moreover, the “Authorize” operation should be done just once and encompass all of the scopes you need. You can’t just add one later. Adding a single scope will revoke access to the previous ones. This behaviour is different from Philip’s (in his screenshots, authorized scopes are indeed visible on Google Apps Domain Control Panel).
This is what I used:

And this is the script:

Or how a single sector can make you ten times more happier.
Today I’ll talk about a client issue: getting (extremely) slow write performance when backing up my laptop over an USB drive (a 200GB Samsung S1 Mini). All I usually do is booting the PC with SystemRescueCd, plug an USB disk in, “ddrescue /dev/sda /mnt/externaldisk/laptop_disk_image.dd“, letting it run overnight. Except that this morning the backup wasn’t finished yet. What’s wrong? Long post (for a simple solution) ahead…

The USB disk (shown below as /dev/sdb*) “feels” fast when reading and awfully slow when writing. The most simple way to do an HDD benchmark is, of course dd. Use it along with dstat (an essential tool when pinpointing performance issues, whatever they may be) and you’ll quickly gather some useful figures. Beware! dd can ruin all your data just by mistaking a “b” for an “a”: triple-check and make sure that you’re running it on the right devices!

A sequential write test:

3.7 MB/s only, definitely slow. Note that you shouldn’t use /dev/{random,urandom} as input file, they’re a bottleneck by themselves. /dev/zero, on the other hand, is super-fast. “dd if=/dev/zero of=/dev/null bs=16384 count=$((10000*1024))” (shove zeros to /dev/null) is bound only by the CPU, running at about 9.3 GB/s here.

A sequential read test:

30 MB/s, that’s the order of magnitude I was expecting (confirmed here).

If I repeat the write test and, at the same time, run dstat, I notice that there are no burst or drops: speed is constant.

Since reading works, kernel and USB Host Controller seem to go along well. Issue should lie on the disk’s side. I had no clue of what was happening until I tried writing straight to the disk instead of the first primary partition (i.e.: /dev/sdb instead of /dev/sdb1), thus trashing the filesystem (I’ve got no data to lose on that disk: no worries).

Even though the difference between read and write throughput seems to be too much (almost one order of magnitude), this is starting to look like a FS blocksize/partition aligment issue. Well, some disks use a physical sector size (PSS) of 512 bytes. Others use 4096 bytes (4 KiB). Others use the latter but tell the OS that they’re using 512 bytes or more simply the OS can’t figure out the right physical sector size… And USB mass storage devices tell the OS almost nothing (hdparm won’t help this time)…

Filesystem (or other “structured storage” systems like, for instance, datafiles in databases) organize their data in blocks. The block size can sometimes be adjusted. 4096 bytes is a quite common value:

A sector represents the smallest chunk of data that can be read/written from/to a disk. If its size is 512, and the filesystem block size is 4096, the filesystem driver will read/write batches of 8 sectors. Better said: the FS thinks to deal with 4k blocks, not knowing that lower level functions will further split them in eight (if only logically).
Consider another example: the PSS is 4096, but the drive acts as if it was 512. Physical sectors can be found at absolute offset sector_number*512*8 (0, 4096, 8192, …). What if a 4k write operation happens at offset 1*512*8-512? (3584, it doesn’t look like a “bad” offset: as far as the OS is concerned, any multiple of 512 is fine). The drive, being unable to write less than 4k and at proper locations, will: read sector 0, read sector 1, modify the last 512 bytes chunk of sector 0, modify the first three chunks of sector 1 then write both sectors (or something similar). If things were properly aligned, a single write operation would’ve sufficed. Read operations speed, on the other hand, may be almost unaffected. Think about it: unless you’re dealing with tons of 4k files spread across sector couples (i.e.: two sectors read instead of one), large chunks of data are (hopefully) laid out sequentially of the disc. Reading 1GB plus 512 bytes, instead of 1GB alone, won’t change anything benchmarks.

What’s up with my partition?

sfdisk (one of fdisk‘s cousins) shows that the partition starts at byte 63*512=32256. This value isn’t divisible by 4096, yielding a non integer result. Sector 64, instead, is a good place to start an aligned partition:

63*512/4096 = 7.87
64*512/4096 = 8.00

Similarly, other partitions should start at sectors that are multiples of 8 (because 512*8=4096).
This is the corrected partition table. Moving the partition forward (by a mere 512 bytes) causes a 10x write speed increase.

You may still have a question though. Does aligning a partition mean that the contained filesystem is aligned too? You’re right, that assumption should not be taken for a fact.

A filesystem is made up of “your” data and “its” data (the latter being internal structures necessary to organize the former). In any case, a FS will try to pad/align stuff to the block size. That was to say that, if you start a partition and the partition is aligned to a given boundary, the filesystem (all of its composing blocks) will be aligned too.

You can find a description of the ext2 layout here. Partition starts with two sectors reserved for the boot loader. Then comes a 1k chunk holding ext2 “superblock”. At offset 56 within the superblock, we should find the ext2 magic number (0x53EF), and here it is:

The next byte after the superblock, is byte 4096. From then on, everything happens (from the FS point of view) in chunks as big as the configured block size. My disk is a 4k sector disk, formatted with a single partition aligned (as the FS) to a 4K block. FS block size is 4K too. Can’t do really do any better than that besides choosing a filesystem that manages to handle the given workload with fewer read/write operations, but I digress…

So, Customer starts updating all of his VMware ESX hosts and things turn out for the worst. VMs are crawling slow (ping response time from 0 to 1000ms), console access through vSphere client doesn’t always work, and hosts’ CPU percentage is unnaturally high. Cause is apparent: path thrashing.
Path thrashing happens when, for some reason, SCSI LUNs are being continuously reassinged from a controller (Target) to another one. ESX has a hard time “bouncing” I/O back and forth on the right Fibre Channel path. On Active/Passive SAN arrays a LUN can be “owned” by just one controller at a time. If the LUN owner has to be changed because of a hardware failure (path, Controller, SFP/GBIC, FC switch, …) or because the Initiator would like to, the LUN itself has to “trespass” (in EMC parlance), transition to another controller. The “command” to do so can be issued by the Initiator or internally by the storage subsystem.
Back to today’s case, I was dealing with an IBM DS4800 where LUNs flipped like mad between controller A and B. How to stop it quickly?

• If anything, the flipping shows that failover works as expected (VMs don’t crash despite the chaos).
• That said, I could just disconnect a controller. Not really because the same storage system hosts an Oracle RAC cluster, humming along happily, unaffected by the issue.
• I need a way to selectively “hide” a controller from one or more hosts. I can do it easily by tweaking the SAN zoning configuration.

A Zone (much like a VLAN) is basically a group of WWNs (or ports). Objects in the Zone can only talk to each other. While creating Zones, it is common practice to “go minimal”: they should contain as few stuff as possible. I usually name them like this:
    Z_HOSTNAME_P1_DS4800_CA1_CB1
HBA Port 1 of HOSTNAME can see Controller A/Port 1 and Controller B/Port 1 of the DS4800.
Thus, going through each ESX server’s Zone, I just remove the Controller that the host shouldn’t see. Path thrashing is temporarily stopped.
The above rant serves mainly as a pro-zoning argument. “If every HBA port has to access every Controller’s port, why implement zoning?”. As you just read, zoning saved me from serious trouble, today.
About the “real” issue, it was ultimately caused by a thing called “Auto Volume Transfer” (AVT)¹. Let’s say that a LUN is assigned to controller A, but I/O for the LUN is issued to controller B. With AVT switched on the storage system will automatically transfer the LUN from A to B.
The Customer ESX servers are all (correctly) configured to use the “Most Recently Used” (MRU) path to a LUN. It seems that ESX, from a certain version on, issues I/O on the standby path, causing havoc if AVT is on. I can’t tell if that’s because it is fooled into thinking that the storage is an Active/Active one or if it just sort of periodically “probes” standby paths.
How do you switch AVT off? By using the DS “Storage Manager” and changing the ESX Hosts’ type from “Linux” (or whatever) to “LNXCLVMWARE”. This applies to all of the LSI derived Storage Systems (IBM, SUN StorageTek, Engenio, …). The latter host type is the right one to use when hooking an ESX cluster to an IBM DS Storage System. But “Linux” seems to do just fine on not so new ESX hosts version 4.1.x … When AVT is off, the Storage will decide to trespass LUNs only in the event of an internal hardware failure while, normally, LUN ownership will be handled by the multipathing software on the Host.

More reading on the subject:

[1] Differences between the “Linux” and “LNXCLVMWARE” host types.
[2] How does Auto Volume Transfer (AVT) work? Courtesy of Google’s cache. Lists which SCSI commands trigger AVT.
[3] A really nice blog post about the same issue described here. (Found, of course, when I was writing mine)

Quick post to show you how DHCP reservations can be replicated between Windows servers. Why whould you want to do that? Because often, to achieve DHCP service high availability, DHCP scopes are equally divided between servers. When a client PC is connected to the network, it sends out a broadcast to discover which DHCP servers are active on that particular ethernet segment. Depending on their number, the PC will receive one or more answer, each offering an IP address. If a client is to be assigned a fixed IP, all of those offers should bear the same IP address. Hence, DHCP reservations need to be configured the same for every DHCP server in the given scope. As far as I know, this needs to be done by hand. To speed up the process, I use netsh (see Netsh commands for DHCP).

The command below will dump all of the reservations to a file named “reservations.txt”. findstr filters netsh output keeping just the info we need.

Each line in “reservations.txt” should look like this:

10.4.1.1 is the IP address for dhcpsrv1, the “source” DHCP server.

Open “reservations.txt” in a text editor, check that everything is fine and substitute the source DHCP server IP with the target’s one (i.e.: 10.4.1.1 becomes 10.4.1.2), save the file and run:

That’s it; not a fancy trick, but it may be useful nonetheless. Just beware that, when there are thousands of clients, netsh could take a while to complete its job (especially the “dump” step)…

Following up on the previous post, let me show you other ways to trick web application into doing what they’ve not been designed to do: saving the Sysadmin some typing and avoding errors. I’ll use JavaScript, jQuery, Greasemonkey and Perl to automate Firefox and implement a sort of dynamic form filling. Let’s start with a screencast:

The config I had to do (on a SonicWALL firewall), involves about 70 subnets, each similar to the other. Only the subnet’s addressing scheme changes, making creation of VLANs/objects/rules a repetitive and error-prone task.

What’s happening in the screencast? VLAN sub-interfaces are being automatically created without me having to type anything at all. How could that be? A simple Greasemonkey script calls a “web service” (AJAX style) fetching the needed data and filling the form for me (I just click the “OK” or “Cancel” buttons). Why this whole Greasemonkey/web service mess? Because here, as in the previous post, JavaScript code is basically being injected into a “page”. Pages (or tabs, or windows) are ran by the browser into a sandbox: they can’t exchange data between each other. Thus page A (the webapp) can’t access code/data in page B (our code). Moreover, injected JavaScript gets lost when the page is closed (think ugly GUIs where dialog windows pop up just to be destroyed shortly after). We need a way (Greasemonkey) to re-inject the code each time our page is shown and some external, long lived, entity to update/keep status (the web service).

Thanks to the HTTP::Server::Simple Perl module, building the web service is trivial. The only logic behind it is keeping track of the current VLAN’s index, iterating through each value on subsequent web service calls:

And here’s the Greasemonkey script. It runs automatically when the “add VLAN” page pops up, does the sort-of-AJAX call, uses jQuery to properly fill the form.

I built similar scripts to create static ARP entries and routes. Another one took care of firewall symbolic objects (names) but, as that part of the config is itself carried out by AJAX (in a non reloading window), I didn’t need Greasemonkey, just Firebug.
No kidding, the above tricks saved me half a day of tedium…

Or how to toggle a thousand checkboxes clicking none.
Have you ever wondered that jQuery may be relevant to your everyday sysadmin job? It is: web-based GUIs proliferate and the Command Line is sooo nineties (this one’s not mine)…
Working on a SonicWALL/Aventail SSL VPN box, I was asked to simplify how permissions were mapped to Users. What a User could or couldn’t do, was defined at the User level. Ok, let’s just:

• Create an A/D group for each role/profile.
• Configure permissions (rules, resources access, …) on various Communities (Aventail parlance for roles/profiles).
• Put the right Users into the right A/D group.
• Cleanup: generally, each Community should just have A/D groups assigned to it. Get rid of unnecessary Community memberships.

First three tasks were easy (thanks also to the DS* commands). The fourth, uhm:

Removing members from a Community means unchecking each User. In my case, more than one thousand clicks and a beginning of carpal tunnel syndrome. At page three I started to think about a less saddening way.

FireQuery is a Firefox extensions that let’s you “inject” jQuery into any webpage. Here’s what I did:

• Went to the page depicted above (the one that let’s you assign members to a Community).
• Launched Firebug.
• Inspected the DOM and noticed that each of the checkboxes value begins with “AV”.
• Used the debugger to see what’s going on when checking/unchecking members. Nothing really strange: just a bit of JavaScript to highlight a row depending on its checkbox’s status.
• Hit the jQuerify button. FireQuery is needed because Aventail’s web-based GUI doesn’t use jQuery.
• Went to Console, typed the JavaScript one-liner below and hit Run

which translates to: use jQuery to select all of the checkboxes whose value starts with “AV”. Uncheck the selected checkboxes.

See? No useless clicking: a GUI has been CLI-fied.

Today I ran into a weird issue while installing Oracle Grid Control Agent 10.2.0.3 on Linux. Right after typing “runInstaller”, OUI crashed because of segmentation fault… Let me talk about some of the troubleshooting maneuvers you may need to perform should you find yourself in similar troubles.

Here are the relevant details:

• OS: Red Hat Enterprise Linux Server 5.3 x86-64
• GC Agent: Oracle Enterprise Manager 10g Grid Control Release 3 (10.2.0.3) for Linux x86-64
• GC Console: Oracle Enterprise Manager 10g Release 5 (10.2.0.5) Grid Control for Microsoft Windows 32-bit

And here’s the error message (the most interesting portions):

To go past this show-stopper I tried a few things…

The Heap report produced by java at crash time, seemed to indicate a memory shortage. By editing the “install/oraparam.ini” file, you can tweak how much RAM is available for OUI’s JVM. Just alter “JRE_MEMORY_OPTIONS” value.

This is also a safe place to put additional command line parameters: they’ll mostly be passed to java’s command line. I said “mostly” because OUI wrapper/launcher seems to check some sort of allowed parameters list and may refuse to go on if somethings doesn’t look right.

The “-XX:MaxPermSize=32m” is one of the knobs that doesn’t pass the sanity check. In order to run OUI’s JVM by hand, with the right parameters, just keep the first lines of runInstaller (the ones starting with ‘Arg:‘):

Strip “^Arg:“, “^\d*:“, “:$“, add a trailing “ \” and you’ll have an OUI launching shell script you can alter at will.

Increasing JVM’s memory led to no effect. Heap report looked fine (usage percentages went down) but crash was still there.

Another useful switch is “-XX:+ShowMessageBoxOnError“. It makes java halt on error, allowing us to attach a debugger and perform a stack backtrace, e.g.:

I also tried to “inject” a couple of newer JVM’s into the stage directory. The quickest way is to borrow it from another installer.

The server’s has a “working” directory were Oracle patches/products are stored before use. In my case, changing OUI’s JVM from 1.4.2.8 to 1.4.2.14 is a matter of copying:

to:

Then modifing the same “oraparam.ini” file mentioned before.

You could as well download a specific JRE from http://java.sun.com (sorry: from Oracle) and:

• install the new JRE somewhere
• unzip (-t) the “filegroup1.jar” file that corresponds to OUI’s “factory” JRE. Note how the directories are laid out (something like: “jre/1.4.2″). Modify the new JRE accordingly.
• zip the new JRE, rename the resulting file to “filegroup1.jar”, copy it in the right place.
• modify oraparam.ini and choose the JVM version you’ll boot OUI into.

Three different JREs, each of them segfaulting in the same spot, as we saw in the backtrace:

Who’s the owner of libXt?

After making sure that none of the running processes was using that package contents, I decided to remove it (rpm -e –nodeps libXt-1.0.2-3.1.i386) and reinstall it. Surprisingly, OUI worked flawlessy after this last action. Too bad I can’t really explain why. libXt version didn’t change before/after reinstall. I should diff it anyway with what’s left untouched on other RAC cluster members. I’ll update the post when I have a stricter explanation…

Today I was handed a hard disk, removed from from a CCTV DVR about which we knew nothing (no model/make). The request was to extract as much footage as possible from it. I hooked the disk to my laptop via a SATA/USB adaptor but, unsurprisingly, wasn’t able to find/mount a filesystem. I dumped the first 30 megabytes off the device (dd if=/dev/sdb of=x.bin bs=1M count=30) and opend the resulting file with a hex editor:

The first bytes read “DHFS4.1″. Various Google searches return just one english hit: a machine translated page from chinese mentioning a company named “Dahua” that manufactures some video surveillance equipment. On their support page an HDD Download Tool can be found (filename: HDD Download Tool.rar). It can look up video clips stored on a disk by date, time and channel (input number), and extract them as “.dav” files (e.g: 01.23.03-01.33.21[M][@2a2ed][0].dav). You can convert .dav files to .avi using “dhavi.exe” (from bahamassecurity.com). I was able to run the latter with Wine, which probably means that .dav content can be packed into an .avi container without transcoding (even though the website says something about H.264 codecs). The former tool, instead, requires Windows because of the raw disk I/O going on.

Hopefully, this post gathered some difficult to find info, coming either from Chinese pages or sites not indexed by Google…

Or how to use the dsquery/dsget/dsmod commands to copy all the members from an
Active Directory group (source), to another one (destination).

If, like me, you are on a neverending quest to click less and script more, you can solve the problem this way:

• Create the destination group, should it not exist.
• Find the source group’s DN:
  >dsquery group -samid sourcegroup
  "CN=sourcegroup,OU=Groups,DC=contoso,DC=com"

  “-samid” argument is the group name whose DN you’re looking for. You can use “*” as a wildcard.

• Ditto for the destination group:
  >dsquery group -samid destinationgroup
  "CN=destinationgroup,OU=Groups,DC=contoso,DC=com"</li>
• On with the copy itself:
  >dsget group "CN=sourcegroup,OU=Groups,DC=contoso,DC=com" -members -expand | dsmod group "CN=destinationgroup,OU=Groups,DC=contoso,DC=com" -addmbr -c
  dsmod succeeded:CN=destinationgroup,OU=Groups,DC=contoso,DC=com

  These are two commands: “dsget group” and “dsmod group“. Output from the first is piped to the second. “-members” causes the group members’ DNs to be listed on standard output (one by line, quoted). “-expand” makes dsget to recursively expand the sub-groups that sourcegroup may hold.
  Conversely, dsmod modifies destinationgroup adding members to it.
  Very cool, so far. The only caveat is that the “-c” switch doesn’t work as advertised. It should copy members over destinationgroup even if already exist, but it doesn’t. If you need to re-sync source and dest, delete source’s contents from dest.

Bonus note; here’s a quick way to discover a user’s DN given his username:

One of the most procrastinated issues I had at a Customer’s, was the proliferation of errors like these (as shown in servers/clients Event Viewer):

There are several posts mentioning the issue, this one pointed me in the right direction. Basically, because of how SEP components communicate, Windows is triggered into updating the list of trusted root Certification Authorities. It tries to do so through the Internet using the Computer account. The latter may not have any proxy configured. Being unable to reach outside, the host gets flooded by crypt32 errors.

In order to solve the issue, I decided to deploy a valid proxy configuration, for the Computer account (SYSTEM user), on a subset of the Domain’s hosts.
One of the ways to script that is the “proxycfg -u” command¹ that works by copying the current user proxy settings to the SYSTEM’s registry. Sounds cool but if the current user is not a member of the local Administrators group, he won’t have the necessary rights. The following script instead, can be launched via Group Policy² during operating system startup, and since it’s a startup script rather than a login one, it will run with administrative privileges.

Nothing fancy in the below source. It creates the registry key if it doesn’t exist, then sets the right value for WinHttpSettings which I obtained this way:

• use “proxycfg -u” on a test host
• use the Registry editor to export the contents of HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\Connections

The value is of type REG_BINARY. Since the RegWrite API (method of class WScript.Shell) cannot deal with binary values, WMI (StdRegProv registry provider) needs to be used. Also, SetBinaryValue expects an array of decimal values, while Regedit exports them as hexadecimal digits (you’ll have to take care of the conversion yourself).

If the scripts works as it should, you’ll be greeted by these events:

And, hopefully, crypt32 errors will be gone for good.