• If anything, the flipping shows that failover works as expected (VMs don’t crash despite the chaos).
• That said, I could just disconnect a controller. Not really because the same storage system hosts an Oracle RAC cluster, humming along happily, unaffected by the issue.
• I need a way to selectively “hide” a controller from one or more hosts. I can do it easily by tweaking the SAN zoning configuration.

A Zone (much like a VLAN) is basically a group of WWNs (or ports). Objects in the Zone can only talk to each other. While creating Zones, it is common practice to “go minimal”: they should contain as few stuff as possible. I usually name them like this:
    Z_HOSTNAME_P1_DS4800_CA1_CB1
HBA Port 1 of HOSTNAME can see Controller A/Port 1 and Controller B/Port 1 of the DS4800.
Thus, going through each ESX server’s Zone, I just remove the Controller that the host shouldn’t see. Path thrashing is temporarily stopped.
The above rant serves mainly as a pro-zoning argument. “If every HBA port has to access every Controller’s port, why implement zoning?”. As you just read, zoning saved me from serious trouble, today.
About the “real” issue, it was ultimately caused by a thing called “Auto Volume Transfer” (AVT)¹. Let’s say that a LUN is assigned to controller A, but I/O for the LUN is issued to controller B. With AVT switched on the storage system will automatically transfer the LUN from A to B.
The Customer ESX servers are all (correctly) configured to use the “Most Recently Used” (MRU) path to a LUN. It seems that ESX, from a certain version on, issues I/O on the standby path, causing havoc if AVT is on. I can’t tell if that’s because it is fooled into thinking that the storage is an Active/Active one or if it just sort of periodically “probes” standby paths.
How do you switch AVT off? By using the DS “Storage Manager” and changing the ESX Hosts’ type from “Linux” (or whatever) to “LNXCLVMWARE”. This applies to all of the LSI derived Storage Systems (IBM, SUN StorageTek, Engenio, …). The latter host type is the right one to use when hooking an ESX cluster to an IBM DS Storage System. But “Linux” seems to do just fine on not so new ESX hosts version 4.1.x … When AVT is off, the Storage will decide to trespass LUNs only in the event of an internal hardware failure while, normally, LUN ownership will be handled by the multipathing software on the Host.

More reading on the subject:

[1] Differences between the “Linux” and “LNXCLVMWARE” host types.
[2] How does Auto Volume Transfer (AVT) work? Courtesy of Google’s cache. Lists which SCSI commands trigger AVT.
[3] A really nice blog post about the same issue described here. (Found, of course, when I was writing mine)

• Mount the host C$ share on the Linux laptop, using CIFS.
• Unexpectedly, the VM has got some active snapshots. Since the whole snapshot machinery is not available in VMware Player, the VM should come out of a VMware Server/ESXi/vSphere host.
  # vmrun -T ws listSnapshots "Windows 2000 Professional.vmx"
  Total snapshots: 2
  Snapshot 1
  Snapshot 2
• You can’t expand a disk that underwent some snapshots, they’ve got to be removed first:
  # vmrun -T ws deleteSnapshot "Windows 2000 Professional.vmx" "Snapshot 1"

  First caveat: the snapshot removal process causes vmrun to read/write large amount of data, depending on the snapshot’s size itself, I guess. You’re running it across the network: you’ll have to wait a while until it’s done. I’ve been monitoring the process with iftop -B -i eth0 -f ‘host hostname‘ to have an estimate of how long it would take to complete. Each snapshot needs to be removed by means of vmrun.

• Expansion time:
  # vmware-vdiskmanager -x 25GB "Windows 2000 Professional.vmdk"
    Grow: 100% done.
  Disk expansion completed successfully.
  
  WARNING: If the virtual disk is partitioned, you must use a third-party
           utility in the virtual machine to expand the size of the
           partitions. For more information, see:
           http://www.vmware.com/support/kb/enduser/std_adp.php?p_faqid=1647

  vmware-vdiskmanager appends X bytes of space (where X = new_capacity – previous_capacity) to the chosen vmdk. Again, those bytes travel through the network, be patient.

• As vmware-vdiskmanager was so kind to remind us, we just expanded the disk, not the logical partitions contained therein. Some kind of partition manager software is needed to complete the job, my fav being Acronis Disk Director.

Summing things up, it’s indeed possible to operate remotely on vmdk disks. Did we save any time by doing things this way, that is without copying the VM “out” of its host then moving it back onto it? If the snaphots are more than one, no doubt that “local beats network”. And what about the expansion? If we add some (empty) data to a vmdk, then need to move it back to the host, then local shouldn’t be any faster than network. Better would be to use an external drive of sorts. Better yet is to temporarily copy/install VMware workstation on the host, or just vmrun/vmware-vdiskmanager executable files and their dependencies. That’s what I’ll try the next time.

We created a bunch of new LUNs, planning to increase an existing Datastore’s capacity.
VMware side, the operation should be a matter of simply firing up vSphere Client, choosing a host, Configuration tab, viewing the Datastore’s properties, then clicking on the Increase button. Except that no Extent device seems to be found. That’s weird because we already did (multiple times) a rescan of each Storage Adapter/HBA. Moreover, selecting “Add Storage” as if we were to create a new Datastore, indeed shows the expected volumes.
The solution turned out to be this one:

• Connect vSphere Client directly to the host (thus logging in as root), and not to the vCenter Server.