Company went cheap on storage, against my advice.  Storage vendor claims compatibility with IBM AIX, but we've had some continual problems suggesting that they aren't truly compatible.  Sunday we were using IBM's Live Partition Mobility to move partitions between servers so we could do hardware maintenance on one of the servers.  One of the storage arrays fails.  OK, it's a pain in the ass, but we had all of the data mirrored to another array in a location a couple of miles away, we start breaking the mirrors to continue our work.  We're almost done with that when the second array fails hard, to the point that 36 hours later, the vendor got it to the point they could START copying data volumes off to another array.  Fortunately, the first array, after a couple of reboots, finally came up, so in most cases I've been able to rebuild the OS volume groups from backups and then play some games to convince the partitions to read from the volumes off the first array enough to get up and running again.  I'm at 48 hours from the initial failure and the vendor still hasn't given me a usable recovered volume from the failed array.

Yup buy cheap and that's what you get.  Live partition migration sounds sketchy.  Sad thing is company probably won't learn from this lesson and will still continue to buy cheap.  I just wanted to say good luck and we're all counting on you.

At least you got them to pay for an actual backup, as insufficient as it was.  I've got 2 small city governments that have absolutely nothing for backup, across all departments.  A few people have taken it upon themselves to backup files to thumb drives.

Well, I run the storage in addition to the AIX servers, so was able to tuck away a terabyte or so for my network install manager, including an NFS filesystem for operating system image backups.  Fortunately, with issues we've been having, along with some lower compression ratios than promised, we hadn't moved the AIX systems to the same storage arrays used by the ESX cluster, if we'd lost that, life would be even more painful.  At least one of the earliest systems recovered was the payroll system :-)



 

I've done some....questionable things...with rsync but yeah, I like my VMs.

Right?  No one can make promises on compression.  As the storage guy, you clearly understand the importance of always getting more than you actually need.

That sort of fits into why I recommended against this particular vendor :-)  I dislike having my production systems on bleeding edge hardware, especially from vendors that are too new to have much of a track record.  Management was already discussing getting replacement storage, looks like they'll be going with IBM to avoid any compatibility issues.  As for more than I need, unfortunately, management decided the ESX admins needed to be storage admins as well, and the ESX environment's storage needs have been growing considerably.  There may be pressure to force their storage requests to go through me in the future, but that's a political fight I'm not inclined to push right now.  Of course, it is a factor in why I may not stay here much longer...



 

Hyperconverge ALL THE THINGS

I'm trying to procure a couple of Dell Ready nodes with SSD.  To replace some Old IBM X Servers and an HP DL380 G7 server.  Luckily we have good storage (NetApp) and a good Server Admin/Storage Admin (Me).  Sad thing is I was recently denied the purchase of Veeam and was told that Symantec is the only thing I can have (Garbage).

Symantec is garbage, and I think Veeam even costs less.  Have you looked at a Veeam rental service?  I can rent Veeam to customers pretty damn cheap.