Posted: 7/18/2011 7:12:26 PM EDT
|
Any storage gurus out there?
I'm hoping someone might have a few suggestions. Say you are using an EMC CX4 240 with a mix of FC and SATA disks. You have an app running on a VM with an Raw Mapped Disk that is a single 250GB LUN. It spikes to over 1800 IOPS and at times runs constantly in the 1200 IOPS range. How do you configure it to get the IOPS it needs and avoid high latency problems but not waste storage? |
|
I would configure a RAID 1/0 LUN on the FC disks and use the following as a guide:
FC 10K disk = 125 IOPS FC 15K disk = 150 IOPS Assuming you have a mixture of 70% read and 30 % write, your 1800 IOPs breaks down to: 1260 Reads 540 Writes ( x 2 for mirroring) 2340 IOPS Total 2340/ 125 IOPS (10K) = 18.72 disks 2340/ 150 IOPS (15K) = 15.6 disks Since you can only have 16 disks in the RAID Group, consider the faster disks or use virtual provisioning or meta luns. This is VERY conservative because it doesn't take into account read and write caching. YMMV Good luck. |
|
Quoted:
I would configure a RAID 1/0 LUN on the FC disks and use the following as a guide: FC 10K disk = 125 IOPS FC 15K disk = 150 IOPS Assuming you have a mixture of 70% read and 30 % write, your 1800 IOPs breaks down to: 1260 Reads 540 Writes ( x 2 for mirroring) 2340 IOPS Total 2340/ 125 IOPS (10K) = 18.72 disks 2340/ 150 IOPS (15K) = 15.6 disks Since you can only have 16 disks in the RAID Group, consider the faster disks or use virtual provisioning or meta luns. This is VERY conservative because it doesn't take into account read and write caching. YMMV Good luck. I moved it to a metalun that was part of two RAID1/0 groups, one 8 drives and one 6 drives. With the other stuff running in the RAID groups it was just too much IO. Currently we have it running in a FAST Pool with SSD and FC. I have it set to the highest tier so it's all on SSD drives. I was trying to free up the SSD drives so I could use them as FAST cache but I don't want to waste and entire shelf of 300 or 600GB drives to get the IOPS I need for the 250GB that the app needs. |
|
mkhllr summed it up pretty good. I would suggest that if there's anyway you could move it off of the SSDs and onto some 15k drives in a RAID 1+0 group and then setup your FAST cache to do it and see if that helps alleviate the issues. I haven't had any spikes that high in our environment, but the ones I normally have are due to poor timing by the developers who all want to run their hard hitting jobs at the same time.
 We just installed 1TB of FAST cache about 3 months ago on our 960 and saw an immediate performance increase on the LUNs we enabled it on. Anybody have any experience with the new VNX arrays? We're trying to consolidate our 960 and NS-20 and wanted to see if others have gotten one yet and how it stacks up against the CX4s. |
|
Unfortunately I don't have the drives to setup another RAID1/0 group, at least not enough to get the IOPS that app requires. I really wanted to add the SSDs to FAST cache but for now I think I'm stuck with it the way it is. We've probably purchased over 300k on drives this year. It's going to be hard to get them to spend more money right now I think.
I've haven't worked with the VNX, but seem some demos. It looks promising, the way it combines the Clariion and Centerra series. I think it would be easier to manage shares by being able to do it straight from the storage and not a file server. There was a Gartner study a while back saying storage needs would increase by 650% over the next 5 years. The study may be a year or two old, but we have more than doubled our Storage in the last 6-7 months and are already running low. How the hell do you size storage needs correctly for the future? the guy that really maintained our storage left 2 months ago and I've kind of taken it over but I don't have much background in it. He didn't either and I think that's part of our sizing problems. Our CX4 is 2 years old, initially bought with 20TB, last year we upped it to 40TB and now we are at about 90TB. We probably only have room to increase add another 3 shelves and we have to upgrade the SAN or add another one. Any ideas how to approach this subject with our CIO? |
Ew, guess you're kind of stuck right now. I hate when that happens. It's always hard going back to management and asking for more money. Only thing you could really do is force the issue and get them to let you make the change during a maintenance window or something. Rock and a hard place.  We have a NS-960 at our DR site, which does the same as the VNX by combining NAS and fibre. Just haven't really gotten to play with it much yet. The way we've been doing storage planning is pretty much shooting from the hip. We cannot do charge backs ( ) so we try to estimate general growth and put that in my budget and when a project comes along needing space, we will ask for money from it depending on the requirements. We usually come out ahead as I'm not going to ask EMC for a quote for 500GB of space, I'll ask for 5TB and the project team is none the wiser. Shhhh.  One that that you might be able to do is have EMC run a data analysis on the storage you need/want to move. Make sure your performance monitoring is enabled and logging/rotating on a normal basis. You can either run the NAR files through analyzer and see what you find or you can have EMC analyze them and provide data. You will have to provide them with specific LUNs and dates/times cause they refuse to look at the entire array. Sucks, lol. Would it be any cheaper to ask for money for more SSDs? I know it's not chump change, but might be easier. If you already have the license, then you're ahead of the curve as that alone is like $90k, or at least it was for us. When we purchased the hardware back in January, we got 11 200GB SSDs and a DAE for $80k. Again, not chump change but if you can sell it as an overall improvement to the array it might go down easier. Worst thing I found about FAST cache, is that you have to go back and enable it on LUNs already created if you didn't have the license installed when they were carved. Oh well, c'est la vie. |
|
I brought up our problem to her this morning and we had a meeting this afternoon with her and our team. She wasn't too thrilled. We are going to have to rearrange stuff and get creative. We've got a small amount, like 20k for a project that we'll use to buy more FC drives.
We apparently have a possible very large project coming up, like an entire new building that I think we'll try to leverage to get what we need out of it. I have to go through and look at usage, probably will get EMC involved and try to see what can be migrated to where to consolidate space. I know we have a few 300GB FC drives that can probably replaced with 600GB ones. I'm hoping I/O isn't a problem. I just got a new "Technical Account Manager" (whatever the title is) about 6 weeks ago and he actually used to be my director when I worked at Methodist so that helps. We get along great and I know he won't bullshit me. Love the avatar by the way. Good show, and the chick that plays Sarah is fucking hot! 
|
|
Yeah, management is never thrilled when you want more money. lol Hopefully you can find a happy medium to get things right. Sounds like an interesting project! Good luck with it. One of my good friends works for EMC and he more or less offered me a job to be a solutions architect in some of the southern states. Not sure I really want to do that due to the travel. But it would be nice to not have to work on anything other than storage. You hiring? lol And yes, Yvonne is smoking hot...and my future wife! But shhhhh, don't tell my current wife!  Too bad we only get to see Chuck for one more season, but that's cool. Can't wait to see her in The Killer Elite! |
|
We aren't hiring right now. Just added a few positions at the end of last year. It's a pretty good gig for the most part. Constantly got something new going on.
Yeah I wouldn't mind working for a vendor and specializing. I pretty much focus on VMware, Cisco and storage stuff. Add in MS server and all the effing apps we support and it's hard to be really good at anything. They are only doing one more season? I guess I'll have to watch all of it. I wasn't thrilled with the end of last season and the other guy getting the intersect. I'm not a big fan of his character, hopefully he only has it for a few episodes or something. |
|
Yeah, I'd love to just focus on one thing. Right now I'm doing UNIX, SAN and the apps that run on UNIX support. But I'm being moved to a more management and advisory role, so I might get to take a step back. One good thing is...no more on-call. lol Yeah, one more 13 episode season. I'm not sure how the Morgan + intersect thing will play out, guess we'll see. As long as there is plenty Sarah Walker (Bartowski now?) I'm good.
|
