A question for IT professionals

General » General Discussion

[ARCHIVED THREAD] - A question for IT professionals

Posted: 8/21/2006 9:36:04 AM EDT

How long would you have a job if you took a customer facing web site down multiple times in one day?

Posted: 8/21/2006 9:37:42 AM EDT

[#1]

Quoted:
How long would you have a job if you took a customer facing web site down multiple times in one day?

10 mins.. Unless its arfcom.. then you get a free pass..

Posted: 8/21/2006 9:38:43 AM EDT

[#2]

Depends.

Do I own the business?

Posted: 8/21/2006 9:38:58 AM EDT

[#3]

Depends on your CYA skills.

Posted: 8/21/2006 9:39:43 AM EDT

[#4]

Quoted:
Depends on your CYA skills.

Exactly

Posted: 8/21/2006 9:40:00 AM EDT

[#5]

Depends what the web site was for, who it was for, and why I did it.

Basically, the standard "it depends" computer guy answer.

Posted: 8/21/2006 9:40:32 AM EDT

[#6]

Quoted:
How long would you have a job if you took a customer facing web site down multiple times in one day?

Depends on whether or not the downtime is planned.

If it's not planned, it depends on who/how many people notice, and how well you can cover your ass.

Posted: 8/21/2006 9:41:49 AM EDT

[#7]

Gimme 20 minutes, and I'll find out

Posted: 8/21/2006 9:43:04 AM EDT

[#8]

I did forget the "cover your ass" part. You guys have obviously been at this a while.

I've done worse and remained gainfully employed. Admit nothing. Deny everything. Make counter accusations.

Posted: 8/21/2006 9:45:30 AM EDT

[#9]

we've had 40+ sites down over a weekend.

so, i'd say, yes, you'd be ok.

edit: that web sites + email server.

Posted: 8/21/2006 9:45:34 AM EDT

[#10]

We have unplanned outages a couple of times a month...shit happens.

I don't think that we've ever lost a customer because of it.

eta: No one has ever been so much as yelled at because of it (even cases in which someone fucked up).

Posted: 8/21/2006 9:45:51 AM EDT

[#11]

Quoted:

Admit nothing. Deny everything. Make counter accusations.

First thing we were taught in Intro to MIS was CYA.

Posted: 8/21/2006 9:58:40 AM EDT

[#12]

Quoted:

Quoted:

Admit nothing. Deny everything. Make counter accusations.

First thing we were taught in Intro to MIS was CYA.

Posted: 8/21/2006 10:50:33 AM EDT

[#13]

CYA in intro to programming, that's funny

I dunno, sites go down all the time, it happens, people expect it to happen (the reason that VOIP will never take over the phones, it doesn't work).
you just work hard and get it back up.
computers crash, it happens, they didn't pay for redundancy, they don't get 24/7/365

now, if you unplugged it, well, better start walking.

Posted: 8/21/2006 12:06:09 PM EDT

[#14]

Quoted:
CYA in intro to programming, that's funny

FYI: MIS is not a software degree.

Posted: 8/21/2006 12:13:54 PM EDT

[#15]

Really depends on why the site went down.
Stupid configuration mistakes or accidently tripped a breaker that the server is plugged into? There are varying degrees of error.

Accidently clicking 'shut down' when remoted into a box is inexcusable....trippping over a wire in a poorly organized data center is a whole different issue.

Posted: 8/21/2006 12:15:03 PM EDT

[#16]

It never happens when you are runnning quad redundant firewalls with virtual round-robin servers and dual homed datacenters. Course.... That 99.999% uptime costs a shitload

-Foxxz

Posted: 8/21/2006 12:19:05 PM EDT

[#17]

Use the standard IT answer: "the problem is on your end"

Posted: 8/21/2006 12:30:18 PM EDT

[#18]

Quoted:
I did forget the "cover your ass" part. You guys have obviously been at this a while.

I've done worse and remained gainfully employed. Admit nothing. Deny everything. Make counter accusations.

this one has learned well.

pfffttt websites are small time in the grand scheme of the wonders we call IT.

if your the web guy it's ALWAYS an application problem.

if your the application guy it's ALWAYS a hardware problem.

if your the hardware guy it's ALWAYS the application developers problem.

If your the application developer then your ALWAYS busy no time to deal directly with the customer. you pass it off to the applications guy and the cycle repeats.

Posted: 8/21/2006 12:35:01 PM EDT

[#19]

Quoted:

Quoted:
I did forget the "cover your ass" part. You guys have obviously been at this a while.

I've done worse and remained gainfully employed. Admit nothing. Deny everything. Make counter accusations.

this one has learned well.

pfffttt websites are small time in the grand scheme of the wonders we call IT.

if your the web guy it's ALWAYS an application problem.

if your the application guy it's ALWAYS a hardware problem.

if your the hardware guy it's ALWAYS the application developers problem.

If your the application developer then your ALWAYS busy no time to deal directly with the customer. you pass it off to the applications guy and the cycle repeats.

You have to get creative when you wear *all* those hats.

Posted: 8/21/2006 12:35:48 PM EDT

[#20]

I hosed the NDS directory late one night running updates.

650+ staff and 4800 student user logins.

I've been promoted to head of the department since then!

Av.

Posted: 8/21/2006 12:35:53 PM EDT

[#21]

Quoted:

Quoted:
How long would you have a job if you took a customer facing web site down multiple times in one day?

10 mins.. Unless its arfcom.. then you get a free pass..

Obviously.

Posted: 8/21/2006 12:46:34 PM EDT

[#22]

Quoted:

Quoted:
I did forget the "cover your ass" part. You guys have obviously been at this a while.

I've done worse and remained gainfully employed. Admit nothing. Deny everything. Make counter accusations.

this one has learned well.

pfffttt websites are small time in the grand scheme of the wonders we call IT.

if your the web guy it's ALWAYS an application problem.

if your the application guy it's ALWAYS a hardware problem.

if your the hardware guy it's ALWAYS the application developers problem.

If your the application developer then your ALWAYS busy no time to deal directly with the customer. you pass it off to the applications guy and the cycle repeats.

You have to get creative when you wear *all* those hats.

easily handled..... blame the vendor that is hardest to get in touch with. by the time they track them down on a wild goose chase the problem is resolved and you have had PLENTY of time to make the approprite cover story.

remember...

when doing the after action autopsy of the down time use BIG words, even invent a few if you have to. It makes management feel like you know exactly what is going on and confuses them. They won't ask questions becuase that would force them to admit they have no clue what you do.

Posted: 8/21/2006 12:47:40 PM EDT

[#23]

Quoted:
How long would you have a job if you took a customer facing web site down multiple times in one day?

Most of the morons I work with wouldn't notice.

They blame me for stuff that I have absolutely no control over.

Posted: 8/21/2006 12:49:44 PM EDT

[#24]

Quoted:

Quoted:
I did forget the "cover your ass" part. You guys have obviously been at this a while.

I've done worse and remained gainfully employed. Admit nothing. Deny everything. Make counter accusations.

this one has learned well.

pfffttt websites are small time in the grand scheme of the wonders we call IT.

if your the web guy it's ALWAYS an application problem.

if your the application guy it's ALWAYS a hardware problem.

if your the hardware guy it's ALWAYS the application developers problem.

If your the application developer then your ALWAYS busy no time to deal directly with the customer. you pass it off to the applications guy and the cycle repeats.

You have to get creative when you wear *all* those hats.

easily handled..... blame the vendor that is hardest to get in touch with. by the time they track them down on a wild goose chase the problem is resolved and you have had PLENTY of time to make the approprite cover story.

remember...

when doing the after action autopsy of the down time use BIG words, even invent a few if you have to. It makes management feel like you know exactly what is going on and confuses them. They won't ask questions becuase that would force them to admit they have no clue what you do.

You are wise in the ways of IT.

Posted: 8/21/2006 12:54:34 PM EDT

[#25]

Tell them the tubes were full.

Posted: 8/21/2006 12:56:29 PM EDT

[#26]

Quoted:
Tell them the tubes were full.

The Senator Byrd defense?

Posted: 8/21/2006 12:56:32 PM EDT

[#27]

Quoted:

Quoted:
I did forget the "cover your ass" part. You guys have obviously been at this a while.

I've done worse and remained gainfully employed. Admit nothing. Deny everything. Make counter accusations.

this one has learned well.

pfffttt websites are small time in the grand scheme of the wonders we call IT.

if your the web guy it's ALWAYS an application problem.

if your the application guy it's ALWAYS a hardware problem.

if your the hardware guy it's ALWAYS the application developers problem.

If your the application developer then your ALWAYS busy no time to deal directly with the customer. you pass it off to the applications guy and the cycle repeats.

You have to get creative when you wear *all* those hats.

easily handled..... blame the vendor that is hardest to get in touch with. by the time they track them down on a wild goose chase the problem is resolved and you have had PLENTY of time to make the approprite cover story.

remember...

when doing the after action autopsy of the down time use BIG words, even invent a few if you have to. It makes management feel like you know exactly what is going on and confuses them. They won't ask questions becuase that would force them to admit they have no clue what you do.

You are wise in the ways of IT.

i work for big blue..... 2/3 of my day is making excuses

Posted: 8/21/2006 12:59:44 PM EDT

[#28]

Posted: 8/21/2006 1:17:01 PM EDT

[#29]

Production or Test/Development?

Test, we take them down all the time for various maintenance and updates, but the customers knows that can happen (and usually has requested the work being done.)

If Production, you could probably get away with that once around here, get chastised, and move on. Twice you'd be in deep shit. Three times you'd likely be out the door.

Posted: 8/21/2006 1:36:50 PM EDT

[#30]

Tell them you experienced the dreaded error code ID-10T.

Posted: 8/21/2006 3:20:11 PM EDT

[#31]

Well, I usually CYA, then turn it into a Sales Opportunity.

"You know, this is a prime example of why you need a Hot Standby Disaster Recovery Site, which would also protect you in case of Hurricane, Terrorist Attack, or the 2nd Coming of the Flying Spaghetti Monster. We have some really cost-effective DR options that I could quote for you, if you're interested?"

Works like a champ.

Futuristic

Posted: 8/21/2006 3:30:37 PM EDT

[#32]

Quoted:
Well, I usually CYA, then turn it into a Sales Opportunity.

"You know, this is a prime example of why you need a Hot Standby Disaster Recovery Site, which would also protect you in case of Hurricane, Terrorist Attack, or the 2nd Coming of the Flying Spaghetti Monster. We have some really cost-effective DR options that I could quote for you, if you're interested?"

Works like a champ.

Futuristic

Hell yes, I love this one. For me (I'm a Cisco IP Telephony phone geek/consulatant/engineer), it works something like this:

"Your IPCC Express 3.5 server went down, which completely killed your call center. It's a shame that this happned, and the impact that it had on your business is substantial. But I've gotta tell ya - Cisco has listened to it's customers and has come up with a solution. With IPCC Express 4.x, you can have a hot standby server available that will fail over in less than 5 seconds. If you had this right now, the faulty server could have been fixed at your leisure. Now, I noticed that you've got a Smartnet SAU contract, so it won't cost you a dime for the software. I've got some availability next week, and I can take care of this for you for about $3500 in services. You'll have to buy another server for reduncancy, but that should only run you about $4000. So for $7500, we can make sure this never happens again. I bet that the downtime you experienced today cost you alot more than $7,500. the potential sales lost were no doubt at least triple that.

What time do you have available next week for me to begin implementing the solution that's going to make you look better to your boss?"

Tee hee.

Posted: 8/21/2006 3:44:12 PM EDT

[#33]

Quoted:

Quoted:
Well, I usually CYA, then turn it into a Sales Opportunity.

"You know, this is a prime example of why you need a Hot Standby Disaster Recovery Site, which would also protect you in case of Hurricane, Terrorist Attack, or the 2nd Coming of the Flying Spaghetti Monster. We have some really cost-effective DR options that I could quote for you, if you're interested?"

Works like a champ.

Futuristic

Hell yes, I love this one. For me (I'm a Cisco IP Telephony phone geek/consulatant/engineer), it works something like this:

"Your IPCC Express 3.5 server went down, which completely killed your call center. It's a shame that this happned, and the impact that it had on your business is substantial. But I've gotta tell ya - Cisco has listened to it's customers and has come up with a solution. With IPCC Express 4.x, you can have a hot standby server available that will fail over in less than 5 seconds. If you had this right now, the faulty server could have been fixed at your leisure. Now, I noticed that you've got a Smartnet SAU contract, so it won't cost you a dime for the software. I've got some availability next week, and I can take care of this for you for about $3500 in services. You'll have to buy another server for reduncancy, but that should only run you about $4000. So for $7500, ~~we can make sure this never happens again.~~ I bet that the downtime you experienced today cost you alot more than $7,500. the potential sales lost were no doubt at least triple that.

What time do you have available next week for me to begin implementing the solution that's going to make you look better to your boss?"

Tee hee.

I might change that to something more along the lines of, ‘We can dramatically reduce the chances of this happening again.’

Just adding a little more CYA

edit for smiley

Posted: 8/21/2006 3:45:19 PM EDT

[#34]

Quoted:

Quoted:
Well, I usually CYA, then turn it into a Sales Opportunity.

"You know, this is a prime example of why you need a Hot Standby Disaster Recovery Site, which would also protect you in case of Hurricane, Terrorist Attack, or the 2nd Coming of the Flying Spaghetti Monster. We have some really cost-effective DR options that I could quote for you, if you're interested?"

Works like a champ.

Futuristic

Hell yes, I love this one. For me (I'm a Cisco IP Telephony phone geek/consulatant/engineer), it works something like this:

"Your IPCC Express 3.5 server went down, which completely killed your call center. It's a shame that this happned, and the impact that it had on your business is substantial. But I've gotta tell ya - Cisco has listened to it's customers and has come up with a solution. With IPCC Express 4.x, you can have a hot standby server available that will fail over in less than 5 seconds. If you had this right now, the faulty server could have been fixed at your leisure. Now, I noticed that you've got a Smartnet SAU contract, so it won't cost you a dime for the software. I've got some availability next week, and I can take care of this for you for about $3500 in services. You'll have to buy another server for reduncancy, but that should only run you about $4000. So for $7500, ~~we can make sure this never happens again.~~ I bet that the downtime you experienced today cost you alot more than $7,500. the potential sales lost were no doubt at least triple that.

What time do you have available next week for me to begin implementing the solution that's going to make you look better to your boss?"

Tee hee.

I might change that to something more along the lines of, ‘We can dramatically reduce the chances of this happening again.’

Just adding a little more CYA

Good point.

Posted: 8/21/2006 3:46:57 PM EDT

[#35]

It would depend on which site, specifically how much money was lost and how much customer ill will was generated as a result.

Where I work, there are about a half dozen that I wouild get seriously yelled at if I did something stupid that brought them down during business hours. There are also a dozen or so that nobody would be likely to notice.

Posted: 8/21/2006 3:49:15 PM EDT

[#36]

I'd say you need to dump your Winblows web server and get a real hosting platform.

Posted: 8/21/2006 3:50:27 PM EDT

[#37]

Quoted:
I'd say you need to dump your Winblows web

server and get a real hosting platform.

+ LAMP

Posted: 8/21/2006 4:03:22 PM EDT

[#38]

Quoted:

Quoted:
Well, I usually CYA, then turn it into a Sales Opportunity.

"You know, this is a prime example of why you need a Hot Standby Disaster Recovery Site, which would also protect you in case of Hurricane, Terrorist Attack, or the 2nd Coming of the Flying Spaghetti Monster. We have some really cost-effective DR options that I could quote for you, if you're interested?"

Works like a champ.

Futuristic

Hell yes, I love this one. For me (I'm a Cisco IP Telephony phone geek/consulatant/engineer), it works something like this:

"Your IPCC Express 3.5 server went down, which completely killed your call center. It's a shame that this happned, and the impact that it had on your business is substantial. But I've gotta tell ya - Cisco has listened to it's customers and has come up with a solution. With IPCC Express 4.x, you can have a hot standby server available that will fail over in less than 5 seconds. If you had this right now, the faulty server could have been fixed at your leisure. Now, I noticed that you've got a Smartnet SAU contract, so it won't cost you a dime for the software. I've got some availability next week, and I can take care of this for you for about $3500 in services. You'll have to buy another server for reduncancy, but that should only run you about $4000. So for $7500, ~~we can make sure this never happens again.~~ I bet that the downtime you experienced today cost you alot more than $7,500. the potential sales lost were no doubt at least triple that.

What time do you have available next week for me to begin implementing the solution that's going to make you look better to your boss?"

Tee hee.

I might change that to something more along the lines of, ‘We can dramatically reduce the chances of this happening again.’

Just adding a little more CYA

edit for smiley

'tis true. Many a quota were exceeded on the basis of an irate customer being upsold because some dipshit tripped over a power cable. Inconsolably screaming CEOs want immediate rectification for their pain. Most times that level of service costs obscene amounts of money to provide. Our job as hardware and software vendors and developers is to have that solution on the shelf ready for deployment when those pure margin deals happen.

Going from one "9" to the next is an exponential journey of the pocketbook. But that is why you pay for insurance, and good IT is just another form of risk management.

[ARCHIVED THREAD] - A question for IT professionals

General » General Discussion

Warning

Confirm Action

About AR15.COM

Stay Connected

Newsletter

Contact Us