Any of you hogs want to take a stab at this one?

General » Urban Commandos

Site Notices

[ARCHIVED THREAD] - Any of you hogs want to take a stab at this one?

Posted: 11/2/2009 2:46:03 PM EDT

Got a report that a server was running slowly. Jumped on to see this under CPU utilization:

Logs show nothing was happening at the time.

My response was:

Dunno.

Posted: 11/2/2009 3:33:33 PM EDT

[#1]

It could be a driver problem , that is where I would start IMO .

Posted: 11/2/2009 3:42:41 PM EDT

[#2]

Database Query ?

Posted: 11/2/2009 3:46:18 PM EDT

[#3]

that pattern means this:

That means you need a bigger boat.

Other than that, I got nothing.

TRG

Posted: 11/2/2009 9:01:30 PM EDT

[#4]

At least help us out enough to tell us what OS is on the server, and what it's duties are...

Posted: 11/3/2009 8:15:25 AM EDT

[#5]

Quoted:

At least help us out enough to tell us what OS is on the server, and what it's duties are...

Sorry, I figured the candy coat would be enough.

Mac OSX 10.5.8 Server.

It is running MySQL, Webserver and file sharing.

Not that that helps anything.

Posted: 11/3/2009 8:38:57 AM EDT

[#6]

was mysql running a vacuum process, or a re-indexing?

Posted: 11/3/2009 8:40:47 AM EDT

[#7]

Any automated tasks set up in MySQL? Maybe table index optimization or backup/replication?

Posted: 11/3/2009 9:45:57 AM EDT

[#8]

if it was memory, i'd say it was a memory leak.

i've never seen cpu usage climb stair like that, but only not busy or straight to the ceiling.

Posted: 11/3/2009 11:04:43 AM EDT

[#9]

Quoted:

if it was memory, i'd say it was a memory leak.

i've never seen cpu usage climb stair like that, but only not busy or straight to the ceiling.

It was CPU usage.

The climb occurred over about 1.5 hours, then that nice wave pattern for a few more. Then it just stops. Logs show nothing.

Posted: 11/3/2009 11:05:34 AM EDT

[#10]

Quoted:

was mysql running a vacuum process, or a re-indexing?

Not according to anything I can see. Drive activity, network access was damn near flat line during this time.

I have NEVER seen this, never.
Maybe it become self aware...

Posted: 11/3/2009 11:06:28 AM EDT

[#11]

One-time event (shrug) or periodic?

Posted: 11/3/2009 11:08:54 AM EDT

[#12]

Quoted:

One-time event (shrug) or periodic?

This is it. You see that perfect slow escalation?

Posted: 11/3/2009 11:24:44 AM EDT

[#13]

does the sql server have a query log??

maybe someone ran some odd recursive query or perhaps something like a regex expression that got caught in a backtrace.

i only say that because a regex that got caught in a backtrace would not use disk/network or memory usage, but would use a lot of cpu usage. ... I think.

Posted: 11/3/2009 11:35:35 AM EDT

[#14]

Quoted:

Quoted:

One-time event (shrug) or periodic?

This is it. You see that perfect slow escalation?

I do; it's ringing no bells. By periodic I meant (should have been clearer) was it an event that occurred once yesterday, never seen before or since –– which is clearly what I think you're saying. It's interesting enough that if it was periodically recurring, you could meter the hell out of the system.

Sorry, not a Mac guy at all. So the normal tools I'd suggest for a real-time analysis (top to figure out which process(es) looked strange; strace -f -p $PID -o $OUTPUT_FILE to attach and try to snag a clue [or truss or dtrace or ...) may not help.

NimmerMehr's suggestion about a bad regex is interesting; on the other hand, I would expect a much steeper curve for that sort of cause. I have also seen VMs and their launched applications slowly chew up CPU and/or memory.

Posted: 11/3/2009 11:43:26 AM EDT

[#15]

The timeline on the bottom part of the screenshot is very blurry.

It appears that the server was being used during business hours and ramped up accordingly, as well as dropped off as the work day finished.

What is on that server?

Posted: 11/3/2009 1:20:30 PM EDT

[#16]

Quoted:

does the sql server have a query log??

maybe someone ran some odd recursive query or perhaps something like a regex expression that got caught in a backtrace.

i only say that because a regex that got caught in a backtrace would not use disk/network or memory usage, but would use a lot of cpu usage. ... I think.

Nobody besides me in the building would know how to do that, nor would they need to. The SQL logs do not reflect anything like this.

Posted: 11/3/2009 1:21:53 PM EDT

[#17]

Quoted:

Quoted:

One-time event (shrug) or periodic?

This is it. You see that perfect slow escalation?

I do; it's ringing no bells. By periodic I meant (should have been clearer) was it an event that occurred once yesterday, never seen before or since –– which is clearly what I think you're saying. It's interesting enough that if it was periodically recurring, you could meter the hell out of the system.

Sorry, not a Mac guy at all. So the normal tools I'd suggest for a real-time analysis (top to figure out which process(es) looked strange; strace -f -p $PID -o $OUTPUT_FILE to attach and try to snag a clue [or truss or dtrace or ...) may not help.

NimmerMehr's suggestion about a bad regex is interesting; on the other hand, I would expect a much steeper curve for that sort of cause. I have also seen VMs and their launched applications slowly chew up CPU and/or memory.

Sorry, I know what you meant. I meant to say, this has only happened this one time.

There are no VM's on this bad boy. The damn thing is bizarre. But you have to love that wave after the initial climb.

Posted: 11/3/2009 1:28:58 PM EDT

[#18]

Quoted:

The timeline on the bottom part of the screenshot is very blurry.

It appears that the server was being used during business hours and ramped up accordingly, as well as dropped off as the work day finished.

What is on that server?

Did even you read the thread?

The only thing most people use this server for is file access and it serves a Gallery 2 photo library. There was no activity during the time from any outside source. No SQL activity in the logs, no disk activity at the time, no network activity...

Posted: 11/3/2009 8:09:01 PM EDT

[#19]

Quoted:

Quoted:

At least help us out enough to tell us what OS is on the server, and what it's duties are...

Sorry, I figured the candy coat would be enough.

Mac OSX 10.5.8 Server.

It is running MySQL, Webserver and file sharing.

Not that that helps anything.

Sorry, that's the theme I use with webmin with most of the servers I have to tinker with.

Check your cron logs. Might be something noted there during the timeframe of the resource hogging.

Posted: 11/3/2009 8:25:48 PM EDT

[#20]

Quoted:

Quoted:

At least help us out enough to tell us what OS is on the server, and what it's duties are...

Sorry, I figured the candy coat would be enough.

Mac OSX 10.5.8 Server.

It is running MySQL, Webserver and file sharing.

Not that that helps anything.

Sorry, that's the theme I use with webmin with most of the servers I have to tinker with.

Check your cron logs. Might be something noted there during the timeframe of the resource hogging.

Already looked, nothing.

Posted: 11/4/2009 5:49:26 AM EDT

[#21]

Quoted:

Quoted:
The timeline on the bottom part of the screenshot is very blurry.

It appears that the server was being used during business hours and ramped up accordingly, as well as dropped off as the work day finished.

What is on that server?

Did even you read the thread?

The only thing most people use this server for is file access and it serves a Gallery 2 photo library. There was no activity during the time from any outside source. No SQL activity in the logs, no disk activity at the time, no network activity...

We had a problem at our work with part of the Mac OS called "Spotlight".

It would start indexing at times and would slow down the whole shebang.

Don't tase me if you already considered this.

ETA: I just recalled having yet another problem with a RAID issue involving something called "Allow Host Cache Flushing." It was something you could disable. Check your RAID admin if applicable. Do a searcheroo on it.

Again,
Tase me gently.

Posted: 11/4/2009 7:37:09 AM EDT

[#22]

Quoted:

Quoted:

The timeline on the bottom part of the screenshot is very blurry.

It appears that the server was being used during business hours and ramped up accordingly, as well as dropped off as the work day finished.

What is on that server?

Did even you read the thread?

The only thing most people use this server for is file access and it serves a Gallery 2 photo library. There was no activity during the time from any outside source. No SQL activity in the logs, no disk activity at the time, no network activity...

We had a problem at our work with part of the Mac OS called "Spotlight".

It would start indexing at times and would slow down the whole shebang.

Don't tase me if you already considered this.

ETA: I just recalled having yet another problem with a RAID issue involving something called "Allow Host Cache Flushing." It was something you could disable. Check your RAID admin if applicable. Do a searcheroo on it.

Again,

Tase me gently.

Spotlight... I will have to check that. The "RAID" system is shit anyway, but I see no activity there.

I don't use these fucking things. These Xservers were an abortion of a mistake, fucking pieces of shit. I would list the official reasons that they were purchased over something better and cheaper, but your heads would explode.

Posted: 11/5/2009 5:01:14 AM EDT

[#23]

These Xservers were an abortion of a mistake, fucking pieces of shit. I would list the official reasons that they were purchased over something better and cheaper, but your heads would explode.

Apple always fails.

[ARCHIVED THREAD] - Any of you hogs want to take a stab at this one?

General » Urban Commandos

Warning

Confirm Action

About AR15.COM

Stay Connected

Newsletter

Contact Us