Warning

 

Close

Confirm Action

Are you sure you wish to do this?

Confirm Cancel
Member Login

Posted: 11/2/2009 1:46:03 PM EST
Got a report that a server was running slowly. Jumped on to see this under CPU utilization:



Logs show nothing was happening at the time.

My response was:





Dunno.
Link Posted: 11/2/2009 2:33:33 PM EST
It could be a driver problem , that is where I would start IMO .
Link Posted: 11/2/2009 2:42:41 PM EST
Database Query ?
Link Posted: 11/2/2009 2:46:18 PM EST
that pattern means this:



That means you need a bigger boat.

Other than that, I got nothing.

TRG
Link Posted: 11/2/2009 8:01:30 PM EST
At least help us out enough to tell us what OS is on the server, and what it's duties are...
Link Posted: 11/3/2009 7:15:25 AM EST

Originally Posted By JeepAR15:
At least help us out enough to tell us what OS is on the server, and what it's duties are...

Sorry, I figured the candy coat would be enough.

Mac OSX 10.5.8 Server.

It is running MySQL, Webserver and file sharing.

Not that that helps anything.
Link Posted: 11/3/2009 7:38:57 AM EST
was mysql running a vacuum process, or a re-indexing?
Link Posted: 11/3/2009 7:40:47 AM EST
Any automated tasks set up in MySQL? Maybe table index optimization or backup/replication?
Link Posted: 11/3/2009 8:45:57 AM EST
if it was memory, i'd say it was a memory leak.

i've never seen cpu usage climb stair like that, but only not busy or straight to the ceiling.
Link Posted: 11/3/2009 10:04:43 AM EST

Originally Posted By NimmerMehr:
if it was memory, i'd say it was a memory leak.

i've never seen cpu usage climb stair like that, but only not busy or straight to the ceiling.

It was CPU usage. The climb occurred over about 1.5 hours, then that nice wave pattern for a few more. Then it just stops. Logs show nothing.
Link Posted: 11/3/2009 10:05:34 AM EST
[Last Edit: 11/3/2009 10:06:45 AM EST by schizrade]

Originally Posted By RyJones:
was mysql running a vacuum process, or a re-indexing?

Not according to anything I can see. Drive activity, network access was damn near flat line during this time.

I have NEVER seen this, never.



Maybe it become self aware...
Link Posted: 11/3/2009 10:06:28 AM EST
One-time event (shrug) or periodic?
Link Posted: 11/3/2009 10:08:54 AM EST

Originally Posted By vim:
One-time event (shrug) or periodic?

This is it. You see that perfect slow escalation?
Link Posted: 11/3/2009 10:24:44 AM EST
does the sql server have a query log??

maybe someone ran some odd recursive query or perhaps something like a regex expression that got caught in a backtrace.

i only say that because a regex that got caught in a backtrace would not use disk/network or memory usage, but would use a lot of cpu usage. ... I think.
Link Posted: 11/3/2009 10:35:35 AM EST

Originally Posted By schizrade:

Originally Posted By vim:
One-time event (shrug) or periodic?

This is it. You see that perfect slow escalation?
I do; it's ringing no bells. By periodic I meant (should have been clearer) was it an event that occurred once yesterday, never seen before or since –– which is clearly what I think you're saying. It's interesting enough that if it was periodically recurring, you could meter the hell out of the system.

Sorry, not a Mac guy at all. So the normal tools I'd suggest for a real-time analysis (top to figure out which process(es) looked strange; strace -f -p $PID -o $OUTPUT_FILE to attach and try to snag a clue [or truss or dtrace or ...) may not help.

NimmerMehr's suggestion about a bad regex is interesting; on the other hand, I would expect a much steeper curve for that sort of cause. I have also seen VMs and their launched applications slowly chew up CPU and/or memory.

Link Posted: 11/3/2009 10:43:26 AM EST
The timeline on the bottom part of the screenshot is very blurry.

It appears that the server was being used during business hours and ramped up accordingly, as well as dropped off as the work day finished.

What is on that server?
Link Posted: 11/3/2009 12:20:30 PM EST

Originally Posted By NimmerMehr:
does the sql server have a query log??

maybe someone ran some odd recursive query or perhaps something like a regex expression that got caught in a backtrace.

i only say that because a regex that got caught in a backtrace would not use disk/network or memory usage, but would use a lot of cpu usage. ... I think.

Nobody besides me in the building would know how to do that, nor would they need to. The SQL logs do not reflect anything like this.
Link Posted: 11/3/2009 12:21:53 PM EST
[Last Edit: 11/3/2009 12:29:51 PM EST by schizrade]

Originally Posted By vim:

Originally Posted By schizrade:

Originally Posted By vim:
One-time event (shrug) or periodic?

This is it. You see that perfect slow escalation?
I do; it's ringing no bells. By periodic I meant (should have been clearer) was it an event that occurred once yesterday, never seen before or since –– which is clearly what I think you're saying. It's interesting enough that if it was periodically recurring, you could meter the hell out of the system.

Sorry, not a Mac guy at all. So the normal tools I'd suggest for a real-time analysis (top to figure out which process(es) looked strange; strace -f -p $PID -o $OUTPUT_FILE to attach and try to snag a clue [or truss or dtrace or ...) may not help.

NimmerMehr's suggestion about a bad regex is interesting; on the other hand, I would expect a much steeper curve for that sort of cause. I have also seen VMs and their launched applications slowly chew up CPU and/or memory.


Sorry, I know what you meant. I meant to say, this has only happened this one time.

There are no VM's on this bad boy. The damn thing is bizarre. But you have to love that wave after the initial climb.
Link Posted: 11/3/2009 12:28:58 PM EST

Originally Posted By Cheesebeast:
The timeline on the bottom part of the screenshot is very blurry.

It appears that the server was being used during business hours and ramped up accordingly, as well as dropped off as the work day finished.

What is on that server?

Did even you read the thread?

The only thing most people use this server for is file access and it serves a Gallery 2 photo library. There was no activity during the time from any outside source. No SQL activity in the logs, no disk activity at the time, no network activity...
Link Posted: 11/3/2009 7:09:01 PM EST

Originally Posted By schizrade:

Originally Posted By JeepAR15:
At least help us out enough to tell us what OS is on the server, and what it's duties are...

Sorry, I figured the candy coat would be enough.

Mac OSX 10.5.8 Server.

It is running MySQL, Webserver and file sharing.

Not that that helps anything.

Sorry, that's the theme I use with webmin with most of the servers I have to tinker with.

Check your cron logs. Might be something noted there during the timeframe of the resource hogging.
Link Posted: 11/3/2009 7:25:48 PM EST

Originally Posted By JeepAR15:

Originally Posted By schizrade:

Originally Posted By JeepAR15:
At least help us out enough to tell us what OS is on the server, and what it's duties are...

Sorry, I figured the candy coat would be enough.

Mac OSX 10.5.8 Server.

It is running MySQL, Webserver and file sharing.

Not that that helps anything.

Sorry, that's the theme I use with webmin with most of the servers I have to tinker with.

Check your cron logs. Might be something noted there during the timeframe of the resource hogging.

Already looked, nothing.
Link Posted: 11/4/2009 4:49:26 AM EST
[Last Edit: 11/4/2009 5:13:57 AM EST by Cheesebeast]
Originally Posted By schizrade:

Originally Posted By Cheesebeast:
The timeline on the bottom part of the screenshot is very blurry.

It appears that the server was being used during business hours and ramped up accordingly, as well as dropped off as the work day finished.

What is on that server?

Did even you read the thread?

The only thing most people use this server for is file access and it serves a Gallery 2 photo library. There was no activity during the time from any outside source. No SQL activity in the logs, no disk activity at the time, no network activity...


We had a problem at our work with part of the Mac OS called "Spotlight".

It would start indexing at times and would slow down the whole shebang.

Don't tase me if you already considered this.

ETA: I just recalled having yet another problem with a RAID issue involving something called "Allow Host Cache Flushing." It was something you could disable. Check your RAID admin if applicable. Do a searcheroo on it.

Again,
Tase me gently.
Link Posted: 11/4/2009 6:37:09 AM EST

Originally Posted By Cheesebeast:
Originally Posted By schizrade:

Originally Posted By Cheesebeast:
The timeline on the bottom part of the screenshot is very blurry.

It appears that the server was being used during business hours and ramped up accordingly, as well as dropped off as the work day finished.

What is on that server?

Did even you read the thread?

The only thing most people use this server for is file access and it serves a Gallery 2 photo library. There was no activity during the time from any outside source. No SQL activity in the logs, no disk activity at the time, no network activity...


We had a problem at our work with part of the Mac OS called "Spotlight".

It would start indexing at times and would slow down the whole shebang.

Don't tase me if you already considered this.

ETA: I just recalled having yet another problem with a RAID issue involving something called "Allow Host Cache Flushing." It was something you could disable. Check your RAID admin if applicable. Do a searcheroo on it.

Again,
Tase me gently.

Spotlight... I will have to check that. The "RAID" system is shit anyway, but I see no activity there.


I don't use these fucking things. These Xservers were an abortion of a mistake, fucking pieces of shit. I would list the official reasons that they were purchased over something better and cheaper, but your heads would explode.
Link Posted: 11/5/2009 4:01:14 AM EST
These Xservers were an abortion of a mistake, fucking pieces of shit. I would list the official reasons that they were purchased over something better and cheaper, but your heads would explode.


Apple always fails.
Top Top