APLawrence.com -  Resources for Unix and Linux Systems, Bloggers and the self-employed

Debugging a Mountain Lion slowdown

I have experienced some annoying performance issues since upgrading to Mountain Lion. So far, I have not been able to link cause and effect, but the symptoms are easy enough to describe: the system slows down so much that it's difficult to even get the Apple Menu to come up so I can logout or reboot. Load average will be high and there will be multiple swapfiles in /var/vm.

That's all I really know right now. Part of the problem, of course, is that things like this are annoying. When it happens, I'm almost always in the middle of something important and do not have the time to do extensive probing. It's hard to do that when the system is dragging and I want to get on with whatever I was doing, not paw through sar or try to fire up Activity Reporter.

I'm apparently not the only one experiencing this sort of problem. If you Google for "Over 3GB of swap files with 16GB of RAM?", you'll find discussions where people argue over whether this is really happening and if it is happening for the reasons other people think it is happening. I'm not giving links because most of these are at Apple and Apple has the nasty habit of removing threads it thinks are past their sell date. If you don't find any discussions.apple.com results that seem relevant, that's what happened.

Apple's parsimony with their disk space aside, those discussions seem to be similar to what I'm seeing, but there is so much blame thrown around that I suspect it's something more basic: some flaw in Mountain Lion that will probably quietly disappear in some future update. The discussions can be interesting, but a lot of it is people loudly insisting that the people experiencing the problem don't understand virtual memory and are therefore falsely maligning Mountain Lion. Those folks tend to launch into long explanations of how paging and swapping works and why none of that could possibly be an issue..

Yeah, fine: I get that and I agree. I'm just going by what I see. When I see this, there are multiple swap files and the load average is high and I don't know why.

I agree that paging shouldn't cause this. But that's "shouldn't". What if something is fundamentally flawed in Mountain Lion? What if paging is getting confused deep in the kernel and not popping back quickly enough under some circumstances? I'm not talking about design, I'm talking about accidental flaw. That's what I think the long explanations at those threads are missing: the possibility of a glitch in the kernel code.

But it's way too early for me to weigh in on that. I'm not even sure what conditions cause this. As I said, when it happens, I'm usually busy and just want to fix it quickly so I can get back to my work.

Suspicions

Oh, I have suspicions. I'm suspicious of Chrome, for one. Right now, with just 5 measly tabs open, Chrome takes up a huge chunk of processes because it puts each tab in its own space. That's good for preventing a misbehaving tab from affecting others, but I wonder if a particular tab starts leaking memory, paging that out might slow things down. It shouldn't, but I'm still suspicious:

$ ps -A -o pcpu,pmem,rss,vsz,comm | grep -i chrome 
  0.0  0.1  16152   790864 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper
  0.1  1.8 223920  1262416 /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
  0.0  0.0      0        0 (Google Chrome He)
  0.0  0.2  23280   796156 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper
  0.0  0.3  37144   794204 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper
  0.0  0.3  34380   788524 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper
  0.0  0.3  34660   795572 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper
  0.0  0.2  30504   783472 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper
  0.0  0.3  40512   792820 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper
  0.0  0.3  35516   818572 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper
  0.0  0.2  27512   782788 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper
  0.0  0.2  28212   782296 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper
  0.0  0.3  36252   792088 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper
  0.0  1.1 142800   976176 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper
  0.0  0.4  45848   834748 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper
  0.0  0.4  49528   924572 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper EH.app/Contents/MacOS/Google Chrome Helper EH
  0.0  0.1  12512   750632 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper EH.app/Contents/MacOS/Google Chrome Helper EH
  0.0  1.1 134260   920352 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper
  0.0  0.7  86048   869264 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper
  0.1  0.6  78452   861224 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper
 

I've also been suspicious of my screen saver, mostly because several people in those threads thought that could be a culprit. But mostly I'm suspicious of those damn files in /var/vm because it seems to me that when those build up, trouble soon follows.

Swapping or paging?

That's a good question, and I don't have the answer yet. As I have 12GB of RAM and am not doing anything extreme when this happens, I assume it's just paging. Maybe I opened up Parallels and haven't used it for a while, so a lot of it paged out. I tried that this morning and see this right now in /var/vm:

-rw-------  1 root  wheel   67108864 Oct 16 04:54 swapfile0
-rw-------  1 root  wheel   67108864 Oct 16 06:58 swapfile1
-rw-------  1 root  wheel  134217728 Oct 16 06:58 swapfile2
 

But the load isn't high now and paging shouldn't cause performance issues for programs I'm USING. That is, if I'm typing in vim and Parallels gets paged out, my vim session shouldn't be affected. If it is, the paging is way too aggressive or it is getting mired down in the kernel. So, I don't think that is the issue.

On the other hand, with 12GB or RAM, I sure as heck shouldn't be swapping, yet this slowdown feels just like that. I need to find out what is going on.

With that in mind, I wrote this script:

#!/usr/bin/perl
# watchme - Tony Lawrence, Oct 2012
$now=time();
unlink "/Users/tony/dumpstats";
$myfile="/Users/tony/problem$now.html";
while (1) {
 $uptime=`uptime`;
 @up=split /\s+/,$uptime;
 $swapfiles=`ls -l /var/vm/swap*`;
 $swap=`ls -l /var/vm/swap* | wc -l`;
 chomp $swap;
 print "$swap $uptime\n"; 
 $processes=`ps -A -o pid,ppid,pcpu,pmem,rss,vsz,comm`;

 if ($processes =~ /DashBoard/ or ($swap > 2 and $up[8] > 2) or (-e "/Users/tony/dumpstats")) {
   $iostat=`iostat`;
   $vmstat=`vm_stat`;
   $sar=`sar -A`;
   open(O,">$myfile");
   print O <<EOF;
<h2>System slow</h2>
<pre>
$uptime
$iostat
$vmstat
$sar
$swapfiles
$processes
</pre>

EOF
close O;
system("open $myfile");
exit 0;
}
 sleep(120);
}
 

Sar was already enabled with root crontab per "man sa1"; nothing else requires anything special. What this does is look for more than two swapfiles and a high load average. If it sees that, it writes out a pile of possible interesting stuff to a html file and then opens that file, interrupting my work very effectively. I set it running in a Terminal window and if it triggers, I have something to look at and can examine it again later (because the file name is timestamped).

By the way, opening a browser window definitely gets your attention. I had thought of using Growl or one of the suggestions at How to invoke alerts from the OS X Terminal, but opening a web page is far more disruptive and disruptive is what I want: I want to notice this when it triggers!

I can also trigger it with "touch ~/dumpstats" if I think things are getting slow. In either case, I exit after the trigger so will need to restart it if I'm not going to reboot or logout.

output produced on trigger

Why do I say "reboot or logout"? Because I'm not yet sure which really fixes this best. I know that logging out sometimes returns things to normal, but it hasn't always. That's another reason I suspect Chrome, by the way - Chrome may reopen the same tabs and get screwy again quickly.. but I don't KNOW that yet.

So, that's my plan: let this puppy run and see what happens. I'll report back if I learn anything.

Found it!

Found something, anyway:

 %CPU %MEM    RSS      VSZ COMM
 99.0 75.9 9556428 86272888 /System/Library/CoreServices/Dock.app/Contents/Resources/DashboardClient.app/Contents/MacOS/DashboardClient
 

I forgot to put pid and ppid into the script (changed it just now) so I don't know yet who called it, but I bet that's it.

Googling this says it's a leaky widget but I don't use any widgets, so something else has called it.. I'll find it.

Just the Dock..

Two days later.. I'm just doing my normal work and this triggered. There were 5 swapfiles of increasing size:

-rw-------  1 root  wheel   67108864 Oct 17 07:14 /var/vm/swapfile0
-rw-------  1 root  wheel   67108864 Oct 17 08:33 /var/vm/swapfile1
-rw-------  1 root  wheel  134217728 Oct 17 08:34 /var/vm/swapfile2
-rw-------  1 root  wheel  268435456 Oct 17 08:34 /var/vm/swapfile3
-rw-------  1 root  wheel  536870912 Oct 17 08:33 /var/vm/swapfile4
 

And "uptime" was high:

 8:34  up  1:20, 5 users, load averages: 2.12 1.75 1.41
 

I found the DashBoardClient consuming 100% CPU:

1922   339 100.1 11.5 1444524 10855052 /System/Library/.../Contents/MacOS/DashboardClient
 

But its parent is just the Dock:

  339   315   0.0  0.3  40672  2562304 /System/Library/CoreServices/Dock.app/Contents/MacOS/Dock
 

So I do not know what caused that to suddenly start running.

Moreover, I can't kill it. Or rather, I can, but the Dock starts up another instance which immediately grabs 100% of CPU.

A "killall Dock" got rid of it and it didn't restart immediately..

A sixth swap file has been added:

-rw-------  1 root  wheel  1073741824 Oct 17 08:44 swapfile5
 

Now that I know I'm looking for extreme cpu usage, I changed the code to this:

  $cpu=`ps -A -o pcpu | sort -rn | head -1`;
  $highcpu=0;
  $highcpu=1 if $cpu > 80;
  print "cpu $cpu";
 
 if ( $highcpu  or (-e "/Users/tony/dumpstats")) {
 


Got something to add? Send me email.





(OLDER)    <- More Stuff -> (NEWER)    (NEWEST)   

Printer Friendly Version

-> -> Debugging a Mountain Lion slowdown


8 comments



Increase ad revenue 50-250% with Ezoic


More Articles by

Find me on Google+

© Anthony Lawrence







Sat Oct 27 18:24:23 2012: 11404   TonyLawrence

gravatar


So "killall Dock" has become a fairly regular thing.. but I still don't know why those things get started.





Thu Nov 8 08:32:45 2012: 11418   NickBarron

gravatar


I do like a good mystery. Do let us know if you narrow down the problem!



Thu Nov 8 23:17:25 2012: 11425   TonyLawrence

gravatar


I still don't know why that DashBoardClient gets started, but killall Dock fixes it..



Fri Apr 12 03:19:52 2013: 12023   anonymous

gravatar


I wish I had that problem!

----------- 1 root wheel 67108864 Apr 11 08:56 swapfile0
-rw------- 1 root wheel 67108864 Apr 11 20:17 swapfile1
-rw------- 1 root wheel 134217728 Apr 11 20:17 swapfile2
-rw------- 1 root wheel 268435456 Apr 11 20:17 swapfile3
-rw------- 1 root wheel 536870912 Apr 11 20:09 swapfile4

Of course, I only have 4 Gb of RAM.

Per your analysis, I killed all the widgets I never look at anyway and will see how things go.



Mon May 20 23:40:49 2013: 12078   TonyLawrence

gravatar


This bit of code at (link) could be helpful too.



Fri Jun 7 11:00:32 2013: 12109   LauriRanta

gravatar


There is a hidden preference for disabling Dashboard:

defaults write com.apple.dashboard mcx-disabled -bool true; killall Dock

It might also prevent DashboardClient from being launched.



Fri Jun 7 11:34:23 2013: 12110   TonyLawrence

gravatar


Oh, that's great!

Thanks!



Mon Jul 1 21:54:52 2013: 12201   anonymous

gravatar


I want/need many tabs and so Google Chrome "Renderer" processes rapidly suck the life out Mountain Lion's real memory. Until I get motivated to create/find a more elegant solution I'm just opening Activity Monitor.app and slaying the smaller Renderer processes and then running /usr/bin/purge to de-zombie-fy my macosx box. When I need the tab's content again I just refresh the page. This is sort of crap I've come to expect from fat squabbling American Corporates I'm sorry to say.

------------------------
Kerio Samepage


Have you tried Searching this site?

Unix/Linux/Mac OS X support by phone, email or on-site: Support Rates

This is a Unix/Linux resource website. It contains technical articles about Unix, Linux and general computing related subjects, opinion, news, help files, how-to's, tutorials and more.

Contact us





I just invent, then wait until man comes around to needing what I've invented (R. Buckminster Fuller)

That's the thing about people who think they hate computers. What they really hate is lousy programmers. (Larry Niven)












This post tagged: