I have experienced some annoying performance issues since upgrading to Mountain Lion. So far, I have not been able to link cause and effect, but the symptoms are easy enough to describe: the system slows down so much that it's difficult to even get the Apple Menu to come up so I can logout or reboot. Load average will be high and there will be multiple swapfiles in /var/vm.
That's all I really know right now. Part of the problem, of course, is that things like this are annoying. When it happens, I'm almost always in the middle of something important and do not have the time to do extensive probing. It's hard to do that when the system is dragging and I want to get on with whatever I was doing, not paw through sar or try to fire up Activity Reporter.
I'm apparently not the only one experiencing this sort of problem. If you Google for "Over 3GB of swap files with 16GB of RAM?", you'll find discussions where people argue over whether this is really happening and if it is happening for the reasons other people think it is happening. I'm not giving links because most of these are at Apple and Apple has the nasty habit of removing threads it thinks are past their sell date. If you don't find any discussions.apple.com results that seem relevant, that's what happened.
Apple's parsimony with their disk space aside, those discussions seem to be similar to what I'm seeing, but there is so much blame thrown around that I suspect it's something more basic: some flaw in Mountain Lion that will probably quietly disappear in some future update. The discussions can be interesting, but a lot of it is people loudly insisting that the people experiencing the problem don't understand virtual memory and are therefore falsely maligning Mountain Lion. Those folks tend to launch into long explanations of how paging and swapping works and why none of that could possibly be an issue..
Yeah, fine: I get that and I agree. I'm just going by what I see. When I see this, there are multiple swap files and the load average is high and I don't know why.
I agree that paging shouldn't cause this. But that's "shouldn't". What if something is fundamentally flawed in Mountain Lion? What if paging is getting confused deep in the kernel and not popping back quickly enough under some circumstances? I'm not talking about design, I'm talking about accidental flaw. That's what I think the long explanations at those threads are missing: the possibility of a glitch in the kernel code.
But it's way too early for me to weigh in on that. I'm not even sure what conditions cause this. As I said, when it happens, I'm usually busy and just want to fix it quickly so I can get back to my work.
Oh, I have suspicions. I'm suspicious of Chrome, for one. Right now, with just 5 measly tabs open, Chrome takes up a huge chunk of processes because it puts each tab in its own space. That's good for preventing a misbehaving tab from affecting others, but I wonder if a particular tab starts leaking memory, paging that out might slow things down. It shouldn't, but I'm still suspicious:
$ ps -A -o pcpu,pmem,rss,vsz,comm | grep -i chrome 0.0 0.1 16152 790864 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper 0.1 1.8 223920 1262416 /Applications/Google Chrome.app/Contents/MacOS/Google Chrome 0.0 0.0 0 0 (Google Chrome He) 0.0 0.2 23280 796156 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper 0.0 0.3 37144 794204 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper 0.0 0.3 34380 788524 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper 0.0 0.3 34660 795572 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper 0.0 0.2 30504 783472 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper 0.0 0.3 40512 792820 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper 0.0 0.3 35516 818572 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper 0.0 0.2 27512 782788 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper 0.0 0.2 28212 782296 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper 0.0 0.3 36252 792088 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper 0.0 1.1 142800 976176 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper 0.0 0.4 45848 834748 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper 0.0 0.4 49528 924572 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper EH.app/Contents/MacOS/Google Chrome Helper EH 0.0 0.1 12512 750632 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper EH.app/Contents/MacOS/Google Chrome Helper EH 0.0 1.1 134260 920352 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper 0.0 0.7 86048 869264 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper 0.1 0.6 78452 861224 /Applications/Google Chrome.app/Contents/Versions/22.0.1229.94/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper
I've also been suspicious of my screen saver, mostly because several people in those threads thought that could be a culprit. But mostly I'm suspicious of those damn files in /var/vm because it seems to me that when those build up, trouble soon follows.
That's a good question, and I don't have the answer yet. As I have 12GB of RAM and am not doing anything extreme when this happens, I assume it's just paging. Maybe I opened up Parallels and haven't used it for a while, so a lot of it paged out. I tried that this morning and see this right now in /var/vm:
-rw------- 1 root wheel 67108864 Oct 16 04:54 swapfile0 -rw------- 1 root wheel 67108864 Oct 16 06:58 swapfile1 -rw------- 1 root wheel 134217728 Oct 16 06:58 swapfile2
But the load isn't high now and paging shouldn't cause performance issues for programs I'm USING. That is, if I'm typing in vim and Parallels gets paged out, my vim session shouldn't be affected. If it is, the paging is way too aggressive or it is getting mired down in the kernel. So, I don't think that is the issue.
On the other hand, with 12GB or RAM, I sure as heck shouldn't be swapping, yet this slowdown feels just like that. I need to find out what is going on.
With that in mind, I wrote this script:
#!/usr/bin/perl
# watchme - Tony Lawrence, Oct 2012
$now=time();
unlink "/Users/tony/dumpstats";
$myfile="/Users/tony/problem$now.html";
while (1) {
$uptime=`uptime`;
@up=split /\s+/,$uptime;
$swapfiles=`ls -l /var/vm/swap*`;
$swap=`ls -l /var/vm/swap* | wc -l`;
chomp $swap;
print "$swap $uptime\n";
$processes=`ps -A -o pid,ppid,pcpu,pmem,rss,vsz,comm`;
if ($processes =~ /DashBoard/ or ($swap > 2 and $up[8] > 2) or (-e "/Users/tony/dumpstats")) {
$iostat=`iostat`;
$vmstat=`vm_stat`;
$sar=`sar -A`;
open(O,">$myfile");
print O <<EOF;
<h2>System slow</h2>
<pre>
$uptime
$iostat
$vmstat
$sar
$swapfiles
$processes
</pre>
EOF
close O;
system("open $myfile");
exit 0;
}
sleep(120);
}
Sar was already enabled with root crontab per "man sa1"; nothing else requires anything special. What this does is look for more than two swapfiles and a high load average. If it sees that, it writes out a pile of possible interesting stuff to a html file and then opens that file, interrupting my work very effectively. I set it running in a Terminal window and if it triggers, I have something to look at and can examine it again later (because the file name is timestamped).
By the way, opening a browser window definitely gets your attention. I had thought of using Growl or one of the suggestions at How to invoke alerts from the OS X Terminal, but opening a web page is far more disruptive and disruptive is what I want: I want to notice this when it triggers!
I can also trigger it with "touch ~/dumpstats" if I think things are getting slow. In either case, I exit after the trigger so will need to restart it if I'm not going to reboot or logout.
Why do I say "reboot or logout"? Because I'm not yet sure which really fixes this best. I know that logging out sometimes returns things to normal, but it hasn't always. That's another reason I suspect Chrome, by the way - Chrome may reopen the same tabs and get screwy again quickly.. but I don't KNOW that yet.
So, that's my plan: let this puppy run and see what happens. I'll report back if I learn anything.
Found something, anyway:
%CPU %MEM RSS VSZ COMM 99.0 75.9 9556428 86272888 /System/Library/CoreServices/Dock.app/Contents/Resources/DashboardClient.app/Contents/MacOS/DashboardClient
I forgot to put pid and ppid into the script (changed it just now) so I don't know yet who called it, but I bet that's it.
Googling this says it's a leaky widget but I don't use any widgets, so something else has called it.. I'll find it.
Two days later.. I'm just doing my normal work and this triggered. There were 5 swapfiles of increasing size:
-rw------- 1 root wheel 67108864 Oct 17 07:14 /var/vm/swapfile0 -rw------- 1 root wheel 67108864 Oct 17 08:33 /var/vm/swapfile1 -rw------- 1 root wheel 134217728 Oct 17 08:34 /var/vm/swapfile2 -rw------- 1 root wheel 268435456 Oct 17 08:34 /var/vm/swapfile3 -rw------- 1 root wheel 536870912 Oct 17 08:33 /var/vm/swapfile4
And "uptime" was high:
8:34 up 1:20, 5 users, load averages: 2.12 1.75 1.41
I found the DashBoardClient consuming 100% CPU:
1922 339 100.1 11.5 1444524 10855052 /System/Library/.../Contents/MacOS/DashboardClient
But its parent is just the Dock:
339 315 0.0 0.3 40672 2562304 /System/Library/CoreServices/Dock.app/Contents/MacOS/Dock
So I do not know what caused that to suddenly start running.
Moreover, I can't kill it. Or rather, I can, but the Dock starts up another instance which immediately grabs 100% of CPU.
A "killall Dock" got rid of it and it didn't restart immediately..
A sixth swap file has been added:
-rw------- 1 root wheel 1073741824 Oct 17 08:44 swapfile5
Now that I know I'm looking for extreme cpu usage, I changed the code to this:
$cpu=`ps -A -o pcpu | sort -rn | head -1`;
$highcpu=0;
$highcpu=1 if $cpu > 80;
print "cpu $cpu";
if ( $highcpu or (-e "/Users/tony/dumpstats")) {
More Articles by Anthony Lawrence - Find me on Google+ 2012-10-16
Click here to add your comments - no registration needed!
Sat Oct 27 18:24:23 2012: 11404 TonyLawrence
So "killall Dock" has become a fairly regular thing.. but I still don't know why those things get started.
Thu Nov 8 08:32:45 2012: 11418 NickBarron
I do like a good mystery. Do let us know if you narrow down the problem!
Thu Nov 8 23:17:25 2012: 11425 TonyLawrence
I still don't know why that DashBoardClient gets started, but killall Dock fixes it..
Fri Apr 12 03:19:52 2013: 12023 anonymous
I wish I had that problem!
----------- 1 root wheel 67108864 Apr 11 08:56 swapfile0
-rw------- 1 root wheel 67108864 Apr 11 20:17 swapfile1
-rw------- 1 root wheel 134217728 Apr 11 20:17 swapfile2
-rw------- 1 root wheel 268435456 Apr 11 20:17 swapfile3
-rw------- 1 root wheel 536870912 Apr 11 20:09 swapfile4
Of course, I only have 4 Gb of RAM.
Per your analysis, I killed all the widgets I never look at anyway and will see how things go.
Mon May 20 23:40:49 2013: 12078 TonyLawrence
This bit of code at http://www.paulbeard.org/wordpress/2013/05/20/swap-usage-monitor/ could be helpful too.
Fri Jun 7 11:00:32 2013: 12109 LauriRanta
http://lri.me/
There is a hidden preference for disabling Dashboard:
defaults write com.apple.dashboard mcx-disabled -bool true; killall Dock
It might also prevent DashboardClient from being launched.
Fri Jun 7 11:34:23 2013: 12110 TonyLawrence
Oh, that's great!
Thanks!
Mon Jul 1 21:54:52 2013: 12201 anonymous
I want/need many tabs and so Google Chrome "Renderer" processes rapidly suck the life out Mountain Lion's real memory. Until I get motivated to create/find a more elegant solution I'm just opening Activity Monitor.app and slaying the smaller Renderer processes and then running /usr/bin/purge to de-zombie-fy my macosx box. When I need the tab's content again I just refresh the page. This is sort of crap I've come to expect from fat squabbling American Corporates I'm sorry to say.
Don't miss responses! Subscribe to Comments by RSS or by Email
Click here to add your comments
If you want a picture to show with your comment, go get a Gravatar
Printer Friendly Version
Debugging a Mountain Lion slowdown Copyright October 2012 Tony Lawrence
Have you tried Searching this site?
Unix/Linux/Mac OS X support by phone, email or on-site: Support Rates
This is a Unix/Linux resource website. It contains technical articles about Unix, Linux and general computing related subjects, opinion, news, help files, how-to's, tutorials and more. We appreciate comments and article submissions.
Publishing your articles here
Jump to Comments
Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them. I resell or can earn commissions from the sale of some of these items. Links within these pages may be affiliate links that pay me for referring you to them. That's mostly insignificant amounts of money; whenever it is not I have made my relationship plain. I also may own stock in companies mentioned here. If you have any question, please do feel free to contact me.
I am a Kerio reseller. Articles here related to Kerio products reflect my honest opinion, but I do have an obvious interest in selling those products also.
Specific links that take you to pages that allow you to purchase the item I reviewed are very likely to pay me a commission. Many of the books I review were given to me by the publishers specifically for the purpose of writing a review. These gifts and referral fees do not affect my opinions; I often give bad reviews anyway.
We use Google third-party advertising companies to serve ads when you visit our website. These companies may use information (not including your name, address, email address, or telephone number) about your visits to this and other websites in order to provide advertisements about goods and services of interest to you. If you would like more information about this practice and to know your choices about not having this information used by these companies, click here.
Printer Friendly Version