@Kane,

Welcome backk ;-)

FKKids.Com has been on v.1.5 since forever on 8GB RAM, 1TB HD, some (my personal;) tweaks to MySQL parms for perf. FBFK does 200,000+ users, 1,000,000+ pageviews/mnth. We've not seen problems of the style that you're facing.

You really should not be having those issues - given that it is a dedicated box, unless MySQL needs fine-tuning. I use some monitoring tools on FBFK to check on MySql.

Be wary of 'cloud' hosting ! Very few Cloud hosters offer true Clouds (maybe Amazon among the few). Most offer some kinda souped-up, relabelled VPS as "Cloud".

PS: I could probably put your site on my FBFK Server with no hiccups ;-P

btw - "..12,000 users, 40,000 visitors, 300,000 pageviews a month.." we had similar numbers circa May 2009 running on a VPS. We moved to dedicated abt September 2009 with 30,000 users.

Hey Dhrup, nice to see ya!

Part of our issue could be non-optimized custom code.  We have a 1-min multi-threaded cron job that takes 5-10 minutes to complete, so there could be 20 or so of the same PHP process running concurrently.  But our host is also pretty bad with support so we want to get off them regardless. 

Do you think the performance improvements in Elgg 1.8 are worth the upgrade?

We're definitely gonna do some thorough testing with whatever new host we decide to go to before moving the live site.  Thanks for the heads-up about cloud hosting.

BTW, this is who we're looking at using for cloud servers:

http://www.gogrid.com/cloud-hosting/compare-gogrid-ec2-rackspace.php

A client of mine last year had $50 "Cloud" with same hosting company as mine when I was using a $100 small dedicated play-server for my fun-stuff - the difference in performance was abt 5-10x ;-)

I understand re: your tech support issues..

That CRON'ed code should not quite take 10 min exec time ! Probably might need code fine-tuning.

1.8 is quite good (as is 1.7.10) but, but.. lack of *PlugIns will getya ;)

We've wanted to move from 1.5 to 1.7.X but... who's got the time and energy to upgrade all he PlugIns we've got ?

I'd recommend a good reliable dedicated box (4GB+ RAM) for your next move, maybe some php code tuning to up the power on yr CRON scripts. PM me is you wanna have me study yr script/s code. It's what I do professionally - esp after all the tweaks we've needed for FBFK ;)

Kane, splitting to three servers as you mentioned should give you some performance boost. You can also have separate server for files. After that, you would have to use cloud architecture for database servers, or partition your data.

Thanks guys.  We're thinking that we should spend time analyzing the slow queries, maybe tweak some MySQL settings, before moving hosts.

Our server specs are pretty insane (quad-core Xeon processor, 8 GB RAM) for the size of our site, and our average memory and CPU loads are very, very low.  So obviously something else is going on and the hardware isn't to blame.

@Dhrup, the reason the cron takes so long is it's sending and receiving data from 20 or so other sites via APIs and whatnot, so that wait time is really the bottleneck.

Cheers!

cron - ;) i see..
enabled slow queries log ?
disable elgg log. 
cut down php error reporting
cut down apache logging