Long waiting time before transfering data to browser

Hi please take a look on this, I notice that my site's waiting for response time is very long above 2 sec. with single php.info file or any other small pic .

Rest of all processes take only ms to respond .

I am on VPS with 1.5 GB RAM, 2.27 GHZ CPU and 100 Mbps netspeed. In below snapshot no dns and no connectivity issue is looking. only waiting time is long which hurts.

I am using e-accelerator, mem cache, query cache for MySQL. Simple cache on. I test this on my local LAMP server and found same results which shows that no problem with server.

On peak time 50-70% of  CPU use and 40-50% of Ram usea So it is look alike no issue with Server stats.

Please advise me how to fix this issue , I think if we resolve this it helps all of elgg users.