Well, there is no cut and dry exact answer. It depends on what your running, what the server is for (web, email, gaming, VOIP, etc), etc
Generally speaking, you want to start by closing/disabling/removing any services that are not needed.
Consider what services you are running, and research if a different program my be better and faster. (i.e. nginx vs apache)
Do a full port scan and close any unneeded ports.
Monitor your server resources and see where you may be running short or need to further optimize.
One thing that is often overlooked, is make sure your site/code is properly secured and optimized! you can have a very secure and optimized server, and still see poor performance and security issues if the site is not properly coded!