Load speed is one of the ranking factors that the search engines use in their ranking algorithm, but it's just one of very many. Page load speed matters, but not as much as others would lead you to believe.
Unless your website is drastically slow then speed isn't going to have a huge impact on your rankings. If you site is very slow, then speeding it up will help it out. However, shaving off a ten of a second or 2 isn't going to make any kind of big impact.
The search engines use it as one of their ranking factors because if someone goes to a site in their results and it takes too long to load, then many people will just hit the back button on their browser and not wait for a site to load. People are getting more and more impatient these days, so if a site is just super slow, then they won't want to stick around to see what loads. The SE's want to try to give the searchers a site they will go to and look at so that reflects on the quality of their rankings.
All said, if your site is very slow, then speeding it up will help it's rankings. If it's already loading at a decent time (2-4 secs) then speeding it up a little isn't going to launch your site a few pages up in the rankings.