Is it worth trying to get ranked on the first page of Google for highly competitive keywords or should I move on to other forms of internet advertising?
which one is better? why?
It depends on your situation. If you are asking this question then I assume you probably aren't all that experienced with SEO. Assuming that is the case, then no I wouldn't recommend you try to target high competition phrases. If you are going to target those phrases, then you probably ought to think about outsourcing some of your SEO especially link building. And if you need to outsource your linkbuilding then it might cost you more than it is worth depending on your site and how much you stand to profit from a top ranking.
Without having specific details of your situation I would say you are probably better off targeting lower competition keywords and/or looking for other sources of generating publicity and traffic. That might include paid advertising and it might not. Again it depends on how much money you are likely to make from the traffic.
My advice to getting traffic for a website usually goes like this - First build a site that benefits a specific type of person and make the site good enough that this person will like the site once they discover it. Then identify places online where this person is spending time (might be certain blogs, forums, Facebook groups, Twitter, search engines, other websites, etc.) and figure out what you need to do to get those people to discover your site for the first time. Maybe that means you join a community and participate in it like you are doing on this forum. Maybe that means you need to buy advertising. Maybe that means you need to use SEO to target a certain phrase that you know your audience is searching for in Google. Maybe that means you should try to publish a guest blog at a popular blog in a certain niche. All of those are possibilities. It depends on your situation.