SimpleMachines & phpBB are good free choices to get you started to see if it's something worth pursuing. Niche forums can still make vibrant communities if your discussions are based around something your members are passionate about.
It's also easier to make money with certain niches, such as specific car models because, even if the members are not that active, if you use good SEO to bring in targeted visitors those visitors will click through your ads for specific car parts, etc., because they are genuinely interested in spending money on their car hobby, which can get very expensive.
For example, I collect classic cars and found a Porsche forum that was pretty much dead at only 1 new post a week from new members, but the front page had a big ad for an online Porsche parts store, so I clicked on it to see what they had. The forum owner earns for the advertising, whether there are new posts or not. My own niche forums earn from advertising 24/7, even when there is no new content. Of course being active is preferred!
People are passionate about their cars, pets, politics, religion, etc, so do your research before jumping in the deep end.
Any basic, shared hosting will do to get you started. You can always upgrade later.