You can start any business if you apply yourself day and night. If you want to start a business to enjoy your free time then you are doing it for the wrong reason. Starting a business is HARD work and DEDICATION.
I don't do graphic designing but I outsource and resell logos for my web designs.
Just to help you out, here are a couple of lead generation tactics that will help you out!
Sniff out companies spending money on advertising that have crappy designs. Logos, website, etc... You can find these businesses by simply googling keywords.
For instance I targeted plumbers. I Googled (now I use SEMrush too, but sometimes SEMrush is inaccurate with companies that just began advertising in last couple months) keywords relevant to plumbing. Then I looked for small plumbing companies paying for Google ads. If a company is paying for Google ads then they have $100 to spend on a logo. I pay $28 for a high quality from Pakistan. Pakistani graphic designers are the best bang for your buck. Just an FYI if you feel your work load getting too heavy.
Here's some examples:
As you can see the quality is pretty good and companies didn't have a problem giving me $100.
Mind you, after pin pointing the companies that needed graphic design, I woud call them on the phone. I used to be a waiter so my people/communication skills are pretty solid. People tend to like me on the phone. Im personable and crack jokes with them. Im not a comedian but business owners seem to be pretty quirky and its not hard to make them laugh. Make em' laugh and you've got em' in the bag.
Another thing is getting familiar with IFTTT. There are tons of ways to streamline leads into an account of your choosing.
You could put out ;nationwide alerts on craigslist for everytime someone is looking for graphic design.
Get creative with your lead gen and you'll have no problem.