Life is going to have challenges, problems and disappointments no matter what. So wouldn't you rather have the challenges, problems and disappointments that come with building the dream life you actually want to live, rather than the ones that come with settling for a tolerable life that you're just trying to get through every day?
Avoiding the challenges of your dream life doesn't spare you problems and pain; avoiding the challenges of your dream life just means you end up with life's problems and pain and a sub-optimal life.
Better to have problems and pain and your dream life (or something much closer to it than what you have by settling and avoiding).
So go for it! Start building your dream life! What do you have to lose? Life can only get better if you lean in to your dreams and make your life more of what you've always wanted it to be.