I have never felt unsafe around the Formosa Area the many times I have been there. I was there recently and took my 2 kids to play Jungle mini golf and then crossed the street to Walgreens. Not an issue.
As others have mentioned, Old Town is a little more 'colorful'.
However, as with anywhere, you should still be aware of your surroundings at all times and not let your guard down.