I've been fishing many times at WDW and it's worth it to go earlier in my opinion. The fish are hungry after a long dark night and the light is beautiful early in the morning. We've actually seen deer at the water's edge at Bay Lake at first light.
Depending upon where you are fishing out of, you can make a breakfast reservation in advance and enjoy breakfast as soon as you get off the water.
For example, when we fish at Bay Lake we have the boat take us over to the Poly at trips end for a breakfast reservation at Kona. Failing to secure a reservation there, we just head over to Captain Cooks for a quick service breakfast and then hit MK for a couple of hours. There are lots of great breakfast options at all of the fishing locations.
Earlier is better for many reasons, the best of which is that the fishing is better. Whatever you decide, you are in for a treat as the fishing at WDW is a memorable experience.
Have a "fin"tastic trip!
~NM