You probably have SWD, or Spotted Wing Drosophila. 

This is a new (to us in Minnesota) insect unintentionally imported from Asia. It is a truly evil insect with no known predators here in the US.

This insect has an ovipositor that pierces the fruit before it has reached maturity. Most insects have to wait until the fruit is ripe and soft to pierce their skin. Because the fly gets into the fruit when they are just forming, it is difficult to control.

You can pick the fruits young and use them. As they mature they will mold due to other fungus getting into the wound.

Clean-up is imperative and it is your best defense. Clean up all dropped fruit and clean off the tree if you can at the end of your harvest season. Do not home compost the fruit or bury it. Use your city compost facility.

Next year you can monitor for the flies early by using sticky traps. Depending on the size of your trees you can cover them.

These insects infect all soft fruit like plums, cherries, raspberries, blueberries, and strawberries.

Sorry for the bad news.

https://extension.umn.edu/yard-and-garden-insects/spotted-wing-drosophila#monitor-your-garden-for-swd-992610

https://www.canr.msu.edu/news/spotted_wing_drosophila_in_summer_plums_and_nectarines_are_a_concern_for_gr

https://bpb-us-e1.wpmucdn.com/blogs.cornell.edu/dist/0/7265/files/2016/12/SWDgarden-22blxxn.pdf