Lots of factors can contribute to premature fruit drop in peaches and their relatives. Is the tree managed with pesticides (organic or otherwise) to prevent the array of common pests and diseases that can affect peach? If not, they are likely at least a partial cause for this fruit shedding. Our Growing Stone Fruits web page goes over the typical ailments of peaches and how/when to spray for them (in the link by Virginia Tech included in the "pest management" section. There is also a phenomenon called June Drop, where immature fruits are shed naturally by the tree, which may be playing a role despite it still being late May.

Miri