This is usually known as June fruit drop. The primary (and usually only) symptom of June fruit drop is small, immature fruit falling off trees. This can happen in apple and citrus trees and also in stone fruit like plums. It occurs usually because there is too much fruit trying to develop on the tree. Early thinning of excess fruit can solve most of the problem. Drop can also be caused by environmental extremes. The tree is conserving its energy for next year's buds.