Based on your description, I'm going to offer several possibilities for your plum trees lack of flowers and fruit. 

Problems are generally divided into two categories, biotic or abiotic.  Biotic include bacteria, fungi, pests (bugs and animals).  Abiotic are non-living factors, such as weather, light, water, minerals, soil temperature, or mechanical.

Let's discuss possible abiotic factors first.  

1.  Weather can be inhibiting fruit set.  Tree blossoms need to be pollinated, and late frosts can damage blossoms, preventing this. Also, if there is a frost just after pollination has occurred, the initial stages of fruit formation can be damaged.  Your plums may have been in the critical stage, while your other fruits were not.

2. Proper pruning techniques are key to fruit production.  Your tree may be putting a lot of energy into making leaves, so remember to prune the tree once a year, in early spring or late winter, to remove unwanted growth. This will also maximize fruit-bearing surface and permit sunlight and air to circulate through the branches. Bear in mind that, with plum trees, flowers bloom on wood that is at least one year old, so proper pruning technique is essential.  

Now for biotic factors.

1. Pests.  As I look at the picture, I cannot see any gall formations or bug infestations, but you should do a close inspection to rule those out.

2.  The other possibility is root damage.  Voles and gophers can cause damage to the roots and girdle the trunks of trees, which will prevent the uptake of water and nutrients into the tree needed for blossom and fruit production.

I'm attaching several publications that provide more detail and offer other factors that could be impacting your trees.  

Why is there no fruit on my Tree?

Growing Fruits and Nuts in the Home Orchard

Training and Pruning your Home Orchard