The material growing on your branches is not mold, but rather lichens. Lichens are harmless, but their presence can sometimes indicate environmental problems, such as lack of nutrients or problems with soil moisture. I have uploaded a marked-up copy of one of your photos. The blue circle shows signs of cedar-apple rust. The red circle shows what looks like necrotic leaf blotch. The former is largely cosmetic, the latter can cause problems when widespread.

I would recommend a soil test in the fall to be sure soil nutrients are at adequate levels. Make sure there is adequate air circulation around the base of the tree (I can't tell how close the other plants are to the trunk, but they can harbor insects and encourage disease if air cannot circulate around the trunk at the ground). Annual pruning during late winter while the tree is dormant will also allow for better air circulation within the tree, which can reduce the incidence of disease. Provide supplemental irrigation equivalent to 1 inch of rain around the root zone during prolonged dry periods.

Lichens: 

https://www.canr.msu.edu/news/what_are_lichens_and_what_are_they_doing_on_my_tree

Cedar-apple rust:

 https://www.canr.msu.edu/news/cedar_apple_rust_a_tale_of_a_fungal_disease_with_two_hosts

https://www.canr.msu.edu/ipm/diseases/cedar_apple_rust

Necrotic leaf blotch:

https://www.canr.msu.edu/ipm/diseases/necrotic_leaf_blotch

Soil testing:

https://www.canr.msu.edu/news/soil-testing-returns-through-msu-extension

Growing fruit trees:

https://www.canr.msu.edu/resources/considerations_for_growing_backyard_tree_fruit

https://www.canr.msu.edu/resources/smart-gardening-pruning-and-training-backyard-apple-and-pear-trees

https://www.canr.msu.edu/news/spring_fertilization_of_garden_fruits

https://www.canr.msu.edu/courses/backyard-fruit-101