Thank you for your question to Extension and the pictures, which are very helpful in diagnosing the issue with your tree. I believe your tree is a crabapple tree by the way you describe it flowers in the spring and small fruit later in the summer. It looks like your crabapple has apple scab which infects apple and crabapple. It is a fungus disease that affects the leaves and eventually the tree. The close-up you shared of the trunk is lichen on the trunk because the tree is infected with the fungus. Here is information about what to do if your tree is infected with apple scab including applying a fungicide:

Apple scab of apples and crabapples | UMN Extension

Garden centers and nurseries sell crabapple trees that are resistant to apple scab. The article above tells you which ones to purchase.