The patches on the bark are an interesting life form called lichens which are harmless to the tree. You can learn about them here: 
https://extension.umd.edu/resource/lichen-algae-and-moss-trees/ They  tend to grow on still objects or slow growing trees that are not producing much new bark or limbs however.

Your tree looks like it is very stressed and may be in decline or dying. 
Did it leaf out well this spring with normal, healthy green leaves?
Many plants and trees in our landscapes are struggling with heat and drought stress, both this year and last.

You don't necessarily have to remove it right now, but if it hasn't looked healthy and full recently it is likely not going to recover well. As long as its fall wouldn't threaten people and property you can wait and see.
Here is our page that helps people decide when to remove a tree: 
https://extension.umd.edu/resource/how-do-you-decide-when-remove-tree/


Christine

Thank you so much for your prompt reply.  My tree is 22 yr old and I recently read that the life span of red buds is about 20 years. It did not leaf out well this spring and last year I noticed a decline in hardiness. I will probably have it taken down as it is about 12’ from the house. I appreciate the information you sent.

Should you decide to replace the tree, here is a link to the Marylander's Plant Trees program which offers a $25 off coupon on the purchase of a new native trees from participating nurseries: 
https://dnr.maryland.gov/forests/Pages/MarylandersPlantTrees/How-to-Choose-and-Care-for-your-Tree.aspx
You can see a rundown of the recommended trees to the bottom left of the page.


Christine