Should I be worried about this tree?
These are lichens. Lichens are neat in that they are a fungus and an alga living symbiotically in one body. They are not hurting the tree, simply using it as a place to hang out. No treatment is necessary.
Check out this Clemson HGIC video to learn more about Lichens: http://www.clemson.edu/extension/hgic/videos_posters/videos/lichens.html.
After you watch the Clemson Video, take a look at this University of Maryland Extension article from a couple years ago about exploring lichens: https://extension.umd.edu/learn/i%E2%80%99m-lichen-it-peak-season-lichen-peeping