Watch your bees carefully. Are they going in and out of one hole or are there multiple holes? It is possible that what you have are carpenter bees. Carpenter bees are solitary bees and there will be multiple holes. https://www.extension.umd.edu/hgic/carpenter-bees https://www.extension.umd.edu/sites/default/files/_images/programs/hgic/Publications/HG29_Carpenter_...
If you can disrupt a bumble bee nest, they are likely to go elsewhere. If you try this, be sure to wear protective clothing and do it at dusk and go inside quickly when finished. Bumble bee nests do not usually persist like yellow jacket nests. Bumble bees are not as aggressive as yellow jackets. You might look in the Howard County library for childrens' books on bees. Bumble bees are very beneficial. vw