Hi Lisa 
Thank you for your gardening question. It appears that you have ground nesting bees. I am attaching a few articles that you may find helpful. I am also attaching an article on yellow jackets which also can make nest underground. You would want to be very careful around them and identify them correctly. Ground bees are very important early season pollinators.  It is important to know their life cycle and using only cultural methods are recommended. The best method is to establish a thick lawn, because they will find bare ground, thin lawn very favorable. Fall is a great time to sow grass sees and help establish a thicker lawn.  I have attached a video regarding this.

 https://www.canr.msu.edu/news/solitary_bees_in_yards
https://insectlab.russell.wisc.edu/2021/04/22/in-defense-of-ground-bees/ https://ohioline.osu.edu/factsheet/HYG-2075-11
https://extension.osu.edu/today/fall-lawn-care