This looks like Symphoricarpos albus, also known as common snowberry. It's a neat native plant that is great for shady spots and streambanks, and makes good habitat for wildlife.
Here at Ask an Expert we're not medical professionals so we can't advise on the the potential toxicity of plants. However you can share the plant ID with your family healthcare provider to get their input. To the best of my knowledge the fruits aren't edible for humans, though they are enjoyed by some birds.
Whether any plant that is not edible in all its parts is "safe" for kids depends on level of supervision, and the child's inclination (or not) to taste unfamiliar plants. After all, some food crops have parts that can be toxic to eat, such as the leaves of rhubarb. An older child who can be trusted not to eat non-food plants or parts of plants would probably be fine just playing in a yard with snowberry, which isn't typically known to cause contact skin reactions, have thorns, etc.
I hope that information is helpful in figuring out what makes most sense for your family.