This appears to be some kind of insect damage. There are several insects that enjoy Broccoli and cauliflower plants. I cannot say from the picture what insect; it may be more than one. You will likely need to do some investigation to figure out what insect(s) are causing the damage. Often if the damage is less than 30% of the plant it does not need to be treated. If you do need to treat, the following articles give recommendations to treat. If you choose to treat, make sure you read the label of any insecticide you use and follow all label directions including the insect that is found.

It appears there may be some frass (insect droppings) on the leaves, but I cannot be sure from the pictures. The following articles show typical insects that damage broccoli family plants and how to identify them. To find insects you may need to check at night with a flashlight. If you do find pictures of specific insects that you cannot identify from the photos in the article you can send in pictures of insects.

https://extension.umn.edu/vegetables/growing-broccoli#disorders-3186112

https://extension.umn.edu/vegetables/growing-cauliflower#insects-3188060