Congratulations on your first vegetable garden! I hope you've been harvesting and eating those lovely kale leaves. Kale is a biennial plant, so you won't see flowers until next summer. Vegetable gardening can be frustrating for all of us not only first timers. Hang in there -- its worth the effort.

Cauliflower, broccoli and kale are in the plant family Brassica which are generally cool weather plants. However, cauliflower tends to like warmer temperatures while broccoli and kale like early spring and fall. Some growers recommend separating cauliflower, broccoli, and kale due to susceptibility to cabbage moths, aphids and other pests that attack the Brassica family. You don't mention seeing any insect pests, so this doesn't seem an issue.

There are a variety of reasons why your broccoli and cauliflower fail to produce heads. I'm assuming that you planted from seed which means it takes up to 100 days to for the plants to produce heads. You may want to use seedlings for your fall planting. 

Crowding the plants (18-24" between plants recommended), lack of soil nutrients (nitrogen, potassium, phosphorus) for cauliflower and boron for broccoli, fluctuation in temperatures and inadequate water are listed as the primary causes for flower (heads) failure. 

Since this is your first garden and it looks like a raised bed, I'm assuming that you purchased soil or created a mix. If you've used existing soil, you ought to have it tested to confirm that it has a balance of nutrients. Here's link for Franklin County if you need to get testing. Since the cauliflower and broccoli are not producing flowers (the heads that we eat) and leaves look good, the nutrient proportions could be off. 

Given our weather this spring, my guess is that the primary cause could be the early heat and plants too close together. Not much you can do about the weather, but you need to thin the seedlings so that they have 12-24" between them. Depending on your soil, you may need to add some fertilizer or something to the soil to create a better nutrient balance. I would also separate the cauliflower from broccoli -- kale is less fussy. 

Enjoy your gardens.

Michael:  I did overlook your primary question.  It doesn't matter is the short answer -- you can leave the plants or remove them. Either way, you will need to weed and pull old plants before you re-plant in the fall. Leaving the plants in the ground won't have a significant impact of soil nutrients.

If you do clear the plants now, you could cover the bare soil with newspaper or cardboard which will reduce some of the weeds. It might make your late summer weeding easier. 

Jane