Unfortunately it's very likely that a domesticated (whether feral or owned) cat is using your growing beds as a litterbox. It will be important to glove up (and then disinfect gloves) to dig out and dispose of these droppings and the soil immediately in contact with them. Toxoplasma gondii is the foremost health concern from cat feces, but there are others - The main thing is to be safe in removing the scats, and then assess to what extent you'd need to remove and replace soil - if this has been going on awhile. To prevent further use, you might consider an electrified chicken fence or using a smaller mesh for the fence enclosing the area with your beds and then maybe an electric wire and/or "outrigger" to prevent climbing-in by the cat(s). Once you install a more cat-proof exclusion barrier, it would also be good to use (or borrow from a friend/neighbor) a game camera to confirm that nobody (cats) is making it inside.