Good Morning, 

Thanks for your question. 

Please click on the following link about cankerworms: https://extension.umn.edu/yard-and-garden-insects/spring-and-fall-cankerworms  This may be the culprit. 

Or it could be this: https://extension.umn.edu/yard-and-garden-insects/sawflies

Both of the above links discuss how to deal with the worms. You did not indicate how widespread the infestation is or if they are damaging the leaves. 

In either of the above, if you decide to use a pesticide on them, a low impact pesticide is best. The sawfly information states: 

Low impact pesticides

Insecticidal soap and horticultural oil are effective when managing small numbers of young sawfly larvae.

• These products have minimal impact on natural enemies.
• Repeat applications might be needed as the product only affects sawflies it directly contacts.

Azadirachtin and spinosad are effective for one or two weeks so sawflies that feed on treated foliage are still affected.

Bacillus thuringiensis var. kurstaki (Btk) is a common low impact insecticide used against moth and butterfly caterpillars. It is specific to caterpillars and does not kill sawfly larvae.

The cankerworm website also states:

Use the bacterial insecticide Bacillus thuringiensis, whenever possible. Bacillus thuringiensis must be applied while the cankerworms are still small (less than 1/2") to get good results.

You should be able to purchase the above low impact pesticides at a nursery, hardware store or big box store. 

You might inspect your plants closely to see how numerous the infestation is.  If we get heavy rain today, they might get knocked off your plant. 

If this does not solve the issue, please let us know. 

Good Luck!