Yes, you can. Bacillus thuringiensis (Bt) (Dipel, Thuricide and other brand names) is very effective on the younger, early instar caterpillars and does not affect insects other than caterpillars. Products containing Bt are available from many garden centers, big box stores, farm supply stores as well as online. You should apply this as soon as possible.
If you do not get good control with that, you can follow up with a product that contains spinosad (Captain Jack's Dead Bug Brew, Monterey Garden Insect Spray, Conserve Naturalyte and others).
As with any pesticide, be sure to read and follow all label instructions.