It is imperative that the male, Jim Dandy, flowers at the same time as the females. They usually start flowering earlier but continue through the female bloom time. 

We have had drought for that last 3 years so that could cause them to drop their berries. Drynesss shouldn't be a problem this year. 

Do your berries get large and red and then disappear? That is probably birds. I know the Cedar Waxwings will come in and clean off my Winterberries within a couple days. 

Have you had your soil tested recently? You could give them a diluted fertilizer but be sure to not overfeed them. Look for a product like Berry-Tone  and follow the package instructions. winterberries like acidic soil but can tolerate many different types.

https://extension.umn.edu/trees-and-shrubs/winterberry

https://soiltest.cfans.umn.edu/

There are tiny buds on Jim Dandy, but they don't look like they've flowered to me.There also seem to be a lot fewer of them. One thing I noticed about Jim is that he's a lot smaller than the females. I wonder if that's natural or if it's because he's literally in their shadows and not getting as much sun. I'm going to trim some of the female branches back to make sure he gets enough sun.

I guess it's possible that Jim already did most of his blooming, I thought I saw blooms on him before, but I might have been mistaking blooms that were actually on the females since they are all pretty close together.

The berries never get beyond the green stage. They do form but they don't change color and they are smaller and fewer in number than my parents' winterberries (they live in Delaware). I haven't seen birds eating them but I don't see any on the ground either so that seems like a good bet. Will birds or other animals eat then when they're green? Would it help if I sprayed the berries with something to make them less palatable to birds?

I've attached a photo of the buds next to my thumb so you can see how tiny they are. My camera doesn't focus well enough when zoomed in that far and it doesn't help that it's twilight right now, but it should give you some idea.

The bushes get a lot of morning sun but are shaded in the afternoon.

I haven't had this soil tested. 

I'll look for Berry-Tone.

Both the male and female plants need sun to flower. More sun = more flowers. 

Generally the male is not as large or showy as the females. The flowers are smaller too. 

If you have morning sun until about 2:00, that should be sufficient. 

I wouldn’t spray them with anything. They probably just need more sun.