You can and should remove bagworms any time you see them.
Half of the bags will be empty because the male leaves to visit/fertilize the female in her bag. The females stay in their bags, lay her eggs and die there, so half of the bags are full of lots of eggs that you don't want hatching.
Be sure to dispose of them in the trash.
Here is our page about them, including their control: http://www.extension.umd.edu/hgic/bagworms-trees-and-shrubs