The wood in your photo looks old and It is hard to tell what you are dealing with. This does not look like bagworms but you will have to look. Bagworms spin a cocoon like bag which consists of leaves from the plants they feed on. Open up the possible cocoon and look for larvae. http://extension.umd.edu/sites/default/files/_images/programs/hgic/Publications/HG32_Bagworms.pdf
Knock down the debris. Also, Look and see if there is a hole under the possible frass/cocoon. Look for any type of larvae. Some insects will pupate and this is not a problem.