Unfortunately, yes, there are other issues besides borer insects - elderberry is susceptible to verticillium wilt, cankers can girdle stems and cause dieback, too. Aphids and spider mites can cause leaves to dry out and die.
You should get a positive diagnosis:
Submit branch and root samples to MSU Plant Diagnostic Lab. Submit branches that are still partly alive, and be sure to include the base of the stems where they have roots connected. There is a fee of $20 dollars or so. Their website has the form, fee schedule and instructions at this link: https://pestid.msu.edu
Getting an exact diagnosis would be best because it may not be an issue that requires the removal of the shrubs.
The concern here is verticillium wilt - If this turns out to be cankers, pruning out the affected stems 6 inches below the insect hole, or the discolored area on the stem, and destroying these will reduce the disease or insects. Aphids and spider mites are managed by a strong spray of water on tops and bottoms of leaves, or use insecticidal soap or an insecticide labeled for control of these insects.
Here are some references (links) for you:
MSU - Integrated Pest Management - Verticillium wilt - https://www.canr.msu.edu/resources/verticillium_wilt
University of Wisconsin - Madison - Verticillium Wilt of Trees and Shrubs Brian Hudelson, UW-Madison Plant Pathology -https://pddc.wisc.edu/wp-content/blogs.dir/39/files/Fact_Sheets/FC_PDF/Verticillium_Wilt_of_Trees_and_Shrubs.pdf
I hope this is helpful. Thank you for using our service.