Lavender is challenging to grow well in the mid-Atlantic, and this is fairly normal for older plants in area gardens -- they tend to get a bit woody, open, and somewhat bedraggled over time. Pruning should be limited to dead growth, minor shaping, and dead-heading (removing spent blooms), since regrowth generally does not occur from other types of cuts the way it can on other shrubs. For the time being, just discontinue trimming (though any dead stems are always fine to remove) and try to keep the plant in the "sweet spot" of neither too wet nor too dry. That said, lavender is much more tolerant of drought than it is wet soil. Going forward, if it declines, that suggests the roots are stressed or dying, and our main suspect in that case would be overwatering or poor soil drainage. If you wanted to try again, using a raised bed or container gives you better management over soil wetness and drainage for herbs that can be picky about this condition.

Miri