fruit on cholla cactus

now that last year's fruits are withered should I remove them from branches this spring to help pormote new growth or leave them alone. They look bad