Hello,

Thank you for reaching out to Ask Extension!

Those sound like pretty large roots, especially at 4 inch diameter. In general, removing any roots larger than a 1" diameter can negatively affect the tree. You might not notice the damage right away, but eventually you will see signs of the root removal in the canopy with stress in the leaves and possible dieback.  Another main concern would be the structural integrity of the tree. Tree roots anchor the tree in the ground, and removing roots as large as 4" would leave the tree open to failure in a storm and/or high winds. 

It sounds like this tree was perhaps planted too close to the fence. We often see trees that are too big for the space they were planted in, or that were planted too close to a building or other structure, and it sounds like this might be your case too. 

Here is more information about root pruning and about tree roots in general and keeping them healthy:

https://hort.ifas.ufl.edu/woody/root-prune-guidelines.shtml

https://extension.colostate.edu/topic-areas/yard-garden/healthy-roots-and-healthy-trees-2-926/

I hope this helps!

Dawn