Wow!  I think you need to talk to your neighbors to get an accurate time line.  If the tree was originally planted at street grade and the original owner did that to it 40 years ago, I am astonished that the tree is still alive.

The second thing I'd do before digging around the tree is to consult a certified arborist:
http://www.isa-arbor.com/findanarborist/findanarborist.aspx

Read here about changing the grade around a tree:
http://www.extension.umn.edu/garden/yard-garden/trees-shrubs/protecting-trees-from-construction-dama...

I see three possibilities:  the tree has adapted, somehow, to this strange situation - if so, leave it alone, don't change the grade.  Or this pile was applied more recently and the tree is now declining. I doubt there is much you can do in this case.  Removal now before it becomes a hazard might be your best choice.  Finally, if the pile was very recent, perhaps the tree can be salvaged by removing the rock/dirt mix. But you need to have an arborist look to see what your best course of action is.

Thanks for the information. The tree has been in this state for over 3 decades. I think it should be left alone. No neighbor was there since the original owners of the property bought it in 1960.

Amazing.  Raise the grade six inches usually does serious harm.  We say twelve inches will kill the tree.  Your photo looks like 18 inches or more.

I agree with your decision. The tree has survived 30 years - don't mess with it.