Thank you for reaching out to Ask Extension. To better answer your question, it would be helpful to know why you want to attach the fence to the tree. Drilling holes into a tree to attach hardware comes with inherent risks to the tree that make it susceptible to disease and pathogens, and, as a consequence, decay. The fewer holes used the better. Using thick dynamic straps rather than metal hardware is better yet. Finding alternative ways to support the fence is even better. Keep in mind that steel nails and screws left in a tree can become hazardous for tree care professionals in the event that the tree needs to be removed in the future. It may also be useful to consider that the tree will continue to grow towards the fence, so anything attached to the tree will need to be adjusted periodically. 

Let us know if you have additional questions.