Sweet potatoes can be harvested at any point once they have 'sized up'. That basically means-when the sweet potatoes have reached a size worth harvesting. The roots will keep growing throughout the season if the conditions stay right. Most sweet potatoes are ready to harvest between 90-110 days after planting depending on the variety and the growing conditions.
You might be able to gently dig down and peek at the roots (they will form around the base of the plant). If they are too small then rebury, water and let them keep growing. Or you can wait until the deer or the weather cause the foliage to die and then dig them out. Try to harvest before the fall rains begin so the roots don't rot in the ground.
Sweet potatoes have very delicate skin and need to be handled gently during harvest. You will also want to put them through a curing process to set the skin and increase ability to last in storage. This publication is from Arkansas (so use the growing info carefully, but describes the universal curing process very well.
Good luck and thanks for using Ask An Expert!