They will not leave. You will need to get a can of wasp and hornet spray and spray late evening or early morning when they are all in the nest. You may have to spray twice to get them all.
How does spraying get to the underground nest?
Usually there is only one entrance and when the spray foams and forms a "plug" the bees are trapped on one side or the other. They have to try to pass through it and die. Depending on the size of the nest more than one application may be needed to kill them. Make sure you get a spray that foams up and is designated for yellowjackets.