Depending on the species you have and the location of the nest, winter may or may not kill off the nest. The new queens would try to find a place where they can hibernate and are usually attracted to sunlight to find a spot outside a house, though it may be under siding someplace else on the house.
The Vespa are interesting in that they chew up wood to make a wood pulp used to make their nests. This also means that they can and will chew through ceiling tiles, plaster board and similar material to get to the other side. A tiny hole which provides them an incentive to make it bigger, such as the smell of food or seeing sunlight, will encourage them. They only need a quarter inch or less to get through.
In your case, besides looking closely at the bedroom, check the area outside the room. Due to the cold weather, they may have found an opening which was warm enough for them to enter your house and then were attracted to the windows in an attempt to get outside to find food and possibly mate. If the weather warms up enough, they will most likely use the original entrance.
So the next step is to look around the outside of your house to find the external exit. This may not be where the nest is located. It may only be a means to get to an attic space or crawl space where the nest could be located feet away from the entrance, depending on the available space. I have seen nests in attics which were hanging from beams or walls 5 feet from the external exit though usually they are much closer.
Taking a vacuum cleaner to the ones on your window could solve the immediate problem. Finding the nest may require a professional if the problem persists in an area where people may be stung by a dying wasp which happens to crawl into a show or on the bed.