Likely one in the area...it's also probable you have something in your house drawing their attention. Something sweet left out maybe? Were they hanging around anything in particular, or just trying to get out?

Check around your home, and see if you can check around your neighbors or ask if they have any issues. Try to pinpoint a nest on the house or around it in trees, bushes, heavy wooded areas(like thick wooden beams, they like those here, we have a big chunk of wood holding up the gardens they and yellow jackets like to try and nest in, some may even have a hold under something or in the ground...though I'm not sure if hornets burrow like yellow jackets do).

Four is a lot. It doesn't mean that you have a nest in your house, but you probably have one nearby. I would at least suspect it.