Weird indeed. I was able to duplicate it. Selected 4 jpgs on my desktop and double-clicked. One window with 3 images, a second window with the fourth. Closed all windows and double-clicked selection again and all four images in one window. Quit Preview and made a new selection of four images (2 from original group, two other images) and same deal. 1st double-click resulted in two windows, close and re-double-click selection and got one window. Hmmm, perhaps a threading issue with images handled by a different thread opening in another window? (WAG?)
Running 10.8.4/Preview 6.0.1 with the Preview option of open groups of files in the same window selected.