Yeah, the problems we've seen with physical media are intermittent and seem to also be model-specific and guest OS-specific... so sometimes it'll work on the second try, sometimes it'll fail in a different place every time... and on many machines it'll work just fine every time. For whatever reason (disc access patterns?), installation of specifically Mac OS X Server 10.6.3 guests seems to be one of the worst affected...
Making an .iso is definitely the "safe" route, and is usually more efficient and reliable, but it isn't always necessary.
Enjoy your Snow Leopard Server, and thanks for sharing your experiences with the community. 
--
Darius