I have a very similar occurrence. However, instead of changing the buffer size, I disable/enable the device in Windows. Go to the speaker icon in the bottom right of windows, right click, go to playback devices, go to Speakers/Saffire Audio option, right click, disable, right click again, enable.
I know this isn't really a fix, but it might be a little faster than opening up the mix control and all that. Plus, you won't have to change the buffer size.
I hope somebody comes along and explains why it's actually happening, though.