Thread drifting Input latency?

so I've been trying to figure this one out for days now, hopefully someone here can give me a answer.

I have my sapphire pro 40 set up as my default system in/output everything goes through the 40, this works a dream, everything in the system is set to run at 24-bit 48k, however I am experiencing a type of input latency that seems entirely unexplained at least to my eyes, the phenomena is as follows.

When dealing with any other software (Reaper, Obs, Discord, and so on) there seems to be a delay on the sound reaching my system and making its way to any target software that only manifests over time, initially the signal will reach it's goal with little no to latency at all, despite testing under a session with nothing else running for a low processing load multiple times I can't pin down any time scale for it to happen, after said arbitrary time limit a repetitively large delay begins to show, me being able to clap and have to wait over a second at time before the signal finally reaches the software it's hooked up with, if i continue to work regardless this delay can even shrink at times, Re initializing the connection between my software and the 40 has no effect on fixing this.

the only thing I could think of it being after discussion with a relative who is a Hifi buff was Clock drift between the pc and sapphire, but again this doesn't add up because my mix control software does not suffer from this delay, even when the latency is at it's worst Mix control will visually respond to a clap immediately without fail at the same time as responding to system sounds meaning the signal does make it to the system with no delay. on top of this the pc is sending output back to the sapphire perfectly throughout all of this and "Sync status" remains locked at all times so I feel clock drift is a red herring, the issue "seems" to exclusively lie somewhere past the point of the signal taking the signal, but before the point of the signal reaching any software that I have any control over said signal in.

i just don't know where or how to approach fixing this since every attempt to look online for similar issues only brings up posts on more common problems that don't help me.

Below is a quick visualization of exactly where the issue seems to be as to make sure no-one misunderstands.


if anyone has any idea's whats happening that would be great, thank you in advance.