Not sure why the latency would be variable. There should always be a minimum of a 75ms delay on any analog input to the Sonos system, so that Sonos can buffer the signal for play across other grouped speakers. But there shouldn’t be any variation over time on that. I think it can be longer, depending on the settings you put on that line in, but the ~75ms delay is built in to the software, and can’t be changed. A digital input, such as on the home theater soundbars don’t have this delay, until you then go and group additional rooms.
If you’re able to reproduce this issue on demand, it might be worth, within 10 minutes, submitting a system diagnostic and calling Sonos Support to discuss it.
There may be information included in the diagnostic that will help Sonos pinpoint the issue and help you find a solution.
When you speak directly to the phone folks, they have tools at their disposal that will allow them to give you advice specific to your Sonos system and network.
I’m not sure what you call major latency, but the way Sonos Five speakers work is to build a small buffer of data to ensure any grouped speakers work in sync. This 70msec delay applies even when the Five is not grouped. Only home theatre devices avoid this delay, which would otherwise introduce terrible lip-sync issues. But, when other speakers are grouped those other speakers will experience the delay.
Edited to add: Ok, Bruce got there first!