I suspect the system isn’t getting the ‘I’ve stopped at this location’ information when you’re disconnecting. Make sure you stop playback in the Sonos controller, before you disconnect the speaker. That ‘should’ send the appropriate termination request through.
However, if you’re already doing this, I’d be awfully tempted to submit a system diagnostic within 10 minutes of experiencing this problem, and call Sonos Support to discuss it.
There may be information included in the diagnostic that will help Sonos pinpoint the issue and help you find a solution.
When you speak directly to the phone folks, they have tools at their disposal that will allow them to give you advice specific to your Sonos system and network.
Which music service are you playing your podcasts through?