Investigating my wonky groups

So why would 4 devices cause it to take up to 12 seconds for group members to stabilise and play when the coordinator plays constantly?

• The issue only consistently occurred when the stream changed. So starting to play, changing track.
• Adding/removing group members seemed more random. Most of the time they would join without issues.
• If left to play an album or playlist, the disturbance only occurred within the first 12 seconds, then never again.
• Volume worked perfectly all the time

That sounds like some sort of change in the other group members or the stream being modified/shaped as more players are added.

Receive buffer being increased in group members? Has to be something which doesn't interrupt the coordinator because that plays perfectly, so not on the fly compression, that would affect everything.

Due to having to reset the first 4 speakers to add wifi settings, my local library wasn't available so I'd been using Qobuz as my only source. Switching back to SonosNet to remove wifi and all it's fun and games from the equation, I checked the Network Matrix.

• ODFM 0 for everything
• Noisefloor between -100 and -110 for everything
• Green boxes everywhere. This looked better connected than my previous years of yellow boxes that worked.

One concern on wifi, the RX/TX rates didn't look great. 54/24 and 60/130... what on earth is inside these things? Nothing great for the Gen 1 Symfonisks unfortunately

https://support.sonos.com/en-gb/article/supported-wifi-modes-and-security-standards-for-sonos-products

Hmmm, so I have 5 x 802.11a and 2 802.11n devices. What is wrong with that gen 2 bookshelf it only connects at 60/130 when n is capable of up to 600? What's the betting that Port I got rid of had a better wifi connection? And the group controller I'd been using only had 24 rx. on a good day with the wind behind it 😕

Maybe the devices aren't doing anything funky with buffers or shaping, are the just being maxed out?
How does the coordinator download the stream when playing?

Well that's going to hurt 14/15Mbit initially which calms down until the whole track is cached, then -5 seconds before the next track same thing. So the first 12 seconds fits right in the initial burst when a stream changes and no player has anything in it's buffer. So if transmit was maxed sending to the group, until the buffer/cache files. 🤔

Starting before the current track ends would prevent a break in the stream so is seamless playing the whole album or playlist.

No bandwidth would also mean wonky controller and why adding the group member mid-track didn't impact.
 

What was I actually playing to test things? 24/44.1 and 24/48 albums as it turns out. Something the Sonos could actually retrieve from Qobuz, so wouldn't use the 16/44.1 version.

Right, need something smaller. No local library yet, so 16/44.1 flac it is. Flawless. Lets add more speakers until it breaks. 1 short of getting them all.

Now we're getting somewhere. What about a bit of Daft Punk, Qobuz provides as 24/88.2, so that means in reality Sonos plays the 16/44.1 flac. Again flawless.

So is this a firmware change that has altered the download behaviour? It's clearly being held back only reaching 17Mbps on the 24/48 albums and is lower for 16/44.1
What did I used to play to all 11/12 devices?
Radio usually for background, so 128Kbps on a good day if really lucky 256kbit.
And for 4 devices between two rooms. Definitely some of my 24/44.1 collection, but that would be full album and not in the same room when starting. So maybe it's always been this way.
What did that update which fixed the volume control change? Websocket to the group so always a connection available?

While waiting for the library to index, time to find out what Yamaha and Airplay do different downloading 24/192 sources, because they did much better. Three notable things
The yamaha gear and ipad have significantly faster wifi connections both up and down

Yamaha pulls a max internet speed chunk in before starting to play and then 5-6 mbps chunks, so lower download Qobuz & Airplay doesn't even bother downloading if it's cached, then it's a fast chunk before limiting the speed. The first just catches that my dlna->openhome converter pulls everything asap, but it then streams to the devices at whatever rate they want.

Back to the local library, 330kbit mp3s first. Flawless to all speakers grouped.

What about a 16/44.1 flac... same device limit when streaming 16/44.1 from Qobuz

And a 24/48 concert track? Less speakers again.
 

Where does this fit with random behaviour we've been seeing across Sonos users?

Is this a firmware change that has altered the download behaviour? It's clearly being held back only reaching 17Mbps on the 24/48 albums and is lower for 16/44.1

• What did I used to play to all 11/12 devices?
• Radio usually for background, so 128Kbps on a good day if really lucky 256kbit.
• And for 4 devices between two rooms. Definitely some of my 24/44.1 collection, but that would be full album and not in the same room when starting. So maybe it's always been this way.
• What did that update which fixed the volume control change? Websocket to the group so always a connection available?
   

Not all devices are created equal. If you have a mix of old and new and an old one is the group controller, it's going having a touch time with lossless and higher content. If your wifi design (quite possibly antenna) causes lower than max connection rates, then again the group controller will have a hard time.

Combine that with not knowing what bit rate you're actually streaming and the rise of lossless/hires content. On lower bandwidth wifi or SonosNet that puts a strain and with more speakers increases the likely hood of saturating what's available. Maybe that's why the Boost wasn't/isn't being supported in the brave new world. With the way the group controller downloads it doesn't take much to overload SonosNet if you have lossless content.

When the connections are saturated the controller can't get requests and responses from the devices. In the first volume slider dance people were suffering, when do most people change the volume of a track? At the beginning during the peak download rate maybe?

With Ethernet only devices you have more bandwidth than you'll ever need. A couple of separate SonosNets and an 802.11N nearer it's 600M max, you'll be wondering why everyone else is having troubles.

So between devices with different wifi capability, SonosNet that can't cope with lossless to more than a handful of devices and people streaming at many different bit/sample sizes, but not really knowing what, when can then send the controller app into an unresponsive or slowdown frenzy it doesn't recover from.

It's not much of a surprise trying to find commonality has been difficult. While it is ultimately network related, especially for SonosNet or the devices with old/odd wifi speeds, it's not something I'd expect to be for users to fix or even know.

It does raise even more questions in this increasingly bandwidth hungry streaming world, especially for those with devices that only have 54Mpbs Wifi connections.

Combine that with not knowing what bit rate you're actually streaming and the rise of lossless/hires content. On lower bandwidth wifi or SonosNet that puts a strain and with more speakers increases the likely hood of saturating what's available. Maybe that's why the Boost wasn't/isn't being supported in the brave new world. With the way the group controller downloads it doesn't take much to overload SonosNet if you have lossless content.

When the connections are saturated the controller can't get requests and responses from the devices. In the first volume slider dance people were suffering, when do most people change the volume of a track? At the beginning during the peak download rate maybe?

With Ethernet only devices you have more bandwidth than you'll ever need. A couple of separate SonosNets and an 802.11N nearer it's 600M max, you'll be wondering why everyone else is having troubles.

So between devices with different wifi capability, SonosNet that can't cope with lossless to more than a handful of devices and people streaming at many different bit/sample sizes, but not really knowing what, when can then send the controller app into an unresponsive or slowdown frenzy it doesn't recover from.

It's not much of a surprise trying to find commonality has been difficult. While it is ultimately network related, especially for SonosNet or the devices with old/odd wifi speeds, it's not something I'd expect to be for users to fix or even know.

It does raise even more questions in this increasingly bandwidth hungry streaming world, especially for those with devices that only have 54Mpbs Wifi connections.

Awesome work, @sigh, bravo!  👏👏👏

I was on the edge of my seat throughout … cannot wait for the movie version. 

It’s always fun to understand why things are broken. It’s why we all like major incidents at work, the customers and incident managers not so much 😂

One thing I didn’t mention is what appeared to cause the controller app to corrupt the user data and one speaker to give up providing airplay. Nothing unusual from my perspective, my wifi access point needed a firmware update, so had to reboot.

The iPad unexpectedly losing wifi gave me a broken app. Doesn’t seem to be something unusual for a tablet. Walk out of range before sending it to sleep, restarting a wifi AP. Shouldn’t have to stop everything or shut it down to do that.

I tried deleting the app only, leaving user data. After reinstalling it was still broken the same way. Had to delete the user data as well, so something in there was corrupt and preventing the app launching enough to even make internet requests. First time I’ve had an app do that.

The wayward speaker seemed to get it’s software in a mess and when it rejoined wifi there was no airplay, just Sonos playback. All the others were fine. 🤷

I’m not sure I’m understanding what you’re saying the issue is, which is not to dispute your findings, just that I’m not getting my head round it. 

16/44.1 is only around 1.4Mbps, 24/44.1 around 2.1Mbps and 24/48 will be around 2.3Mbps. 

All of which are way below even an ancient 54Mbps WiFi connection’s theoretical maximums and hopefully still way under its achievable bandwidth with decent-ish signals. 

One thing I’ve noticed on all my devices bar somewhat ironically the oldest Play:3 is that I get a large number of dropped packets on br0. Even on a relatively new Port which is ethernet connected with WiFi off… For example:

br0       Link encap:Ethernet  HWaddr 38:42:0B:89:36:90            inet addr:192.168.11.190  Bcast:192.168.11.255  Mask:255.255.255.0          inet6 addr: fe80::3a42:bff:fe89:3690/64 Scope:Link          inet6 addr: fddb:cd92:62a7:d643:3a42:bff:fe89:3690/64 Scope:Global          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1          RX packets:11186645 errors:0 dropped:2210960 overruns:0 frame:0          TX packets:2964160 errors:0 dropped:0 overruns:0 carrier:0          collisions:0 txqueuelen:1000           RX bytes:331790708 (316.4 MiB)  TX bytes:1596358387 (1.4 GiB)

Now in general, everything seems to be ok, bar known functional limitations of the app, However, I don’t really understand why I’m getting so many dropped frames on all bar the Play:3? The Play:3 has actively been involved in groups with other speakers too, so isn’t used completely in isolation. 

Does handling large amounts of dropped frames use more processing power? What’s ‘special’ about the Play:3 ? 

So they issue I had is down to the 54Mbit Wifi Bandwidth when a track starts playing.

The bandwidth claimed on Wifi is unfortunately, always the theoretical max.

Among other things, when playing music the group coordinator retrieves the file/stream to play, decodes, plays locally and sends the data to the other devices in the group.

As you say Stereo 16/44.1 is ~1.4Mbit per second to play. From that outgoing bandwidth needed from the group coordinator is easy to work out. 1.4Mbps X number of member devices. 5 devices ~ 7Mbps, 10 devices ~ 14Mbps

Picking a couple of file sizes from my nas, which are in bytes, so x8 for bits.

Flac 16/44 is 35 MByte or 280 Mbit

Flac 26/48 is 57 MByte or 456 Mbit

The group coordinator the needs to retrieve the file/stream and send to the other devices to play. The group coordinator isn’t streaming the source at playback speed it is retrieving much faster into a local buffer, while also trying to send data to the group members, who will most likely also have a buffer to fill.

The 24/96 graph shows the speed the coordinator was retrieving the 456 Mbit long source. From Qobuz, or my NAS it loads as much of the data in as it can to fill the local buffer, then during playback if there is more data it keeps the buffer full until all of 456 Mbit Flac content is retrieved.

So with actual usable bandwidth of ~20 Mbps at the start of playback, the group coordinator was using up to ~15 Mbps to download, leaving only ~5 Mbps free to send the data to the group members during that time.

Once the download completes or the buffer fills the ~20 Mbps on the wifi can be used just to send data to the other members. When 5 seconds from the end of the track, the coordinator starts loading the next track. This doesn’t cause drop out if left to play. Most likely because the buffer in the group members is full enough it never empties during the burst to load the next track.

The group coordinator download speed appears to being limited by the device, because even changing from wifi to ethernet → sonosnet, the download speed into the group coordinator didn’t increase, but a larger number of group members could play.

Hi @sigh, great analysis as always. I was typing a reply to say “no way are any of your Sonos devices using 802.11a” when I read this …

https://support.sonos.com/en/article/supported-wifi-modes-and-security-standards-for-sonos-products

… and I stand corrected! Also surprising to see how few Sonos devices support 802.11ac (aka Wi-Fi 5) which was broadly rolled out in routers a decade ago.

I think you may be onto a critical underlying challenge: the much chattier APIs in the new app / new device firmware present tremendous bandwidth issues for the 802.11n protocol used by the majority of contemporary Sonos device.

Modern mesh Wi-Fi is fortunately backward compatible with the older legacy protocols, but that is at best a ‘patch’ for the bandwidth issues inherent to 802.11n.

br0       Link encap:Ethernet  HWaddr 38:42:0B:89:36:90            inet addr:192.168.11.190  Bcast:192.168.11.255  Mask:255.255.255.0          inet6 addr: fe80::3a42:bff:fe89:3690/64 Scope:Link          inet6 addr: fddb:cd92:62a7:d643:3a42:bff:fe89:3690/64 Scope:Global          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1          RX packets:11186645 errors:0 dropped:2210960 overruns:0 frame:0          TX packets:2964160 errors:0 dropped:0 overruns:0 carrier:0          collisions:0 txqueuelen:1000           RX bytes:331790708 (316.4 MiB)  TX bytes:1596358387 (1.4 GiB)

Unfortunately that is just the totals, so without being able to get a more detailed breakdown of the network stats or capture what the bridge is seeing using additional tools it is hard to say. 

There can be various reasons including but not limited to

• software bridges have input/output buffers, so the input buffer could be near full and receiving bursts of traffic larger than the free space
• there could be unknown packet or protocols being received, some smart home devices send out regular bursts of traffic with a custom protocol type for example.
• if there are firewall rules on the bridge they could drop traffic

Identifying things like is it a regular repeating increase every X minutes, is it only when playing music, is it all the time may help point in directions, but without being able to see the details of what/why it’s dropping it’s going to be a tricky one to pinpoint.

As for why the play 3 is the odd one out, without a better idea of what is being dropped it’s difficult to know whether it is traffic the 3 would even receive. Maybe it’s filtering it out before the reaching the 3, maybe the 3 is the source 😃