Sonos making thousands of SMB3 connections to Synology

  • 27 January 2022
  • 49 replies
  • 1338 views

Badge +1

Not sure if anyone else is experiencing issues with Sonos Music Library and the Synology NAS becoming unresponsive or very sluggish, or with random speakers disappearing from the S2 app.

 

I recently enabled the Music Library feature in Sonos (before the 13.4.1 update in Dec 2021 which introduced SMB3 support). At that time, I configured my Synology NAS to use SMB1 to allow the connection to work. I was on DSM Version: 7.0.1-42218 and running S2 13.3.2. Somewhere around 11/25/21, my Synology started going offline and become unresponsive. I was running a DS1517+ with 2GB of RAM and noticed that the SMB process was eating a lot more RAM than usual. Also, started seeing thousands of connections from the Sonos speakers, multiple times for each speaker. My Sonos updated to 13.4, but the issue still remained.

I opened a case with Synology and they determined that the Sonos speakers were opening hundreds of connections per speaker and leaving the connections open. This was slowly draining the memory until the NAS would become unresponsive.

On 12/7/21, S2 updated to 13.4.1 which introduced the SMB3 support. I turned off SMB1 on the Synology NAS and still saw the hundreds and thousands of connections. I would also see the speakers attempt an SMB1 connection still which was denied by the speakers. For testing, we tried leaving SMB1 on, but still saw the large amount of connections from the Sonos speakers.

I ended up switching to a newer DS1621+ with 8GB of RAM and the issue still continued to happen, but took several days instead of just one. At one point, the number of active connections from Sonos was near 4000.

 

After switching to the DS1621+ with more memory, the issues started getting worse on the Sonos side. Several speakers would start to show up as ZP100 (would be various speakers and not always the same ones). This marked these speakers as incompatible with the S2 app and so I couldn’t do anything relating to the music library (like remove it) until the affected speakers were rebooted. If you tried going to http://<ip of speaker>:1400/xml/device_information.xml nothing would show up, or just the XML banner stating “This XML file does not appear to have any style information associated with it. The document tree is shown below.” The speakers were acting similar to the old NAS where they would start to become unresponsive. I can only guess that with the hundreds of open connections to the NAS, the memory utilization would get very high and the speaker would stop responding correctly.

 

After rebooting the speaker that showed up as a ZP100 you could then hit the XML page and the speaker showed up correctly as a Sonos One. I would also get some speakers that would just completely drop from the application, but after a reboot, they’d show up just fine.. In all cases, my wireless controller showed all speakers connected on the network and they all responded to pings. After a reboot of all the speakers, they all showed up correctly in the S2 app and could all play music from the Synology. Also, after a reboot of the speaker, all the connections would slowly drop on the NAS and memory usage would return to normal.
 

The whole system would run fine for a couple of days and then the connections would start appearing on the NAS at various times (not just at the 2am re-indexing time). I also noticed that whenever I had the S2 app up and when I would explore a Sonos playlist that contained music from the NAS, I would get SMB1 failures in the NAS logs for the associated speaker. I believe that the code is still set to use SMB1 for some functions and is getting blocked by the NAS..

 

If I disable the Music Library, all the speakers stay connected to the wifi network and I can stream music with no issues for several days. The memory on the NAS remains at normal levels and doesn’t climb over time. I still see SMB1 connections whenever I pull up the S2 app as long as the Sonos playlists exist that point to music on the NAS, even if the Muisc Library is disabled. I still think the Sonos code is trying to use SMB1 to update something either with album art or the track icons and is having trouble. If I browse to the album directly under Music Library, the album art does show correctly. It just seems to be the track icons in the playlist has trouble populating. These playlists were created by selecting various tracks in the Music Library and adding them to the Sonos playlist.

 

I have tickets open with both Synology and Sonos and currently waiting to hear back from Sonos after giving them countless diagnostics and logs. It seems that the diagnostic logs don’t show the connections to the NAS and so support doesn’t have any visibility into the problem other than looking at the thousands of connections in the Synology logs that I’m providing them.

James L. 4 months ago

Hi folks,

 

I’m happy to say that this issue has been resolved with the latest S2 update (14.4). Sonos players should no longer be sending excessive requests to your NAS drives when using the music library when updated to this version.

Please let our customer care team know if you’re still experiencing this issue after updating all players and controllers to 14.4. 🙂

View original

49 replies

Badge +1

The previous connections went away on their own and now another 361 connections have been made by the same speaker as before. This speaker is not currently the associated product and is a speaker that has a playlist queued up from Spotify so not sure why the speaker is still making persistent connections to the NAS.

I’ve reported this to support as well.

Userlevel 1
Badge

I had some time so I updated everything to 14.6 (5 devices) and still have the problem. 403 connections to the NAS from the speakers.

 

root@ix4-300d:/# smbstatus | grep Music | wc -l
403

 

I also wired everything in case the connection were getting deauths and that was causing it, but the same problem happens even when the speakers are physically wired, so it has nothing to do with that. I will call support.

 

Badge +1

I’ve updated to 14.6 and still have the same problem, although it appears it isn’t quite as bad as before. I have 427 connections, but it doesn’t appear to be growing until there is no more memory on the NAS. Before, the connections would just continue to grow by the hundreds until there was several thousands of them. Now it appears to be under 500 connections.

I’ve let support know and have been keeping them updated. They said they are looking into it, but haven’t heard anything back from them.

Userlevel 1
Badge

Hi folks,

 

I’m happy to say that this issue has been resolved with the latest S2 update (14.4). Sonos players should no longer be sending excessive requests to your NAS drives when using the music library when updated to this version.

Please let our customer care team know if you’re still experiencing this issue after updating all players and controllers to 14.4. 🙂

 

James since it is not resolved I just gave a courtesy call to support and uploaded my logs which are 14.6 and the problem appears hardwired and wifi so as to rule that out.

 

Support said:

“Basically what  our support doing is they will going to look at the forum and check a work around to fix the issue. You will be updated on the link that you've provided to us in the community.”

 

 

So possibly if there is anything to say they will reply back here.  I guess my next step would be to try out alternate SMB servers to see maybe there is some bug in a specific SAMBA version. I’m a little perplexed that more people haven’t complained en mass, but maybe nobody uses direct file share anymore or people don’t update beyond 13.x. It is very mysterious to me why only two people seem to reliably have the issue.

 

If it is of any help, updating the sonos firmware clears all the connection, as does switching from wifi to hardwire, or from hardwire to wifi. You can actually see all the connections cleaned up, but then the problem slowly grows again until the sonos is again switched or reset.  The Sonos seems to be holding those connections is some sort of connection queue and forgetting about them.

 

Userlevel 1
Badge

Update: I mirrored my NAS to a rasperry pi via NFS, and then served that to the sonos via smb, in effect creating a 2nd smb server, and the problem still happens very quickly and is easily reproducible. So 99% sure it is a problem with Sonos.

 

Userlevel 3
Badge +2

Interesting, I have two Linux servers running serving music to Sonos via SMB and I have no issues at all. What version of the Samba are you using?

Userlevel 1
Badge

 

root@ix4-300d:/usr/local/samba/bin# /usr/local/samba/sbin/smbd --version
Version 4.9.3
root@ix4-300d:/usr/local/samba/bin# smbstatus | grep Music | wc -l
400
^^ no kidding

root@rpi4:~# smbd --version
Version 4.13.13-Debian

root@rpi4:~# smbstatus | grep Music | wc -l
50

It should use and maintain 5 connections only.

 

I’ve talked to friends and they have the same issue. You need to run 14.x not 13.x. S2 only.

Pretty easy to reproduce. Play from a share with a few thousand songs on it, scroll through album art in the queue to force connections, etc…

It’s most likely a connection pool issue.

 

 

Userlevel 3
Badge +2

Weird and interesting… as said, I have no issues here. I’m running Samba 4.9.5 and Samba 4.15.5 on two different machines and can’t reproduce this issue.

What exactly do you do? You mention a share with a few thousand tracks, do you have them all in the queue? I added approx. 1000 tracks to the queue, played it and scrolled through and had no issues. Also browsing through the library doesn’t create any excess connections that are not closed. 

Userlevel 1
Badge

I load the entire NAS into the queue and just shuffle them. Music share is just the root of a bunch of folders with tracks.  So that’s about 19,000.

I’ve tried splitting the rooms, vs combining them and it has no effect on this problem.

I should point out that there was never a problem until 14.0 came out, and as I understand that changed the NAS protocol to SMB3. So that seems to suggest something related in that new feature.

 

It’s not a wifi issue because the problem occurs when everything is wired. How many connections do you have to your NAS that are reused properly?

 

 

 

 

 

 

Userlevel 1
Badge

Just for fun I tried loading a subfolder into the queue so it was only around 200 songs and let it play and then messed around the S2 app a bit to force it to load album art, etc.., and was easily able to reproduce the problem. 9 new connections in a minute.

root@ix4-300d:/# date
Fri May 6 17:00:59 EDT 2022
root@ix4-300d:/# smbstatus | grep Music | wc -l
17

root@ix4-300d:/# date
Fri May 6 17:01:58 EDT 2022
root@ix4-300d:/# smbstatus | grep Music | wc -l
26

You can see all the new ones started at 5:01 just as I started using the S2 app to force the album art load.

root@ix4-300d:/# smbstatus | grep Music
Music 15693 192.168.1.168 Fri May 6 04:54:23 PM 2022 EDT - -
Music 15484 192.168.1.113 Fri May 6 04:53:21 PM 2022 EDT - -
Music 15676 192.168.1.168 Fri May 6 04:54:20 PM 2022 EDT - -
Music 15712 192.168.1.168 Fri May 6 04:54:25 PM 2022 EDT - -
Music 15710 192.168.1.168 Fri May 6 04:54:25 PM 2022 EDT - -
Music 16445 192.168.1.168 Fri May 6 05:01:36 PM 2022 EDT - -
Music 16433 192.168.1.168 Fri May 6 05:01:33 PM 2022 EDT - -
Music 16456 192.168.1.168 Fri May 6 05:01:40 PM 2022 EDT - -
Music 15465 192.168.1.168 Fri May 6 04:53:19 PM 2022 EDT - -
Music 16425 192.168.1.168 Fri May 6 05:01:32 PM 2022 EDT - -
Music 16435 192.168.1.168 Fri May 6 05:01:34 PM 2022 EDT - -
Music 16429 192.168.1.168 Fri May 6 05:01:32 PM 2022 EDT - -
Music 15704 192.168.1.168 Fri May 6 04:54:24 PM 2022 EDT - -
Music 16421 192.168.1.168 Fri May 6 05:01:30 PM 2022 EDT - -
Music 15674 192.168.1.168 Fri May 6 04:54:20 PM 2022 EDT - -
Music 16443 192.168.1.168 Fri May 6 05:01:36 PM 2022 EDT - -
Music 15488 192.168.1.113 Fri May 6 04:53:22 PM 2022 EDT - -
Music 16458 192.168.1.168 Fri May 6 05:01:41 PM 2022 EDT - -
Music 16185 192.168.1.139 Fri May 6 04:59:18 PM 2022 EDT - -
Music 15686 192.168.1.168 Fri May 6 04:54:21 PM 2022 EDT - -
Music 15708 192.168.1.168 Fri May 6 04:54:24 PM 2022 EDT - -
Music 16439 192.168.1.168 Fri May 6 05:01:35 PM 2022 EDT - -
Music 15477 192.168.1.139 Fri May 6 04:53:21 PM 2022 EDT - -
Music 15690 192.168.1.168 Fri May 6 04:54:22 PM 2022 EDT - -
Music 15483 192.168.1.139 Fri May 6 04:53:21 PM 2022 EDT - -
15477 99 DENY_NONE 0x89 RDONLY NONE /mnt/pools/A/A0/Music Pink Floyd/04 - A Saucerful Of Secrets.flac Fri May 6 17:05:38 2022

 

Userlevel 7
Badge +21

Debian Arm Linux (Raspberry PI) loading a 1034 song playlist and scrolling to the end using the noson app.

pi@pi-3b-2:~ $ sudo smbstatus

Samba version 4.9.5-Debian
PID     Username     Group        Machine                                   Protocol Version  Encryption           Signing              
----------------------------------------------------------------------------------------------------------------------------------------
15839   nobody       nogroup      172.16.1.114 (ipv4:172.16.1.114:40048)    SMB3_02           -                    -                    
15816   nobody       nogroup      172.16.1.115 (ipv4:172.16.1.115:42990)    SMB3_02           -                    -                    
15814   nobody       nogroup      172.16.1.115 (ipv4:172.16.1.115:42988)    SMB3_02           -                    -                    
15841   nobody       nogroup      172.16.1.114 (ipv4:172.16.1.114:40050)    SMB3_02           -                    -                    

Service      pid     Machine       Connected at                     Encryption   Signing     
---------------------------------------------------------------------------------------------
music        15841   172.16.1.114  Fri May  6 02:59:14 PM 2022 MST  -            -           
music        15839   172.16.1.114  Fri May  6 02:59:14 PM 2022 MST  -            -           
music        15816   172.16.1.115  Fri May  6 02:59:04 PM 2022 MST  -            -           
music        15814   172.16.1.115  Fri May  6 02:59:04 PM 2022 MST  -            -   

 

I was going to post the same using a Sonos controller but it looks like my playlists aren’t available using it, neither is my Recently Played list.

=====

Edit: loaded the queue manually from Sonos controller, scrolled to the end to populate the thumbnails. Aside from being far slower to populate than the noson program there was no difference in the smbstatus output.

 

Userlevel 1
Badge

Yes that is the way it is supposed to look. Your connection time is recent however (3pm). The people that have the issue report that it gets worse over time because the old connections are not closed or reused. It would therefore be more conclusive to show the results of an smbstatus on Sunday, still showing the connections in use from Friday. Or perhaps the way it is supposed to work is that the Sonos closes the old connections before starting the new ones in which case the 5 connections will always be recent - in our case we get new ones but the old ones stay orphaned, and they are definitely orphans because setting the smb deadtime parameter works to cleanup the connections. Unfortunately deadtime is a global and not a share property, so I can’t set it to some relatively low value otherwise I’ll time out other shares that I want to leave alone.

 

Userlevel 7
Badge +21

On my Pi SMB server the old connections seem to disappear over time. Nothing showing today.

I’m using Raspbian GNU/Linux 10 (buster) are you on a newer release?

I don’t get offered Samba 4.15.x on this system.

 

Userlevel 3
Badge +2

Ok, looks like I can somewhat repro this on my Debian 10 machine with Samba 4.9.5. It’s not as excessive as for billfor, but I have some open connections. My Samba 14.15.5 machine however doesn’t show this behaviour. It’s running in a Docker btw. and I use it to rip my audio CDs and Blurays. 

Badge +1

For reference, my Synology NAS is running the following SMB version:

Version 4.10.18
Synology Build 42184, Feb 17 2022 19:04:31

 

Like billfor mentioned, I didn’t notice this on 13.x, but on 14.x around the time the SMB3 fix came out. I had a library setup around Halloween last year to play music from the NAS, forgot that I had left the music library enabled and didn’t have a problem till Thanksgiving week or soon after when I had upgraded to 14.x. That’s when I noticed my old NAS kept running out of memory and required almost every other day reboots before I tracked the issue to the Sonos speakers keeping thousands of connections open.

 

14.6 seems to have helped slightly as I don’t see the connections growing out of control now.

Badge +1

Just checked and I’m at 1095 connections to the NAS. Before, the connections would just continue growing unchecked. I’ll let you know if the number goes back down by itself over time.

Userlevel 1
Badge

4.15.5 seems to do better w/r to connections but I need to let it run longer. Though they still grow It seems slower and has a slightly different behavior insofar as the older connections are actually holding open older song fiiles as opposed to being completely orphaned. 

 

I am using servercontainers/samba from dockerhub. One other setting that is hardcoded/defaulted in their config is the following:

 

 socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=524288 SO_SNDBUF=524288

 

That may or may not help the situation (I have not had time to test it).

 

Not sure what smb config Sonos uses to do their QA but would be good to know. Sonos should buy a raspberry pi and just use the current vanilla version of smbd off debian bullseye to do their testing. For most NAS users, changing the version to something as current as 4.15.5 may not be supportable, and certainly setting socket_options is not recommended and may not be supported.

 

Only one of the connections below is actually active w/r to the song playing...

 

root@rpi4:~# docker exec -it samba smbstatus

Samba version 4.15.5
PID     Username     Group        Machine                                   Protocol Version  Encryption           Signing
----------------------------------------------------------------------------------------------------------------------------------------
20148   nobody       nobody       192.168.1.113 (ipv4:192.168.1.113:41080)  SMB3_02           -                    -
20120   nobody       nobody       192.168.1.113 (ipv4:192.168.1.113:41032)  SMB3_02           -                    -
26510   nobody       nobody       192.168.1.113 (ipv4:192.168.1.113:51914)  SMB3_02           -                    -
31288   nobody       nobody       192.168.1.113 (ipv4:192.168.1.113:55252)  SMB3_02           -                    -
12796   nobody       nobody       192.168.1.113 (ipv4:192.168.1.113:42204)  SMB3_02           -                    -
31315   nobody       nobody       192.168.1.113 (ipv4:192.168.1.113:55272)  SMB3_02           -                    -
12794   nobody       nobody       192.168.1.113 (ipv4:192.168.1.113:42202)  SMB3_02           -                    -
12694   nobody       nobody       192.168.1.113 (ipv4:192.168.1.113:42102)  SMB3_02           -                    -
20146   nobody       nobody       192.168.1.113 (ipv4:192.168.1.113:41078)  SMB3_02           -                    -

Service      pid     Machine       Connected at                     Encryption   Signing
---------------------------------------------------------------------------------------------
Music        12794   192.168.1.113 Thu May 12 11:29:53 2022 UTC     -            -
Music        20120   192.168.1.113 Fri May 13 14:09:58 2022 UTC     -            -
Music        12796   192.168.1.113 Thu May 12 11:29:53 2022 UTC     -            -
Music        12694   192.168.1.113 Thu May 12 11:25:48 2022 UTC     -            -
Music        20146   192.168.1.113 Fri May 13 14:11:21 2022 UTC     -            -
Music        31288   192.168.1.113 Thu May 12 23:45:23 2022 UTC     -            -
Music        31315   192.168.1.113 Thu May 12 23:46:06 2022 UTC     -            -
Music        20148   192.168.1.113 Fri May 13 14:11:22 2022 UTC     -            -
Music        26510   192.168.1.113 Thu May 12 20:36:50 2022 UTC     -            -

Locked files:
Pid          User(ID)   DenyMode   Access      R/W        Oplock           SharePath   Name   Time
--------------------------------------------------------------------------------------------------
31315        65534      DENY_NONE  0x89        RDONLY     NONE             /Music   ripped/King’s X - Ear Candy - 07 - 67.flac   Fri May 13 00:29:41 2022
26510        65534      DENY_NONE  0x89        RDONLY     NONE             /Music   Indie/VA - BP TOP 100 of June 2012 - 26 - Knife Party - Centipede (Original Mix).mp3   Thu May 12 22:14:16 2022
12796        65534      DENY_NONE  0x89        RDONLY     NONE             /Music   80s/James - Say Something - Laid.mp3   Thu May 12 20:35:58 2022
31288        65534      DENY_NONE  0x89        RDONLY     NONE             /Music   Indie/Emiliana Torrini - Rarities - 01 - Weird Friendless Kid.mp3   Fri May 13 00:29:27 2022
20120        65534      DENY_NONE  0x89        RDONLY     NONE             /Music   80s/Kinks, The - Better Ship - Ultimate Collection (1 of 2).mp3   Fri May 13 14:39:11 2022
12794        65534      DENY_NONE  0x89        RDONLY     NONE             /Music   Indie/Bloc Party - Weekend in the City - 08 - Kreuzberg.mp3   Thu May 12 23:40:22 2022

 

Userlevel 1
Badge

My comment about 14.15.5 appears to have been blocked for moderation for some apparent reason. TLDR; better but still an issue.

Try this in smb.conf

 socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=524288 SO_SNDBUF=524288

 

Userlevel 7
Badge +18

Hi @billfor 

Due to spammers now posting a lot of phone numbers, our filters now occasionally mistake genuine posts with strings of numbers as spam. I’ve corrected this for your post above.

Badge +1

I have installed 14.8 and so far, things seem to be better. I’m not seeing the connections staying open like before and the SMB1 connections stopped over a month ago. How does it look for everyone else? The SMB service on my Synology is still running 4.10.18-0434.

Userlevel 1
Badge

No I still have 218 connections to my Music share from 5 Sonos devices running 14.8.

 

Badge +1

I spoke too soon. I was playing some songs from the NAS and now I’m up to 272 connections.

Userlevel 1
Badge

I would have much more than 218 connections I just set the deadtime global in samba to kill anything older than 3 days.

Badge +1

Sad to report that my connections grew to over 2700 and also the SMB1 connection issue is still there. I have 98 new failed attempts to the NAS for SMB1 connections which are denied. At first, it seemed like they were fixed since I didn’t have any since 2022-05-07, but then starting on 2022-06-15 through 2022-06-16, they came back. i don’t recall exactly what triggered the connections, but the speaker that made them was recently factory reset and re-added because it dropped off the network. The reason it dropped may be related to my home automation trying to play a non-existing Pandora radio station. I only mention it here, because there may be something in the onboarding process that is causing the SMB1 connections.

Reply