Sonos making thousands of SMB3 connections to Synology

  • 27 January 2022
  • 66 replies
  • 2525 views

Userlevel 1
Badge +1

Not sure if anyone else is experiencing issues with Sonos Music Library and the Synology NAS becoming unresponsive or very sluggish, or with random speakers disappearing from the S2 app.

 

I recently enabled the Music Library feature in Sonos (before the 13.4.1 update in Dec 2021 which introduced SMB3 support). At that time, I configured my Synology NAS to use SMB1 to allow the connection to work. I was on DSM Version: 7.0.1-42218 and running S2 13.3.2. Somewhere around 11/25/21, my Synology started going offline and become unresponsive. I was running a DS1517+ with 2GB of RAM and noticed that the SMB process was eating a lot more RAM than usual. Also, started seeing thousands of connections from the Sonos speakers, multiple times for each speaker. My Sonos updated to 13.4, but the issue still remained.

I opened a case with Synology and they determined that the Sonos speakers were opening hundreds of connections per speaker and leaving the connections open. This was slowly draining the memory until the NAS would become unresponsive.

On 12/7/21, S2 updated to 13.4.1 which introduced the SMB3 support. I turned off SMB1 on the Synology NAS and still saw the hundreds and thousands of connections. I would also see the speakers attempt an SMB1 connection still which was denied by the speakers. For testing, we tried leaving SMB1 on, but still saw the large amount of connections from the Sonos speakers.

I ended up switching to a newer DS1621+ with 8GB of RAM and the issue still continued to happen, but took several days instead of just one. At one point, the number of active connections from Sonos was near 4000.

 

After switching to the DS1621+ with more memory, the issues started getting worse on the Sonos side. Several speakers would start to show up as ZP100 (would be various speakers and not always the same ones). This marked these speakers as incompatible with the S2 app and so I couldn’t do anything relating to the music library (like remove it) until the affected speakers were rebooted. If you tried going to http://<ip of speaker>:1400/xml/device_information.xml nothing would show up, or just the XML banner stating “This XML file does not appear to have any style information associated with it. The document tree is shown below.” The speakers were acting similar to the old NAS where they would start to become unresponsive. I can only guess that with the hundreds of open connections to the NAS, the memory utilization would get very high and the speaker would stop responding correctly.

 

After rebooting the speaker that showed up as a ZP100 you could then hit the XML page and the speaker showed up correctly as a Sonos One. I would also get some speakers that would just completely drop from the application, but after a reboot, they’d show up just fine.. In all cases, my wireless controller showed all speakers connected on the network and they all responded to pings. After a reboot of all the speakers, they all showed up correctly in the S2 app and could all play music from the Synology. Also, after a reboot of the speaker, all the connections would slowly drop on the NAS and memory usage would return to normal.
 

The whole system would run fine for a couple of days and then the connections would start appearing on the NAS at various times (not just at the 2am re-indexing time). I also noticed that whenever I had the S2 app up and when I would explore a Sonos playlist that contained music from the NAS, I would get SMB1 failures in the NAS logs for the associated speaker. I believe that the code is still set to use SMB1 for some functions and is getting blocked by the NAS..

 

If I disable the Music Library, all the speakers stay connected to the wifi network and I can stream music with no issues for several days. The memory on the NAS remains at normal levels and doesn’t climb over time. I still see SMB1 connections whenever I pull up the S2 app as long as the Sonos playlists exist that point to music on the NAS, even if the Muisc Library is disabled. I still think the Sonos code is trying to use SMB1 to update something either with album art or the track icons and is having trouble. If I browse to the album directly under Music Library, the album art does show correctly. It just seems to be the track icons in the playlist has trouble populating. These playlists were created by selecting various tracks in the Music Library and adding them to the Sonos playlist.

 

I have tickets open with both Synology and Sonos and currently waiting to hear back from Sonos after giving them countless diagnostics and logs. It seems that the diagnostic logs don’t show the connections to the NAS and so support doesn’t have any visibility into the problem other than looking at the thousands of connections in the Synology logs that I’m providing them.


This topic has been closed for further comments. You can use the search bar to find a similar topic, or create a new one by clicking Create Topic at the top of the page.

66 replies

Userlevel 1
Badge +1

In the [global] section ,

deadtime=4320

max smbd processes = 500

 

and in the section for the specific share,

max connections = 400

 

On my old NAS, even though the max connections is technically per share, my NAS autogenerates the smb.conf from another config, so I had to make all my max connections 400.

 

Also in [global] you could try adding:

socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=524288 SO_SNDBUF=524288

 

But I don’t think it makes a difference and it might cause other issues, so best to leave it alone unless you want to experiment.

 

 

Userlevel 1
Badge +1

I’d prefer to use your settings billfor since doing the single connection breaks things.

What are the settings in your smb.conf file that you’re using? I’ll see if there’s a way to modify this on the Synology so that it persists across a reboot. Typically, you aren’t supposed to modify config files from the CLI, but there’s no GUI settings to update the deadtime and max connections.

Userlevel 1
Badge +1

Yeah this was the original problem whereby the max connections was too low, and then you wind up skipping songs if you are just doing a shuffle.  But the bigger issue when you set max connections to a low value is that the music library scan fails, so I was having a problem updating it until I set the maxconnections to a higher number like 500.  A deadtime of 3 days and maxcon 500 seems to avoid problems at least for me.

 

Userlevel 1
Badge +1

For my particular use case, I have various songs on my NAS that I then create sonos playlists and then these are triggered by home automation. Rather than connect to the root music share that has many thousands of songs, I now have a new folder for sonos that I put the songs I need to play on the speakers and nothing else. Connecting to this sonos only share seems to have helped since there are now just a dozen songs or less at any one time.

If you’re wanting to have access to your entire library and you can’t modify the smb.conf file to close stale connections, you can go into the Synology Control Panel, File Services, SMB, Advanced Settings, Others, and then check the box next to “Disable multiple connections from the same IP address”. This keeps the connections from getting out of control, but does introduce an issue where the first song of the playlist won’t play and you have to hit play a second time before the music will start playing.

Userlevel 7
Badge +22

I really should update that Pi but since it just sits in the rack quietly running I hate to disturb it. I swapped power bricks a couple months ago so this is a low number.

pi@pi-3b-2:~ $ uptime
 17:05:56 up 65 days,  4:33,  2 users,  load average: 0.23, 0.40, 0.35

pi@pi-3b-2:~ $ lsb_release -a
No LSB modules are available.
Distributor ID:    Raspbian
Description:    Raspbian GNU/Linux 10 (buster)
Release:    10
Codename:    buster
 

Userlevel 1
Badge +1

4.13.13 ships with bullseye

root@rpi4:~# lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye
root@rpi4:~# samba -V
Version 4.13.13-Debian

 

I had downloaded the source and built the latest 4.16 a few months ago, and had the same problem. Went back to older containerized versions and same problem. Given that my NAS is a 10 year old lenovo ix4 that has never had any problems with leaving open files with any other application, including the 13.x version of the sonos fw, I’m inclined to believe that this is a sonos problem. My library is 20,000 songs though, so it may be quantity. Not ruling out there is some global config that could be set to fix it though.

 

 

Userlevel 7
Badge +22

Samba 4.9.5-Debian that ships with the  5.10.103-v7+ Raspberrian I have running and whatever OpenSuse 15.3 used, haven’t tried it on 15.4.

Userlevel 1
Badge +1

Which version of samba are you using and can you send the global config section?

Userlevel 7
Badge +22

That is sounding more and more like a NAS issue. 

While playing with the Pi as a gateway I connected it to several different data sources from a Windows PC to Linux systems and with different protocols with no issues.

Userlevel 1
Badge +1

I’ve run an SMB gateway my pi to the NAS but it didn’t work. I tried various versions of samba.

Userlevel 7
Badge +22

Sonos seems to prioritize problems by the number of folks complaining to them so unless a bunch of folks with your hardware/problem complain it isn’t likely to top the list.

It would be an interesting experiment to see how your setup would respond to using an SMB gateway between the NAS and your Sonos. I’d say use a Raspberry Pi but they are about impossible to find and priced way too high due to the chip shortage. Maybe another SMB capable system instead? Same basic setup.

SMB v1 Gateway  https://stan-miller.livejournal.com/357.html

Userlevel 1
Badge +1

If you set the “deadtime” parameter in smb.conf to close out the orphaned connections after 3 days,  the speakers will not need to reboot. I’ve never rebooted my speakers. Average use for 5 connected sonos clients is about 200 connections for a rolling 3 day average.  Amazed Sonos doesn’t seem to do anything about the issue though.

Userlevel 1
Badge +1

I’m also still seeing the issue on S2 14.14 with SMB on Syno 4.10.18-0434.

 

At some point, I have to reboot all my speakers. There is a workaround on Synology where you can block multiple connection attempts by the same IP address under File Services->SMB->Advanced Settings->Others “Disable multiple connections from the same IP address”. This keeps the syno from running out of memory and helps to keep the speakers from running out of memory. I’ve noticed when enabling this setting that sometimes you get an error when first playing a song from the Syno, but trying it again works. With this setting, the speakers stay online for much longer before they start having problems.

Userlevel 1
Badge +1

Still seeing the same issue on my system. 362 open connections on my DS214 from three different speakers not playing anything.

S2-System is on 14.14.

SMB-Service on Syno is 4.10.18-0327

Please help...

 

I think the easiest workaround at the moment is to set deadtime on the smb.conf to 4320 which is 3 days, and also make sure that the “max connections” setting for your music shares is >= 400. Not sure how to do that on a Synology but those are pretty common adjustments.

 

If you want to live life dangerously you could also try this in the global, but I don’t think it will help much.

socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=524288 SO_SNDBUF=524288

 

 

 

Userlevel 1
Badge +1

I called them months ago and they asked me to upload logs which I did. Never heard back from them. I think OP also said he called and gave logs. Considering the issue has been open for about a year now, I disagree with your premise.

Userlevel 7
Badge +22

You really need to call and talk with Sonos, they are most likely to find your problem.

Userlevel 2

Still seeing the same issue on my system. 362 open connections on my DS214 from three different speakers not playing anything.

S2-System is on 14.14.

SMB-Service on Syno is 4.10.18-0327

Please help...

Userlevel 1
Badge +1

Sad to report that my connections grew to over 2700 and also the SMB1 connection issue is still there. I have 98 new failed attempts to the NAS for SMB1 connections which are denied. At first, it seemed like they were fixed since I didn’t have any since 2022-05-07, but then starting on 2022-06-15 through 2022-06-16, they came back. i don’t recall exactly what triggered the connections, but the speaker that made them was recently factory reset and re-added because it dropped off the network. The reason it dropped may be related to my home automation trying to play a non-existing Pandora radio station. I only mention it here, because there may be something in the onboarding process that is causing the SMB1 connections.

Userlevel 1
Badge +1

I would have much more than 218 connections I just set the deadtime global in samba to kill anything older than 3 days.

Userlevel 1
Badge +1

I spoke too soon. I was playing some songs from the NAS and now I’m up to 272 connections.

Userlevel 1
Badge +1

No I still have 218 connections to my Music share from 5 Sonos devices running 14.8.

 

Userlevel 1
Badge +1

I have installed 14.8 and so far, things seem to be better. I’m not seeing the connections staying open like before and the SMB1 connections stopped over a month ago. How does it look for everyone else? The SMB service on my Synology is still running 4.10.18-0434.

Userlevel 7
Badge +18

Hi @billfor 

Due to spammers now posting a lot of phone numbers, our filters now occasionally mistake genuine posts with strings of numbers as spam. I’ve corrected this for your post above.

Userlevel 1
Badge +1

My comment about 14.15.5 appears to have been blocked for moderation for some apparent reason. TLDR; better but still an issue.

Try this in smb.conf

 socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=524288 SO_SNDBUF=524288

 

Userlevel 1
Badge +1

4.15.5 seems to do better w/r to connections but I need to let it run longer. Though they still grow It seems slower and has a slightly different behavior insofar as the older connections are actually holding open older song fiiles as opposed to being completely orphaned. 

 

I am using servercontainers/samba from dockerhub. One other setting that is hardcoded/defaulted in their config is the following:

 

 socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=524288 SO_SNDBUF=524288

 

That may or may not help the situation (I have not had time to test it).

 

Not sure what smb config Sonos uses to do their QA but would be good to know. Sonos should buy a raspberry pi and just use the current vanilla version of smbd off debian bullseye to do their testing. For most NAS users, changing the version to something as current as 4.15.5 may not be supportable, and certainly setting socket_options is not recommended and may not be supported.

 

Only one of the connections below is actually active w/r to the song playing...

 

root@rpi4:~# docker exec -it samba smbstatus

Samba version 4.15.5
PID     Username     Group        Machine                                   Protocol Version  Encryption           Signing
----------------------------------------------------------------------------------------------------------------------------------------
20148   nobody       nobody       192.168.1.113 (ipv4:192.168.1.113:41080)  SMB3_02           -                    -
20120   nobody       nobody       192.168.1.113 (ipv4:192.168.1.113:41032)  SMB3_02           -                    -
26510   nobody       nobody       192.168.1.113 (ipv4:192.168.1.113:51914)  SMB3_02           -                    -
31288   nobody       nobody       192.168.1.113 (ipv4:192.168.1.113:55252)  SMB3_02           -                    -
12796   nobody       nobody       192.168.1.113 (ipv4:192.168.1.113:42204)  SMB3_02           -                    -
31315   nobody       nobody       192.168.1.113 (ipv4:192.168.1.113:55272)  SMB3_02           -                    -
12794   nobody       nobody       192.168.1.113 (ipv4:192.168.1.113:42202)  SMB3_02           -                    -
12694   nobody       nobody       192.168.1.113 (ipv4:192.168.1.113:42102)  SMB3_02           -                    -
20146   nobody       nobody       192.168.1.113 (ipv4:192.168.1.113:41078)  SMB3_02           -                    -

Service      pid     Machine       Connected at                     Encryption   Signing
---------------------------------------------------------------------------------------------
Music        12794   192.168.1.113 Thu May 12 11:29:53 2022 UTC     -            -
Music        20120   192.168.1.113 Fri May 13 14:09:58 2022 UTC     -            -
Music        12796   192.168.1.113 Thu May 12 11:29:53 2022 UTC     -            -
Music        12694   192.168.1.113 Thu May 12 11:25:48 2022 UTC     -            -
Music        20146   192.168.1.113 Fri May 13 14:11:21 2022 UTC     -            -
Music        31288   192.168.1.113 Thu May 12 23:45:23 2022 UTC     -            -
Music        31315   192.168.1.113 Thu May 12 23:46:06 2022 UTC     -            -
Music        20148   192.168.1.113 Fri May 13 14:11:22 2022 UTC     -            -
Music        26510   192.168.1.113 Thu May 12 20:36:50 2022 UTC     -            -

Locked files:
Pid          User(ID)   DenyMode   Access      R/W        Oplock           SharePath   Name   Time
--------------------------------------------------------------------------------------------------
31315        65534      DENY_NONE  0x89        RDONLY     NONE             /Music   ripped/King’s X - Ear Candy - 07 - 67.flac   Fri May 13 00:29:41 2022
26510        65534      DENY_NONE  0x89        RDONLY     NONE             /Music   Indie/VA - BP TOP 100 of June 2012 - 26 - Knife Party - Centipede (Original Mix).mp3   Thu May 12 22:14:16 2022
12796        65534      DENY_NONE  0x89        RDONLY     NONE             /Music   80s/James - Say Something - Laid.mp3   Thu May 12 20:35:58 2022
31288        65534      DENY_NONE  0x89        RDONLY     NONE             /Music   Indie/Emiliana Torrini - Rarities - 01 - Weird Friendless Kid.mp3   Fri May 13 00:29:27 2022
20120        65534      DENY_NONE  0x89        RDONLY     NONE             /Music   80s/Kinks, The - Better Ship - Ultimate Collection (1 of 2).mp3   Fri May 13 14:39:11 2022
12794        65534      DENY_NONE  0x89        RDONLY     NONE             /Music   Indie/Bloc Party - Weekend in the City - 08 - Kreuzberg.mp3   Thu May 12 23:40:22 2022