Sonos making thousands of SMB3 connections to Synology

  • 27 January 2022
  • 66 replies
  • 2537 views


Show first post
This topic has been closed for further comments. You can use the search bar to find a similar topic, or create a new one by clicking Create Topic at the top of the page.

66 replies

Userlevel 1
Badge +1

Just for fun I tried loading a subfolder into the queue so it was only around 200 songs and let it play and then messed around the S2 app a bit to force it to load album art, etc.., and was easily able to reproduce the problem. 9 new connections in a minute.

root@ix4-300d:/# date
Fri May 6 17:00:59 EDT 2022
root@ix4-300d:/# smbstatus | grep Music | wc -l
17

root@ix4-300d:/# date
Fri May 6 17:01:58 EDT 2022
root@ix4-300d:/# smbstatus | grep Music | wc -l
26

You can see all the new ones started at 5:01 just as I started using the S2 app to force the album art load.

root@ix4-300d:/# smbstatus | grep Music
Music 15693 192.168.1.168 Fri May 6 04:54:23 PM 2022 EDT - -
Music 15484 192.168.1.113 Fri May 6 04:53:21 PM 2022 EDT - -
Music 15676 192.168.1.168 Fri May 6 04:54:20 PM 2022 EDT - -
Music 15712 192.168.1.168 Fri May 6 04:54:25 PM 2022 EDT - -
Music 15710 192.168.1.168 Fri May 6 04:54:25 PM 2022 EDT - -
Music 16445 192.168.1.168 Fri May 6 05:01:36 PM 2022 EDT - -
Music 16433 192.168.1.168 Fri May 6 05:01:33 PM 2022 EDT - -
Music 16456 192.168.1.168 Fri May 6 05:01:40 PM 2022 EDT - -
Music 15465 192.168.1.168 Fri May 6 04:53:19 PM 2022 EDT - -
Music 16425 192.168.1.168 Fri May 6 05:01:32 PM 2022 EDT - -
Music 16435 192.168.1.168 Fri May 6 05:01:34 PM 2022 EDT - -
Music 16429 192.168.1.168 Fri May 6 05:01:32 PM 2022 EDT - -
Music 15704 192.168.1.168 Fri May 6 04:54:24 PM 2022 EDT - -
Music 16421 192.168.1.168 Fri May 6 05:01:30 PM 2022 EDT - -
Music 15674 192.168.1.168 Fri May 6 04:54:20 PM 2022 EDT - -
Music 16443 192.168.1.168 Fri May 6 05:01:36 PM 2022 EDT - -
Music 15488 192.168.1.113 Fri May 6 04:53:22 PM 2022 EDT - -
Music 16458 192.168.1.168 Fri May 6 05:01:41 PM 2022 EDT - -
Music 16185 192.168.1.139 Fri May 6 04:59:18 PM 2022 EDT - -
Music 15686 192.168.1.168 Fri May 6 04:54:21 PM 2022 EDT - -
Music 15708 192.168.1.168 Fri May 6 04:54:24 PM 2022 EDT - -
Music 16439 192.168.1.168 Fri May 6 05:01:35 PM 2022 EDT - -
Music 15477 192.168.1.139 Fri May 6 04:53:21 PM 2022 EDT - -
Music 15690 192.168.1.168 Fri May 6 04:54:22 PM 2022 EDT - -
Music 15483 192.168.1.139 Fri May 6 04:53:21 PM 2022 EDT - -
15477 99 DENY_NONE 0x89 RDONLY NONE /mnt/pools/A/A0/Music Pink Floyd/04 - A Saucerful Of Secrets.flac Fri May 6 17:05:38 2022

 

Userlevel 7
Badge +22

Debian Arm Linux (Raspberry PI) loading a 1034 song playlist and scrolling to the end using the noson app.

pi@pi-3b-2:~ $ sudo smbstatus

Samba version 4.9.5-Debian
PID     Username     Group        Machine                                   Protocol Version  Encryption           Signing              
----------------------------------------------------------------------------------------------------------------------------------------
15839   nobody       nogroup      172.16.1.114 (ipv4:172.16.1.114:40048)    SMB3_02           -                    -                    
15816   nobody       nogroup      172.16.1.115 (ipv4:172.16.1.115:42990)    SMB3_02           -                    -                    
15814   nobody       nogroup      172.16.1.115 (ipv4:172.16.1.115:42988)    SMB3_02           -                    -                    
15841   nobody       nogroup      172.16.1.114 (ipv4:172.16.1.114:40050)    SMB3_02           -                    -                    

Service      pid     Machine       Connected at                     Encryption   Signing     
---------------------------------------------------------------------------------------------
music        15841   172.16.1.114  Fri May  6 02:59:14 PM 2022 MST  -            -           
music        15839   172.16.1.114  Fri May  6 02:59:14 PM 2022 MST  -            -           
music        15816   172.16.1.115  Fri May  6 02:59:04 PM 2022 MST  -            -           
music        15814   172.16.1.115  Fri May  6 02:59:04 PM 2022 MST  -            -   

 

I was going to post the same using a Sonos controller but it looks like my playlists aren’t available using it, neither is my Recently Played list.

=====

Edit: loaded the queue manually from Sonos controller, scrolled to the end to populate the thumbnails. Aside from being far slower to populate than the noson program there was no difference in the smbstatus output.

 

Userlevel 1
Badge +1

Yes that is the way it is supposed to look. Your connection time is recent however (3pm). The people that have the issue report that it gets worse over time because the old connections are not closed or reused. It would therefore be more conclusive to show the results of an smbstatus on Sunday, still showing the connections in use from Friday. Or perhaps the way it is supposed to work is that the Sonos closes the old connections before starting the new ones in which case the 5 connections will always be recent - in our case we get new ones but the old ones stay orphaned, and they are definitely orphans because setting the smb deadtime parameter works to cleanup the connections. Unfortunately deadtime is a global and not a share property, so I can’t set it to some relatively low value otherwise I’ll time out other shares that I want to leave alone.

 

Userlevel 7
Badge +23

Have you tried a 3rd party app? Such as SonoPhone or mine for Windows?

Userlevel 7
Badge +22

On my Pi SMB server the old connections seem to disappear over time. Nothing showing today.

I’m using Raspbian GNU/Linux 10 (buster) are you on a newer release?

I don’t get offered Samba 4.15.x on this system.

 

Userlevel 1
Badge +1

I’m also still seeing the issue on S2 14.14 with SMB on Syno 4.10.18-0434.

 

At some point, I have to reboot all my speakers. There is a workaround on Synology where you can block multiple connection attempts by the same IP address under File Services->SMB->Advanced Settings->Others “Disable multiple connections from the same IP address”. This keeps the syno from running out of memory and helps to keep the speakers from running out of memory. I’ve noticed when enabling this setting that sometimes you get an error when first playing a song from the Syno, but trying it again works. With this setting, the speakers stay online for much longer before they start having problems.

Userlevel 1
Badge +1

Update 14.2 was released yesterday. I’ve re-enabled my music library to my NAS and will monitor for the next day or so to see if the connections continue.

Something else I’ve noticed, is that if you have any playlists created that contain music from your NAS, even if you don’t have the music library feature enabled, Sonos will still make SMB connections to the NAS. I’m assuming it’s to grab artwork for the songs in the playlist. I’ve also noticed that Sonos uses SMB1 connections first and then falls back to SMB3 which seems backwards to me. It should try the more secure method first and only fallback to SMB1 if it can’t connect via SMB3.

Userlevel 1
Badge +1

What I’ve done to help the problem is to set “deadtime” parameter in the [global] section of the NAS smb.conf to around 3 days. This will time out the orphaned connections -- I run with about 200 connections now since there are always going to be some dead ones  -- there should normally only be 5 or so connections I think.  You could drop deadtime lower but then if you have other devices or windows workstations using the NAS they may have to reopen connections.

 

The other parameter you could set is the max connections setting in the sections for the specific shares. I set mine to 500 which gives me enough room, considering the extent of this bug :-)

 

Some NASs may do this already, and others may have a UI to change these values, but my old NAS doesn’t so I need to edit the smb.conf

 

 

 

 

On Synology, there is a setting to allow only one SMB connection per IP address. This definitely prevents the issue from happening, however, when you go to play songs from the NAS, the initial song will fail to play, but then subsequent songs will play without issue. If development can’t figure out how to keep the speakers from making hundreds of calls to the NAS, I may just end up turning on the single connection option on the Synology and be done with it.

I also think the number of connections to the NAS may be related to the number of songs you have in your library. I’m thinking the more songs you have, the more connections that are made and so might affect people differently depending upon the number of songs in the library.

Userlevel 1
Badge +1

If I switch the Sonos to a dhcp reservation and put that in “hosts allow”, I still have to contend with some other non-Sonos devices connecting to the same shares. I don’t see anyway to limit the connection count per ip address, at least in the smb.conf. I could set max connections really low but don’t want to limit other well-behaved clients.

 

Userlevel 1
Badge +1

Update 14.2 was released yesterday. I’ve re-enabled my music library to my NAS and will monitor for the next day or so to see if the connections continue.

Something else I’ve noticed, is that if you have any playlists created that contain music from your NAS, even if you don’t have the music library feature enabled, Sonos will still make SMB connections to the NAS. I’m assuming it’s to grab artwork for the songs in the playlist. I’ve also noticed that Sonos uses SMB1 connections first and then falls back to SMB3 which seems backwards to me. It should try the more secure method first and only fallback to SMB1 if it can’t connect via SMB3.

A slight correction to the above, Sonos doesn’t consistently make SMB1 connections. I just tested browsing playlists after adding the music library back, and only see SMB3 connections. I’m not sure what actions are triggering the SMB1 connections to the NAS at this time. I can just see that there are failed attempts from the speakers because I have SMB1 disabled.

Userlevel 1
Badge +1

If I switch the Sonos to a dhcp reservation and put that in “hosts allow”, I still have to contend with some other non-Sonos devices connecting to the same shares. I don’t see anyway to limit the connection count per ip address, at least in the smb.conf. I could set max connections really low but don’t want to limit other well-behaved clients.

 

I have several SMB shares on my NAS as well and several Windows clients are connecting to them. My experience has been that each of these Windows machines only has one connection open to the NAS at a time. I’m assuming that “well behaved” clients only make a single connection to the SMB share and so limiting SMB to a single connection per IP shouldn’t break those clients. I agree with you though, we shouldn’t have to enable such strict measures just for the Sonos speakers.

Userlevel 1
Badge +1

Just as an FYI, at the direction of support, I removed all songs in the playlists and I no longer saw any connections from the speakers as expected.

I’ve installed version 14.4 and have re-enabled the Music Library. I’ll keep an eye on the connections to the NAS to see if the issue remains with the hundreds of connections and SMB1 attempted connections.

Userlevel 1
Badge +1

I had some time so I updated everything to 14.6 (5 devices) and still have the problem. 403 connections to the NAS from the speakers.

 

root@ix4-300d:/# smbstatus | grep Music | wc -l
403

 

I also wired everything in case the connection were getting deauths and that was causing it, but the same problem happens even when the speakers are physically wired, so it has nothing to do with that. I will call support.

 

Userlevel 1
Badge +1

BTW, I don’t believe I’ve noticed this behavior until sometime around the release of 13.4 back in November. I had the music library enabled since Halloween to play songs on the Sonos speakers and it was working fine until about Thanksgiving. That’s when I noticed my NAS starting to hang and go offline and I opened a ticket with Synology and that’s when we discovered the thousands of connections from the Sonos speakers.

My old NAS had 2GB of RAM and would lock up after a day or 2. I recently replaced that older NAS with one that has 8GB of RAM and it lasts a lot longer before becoming unresponsive. Most of the time, the Sonos speakers will start to crash first before the NAS does since the speakers don’t have enough RAM to maintain the high number of SMB connections to the NAS.

Userlevel 1
Badge +1

I’ve updated to 14.6 and still have the same problem, although it appears it isn’t quite as bad as before. I have 427 connections, but it doesn’t appear to be growing until there is no more memory on the NAS. Before, the connections would just continue to grow by the hundreds until there was several thousands of them. Now it appears to be under 500 connections.

I’ve let support know and have been keeping them updated. They said they are looking into it, but haven’t heard anything back from them.

Userlevel 4
Badge +4

Ok, looks like I can somewhat repro this on my Debian 10 machine with Samba 4.9.5. It’s not as excessive as for billfor, but I have some open connections. My Samba 14.15.5 machine however doesn’t show this behaviour. It’s running in a Docker btw. and I use it to rip my audio CDs and Blurays.