Which speaker model? Via your wifi, or by Bluetooth, Airplay? You’ve not given much info for us to go on…

Most of that issue will go away if you change your Wi-Fi SSID and password,  so your neighbor doesn’t have access to it, but as @nik9669a asks, it potentially isn’t all. This is why type of speaker and how your neighbor is connecting is important to us in order to provide a more robust answer. 

Kicking them off your WiFi will do it, unless you let them use their Spotify account on your system once. You have to play something other than Spotify on each speaker to break that Direct connection.