The TV shouldn’t really be a factor since the playbar uses an optical connection. Your remote will be directly communicating with the playbar. That said, I have both my Samsung TV remote and cable remote setup to control the playbar, and the cable remote works a lot better. I couldn’t really say the TV remote is delayed so much as it seems to raise the volume in much smaller increments...at least it feels that way.
So I guess I don’t really have an answer for you, other than I think I’ve seen the same issue. I work around it by using a different remote.