Since the issue is only on the ssh server side, it does not even matter if it is installed, because it would not normally be running. Ssh’ing to containers does not seem like a common practice at all.

AllowedIPs is a client side restriction that can be ignored anyway, no real security benefit from not just putting your entire network there.

You can just specify AllowedIPs in the wireguard client config to route only traffic to your services through the VPN, works great.