ESI PBX LAN-only VoIP phone forwarding thru IPSec tunnel – remote router sees packets but not phone, doesn’t forward

        Okay, so we have an older X-Class ESI PBX system, an ESI IVX 128x FSIII [see below for link]. We have an LNC card installed in it, which allows for the use of up to 12 <a href="http://refurbphoneexchange.com/products/esi-ip-48-key-h-ipfp-phone" rel="nofollow noreferrer">LAN-only VoIP phones</a>. We have an IPSec tunnel between the main site (with the PBX, using a pfSense router, build 2.3.2-RELEASE amd64) and the remote site (with the phone, using a Cisco RV220w router). I'm trying to get the phone and the PBX to talk, and right now they won't. Here's what I've discovered:
  • The phone works fine when plugged into the same fabric the PBX is connected to, boots up and grabs extension info. The PBX and extension settings are viewable here, but that may be a red herring for now.

  • The IPSec tunnel is set to allow all connection attempts between the subnets, and I've explicitly forwarded all ports in the range related to the PBX coming thru the tunnel to the private IP for the PBX (2.201). On the remote site's end, I had plugged the phone directly into a router port, and the router saw packets moving on the interface, but wouldn't assign an IP to the phone or acknowledge a device was connected (nothing in the clients list matching the phone's MAC).

  • I plugged the phone into my PC and ran a pcap with Wireshark. I'm not an expert with Wireshark, but it looks to me like the phone is sending broadcast/unicast packets to destination address Esi_ff:ff:ff (01:30:4d:ff:ff:ff) using the protocol 0x887f, which isn't an EtherType that google seems to have heard of. Packet length is 82 bits, frequency is every couple of seconds, same packet each time.

IPSec info:
Local router (pfSense) config:
Subnet: 192.168.2.0/23
firewall rules:

scrub on bge0 all fragment reassemble
scrub on re0 all fragment reassemble
scrub on re1 all fragment reassemble
anchor "relayd/*" all
anchor "openvpn/*" all
anchor "ipsec/*" all
block drop in log quick inet from 169.254.0.0/16 to any label "Block IPv4 link-local"
block drop in log quick inet from any to 169.254.0.0/16 label "Block IPv4 link-local"
block drop in log inet all label "Default deny rule IPv4"
block drop out log inet all label "Default deny rule IPv4"
block drop log quick inet proto tcp from any port = 0 to any label "Block traffic from port 0"
block drop log quick inet proto udp from any port = 0 to any label "Block traffic from port 0"
block drop log quick inet proto tcp from any to any port = 0 label "Block traffic to port 0"
block drop log quick inet proto udp from any to any port = 0 label "Block traffic to port 0"
block drop log quick from <snort2c> to any label "Block snort2c hosts"
block drop log quick from any to <snort2c> label "Block snort2c hosts"
block drop in log quick proto carp from (self) to any
pass quick proto carp all no state
block drop in log quick proto tcp from <sshlockout> to (self) port = ssh label "sshlockout"
block drop in log quick proto tcp from <webConfiguratorlockout> to (self) port = 8080 label "webConfiguratorlockout"
block drop in log quick from <virusprot> to any label "virusprot overload table"
block drop in log quick on bge0 from <bogons> to any label "block bogon IPv4 networks from WAN"
block drop in log quick on bge0 from <bogonsv6> to any label "block bogon IPv6 networks from WAN"
block drop in log on ! bge0 inet from %%main_site_upstreamSubnet%% to any
block drop in log inet from %%main_site%% to any
block drop in log inet from %%main_site_IPtwo%% to any
block drop in log on ! re0 inet from 192.168.2.0/23 to any
block drop in log inet from 192.168.2.20 to any
block drop in log on ! re1 inet from 10.0.0.0/24 to any
block drop in log inet from 10.0.0.1 to any
pass in on lo0 inet all flags S/SA keep state label "pass IPv4 loopback"
pass out on lo0 inet all flags S/SA keep state label "pass IPv4 loopback"
pass out inet all flags S/SA keep state allow-opts label "let out anything IPv4 from firewall host itself"
pass out route-to (bge0 %%main_site_upstream%%) inet from %%main_site%% to ! %%main_site_upstreamSubnet%% flags S/SA keep state allow-opts label "let out anything from firewall host itself"
pass out route-to (bge0 %%main_site_upstream%%) inet from %%main_site_IPtwo%% to ! %%main_site_upstreamSubnet%% flags S/SA keep state allow-opts label "let out anything from firewall host itself"
pass out on enc0 all flags S/SA keep state label "IPsec internal host to host"
pass in quick on re0 proto tcp from any to (re0) port = 8080 flags S/SA keep state label "anti-lockout rule"
anchor "userrules/*" all
pass in log quick on enc0 inet proto tcp from 192.168.5.0/24 to any flags S/SA keep state allow-opts label "USER_RULE: warehouse to LAN (IPSec VPN tunnel passthru enable)"
pass in quick on enc0 inet from 192.168.5.0/24 to 192.168.2.0/23 flags S/SA keep state label "USER_RULE"
pass in quick on enc0 inet from 192.168.2.0/23 to 192.168.5.0/24 flags S/SA keep state label "USER_RULE"
pass in log quick on enc0 inet proto tcp from 192.168.2.8 to 192.168.5.0/24 flags S/SA keep state label "USER_RULE: email to warehouse (outgoing) pass all"
pass in log quick on enc0 inet proto tcp from 192.168.5.0/24 to 192.168.2.3 flags S/SA keep state label "USER_RULE: primary dc incoming"
pass in log quick on enc0 inet proto tcp from 192.168.2.3 to 192.168.5.0/24 flags S/SA keep state label "USER_RULE: primary dc outgoing"
pass in log quick on enc0 inet proto tcp from 192.168.2.6 to 192.168.5.0/24 flags S/SA keep state label "USER_RULE: backup dc outgoing"
pass in log quick on enc0 inet proto tcp from 192.168.5.0/24 to 192.168.2.6 flags S/SA keep state label "USER_RULE: backup dc incoming"
pass in quick on enc0 inet proto tcp from any to 192.168.2.8 port = ldap flags S/SA keep state label "USER_RULE: NAT email LDAP (ipsec tunnel)"
pass in quick on enc0 inet proto tcp from any to 192.168.2.8 port = smtp flags S/SA keep state label "USER_RULE: NAT email smtp incoming (ipsec tunnel)"
pass in quick on enc0 inet proto tcp from any to 192.168.2.8 port = nntp flags S/SA keep state label "USER_RULE: NAT email nntp incoming (ipsec tunnel)"
pass in quick on enc0 inet proto udp from any to 192.168.2.8 port = snmp keep state label "USER_RULE: NAT email snmp (udp) incoming (ipsec tunnel)"
pass in quick on enc0 inet proto tcp from any to 192.168.2.8 port = netbios-ns flags S/SA keep state label "USER_RULE: NAT email mpls-in incoming (ipsec tunnel)"
pass in quick on enc0 inet proto tcp from any to 192.168.2.8 port = kerberos-sec flags S/SA keep state label "USER_RULE: NAT email IPsec incoming (ipsec tunnel)"
pass in quick on enc0 inet proto tcp from any to 192.168.2.8 port = sftp flags S/SA keep state label "USER_RULE: NAT email L2TP incoming (look into this) (ipsec t..."
pass in quick on enc0 inet proto tcp from any to 192.168.2.8 port = https flags S/SA keep state label "USER_RULE: NAT email ActiveSync (TDP/UDP both) (ipsec tunnel)"
pass in quick on enc0 inet proto udp from any to 192.168.2.8 port = https keep state label "USER_RULE: NAT email ActiveSync (TDP/UDP both) (ipsec tunnel)"
pass in quick on enc0 inet proto tcp from any to 196.168.2.8 port = 6001 flags S/SA keep state label "USER_RULE: NAT email RPC 6001 (ipsec tunnel)"
pass in quick on enc0 inet proto tcp from any to 196.168.2.8 port = 6002 flags S/SA keep state label "USER_RULE: NAT email RPC 6002 (ipsec tunnel)"
pass in quick on enc0 inet proto tcp from any to 196.168.2.8 port = 6003 flags S/SA keep state label "USER_RULE: NAT email RPC 6003 (ipsec tunnel)"
pass in quick on enc0 inet proto tcp from any to 196.168.2.8 port = 6004 flags S/SA keep state label "USER_RULE: NAT email RPC 6004 (ipsec tunnel)"
pass in quick on enc0 inet proto udp from any to 192.168.2.8 port = ntp keep state label "USER_RULE: NAT email ntp incoming (ipsec tunnel)"
pass in quick on enc0 inet proto tcp from any to 192.168.2.8 port = pop3 flags S/SA keep state label "USER_RULE: NAT email pop3 incoming (ipsec tunnel)"
pass in quick on enc0 inet proto tcp from any to 192.168.2.3 port = domain flags S/SA keep state label "USER_RULE: NAT dns server forward (ipsec tunnel)"
pass in quick on enc0 inet proto udp from any to 192.168.2.3 port = domain keep state label "USER_RULE: NAT dns server forward (ipsec tunnel)"
pass in quick on enc0 inet proto tcp from any to 192.168.2.3 port = ldap flags S/SA keep state label "USER_RULE: NAT dns server forward (ipsec tunnel)"
pass in quick on enc0 inet proto udp from any to 192.168.2.3 port = ldap keep state label "USER_RULE: NAT dns server forward (ipsec tunnel)"
block drop in quick on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto tcp from %%main_site_upstreamSubnet%% to 192.168.2.0/23 port = ms-sql-s flags S/SA label "USER_RULE: drop all sql incoming"
block drop in quick on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto tcp from %%main_site_upstreamSubnet%% to 192.168.2.0/23 port = ncube-lm flags S/SA label "USER_RULE: drop all sql-net incoming"
pass in quick on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto tcp from any to 192.168.2.10 port = http flags S/SA keep state label "USER_RULE: NAT camera server http incoming"
pass in quick on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto tcp from any to 192.168.2.12 port = http flags S/SA keep state label "USER_RULE: NAT webserv http incoming"
pass in quick on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto tcp from any to 192.168.2.8 port = smtp flags S/SA keep state label "USER_RULE: NAT email smtp incoming"
pass in quick on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto udp from any to 192.168.2.8 port = ntp keep state label "USER_RULE: NAT email ntp incoming"
pass in quick on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto tcp from any to 192.168.2.8 port = pop3 flags S/SA keep state label "USER_RULE: NAT email pop3 incoming"
pass in quick on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto tcp from any to 192.168.2.8 port = nntp flags S/SA keep state label "USER_RULE: NAT email nntp incoming"
pass in quick on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto udp from any to 192.168.2.8 port = snmp keep state label "USER_RULE: NAT email snmp (udp) incoming"
pass in quick on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto tcp from any to 192.168.2.8 port = netbios-ns flags S/SA keep state label "USER_RULE: NAT email mpls-in incoming"
pass in quick on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto tcp from any to 192.168.2.8 port = kerberos-sec flags S/SA keep state label "USER_RULE: NAT email IPsec incoming"
pass in quick on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto tcp from any to 192.168.2.8 port = sftp flags S/SA keep state label "USER_RULE: NAT email L2TP incoming (look into this)"
pass in quick on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto tcp from any to 192.168.2.8 port = https flags S/SA keep state label "USER_RULE: NAT email ActiveSync (TDP/UDP both)"
pass in quick on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto udp from any to 192.168.2.8 port = https keep state label "USER_RULE: NAT email ActiveSync (TDP/UDP both)"
pass in quick on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto tcp from any to 192.168.2.8 port = ldap flags S/SA keep state label "USER_RULE: NAT email LDAP"
pass in quick on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto tcp from any to 196.168.2.8 port = 6001 flags S/SA keep state label "USER_RULE: NAT email RPC 6001"
pass in quick on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto tcp from any to 196.168.2.8 port = 6002 flags S/SA keep state label "USER_RULE: NAT email RPC 6002"
pass in quick on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto tcp from any to 196.168.2.8 port = 6004 flags S/SA keep state label "USER_RULE: NAT email RPC 6004"
pass in quick on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto tcp from any to 196.168.2.8 port = 6003 flags S/SA keep state label "USER_RULE: NAT email RPC 6003"
pass in quick on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto tcp from any to 192.168.2.19 port = 1297 flags S/SA keep state label "USER_RULE: NAT visibar gun incoming"
pass in quick on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto udp from any to 192.168.2.19 port = 1297 keep state label "USER_RULE: NAT visibar gun incoming"
pass in quick on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto icmp from %%remote_site%% to %%main_site%% icmp-type echoreq keep state label "USER_RULE: Easy Rule: Passed from Firewall Log View"
pass in log quick on re0 inet from any to 192.168.2.201 flags S/SA keep state label "USER_RULE: allow to PBX"
pass in log quick on re0 inet from any to 192.168.5.0/24 flags S/SA keep state allow-opts label "USER_RULE: warehouse to LAN (IPSec VPN tunnel passthru enable)"
block drop in quick on re0 inet proto tcp from 192.168.2.0/23 to any port = smtp label "USER_RULE: disallow smtp for subnet"
block drop in quick on re0 inet proto udp from 192.168.2.0/23 to any port = smtp label "USER_RULE: disallow smtp for subnet"
pass in quick on re0 inet from 192.168.2.0/23 to any flags S/SA keep state label "USER_RULE: Default allow LAN to any rule"
pass in log quick on re1 inet proto tcp from any to (self) flags S/SA keep state label "USER_RULE"
pass out route-to (bge0 %%main_site_upstream%%) inet proto udp from (self) to %%remote_site%% port = isakmp keep state label "IPsec: warehouse - outbound isakmp"
pass in on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto udp from %%remote_site%% to (self) port = isakmp keep state label "IPsec: warehouse - inbound isakmp"
pass out route-to (bge0 %%main_site_upstream%%) inet proto udp from (self) to %%remote_site%% port = sae-urn keep state label "IPsec: warehouse - outbound nat-t"
pass in on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto udp from %%remote_site%% to (self) port = sae-urn keep state label "IPsec: warehouse - inbound nat-t"
pass out route-to (bge0 %%main_site_upstream%%) inet proto esp from (self) to %%remote_site%% keep state label "IPsec: warehouse - outbound esp proto"
pass in on bge0 reply-to (bge0 %%main_site_upstream%%) inet proto esp from %%remote_site%% to (self) keep state label "IPsec: warehouse - inbound esp proto"
anchor "tftp-proxy/*" all
pass in on re0 proto udp from any to any port = sip keep state
pass in on re0 proto udp from any to any port 64000:64999 keep state
NAT rules:
no nat proto carp all
nat-anchor "natearly/*" all
nat-anchor "natrules/*" all
nat on bge0 inet from 127.0.0.0/8 to any port = isakmp -> %%main_site%% static-port
nat on bge0 inet from 192.168.2.0/23 to any port = isakmp -> %%main_site%% static-port
nat on bge0 inet from 10.0.0.0/24 to any port = isakmp -> %%main_site%% static-port
nat on bge0 inet from 127.0.0.0/8 to any -> %%main_site%% port 1024:65535
nat on bge0 inet from 192.168.2.0/23 to any -> %%main_site%% port 1024:65535
nat on bge0 inet from 10.0.0.0/24 to any -> %%main_site%% port 1024:65535
no rdr proto carp all
rdr-anchor "relayd/*" all
rdr-anchor "tftp-proxy/*" all
rdr on bge0 inet proto tcp from any to any port = smtp -> 192.168.2.8
rdr on bge0 inet proto udp from any to any port = ntp -> 192.168.2.8
rdr on bge0 inet proto tcp from any to any port = pop3 -> 192.168.2.8
rdr on bge0 inet proto tcp from any to any port = nntp -> 192.168.2.8
rdr on bge0 inet proto udp from any to any port = snmp -> 192.168.2.8
rdr on bge0 inet proto tcp from any to any port = netbios-ns -> 192.168.2.8
rdr on bge0 inet proto tcp from any to any port = kerberos-sec -> 192.168.2.8
rdr on bge0 inet proto tcp from any to any port = sftp -> 192.168.2.8
rdr on bge0 inet proto tcp from any to any port = ldap -> 192.168.2.8
rdr on bge0 inet proto tcp from any to any port = https -> 192.168.2.8
rdr on bge0 inet proto udp from any to any port = https -> 192.168.2.8
rdr on bge0 inet proto tcp from any to %%main_site%% port = 6001 -> 196.168.2.8
rdr on bge0 inet proto tcp from any to %%main_site%% port = 6002 -> 196.168.2.8
rdr on bge0 inet proto tcp from any to %%main_site%% port = 6003 -> 196.168.2.8
rdr on bge0 inet proto tcp from any to %%main_site%% port = 6004 -> 196.168.2.8
rdr on bge0 inet proto tcp from any to any port = 1297 -> 192.168.2.19
rdr on bge0 inet proto udp from any to any port = 1297 -> 192.168.2.19
rdr on bge0 inet proto tcp from any to %%main_site%% port = http -> 192.168.2.12
rdr on bge0 inet proto tcp from any to %%main_site_IPtwo%% port = http -> 192.168.2.10
rdr on enc0 inet proto tcp from any to 192.168.2.3 port = domain -> 192.168.2.3
rdr on enc0 inet proto udp from any to 192.168.2.3 port = domain -> 192.168.2.3
rdr on enc0 inet proto tcp from any to 192.168.2.3 port = ldap -> 192.168.2.3
rdr on enc0 inet proto udp from any to 192.168.2.3 port = ldap -> 192.168.2.3
rdr on re0 inet proto udp from any to ! (re0) port = sip -> 127.0.0.1 port 5060
rdr-anchor "miniupnpd" all
binat on bge0 inet from 192.168.2.10 to any -> %%main_site_IPtwo%%
binat on enc0 inet from 192.168.2.0/23 to 192.168.5.0/24 -> 192.168.2.0/23
Remote site (Cisco) config:
Subnet: 192.168.5.0/24
Default allow all outbound, I'm not sure how to export a nice list from an RV220w like the above, but there's not a lot going on with this thing. I'll remote in shortly and try to take some screenshots. I don't really know where to go from here, but there has to be a way to make this work, right? I can provide more information as requested. Link to the PBX administrator's manual will be below in a comment since I apparently don't have the reputation yet to post more than 2 links at a time.

Leave Your Comment

Leave a Reply