Unable to browse internal IPs but can ping PPPOE + NAT Cisco Lab

[Gnomesenpai1]

Hey all, Ive been setting up all my Cisco lab again and got around to playing with PPPOE and NAT. I have the setup 90% working and I can use internet services just fine, however I am finding i cant use web services i have hosted on my regular LAN. Im running OPNsense as my main router with a C2901 as my lab core and an 881 as one of my many lab routers.

OPNSense > OSPF > C2901 > PPPOE > 881 > VM Client.

The VM client can ping the IP of the local service im trying to reach but web is timing out and ive confirmed my FW rules on OPNSense for my lab networks and all my other lab routers can go to this service so i believe this is an issue either relating to NAT or MTU.

DNS is being handled by 192.168.3.3, but i cannot access a webservice on port 81 on 192.168.3.2 from this PPPOE setup.

C2901:
bba-group pppoe global
virtual-template 1
!
ip dhcp pool pppoe-client
network 10.10.50.0 255.255.255.0
default-router 10.10.50.1
dns-server 192.168.3.3
!
interface GigabitEthernet0/0
ip address 10.10.20.2 255.255.255.252
duplex auto
speed auto
!
interface GigabitEthernet0/1
mtu 1492
no ip address
duplex auto
speed 100
pppoe enable group global

interface Virtual-Template1
mtu 1492
ip address 10.10.50.1 255.255.255.0
peer default ip address dhcp-pool pppoe-client
ppp authentication chap callin

C881G:
interface FastEthernet4
mtu 1492
no ip address
no ip redirects
no ip unreachables
no ip proxy-arp
ip virtual-reassembly in
ip tcp adjust-mss 1451
duplex auto
speed auto
pppoe-client dial-pool-number 1

interface Vlan1
ip address 192.168.103.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
ip tcp adjust-mss 1451
!
interface Dialer1
mtu 1492
ip address negotiated
ip nat outside
ip virtual-reassembly in
encapsulation ppp
ip tcp adjust-mss 1452
dialer pool 1
dialer-group 1
ppp chap hostname customer
ppp chap password 7 01100F175804
ppp pap sent-username customer password 7 01100F175804
ppp ipcp route default
no cdp enable

ip nat inside source list 1 interface Dialer1 overload
ip route 0.0.0.0 0.0.0.0 Dialer1
!
access-list 1 remark ==NAT==
access-list 1 permit 192.168.103.0 0.0.0.255

[MHM Cisco World]

Can you more elaborate from where to where you want to access web

Thanks 

MHM

[Gnomesenpai1]

I can access external services that are outside of my networks fine e.g. google, fast.com via the setup. I can ping to my real local lan addresses hanging off my OPNSense router, but unable to browse to any web pages that are running like my local openspeedtest at 192.168.3.2:8.

[MHM Cisco World]

debug pppoe packet <<- run this command in both pppoe server and client

MHM

[Gnomesenpai1]

Unplugged and replugged the WAN from the 881 to get PPPOE to bounce,  PPPOE has re-established, the 881 spat out the following (ignore the date/time, just brought my lab back online, it was correct earliler):

.Aug 1 21:06:15.228: pppoe_send_padi:
contiguous pak, size 60
FF FF FF FF FF FF F0 F7 55 72 52 B2 88 63 11 09
00 00 00 10 01 01 00 00 01 03 00 08 7E 00 00 01
00 00 1C 96 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00
.Aug 1 21:06:15.228: PPPoE 0: I PADO R:188b.9d64.2721 L:f0f7.5572.52b2 Fa4
contiguous pak, size 65
F0 F7 55 72 52 B2 18 8B 9D 64 27 21 88 63 11 07
00 00 00 2D 01 01 00 00 01 03 00 08 7E 00 00 01
00 00 1C 96 01 02 00 05 63 32 39 30 31 01 04 00
10 A7 79 A2 39 86 A0 9C 5B F8 35 BB E2 40 43 C2
CF
.Aug 1 21:06:17.276: OUT PADR from PPPoE Session
contiguous pak, size 65
18 8B 9D 64 27 21 F0 F7 55 72 52 B2 88 63 11 19
00 00 00 2D 01 01 00 00 01 03 00 08 7E 00 00 01
00 00 1C 96 01 02 00 05 63 32 39 30 31 01 04 00
10 A7 79 A2 39 86 A0 9C 5B F8 35 BB E2 40 43 C2
CF
.Aug 1 21:06:17.276: PPPoE 2: I PADS R:188b.9d64.2721 L:f0f7.5572.52b2 Fa4
contiguous pak, size 65
F0 F7 55 72 52 B2 18 8B 9D 64 27 21 88 63 11 65
00 02 00 2D 01 01 00 00 01 03 00 08 7E 00 00 01
00 00 1C 96 01 02 00 05 63 32 39 30 31 01 04 00
10 A7 79 A2 39 86 A0 9C 5B F8 35 BB E2 40 43 C2
CF
.Aug 1 21:06:17.276: %DIALER-6-BIND: Interface Vi1 bound to profile Di1
.Aug 1 21:06:17.284: %LINK-3-UPDOWN: Interface Virtual-Access1, changed state to up
.Aug 1 21:06:17.344: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state to up

C2901:
Aug 1 21:06:15.219: PPPoE 0: I PADI R:f0f7.5572.52b2 L:ffff.ffff.ffff Gi0/1
contiguous pak, size 60
FF FF FF FF FF FF F0 F7 55 72 52 B2 88 63 11 09
00 00 00 10 01 01 00 00 01 03 00 08 7E 00 00 01
00 00 1C 96 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00
Aug 1 21:06:15.219: PPPoE 0: O PADO, R:188b.9d64.2721 L:f0f7.5572.52b2 Gi0/1
Aug 1 21:06:15.219: Service tag: NULL Tag
contiguous pak, size 65
F0 F7 55 72 52 B2 18 8B 9D 64 27 21 88 63 11 07
00 00 00 2D 01 01 00 00 01 03 00 08 7E 00 00 01
00 00 1C 96 01 02 00 05 63 32 39 30 31 01 04 00
10 A7 79 A2 39 86 A0 9C 5B F8 35 BB E2 40 43 C2
CF
Aug 1 21:06:17.267: PPPoE 0: I PADR R:f0f7.5572.52b2 L:188b.9d64.2721 Gi0/1
contiguous pak, size 65
18 8B 9D 64 27 21 F0 F7 55 72 52 B2 88 63 11 19
00 00 00 2D 01 01 00 00 01 03 00 08 7E 00 00 01
00 00 1C 96 01 02 00 05 63 32 39 30 31 01 04 00
10 A7 79 A2 39 86 A0 9C 5B F8 35 BB E2 40 43 C2
CF
Aug 1 21:06:17.267: [2]PPPoE 2: O PADS R:f0f7.5572.52b2 L:188b.9d64.2721 Gi0/1
contiguous pak, size 65
F0 F7 55 72 52 B2 18 8B 9D 64 27 21 88 63 11 65
00 02 00 2D 01 01 00 00 01 03 00 08 7E 00 00 01
00 00 1C 96 01 02 00 05 63 32 39 30 31 01 04 00
10 A7 79 A2 39 86 A0 9C 5B F8 35 BB E2 40 43 C2
CF
Aug 1 21:09:59.063: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down
Aug 1 21:10:00.063: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to down
Aug 1 21:10:04.063: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up
Aug 1 21:10:05.063: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up

[MHM Cisco World]

Debug not helpful 

Did yoh try access web server via IP not url?

Try lookup also

It can dns issue if you can ping web server 

MHM

[paul driver]

Hello 

---

@Gnomesenpai1 wrote
:but i cannot access a webservice on port 81 on 192.168.3.2 from this PPPOE setup.

---

try :
access-list 1 deny host 192.168.103.2 any
access-list 1 permit 192.168.103.0 0.0.0.255
ip nat inside source static tcp 192.168.103.2 81 interface dialer 1 81

So externally you will connect via: 
http <public ip opensource> 81

[Gnomesenpai1]

Those did not work, I have applied the "ip nat inside source static tcp 192.168.103.2 81 interface dialer 1 81", I see NAT translations under sh ip nat translations.

881G(config)#do sh ip nat translation | inc 192.168.3.2

Pro Inside global Inside local Outside local Outside global
tcp 10.10.50.2:58574 192.168.103.2:58574 192.168.3.2:443 192.168.3.2:443
tcp 10.10.50.2:58581 192.168.103.2:58581 192.168.3.2:443 192.168.3.2:443
tcp 10.10.50.2:58593 192.168.103.2:58593 192.168.3.2:80 192.168.3.2:80
tcp 10.10.50.2:58604 192.168.103.2:58604 192.168.3.2:81 192.168.3.2:81
tcp 10.10.50.2:58605 192.168.103.2:58605 192.168.3.2:81 192.168.3.2:81
tcp 10.10.50.2:58606 192.168.103.2:58606 192.168.3.2:81 192.168.3.2:81
tcp 10.10.50.2:58607 192.168.103.2:58607 192.168.3.2:81 192.168.3.2:81
tcp 10.10.50.2:58608 192.168.103.2:58608 192.168.3.2:81 192.168.3.2:81
tcp 10.10.50.2:58609 192.168.103.2:58609 192.168.3.2:81 192.168.3.2:81
tcp 10.10.50.2:58617 192.168.103.2:58617 192.168.3.2:81 192.168.3.2:81
tcp 10.10.50.2:58618 192.168.103.2:58618 192.168.3.2:81 192.168.3.2:81
tcp 10.10.50.2:58620 192.168.103.2:58620 192.168.3.2:80 192.168.3.2:80
tcp 10.10.50.2:58622 192.168.103.2:58622 192.168.3.2:81 192.168.3.2:81
tcp 10.10.50.2:58623 192.168.103.2:58623 192.168.3.2:81 192.168.3.2:81
tcp 10.10.50.2:58624 192.168.103.2:58624 192.168.3.2:81 192.168.3.2:81
tcp 10.10.50.2:58626 192.168.103.2:58626 192.168.3.2:81 192.168.3.2:81

However still the same situation of it not getting a SYN-ACK and when doing CURL to the web service it fails with a connection reset. 

I couldnt apply "access-list 1 deny host 192.168.103.2 any" as it was "invalid input detected at 'any'". I am wondering if maybe its MTU related? But ive scanned over the configuration several times and everything appears correctly. As I have previously stated, I can access external sites that are outside my RFC1918 prefixes like google. But internal addresses are pingable but not browsable which im assuming due to UDP vs TCP and i can demonstrate external NAT happening.

tcp 10.10.50.2:58630 192.168.103.2:58630 142.250.151.103:443 142.250.151.103:443
tcp 10.10.50.2:58631 192.168.103.2:58631 142.251.30.100:443 142.251.30.100:443

[paul driver]

hello
try applying the ace again without “any”
access-list 1 deny host 192.168.103.2 

Can you elaborate on “internal addresses are ping able but not browsable” ?  
where are you trying to browse from outside or inside your network?

[Gnomesenpai1]

As a test, I removed NAT from the 881 and configured static routes on opnsense and C2901, I am seeing the same behaviour of WAN services accessible and LAN services inaccessible which does make me wonder if this is infact an issue with MTUs? From the guides I have read the MTU values I have chosen should be OK though.

to add to this - I have tested and can SSH into 192.168.3.2, on other networks I have confirmed accessibility to 192.168.3.2:81.

Update - I removed all set MTU valuves and things to appear to be working. Going to re-apply NAT.

Update 2 - re-applied NAT and everything is fine, NAT working as intended so it looks like it was an MTU issue. But i'm still not sure why exactly. If anyone could explain it that would be amazing.

[MHM Cisco World]

We nee to see in which point traffic stop

Can you use tcpdump abd share result here ?

Also for DNS did you check it?

MHM