Cisco 4500-X, ASR-1000, and a GPON stick that can't be reached?

[firestorm-v1]

Hello Community:

I seem to have myself up a river of effluence without a means of locomotion and am hoping someone can give me some help so I can get unstuck.   I've got a Cisco 4500-X switch running IOS-SE v3.11.04E and I've encountered problems with trying to make sure I can talk to a GPON ONT-on-a-stick that's plugged into an attached ASR. Thanks to the help of a few members in the Routing topic, I was able to make progress and get some configuration set on the ASR to allow me to manage a GPON stick, but I'm finding that the 4500 is just not acting right and I can't seem to figure out why.

The GPON stick has an untagged interface (IP: 10.0.5.10) for stick management and monitoring and a tagged interface (VLAN242) that is used by the ASR to obtain DHCP from the service provider.

With that said, the ASR's config is pretty basic:

• The stick is inserted into Gi0/0/0.  There is a service instance defined with untagged encapsulation and bridge-domain 50. The interface also has a directive that changes the MAC address due to provider requirements.
• Gi0/0/0.242 is the DHCP interface to the provider.  It is attached to a "PUBLIC_ROUTING" vrf for routing the public subnet.
• Gi0/0/1 is the downstream trunk attached to the 4500-X on Te1/13.  It has a service instance defined with untagged encapsulation and bridge-domain 50. It has no IP address.
• Gi0/0/1.900 is the public subnet I have allocated from the ISP.  The IP address on this interface is the "default gateway" for the public subnet.  It is also attached to the "PUBLIC_ROUTING" vrf.
• Gi0/0/2 has a service instance defined with untagged encapsulation and bridge-domain 50.  It has no IP address.
• Interface BDI50 has a DHCP assigned address from VLAN50 (currently 10.0.5.77).

With regards to VLAN900, the public subnet, this is working 100% as expected (which is good otherwise I wouldn't be able to post this).

The 4500-X is similarly configured in a basic fashion:

• The two firewalls are at Te1/1 and Te1/2.  Both firewalls have VLAN50 present and provides DHCP and routing on that VLAN.
• The distribution switch and the downstream switch that the ASR was connected to are on Te1/3 and 1/4 respectively.
• The ASR is connected to Te1/13 and is configured as a trunk with VLAN50 native and allowed VLANs set to 50 and 900.  It has spanning-tree portfast edge trunk set and spanning-tree bpdufilter enable set.
• A test port on Te1/14 has also been configured as an access port for VLAN50.  Spanning-tree portfast edge has been set on this port. 

Now that the layout has been described, here's where things get weird.

• When the ASR was attached to the downstream switch, the BDI interface could pull a DHCP address but couldn't ping the stick.  My laptop attached to Gi0/0/2 was able to pull DHCP and ping the stick. When my laptop was attached to the downstream switch, it could pull DHCP and ping the stick.  When the laptop was moved to the 4500-X, it could still pull DHCP but could no longer ping the stick.
• After the ASR was moved to the 4500-X. the BDI interface could still pull DHCP from VLAN50 but it still couldn't ping the stick.  My laptop attached to Gi0/0/2 was still able to pull DHCP but lost access to the stick.
• VLAN50 DHCP interfaces were created on the 4500X, the downstream switch, and other switches in the infrastructure.  All VLAN50 interfaces could ping each other and the other hosts in the VLAN50 subnet except the GPON stick.
• All this time, VLAN900 continues to provide Internet access as expected but the GPON stick remains elusive.

Here's the configs from the 4500-X:

interface TenGigabitEthernet1/13
 description ASR-1000 - Gi0/0/1
 switchport trunk allowed vlan 50,900
 switchport trunk native vlan 50
 switchport mode trunk
 spanning-tree portfast edge trunk
 spanning-tree bpdufilter enable
!
interface TenGigabitEthernet1/14
 description VLAN 50 TEST PORT
 switchport access vlan 50
 switchport mode access
 spanning-tree portfast edge
!

 Here's the relevant configs from the ASR (with minor redaction):

interface GigabitEthernet0/0/0
 description GPON_ONT_STICK
 mac-address xxxx.yyyy.zzzz
 no ip address
 negotiation auto
 cdp enable
 service instance 50 ethernet
  encapsulation untagged
  bridge-domain 50
 !
!
interface GigabitEthernet0/0/1
 description INTERNAL_VLANS
 no ip address
 negotiation auto
 service instance 50 ethernet
  encapsulation untagged
  bridge-domain 50
 !
!
interface GigabitEthernet0/0/2
 no ip address
 negotiation auto
 service instance 50 ethernet
  encapsulation untagged
  bridge-domain 50
 !
!
interface BDI50
 vrf forwarding INTERNAL_MGMT
 ip address dhcp

At some point in the future, I want the 4500-X to take the place of the ASR with respect to this GPON stick, but I can't fully get this working with the ASR first before I start looking at whether or not this is feasible.  For the time being, getting the GPON stick's management interface working as intended would be a huge plus.

Any ideas on this?  Thank you in advance.

[firestorm-v1]

Just updating the thread since I haven't been able to make any progress on my own...

It would appear that the GPON stick isn't compatible with the 4500-X because it's either 1G or 2.5G and the switch only offers 1G/10G without a way to hard set the port speed to 1G.  Since I don't have a 10G GPON stick, the switch will "see" the stick, it will show in "show interface status" and for all intents "appears" to be working, but in actuality it does not.  To further exacerbate the situation, the GPON stick appears to drop its untagged management interface when the PON interface establishes connectivity. 

In the meantime, I'm using a Cisco ASR-1000 to perform routing capability for the edge and it's been working well so far.  If I do happen to make any breakthrough discoveries, I'll update the thread.

[stephan.l.martin1]

It would appear at face value that the issue you are running into may be centered around your VRF instances. It would be beneficial to see the complete topology as unfortunately the description (although very detailed) is a bit hard to follow completely. I suspect that your vlan 900 traffic and vlan 50 traffic are having some issues given the separate VRF instances.

[firestorm-v1]

Hello Stephan:

That's the weird part, the VRF is working correctly and is fully operational as far as I can tell (I'm routing through the VRF to type this).  I diagrammed it in MS Paint (forgive me) but this is how it's working at the moment albeit with no access to the stick itself. 
What's not shown are the LAN-side connections. Te1/13 on the 4500-X is configured as trunk with VLAN50 being native and VLAN900 being tagged.  Some of the IP addresses have been changed for obfuscation purposes but all are configured in the same public subnet (e.g. the x.x.x.1 address is actually x.x.x.254 but are still /29).

For all intents in testing, I used Te1/14 on the 4500-X as an access port on VLAN50, and previously had Gi0/0/2 on the ASR attached to bridge-domain 50 for further testing.  I was able to pull DHCP (VLAN50 subnet) through Gi0/0/2 and I could intermittently access the ONT stick on its untagged interface so I am fairly confident the bridge-domain is working correctly.  The Te1/15 and Te1/16 interfaces are access interfaces for VLAN900 and connect to the WAN ports on my OPNsense boxes. They share a VIP for high availability although I realize they're not really HA due to connecting to the same switch.

As far as the VRF is concerned, only the VLAN tagged interfaces are members (Gi0/0/0.242 required by the ISP, and Gi0/0/1.900, the public VLAN).  Their parent interfaces, (Gi0/0/0 and Gi0/0/1 respectively) are not associated with the VRF and only have service instances defined with the common bridge domain.

The only other VRF is the Mgmt-vrf but that's used exclusively with Gi0, the management interface for the ASR and is an internal only interface.

[stephan.l.martin1]

Your system is operating in the ATT_PUBLIC_ROUTING vrf as where the ONT stick management is operating in the INT_MGMT vrf. Unless you have route leaking configured within the VRF Forwarding instances your device doesn't know of the subnet as the routing table is a separate instance.

That is what I am reading from all of this- The traffic will make it to the router within the VRF but doesn't have a direct path. If you were to show IP cef to the ONT address within the VRF that would indicate whether a route is known or not.

Edit: Admittedly my knowledge of BDIs is a bit fuzzy as I haven't worked with them in a while. Your ONT is either in the global routing table or your INT_MGMT VRF. I would be looking to see which routing VRF instance the ONT is in as that will point to why you cannot access it.

[firestorm-v1]

Hello Stephan:

I owe you an apology, I got this thread confused with my other thread about the ASR when I replied to you initially.

At this point, I've about given up on ever getting access to the stick while it's linked up and running.  I've done several tests that prove the ASR and the bridge-domain are working as expected, I can pull DHCP through the ASR on Gi0/0/2 (also part of the bridge-domain), the ASR can pull DHCP and both pulls are on the VLAN50 subnet as expected.  As far as the BDI/ bridge domain is concerned, it's doing its job.  I think it's the GPON stick that has the problem.  If it were a misconfiguration somewhere, I'd expect to either pull the wrong subnet or get a DHCP timeout. Since the stick is joined by the bridge-domain, I would expect it to not need a VRF since the bridge-domain just forwards the packets onward.  Even removing the VRF that's shown for the BDI interface had no effect.

I'm not 100% convinced that the stick's untagged management interface stays reachable when fiber is attached as I've only been able to test this with a media converter but even then it wasn't consistent.  While others using the same model stick on other machines/routers/switches comment that it should stay reachable as long as the port is up, that's not been my experience and even with a basic media converter, I've seen the management interface stop responding once VLAN242 pulls an IP and data starts flowing.  To further complicate matters, the 4500-X does obey RX_LOS from the SFP and will down the switchport if there's no fiber attached which makes troubleshooting difficult when you can't talk to the management interface without the fiber.  I wish I knew of a way around that, but I am not optimistic.

I had hoped that at some point in the near future (when I recover from the insanity of the last two weeks at this), I could roll the ASR's config into the 4500 and use it for edge routing and my core switching (it would be practically the same setup) but it feels like the deck is stacked against me and I've shelved the idea for now.  

[stephan.l.martin1]

Have you verified that the management is reachable within the VRF? If you are unsure which VRF it is in you can do “show vrf” and then show ip arp vrf (name) until you figure it out. If you can ping it within the vrf but not from your workstation you need to configure route leaks for the two vrf instances if the end state is management reachability. 

[firestorm-v1]

Hello Stephan:

Sorry about leaving you hanging, work and home have been getting more chaotic the closer we get to christmas and I haven't had much time to dive into this.  That being said, I was able to do a couple of tests on the stick and found that the stick just stops communicating on the management VLAN after a period of time if there's no activity on that interface.  To get around this, I set up a ping running from another host on the management VLAN to ping the stick's IP address.  It's been 100% reachable for the week I've been running it and the Web-UI for the stick is also reachable as well.  Even with it pinging and being reachable, there's no entries in the ARP table on either the 4500-X or the ASR which I find unusual.

In talking with other users that have this ONT stick and similar sticks, apparently this is a somewhat known thing. The community firmware for a similar ONT stick actually includes a ping function that pings the management VLAN's default gateway in order to keep the stick reachable.  That being said, I am fairly confident that this is not an issue in either the 4500-X or the ASR, but rather something in the stick falling asleep if there's no traffic on the management interface.

I did find some promising configuration options for the ONT stick that may prove useful, but just to make sure I don't get my posts muddled together with multiple issues, I'll create a separate post to keep them siloed.

Thank you!

[stephan.l.martin1]

Thank you for the follow-up! I would recommend setting up a ICMP-Echo SLA as this will remain persistent vs. a workstation that will stop pinging if someone exits the terminal or reboots the system.

Interesting "feature"