need help in Nexus switch

[mabuzaid1]

Dear Cisco Support Community ,

Thank you all in advance .

I am new to Cisco nexus switch and as of now i have simple question for connection with layer 2 uplink (firewall):

our Scenario is we have two c9000 series and we have two fortigate firewall.

the Scenario is we need to connect a server with two links (one link to each switch "9000 series") and then the server Shall ping to fortigate (through the trunk port connected between the switch and Fortigate ).

Here i have attached the schame for the network .

I had also posted this issue before (i got it till some stage and then after that we are unable to complete what ever appeared later) .

I will share also the link of the previous post. 

https://community.cisco.com/t5/data-center-switches/need-help-with-nexus-switch/m-p/4105322#M5991

Note The VPC seems to be working (show VPC shows everything is good , however we can't ping from / to switch and fortgiate .

Eagerly waiting your help.

Config for Link between fortigate and nexus switch :

Nexus 1
===========
interface port-channel17
description ***To-FW-1***
switchport
switchport mode trunk
vpc 17

interface port-channel18
description ***To-FW-2****
switchport
switchport mode trunk
vpc 18

interface Ethernet1/17
switchport
switchport mode trunk
channel-group 17
no shutdown

interface Ethernet1/18
description ***To-FW-2***
switchport
switchport mode trunk
channel-group 18
no shutdown

Nexus SW 2
==========

!
interface port-channel17
description ***To-FW-1***
switchport
switchport mode trunk
vpc 17
!
interface port-channel18
description ***To-FW-2****
switchport
switchport mode trunk
vpc 18
!
interface Ethernet1/17
description ***To-FW-1***
switchport
switchport mode trunk
channel-group 17
no shutdown
!
interface Ethernet1/18
description ***To-FW-2****
switchport
switchport mode trunk
channel-group 18
no shutdown

Thanks

[TONY SMITH]

Sounds like something is wrong in the IP configuration somewhere, but you don't show any of this in your post.  Where are you pinging from, how is it configured and where is it connected?

[mabuzaid1]

Dear Tony ,,,,,

Thanks for your reply .....

Our Scenario is we have two Fortigate ( Active / Passive ) with up-link to each nexus switch (we have 4 fortigate).

Here are configuration for first two nexus switches :

Nexus 1:

feature vpc

feature lacp

feature inter-vlan

conf t

interface eth 1/48(for keep alive)

switchport

switchport mode access

switchport access vlan 10

no shut

exit

interface vlan 10

no shut

ip add 10.10.10.1 / 30

exit

vpc domain 1

peer-keep alive dest 10.10.10.2 source 10.10.10.1 vrf def

exit

interface eth 1/49

channel-group 1 mode actvie

exit

interface port-channel 1

switchport

switchport mode trunk

switchport trunk allowed vlan all 

peer-link

exit

interface eth1/1-2

switchport

channel-group 20 mode active

exit

interface port-chnnel 20

switchport

switchport mode trunk

switchport trunk allowed vlan all 

vpc 10

exit

same configuration for Nexus 2 :

different is vpc domain 1 

peer-keep alive dest 10.10.10.1 source 10.10.10.2 vrf defualt

rest is same .

Now with enabling the port mirroing at fortigate these two switches works fine (connected to servers using one link from each switch to the server) .

Problem :

we can't introduce the other two nexus (all switches are 9000 series ) to the fortigate as it flaps all the links and start dropping the packets .

note:

fortigate uses channel group between all the ports (single channel group , trunk port from passive and active).

We dont want to change the hirarchy of fortigate (as less as possible if needed)

how do you think we can link these other two switches throught the trunk port of fortigate (they will be connected with different end devices "servers") also we really want to have link from eatch fortigate to each switch in the network (all 4 f them )

Thanks in advnace

[TONY SMITH]

OK it may not be relevant but it's not good practice to use operational SVIs as your VPC keepalive.  You should configure your E1/48 as no switch port, give it an IP address and use a separate VRF.  However that's by the by.

In terms of your design, at Layer 2 do the two Fortigates act as two separate devices, or do they share some sort of entity?  Ideally you want each Fortigate to have a "normal" port channel configuration, they don't need to know it's actually a VPC but they do need to know it's a port channel and therefore they should expect inbound on either of the ports.   Their load balancing algorithm doesn't necessarily have to match that of the VPC.

Can you draw how you'd ideally like it to look like when you have four Nexuses?   You are aware the two pairs of Nexuses need to use different VPC domain IDs if they are linked at L2.

[mabuzaid1]

Thanks Tony for you valuable comments .

I have attached the required design .For the Layer two connection with Firewall its their request as they have 

a Vlan domain on this .

hope you got my idea .

[TONY SMITH]

OK so the other two Nexuses will be a separate VPC pair, with VPC port channels to the two firewalls.  Is that correct? 

If that's the case then each Fortigate needs to have two separate port channels configured, two ports in each.  One port channel goes to your original two Nexuses, and one goes to the 3rd and 4th.

Can you post up "sh vpc brief" from the VPC Primary of each pair?

[mabuzaid1]

Good morning Dear Toney

I have seen the vpc status its working well and forming adjacency but switches are at the site not with me currently, however whenever possible i would show you the vpc status.

So, you mean that each fortigate will have two port channels trunk with different vlan tag. 

In that scnario do we need also to connect the two vpc domain devices togather, if needed how the connection will be and what are the configuration for this connection a side from the two vpc domain config. 

One last question, vpc domain can bundle two devices only right? 

Because if they requested to keep the fortigate config downstream port as one channel group for all 4 ports, is it possible? 

[TONY SMITH]

A VPC domain comprises two Nexuses.  If you have four, they need to be two VPC domains, with different domain IDs if they have any L2 connectivity between them.

A channel group on an external device, like your Fortigate, must go to a single switch or a single VPC pair.  So your four ports could be in one channel group with two ports going to Nexus 1 and two to Nexus 2.  Or two channel groups one going to each VPC pair.

Why do they want four ports to go as one channel group to two separate sets of switches?  I can't see any logical reason.

[mabuzaid1]

From you post you mentioned that , With 2 VPC domain we can create one channel group at two fortigates (all downstream link ) could be linked with one channel group (trunk port with Vlan tagged at fortigate ) .

I have checked the scenario as below :

- Created two VPC domain (1 pair of UTP & 1 pair of Fiber ).

- Didn't link the two VPC pairs together (as that the end devices connected to them are total separate either fiber link or     UTP link).

- Assigned the downstream ports at each VPC domain as one channel group , trunk port with VPC no 15,16 respectively .

Result :

- The ports (at each fortigates were not working normaly it goes up/down and it was abnormal .

- I thought this scenario is not possible as the fortigate will think each VPC pair as standalone device which couldn't       be possible to use one channel group with multiple devices (as there are multiple VPC pairs ).

Can you explain please how to use multiple VPC domains and connect all of them in one port channel at the pairs of fortigate devices ?

Thanks

[TONY SMITH]

You need to stick to the principles.  A VPC Domain has exactly two Nexus switches as members.  A VPC has member interfaces on each of the two VPC Domain members.  Incoming frames can be received on any member port, and outgoing frames sent on any member port.  

So your four Nexuses can't be all one VPC domain, they will be two pairs.  VPC Domain IDs must be different.

On the Fortigate a channel group must have all it's member interfaces connected to the same VPC pair.

What I can't understand is the desire to have one channel group on the Fortigate connected to different VPC pairs.  Inherently in a channel group the device is free to use any of the member ports.  So how would it be expected to work if a frame was received on say port 1, from Nexus one, but the Fortigate chooses to send its reply out of port 4 going to Nexus 4?  

Why can't you have all traffic from the Fortigate going to Nexus 1 and 2?

What else is connected?  Presumably you have other devices and maybe a whole network connected to the Nexuses in some way.

[mabuzaid1]

Yes i understand that but i juat wanted to clarify. 

So, what are the other options to keep the network configuration same as its. 

Does interconnecting two vpc domain with each other and then connecting them to firewalls using links to nexus 1 & links 2 works fine (all 4 nexus switches has one vlan for member ports).

Kindly advice. 

[TONY SMITH]

I'm afraid I really don't follow what function you're trying to achieve.

Let's start with the Nexuses, and forget the connection to the Fortigates.  How is the LAN configured with the Nexuses and other devices.  With that defined, what functions are the connections to the Fortigates providing?

I think there's a danger of getting bogged down in "can I connect this to that" without knowing exactly what function is wanted.

[mabuzaid1]

Thanks Tony & sorry for the late reply as i was traveling & couldn't check the Community .

In reference to the main function wanted from the VPC connection it is simple as this :

- We have a single Vlan for all devices connected to the Nexus switches.

- Each end device connected to nexus has dual link (channel - group).

- Each Nexus shall has two connection with gateway (Fortigate "Active/Passive").

- All uplinks and member ports are assigned to single same Vlan .

- Some Servers connected through UTP switches & some connected through Fiber switches .

- Now two nexus switches (UTP ports ) function properly without any issue  between the uplinks & member ports.

This is our simple scenario and I think the easy way is to change the network architecture by creating another ether channel to connect to the other two nexus fiber switches (with different vlan id as SVI ).

If this is the only possible scenario (to change port channel configuration at firewall) , Then (correct me if i am wrong ) we don't need to do any physical connection between the two multiple VPC domain as the downstream devices are either connected using fiber or UTP connection !

Do you think there is any other possible solution with the said scenario ?.

Please advice.

Thanks 

[TONY SMITH]

---

@mabuzaid1 wrote:

This is our simple scenario and I think the easy way is to change the network architecture by creating another ether channel to connect to the other two nexus fiber switches (with different vlan id as SVI ).

If this is the only possible scenario (to change port channel configuration at firewall) , Then (correct me if i am wrong ) we don't need to do any physical connection between the two multiple VPC domain as the downstream devices are either connected using fiber or UTP connection !

Do you think there is any other possible solution with the said scenario ?.

Please advice.

Thanks 

---

Let's see if I have this correct.  What you're suggesting is that you'll create a separate channel group on the Fortigates, with a different subnet and VLAN ID, and connect this to a VPC channel group on the other two Nexuses.   No Layer 2 connection between the first VPC pair and the second.   Communications between devices on Nexus 1&2, and devices on Nexus 3&4 will go via the Fortigate, these being the only devices with connections to both Nexus pairs.

All correct so far?

If so then depending on how much internal traffic you have that could be inefficient, considering the Nexuses can do L3 switching and routing.

What is the reason for the second two Nexuses?  Why do they have to connect to the Fortigates rather than having let's say Fortigates connecting Nexus 1&2, then Nexus 3&4 connecting to 1&2 with a double ended VPC.  Depending on the models you could use direct attach cables and connect the Nexus to Nexus links at 40 or even 100gig.

[mabuzaid1]

Hi Tony ,

Yes you got the idea , The reason why he have multiple nexus switches is because some end devices has only

Fiber NIC And some has only UTP , with that said we have multiple VPC devices .

The reason not to interconnect multiple VPCs together and connect them through fortigate is that we would like

To utilize The full speed of each firewall port (10GB) with a single devices (in this case channel group).

In case of interconnection between multiple VPC (we will need to create multiple VPC domain and configure it

Thanks