The packet flow is not the same for different ports

[meidanmeshulam]

Hi all,

I got the next scenario:
Two VMs, each VM is under its own EPG and I got a standard contract between the EPGs.

I noticed that the "stateful" check box in the filter is behaving a little strangely, for example:
If the destination port is set to TCP 80(HTTP), the provider can't initiate a session with the consumer, no matter if the "stateful" checkbox is checked or not, the "stateful" checkbox has no effect at all.

Now, if I'll change the destination port to TCP 22(SSH), the "stateful" button will affect the packet flow, and now if I'll leave it unchecked, both the consumer and the provider can initiate the session.

Can anyone please help me understand this behavior?

Thanks a lot!

[Sergiu.Daniluk]

Hi @meidanmeshulam 

Can you collect the commands as listed in this post: https://community.cisco.com/t5/application-centric/meaning-of-stateful-flag-in-aci-filter/td-p/4278713 and share the results here so we can analyze them together.

Cheers,

Sergiu

[meidanmeshulam]

@Sergiu.Daniluk  Thanks for your time and help

Here is the output:
sb-lab-leaf-1# show zoning-rule
+---------+--------+--------+----------+----------------+---------+----------+------------------------------+----------+----------------------+
| Rule ID | SrcEPG | DstEPG | FilterID | Dir | operSt | Scope | Name | Action | Priority |
+---------+--------+--------+----------+----------------+---------+----------+------------------------------+----------+----------------------+
| 4096 | 0 | 0 | implicit | uni-dir | enabled | 16777200 | | deny,log | any_any_any(21) |
| 4097 | 0 | 0 | implarp | uni-dir | enabled | 16777200 | | permit | any_any_filter(17) |
| 4098 | 0 | 0 | implicit | uni-dir | enabled | 2818048 | | deny,log | any_any_any(21) |
| 4099 | 0 | 0 | implarp | uni-dir | enabled | 2818048 | | permit | any_any_filter(17) |
| 4100 | 0 | 15 | implicit | uni-dir | enabled | 2818048 | | deny,log | any_vrf_any_deny(22) |
| 4102 | 0 | 0 | implicit | uni-dir | enabled | 2949120 | | deny,log | any_any_any(21) |
| 4120 | 0 | 0 | implarp | uni-dir | enabled | 2949120 | | permit | any_any_filter(17) |
| 4118 | 0 | 15 | implicit | uni-dir | enabled | 2949120 | | deny,log | any_vrf_any_deny(22) |
| 4111 | 0 | 49160 | implicit | uni-dir | enabled | 2949120 | | permit | any_dest_any(16) |
| 4104 | 0 | 49162 | implicit | uni-dir | enabled | 2949120 | | permit | any_dest_any(16) |
| 4101 | 0 | 49164 | implicit | uni-dir | enabled | 2949120 | | permit | any_dest_any(16) |
| 4103 | 49171 | 16388 | 5 | bi-dir | enabled | 2949120 | Tenant_Skybox1:icmp-contract | permit | fully_qual(7) |
| 4123 | 16388 | 49171 | 5 | uni-dir-ignore | enabled | 2949120 | Tenant_Skybox1:icmp-contract | permit | fully_qual(7) |
| 4126 | 0 | 49168 | implicit | uni-dir | enabled | 2949120 | | permit | any_dest_any(16) |
| 4106 | 49171 | 16388 | 13 | bi-dir | enabled | 2949120 | Tenant_Skybox1:web-contract | permit | fully_qual(7) |
| 4119 | 16388 | 49171 | 14 | uni-dir-ignore | enabled | 2949120 | Tenant_Skybox1:web-contract | permit | fully_qual(7) |
| 4109 | 16387 | 16386 | 64 | uni-dir-ignore | enabled | 2949120 | Tenant_Skybox1:ssh-contract | permit | fully_qual(7) |
| 4124 | 16386 | 16387 | 63 | bi-dir | enabled | 2949120 | Tenant_Skybox1:ssh-contract | permit | fully_qual(7) |
| 4112 | 0 | 0 | implicit | uni-dir | enabled | 2523139 | | deny,log | any_any_any(21) |
| 4115 | 0 | 0 | implarp | uni-dir | enabled | 2523139 | | permit | any_any_filter(17) |
| 4117 | 0 | 15 | implicit | uni-dir | enabled | 2523139 | | deny,log | any_vrf_any_deny(22) |
+---------+--------+--------+----------+----------------+---------+----------+------------------------------+----------+----------------------+

The relevant contract are highlighted.
Basically we have a contract between two VMs for SSH traffic, and we have a contract between two VMs for HTTP traffic.
The "Stateful" checkbox in unchecked in both of them (meaning traffic can be initiated from both VMs).

In reality,
SSH is allowed from both epgs (from both VMs).
HTTP is allowed only from consumer to provider epg.

Thanks again!

[Sergiu.Daniluk]

For the sake of t-shooting, if you add SSH entry into the HTTP contract, is it behaving the same way as HTTP (only consumer -> provider direction is allowed)?

How about adding HTTP to SSH-contract?

Cheers,

Sergiu

[Sergiu.Daniluk]

Hi @meidanmeshulam 

Can you share the output of the following commands:

show zoning-filter filter 13
show zoning-filter filter 14
show zoning-filter filter 63
show zoning-filter filter 64

Additionally, also share some screen captures with the configuration of the SSH and HTTP contract (contract, subject, filter)

And to have a complete picture view, please share the ACI version running on APIC/Leafs and also the hardware model for the leafs.

Cheers,

Sergiu

[meidanmeshulam]

@Sergiu.Daniluk  yes of course


sb-lab-leaf-2# show zoning-filter filter 13
+----------+------+--------+-------------+------+-------------+----------+-------------+-------------+-----------+---------+-------+-------------+-------------+----------+
| FilterId | Name | EtherT | ArpOpc | Prot | ApplyToFrag | Stateful | SFromPort | SToPort | DFromPort | DToPort | Prio | Icmpv4T | Icmpv6T | TcpRules |
+----------+------+--------+-------------+------+-------------+----------+-------------+-------------+-----------+---------+-------+-------------+-------------+----------+
| 13 | 13_0 | ip | unspecified | tcp | no | no | unspecified | unspecified | http | http | dport | unspecified | unspecified | |
+----------+------+--------+-------------+------+-------------+----------+-------------+-------------+-----------+---------+-------+-------------+-------------+----------+

sb-lab-leaf-2# show zoning-filter filter 14
+----------+------+--------+-------------+------+-------------+----------+-----------+---------+-------------+-------------+-------+-------------+-------------+----------+
| FilterId | Name | EtherT | ArpOpc | Prot | ApplyToFrag | Stateful | SFromPort | SToPort | DFromPort | DToPort | Prio | Icmpv4T | Icmpv6T | TcpRules |
+----------+------+--------+-------------+------+-------------+----------+-----------+---------+-------------+-------------+-------+-------------+-------------+----------+
| 14 | 14_0 | ip | unspecified | tcp | no | no | http | http | unspecified | unspecified | sport | unspecified | unspecified | |
+----------+------+--------+-------------+------+-------------+----------+-----------+---------+-------------+-------------+-------+-------------+-------------+----------+
sb-lab-leaf-2#

sb-lab-leaf-2# show zoning-filter filter 63
+----------+------+--------+-------------+------+-------------+----------+-------------+-------------+-------------+-------------+-------+-------------+-------------+----------+
| FilterId | Name | EtherT | ArpOpc | Prot | ApplyToFrag | Stateful | SFromPort | SToPort | DFromPort | DToPort | Prio | Icmpv4T | Icmpv6T | TcpRules |
+----------+------+--------+-------------+------+-------------+----------+-------------+-------------+-------------+-------------+-------+-------------+-------------+----------+
| 63 | 63_0 | ip | unspecified | tcp | no | no | unspecified | unspecified | unspecified | unspecified | proto | unspecified | unspecified | |
+----------+------+--------+-------------+------+-------------+----------+-------------+-------------+-------------+-------------+-------+-------------+-------------+----------+
sb-lab-leaf-2#

sb-lab-leaf-2# show zoning-filter filter 64
+----------+------+--------+-------------+------+-------------+----------+-------------+-------------+-------------+-------------+-------+-------------+-------------+----------+
| FilterId | Name | EtherT | ArpOpc | Prot | ApplyToFrag | Stateful | SFromPort | SToPort | DFromPort | DToPort | Prio | Icmpv4T | Icmpv6T | TcpRules |
+----------+------+--------+-------------+------+-------------+----------+-------------+-------------+-------------+-------------+-------+-------------+-------------+----------+
| 64 | 64_0 | ip | unspecified | tcp | no | no | unspecified | unspecified | unspecified | unspecified | proto | unspecified | unspecified | |
+----------+------+--------+-------------+------+-------------+----------+-------------+-------------+-------------+-------------+-------+-------------+-------------+----------+
sb-lab-leaf-2#

[Sergiu.Daniluk]

Hi @meidanmeshulam 

Filter 63 and 64 are part of "ssh" contract, while filter 13 and 14 are part of "web" contract.

| 4106 | 49171 | 16388 | 13 | bi-dir | enabled | 2949120 | Tenant_Skybox1:web-contract | permit | fully_qual(7) |
| 4119 | 16388 | 49171 | 14 | uni-dir-ignore | enabled | 2949120 | Tenant_Skybox1:web-contract | permit | fully_qual(7) |
| 4109 | 16387 | 16386 | 64 | uni-dir-ignore | enabled | 2949120 | Tenant_Skybox1:ssh-contract | permit | fully_qual(7) |
| 4124 | 16386 | 16387 | 63 | bi-dir | enabled | 2949120 | Tenant_Skybox1:ssh-contract | permit | fully_qual(7) |

Looking at them how they are programmed in software ("show zoning-filter filter id"), there are two problems I see:

1. The "ack" TcpRule is missing from both contracts and "stateful" flag is set to "no". I would expect something like this in the example below (example from lab) where the "acl" is present for direction provider pcTag -> consumer pcTag. And since you confirmed is missing from hardware as well, I am confused how it's working for HTTP. Anyway, second problem:

2. For the "ssh-contract" you have definitely encountered a bug. You do not see "ssh" port present for DestinationPort neither for SourcePort

+----------+------+--------+-------------+------+-------------+----------+-------------+-------------+-------------+-------------+-------+-------------+-------------+----------+
| FilterId | Name | EtherT | ArpOpc | Prot | ApplyToFrag | Stateful | SFromPort | SToPort | DFromPort | DToPort | Prio | Icmpv4T | Icmpv6T | TcpRules |
+----------+------+--------+-------------+------+-------------+----------+-------------+-------------+-------------+-------------+-------+-------------+-------------+----------+
| 63 | 63_0 | ip | unspecified | tcp | no | no | unspecified | unspecified | unspecified | unspecified | proto | unspecified | unspecified | |
+----------+------+--------+-------------+------+-------------+----------+-------------+-------------+-------------+-------------+-------+-------------+-------------+----------+
sb-lab-leaf-2#

+----------+------+--------+-------------+------+-------------+----------+-------------+-------------+-------------+-------------+-------+-------------+-------------+----------+
| FilterId | Name | EtherT | ArpOpc | Prot | ApplyToFrag | Stateful | SFromPort | SToPort | DFromPort | DToPort | Prio | Icmpv4T | Icmpv6T | TcpRules |
+----------+------+--------+-------------+------+-------------+----------+-------------+-------------+-------------+-------------+-------+-------------+-------------+----------+
| 64 | 64_0 | ip | unspecified | tcp | no | no | unspecified | unspecified | unspecified | unspecified | proto | unspecified | unspecified | |
+----------+------+--------+-------------+------+-------------+----------+-------------+-------------+-------------+-------------+-------+-------------+-------------+----------+

For a working contract, the filters would look like this (again, example from lab):

Leaf1201# show zoning-filter filter 38
+----------+------+--------+-------------+------+-------------+----------+-------------+-------------+-----------+---------+-------+-------------+-------------+----------+
| FilterId | Name | EtherT |    ArpOpc   | Prot | ApplyToFrag | Stateful |  SFromPort  |   SToPort   | DFromPort | DToPort |  Prio |   Icmpv4T   |   Icmpv6T   | TcpRules |
+----------+------+--------+-------------+------+-------------+----------+-------------+-------------+-----------+---------+-------+-------------+-------------+----------+
|    38    | 38_0 |   ip   | unspecified | tcp  |      no     |   yes    | unspecified | unspecified |    ssh    |   ssh   | dport | unspecified | unspecified |          |
+----------+------+--------+-------------+------+-------------+----------+-------------+-------------+-----------+---------+-------+-------------+-------------+----------+
Leaf1201# show zoning-filter filter 39
+----------+------+--------+-------------+------+-------------+----------+-----------+---------+-------------+-------------+-------+-------------+-------------+----------+
| FilterId | Name | EtherT |    ArpOpc   | Prot | ApplyToFrag | Stateful | SFromPort | SToPort |  DFromPort  |   DToPort   |  Prio |   Icmpv4T   |   Icmpv6T   | TcpRules |
+----------+------+--------+-------------+------+-------------+----------+-----------+---------+-------------+-------------+-------+-------------+-------------+----------+
|    39    | 39_0 |   ip   | unspecified | tcp  |      no     |   yes    |    ssh    |   ssh   | unspecified | unspecified | flags | unspecified | unspecified |   ack    |
+----------+------+--------+-------------+------+-------------+----------+-----------+---------+-------------+-------------+-------+-------------+-------------+----------+

I suspect you are facing the same bug detailed by @RedNectar in his post: https://rednectar.net/2020/09/05/aci-version-mismatch-alert-dont-use-v5-on-apic-and-v14-on-leaves/

My recommendation is to:

1. Reapply the contract on EPGs

2. If that doesn't solve the problem, reconfigure the contract from scratch (including the subject/filters)

3. If still doesn't solve the problem, try with a range of ports instead of specific to ssh

Once again, what version you have (APIC and leafs) and what hw model are your leafs?

Stay safe,

Sergiu

[Sergiu.Daniluk]

Also, there is an explanation with some diagrams about stateful flag here:  https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-743951.html#Filterentryconfigurationoption

Take care,

Sergiu

[meidanmeshulam]

Thanks @Sergiu.Daniluk !
I read all of that like 100 times already : ]
This behavior is mentioned nowhere.

I will get the output of the command you mentioned earlier and hopefully well figure it out
Thanks again!

[RedNectar]

Hi @meidanmeshulam ,

---

I got the next scenario:
Two VMs, each VM is under its own EPG and I got a standard contract between the EPGs.

OK Let's add a picture

I noticed that the "stateful" check box in the filter is behaving a little strangely, for example:
If the destination port is set to TCP 80(HTTP), the provider can't initiate a session with the consumer, no matter if the "stateful" checkbox is checked or not, the "stateful" checkbox has no effect at all.

Functioning PERFECTLY then - just as designed for port 80. To check the functioning of the STATEFUL FLAG, you'd have to spoof a packet from the Provider to the Consumer with a SOURCE PORT of 80 - like this where an evil person who has already compromised the web server has found port 23 open on a consumer device:

The whole idea of the Stateful option is to FORCE traffic from the Provider to the Consumer to have the ACK flag set, thereby preventing the Provider from sending a SYN packet to initiate a connection. It worries me that in one of your other replies to @Sergiu.Daniluk you state:

The "Stateful" checkbox in unchecked in both of them (meaning traffic can be initiated from both VMs).

Leaving the stateful checkbox unchecked DOES NOT MEAN TRAFFIC CAN BE INITIATED FROM BOTH VMs - unless you are talking about spoofing source ports.

Now, if I'll change the destination port to TCP 22(SSH), the "stateful" button will affect the packet flow, and now if I'll leave it unchecked, both the consumer and the provider can initiate the session.

Can anyone please help me understand this behavior?

---

Then I'm pretty sure you must have another contract somewhere that allows that TCP SYN packet to flow from the Provider to the Consumer with a Destination Port=22.  I'd have to see your configuration to find it, but I'll bet it's there somewhere. But before I go chasing down that rabbit hole, can you please look at your configuration with a renewed understanding of how the Stateful Flag works, and see if your contracts make more sense.  You may find reading this post useful too.

And finally - you may also find the built-in contract_parser.py python script useful too - it needs to be run from a leaf switch. I've figured out your Tenant name from another reply, but you'll have to add your Application Profile and EPG names

Leafxxx# contract_parser.py --epg tn-Tenant_Skybox1/ap-<YOUR_AP>/epg-<YOUR_EPG>

or even

Leafxxx# contract_parser.py --epg tn-Tenant_Skybox1/ap-<YOUR_AP>/epg-<YOUR_EPG> --port 22

I hope this helps.

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem.

---

[meidanmeshulam]

@RedNectar  Thank you for the beautifully explained answer.
As you said, the "Stateful" checkbox is for "preventing the Provider from sending a SYN packet to initiate a connection".
That is exactly how I see it, and if the provider is unable to send a SYN packet, obviously he shouldn't be able to initiate a connection.
Having said that, how come my provider is able to initiate a connection when I'm using destination port TCP 22?
And why he cannot when I'm using destination port TCP 80?

There is no contract to allow this traffic. I am using an ACI mini-environment for lab and testing purposes. I deployed it myself and there is absolutely no other contract that allows this traffic or other contract in general (unless there is a hidden one that I cannot see?).
I tried testing it with only a single contract and the result is the same.
I will try the python script tomorrow and keep you posted!

If you have any other ideas I would love to hear
Thanks a lot!

[RedNectar · Accepted solution]

Hi @meidanmeshulam ,

I've just had a related thought when you said

Having said that, how come my provider is able to initiate a connection when I'm using destination port TCP 22?

---

and I remembered writing this at some stage

https://rednectar.net/2020/09/05/aci-version-mismatch-alert-dont-use-v5-on-apic-and-v14-on-leaves/

[Moderators: I beg forgiveness for linking to my own blog, but I don't believe you'll find any official Cisco documentation of this bug other than the confirmation that it IS a bug in this Facebook post. And begging DOUBLE forgiveness for linking to Facebook]

So if you are using a combination of V5 APIC and v14 NXOS on the leaves, filters for port 22 get instantiated on the leaves as filters for ANY port.

BUT...

you still shouldn't have been able to INITIATE a connection from the Provider, even with that bug, but it just MIGHT have something to do with your problem.

[meidanmeshulam · Accepted solution]

@Sergiu.Daniluk , Thanks for your help mate, seems like you are right and the DToPort remains unspecified when I use SSH.
@RedNectar Thanks champ, your discovery really helped me out. I am using APIC version  5.0(2e) just as you specified on your FB page and blog, and thanks to Sergius's observations the picture is now clear : ] It's a Cisco bug.
I will say that my switches are all running nxos 15 and not 14 as you are showing in the blog but the issue remains.

@Robert Burns Thanks mate for your help as well, much appreciated.

Case closed!

[Sergiu.Daniluk]

Did you tried reapplying or reconfiguring the ssh contract? I am curios if this fixes the programming.

Cheers,

Sergiu

[Robert Burns]

The reason why you're not seeing the filter rules for those commands is because you ran them on the wrong switch (sb-lab-leaf-2).  The Rule IDs originally were shown from Leaf1, and the Filter IDs are locally significant.  
Re-collect the following from sb-lab-leaf-1:

show zoning-filter filter 13
show zoning-filter filter 14
show zoning-filter filter 63
show zoning-filter filter 64

Robert