cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2804
Views
10
Helpful
10
Replies

ACI VMM VMWare Integration Debugging

nplusplus
Level 1
Level 1

Hello, Everyone,

 

We have just implemented a VMM integration with a vCenter 7.0.0.10600 environment.  The vDS was created, along with a single port group corresponding to the pre-provisioned management EPG.  We then associated a pre-provisioned Storage EPG with the VMM domain, made a number of related changes to it, and eventually removed and attempted to re-add it.  However, it never was recreated, and now, ACI seems unable to create any new port groups.  After associating other EPGs with the domain, I see vCenter logs indicating the ACI account is logging in successfully and executing API calls, but the new port groups do not appear.  Does anyone have any advice for how to debug such vCenter API calls.  I would like to be able to see what API calls are being made, and how vCenter is handling them because, despite the ACI faults that say ACI cannot connect to the VMM domain, it clearly is connecting and attempting some interaction, so I feel I cannot trust the faults.

 

Thank you,

Nathan

1 Accepted Solution

Accepted Solutions

I think I just found the root cause of your problems:

ACI 4.2.2f does not support vCenter 7.x

https://www.cisco.com/c/dam/en/us/td/docs/Website/datacenter/aci/virtualization/matrix/virtmatrix.html

This is most likely because some VMware APIs changed.

 

Stay safe,

Sergiu

 

View solution in original post

10 Replies 10

Sergiu.Daniluk
VIP Alumni
VIP Alumni

Hi @nplusplus 

"We then associated a pre-provisioned Storage EPG with the VMM domain, made a number of related changes to it, and eventually removed and attempted to re-add it"

What changes were done? Have you performed any changes manually in vCenter?

 

Stay safe,

Sergiu

Thank you, Sergiu,

 

I am not the vCenter administrator, and while I believe no changes were made to the ACI vDS in vCenter, I am certain changes were made to other settings, including changes to a manually created vDS to enable access to vCenter within the environment for ACI (i.e. manually created vDS w/ port groups for management and storage with static tags).  In ACI, we added two additional EPGs in our "storage" BD to create VMM domain associations and restrict them to single uplinks per VMWare iSCSI multipathing requirements.  The related storage port groups were not created, and at some point in the process we removed the original storage EPG association, which removed the previously ACI-created port group for it.  Since then, we have been unable to affect the creation of new port groups.  Of note, we added back the original storage EPG association and regained access to vCenter b/c its removal broke such access.  However, the storage port group still was not recreated.

 

Regards,

Nathan

Can you share (print screen?) the faults you have listed in the VMM domain?

Thanks,

Sergiu

Hello, Sergiu,

 

Thank you for your attention to this.  I am working on posting a screenshot, it is just that posting to the community is a luxury that I cannot always find time to do, and obfuscating a screenshot of an ACI fault listing is tedious.

 

I also have a TAC case open, and if anything comes out of that I will be sure to post it.  Additionally, I have considered just recreating the VMM domain.  I'll let you know if we go that route also.

 

Thank you,

Nathan

Sergiu.Daniluk
VIP Alumni
VIP Alumni

Hello again,

 

I found that there are some logs generated in /var/log/vmware/vpxd/vpxd.log:

 

2020-09-18T11:02:06.761Z info vpxd[04199] [Originator@6876 sub=vpxLro opID=5d730ab6] [VpxLRO] -- BEGIN task-33332 -- dvs-861 -- vim.DistributedVirtualSwitch.addPortgroup -- 52fb40a4-f26e-63f6-5a15-0bf2faaa44db(52794cc1-6451-fb32-a7f5-c15362a45ee4)
2020-09-18T11:02:06.807Z info vpxd[04199] [Originator@6876 sub=vpxLro opID=5d730ab6] [VpxLRO] -- FINISH task-33332

dvs-861 is just the Object ID of the DVS, not the actual nam, but you can see that there is an addPortgroup method called.

 

However, the API access logs are not listed here.

 

Cheers,

Sergiu

Robert Burns
Cisco Employee
Cisco Employee

What version of ACI are you running?

Robert

nplusplus
Level 1
Level 1

Thank you @Robert Burns and @Sergiu.Daniluk for your inquiries and interest in this.  The TAC case never really went anywhere before I just removed and readded the VMM domain, which resolved but also means I never found a cause for the problem.  Based on some subsequent behavior I have observed, in which I have been unable to remove and readd a vDS of the same name without removing the containing vSwitch folder in VCenter in between, I suspect we just encountered an error removing the problematic vDS in an earlier change iteration such that it appeared unchanged but threw no further errors in VCenter when trying to update it.

 

It is worth noting this made very clear a few things.

1) One must remove all guest and vmk attachments from any related port groups before deleting a port group or vDS through ACI.  This, of course, is documented in the ACI docs.  I'm just calling it out here.

2) In the combination of our current version of ACI (4.2(2f)) and VCenter (7.x), removing a vDS does not remove the containing folder in VCenter that gets created upon creation of a vDS.  This appears to prevent successful complete deletion of the vDS and does not allow replacing the vDS with another one of the same name.  I suspect this could be called a bug.  Not sure if I'll get around to submitting this as a ticket if anyone wants to take it on with a lab environment or something.  I don't know.  Maybe it's intentional.

3) ACI creates events in VCenter for vDS and port group deletion with "critical" status, which seems a little overkill.  It would be better to have such an intentional action be flagged as something other than critical IMO.  This feels like a customer had a problem with someone removing VMM domain associations unexpectedly and asked Cisco to make it a critical alert when a logged event would have been just fine such that the customer could then set their own severity in a secondary monitoring system.

 

Thank you,

Nathan

I think I just found the root cause of your problems:

ACI 4.2.2f does not support vCenter 7.x

https://www.cisco.com/c/dam/en/us/td/docs/Website/datacenter/aci/virtualization/matrix/virtmatrix.html

This is most likely because some VMware APIs changed.

 

Stay safe,

Sergiu

 

You know, I have been through the "Cisco ACI with VMware VDS Integration" document I don't know how many times, and I never followed that link. 🤦

Thank you, @Sergiu.Daniluk .

At the same time, I am a little disappointed that I had a ticket open about this with TAC for a couple of weeks and no one pointed me to that.

Looking through the ACI versions now to determine my update path.

Regards,

Nathan

You may want to check this tool: https://www.cisco.com/c/dam/en/us/td/docs/Website/datacenter/apicmatrix/index.html

It will give you the upgrade path, supported version for integration technologies and ACI software caveats.

 

Take care,

Sergiu

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Save 25% on Day-2 Operations Add-On License