600 Virtual Machines distributed throughout the ESX cluster.
300 active VLANs
According to the Cisco Nexus 1000v Getting Started Guide, Release 4.0(4)SV1(3) - Configuration Limits, one can configure up to 512 active VLANs across all VEMs in a single 1000v instance; but can only configure 256 port profiles. Am I reading that right?
I see several ways to get around this, but none of them are as desirable as being able to configure more Port Profiles in a single 1000v instance.
One workaround is to not assign VLANs in a Port Profile and instead manually configure the VLAN on the Vethernet port after it has been created. However, this would all but eliminate the usefulness of Port Profiles. Furthermore, Cisco has stated in the Interface Configuration Guide "[w]e do not recommend that you override port profile configurations by making changes to the assigned interface configurations. Only make configuration changes to interfaces to quickly test a change or to disabled a port." I don't know why they don't recommend it, but I would guess that there could be a chance that a Port Profile configuration could somehow be reapplied to an interface and you would lose the custom configuration.
Another workaround is to deploy more than one 1000v instance. Using the example above, I could put half of the VEMs on one 1000v instance, and half on another. Neither instance could have all of the VLANs, so the 1000v would essentially force a redesign of the VMware infrastructure; affecting things such as DRS, VMotion, and potentially other VMware design considerations of which I'm not even aware (I'm not a VMware guy).
Assuming for a moment that I'm not wrong about this limitation and its effects, which I could very well be, why does the limitation exist? VMware supports up to 512 port groups on a host and apparently 4094 VLANs on a vSwitch. Why are the limits of the 1000v so much lower? Is that just the price that has to be paid in order to enjoy the other features of the 1000v? While the 1000v would enhance the security and manageability of the network, it seems like it would actually limit the otherwise bountiful capacity of an environment that utilizes a lot of VLANs.
I'd really like to hear from someone that I've got it all wrong and that there's a simple solution to my perceived problem.
You're read correctly. Though some of the limitations were historically from VMware, we have "throttled" some of the limitations in our early releases.
The next release (SV1.4) which should be out in a couple months will increase the following limits:
I understand some of the limits "don't make sense", but our dev teams try to roll out features in a controlled manner and sometimes they don't always add up logically
Additionally, the default maximum number of ports per port-profile is currently hardcoded to 32. In VMware vSphere 4.1, the maximum number of supported ports per DVS is increasing from 8000 ports to 30000. With the vSphere 4.0 maximum, you might want to have the maximum ports per port-profile set to a lower value say four. For vSphere 4.1, the user might want to set the maximum to fifteen per port-profile. We've added the ability for the user to set the default maximum number of ports per port-profile. To do this, we will introduce a new configuration command:
Thanks, Robert. I'm glad the limitation is only a cautionary throttling and not something worse. 2048 port-profiles per DVS would definitely be sufficient for the foreseeable future. I also see that the stats I grabbed for VMWare with respect to maximum port groups and vlans were old; vSphere 4.1 drastically increased a lot of maximums. http://www.vmware.com/pdf/vsphere4/r41/vsp_41_config_max.pdf
Thanks for pointing out the maximum ports per DVS. I don't think we're on 4.1 yet in production, but I have been testing the 1000v on vSphere 4.1 in the lab. I'll make sure that vSphere 4.1 is a pre-requisite to deployment of the 1000v in production so that max-ports is not as much of a concern. I think the VM admins have already planned for an upgrade to 4.1 soon.