CML Server FLAT network setup issue

[Elleku1985]

Hi,

I installed, for the client I am working for, CML v1.2.

Performing the first setup according to the Cisco Modeling Labs 1.2 User Guide, we specified in UWM --> CML Server --> System Configuration --> Networks tab, the FLAT-FLAT1-SNAT subnets information according to the customer's addressing plan.

After rehost and reboot, thus, we are not able to launch any simulation from the CML client, receiving the following error message:

(ERROR) [Sep/14/2016 13:00:32] Failed to start simulation "2-node-zaSBza": Exactly 1 subnet needed for LXC FLAT network "flat"

I performed the following checks:

• UWM --> CML Server --> System Status --> Check Health Status : All checks passed and in a green status
• ifconfig on the CML Server, and all interfaces IP Addresses are consistent with the subnets specified in UWM
• /etc/virl.ini: all the information inside the file are correct and consistent with the subnets configured in UWM
• /etc/settings.ini: all the information inside the file are correct and consistent with the subnets configured in UWM
• UWM --> Connectivity: Here we have that L2 FLAT IP Address allocations shows all networks information set to none, while L3 SNAT IP Address allocations and Project Management IP address allocations have all the information wrong and different from the networks configured in the System Configuration.

I hope someone can support me with any suggesiton.

Thanks in advance

Lucio

[ssnider]

can you provide a copy of the topology.virl project file; a description of your installation (vm vs baremetal + any platform details, and a copy of the virl.ini file.

Is this a trial set up, or is this an active subscription?

[Elleku1985]

Thanks for the answer.

Attached to the post you can find a copy of a project file we can not simulate, but consider that the error happens with any kind of topoplogy, even with a single IOSv router.

The CML Server is a Virtual Machine running on ESXi 6.0.0; the host server is a Cisco UCSC-C220-M4S. It is an active subscription, not a trial.

CML version is 1.2.0

I also attached a copy of the virl.ini file; inside of it, you can notice the following networks that we configured via the User Workspace Management:

• FLAT (eth1): 10.12.250.0/24
• FLAT1 (eth2): 10.12.251.0/24
• SNAT (eth3): 10.12.252.0/24

Thanks for your help.

Regards.

[ssnider]

For whatever reason, it would seem like the rehosting process did not complete properly.  In that this is a subscription, it is appropriate to open a ticket with TAC - that will enable further investigation of this deployment.  Alternatively, the CML app may be redeployed - however, this will create a new vMAC for the VM and necessitate getting a new license to replace the initial.  Thus before doing so, it may be best to try a "rehosting" manually as to avoid the relicensing ...

If the virl.ini file accurately reflects what your integration demands, via the CML VM's vConsole, or an SSH session to the VM:

execute  /usr/local/bin/vinstall rehost

allow to complete - you will get an explicit message to reboot - be patient and allow the process to complete.

Upon getting that message, reboot the CML VM.

log back into the CML's console, then execute   /usr/local/bin/vinstall bridge

This will be a bit quicker.. when complete, reboot again.

when the CML VM completes the boot cycle, give it an extra 4-5 minutes before attempting to login to uwmadmin pages or initiate a CML_Client session. 

following the manual rehosting, perform the system verification checks and attempt running your sample topology again. 

If this doesn't resolve your issue - it would be advisable to open a TAC case for further investigation.  you will need either the contract number or sales order number.  This will enable further diagnosis of your installation.

cheers,

steve

[Elleku1985]

Thanks againg Steve for your support.

Unfortunately the suggested procedure doesn't solve the problem.

During the rehost process we could notice that several times the correct information about networks are listed; here it is an example extraction of this output:

l2_address:
        10.12.250.254/24
    l2_address2:
        10.12.251.254/24
    l2_end_address:
        10.12.250.253
    l2_end_address2:
        10.12.251.253
    l2_mask:
        255.255.255.0
    l2_mask2:
        255.255.255.0
    l2_network:
        10.12.250.0/24
    l2_network2:
        10.12.251.0/24
    l2_network_gateway:
        10.12.250.1
    l2_network_gateway2:
        10.12.251.1
    l2_port:
        eth1
    l2_port2:
        eth2
    l2_port2_enabled:
        True
    l2_start_address:
        10.12.250.50
    l2_start_address2:
        10.12.251.50

But at the end of the process, one of the last logs states that it was unable to find flat, fat1 and ext-net subnets:

Unable to find subnet with name 'flat'
Unable to find subnet with name 'flat1'
Unable to find subnet with name 'ext-net'
['id']
Agent id could not be found
+------------+-------+
| Field      | Value |
+------------+-------+
| floatingip | 50    |
| network    | -1    |
| port       | -1    |
| router     | -1    |
| subnet     | -1    |
+------------+-------+
Request Failed: internal server error while processing your request.
Request Failed: internal server error while processing your request.
Request Failed: internal server error while processing your request.

We tried anyway to complete the procedure with reboot, vinstall bridge and reboot again. After 10 minutes I looked at uwmadmin panel, and still FLAT, FLAT1 and SNAT subnets are set to none in the connectivity section; simulation also fails again with the same error.

At this point we will proceed with a request to Cisco TAC for further investigation.

Thank you

Lucio

[Elleku1985 · Accepted solution]

I could finally find the source of the problem.

With a better analysis of the rehost process output, I could notice the following error (the same for SNAT, FLAT and FLAT1 subnet creation):

 ID: create snat subnet
    Function: cmd.run
        Name: neutron --os-tenant-name admin --os-username admin --os-password password --os-auth-url=http://127.0.1.1:5000/v2.0 subnet-create ext-net 10.12.252.0/24 --allocation-pool start=10.12.252.50,end=10.12.252.253 --gateway 10.12.252.1 --name ext-net --dns-nameservers list=true
      Result: False
     Comment: Command "neutron --os-tenant-name admin --os-username admin --os-password password --os-auth-url=http://127.0.1.1:5000/v2.0 subnet-create ext-net 10.12.252.0/24 --allocation-pool start=10.12.252.50,end=10.12.252.253 --gateway 10.12.252.1 --name ext-net --dns-nameservers list=true" run
     Started: 09:40:21.440097
    Duration: 1398.836 ms
     Changes:   
              ----------
              pid:
                  26064
              retcode:
                  1
              stderr:
                  Invalid values_specs --dns-nameservers list=true
              stdout:

As we actually do not need DNS services for our simulations, we were not specifying any DNS for SNAT, FLAT and FLAT1 IP address leaving blank the related field on CML Server configuration via uwmadmin, but it seems that if you don't specify a DNS the subnet creation fails (but no error messages are shown in Web GUI, you should dig inside the ssh output).

Leaving the default DNS IPs (Google's), or custum IPs, allows the subnet creation to complete and simulations can be launched.

Thanks for support.

Regards,

Lucio