impossible to connect to SDWAN manager when using sdwan-lab-deployment

[thibaultm]

Now that SDWAN manager loads in CML,
connection to it whether using pat (after having configured it in cockpit): export MANAGER_IP='pat:2002'
or whether a new IP is used : export MANAGER_IP='192.168.1.41'
never works (attempt 27 and counting...) :

Trying to look at logs here is what I get :

and that :

I don't know what to think now ?
could it be because of disk issues ? or something else ?

when trying to reach the manager from my ubuntu VM, it does not work :

 

[rkochery]

Hi @thibaultm , Could you please refer the instructions in this training video. 

https://learningnetwork.cisco.com/s/learning-plan-detail-standard?ltui__urlRecordId=a1c6e00000975VjAAI&ltui__urlRedirect=learning-plan-detail-standard

Setting Up a Catalyst SD-WAN Fabric Quickly in CML

You need to login with your CCO to be able to access it. 

[thibaultm]

Hello, nice try, but I know very well that video by Joe Clark, I've seen it several times and I followed it VERY carefully as well as the other one I found :
Build your own Cisco Catalyst SD-WAN Virtual Lab with free Lab Deployment Tool in CML - YouTube

By the way that same video by Joe Clark can also be seen on youtube at :
Easy SD-WAN in CML with catalyst-sdwan-lab - YouTube

That is only AFTER have seen several times that video, trying to implement BOTH options mentionned that I realised it doesn't work in my case hence the post above.

That being said I might have misunderstood it or something like that but watched it I surely did several times.

[rkochery]

Okay, few questions. 

1. Does regular external connectivity work using bridge mode in external connectivity? 

2. Have you tried without PAT? I see that you mentioned you configured PAT in cockpit, Have you verified that PAT is working on other nodes and just does not work in SD-WAN Manager? 

3. Could you check the status of the nms 

$ request nms all status

Also did you try restarting the nms? 

$ request nms all restart

[thibaultm]

Thanks for your reply

1. Yes
On my (W11) laptop on top of CML VM, I have an ubuntu 22.04 VM I use for network automation.
it is on that VM that I implemented :
catalyst-sdwan-lab · PyPI
So I never created the lab myself on CML, it was created by the script based on the data I  entered.
So clearly external connectivity to the CML VM works well (in that case) otherwise it would not have created the lab that shows on the screenshot.

2. Yes. I have tried with and without PAT.
but I haven't tried PAT on other nodes, just for SD-WAN Manager

3. What do you mean by "nms" ?
Where should "request nms all restart" be entered ?

For your information this is the second post I entered for those tests, the first one I posted is :
Solved: resource issue when using sdwan-lab-deployment-tool in CML 2.8 - Cisco Community
I must admit I did not repeat what I presented in that post.

Now as to Restart, that previous post was yesterday, I stopped My CML VM and my Ubtunu VM (both configured in VMworkstation Pro 17) between yesterday and today and I still have the same behaviour of impossibility to connect to SDWAN manager either with pat:2002 or with a new IP address.

[rkochery]

Hi, Are you referring to PATty tool inside CML? https://developer.cisco.com/docs/modeling-labs/patty-tool-mapping-configuration/

Please confirm that syntax that you are using for creating the mapping, it should be pat:5443:443

The command  "request nms all restart" should be performed on the sd-wan manager node. 

I would strongly recommend, testing PAT on a regular node and make sure it is working, before trying on sd-wan nodes. 

[thibaultm]

Hello some news,
1) I did not enter the pat configuration manually, I simply followed Joe clark instruction in his video, i.e enter :
export MANAGER_IP="pat:2002"
in his sdwan.sh file where he puts all his values for the lab as the IP of CML and admin password etc. and then he sources it.
May be it is not the good solution when running CML with a personal license on a single server ?

Now that being said, I also tried the manager with a new IP taken from the CML IP range, and it works.
I can log in the manager, so for the moment since it works I'll stick with that trying to go further if possible.

Though I don't know why the deploy script could not reach it.

Now comparing with Joe Clark video, when logged in the manager I do not see vSmart and vBond seen in the controllers windows of the manager, I only see the manager, so it is not complete but I don't know why

[thibaultm]

Ok I made another attempt from the script (still with a dedicated IP for vmanage), and it worked :
(sdwan_lab1) thibault@NetAutoU2204:~/csdwan$ vi sdwan_lab2.sh
(sdwan_lab1) thibault@NetAutoU2204:~/csdwan$ source sdwan_lab2.sh
(sdwan_lab1) thibault@NetAutoU2204:~/csdwan$ csdwan setup
Setup task done
(sdwan_lab1) thibault@NetAutoU2204:~/csdwan$ csdwan deploy 20.15.1
Deploy task done
#############################################
Lab is deployed.
CML URL: https://192.168.1.29
SD-WAN Manager URL: https://192.168.1.41:443
Use the username/password set with the script for CML and SD-WAN Manager login.
All other nodes use default username/password.
#############################################
(sdwan_lab1) thibault@NetAutoU2204:~/csdwan$

and this time when logging in vmanage I can see vSmart and vBond, great I can try to proceed

[thibaultm · Accepted solution]

OK So  I implemented the end of the video from Joe clark by adding 1 node (alpine linux) behind each Edge device and it worked well :
(sdwan_lab1) thibault@NetAutoU2204:~/csdwan$ csdwan add 2 edges 17.15.01a --lab "sdwan_lab2"
Add task done
(sdwan_lab1) thibault@NetAutoU2204:~/csdwan$

I managed to get each alpine linux ping each other as in the video by Joe Clark.

Aftarwards I wanted to test the backup feature as in the video of tzarski :
Build your own Cisco Catalyst SD-WAN Virtual Lab with free Lab Deployment Tool in CML

But I got the following error (N.B. that is OK  for me I have a SDWAN lab cool) :

(sdwan_lab1) thibault@NetAutoU2204:~/csdwan$ csdwan backup
Creating backup of Manager01 node (1/13)...Traceback (most recent call last):
File "src/pyats/connections/manager.py", line 499, in pyats.connections.manager.ConnectionManager.connect
File "src/unicon/bases/connection.py", line 714, in unicon.bases.connection.Connection.connect
TypeError: connect() got an unexpected keyword argument 'log_stdout'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "src/unicon/statemachine/statemachine.py", line 737, in unicon.statemachine.statemachine.StateMachine.go_to
File "src/unicon/statemachine/statetransition.py", line 478, in unicon.statemachine.statetransition.AnyStateTransition.do_transitions
File "src/unicon/eal/dialogs.py", line 479, in unicon.eal.dialogs.Dialog.process
File "src/unicon/eal/dialog_processor.py", line 339, in unicon.eal.dialog_processor.SimpleDialogProcessor.process
File "src/unicon/eal/dialog_processor.py", line 219, in unicon.eal.dialog_processor.SimpleDialogProcessor.expect_eval_statements
File "src/unicon/eal/backend/spawn.py", line 111, in unicon.eal.backend.spawn.RawSpawn.read_update_buffer
File "src/unicon/eal/backend/spawn.py", line 96, in unicon.eal.backend.spawn.RawSpawn.read
File "src/unicon/eal/backend/pty_backend.py", line 174, in unicon.eal.backend.pty_backend.Spawn._read
unicon.core.errors.EOF: Unable to read. Connection closed or not available

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "src/unicon/bases/connection.py", line 830, in unicon.bases.connection.Connection.proxy_connect
File "src/unicon/bases/linux/connection_provider.py", line 62, in unicon.bases.linux.connection_provider.BaseLinuxConnectionProvider.connect
File "src/unicon/bases/linux/connection_provider.py", line 94, in unicon.bases.linux.connection_provider.BaseLinuxConnectionProvider.establish_connection
File "src/unicon/statemachine/statemachine.py", line 740, in unicon.statemachine.statemachine.StateMachine.go_to
unicon.core.errors.StateMachineError: Failed while bringing device to "any" state

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "src/unicon/bases/connection.py", line 770, in unicon.bases.connection.Connection.connect
File "src/unicon/bases/connection.py", line 833, in unicon.bases.connection.Connection.proxy_connect
unicon.core.errors.ConnectionError: failed to connect via proxy <unicon.plugins.linux.LinuxConnection object at 0x7a31e02906d0>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/thibault/csdwan/sdwan_lab1/bin/csdwan", line 8, in <module>
sys.exit(main())
File "/home/thibault/csdwan/sdwan_lab1/lib/python3.10/site-packages/catalyst_sdwan_lab/__main__.py", line 578, in main
backup.main(
File "/home/thibault/csdwan/sdwan_lab1/lib/python3.10/site-packages/catalyst_sdwan_lab/tasks/backup.py", line 237, in main
[personality, node_type, pylab] = check_pyats_device_connectivity(
File "/home/thibault/csdwan/sdwan_lab1/lib/python3.10/site-packages/catalyst_sdwan_lab/tasks/backup.py", line 96, in check_pyats_device_connectivity
if validate_credentials(pylab, node_label):
File "/home/thibault/csdwan/sdwan_lab1/lib/python3.10/site-packages/catalyst_sdwan_lab/tasks/backup.py", line 35, in validate_credentials
pylab.run_command(node_label, "show version")
File "/home/thibault/csdwan/sdwan_lab1/lib/python3.10/site-packages/virl2_client/models/cl_pyats.py", line 208, in run_command
return self._execute_command(
File "/home/thibault/csdwan/sdwan_lab1/lib/python3.10/site-packages/virl2_client/models/cl_pyats.py", line 180, in _execute_command
pyats_device.connect(log_stdout=False, learn_hostname=True)
File "src/pyats/connections/manager.py", line 501, in pyats.connections.manager.ConnectionManager.connect
File "src/unicon/bases/connection.py", line 799, in unicon.bases.connection.Connection.connect
unicon.core.errors.ConnectionError: failed to connect to Manager01
failed to connect via proxy <unicon.plugins.linux.LinuxConnection object at 0x7a31e02906d0>
(sdwan_lab1) thibault@NetAutoU2204:~/csdwan$

I must say I already installed pyats some time ago and I use it on my ubuntu VM so it could be related to that.
That being said great job making those script to help deploy a 

SD WAN lab in CML.

I agree with Joe Clark in his video that even past sdwan labs are consumming disk space so cleaning is needed and it's true it takes time to boot but very interesting indeed