*** ANNOUNCEMENT *** Experiencing High Failure Rates

[devnet_geek]

All - We are experiencing high failure rates in our sandbox environment. Work is in progress for the resolution. We will update soon

[Leo Laohoo]

---

@devnet_geek wrote:
We will update soon

---

@devnet_geek, 

Is/Are there any update(s)?  

[Blake B]

<disregard this post> (*I've updated my previous post)

[Blake B]

---

@devnet_geek wrote:

We will update soon

---

I understand there are ongoing issues; have there been any updates since Friday? By the way, thank you for working the weekend to resolve it! For many of us, the weekends are the only days we can really squeeze in quality lab/study, so we definitely appreciate it.
    I also wanted to provide feedback from the end-user perspective that the issue(s) does still exist. From my limited perspective, I am getting authentication failures when establishing a ssh connection on standard port 22. When debugging the connection attempts, symptoms present similar to that of when PublicKeyAuthentication is required for remote connections. However, an error I have not seen before(*see reference below) involving, $SSH_SK_PROVIDER, would point me to investigate how MFA/FIDO is configured on the server-side.
*REFERENCE: https://www.openssh.com/releasenotes.html

devvy[@]fedora $ ssh-vvv admin@devnetsandboxiosxe.cisco.com
...
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug2: resolving "sandbox-iosxe-latest-1.cisco.com" port 22
debug3: ssh_connect_direct: entering
debug1: Connecting to sandbox-iosxe-latest-1.cisco.com [131.226.217.181] port 22.
debug1: Connection established
...
<multiple failed attempts at reading my local ~/.ssh/config file for matching keys..>
...
debug3: recv - from CB(2) ERROR:108, io:0000021F1D439BA0
kex_exchange_identification: read: Connection reset
Connection reset by 131.226.217.181 port 22

   Below are the Always-On Lab FQDNs that I have been attempting to connect with since yesterday afternoon:
devnetsandboxiosxe.cisco.com
sandbox-iosxe-recomm-1.cisco.com
sandbox-iosxe-latest-1.cisco.com

[jokearns1]

Hi,

In regards to the following, access is restored.

devnetsandboxiosxe.cisco.com
sandbox-iosxe-recomm-1.cisco.com
sandbox-iosxe-latest-1.cisco.com

However, we aim to reconfigure all of our Always On labs in the near future to become reservation based. Users will need to create a reservation to generate their own credentials to the server. We will no longer be providing the same creds for everyone. It results in far too much abuse of the resource, killing access for everyone else.

With regard to the above Announcement, we have patched the issue from last Friday and labs are spinning up as normal. Thanks fro your patients.

Joe Kearns

[bigevilbeard]

@Blake B i dont think the alway on devices are impacted here in this update from the team. The only now working URL for the IOS XE device is devnetsandboxiosxe.cisco.com - as far as i know the others are not longer active.

Sadly the always on sandboxes are subject to abuse which often locks other users out for periods until the team can reboot or rebuild the devices.

Hope this helps.

[devnet_geek · Accepted solution]

Issue has now been fixed. Sandbox services are stable now.