cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

Parallel commits through REST

khgrant
Cisco Employee
Cisco Employee

 

Hi Team,

 

 

For a PoC with ATT, we are load-testing NSO 3.4.3 with concurrent service-instantiations on ALU devices using commit queues through the REST API. Each service makes changes on a single device.

 

 

One of the tests we ran was executing 10/20/50 service create calls on a single device, with commit-queue globally enabled.

 

 

We repeated this test, executing 10/20/50 service create calls on two devices, again with commit-queue globally enabled.

 

 

What we noticed is that the time it took for instantiating all services was more or less similar in the 1st and 2nd test, while we expected it to take about half the time for the 2nd test (we assume services for device 1 and device 2 are configured concurrently).

 

 

Is there anything we are doing wrong or is our expectation wrong?

 

 

Thanks!

 

8 REPLIES 8

khgrant
Cisco Employee
Cisco Employee

 

Hi Michel,

 

 

South-Bound, we have a lot of threads (I have heard the number being 100) that program up the devices in parallel.

 

 

The slowest of those devices will determine how long it will take for the SB programming to finish.

 

 

What is NTT’s requirement here? Is it to determine how many parallel NB requests that can be handled at a time? I don’t think there is any hardcoding, we keep accepting requests until we run out of OS resources (socket, file-descriptors, things like that).

 

 

There is a limit to the number of CLI sessions we can have at a time (which can be configured in ncs.conf). Don’t think there is any such limit for REST/HTTP connections though. Or at least I haven’t read about it yet.

 

 

Thanks,

 

  1. Bilal.

 

khgrant
Cisco Employee
Cisco Employee

 

Hi Bilal,

 

 

The result we are seeing is that when we run 50 simultaneous service create requests on a single device, all are committed successfully in ~22 seconds.

 

 

When we run the same 50 simultaneous service create requests on 2 devices (25 for each device), we expect them to be committed in ~11 seconds but instead they are all committed in ~22 seconds.

 

 

So the feeling is like there is a single queue, and (no matter what the configuration is) a transaction can only run after the previous transaction in the queue is committed.

 

 

What I am looking to understand is whether we are doing it wrong, or is our understanding wrong?

 

 

Thanks!

 

khgrant
Cisco Employee
Cisco Employee

 

How are you measuring the time taken for the two scenarios?

 

 

If you are using commit-queues, then that is asynchronous. Sender will send the request, the receiver will queue it and ACK the acceptance of the request. The Sender will have to then check back again to see if the request has been successfully completed.

 

 

From what I understand, multiple requests in the commit-queue can be acted upon in parallel.

 

 

Thanks,

 

  1. Bilal.

 

khgrant
Cisco Employee
Cisco Employee

 

Hi Bilal,

 

 

Please see inline.

 

 

Thanks!

 

 

From: Bilal Alam (balam) Sent: Thursday, February 11, 2016 2:08 PM To: Michel Papiashvili (pamichel) <pamichel@cisco.com>; service_orchestration(mailer list) <service_orchestration@cisco.com> Subject: RE: Parallel commits through REST

                         

khgrant
Cisco Employee
Cisco Employee

 

Hi,

 

 

> Michel: in practice, I noticed the rest call is not async

 

The specific REST call is synchronous but the way the desired outcome (programming the SB devices) is achieved via commit-queues is asynchronous.

 

 

> and returns when the commit is finished,

 

The REST call returns when the request has been successfully queued. The assumption that commit is finished is not correct for the commit-queue case.

 

 

> so measuring the time it takes curl command to complete (using -w %{time_total}) will tell us approx.

 

> how long it took NSO to commit the transaction.

 

When using commit-queues then this just measures how long it took for the request to be successfully queued.

 

 

Thanks,

 

  1. Bilal.

 

khgrant
Cisco Employee
Cisco Employee

 

Michel,

 

 

I good way to see what is going on in regards to transactions is pretty verbosely logged in devel.log.

 

Perhaps you can get an idea of what is happening here.

 

 

-Larry

 

 

Example:

 

<DEBUG> 11-Feb-2016::17:57:53.359 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148: entering validate phase for running...
<DEBUG> 11-Feb-2016::17:57:53.359 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   validate: grabbing transaction lock...
<DEBUG> 11-Feb-2016::17:57:53.359 LMANOR-M-GB6B ncs[40244]: devel-c new_trans request daemon id: 1 thandle: 8148
<DEBUG> 11-Feb-2016::17:57:53.360 LMANOR-M-GB6B ncs[40244]: devel-c new_trans succeeded daemon id: 1 session id: 8148 worker id: 1
<DEBUG> 11-Feb-2016::17:57:53.360 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:  ok
<DEBUG> 11-Feb-2016::17:57:53.360 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   validate: creating rollback file...
<DEBUG> 11-Feb-2016::17:57:53.367 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:  ok
<DEBUG> 11-Feb-2016::17:57:53.367 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   validate: run transforms and transaction hooks...
<DEBUG> 11-Feb-2016::17:57:53.368 LMANOR-M-GB6B ncs[40244]: devel-c write_all request for callpoint 'ncs-rfs-service-hook' path
<DEBUG> 11-Feb-2016::17:57:53.370 LMANOR-M-GB6B ncs[40244]: devel-c new_trans request daemon id: 818 thandle: 8148
<DEBUG> 11-Feb-2016::17:57:53.370 LMANOR-M-GB6B ncs[40244]: devel-c new_trans succeeded daemon id: 818 session id: 309 worker id: 2
<DEBUG> 11-Feb-2016::17:57:53.370 LMANOR-M-GB6B ncs[40244]: devel-c service_create request for callpoint 'customer-dns-servicepoint' path /ncs:services/customer-dns{Cust-1 asr1k0}
<DEBUG> 11-Feb-2016::17:57:53.386 LMANOR-M-GB6B ncs[40244]: devel-c service_create succeeded for callpoint 'customer-dns-servicepoint' path /ncs:services/customer-dns{Cust-1 asr1k0}
<DEBUG> 11-Feb-2016::17:57:53.391 LMANOR-M-GB6B ncs[40244]: devel-c write_all succeeded for callpoint 'ncs-rfs-service-hook' path
<DEBUG> 11-Feb-2016::17:57:53.392 LMANOR-M-GB6B ncs[40244]: devel-c write_all request for callpoint 'ncs-rfs-service-hook' path
<DEBUG> 11-Feb-2016::17:57:53.392 LMANOR-M-GB6B ncs[40244]: devel-c write_all succeeded for callpoint 'ncs-rfs-service-hook' path
<DEBUG> 11-Feb-2016::17:57:53.393 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:  ok
<DEBUG> 11-Feb-2016::17:57:53.393 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   validate: mark inactive...
<DEBUG> 11-Feb-2016::17:57:53.393 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:  ok
<DEBUG> 11-Feb-2016::17:57:53.393 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   validate: pre validate...
<DEBUG> 11-Feb-2016::17:57:53.394 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:  ok
<DEBUG> 11-Feb-2016::17:57:53.394 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   validate: run validation over the change set...
<DEBUG> 11-Feb-2016::17:57:53.395 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   validate: validation over the change set done
<DEBUG> 11-Feb-2016::17:57:53.395 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   validate: run dependency-triggered validation...
<DEBUG> 11-Feb-2016::17:57:53.395 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   validate: dependency-triggered validation done
<DEBUG> 11-Feb-2016::17:57:53.395 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   validate: check configuration policies...
<DEBUG> 11-Feb-2016::17:57:53.395 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   validate: configuration policies done
<DEBUG> 11-Feb-2016::17:57:53.395 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148: entering write phase for running...
<DEBUG> 11-Feb-2016::17:57:53.395 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   cdb: write_start
<DEBUG> 11-Feb-2016::17:57:53.397 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148: entering prepare phase for running...
<DEBUG> 11-Feb-2016::17:57:53.397 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   cdb: prepare
<DEBUG> 11-Feb-2016::17:57:53.397 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   cdb: delivering prepare subscription notifications at prio 0
<INFO> 11-Feb-2016::17:57:53.399 LMANOR-M-GB6B ncs[40244]: ncs connecting NED asr1k0
<DEBUG> 11-Feb-2016::17:57:53.399 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   ncs: connecting NED asr1k0
<INFO> 11-Feb-2016::17:57:53.606 LMANOR-M-GB6B ncs[40244]: ncs calculating southbound diffs
<DEBUG> 11-Feb-2016::17:57:53.606 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   ncs: calculating southbound diffs
<INFO> 11-Feb-2016::17:57:53.864 LMANOR-M-GB6B ncs[40244]: ncs device asr1k0: send NED prepare
<DEBUG> 11-Feb-2016::17:57:53.864 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   ncs: device asr1k0: send NED prepare
<INFO> 11-Feb-2016::17:57:54.100 LMANOR-M-GB6B ncs[40244]: ncs device asr1k0: send NED commit
<DEBUG> 11-Feb-2016::17:57:54.100 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   ncs: device asr1k0: send NED commit
<DEBUG> 11-Feb-2016::17:57:54.106 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   cdb: all prepare subscription notifications acknowledged
<DEBUG> 11-Feb-2016::17:57:54.106 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148: entering commit phase for running...
<DEBUG> 11-Feb-2016::17:57:54.106 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   cdb: commit
<DEBUG> 11-Feb-2016::17:57:54.108 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   cdb: delivering commit subscription notifications at prio 0
<INFO> 11-Feb-2016::17:57:54.108 LMANOR-M-GB6B ncs[40244]: ncs device asr1k0: send NED persist
<DEBUG> 11-Feb-2016::17:57:54.108 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   ncs: device asr1k0: send NED persist
<DEBUG> 11-Feb-2016::17:57:54.293 LMANOR-M-GB6B ncs[40244]: ncs commit progress db=running usid=398 thandle=8148:   cdb: all commit subscription notifications acknowledged
<DEBUG> 11-Feb-2016::17:57:54.294 LMANOR-M-GB6B ncs[40244]: devel-c close_trans request daemon id: 818 session id: 309
<DEBUG> 11-Feb-2016::17:57:54.294 LMANOR-M-GB6B ncs[40244]: devel-c close_trans succeeded daemon id: 818 session id: 309
<DEBUG> 11-Feb-2016::17:57:54.294 LMANOR-M-GB6B ncs[40244]: devel-c close_trans request daemon id: 1 session id: 8148
<DEBUG> 11-Feb-2016::17:57:54.294 LMANOR-M-GB6B ncs[40244]: devel-c close_trans succeeded daemon id: 1 session id: 8148

khgrant
Cisco Employee
Cisco Employee

 

Hi,

 

first I paraphrase your question to check I understand it correctly.

 

 

Scenario 1:

 

You have a service manipulating data on one device.

 

You instantiate 10/20/50 services, each service instance affects the same device.

 

 

Scenario 2:

 

You have a service manipulating data on one device.

 

You instantiate 10/20/50 services, each service instance affects randomly one of two devices.

 

Each service still only change one device.

 

 

If I have understood you correctly your measurements are the expected ones.

 

 

For Scenario 1:

 

NSO is serial when it comes to services. The commits to the devices are however done in parallel, the slowest device affects thus the overall time. If one service affects

 

1/2/10 devices you should just detect minimal differences.

 

 

For Scenario 2:

 

If the services just affect different devices no differences should be measured.

 

 

/Dag

 

khgrant
Cisco Employee
Cisco Employee

 

Understood Dag, thank you!

 

So it only runs parallel when the configuration is sent multiple devices within the same service-create transaction. So in that case, if the service creation is the bottleneck, will aggregating multiple service-create requests into a single API/transaction call improve the overall performance in terms of services created/hour?

 

Is there any other (or better) way we can scale the rate of services/hour?

   

Thanks! 

Michel Papiashvili

 

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: