Hello folks, I've got an issue with a new Catalyst 9300 deployment for a customer. We've moved routing from a dedicated router to the 9300 and the 9300 has taken the QoS for the site. We're using standard Auto QoS, trusting DSCP, but I'm seeing significant output drops on the WAN bound interface. Can anyone help shed some light on to what is going on? The stats don't seem to add up correctly on the port-map. The WAN providers interface is nailed to 100/Full and the Auto QoS is set to run in percentages of the interface and has dutifully set it's figures to a 100 Mbps reference bandwidth. I've upgraded the switches from 16.09.01 to 16.12.01, but that didn't clear the problem. I can't quite understand from the show policy-map what the drops in the priority-queue 1 are from, as it doesn't seem to be matching anything and there's no traffic on the site marked that should hit the priority-queue. Here is a show interface:- GigabitEthernet1/0/1 is up, line protocol is up (connected) Hardware is Gigabit Ethernet, address is c4b3.6a05.7c81 (bia c4b3.6a05.7c81) Description: Network::WAN MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec, reliability 255/255, txload 2/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Carrier delay is 0 msec Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX input flow-control is on, output flow-control is unsupported ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:01, output 00:00:08, output hang never Last clearing of "show interface" counters never Input queue: 0/200000/0/0 (size/max/drops/flushes); Total output drops: 35615 Queueing strategy: Class-based queueing Output queue: 0/200000 (size/max) 30 second input rate 34000 bits/sec, 34 packets/sec 30 second output rate 966000 bits/sec, 102 packets/sec 120258 packets input, 66898060 bytes, 0 no buffer Received 2703 broadcasts (2702 multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 2702 multicast, 0 pause input 0 input packets with dribble condition detected 285906 packets output, 310294634 bytes, 0 underruns 0 output errors, 0 collisions, 2 interface resets 0 unknown protocol drops 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier, 0 pause output 0 output buffer failures, 0 output buffers swapped out Show policy-map int g1/0/1:- GigabitEthernet1/0/1 Service-policy input: AutoQos-4.0-Trust-Dscp-Input-Policy Class-map: class-default (match-any) 291057 packets Match: any QoS Set dscp dscp table AutoQos-4.0-Trust-Dscp-Table Service-policy output: AutoQos-4.0-Output-Policy queue stats for all priority classes: Queueing priority level 1 (total drops) 50316915 (bytes output) 52157956 Class-map: AutoQos-4.0-Output-Priority-Queue (match-any) 0 packets Match: dscp cs4 (32) cs5 (40) ef (46) Match: cos 5 Priority: 30% (30000 kbps), burst bytes 750000, Priority Level: 1 Class-map: AutoQos-4.0-Output-Control-Mgmt-Queue (match-any) 0 packets Match: dscp cs2 (16) cs3 (24) cs6 (48) cs7 (56) Match: cos 3 Queueing queue-limit dscp 16 percent 80 queue-limit dscp 24 percent 90 queue-limit dscp 48 percent 100 queue-limit dscp 56 percent 100 (total drops) 0 (bytes output) 915388 bandwidth remaining 10% queue-buffers ratio 10 Class-map: AutoQos-4.0-Output-Multimedia-Conf-Queue (match-any) 0 packets Match: dscp af41 (34) af42 (36) af43 (38) Match: cos 4 Queueing (total drops) 0 (bytes output) 303 bandwidth remaining 10% queue-buffers ratio 10 Class-map: AutoQos-4.0-Output-Trans-Data-Queue (match-any) 0 packets Match: dscp af21 (18) af22 (20) af23 (22) Match: cos 2 Queueing (total drops) 0 (bytes output) 0 bandwidth remaining 10% queue-buffers ratio 10 Class-map: AutoQos-4.0-Output-Bulk-Data-Queue (match-any) 0 packets Match: dscp af11 (10) af12 (12) af13 (14) Match: cos 1 Queueing (total drops) 0 (bytes output) 0 bandwidth remaining 4% queue-buffers ratio 10 Class-map: AutoQos-4.0-Output-Scavenger-Queue (match-any) 0 packets Match: dscp cs1 (8) Queueing (total drops) 0 (bytes output) 0 bandwidth remaining 1% queue-buffers ratio 10 Class-map: AutoQos-4.0-Output-Multimedia-Strm-Queue (match-any) 0 packets Match: dscp af31 (26) af32 (28) af33 (30) Queueing (total drops) 0 (bytes output) 0 bandwidth remaining 10% queue-buffers ratio 10 Class-map: class-default (match-any) 0 packets Match: any Queueing (total drops) 0 (bytes output) 259417210 bandwidth remaining 25% queue-buffers ratio 25 Show run int g1/0/1:- interface GigabitEthernet1/0/1 description Network::WAN switchport access vlan 900 switchport trunk native vlan 999 switchport mode access switchport nonegotiate bandwidth 100000 bandwidth qos-reference 100000 ip flow monitor FNF-monitor-SolarWinds-in input ip flow monitor FNF-monitor-SolarWinds-out output ip arp inspection trust load-interval 30 carrier-delay msec 0 speed 100 duplex full snmp ifindex persist storm-control broadcast level 50.00 storm-control multicast level 50.00 storm-control action trap auto qos trust dscp spanning-tree portfast trunk service-policy input AutoQos-4.0-Trust-Dscp-Input-Policy service-policy output AutoQos-4.0-Output-Policy hold-queue 200000 in hold-queue 200000 out ip dhcp snooping trust end Any insight or thoughts, greatly appreciated. Best, Leigh
... View more
Hello all, Quick update on this one. We've managed to fix the issue, but it appears to be a bug. The problem arises when the ASCII Display (Caller ID) for the phone line starts with a "0" (Zero). Any other digit and it's fine.. For reference, we're running version 22.214.171.12400-11, it affected all of the phone types we tested. Best, Leigh
... View more
Hi there, Thanks for responding, it's much appreciated. Here is a shot of the packet capture of the INVITE that gets the Bad Request back:- The INVITE is sent when I try to dial an external number. Best, Leigh
... View more
In the end, we put in several IP addresses in the ACL on the WLC (which we got from a traffic capture - I don't have access to the WLC's any more) and on the FirePOWER egress firewall, we put in a URL group for DigiCert.
After all that, it still doesn't quite work 100% of the time, as various models of Android devices looked for different paths and some simply won't trust the intermediary server.
... View more
Thanks for all of your help with this, I've managed to get it sorted now. After lots of testing, I was at a point where the RightFax server was sending what seemed like fax transmission into CUCM, but it was never going out of the gateway.
Eventually, I built a new set of SIP trunks into the gateways going to Gamma on port 5066, changed all of the legs to "No Preference" and it worked first time!
Hope that helps someone else in the same situation.
... View more
03.16.06b.S / 15.5(3)S6b We've had some odd issues with the boxes to do with DTMF tones being sent through Lync, other than that, pretty well behaved we believe. Though, it is a big system with 20,000 users connected and many faults don't always get through to us
... View more
Ok, so I've set hardware MTP not both trunks and both ends are set to RFC-2833. I can't alter it to no-preference as it's in there supporting a lot of other systems.
I think I've managed to grab a failed call on the IOS MTP router and there's a funny result in there (I think). The call for the MTP (if it is indeed the call, there's 20,000 devices in the system and there was a lot of lines in the debug to wade through).
Here's the bit I think is the failed MTP attempt:-
Jun 19 10:34:04.867: SCCP:rcvd OpenReceiveChannel Jun 19 10:34:04.867: OpenReceiveChannelMsg Info: conference_id = 69263086, pass_through_party_id = 50769958 msec_pkt_size = 20, compression_type = 2 qualifier_in.ecvalue = 0, g723_bitrate = 0, call_ref = 58554818 stream_pass_through_id = 16777216, rfc2833_payload_type = 0 codec_dynamic_payload = 0, codec_mode = 0 Encryption Info :: algorithm_id 0, key_len 0, salt_len 0 requestedAddrType = 0, source_ip_addr.ipAddrType = 0, source_ip_addr = 10.156.102.36, source_port_number = 4000, audio_level_adjustment = 0 Jun 19 10:34:04.867: sym_xapp_search_for_conn_cb: sess_cb 7FEA8EB1D2A0, sess_id 69263086, conn_id 50769958 Jun 19 10:34:04.867: sym_xapp_create_conn_if_ok: creating new conn_cb for sess_id 69263086, conn_id 50769958 Jun 19 10:34:04.867: sym_xapp_create_and_init_conn_cb: glob_ccm->version 9 Jun 19 10:34:04.867: sccp_extract_and_validate_srtp_context Jun 19 10:34:04.867: sym_xapp_find_peer_conn_cb: find possible peer_conn_cb for cur conn_cb 0x7FEA8F078238: sess_id 69263086 conn_id 50769958 conn_id_tx 0 stream_passthr_id 16777216 call_ref 58554818 peer_conn_cb 0x7FEA8EA1C9F0: sess_id 69263086 conn_id 50769948 conn_id_tx 0 stream_passthr_id 16777216 call_ref 58554818, conn_mode 1 rtp_clr 0x7FEA869C61B0 Jun 19 10:34:04.867: sym_xapp_find_peer_conn_cb: find possible peer_conn_cb for cur conn_cb 0x7FEA8F078238: sess_id 69263086 conn_id 50769958 conn_id_tx 0 stream_passthr_id 16777216 call_ref 58554818 peer_conn_cb 0x7FEA8F077668: sess_id 69263086 conn_id 50769946 conn_id_tx 0 stream_passthr_id 16777216 call_ref 58554818, conn_mode 4 rtp_clr 0x7FEA8F79D9B0 Jun 19 10:34:04.867: sym_xapp_find_peer_conn_cb: found peer_conn_cb, rtp_clr state 4 Jun 19 10:34:04.868: sccp_get_local_address_by_idb: get_physical_ip:1, use IP addr 10.156.102.85 Jun 19 10:34:04.868: sym_xapp_open_receive_chnl agreed_sccp_ver=18, dtmf_method=10, rx_PT=0 conn_info->local_ipaddr=10.156.102.85 conn_info->remote_ipaddr=0.0.0.0 Jun 19 10:34:04.868: sym_xapp_open_receive_chnl: appl_type 2, sess_id 69263086, conn_id 50769958, codec 2, pkt_period 20, sess_type 2 mixed_mode 0, mm_codec 0, mm_pkt_period 0, stream_passthr_id 16777216, rcv_dtmf_payload_type 101, snd_dtmf_payload_type 101, codec_mode 0, rx_dynamic_pt 0, tx_dynamic_pt 0, amr_bitmap 0, call_ref 58554818, profile_id 1, from_dfr_list 0, need_to_deffer 1 Jun 19 10:34:04.868: Enterning sym_xapp_trigger_fsm_or_deffer_event: conn_cb 0x7FEA8F078238 Jun 19 10:34:04.868: sym_xapp_defer_conn_setup_req: setup conn req for sess_id 69263086, conn_id 50769958, conn_id_tx 0, stream_passthr 16777216, call_ref 58554818 is deferred Jun 19 10:34:04.868: sym_xapp_trigger_fsm_or_deffer_event: conn setup req defered for sess_id 69263086, conn_id 50769958, conn_id_tx 0, stream_passthr 16777216, call_ref 58554818
10.156.102.36 is the CUCM server and 10.156.102.85 is the SBC.
Points of note that are different from the other calls is the line "need_to_deffer 1" and then the setup request look to progress through to saying that the " stream_passthr 16777216, call_ref 58554818 is deferred ".
Is that the IOS gateway kicking back the request for a pass-through? I can't find any information anywhere as to what "need_to_defer" means, but I presume it means it's going to defer/kick the request.
I did put the SDL trace in this thread earlier, if you need any more than that, I can pop some more info in too.
Many thanks in advance,
... View more
Many thanks for your reply. I've gone through the configuration of the setup and I've built the SIP trunk down to the RightFax server to have the CM server they're registered with as their primary MTP in the MRGL, the main inbound/outbound SIP trunk is set to have the hardware ISR's as their primary choice.
I've confirmed the CUCM has "Enable Passthrough" set (it was already set) On the IOS MTP, I've got:-
voice service voip
allow-connections sip to sip
fax protocol t38 version 0 ls-redundancy 0 hs-redundancy 0 fallback none
dpsfarm profile 10 mtp
description DSP MTP Resources
maximum sessions hardware 100
associate application SCCP
Again, this was as it's previously been configured - I've not altered anything.
The main inbound/outbound SIP trunk has it's DTMF Signalling set to OOB and RFC2833 (I believe it was to do with an issue getting DTMF to work correctly in Lync), os I've set the RightFax trunk to be the same.
However, I still get told the MTP resource doesn't support pass-thru in the trace. Any thoughts on this? Any idea where I look to see which resource is being asked to be MTP for the DTMF, so I can see why it's not offering the service?
... View more
Thanks for the response. I've got the trace and gone through it. The timings weren't an exact match, but I've found the conversation.
In the attached file, on line 170, the invite arrives asking for T38, on line 224, the Trying response goes back. Then, I think I see the root cause of the issue, on line 248, there is a message saying there is no in band DTMF, so an MTP is invoked on line 249.
On line 309, the server gets a message from the MTP saying it doesn't support pass thru fax.
Do you agree?
What's the best course of action here? Reconfigure the MTP to support pass thru? Alter the order of the MTP servers so there's a CM in there offering the service, rather than the ISR that's currently running the MTP?
... View more
We're trying to configure a SIP trunk to a RightFax server and are having difficulty in getting it working.
Setup wise, we have:-
Gamma (SIP provider) - sip trunk - Sonus SBC - sip trunk - CUCM - sip trunk - RightFax Server
From what we can see, the configuration looks fine, indeed, we have a GoldFax server too which runs more than happily. We've set the SIP trunk down to the RightFax server to be identical to the working GoldFax server.
We've run a Wireshare trace on the RightFax Server and the issue seems to arise when the RightFax server wants to run T38. The trace is here:-
It seems as though the call progresses nicely through normal trying and ringing, but then, when the server wants to talk FoIP using T38, the Call Manager says it's trying and then kicks back a 488 Not Acceptable Media.
I presume the Call Manager is checking with the gateway to see if they can take T38, but gets a "no" back? As I've said, we have another fax server running quite nicely with no problems. We've got the SIP Trunk Security profile set to UDP for outgoing (which is a RightFax requirement). The SIP profile is reatively standard, except we have Outgoing T.38 INVITE include audio mline checked.
Has anyone had this sort of setup before? Can you shed any light on our issue?
... View more
We're looking to run reports based on guest usage from ISE and tally their data usage and roaming between AP's from Prime. We can get the data out of ISE with no problems, but we want to access the Prime database directly to call the usage data. It's in there and we can export it to csv with no problems, but we want to be able to automate it as quickly as possible, as there is a lot of data to get through.
... View more