I have been running some QoS tests lately and wanted to share some of my results. Some of this behavior is described in various documentation guides but it’s not really clearly described in one place. I’ll describe what I have found so far in this post.
QoS is only active during congestion. This is well known but it’s not as well known how congestion is detected. the TX ring is used to hold packets before they get transmitted out on an interface. This is a hardware FIFO queue and when the queue gets filled, the interface is congested. When buying a subrate circuit from a SP, something must be added to achieve the backpressure so that the TX ring is considered full. This is done by applying a parent shaper and a child policy with the actual queue configuration.
The LLQ is used for high priority traffic. When the interface is not congested, the LLQ can use all available bandwidth unless an explicit policer is configured under the LLQ.
A normal queue can use more bandwidth than it is guaranteed when there is no congestion.
When a normal queue wants to use more bandwidth than its guaranteed, it can if there is bandwidth available from other classes not using their full amount of bandwidth.
If multiple queues want to use more bandwidth than they are guaranteed and there is bandwidth available from other classes not using their full amount, they can. The amount of “free” bandwidth seems to be distributed quite evenly although I don’t think there is any guarantee that this is always the case.
When there is congestion, the LLQ does not get assigned additional bandwidth even if there is bandwidth available from any of the normal classes. The only time the LLQ can use more bandwidth is if there is no congestion.
I hope this post gives you some insight on how LLQ and CBWFQ behaves on Cisco devices.
Update:
Some of the readers asked if I could provide the CLI output as well which I am doing below. My setup was two CSR1000v routers and two Ubuntu hosts running Iperf3.
The first test is to test the maximum bandwidth without any QoS applied and sending TCP traffic.
daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 Connecting to host x.x.x.x, port 5001 [ 4] local x.x.x.x port 38941 connected to x.x.x.x port 5001 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 124 MBytes 1036 Mbits/sec 6 837 KBytes [ 4] 1.00-2.00 sec 119 MBytes 996 Mbits/sec 0 940 KBytes [ 4] 2.00-3.00 sec 120 MBytes 1007 Mbits/sec 12 571 KBytes [ 4] 3.00-4.00 sec 119 MBytes 996 Mbits/sec 0 714 KBytes [ 4] 4.00-5.00 sec 119 MBytes 997 Mbits/sec 0 831 KBytes [ 4] 5.00-6.00 sec 120 MBytes 1007 Mbits/sec 6 673 KBytes [ 4] 6.00-7.01 sec 120 MBytes 1001 Mbits/sec 0 800 KBytes [ 4] 7.01-8.00 sec 118 MBytes 991 Mbits/sec 5 650 KBytes [ 4] 8.00-9.00 sec 120 MBytes 1006 Mbits/sec 0 778 KBytes [ 4] 9.00-10.00 sec 119 MBytes 997 Mbits/sec 0 885 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 1.17 GBytes 1003 Mbits/sec 29 sender [ 4] 0.00-10.00 sec 1.17 GBytes 1001 Mbits/sec receiver
Without any shaping the routers were able to use the full bandwidth available. The next test uses UDP.
daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -u -b 1000m Connecting to host x.x.x.x, port 5001 [ 4] local x.x.x.x port 57585 connected to x.x.x.x port 5001 [ ID] Interval Transfer Bandwidth Total Datagrams [ 4] 0.00-1.00 sec 111 MBytes 928 Mbits/sec 14168 [ 4] 1.00-2.00 sec 124 MBytes 1043 Mbits/sec 15923 [ 4] 2.00-3.00 sec 115 MBytes 961 Mbits/sec 14660 [ 4] 3.00-4.00 sec 118 MBytes 991 Mbits/sec 15117 [ 4] 4.00-5.00 sec 127 MBytes 1062 Mbits/sec 16209 [ 4] 5.00-6.00 sec 94.5 MBytes 792 Mbits/sec 12095 [ 4] 6.00-7.00 sec 132 MBytes 1109 Mbits/sec 16913 [ 4] 7.00-8.00 sec 125 MBytes 1047 Mbits/sec 15970 [ 4] 8.00-9.00 sec 114 MBytes 955 Mbits/sec 14572 [ 4] 9.00-10.00 sec 131 MBytes 1099 Mbits/sec 16774 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 0.00-10.00 sec 1.16 GBytes 999 Mbits/sec 0.070 ms 31413/152388 (21%) [ 4] Sent 152388 datagrams
UDP could utilize the full bandwidth as well but only after I had tweaked some buffer settings in Linux. With the default buffer size I could only achieve around 500 Mbit/s. UDP is more reliant on buffer size since it has no window scaling like TCP.
I then applied the following QoS configuration.
class-map match-all CM1 match access-group name ACL1 class-map match-all CM2 match access-group name ACL2 class-map match-all CM3 match access-group name ACL3 class-map match-all CM4 match access-group name ACL4 class-map match-all CM5 match access-group name ACL5 class-map match-all CM6 match access-group name ACL6 class-map match-all CM7 match access-group name ACL7 class-map match-all CM8 match access-group name ACL8 ! policy-map CHILD class CM1 priority 7000 class CM2 bandwidth 5000 class CM3 bandwidth 5000 class CM4 bandwidth 5000 class CM5 bandwidth 15000 class CM6 bandwidth 15000 class CM7 bandwidth 15000 class CM8 bandwidth 15000 ! policy-map SHAPE90M class class-default shape average 90000000 service-policy CHILD ! interface GigabitEthernet0/2 service-policy output SHAPE90M
After applying the policy, the priority class can use the full bandwidth 90Mbit/s. The first test uses TCP.
daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -i 10 -t 120
Connecting to host x.x.x.x, port 5001
[ 4] local x.x.x.x port 38949 connected to x.x.x.x port 5001
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-10.00 sec 106 MBytes 88.6 Mbits/sec 33 624 KBytes
[ 4] 10.00-20.00 sec 102 MBytes 86.0 Mbits/sec 3 706 KBytes
[ 4] 20.00-30.00 sec 102 MBytes 86.0 Mbits/sec 3 631 KBytes
[ 4] 30.00-40.00 sec 102 MBytes 86.0 Mbits/sec 6 556 KBytes
[ 4] 40.00-50.00 sec 102 MBytes 86.0 Mbits/sec 0 697 KBytes
[ 4] 50.00-60.00 sec 102 MBytes 86.0 Mbits/sec 6 617 KBytes
[ 4] 60.00-70.00 sec 101 MBytes 84.9 Mbits/sec 6 667 KBytes
[ 4] 70.00-80.00 sec 100 MBytes 83.9 Mbits/sec 37 591 KBytes
[ 4] 80.00-90.00 sec 101 MBytes 84.9 Mbits/sec 3 624 KBytes
[ 4] 90.00-100.00 sec 102 MBytes 86.0 Mbits/sec 9 540 KBytes
[ 4] 100.00-110.00 sec 102 MBytes 86.0 Mbits/sec 0 680 KBytes
[ 4] 110.00-120.00 sec 102 MBytes 86.0 Mbits/sec 6 612 KBytes
– – – – – – – – – – – – – – – – – – – – – – – – –
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-120.00 sec 1.20 GBytes 85.9 Mbits/sec 112 sender
[ 4] 0.00-120.00 sec 1.20 GBytes 85.6 Mbits/sec receiver
This is the output from the CSR router.
R1#sh policy-map interface gi2.200
GigabitEthernet2.200
Service-policy output: SHAPE90M
Class-map: class-default (match-any)
812652 packets, 1233588696 bytes
30 second offered rate 88625000 bps, drop rate 9000 bps
Match: any
Queueing
queue limit 937 packets
(queue depth/total drops/no-buffer drops) 0/105/0
(pkts output/bytes output) 812550/1233450900
shape (average) cir 90000000, bc 360000, be 360000
target shape rate 90000000
Service-policy : CHILD
queue stats for all priority classes:
Queueing
queue limit 512 packets
(queue depth/total drops/no-buffer drops) 490/105/0
(pkts output/bytes output) 812550/1233450900
Class-map: CM1 (match-all)
812655 packets, 1233610290 bytes
30 second offered rate 88627000 bps, drop rate 8000 bps
Match: access-group name ACL1
Priority: 7000 kbps, burst bytes 175000, b/w exceed drops: 105
The next test is then performed with UDP. Observe that there is some packet loss since UDP does not have acknowledgements.
daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -u -b 90m -i 10 -t 120 Connecting to host x.x.x.x, port 5001 [ 4] local x.x.x.x port 35311 connected to x.x.x.x port 5001 [ ID] Interval Transfer Bandwidth Total Datagrams [ 4] 0.00-10.00 sec 106 MBytes 89.2 Mbits/sec 13604 [ 4] 10.00-20.00 sec 107 MBytes 90.0 Mbits/sec 13734 [ 4] 20.00-30.00 sec 107 MBytes 90.0 Mbits/sec 13732 [ 4] 30.00-40.00 sec 107 MBytes 90.0 Mbits/sec 13733 [ 4] 40.00-50.00 sec 107 MBytes 90.0 Mbits/sec 13733 [ 4] 50.00-60.00 sec 107 MBytes 90.0 Mbits/sec 13734 [ 4] 60.00-70.00 sec 107 MBytes 90.0 Mbits/sec 13733 [ 4] 70.00-80.00 sec 107 MBytes 90.0 Mbits/sec 13734 [ 4] 80.00-90.00 sec 107 MBytes 90.0 Mbits/sec 13731 [ 4] 90.00-100.00 sec 107 MBytes 90.0 Mbits/sec 13734 [ 4] 100.00-110.00 sec 107 MBytes 90.0 Mbits/sec 13737 [ 4] 110.00-120.00 sec 107 MBytes 90.0 Mbits/sec 13729 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 0.00-120.00 sec 1.26 GBytes 89.9 Mbits/sec 0.727 ms 8544/164668 (5.2%) [ 4] Sent 164668 datagrams
R1#sh policy-map interface gi2.200
GigabitEthernet2.200
Service-policy output: SHAPE90M
Class-map: class-default (match-any)
1720938 packets, 2517232191 bytes
30 second offered rate 92407000 bps, drop rate 2099000 bps
Match: any
Queueing
queue limit 937 packets
(queue depth/total drops/no-buffer drops) 0/24449/0
(pkts output/bytes output) 1696469/2482966749
shape (average) cir 90000000, bc 360000, be 360000
target shape rate 90000000
Service-policy : CHILD
queue stats for all priority classes:
Queueing
queue limit 512 packets
(queue depth/total drops/no-buffer drops) 240/24449/0
(pkts output/bytes output) 1696469/2482966749
Class-map: CM1 (match-all)
1720918 packets, 2517251531 bytes
30 second offered rate 92407000 bps, drop rate 2702000 bps
Match: access-group name ACL1
Priority: 7000 kbps, burst bytes 175000, b/w exceed drops: 24449
The next test sends 90Mbit/s UDP in the priority class and 90Mbit/s TCP in another class.
daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -i 10 -t 120 Connecting to host x.x.x.x, port 5001 [ 4] local x.x.x.x port 51066 connected to x.x.x.x port 5001 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-10.00 sec 94.5 MBytes 79.3 Mbits/sec 157 76.4 KBytes [ 4] 10.00-20.00 sec 93.0 MBytes 78.0 Mbits/sec 145 77.8 KBytes [ 4] 20.00-30.00 sec 93.5 MBytes 78.5 Mbits/sec 153 102 KBytes [ 4] 30.00-40.00 sec 91.2 MBytes 76.5 Mbits/sec 172 87.7 KBytes [ 4] 40.00-50.00 sec 93.5 MBytes 78.4 Mbits/sec 132 77.8 KBytes [ 4] 50.00-60.00 sec 93.8 MBytes 78.7 Mbits/sec 117 91.9 KBytes [ 4] 60.00-70.00 sec 87.2 MBytes 73.2 Mbits/sec 101 103 KBytes [ 4] 70.00-80.00 sec 91.7 MBytes 76.9 Mbits/sec 151 102 KBytes [ 4] 80.00-90.00 sec 93.4 MBytes 78.3 Mbits/sec 181 100 KBytes [ 4] 90.00-100.00 sec 93.4 MBytes 78.3 Mbits/sec 144 91.9 KBytes [ 4] 100.00-110.00 sec 93.2 MBytes 78.2 Mbits/sec 125 74.9 KBytes [ 4] 110.00-120.00 sec 95.9 MBytes 80.4 Mbits/sec 128 79.2 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-120.00 sec 1.09 GBytes 77.9 Mbits/sec 1706 sender [ 4] 0.00-120.00 sec 1.09 GBytes 77.9 Mbits/sec receiver
The TCP traffic got 78Mbit/s which was everything that was available after the priority class got its share.
Class-map: CM2 (match-all)
1522624 packets, 2311298853 bytes
30 second offered rate 81086000 bps, drop rate 156000 bps
Match: access-group name ACL2
Queueing
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 61/2926/0
(pkts output/bytes output) 1519698/2306857185
bandwidth 5000 kbps
The priority class got around 8Mbit/s. Iperf is sending at 90Mbit/s but notice the 91% packet loss.
daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -u -b 90m -i 10 -t 120 Connecting to host x.x.x.x, port 5001 [ 4] local x.x.x.x port 56800 connected to x.x.x.x port 5001 [ ID] Interval Transfer Bandwidth Total Datagrams [ 4] 0.00-10.00 sec 106 MBytes 89.2 Mbits/sec 13604 [ 4] 10.00-20.00 sec 107 MBytes 90.0 Mbits/sec 13735 [ 4] 20.00-30.00 sec 107 MBytes 90.0 Mbits/sec 13731 [ 4] 30.00-40.00 sec 107 MBytes 90.0 Mbits/sec 13734 [ 4] 40.00-50.00 sec 107 MBytes 90.0 Mbits/sec 13733 [ 4] 50.00-60.00 sec 107 MBytes 90.0 Mbits/sec 13731 [ 4] 60.00-70.00 sec 107 MBytes 90.0 Mbits/sec 13736 [ 4] 70.00-80.00 sec 108 MBytes 90.2 Mbits/sec 13763 [ 4] 80.00-90.00 sec 107 MBytes 89.8 Mbits/sec 13700 [ 4] 90.00-100.00 sec 107 MBytes 90.0 Mbits/sec 13736 [ 4] 100.00-110.00 sec 107 MBytes 90.0 Mbits/sec 13733 [ 4] 110.00-120.00 sec 107 MBytes 90.0 Mbits/sec 13733 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 0.00-120.00 sec 1.26 GBytes 89.9 Mbits/sec 0.660 ms 149472/164540 (91%) [ 4] Sent 164540 datagrams
The CSR router shows a high drop drate. To get the actual rate take the offered rate minus the drop rate.
Class-map: CM1 (match-all)
3869818 packets, 5535668355 bytes
30 second offered rate 91647000 bps, drop rate 83689000 bps
Match: access-group name ACL1
Priority: 7000 kbps, burst bytes 175000, b/w exceed drops: 1719101
The final test is sending 90 Mbit/s in all eight classes. This means that each class should only get its alotted bandwidth.
The priority class gets 7Mbit/s.
Class-map: CM1 (match-all)
4954923 packets, 7059843887 bytes
30 second offered rate 91638000 bps, drop rate 84681000 bps
Match: access-group name ACL1
Priority: 7000 kbps, burst bytes 175000, b/w exceed drops: 2712171
Then we have three classes getting 5Mbit/s.
daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -i 10 -t 120 Connecting to host x.x.x.x, port 5001 [ 4] local x.x.x.x port 34496 connected to x.x.x.x port 5001 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-10.00 sec 18.1 MBytes 15.2 Mbits/sec 33 77.8 KBytes [ 4] 10.00-20.00 sec 6.96 MBytes 5.84 Mbits/sec 8 73.5 KBytes [ 4] 20.00-30.00 sec 6.84 MBytes 5.73 Mbits/sec 8 72.1 KBytes [ 4] 30.00-40.00 sec 7.15 MBytes 5.99 Mbits/sec 8 66.5 KBytes [ 4] 40.00-50.00 sec 6.84 MBytes 5.73 Mbits/sec 6 93.3 KBytes [ 4] 50.00-60.00 sec 7.08 MBytes 5.94 Mbits/sec 8 91.9 KBytes [ 4] 60.00-70.00 sec 6.90 MBytes 5.79 Mbits/sec 8 91.9 KBytes [ 4] 70.00-80.00 sec 7.08 MBytes 5.94 Mbits/sec 13 82.0 KBytes [ 4] 80.00-90.00 sec 6.96 MBytes 5.84 Mbits/sec 10 72.1 KBytes [ 4] 90.00-100.00 sec 7.02 MBytes 5.89 Mbits/sec 3 93.3 KBytes [ 4] 100.00-110.00 sec 6.77 MBytes 5.68 Mbits/sec 8 91.9 KBytes [ 4] 110.00-120.00 sec 7.15 MBytes 5.99 Mbits/sec 8 91.9 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-120.00 sec 94.9 MBytes 6.63 Mbits/sec 121 sender [ 4] 0.00-120.00 sec 94.9 MBytes 6.63 Mbits/sec receiver
daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -i 10 -t 120 Connecting to host x.x.x.x, port 5001 [ 4] local x.x.x.x port 34496 connected to x.x.x.x port 5001 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-10.00 sec 18.1 MBytes 15.2 Mbits/sec 33 77.8 KBytes [ 4] 10.00-20.00 sec 6.96 MBytes 5.84 Mbits/sec 8 73.5 KBytes [ 4] 20.00-30.00 sec 6.84 MBytes 5.73 Mbits/sec 8 72.1 KBytes [ 4] 30.00-40.00 sec 7.15 MBytes 5.99 Mbits/sec 8 66.5 KBytes [ 4] 40.00-50.00 sec 6.84 MBytes 5.73 Mbits/sec 6 93.3 KBytes [ 4] 50.00-60.00 sec 7.08 MBytes 5.94 Mbits/sec 8 91.9 KBytes [ 4] 60.00-70.00 sec 6.90 MBytes 5.79 Mbits/sec 8 91.9 KBytes [ 4] 70.00-80.00 sec 7.08 MBytes 5.94 Mbits/sec 13 82.0 KBytes [ 4] 80.00-90.00 sec 6.96 MBytes 5.84 Mbits/sec 10 72.1 KBytes [ 4] 90.00-100.00 sec 7.02 MBytes 5.89 Mbits/sec 3 93.3 KBytes [ 4] 100.00-110.00 sec 6.77 MBytes 5.68 Mbits/sec 8 91.9 KBytes [ 4] 110.00-120.00 sec 7.15 MBytes 5.99 Mbits/sec 8 91.9 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-120.00 sec 94.9 MBytes 6.63 Mbits/sec 121 sender [ 4] 0.00-120.00 sec 94.9 MBytes 6.63 Mbits/sec receiver
daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -i 10 -t 120 Connecting to host x.x.x.x, port 5001 [ 4] local x.x.x.x port 52029 connected to x.x.x.x port 5001 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-10.00 sec 13.5 MBytes 11.3 Mbits/sec 15 66.5 KBytes [ 4] 10.00-20.00 sec 6.77 MBytes 5.68 Mbits/sec 5 80.6 KBytes [ 4] 20.00-30.00 sec 7.02 MBytes 5.89 Mbits/sec 8 74.9 KBytes [ 4] 30.00-40.00 sec 7.15 MBytes 5.99 Mbits/sec 8 73.5 KBytes [ 4] 40.00-50.00 sec 6.96 MBytes 5.84 Mbits/sec 8 67.9 KBytes [ 4] 50.00-60.00 sec 6.77 MBytes 5.68 Mbits/sec 6 93.3 KBytes [ 4] 60.00-70.00 sec 7.08 MBytes 5.94 Mbits/sec 8 91.9 KBytes [ 4] 70.00-80.00 sec 7.02 MBytes 5.89 Mbits/sec 12 82.0 KBytes [ 4] 80.00-90.00 sec 6.96 MBytes 5.84 Mbits/sec 8 86.3 KBytes [ 4] 90.00-100.00 sec 6.96 MBytes 5.84 Mbits/sec 7 76.4 KBytes [ 4] 100.00-110.00 sec 7.02 MBytes 5.89 Mbits/sec 6 90.5 KBytes [ 4] 110.00-120.00 sec 7.08 MBytes 5.94 Mbits/sec 8 89.1 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-120.00 sec 90.3 MBytes 6.31 Mbits/sec 99 sender [ 4] 0.00-120.00 sec 89.9 MBytes 6.29 Mbits/sec receiver
Then we have four classes with 15Mbit/s.
daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -i 10 -t 120 Connecting to host x.x.x.x, port 5001 [ 4] local x.x.x.x port 45678 connected to x.x.x.x port 5001 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-10.00 sec 19.0 MBytes 15.9 Mbits/sec 2 189 KBytes [ 4] 10.00-20.00 sec 18.6 MBytes 15.6 Mbits/sec 8 205 KBytes [ 4] 20.00-30.00 sec 18.4 MBytes 15.4 Mbits/sec 6 194 KBytes [ 4] 30.00-40.00 sec 18.6 MBytes 15.6 Mbits/sec 4 189 KBytes [ 4] 40.00-50.00 sec 18.0 MBytes 15.1 Mbits/sec 6 188 KBytes [ 4] 50.00-60.00 sec 18.4 MBytes 15.4 Mbits/sec 6 173 KBytes [ 4] 60.00-70.00 sec 18.4 MBytes 15.4 Mbits/sec 3 219 KBytes [ 4] 70.00-80.00 sec 18.5 MBytes 15.5 Mbits/sec 11 194 KBytes [ 4] 80.00-90.00 sec 18.4 MBytes 15.4 Mbits/sec 6 191 KBytes [ 4] 90.00-100.00 sec 18.4 MBytes 15.4 Mbits/sec 7 192 KBytes [ 4] 100.00-110.00 sec 18.4 MBytes 15.4 Mbits/sec 6 189 KBytes [ 4] 110.00-120.00 sec 24.0 MBytes 20.1 Mbits/sec 5 192 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-120.00 sec 227 MBytes 15.9 Mbits/sec 70 sender [ 4] 0.00-120.00 sec 226 MBytes 15.8 Mbits/sec receiver
daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -i 10 -t 120 Connecting to host x.x.x.x, port 5001 [ 4] local x.x.x.x port 40237 connected to x.x.x.x port 5001 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-10.00 sec 22.4 MBytes 18.8 Mbits/sec 13 205 KBytes [ 4] 10.00-20.00 sec 18.0 MBytes 15.1 Mbits/sec 6 195 KBytes [ 4] 20.00-30.00 sec 18.4 MBytes 15.4 Mbits/sec 6 189 KBytes [ 4] 30.00-40.00 sec 18.4 MBytes 15.4 Mbits/sec 5 188 KBytes [ 4] 40.00-50.00 sec 18.5 MBytes 15.5 Mbits/sec 4 177 KBytes [ 4] 50.00-60.00 sec 18.0 MBytes 15.1 Mbits/sec 4 223 KBytes [ 4] 60.00-70.00 sec 18.4 MBytes 15.4 Mbits/sec 6 205 KBytes [ 4] 70.00-80.00 sec 18.5 MBytes 15.5 Mbits/sec 8 189 KBytes [ 4] 80.00-90.00 sec 18.4 MBytes 15.4 Mbits/sec 8 171 KBytes [ 4] 90.00-100.00 sec 18.3 MBytes 15.3 Mbits/sec 7 188 KBytes [ 4] 100.00-110.00 sec 18.5 MBytes 15.5 Mbits/sec 6 174 KBytes [ 4] 110.00-120.00 sec 18.5 MBytes 15.5 Mbits/sec 4 221 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-120.00 sec 224 MBytes 15.7 Mbits/sec 77 sender [ 4] 0.00-120.00 sec 224 MBytes 15.6 Mbits/sec receiver
daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -i 10 -t 120 Connecting to host x.x.x.x, port 5001 [ 4] local x.x.x.x port 35318 connected to x.x.x.x port 5001 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-10.00 sec 20.7 MBytes 17.4 Mbits/sec 11 195 KBytes [ 4] 10.00-20.00 sec 18.3 MBytes 15.4 Mbits/sec 7 198 KBytes [ 4] 20.00-30.00 sec 18.5 MBytes 15.5 Mbits/sec 6 189 KBytes [ 4] 30.00-40.00 sec 17.9 MBytes 15.0 Mbits/sec 6 189 KBytes [ 4] 40.00-50.00 sec 18.4 MBytes 15.4 Mbits/sec 4 181 KBytes [ 4] 50.00-60.00 sec 18.4 MBytes 15.4 Mbits/sec 6 177 KBytes [ 4] 60.00-70.00 sec 18.5 MBytes 15.5 Mbits/sec 4 208 KBytes [ 4] 70.00-80.00 sec 18.4 MBytes 15.4 Mbits/sec 10 189 KBytes [ 4] 80.00-90.00 sec 18.4 MBytes 15.4 Mbits/sec 7 177 KBytes [ 4] 90.00-100.00 sec 18.3 MBytes 15.3 Mbits/sec 7 189 KBytes [ 4] 100.00-110.00 sec 18.5 MBytes 15.5 Mbits/sec 6 184 KBytes [ 4] 110.00-120.00 sec 19.5 MBytes 16.4 Mbits/sec 7 165 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-120.00 sec 224 MBytes 15.6 Mbits/sec 81 sender [ 4] 0.00-120.00 sec 223 MBytes 15.6 Mbits/sec receiver
daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -i 10 -t 120 Connecting to host x.x.x.x, port 5001 [ 4] local x.x.x.x port 54984 connected to x.x.x.x port 5001 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-10.00 sec 19.5 MBytes 16.3 Mbits/sec 2 222 KBytes [ 4] 10.00-20.00 sec 18.5 MBytes 15.5 Mbits/sec 4 204 KBytes [ 4] 20.00-30.00 sec 18.4 MBytes 15.4 Mbits/sec 6 194 KBytes [ 4] 30.00-40.00 sec 18.2 MBytes 15.3 Mbits/sec 6 189 KBytes [ 4] 40.00-50.00 sec 18.4 MBytes 15.4 Mbits/sec 6 187 KBytes [ 4] 50.00-60.00 sec 18.4 MBytes 15.4 Mbits/sec 6 171 KBytes [ 4] 60.00-70.00 sec 18.4 MBytes 15.4 Mbits/sec 4 218 KBytes [ 4] 70.00-80.00 sec 18.4 MBytes 15.4 Mbits/sec 10 189 KBytes [ 4] 80.00-90.00 sec 18.4 MBytes 15.4 Mbits/sec 8 184 KBytes [ 4] 90.00-100.00 sec 18.1 MBytes 15.2 Mbits/sec 6 189 KBytes [ 4] 100.00-110.00 sec 18.5 MBytes 15.5 Mbits/sec 6 189 KBytes [ 4] 110.00-120.00 sec 20.9 MBytes 17.5 Mbits/sec 6 178 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-120.00 sec 224 MBytes 15.7 Mbits/sec 70 sender [ 4] 0.00-120.00 sec 223 MBytes 15.6 Mbits/sec receiver
Each class only got the bandwidth that was assigned to it. Hopefully the CLI output makes it even more clear how QoS works on Cisco devices.
Hello, Daniel!
At first, thank you for the article. I really enjoy to read your blog.
About a content, I dont actually agree with that sentence – “QoS is only active during congestion.”. That decline the important QoS part as “congestion avoidance”. And as we know, that part works to _prevent_ a congestion but not to manage. So QoS may take a work before a congestion occurs and basically it uses RED/WRED mechanism.
I like how Juniper defines drop profiles to utilize congestion avoidance – http://www.juniper.net/documentation/en_US/junos15.1/topics/concept/red-drop-profile-overview-cos-config-guide.html
Clearly undestandable CLI terms – an example, at 50% drop 10% of random traffic to prevent an avoidance.
edit class-of-service drop-profiles DROP1
fill-level 50
drop-probability 10
top
Thanks Boris!
I should have been a bit more clear that the management of queues is only active during congestion. WRED is a good tool for TCP flows if you can find the right values to use. The syntax on J seems a bit more simple than C.
Nice Article,
could you please share the Cli result as well.
I’ll try to add the CLI results within the next few days.
Hi Daniel
Thanks for all your CCDE posts. I just subscribed and loving it all the way. I’m currently studying for the CCDE so having blogs like yours to read is definitely rewarding. Regarding this post, can you please share the CLi results with your findings when you can?
Thanks for the feedback. I’ll add the CLI results within the next few days.