I have been running some QoS tests lately and wanted to share some of my results. Some of this behavior is described in various documentation guides but it’s not really clearly described in one place. I’ll describe what I have found so far in this post.

QoS is only active during congestion. This is well known but it’s not as well known how congestion is detected. the TX ring is used to hold packets before they get transmitted out on an interface. This is a hardware FIFO queue and when the queue gets filled, the interface is congested. When buying a subrate circuit from a SP, something must be added to achieve the backpressure so that the TX ring is considered full. This is done by applying a parent shaper and a child policy with the actual queue configuration.

The LLQ is used for high priority traffic. When the interface is not congested, the LLQ can use all available bandwidth unless an explicit policer is configured under the LLQ.

A normal queue can use more bandwidth than it is guaranteed when there is no congestion.

When a normal queue wants to use more bandwidth than its guaranteed, it can if there is bandwidth available from other classes not using their full amount of bandwidth.

If multiple queues want to use more bandwidth than they are guaranteed and there is bandwidth available from other classes not using their full amount, they can. The amount of “free” bandwidth seems to be distributed quite evenly although I don’t think there is any guarantee that this is always the case.

When there is congestion, the LLQ does not get assigned additional bandwidth even if there is bandwidth available from any of the normal classes. The only time the LLQ can use more bandwidth is if there is no congestion.

I hope this post gives you some insight on how LLQ and CBWFQ behaves on Cisco devices.

Update:

Some of the readers asked if I could provide the CLI output as well which I am doing below. My setup was two CSR1000v routers and two Ubuntu hosts running Iperf3.

The first test is to test the maximum bandwidth without any QoS applied and sending TCP traffic.

daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001
Connecting to host x.x.x.x, port 5001
[  4] local x.x.x.x port 38941 connected to x.x.x.x port 5001
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   124 MBytes  1036 Mbits/sec    6    837 KBytes       
[  4]   1.00-2.00   sec   119 MBytes   996 Mbits/sec    0    940 KBytes       
[  4]   2.00-3.00   sec   120 MBytes  1007 Mbits/sec   12    571 KBytes       
[  4]   3.00-4.00   sec   119 MBytes   996 Mbits/sec    0    714 KBytes       
[  4]   4.00-5.00   sec   119 MBytes   997 Mbits/sec    0    831 KBytes       
[  4]   5.00-6.00   sec   120 MBytes  1007 Mbits/sec    6    673 KBytes       
[  4]   6.00-7.01   sec   120 MBytes  1001 Mbits/sec    0    800 KBytes       
[  4]   7.01-8.00   sec   118 MBytes   991 Mbits/sec    5    650 KBytes       
[  4]   8.00-9.00   sec   120 MBytes  1006 Mbits/sec    0    778 KBytes       
[  4]   9.00-10.00  sec   119 MBytes   997 Mbits/sec    0    885 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.17 GBytes  1003 Mbits/sec   29             sender
[  4]   0.00-10.00  sec  1.17 GBytes  1001 Mbits/sec                  receiver

Without any shaping the routers were able to use the full bandwidth available. The next test uses UDP.

daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -u -b 1000m
Connecting to host x.x.x.x, port 5001
[  4] local x.x.x.x port 57585 connected to x.x.x.x port 5001
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec   111 MBytes   928 Mbits/sec  14168  
[  4]   1.00-2.00   sec   124 MBytes  1043 Mbits/sec  15923  
[  4]   2.00-3.00   sec   115 MBytes   961 Mbits/sec  14660  
[  4]   3.00-4.00   sec   118 MBytes   991 Mbits/sec  15117  
[  4]   4.00-5.00   sec   127 MBytes  1062 Mbits/sec  16209  
[  4]   5.00-6.00   sec  94.5 MBytes   792 Mbits/sec  12095  
[  4]   6.00-7.00   sec   132 MBytes  1109 Mbits/sec  16913  
[  4]   7.00-8.00   sec   125 MBytes  1047 Mbits/sec  15970  
[  4]   8.00-9.00   sec   114 MBytes   955 Mbits/sec  14572  
[  4]   9.00-10.00  sec   131 MBytes  1099 Mbits/sec  16774  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.00  sec  1.16 GBytes   999 Mbits/sec  0.070 ms  31413/152388 (21%)  
[  4] Sent 152388 datagrams

UDP could utilize the full bandwidth as well but only after I had tweaked some buffer settings in Linux. With the default buffer size I could only achieve around 500 Mbit/s. UDP is more reliant on buffer size since it has no window scaling like TCP.

I then applied the following QoS configuration.

class-map match-all CM1
match access-group name ACL1 
class-map match-all CM2 
match access-group name ACL2 
class-map match-all CM3 
match access-group name ACL3 
class-map match-all CM4 
match access-group name ACL4 
class-map match-all CM5 
match access-group name ACL5 
class-map match-all CM6 
match access-group name ACL6 
class-map match-all CM7 
match access-group name ACL7 
class-map match-all CM8 
match access-group name ACL8
!
policy-map CHILD
class CM1
priority 7000
class CM2
  bandwidth 5000
class CM3
  bandwidth 5000
class CM4
  bandwidth 5000
class CM5
  bandwidth 15000
class CM6
  bandwidth 15000
class CM7
  bandwidth 15000
class CM8
  bandwidth 15000
!
policy-map SHAPE90M
class class-default
  shape average 90000000   
   service-policy CHILD
!
interface GigabitEthernet0/2
service-policy output SHAPE90M

After applying the policy, the priority class can use the full bandwidth 90Mbit/s. The first test uses TCP.

daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -i 10 -t 120
Connecting to host x.x.x.x, port 5001
[ 4] local x.x.x.x port 38949 connected to x.x.x.x port 5001
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-10.00 sec 106 MBytes 88.6 Mbits/sec 33 624 KBytes
[ 4] 10.00-20.00 sec 102 MBytes 86.0 Mbits/sec 3 706 KBytes
[ 4] 20.00-30.00 sec 102 MBytes 86.0 Mbits/sec 3 631 KBytes
[ 4] 30.00-40.00 sec 102 MBytes 86.0 Mbits/sec 6 556 KBytes
[ 4] 40.00-50.00 sec 102 MBytes 86.0 Mbits/sec 0 697 KBytes
[ 4] 50.00-60.00 sec 102 MBytes 86.0 Mbits/sec 6 617 KBytes
[ 4] 60.00-70.00 sec 101 MBytes 84.9 Mbits/sec 6 667 KBytes
[ 4] 70.00-80.00 sec 100 MBytes 83.9 Mbits/sec 37 591 KBytes
[ 4] 80.00-90.00 sec 101 MBytes 84.9 Mbits/sec 3 624 KBytes
[ 4] 90.00-100.00 sec 102 MBytes 86.0 Mbits/sec 9 540 KBytes
[ 4] 100.00-110.00 sec 102 MBytes 86.0 Mbits/sec 0 680 KBytes
[ 4] 110.00-120.00 sec 102 MBytes 86.0 Mbits/sec 6 612 KBytes
– – – – – – – – – – – – – – – – – – – – – – – – –
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-120.00 sec 1.20 GBytes 85.9 Mbits/sec 112 sender
[ 4] 0.00-120.00 sec 1.20 GBytes 85.6 Mbits/sec receiver

This is the output from the CSR router.

R1#sh policy-map interface gi2.200
 GigabitEthernet2.200 

  Service-policy output: SHAPE90M

    Class-map: class-default (match-any)  
      812652 packets, 1233588696 bytes
      30 second offered rate 88625000 bps, drop rate 9000 bps
      Match: any 
      Queueing
      queue limit 937 packets
      (queue depth/total drops/no-buffer drops) 0/105/0
      (pkts output/bytes output) 812550/1233450900
      shape (average) cir 90000000, bc 360000, be 360000
      target shape rate 90000000

      Service-policy : CHILD

        queue stats for all priority classes:
          Queueing
          queue limit 512 packets
          (queue depth/total drops/no-buffer drops) 490/105/0
          (pkts output/bytes output) 812550/1233450900

        Class-map: CM1 (match-all)  
          812655 packets, 1233610290 bytes
          30 second offered rate 88627000 bps, drop rate 8000 bps
          Match: access-group name ACL1
          Priority: 7000 kbps, burst bytes 175000, b/w exceed drops: 105

The next test is then performed with UDP. Observe that there is some packet loss since UDP does not have acknowledgements.

daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -u -b 90m -i 10 -t 120
Connecting to host x.x.x.x, port 5001
[  4] local x.x.x.x port 35311 connected to x.x.x.x port 5001
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-10.00  sec   106 MBytes  89.2 Mbits/sec  13604  
[  4]  10.00-20.00  sec   107 MBytes  90.0 Mbits/sec  13734  
[  4]  20.00-30.00  sec   107 MBytes  90.0 Mbits/sec  13732  
[  4]  30.00-40.00  sec   107 MBytes  90.0 Mbits/sec  13733  
[  4]  40.00-50.00  sec   107 MBytes  90.0 Mbits/sec  13733  
[  4]  50.00-60.00  sec   107 MBytes  90.0 Mbits/sec  13734  
[  4]  60.00-70.00  sec   107 MBytes  90.0 Mbits/sec  13733  
[  4]  70.00-80.00  sec   107 MBytes  90.0 Mbits/sec  13734  
[  4]  80.00-90.00  sec   107 MBytes  90.0 Mbits/sec  13731  
[  4]  90.00-100.00 sec   107 MBytes  90.0 Mbits/sec  13734  
[  4] 100.00-110.00 sec   107 MBytes  90.0 Mbits/sec  13737  
[  4] 110.00-120.00 sec   107 MBytes  90.0 Mbits/sec  13729  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-120.00 sec  1.26 GBytes  89.9 Mbits/sec  0.727 ms  8544/164668 (5.2%)  
[  4] Sent 164668 datagrams
R1#sh policy-map interface gi2.200
 GigabitEthernet2.200 

  Service-policy output: SHAPE90M

    Class-map: class-default (match-any)  
      1720938 packets, 2517232191 bytes
      30 second offered rate 92407000 bps, drop rate 2099000 bps
      Match: any 
      Queueing
      queue limit 937 packets
      (queue depth/total drops/no-buffer drops) 0/24449/0
      (pkts output/bytes output) 1696469/2482966749
      shape (average) cir 90000000, bc 360000, be 360000
      target shape rate 90000000

      Service-policy : CHILD

        queue stats for all priority classes:
          Queueing
          queue limit 512 packets
          (queue depth/total drops/no-buffer drops) 240/24449/0
          (pkts output/bytes output) 1696469/2482966749

        Class-map: CM1 (match-all)  
          1720918 packets, 2517251531 bytes
          30 second offered rate 92407000 bps, drop rate 2702000 bps
          Match: access-group name ACL1
          Priority: 7000 kbps, burst bytes 175000, b/w exceed drops: 24449

The next test sends 90Mbit/s UDP in the priority class and 90Mbit/s TCP in another class.

daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -i 10 -t 120
Connecting to host x.x.x.x, port 5001
[  4] local x.x.x.x port 51066 connected to x.x.x.x port 5001
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-10.00  sec  94.5 MBytes  79.3 Mbits/sec  157   76.4 KBytes       
[  4]  10.00-20.00  sec  93.0 MBytes  78.0 Mbits/sec  145   77.8 KBytes       
[  4]  20.00-30.00  sec  93.5 MBytes  78.5 Mbits/sec  153    102 KBytes       
[  4]  30.00-40.00  sec  91.2 MBytes  76.5 Mbits/sec  172   87.7 KBytes       
[  4]  40.00-50.00  sec  93.5 MBytes  78.4 Mbits/sec  132   77.8 KBytes       
[  4]  50.00-60.00  sec  93.8 MBytes  78.7 Mbits/sec  117   91.9 KBytes       
[  4]  60.00-70.00  sec  87.2 MBytes  73.2 Mbits/sec  101    103 KBytes       
[  4]  70.00-80.00  sec  91.7 MBytes  76.9 Mbits/sec  151    102 KBytes       
[  4]  80.00-90.00  sec  93.4 MBytes  78.3 Mbits/sec  181    100 KBytes       
[  4]  90.00-100.00 sec  93.4 MBytes  78.3 Mbits/sec  144   91.9 KBytes       
[  4] 100.00-110.00 sec  93.2 MBytes  78.2 Mbits/sec  125   74.9 KBytes       
[  4] 110.00-120.00 sec  95.9 MBytes  80.4 Mbits/sec  128   79.2 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-120.00 sec  1.09 GBytes  77.9 Mbits/sec  1706             sender
[  4]   0.00-120.00 sec  1.09 GBytes  77.9 Mbits/sec                  receiver

The TCP traffic got 78Mbit/s which was everything that was available after the priority class got its share.

Class-map: CM2 (match-all)  
          1522624 packets, 2311298853 bytes
          30 second offered rate 81086000 bps, drop rate 156000 bps
          Match: access-group name ACL2
          Queueing
          queue limit 64 packets
          (queue depth/total drops/no-buffer drops) 61/2926/0
          (pkts output/bytes output) 1519698/2306857185
          bandwidth 5000 kbps

The priority class got around 8Mbit/s. Iperf is sending at 90Mbit/s but notice the 91% packet loss.

daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -u -b 90m -i 10 -t 120
Connecting to host x.x.x.x, port 5001
[  4] local x.x.x.x port 56800 connected to x.x.x.x port 5001
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-10.00  sec   106 MBytes  89.2 Mbits/sec  13604  
[  4]  10.00-20.00  sec   107 MBytes  90.0 Mbits/sec  13735  
[  4]  20.00-30.00  sec   107 MBytes  90.0 Mbits/sec  13731  
[  4]  30.00-40.00  sec   107 MBytes  90.0 Mbits/sec  13734  
[  4]  40.00-50.00  sec   107 MBytes  90.0 Mbits/sec  13733  
[  4]  50.00-60.00  sec   107 MBytes  90.0 Mbits/sec  13731  
[  4]  60.00-70.00  sec   107 MBytes  90.0 Mbits/sec  13736  
[  4]  70.00-80.00  sec   108 MBytes  90.2 Mbits/sec  13763  
[  4]  80.00-90.00  sec   107 MBytes  89.8 Mbits/sec  13700  
[  4]  90.00-100.00 sec   107 MBytes  90.0 Mbits/sec  13736  
[  4] 100.00-110.00 sec   107 MBytes  90.0 Mbits/sec  13733  
[  4] 110.00-120.00 sec   107 MBytes  90.0 Mbits/sec  13733  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-120.00 sec  1.26 GBytes  89.9 Mbits/sec  0.660 ms  149472/164540 (91%)  
[  4] Sent 164540 datagrams

The CSR router shows a high drop drate. To get the actual rate take the offered rate minus the drop rate.

Class-map: CM1 (match-all)  
          3869818 packets, 5535668355 bytes
          30 second offered rate 91647000 bps, drop rate 83689000 bps
          Match: access-group name ACL1
          Priority: 7000 kbps, burst bytes 175000, b/w exceed drops: 1719101

The final test is sending 90 Mbit/s in all eight classes. This means that each class should only get its alotted bandwidth.

The priority class gets 7Mbit/s.

Class-map: CM1 (match-all)  
          4954923 packets, 7059843887 bytes
          30 second offered rate 91638000 bps, drop rate 84681000 bps
          Match: access-group name ACL1
          Priority: 7000 kbps, burst bytes 175000, b/w exceed drops: 2712171

Then we have three classes getting 5Mbit/s.

daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -i 10 -t 120
Connecting to host x.x.x.x, port 5001
[  4] local x.x.x.x port 34496 connected to x.x.x.x port 5001
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-10.00  sec  18.1 MBytes  15.2 Mbits/sec   33   77.8 KBytes       
[  4]  10.00-20.00  sec  6.96 MBytes  5.84 Mbits/sec    8   73.5 KBytes       
[  4]  20.00-30.00  sec  6.84 MBytes  5.73 Mbits/sec    8   72.1 KBytes       
[  4]  30.00-40.00  sec  7.15 MBytes  5.99 Mbits/sec    8   66.5 KBytes       
[  4]  40.00-50.00  sec  6.84 MBytes  5.73 Mbits/sec    6   93.3 KBytes       
[  4]  50.00-60.00  sec  7.08 MBytes  5.94 Mbits/sec    8   91.9 KBytes       
[  4]  60.00-70.00  sec  6.90 MBytes  5.79 Mbits/sec    8   91.9 KBytes       
[  4]  70.00-80.00  sec  7.08 MBytes  5.94 Mbits/sec   13   82.0 KBytes       
[  4]  80.00-90.00  sec  6.96 MBytes  5.84 Mbits/sec   10   72.1 KBytes       
[  4]  90.00-100.00 sec  7.02 MBytes  5.89 Mbits/sec    3   93.3 KBytes       
[  4] 100.00-110.00 sec  6.77 MBytes  5.68 Mbits/sec    8   91.9 KBytes       
[  4] 110.00-120.00 sec  7.15 MBytes  5.99 Mbits/sec    8   91.9 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-120.00 sec  94.9 MBytes  6.63 Mbits/sec  121             sender
[  4]   0.00-120.00 sec  94.9 MBytes  6.63 Mbits/sec                  receiver
daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -i 10 -t 120
Connecting to host x.x.x.x, port 5001
[  4] local x.x.x.x port 34496 connected to x.x.x.x port 5001
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-10.00  sec  18.1 MBytes  15.2 Mbits/sec   33   77.8 KBytes       
[  4]  10.00-20.00  sec  6.96 MBytes  5.84 Mbits/sec    8   73.5 KBytes       
[  4]  20.00-30.00  sec  6.84 MBytes  5.73 Mbits/sec    8   72.1 KBytes       
[  4]  30.00-40.00  sec  7.15 MBytes  5.99 Mbits/sec    8   66.5 KBytes       
[  4]  40.00-50.00  sec  6.84 MBytes  5.73 Mbits/sec    6   93.3 KBytes       
[  4]  50.00-60.00  sec  7.08 MBytes  5.94 Mbits/sec    8   91.9 KBytes       
[  4]  60.00-70.00  sec  6.90 MBytes  5.79 Mbits/sec    8   91.9 KBytes       
[  4]  70.00-80.00  sec  7.08 MBytes  5.94 Mbits/sec   13   82.0 KBytes       
[  4]  80.00-90.00  sec  6.96 MBytes  5.84 Mbits/sec   10   72.1 KBytes       
[  4]  90.00-100.00 sec  7.02 MBytes  5.89 Mbits/sec    3   93.3 KBytes       
[  4] 100.00-110.00 sec  6.77 MBytes  5.68 Mbits/sec    8   91.9 KBytes       
[  4] 110.00-120.00 sec  7.15 MBytes  5.99 Mbits/sec    8   91.9 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-120.00 sec  94.9 MBytes  6.63 Mbits/sec  121             sender
[  4]   0.00-120.00 sec  94.9 MBytes  6.63 Mbits/sec                  receiver
daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -i 10 -t 120
Connecting to host x.x.x.x, port 5001
[  4] local x.x.x.x port 52029 connected to x.x.x.x port 5001
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-10.00  sec  13.5 MBytes  11.3 Mbits/sec   15   66.5 KBytes       
[  4]  10.00-20.00  sec  6.77 MBytes  5.68 Mbits/sec    5   80.6 KBytes       
[  4]  20.00-30.00  sec  7.02 MBytes  5.89 Mbits/sec    8   74.9 KBytes       
[  4]  30.00-40.00  sec  7.15 MBytes  5.99 Mbits/sec    8   73.5 KBytes       
[  4]  40.00-50.00  sec  6.96 MBytes  5.84 Mbits/sec    8   67.9 KBytes       
[  4]  50.00-60.00  sec  6.77 MBytes  5.68 Mbits/sec    6   93.3 KBytes       
[  4]  60.00-70.00  sec  7.08 MBytes  5.94 Mbits/sec    8   91.9 KBytes       
[  4]  70.00-80.00  sec  7.02 MBytes  5.89 Mbits/sec   12   82.0 KBytes       
[  4]  80.00-90.00  sec  6.96 MBytes  5.84 Mbits/sec    8   86.3 KBytes       
[  4]  90.00-100.00 sec  6.96 MBytes  5.84 Mbits/sec    7   76.4 KBytes       
[  4] 100.00-110.00 sec  7.02 MBytes  5.89 Mbits/sec    6   90.5 KBytes       
[  4] 110.00-120.00 sec  7.08 MBytes  5.94 Mbits/sec    8   89.1 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-120.00 sec  90.3 MBytes  6.31 Mbits/sec   99             sender
[  4]   0.00-120.00 sec  89.9 MBytes  6.29 Mbits/sec                  receiver

Then we have four classes with 15Mbit/s.

daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -i 10 -t 120
Connecting to host x.x.x.x, port 5001
[  4] local x.x.x.x port 45678 connected to x.x.x.x port 5001
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-10.00  sec  19.0 MBytes  15.9 Mbits/sec    2    189 KBytes       
[  4]  10.00-20.00  sec  18.6 MBytes  15.6 Mbits/sec    8    205 KBytes       
[  4]  20.00-30.00  sec  18.4 MBytes  15.4 Mbits/sec    6    194 KBytes       
[  4]  30.00-40.00  sec  18.6 MBytes  15.6 Mbits/sec    4    189 KBytes       
[  4]  40.00-50.00  sec  18.0 MBytes  15.1 Mbits/sec    6    188 KBytes       
[  4]  50.00-60.00  sec  18.4 MBytes  15.4 Mbits/sec    6    173 KBytes       
[  4]  60.00-70.00  sec  18.4 MBytes  15.4 Mbits/sec    3    219 KBytes       
[  4]  70.00-80.00  sec  18.5 MBytes  15.5 Mbits/sec   11    194 KBytes       
[  4]  80.00-90.00  sec  18.4 MBytes  15.4 Mbits/sec    6    191 KBytes       
[  4]  90.00-100.00 sec  18.4 MBytes  15.4 Mbits/sec    7    192 KBytes       
[  4] 100.00-110.00 sec  18.4 MBytes  15.4 Mbits/sec    6    189 KBytes       
[  4] 110.00-120.00 sec  24.0 MBytes  20.1 Mbits/sec    5    192 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-120.00 sec   227 MBytes  15.9 Mbits/sec   70             sender
[  4]   0.00-120.00 sec   226 MBytes  15.8 Mbits/sec                  receiver
daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -i 10 -t 120
Connecting to host x.x.x.x, port 5001
[  4] local x.x.x.x port 40237 connected to x.x.x.x port 5001
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-10.00  sec  22.4 MBytes  18.8 Mbits/sec   13    205 KBytes       
[  4]  10.00-20.00  sec  18.0 MBytes  15.1 Mbits/sec    6    195 KBytes       
[  4]  20.00-30.00  sec  18.4 MBytes  15.4 Mbits/sec    6    189 KBytes       
[  4]  30.00-40.00  sec  18.4 MBytes  15.4 Mbits/sec    5    188 KBytes       
[  4]  40.00-50.00  sec  18.5 MBytes  15.5 Mbits/sec    4    177 KBytes       
[  4]  50.00-60.00  sec  18.0 MBytes  15.1 Mbits/sec    4    223 KBytes       
[  4]  60.00-70.00  sec  18.4 MBytes  15.4 Mbits/sec    6    205 KBytes       
[  4]  70.00-80.00  sec  18.5 MBytes  15.5 Mbits/sec    8    189 KBytes       
[  4]  80.00-90.00  sec  18.4 MBytes  15.4 Mbits/sec    8    171 KBytes       
[  4]  90.00-100.00 sec  18.3 MBytes  15.3 Mbits/sec    7    188 KBytes       
[  4] 100.00-110.00 sec  18.5 MBytes  15.5 Mbits/sec    6    174 KBytes       
[  4] 110.00-120.00 sec  18.5 MBytes  15.5 Mbits/sec    4    221 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-120.00 sec   224 MBytes  15.7 Mbits/sec   77             sender
[  4]   0.00-120.00 sec   224 MBytes  15.6 Mbits/sec                  receiver
daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -i 10 -t 120
Connecting to host x.x.x.x, port 5001
[  4] local x.x.x.x port 35318 connected to x.x.x.x port 5001
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-10.00  sec  20.7 MBytes  17.4 Mbits/sec   11    195 KBytes       
[  4]  10.00-20.00  sec  18.3 MBytes  15.4 Mbits/sec    7    198 KBytes       
[  4]  20.00-30.00  sec  18.5 MBytes  15.5 Mbits/sec    6    189 KBytes       
[  4]  30.00-40.00  sec  17.9 MBytes  15.0 Mbits/sec    6    189 KBytes       
[  4]  40.00-50.00  sec  18.4 MBytes  15.4 Mbits/sec    4    181 KBytes       
[  4]  50.00-60.00  sec  18.4 MBytes  15.4 Mbits/sec    6    177 KBytes       
[  4]  60.00-70.00  sec  18.5 MBytes  15.5 Mbits/sec    4    208 KBytes       
[  4]  70.00-80.00  sec  18.4 MBytes  15.4 Mbits/sec   10    189 KBytes       
[  4]  80.00-90.00  sec  18.4 MBytes  15.4 Mbits/sec    7    177 KBytes       
[  4]  90.00-100.00 sec  18.3 MBytes  15.3 Mbits/sec    7    189 KBytes       
[  4] 100.00-110.00 sec  18.5 MBytes  15.5 Mbits/sec    6    184 KBytes       
[  4] 110.00-120.00 sec  19.5 MBytes  16.4 Mbits/sec    7    165 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-120.00 sec   224 MBytes  15.6 Mbits/sec   81             sender
[  4]   0.00-120.00 sec   223 MBytes  15.6 Mbits/sec                  receiver
daniel@daniel1:~$ sudo iperf3 -c x.x.x.x -f m -p 5001 -i 10 -t 120
Connecting to host x.x.x.x, port 5001
[  4] local x.x.x.x port 54984 connected to x.x.x.x port 5001
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-10.00  sec  19.5 MBytes  16.3 Mbits/sec    2    222 KBytes       
[  4]  10.00-20.00  sec  18.5 MBytes  15.5 Mbits/sec    4    204 KBytes       
[  4]  20.00-30.00  sec  18.4 MBytes  15.4 Mbits/sec    6    194 KBytes       
[  4]  30.00-40.00  sec  18.2 MBytes  15.3 Mbits/sec    6    189 KBytes       
[  4]  40.00-50.00  sec  18.4 MBytes  15.4 Mbits/sec    6    187 KBytes       
[  4]  50.00-60.00  sec  18.4 MBytes  15.4 Mbits/sec    6    171 KBytes       
[  4]  60.00-70.00  sec  18.4 MBytes  15.4 Mbits/sec    4    218 KBytes       
[  4]  70.00-80.00  sec  18.4 MBytes  15.4 Mbits/sec   10    189 KBytes       
[  4]  80.00-90.00  sec  18.4 MBytes  15.4 Mbits/sec    8    184 KBytes       
[  4]  90.00-100.00 sec  18.1 MBytes  15.2 Mbits/sec    6    189 KBytes       
[  4] 100.00-110.00 sec  18.5 MBytes  15.5 Mbits/sec    6    189 KBytes       
[  4] 110.00-120.00 sec  20.9 MBytes  17.5 Mbits/sec    6    178 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-120.00 sec   224 MBytes  15.7 Mbits/sec   70             sender
[  4]   0.00-120.00 sec   223 MBytes  15.6 Mbits/sec                  receiver

Each class only got the bandwidth that was assigned to it. Hopefully the CLI output makes it even more clear how QoS works on Cisco devices.

General – Behavior Of QoS Queues On Cisco IOS
Tagged on:             

6 thoughts on “General – Behavior Of QoS Queues On Cisco IOS

  • November 16, 2015 at 3:06 pm
    Permalink

    Hello, Daniel!
    At first, thank you for the article. I really enjoy to read your blog.
    About a content, I dont actually agree with that sentence – “QoS is only active during congestion.”. That decline the important QoS part as “congestion avoidance”. And as we know, that part works to _prevent_ a congestion but not to manage. So QoS may take a work before a congestion occurs and basically it uses RED/WRED mechanism.

    I like how Juniper defines drop profiles to utilize congestion avoidance – http://www.juniper.net/documentation/en_US/junos15.1/topics/concept/red-drop-profile-overview-cos-config-guide.html

    Clearly undestandable CLI terms – an example, at 50% drop 10% of random traffic to prevent an avoidance.
    edit class-of-service drop-profiles DROP1
    fill-level 50
    drop-probability 10
    top

    Reply
    • November 16, 2015 at 3:10 pm
      Permalink

      Thanks Boris!

      I should have been a bit more clear that the management of queues is only active during congestion. WRED is a good tool for TCP flows if you can find the right values to use. The syntax on J seems a bit more simple than C.

      Reply
  • November 16, 2015 at 5:50 pm
    Permalink

    Nice Article,
    could you please share the Cli result as well.

    Reply
    • November 16, 2015 at 10:00 pm
      Permalink

      I’ll try to add the CLI results within the next few days.

      Reply
  • November 16, 2015 at 6:46 pm
    Permalink

    Hi Daniel
    Thanks for all your CCDE posts. I just subscribed and loving it all the way. I’m currently studying for the CCDE so having blogs like yours to read is definitely rewarding. Regarding this post, can you please share the CLi results with your findings when you can?

    Reply
    • November 16, 2015 at 10:00 pm
      Permalink

      Thanks for the feedback. I’ll add the CLI results within the next few days.

      Reply

Leave a Reply to muhammad jamshed Cancel reply

Your email address will not be published. Required fields are marked *