Borrowing Credits When Using Shaper on Cisco IOS

Introduction

When using a shaper on IOS, the shaper allows a deficit to be created, borrowing
future credits. It’s common knowledge that a shaper queues or buffers packets but
it’s not common knowledge that the shaper allows a deficit to be created.

To demonstrate the concepts I have setup a very simple network with two routers
connected by a FastEthernet link.

Their clocks have been synchronized to show the timing of the events going on.

This post assumes prior knowledge of QoS with regard to concepts such as Bc, Be
CIR and Tc.

Using a Policer

A policer does not allowed a deficit to be created. This can be proven very easily.
To prove the concept a single rate, two color policer will be used. A two color
policer does not have a Be bucket so no tokens will be spilled over from the Bc
bucket.

The Bc bucket starts out full. When a packet arrives, the packet size is compared
to the number of tokens (bytes) in the Bc bucket. If the packet fits then the appropriate
number of tokens is taken from the Bc bucket and the packet is sent on its way.

The next time a packet arrives, the number of tokens in the bucket will depend on the
time interval between the packets. This is in contrast to a shaper that submits tokens
to the bucket at fixed intervals.

A policer does not allow a deficit to be created. A policer is created with a Bc value
of 1000 bytes. The CIR is set to 10 kbit/s. With such a low value for Bc it means that
any packets with a size over 1000 bytes will be dropped.

R1#sh policy-map
  Policy Map POLICER
    Class class-default
     police cir 10000 bc 1000
       conform-action transmit 
       exceed-action drop

R1#ping 10.0.0.2 size 1000

Type escape sequence to abort.
Sending 5, 1000-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

No packets made it through due to the size of the packet being 1000 bytes payload,
20 bytes of IP and 8 bytes of ICMP which is more than 1000 bytes in total.
The policer does not allow a deficit to be created so all packets had to be dropped.

If we ping with a 972 byte payload some packets should make it through.

R1#ping 10.0.0.2 size 972 ti 1

Type escape sequence to abort.
Sending 5, 972-byte ICMP Echos to 10.0.0.2, timeout is 1 seconds:
!.!.!
Success rate is 60 percent (3/5), round-trip min/avg/max = 32/46/68 ms

The policer shows that some packets have exceeded.

R1#sh policy-map int f0/0
 FastEthernet0/0 

  Service-policy output: POLICER

    Class-map: class-default (match-any)
      35 packets, 12728 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any 
      police:
          cir 10000 bps, bc 1000 bytes
        conformed 5 packets, 3138 bytes; actions:
          transmit 
        exceeded 7 packets, 7042 bytes; actions:
          drop 
        conformed 0 bps, exceed 0 bps

While sending the packets I had debugs going on both devices. This is the timing
of the event.

Apr 15 12:08:10.183: IP: tableid=0, s=10.0.0.1 (local), d=10.0.0.2 (FastEthernet0/0), routed via FIB
Apr 15 12:08:10.187: IP: s=10.0.0.1 (local), d=10.0.0.2 (FastEthernet0/0), len 972, sending
Apr 15 12:08:10.247: IP: tableid=0, s=10.0.0.2 (FastEthernet0/0), d=10.0.0.1 (FastEthernet0/0), routed via RIB
Apr 15 12:08:10.247: IP: s=10.0.0.2 (FastEthernet0/0), d=10.0.0.1 (FastEthernet0/0), len 972, rcvd 3

A packet is sent at 10.183 and received at 10.247. A look at R2 confirms that
it sent the packet at 10.195.

Apr 15 12:08:10.195: ICMP: echo reply sent, src 10.0.0.2, dst 10.0.0.1

The next packet is sent at 10.255 but this does not make it through the policer.

Apr 15 12:08:10.255: IP: tableid=0, s=10.0.0.1 (local), d=10.0.0.2 (FastEthernet0/0), routed via FIB
Apr 15 12:08:10.259: IP: s=10.0.0.1 (local), d=10.0.0.2 (FastEthernet0/0), len 972, sending

With a CIR of 10 kbit/s, we can only send 1250 bytes every second.

The router then waits for the ICMP packet to timeout which was set to one second.
Then the next packet is sent at 11.255 and received at 11.287.

Apr 15 12:08:11.255: IP: tableid=0, s=10.0.0.1 (local), d=10.0.0.2 (FastEthernet0/0), routed via FIB
Apr 15 12:08:11.255: IP: s=10.0.0.1 (local), d=10.0.0.2 (FastEthernet0/0), len 972, sending
Apr 15 12:08:11.287: IP: tableid=0, s=10.0.0.2 (FastEthernet0/0), d=10.0.0.1 (FastEthernet0/0), routed via RIB
Apr 15 12:08:11.287: IP: s=10.0.0.2 (FastEthernet0/0), d=10.0.0.1 (FastEthernet0/0), len 972, rcvd 3

Output from the other router shows it was sent at 11.255.

Apr 15 12:08:11.255: ICMP: echo reply sent, src 10.0.0.2, dst 10.0.0.1

It is clear that a policer does not allow a deficit, either the packet makes it
through or it is dropped.

Using a Shaper

A shaper allows a deficit to be created. This can be proven by creating a shaper
that uses only Bc and no Be. If a packet is sent with a size larger than Bc it
should in theory be dropped. This is however not the case. The following shaper
is used.

R1#sh policy-map
  Policy Map SHAPER
    Class class-default
      Traffic Shaping
         Average Rate Traffic Shaping
         CIR 10000 (bps) Max. Buffers Limit 1000 (Packets)
         Bc 8000 Be 0

If a shaper does not allow a deficit then all packets larger than 1000 bytes should
be dropped.

R1#ping 10.0.0.2 size 972 ti 1

Type escape sequence to abort.
Sending 5, 972-byte ICMP Echos to 10.0.0.2, timeout is 1 seconds:
!!!!.
Success rate is 80 percent (4/5), round-trip min/avg/max = 32/409/808 ms

Almost all packets made it through which could be due to buffering but let’s
have a look at the timing of what happened.

Apr 15 12:19:45.683: IP: tableid=0, s=10.0.0.1 (local), d=10.0.0.2 (FastEthernet0/0), routed via FIB
Apr 15 12:19:45.687: IP: s=10.0.0.1 (local), d=10.0.0.2 (FastEthernet0/0), len 972, sending
Apr 15 12:19:45.775: IP: tableid=0, s=10.0.0.2 (FastEthernet0/0), d=10.0.0.1 (FastEthernet0/0), routed via RIB
Apr 15 12:19:45.775: IP: s=10.0.0.2 (FastEthernet0/0), d=10.0.0.1 (FastEthernet0/0), len 972, rcvd 3

The Bc bucket starts out full so the packet is immediately transmitted.
Packet was sent at 45.683 and received at 45.775. We confirm with output
from the other router.

Apr 15 12:19:45.714: ICMP: echo reply sent, src 10.0.0.2, dst 10.0.0.1

The interesting part is that R1 sent its second packet at 45.783.

Apr 15 12:19:45.783: IP: tableid=0, s=10.0.0.1 (local), d=10.0.0.2 (FastEthernet0/0), routed via FIB
Apr 15 12:19:45.787: IP: s=10.0.0.1 (local), d=10.0.0.2 (FastEthernet0/0), len 972, sending
Apr 15 12:19:45.811: IP: tableid=0, s=10.0.0.2 (FastEthernet0/0), d=10.0.0.1 (FastEthernet0/0), routed via RIB
Apr 15 12:19:45.815: IP: s=10.0.0.2 (FastEthernet0/0), d=10.0.0.1 (FastEthernet0/0), len 972, rcvd 3

This packet was then received at 45.811. Once again output from the other router.

Apr 15 12:19:45.782: ICMP: echo reply sent, src 10.0.0.2, dst 10.0.0.1

R1 should not have been allowed to send this packet so quickly after the first one.
With our shaper applied it should have had to wait around 800ms before sending the
next one. However a deficit was created to allow sending the packet more quickly.

If we look at the five packets that R2 replied to we can see a pattern.

Apr 15 12:19:45.714: ICMP: echo reply sent, src 10.0.0.2, dst 10.0.0.1
Apr 15 12:19:45.782: ICMP: echo reply sent, src 10.0.0.2, dst 10.0.0.1
Apr 15 12:19:46.498: ICMP: echo reply sent, src 10.0.0.2, dst 10.0.0.1
Apr 15 12:19:47.298: ICMP: echo reply sent, src 10.0.0.2, dst 10.0.0.1
Apr 15 12:19:48.918: ICMP: echo reply sent, src 10.0.0.2, dst 10.0.0.1

The first two packets came in very quickly. Between packet two and three there is
a 716ms gap. Between three and four there is a 800ms gap. Between four and five
there is a 1620ms gap.

It is clear that at the end the router had to pay its dues.

Conclusion

Shapers on Cisco IOS allows a deficit to be created. This means that packets larger
than the size of the Bc bucket can be sent. The internals of this mechanism is only
known by Cisco.

What is the reason for this behavior? I can only speculate but it could be to try to
send packets rather than dropping them. What are your ideas?