Raspberry Pi 3B+ Home Router - Beating Bufferbloat with Cake

Evuraan Tue Jan 15 23:44:36 PST 2019

Thing 1 with my Raspberry 3B+ Router

My Pi 3B+ home router

Above is the picture of my son with our Pi3b+ router.

2019 is when people elsewhere are rolling in Full Duplex Gibabit+ speeds to their homes. Not us in the US, where ISPs keep on their shenanigans to hold prices up and speeds down, enforce usage caps and also introduced ill advised legislation to revoke Net Neutrality protections. Amidst all this, my predatory ISP started offering 150mbps down / 5mbps upward speeds only recently - and I decided to move up from my DD-WRT running on a WNDR3400 - it could not handle speeds larger than 100mbps, and I wanted to use a SBC which runs a "fuller" Linux. I chose to buy a Raspberry Pi 3B+, loaded it with raspbian lite, and setup my own Home Router.

Pi 3B+ has a Gigabit NIC, with some caveats as it shares the USB bus, so you never would get the Gigabit speeds. Folks have reported Pi 3B+ Ethernet actually only gives them around ~300mbps - which is more than enough in my case, as my ISP's speed is pretty crappy anyway.

For the other side, I found a USB Ethernet to connect to my TC-7610 Cable Modem - in the photo above, it is the black adapter which is connected to the white ethernet cable.

eth0 is my internal, and eth1 is the USB NIC going to the cable modem. I leveraged ufw to setup the rules to suit my port forwarding needs.


Those two LEDs - what are they for?

Took out my soldering iron, and connected two LEDs to the GPIO ports - a green, and a red - secured them to the raspberry pi's case using transparent tape. The green LED would blink if it detects traffic on eth1. If we loose the internet, or traffic stops on eth1, the red LED would light up. If my sons see the solid red LED lit, they know they cannot watch their TV shows online! Let's leave this as a topic for another time, to go over the Golang code that controls the LEDs, and also exposes a few URLs to tell me the state and statistics of the router.


The menace: Bufferbloat

Under egress load, I started experiencing severe latency - the performance was suffering. DNS lookups would take long, resulting in slow page loads, due to Bufferbloat. There are exhaustive web resources (1, 2) dedicated to the phenomenon of Bufferbloat, I'd leave the avid reader to pursue in her own charter, but suffice to say it percolated in my case, made me take notice. Also note, bufferbloat may become more evident on slow links, where the outbound and inbound speeds are set so wide apart.

To check if you have this menace on your networking, here's the easiest way to test for bufferbloat. If you have bufferbloat, read on, to learn how to use queue management mechanisms to keep it in control.


Curb bufferbloat using fq_codel

One solution to curb Bufferbloat is to use the FQ_Codel AQM (Fair Queueing Controlled Delay Active Queue Management). sch_fq_codel ships with most Linux kernel versions these days, and hence this is presently the easiest fix against bufferbloat. fq_codel is now officially IETF RFC8290.

Running shaper.sh as root will put fq_codel to work right away. I used /etc/crontab to have this script run, see the script section for more.

shaper.sh with fq_codel


# -------------------------------------------------------------- #
# evuraan at gmail -                                             #
# Freely given - use at your own risk!                           #
# Also see: https://wiki.gentoo.org/wiki/Traffic_shaping         #
# -------------------------------------------------------------- #

ext="eth1" 		# Set your Egress interface
ext_up="5000kbit" 	# Set your max theoretical outbound speed


q="1514"       # HTB Quantum = 1500bytes IP + 14 bytes ethernet.
quantum="300"  # fq_codel quantum 300 gives a boost to interactive flows

modprobe ifb
modprobe sch_fq_codel

$tc qdisc add dev $ext root handle 1: htb default 11
$tc class add dev $ext parent 1: classid 1:1 htb rate $ext_up
$tc class add dev $ext parent 1:1 classid 1:11 htb rate $ext_up prio 0 quantum $q
$tc qdisc add dev $ext parent 1:11 fq_codel quantum $quantum noecn

Curb bufferbloat with some Cake!

Just in this peculiar case, Cake stands for "Common Applications Kept Enhanced" queuing discipline, made by Toke Høiland-Jørgensen and Jonathan Morton. Cake includes a variant of the fq_codel algorithm called Cobalt and apparently can work magic! One cool thing that struck me, is when NAT is in use, CAKE reaches into nf_conntrack layer and extracts information to identify flow - the 5-tuple of source IP address, destination IP address, source port, destination port, and protocol. This is seen as a layering infraction, but is pretty clever. There indeed are way better resources (1, 2) out there to read about Cake, it is seen as a better solution to squeeze the most bandwidth and latency out of even the slowest ISP links and routers. (1). Cake can shape more, for example: 115mbps on an Archer C7, where HTB and fq_codel could not (1).

In short, if you can use Cake, you should use it instead of fq_codel and HTB solution. There in lies the rub, at least for now: Cake qdisc has been merged into Linux 4.19, and at the time of this writing, 4.14.92-v7 seems to be the latest on Raspbian 9.6 (stretch). When I upgraded my pi to 4.19.14, there was no loadable sch_cake module - which brings us to the rest of this topic - how to compile and install sch_cake out of tree for Raspberry pi.

If your system happens to come with sch_cake loadable module, skip over to this section.


compile and install sch_cake for Raspberry pi

Step 1 : rpi-source

(If you have the kernel source installed, you can skip this step.)

rpi-source installs the kernel source used to build rpi-update kernels and the kernel on the Raspian image. This makes it possible to build loadable kernel modules.

$ sudo apt-get install bc libncurses5-dev flex bison
$ sudo wget https://raw.githubusercontent.com/notro/rpi-source/master/rpi-source -O /usr/bin/rpi-source && sudo chmod +x /usr/bin/rpi-source && /usr/bin/rpi-source -q --tag-update
$ sudo rpi-source

Step 2 : build sch_cake module

$ git clone https://github.com/dtaht/sch_cake.git
$ cd sch_cake
$ sudo make; sudo make install

Step 3 : build and install the iproute with cake support

$ git clone https://github.com/dtaht/tc-adv
$ cd tc-adv
$ ./configure
$ sudo make
$ sudo make install

shaper.sh for Cake shaper


modprobe sch_cake
modprobe act_mirred
tc qdisc add dev eth1 root cake bandwidth 5mbit

running shaper.sh

Here's how I run the shaper.sh on my pi: Saved the shaper script as /usr/local/bin/shaper.sh, and
$ sudo chmod +x /usr/local/bin/shaper.sh
Then, added a line in /etc/crontab so root runs it shortly after reboots:
@reboot         root    /usr/local/bin/shaper.sh  1>/dev/null 2>/dev/null || :

monitoring your CAKE in realtime

You can watch the schduler in action as:
watch -n 1 tc -s qdisc show dev eth1
 $  tc -s qdisc show dev eth1
qdisc cake 8001: root refcnt 2 bandwidth 5Mbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms raw overhead 0 
 Sent 1039127541 bytes 3111195 pkt (dropped 6122, overlimits 3710115 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 360982b of 4Mb
 capacity estimate: 5Mbit
 min/max network layer size:           42 /    1514
 min/max overhead-adjusted size:       42 /    1514
 average network hdr offset:           14

                   Bulk  Best Effort        Voice
  thresh      312496bit        5Mbit     1250Kbit
  target         58.1ms        5.0ms       14.5ms
  interval      153.1ms      100.0ms      109.5ms
  pk_delay        7.4ms        4.2ms        448us
  av_delay        564us        743us         43us
  sp_delay          6us         11us          6us
  backlog            0b           0b           0b
  pkts             9869      2507301       600147
  bytes          533077    822257322    225237060
  way_inds            2       106513            0
  way_miss         1452       100992          381
  way_cols            0            0            0
  drops               0         6122            0
  marks               0            2            0
  ack_drop            0            0            0
  sp_flows            1            1            0
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len            98         1514         1191
  quantum           300          300          300



* The Bufferbloat projects - undoubtedly the best resource on this topic.
* DSLreports.com Speedtest-The easiest speedtest there is to find out if you have Bufferbloat.
* IETF RFC8290 - The Flow Queue CoDel Packet Scheduler and Active Queue Management Algorithm


Thank you for reading. I did not know of bufferbloat until I stumbled upon the DSLreports.com speedtest page, which gave my setup the lowest grade due to Bufferbloat. Thereon, I struggled a little bit, but the trip has been enjoyable, as the research was fun, I learned a bunch of stuff. I hope you or someone in need find this stuff useful. Please leave your corrections/comments/feedback here.