Pfsense QoS Using CoDel

02/03/2022 14:42 SANS Internet Storm Center SANS Internet Storm Center - A global cooperative cyber threat / internet
/ internet security monitor and …
Threat Level: GREEN
SANS SANS ISC InfoSec Forums Participate: Learn more about our honeypot network
ISC: Keyword, Domain, Port, IP or Head

Search https://isc.sans.edu/honeypot.html
Emai
Sign U
← Next Thread
Previous T
Contact Us
Diary Securing and Optimizing Networks: Using pfSense Traffic Shaper Limiters to Com
Podcasts Bufferbloat
Jobs
Jo
Tools
Data [This is a guest diary by Yee Ching Tok (personal website here (https://poppopretn.com)). Feedback welcome either via
comments or our contact page (https://isc.sans.edu/contact.html)]
FORUMS 4
Auditing In light of recent challenges brought about by COVID-19, the need for remote work, meetings, training, and learning has IS
increased exponentially. A secure and optimized network would certainly allow users to do their best, despite being
Diary Discussions
unable to work from their usual offices. I was recently asked if the addition of an open-sourced firewall, such as
Forensics
pfSense, would introduce network latency, reduce network speeds and affect productivity. A concern brought up was the
General Discussions
issue of Bufferbloat, which is briefly defined as high latency within the network when multiple types of network traffic
Industry News
[for example bulk traffic (think upload/download)] hinders time-sensitive traffic (think gaming, Zoom/Skype, etc) [1].
Network Security
This phenomenon is usually caused by a router lacking the ability to execute Smart Queue Management (SQM) for the
Penetration Testing network traffic. After some experiments and research, I would like to share that using pfSense’s Traffic Shaper Limiters
Software Security could help optimize networks and address potential Bufferbloat issues. I also further discuss some settings that were
tweaked and tested (vis-à-vis what is usually outlined) that ensure conformance to cybersecurity best practices while
Questions? Feedback?
network performance is optimized.
Use our contact form
or
Note: All networks are unique, and multiple factors can affect network performance. Processing power and availability
report bugs here
of memory do affect the capability of a pfSense firewall, so please make sure to buy/build an appropriately sized
For interactive help firewall that matches the network bandwidth requirements. There are also alternative solutions that address
Bufferbloat issues, such as using Ubiquiti products [2] or Untangle NG Firewall [3]. Last but not least, please remember
and to chat with other
to do a config backup on your pfSense firewall before changing anything, and test the changes made in a test network
users, try our Slack
before deploying them into the production environment.
group.
1) Assess network condition

Firstly, assess your network connection with your ISP. The website DSLReports (https://www.dslreports.com/speedtest)
is suitable to start with. Start the test by selecting the appropriate connection type (E.g. Gigabit/Fiber, Cable, DSL, etc).
Note down the final result, especially the grade for Bufferbloat.
2) Set up Download Limiter

Next, log in to the pfSense Web GUI, and navigate to the Limiters section. (Click on “Firewall” > “Traffic Shaper”).
Following that, click on “Limiters”, and select the “New Limiter” button on the left side of the menu (Please refer to
Figure 1 for the areas to select).
Figure 1: Creating a New Limiter (Download Limiter)
Input the following configurations (Please refer to Figure 2 for a screenshot of the configuration) for the new Limiter:
Enable – Enable limiter and its children (Check this box)
Name – WANDownload (Feel free to name it otherwise, no spaces allowed)
Bandwidth – Input the bandwidth for download as subscribed from your ISP, and select the unit (Bit/s, Kbit/s, Mbit/s).
Leave the “Schedule” field unmodified. (In Figure 2, I used 1000 Mbit/s. Please adjust the values here according to your
ISP subscription, or at least the guaranteed speed you are aware that the ISP is giving you.))
Mask – None
Description – WAN Download Speed (Feel free to name it otherwise, spaces allowed)
Queue Management Algorithm – CoDel
Scheduler – FQ_CODEL (Note: You can set the value of the “quantum” parameter to 300 if you want to give priority to
Voice over IP (VoIP) traffic. In addition, if your pfSense firewall is not constrained by memory, you can change the value
of the “limit” parameter to 20480, and the value of “flows” parameter to 65535. Figure 2 does not show this yet, but you
https://isc.sans.edu/forums/diary/Securing+and+Optimizing+Networks+Using+pfSense+Traffic+Shaper+Limiters+to+Combat+Bufferbloat/27102/ 1/16
02/03/2022 14:42 SANS Internet Storm Center SANS Internet Storm Center - A global cooperative cyber threat / internet security monitor and …
will be able to see these fine-grained settings after you save the limiter.)
Queue Length – 1000 (for a connection with higher bandwidth, this value can be increased to 2000/5000/10000. Start
off with 1000 first.)
ECN – Enable Explicit Congestion Notification (Check this box)
At the bottom of the page, click “Save”. Do not navigate away from the page just yet.
Figure 2: Configuration of Download Limiter
Note: You will get the following message “The traffic shaper configuration has been changed. The changes must be
applied for them to take effect.”. This message can be ignored, as the changes can be applied later in one go once the
download and upload limiters and queues are configured.
3) Set up Download Limiter Queue

After clicking the “Save” button, click on the “Add new Queue” button next to it (Please refer to Figure 3 for the
illustration). Note: pfSense may not direct you to the Queue configuration page even after you select the “Add new
Queue” button. You will notice this if you see the input for bandwidth (this is not needed for Queue configuration). In
this case, select the limiter you have just created, e.g. “WANDownload”, on the left side of the menu. Scroll down, and
click on the “Add new Queue” button again. You will then be redirected to the Queue configuration page.
Figure 3: Selecting the “Add new Queue” Button after Configuring Download Limiter
Input the following configurations (Please refer to Figure 4 for a screenshot of the configuration):
Enable – Enable this queue (Check this box)
Name – WANDownloadQueue (Feel free to name it otherwise, spaces not allowed)
Mask – None
Description – WAN Download Queue (Feel free to name it otherwise, spaces allowed)
Queue Length – (Leave this blank)
At the bottom of the page, click “Save”.
Figure 4: Configuring Download Limiter Queue
4) Set up Upload Limiter

It is now time to set up the Upload Limiter. Select “New Limiter” on the left side of the menu (Please refer to Figure 5
on how things should look like now).
Figure 5: Creating a New Limiter (Upload Limiter)
Enable – Enable limiter and its children (Check this box)
Name – WANUpload (Feel free to name it otherwise, no spaces allowed)
Bandwidth – Input the bandwidth for upload as subscribed from your ISP, and select the unit (Bit/s, Kbit/s, Mbit/s).
Leave the “Schedule” field unmodified. (In Figure 6, I used 500 Mbit/s. Please adjust the values here according to your
ISP subscription, or at least the guaranteed speed you are aware that the ISP is giving you.)
Mask – None
Description – WAN Upload Speed (Feel free to name it otherwise, spaces allowed)
Scheduler – FQ_CODEL (Note: You can set the value of the “quantum” parameter to 300 if you want to give priority to
Voice over IP (VoIP) traffic. In addition, if your pfSense firewall is not constrained by memory, you can change the value
of the “limit” parameter to 20480, and the value of “flows” parameter to 65535. Further explanations available here [4].)
Queue Length – 1000 (for a connection with higher bandwidth, this value can be increased to 2000/5000/10000. Start
off with 1000 first.)
At the bottom of the page, click “Save”. Do not navigate away from the page just yet.
Figure 6: Configuration of Upload Limiter
5) Set up Upload Limiter Queue

After clicking the “Save” button, click on the “Add new Queue” button next to it (Please refer to Figure 3 for the
illustration). Note: pfSense may not direct you to the Queue configuration page even after you select the “Add new
Queue” button. You will notice this if you see the input for bandwidth (this is not needed for Queue configuration). In
this case, select the limiter you have just created, e.g. “WANUpload”, on the left side of the menu. Scroll down, and click
on the “Add new Queue” button again. You will then be redirected to the Queue configuration page.
Enable – Enable this queue (Check this box)
Name – WANUploadQueue (Feel free to name it otherwise, spaces not allowed)
Mask – None
Description – WAN Upload Queue (Feel free to name it otherwise, spaces allowed)
Queue Length – (Leave this blank)
At the bottom of the page, click “Save”.
Figure 7: Configuring Upload Limiter Queue
It is now time to apply the changes made. Click on the “Apply Changes” button on the top of the page (Please refer to
Figure 8). After applying the changes, you will see a message stating “The changes have been applied successfully.”.
Figure 8: Applying Changes Made to Traffic Shaping Limiters and Queues
6) Set up Floating Firewall Rules

Navigate to the pfSense Firewall Rules page (Click on “Firewall” > “Rules”). Following that, click on “Floating” (by
default, you will be brought to the “WAN” rules page). Select the first “Add” button (With reference to Figure 9, the
button highlighted by the red box and with the arrow pointing up).
Figure 9: Navigating to pfSense Floating Rules and Adding First Rule
Floating Rule #1 (Please refer to Figure 10 for a screenshot of the configuration):
• Action: Pass
• Quick: Tick Apply the action immediately on match.
• Interface: WAN
• Direction: out
• Address Family: IPv4
• Protocol: ICMP
• ICMP subtypes: Traceroute
• Source: any
• Destination: any
• Description: Traceroute routing workaround
• Advanced Options: Click on “Display Advanced”.
Scroll down.
• Gateway: Do not use Default. Select your firewall WAN gateway.
Click “Save”. You will see a message “The firewall rule configuration has been changed. The changes must be applied for
them to take effect.” on the top of the GUI (with reference to Figure 11). This message can be ignored, as the changes
can be applied later in one go once all the firewall rules have been configured.
Figure 10: Configuration of Floating Rule for Traceroute Routing Workaround
This rule is required for users who experience their pfSense firewall not being able send out ICMP traceroute [5] when
Traffic Shaper Limiters are used. I personally did not experience this when I tested my configuration, and thus disabled
this Floating Rule.
Select the second “Add” button (With reference to Figure 11, the button highlighted by the red box and with the arrow
pointing down).
Figure 11: Addition of Second Floating Rule
Floating Rule #2 (Please refer to Figure 12 for a screenshot of the configuration):
• Action: Pass
• Quick: Tick Apply the action immediately on match.
• Interface: WAN
• Direction: out
• Address Family: IPv4
• Protocol: ICMP
• ICMP subtypes: Echo reply, Echo Request
• Source: any
• Description: Limiter drop ping traffic under load workaround (Bug #9024)
Scroll down.
Click “Save”.
Figure 12: Configuration of Floating Rule for Limiter dropping Ping Traffic Under Load
This rule is required for users who experience their pfSense firewall dropping ping traffic when it is under heavy load
when Traffic Shaper Limiters are used [6]. I personally did not experience this when I tested my configuration, and thus
disabled this Floating Rule. In addition, a particular note has to be taken on the “Direction” option. While it is tempting
to put “any” instead of “out”, “any” will allow the pfSense firewall to respond to incoming ping requests (something
which many users may not want their firewalls to do so).
Next, we have to create the floating rules that will optimize the network traffic. There are some who suggest the usage
of a single “Pass” action floating rule, and with the “Direction” option configured to “out”. This is not recommended.
Firstly, according to Netgate documentation, Traffic Shaping related rules should utilize the “Match” action [7]. Secondly,
by using only “out” for the “Direction” option, only outgoing traffic will be optimized and incoming traffic ignored. As
such, 2 floating rules (one for incoming, and one for outgoing) should be created. They are as follows:
pointing down) again.
Floating Rule #3 (Incoming Traffic) (Please refer to Figure 13 for a screenshot of the configuration):
• Action: Match
• Interface: WAN
• Direction: in
• Address Family: IPv4 (Select IPv4+IPv6 if the network has IPv6 traffic)
• Protocol: Any
• Source: any
• Description: WAN Incoming Traffic Queue
Scroll down.
• In / Out pipe: WANDownloadQueue / WANUploadQueue
Click “Save”.
Figure 13: Configuration of Floating Rule for Incoming Traffic
pointing down) for the last time.
Floating Rule #4 (Outgoing Traffic) (Please refer to Figure 14 for a screenshot of the configuration):
• Action: Match
• Interface: WAN
• Direction: out
• Address Family: IPv4 (Select IPv4+IPv6 if the network has IPv6 traffic)
• Protocol: Any
• Source: any
• Description: WAN Outgoing Traffic Queue
Scroll down.
• In / Out pipe: WANUploadQueue / WANDownloadQueue
Click “Save”.
Figure 14: Configuration of Floating Rule for Outgoing Traffic
Finally, you can go ahead to click the “Apply Changes” button at the top of the GUI to apply the created firewall rules
(Please refer to Figure 15). After applying the changes, you will see a message “The changes have been applied
successfully. The firewall rules are now reloading in the background. Monitor the filter reload progress.”.
Figure 15: Applying Firewall Rules
7) Re-assess network condition

Finally, re-assess your network connection with your ISP at the DSLReports website.
(https://www.dslreports.com/speedtest). Start the test by selecting the appropriate connection type (E.g. Gigabit/Fiber,
Cable, DSL etc), and the grade for Bufferbloat should have improved with the implementation of Traffic Shaping
Limiters and Queues.
That is all! I hope this guide has been useful in introducing network enhancements while maintaining the security of
networks, especially for pfSense users. Do note that implementing Traffic Shaping may require a bit of further tweaking
(e.g. Download and Upload Limiters, especially the parameter values under the Scheduler option highlighted in Steps 2
and 4 previously) due to various factors (e.g. ISP, geolocation, business requirements, equipment, etc). However, the
steps outlined above should be enough to get you started on optimizing networks (especially home networks) that are
increasingly being used for work, studies and personal entertainment.
[1] https://www.bufferbloat.net/
[2] https://help.ubnt.com/hc/en-us/articles/220716608-EdgeRouter-Advanced-queue-CLI-examples
[3] https://wiki.untangle.com/index.php/Bufferbloat
[4] https://forum.netgate.com/post/807490
[5] https://docs.netgate.com/pfsense/en/latest/troubleshooting/traceroute-output.html
[6] https://redmine.pfsense.org/issues/9024
[7] https://docs.netgate.com/pfsense/en/latest/firewall/floating-rules.html
Update: This post has been updated to address an issue reported by a reader where the order of the traffic flow in the
floating firewall rule was configured in the wrong orientation.
I will be teaching next: Application Security: Securing Web Apps, APIs, and Microservices - SANS Security West 2022
Thread locked
Subscribe J
I would like to mention that the person who maintains the code suggests using Tail Drop for the Queue Management
Algorithm to avoid an error. The result is otherwise the same. https://forum.netgate.com/topic/112527/playing-with-
fq_codel-in-2-4/770
If your internet comes from a cable provider you may want to keep in mind a few other things...
First, if you are on a DOCSIS 3.1 network AND have a DOCSIS 3.1 modem then traffic management algorithms are
already in place in the form of DOCSIS-PIE and you likely don't need this.
(https://ieeexplore.ieee.org/document/7060493).
Second, I have seen some cable ISPs (most recently Spectrum but it varies in different regions) where there is no set or
guaranteed upload bandwidth and it fluctuates wildly. In these areas you may get an upload bandwidth of 50 or 500
Mbps depending on network conditions when you test. So I don't think that a limiter is the right tool in this scenario.
You can still test with queues and see what happens.
It's worth noting that you should check in your ISPs modem for traffic shaping too as it might be easier, work better on
their network, and/or require less investment in money or time.
I do have pfSense and over the years tested each of the traffic shaping algorithms. I find that fq_codel is nice because
you can set it and forget it. I also had good results with PRIQ and FAIRQ but I had to monkey with them a little to get it
right. These days I am lucky enough that I can buy more bandwidth than I can use and that will always be better.
Quote F
Following this article drops my speedtest scores from almost 30Mbit/sec to less than 2Mbit/sec. Didn't affect our An
upload speeds. Disabling the rules causes are speedtest scores to go back up to 30Mbit/sec. What's going on with that?
Quote F
Ye
Quoting Anonymous:
Following this article drops my speedtest scores from almost 30Mbit/sec to less than 2Mbit/sec. Didn't affect our
upload speeds. Disabling the rules causes are speedtest scores to go back up to 30Mbit/sec. What's going on
with that?
IS
There could be multiple factors at play over here. For example, Sam raised excellent points about cable ISPs (e.g.
DOCSIS 3.1 network and modem, and not setting/guaranteeing bandwidth).
Some thoughts: Did you set the download bandwidth according to your ISP subscription? In addition, did your
connection experience Bufferbloat before you configured the Traffic Shaping settings? It is not unusual to experience a
drop in the maximum download/upload speed when Traffic Shaping is configured, but the Bufferbloat score should
have improved after the configuration.
Quote F
I have 2 questions on this configuration.

An
Will this work with dual-wan using a load balance scheme/configuration? If so, would I just set it to the load balance
gateway group?
Second question is on a low bandwidth 12mb down and 1mb up connection what would be the queue length to use?
Would this even work on a lower bandwidth connection at all?
Quote M
11
Ye
Quoting Anonymous:
I have 2 questions on this configuration.
Will this work with dual-wan using a load balance scheme/configuration? If so, would I just set it to the load
balance gateway group?
IS
Second question is on a low bandwidth 12mb down and 1mb up connection what would be the queue length to
use? Would this even work on a lower bandwidth connection at all?
Thank you for your questions.
For your first question, I have yet to be able to test the suggestions I am going to put forth to you and thus it is a
theoretical approach. Firstly, I will have to assume that the 2 WAN bandwidths are the same. If they have different
bandwidths, perhaps you should consider weighing the gateways in the load balance group. For a dual WAN using a
load balancing scheme/configuration, you could try to use the alternative configuration as suggested over here:
forum.netgate.com/topic/57476/… (Please change hxxps to https). In summary, what was done was to use 2 parent
limiters (upload and download) and corresponding child queues (the number will depend on the number of subnets you
have). Upload child queues would have source address mask set download queues would have the destination address
mask set.
Second question - An optimal queue length depends on the speed of your link. For high speed WANs, a value of 1000 is
acceptable. Hence, if your WAN speed is expected to be lower, you could set it to perhaps a value of 250 or 500. Do note
that if your queue is too small, you would get packet loss and decreased throughput. However, if a queue is too large,
you could end up with bufferbloat. I would argue that traffic shaping is even more vital in low bandwith networks with
multiple users since there isn't much for everyone. As long as the bandwidth for limiters is configured correctly (e.g.
speeds that you know you ideally would have from your ISP), it would work (Caveat: Sam's comments about a DOCSIS
3.1 network, and also ISPs that do not have guaranteed upload/download bandwidth).
Ultimately, optimization of networks is an ongoing process and certainly requires a fair bit of experimentation, testing
and technological features of networking devices implemented within in the network.
Quote M
11
← Next Thread
Previous T
Sign Up for Free or Log In to start participating in the conversation!
YouTube Twitter LinkedIn ISC Feed

Pfsense QoS Using CoDel

Uploaded by

Copyright:

Available Formats

Pfsense QoS Using CoDel

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Pfsense QoS Using CoDel

Uploaded by

Copyright:

Available Formats

02/03/2022 14:42 SANS Internet Storm Center SANS Internet Storm Center - A global cooperative cyber threat / internet

/ internet security monitor and …

Threat Level: GREEN

ISC: Keyword, Domain, Port, IP or Head

1) Assess network condition

2) Set up Download Limiter

Figure 1: Creating a New Limiter (Download Limiter)

Enable – Enable limiter and its children (Check this box)

Name – WANDownload (Feel free to name it otherwise, no spaces allowed)

Queue Management Algorithm – CoDel

ECN – Enable Explicit Congestion Notification (Check this box)

Figure 2: Configuration of Download Limiter

3) Set up Download Limiter Queue

Enable – Enable this queue (Check this box)

Name – WANDownloadQueue (Feel free to name it otherwise, spaces not allowed)

Queue Management Algorithm – CoDel

Queue Length – (Leave this blank)

ECN – Enable Explicit Congestion Notification (Check this box)

At the bottom of the page, click “Save”.

Figure 4: Configuring Download Limiter Queue

4) Set up Upload Limiter

Figure 5: Creating a New Limiter (Upload Limiter)

Enable – Enable limiter and its children (Check this box)

Name – WANUpload (Feel free to name it otherwise, no spaces allowed)

Queue Management Algorithm – CoDel

ECN – Enable Explicit Congestion Notification (Check this box)

Figure 6: Configuration of Upload Limiter

5) Set up Upload Limiter Queue

Enable – Enable this queue (Check this box)

Name – WANUploadQueue (Feel free to name it otherwise, spaces not allowed)

Queue Management Algorithm – CoDel

Queue Length – (Leave this blank)

ECN – Enable Explicit Congestion Notification (Check this box)

At the bottom of the page, click “Save”.

Figure 7: Configuring Upload Limiter Queue

Figure 8: Applying Changes Made to Traffic Shaping Limiters and Queues

6) Set up Floating Firewall Rules

Figure 9: Navigating to pfSense Floating Rules and Adding First Rule

Floating Rule #1 (Please refer to Figure 10 for a screenshot of the configuration):

• Action: Pass

• Quick: Tick Apply the action immediately on match.

• Interface: WAN

• Direction: out

• Address Family: IPv4

• Protocol: ICMP

• ICMP subtypes: Traceroute

• Source: any

• Destination: any

• Description: Traceroute routing workaround

• Advanced Options: Click on “Display Advanced”.

Figure 10: Configuration of Floating Rule for Traceroute Routing Workaround

Figure 11: Addition of Second Floating Rule

Floating Rule #2 (Please refer to Figure 12 for a screenshot of the configuration):

• Action: Pass

• Quick: Tick Apply the action immediately on match.

• Interface: WAN

• Direction: out

• Address Family: IPv4