MUM Ho Chi Minh, Vietnam

January 2019

Sao Thiên Vương - http://switch-router.com 1

 My name: Binh Nguyen Dang Thai
 Email: binhndt@uranus.com.vn
 Cellphone: 0909006026
 Working for: URANUS INFORMATION
TECHNOLOGY COMPANY

Sao Thiên Vương - http://switch-router.com 2

 URANUS has been focusing on wireless
networking market since 2015.
 Became MikroTik distributor since 2016.
 Website: http://uranus.com.vn
 Support website: http://forum.uranus.com.vn

Sao Thiên Vương - http://switch-router.com 3

 Using PCC (Per Connection Classifier) to load
balancing multiple Internet links correctly.
 Solving common problems beside PCC load
balancing.

Sao Thiên Vương - http://switch-router.com 4

 Load Balancing is a technique to distribute the
workload across two or more network links in
order to maximize throughput, minimize
response time, and avoid overload.
 Distribution may be symmetrical or asymmetrical
depending on rate of network links.
 Useful when the downstream bandwidth
requirement to a single routing device exceeds the
capabilities of a single network link.
 Using multiple network links with load balancing,
instead of single network links, may increase
reliability through redundancy
Sao Thiên Vương - http://switch-router.com 5
 PCC divides the traffic into streams and then
uses routing rules to sort the traffic evenly (or
not) across multiple network links.
 This is done by:
 Using a hashing algorithm to first sort the traffic
based on source address, source port, destination
address, destination port or various combinations
thereof.
 Using packet marking and marking route and
several routing tables to ensure traffic follows a
specified route out the specified WAN interface.

Sao Thiên Vương - http://switch-router.com 6

 MikroTik RouterOS is extremely powerful and
configurable, so this can be a double edged
sword, several possible solutions to the same
problem.
 Each has multiple moving pieces
 Greatest success with any solution by
understanding the pieces and what they do.

Sao Thiên Vương - http://switch-router.com 7

Terms must to understand
 Packet: The container for our data, header and
payload.
 Connection: “Conduit” through which host to host
communication occurs, based on Src/Dst addresses
and ports
 Mangle Facility: is a kind of 'marker' that marks
packets for future processing with special marks.
They identify a packet based on its mark and
process it accordingly. Mangle marks exist only
within the router, they are not transmitted across the
network.
Sao Thiên Vương - http://switch-router.com 8
Terms must to understand
 PCC: PCC matcher will allow to divide traffic
into equal streams with ability to keep packets
with specific set of options in one particular
stream (we can specify this set of options from
src-address, src-port, dst-address, dst-port)

Sao Thiên Vương - http://switch-router.com 9

 PCC (cont.) – How does this work?
PCC takes selected fields from IP header, and
with the help of a hashing algorithm converts
selected fields into 32-bit value. This value then is
divided by a specified Denominator and the
remainder then is compared to a specified
Remainder, if equal then packet will be captured.
You can choose from src-address, dst-address,
src-port, dst-port from the header to use in this
operation.

Sao Thiên Vương - http://switch-router.com 10

 PCC (cont.) How does this work (cont.)

Sao Thiên Vương - http://switch-router.com 11

 PCC (cont.) – Example: 3 WAN Connections
3 PCC rules required
ValuesToHash Denominator Remainder

1st WAN connections

2nd WAN connections
3rd WAN connections

 The 1st line means "produce the output of the hash function given the
packet’s both IP addresses, divide it by 3 and if the remainder is 0, perform
the action of marking the connection as WAN1“
 The 2nd rule means "produce the output of the hash function given the
packet’s both IP addresses, divide it by 3 and if the remainder is 1, perform
the action of marking the connection as WAN2“
 The 3rd rule means "produce the output of the hash function given the
packet’s both IP addresses, divide it by 3 and if the remainder is 2, perform
the action of marking the connection as WAN3“

Sao Thiên Vương - http://switch-router.com 12

 PCC (cont.) - How to set PCC, Remember?
2 WAN connections:
2 / 0 First WAN
2 / 1 Second WAN
3 WAN connections:
3 / 0 First WAN
3 / 1 Second WAN
3 / 2 Third WAN and so on...

Sao Thiên Vương - http://switch-router.com 13

 PCC (cont.): Where does it found?

Sao Thiên Vương - http://switch-router.com 14

Terms must to understand
 Routing Table: Route rules, the rules the
router uses to determine what to do with a
packet. By comparing the destination address
in the packet to the list of routes, the router
decides which interface to send the packet out.
By adding a routing mark with mangle, we can
have multiple routing tables!

Sao Thiên Vương - http://switch-router.com 15

 Routing Table (cont.)

Sao Thiên Vương - http://switch-router.com 16

Terms must to understand
 Routing-mark:
 RouterOS attribute assigned to each packet
 Routing-mark can be changed in firewall mangle
facility just before any routing decision:
 chain Prerouting – for all incoming traffic
 chain Output – for outgoing traffic from router
 Every new routing mark has its own routing
table with the same name
 By default all packets have the “main” routing
mark
Sao Thiên Vương - http://switch-router.com 17
Terms must to understand
 MikroTik packet low: This manual describes
the order in which an IP packet traverses
various internal facilities of the router and
some general information regarding packet
handling, common IP protocols and protocol
options.

Sao Thiên Vương - http://switch-router.com 18

 MikroTik packet low (cont.)
 Overall Packet flow Diagram (RouterOS v6):
https://wiki.mikrotik.com/wiki/Manual:Packet_Fl
ow

Sao Thiên Vương - http://switch-router.com 19

Scenario: One router MikroTik RB1100AHx2,
many clients, 3 FTTH links

Sao Thiên Vương - http://switch-router.com 20

 Set up the basic portion of the network
 Private IP address on Bridge interface, Bridge interface is
logical bridge or CPU bridge, included ether4 – ether13.
 DHCP Server on Bridge interface
 DNS server
 Configure PPPoE clients for 3 WAN links, in this case on
ether1, ether2, ether3
 Masquerade for 3 PPPoE clients
 Defines default route for 3 WAN links
 Firewall if required
 Create load balancing part of the configuration:
 Mangle rules
 Routing tables

Sao Thiên Vương - http://switch-router.com 21

 Assigns an IP address and ports for Bridged interface
/ip address
add address=192.168.100.1/24 interface=bridge network=192.168.100.0
/interface bridge port
add bridge=bridge hw=no interface=ether6
add bridge=bridge hw=no interface=ether9
add bridge=bridge hw=no interface=ether8
add bridge=bridge hw=no interface=ether10
add bridge=bridge hw=no interface=ether11
add bridge=bridge hw=no interface=ether12
add bridge=bridge hw=no interface=ether13
add bridge=bridge hw=no interface=ether7
add bridge=bridge hw=no interface=ether4
add bridge=bridge hw=no interface=ether5

Sao Thiên Vương - http://switch-router.com 22

 Configure PPPoE clients for 3 WAN links
/interface pppoe-client
add disabled=no interface=ether1 keepalive-timeout=60 max-
mru=1480 max-mtu=\
1480 mrru=1600 name=pppoe-out1 password=T9a799
user=t008_ftth_abc
add disabled=no interface=ether2 keepalive-timeout=60 max-
mru=1480 max-mtu=\
1480 mrru=1600 name=pppoe-out2 password=r23hnV
user=t008_ftth_xyz
add disabled=no interface=ether3 keepalive-timeout=60
name=pppoe-out3 \
password=hT2t5g user=t008_ftth_123

Sao Thiên Vương - http://switch-router.com 23

 Masquerade for 3 WAN links
/ip firewall nat
add action=masquerade chain=srcnat out-interface=pppoe-out1
add action=masquerade chain=srcnat out-interface=pppoe-out2
add action=masquerade chain=srcnat out-interface=pppoe-out3
Where can we configure on Winbox?

Sao Thiên Vương - http://switch-router.com 24

 Define default routes for 3 WAN links
/ip route
add check-gateway=ping distance=1 gateway=pppoe-out3
add check-gateway=ping distance=2 gateway=pppoe-out2
add check-gateway=ping distance=3 gateway=pppoe-out1
What does look like on main routing tables

Sao Thiên Vương - http://switch-router.com 25

 Create load balancing part of the configuration:
 Define address list for Local Address Network
(LAN)
 Mangle rules

Sao Thiên Vương - http://switch-router.com 26

 Define LAN address list
/ip firewall address-list
add address=192.168.100.0/24 list=LAN

Sao Thiên Vương - http://switch-router.com 27

 Bypass mangle rules for local traffic
We need to ensure that any traffic going to our
local bypasses all the ‘line balancing’ rules.
We do this with an action of ‘accept’. In this case,
internal traffic is on LAN address list.
/ip firewall mangle
add action=accept chain=prerouting
comment="Accept traffic from LAN" \
dst-address-list=LAN src-address-list=LAN

Sao Thiên Vương - http://switch-router.com 28

 Topology:
 In this topology, there are 4
possible traffic flows
 WAN -> Router
 Router ->WAN
 WAN -> LAN
 LAN -> WAN

Sao Thiên Vương - http://switch-router.com 29

Taking care of incoming connections
 When a connection is initiated from the
internet through one of the ISPs we need to
ensure that this connections is replied through
the same ISP (from the same public IP)
 We need to mark these connections, and then
put them in the proper routing table.

Sao Thiên Vương - http://switch-router.com 30

Router marking – WAN -> Router
 Catch the connection from internet to the router, and mark them

add action=mark-connection chain=input comment="Mark all

connections that are initiated from outsite" connection-mark=no-mark
in-interface=pppoe-out1 new-connection-mark=WAN1-to-ROS
passthrough=no

add action=mark-connection chain=input connection-mark=no-mark

in-interface=pppoe-out2 new-connection-mark=WAN2-to-ROS
passthrough=no

add action=mark-connection chain=input connection-mark=no-mark

in-interface=pppoe-out3 new-connection-mark=WAN3-to-ROS
passthrough=no

Sao Thiên Vương - http://switch-router.com 31

Router marking – WAN -> Router (cont.)
 Then put these connections into the proper routing tables.

add action=mark-routing chain=output comment="Mark

routing for router's replies" connection-mark=WAN1-to-ROS
new-routing-mark=WAN1 passthrough=no

add action=mark-routing chain=output connection-

mark=WAN2-to-ROS new-routing-mark=WAN2
passthrough=no

add action=mark-routing chain=output connection-

mark=WAN3-to-ROS new-routing-mark=WAN3
passthrough=no

Sao Thiên Vương - http://switch-router.com 32

Router marking – WAN -> Router (cont.)
 What does look like on Winbox

Sao Thiên Vương - http://switch-router.com 33

Taking care of the LAN
 Same principle applies to the LAN

 Connections initiated from the internet through

one ISP, should be replied to through the same
ISP

Sao Thiên Vương - http://switch-router.com 34

LAN marking

add action=mark-connection chain=forward

comment="Mark all connections for NAT inbound"
connection-mark=no-mark in-interface=pppoe-out1 new-
connection-mark=WAN1-to-LAN passthrough=no

add action=mark-connection chain=forward connection-

mark=no-mark in-interface=pppoe-out2 new-connection-
mark=WAN2-to-LAN passthrough=no

add action=mark-connection chain=forward connection-

mark=no-mark in-interface=pppoe-out3 new-connection-
mark=WAN3-to-LAN passthrough=no
Sao Thiên Vương - http://switch-router.com 35
LAN marking (cont.)

add action=mark-routing chain=prerouting connection-

mark=WAN1-to-LAN new-routing-mark=WAN1
passthrough=no src-address-list=LAN

add action=mark-routing chain=prerouting connection-

mark=WAN2-to-LAN new-routing-mark=WAN2
passthrough=no src-address-list=LAN

add action=mark-routing chain=prerouting connection-

mark=WAN3-to-LAN new-routing-mark=WAN3
passthrough=no src-address-list=LAN

Sao Thiên Vương - http://switch-router.com 36

Incoming connections - done
 We have ensured that when a connection
from the internet to our router, or services
inside of our network is established, it works.
LAN – partially done
 So what about connections outgoing from our
LAN to the internet?
 These we actually want to load-balance

Sao Thiên Vương - http://switch-router.com 37

LAN -> WAN mangle

add action=mark-connection chain=prerouting comment="PCC rules“

connection-mark=no-mark dst-address-list=!LAN dst-address-
type=!local new-connection-mark=LAN-to-WAN1
passthrough=yes per-connection-classifier=both-addresses:3/0
src-address-list=LAN

add action=mark-connection chain=prerouting connection-mark=no-

mark dst-address-list=!LAN dst-address-type=!local new-
connection-mark= LAN-to-WAN2 passthrough=yes per-
connection-classifier=both-addresses:3/1 src-address-list=LAN

add action=mark-connection chain=prerouting connection-mark=no-

mark dst-address-list=!LAN dst-address-type=!local new-
connection-mark=LAN-to-WAN3 passthrough=yes per-
connection-classifier=both-addresses:3/2 src-address-list=LAN

Sao Thiên Vương - http://switch-router.com 38

 LAN -> WAN mangle (cont.)

add action=mark-routing chain=prerouting comment="Mark

routing for upload packets from marked connections“
connection-mark=LAN-to-WAN1 dst-address-list=!LAN
new-routing-mark=WAN1 passthrough=no src-address-
list=LAN

add action=mark-routing chain=prerouting connection-mark=LAN-

to-WAN2 dst-address-list=!LAN new-routing-mark=WAN2
passthrough=no src-address-list=LAN

add action=mark-routing chain=prerouting connection-mark=LAN-

to-WAN3 dst-address-list=!LAN new-routing-mark=WAN3
passthrough=no src-address-list=LAN

Sao Thiên Vương - http://switch-router.com 39

 Mangle in GUI

Sao Thiên Vương - http://switch-router.com 40

 Routing tables

Sao Thiên Vương - http://switch-router.com 41

 I can access just 1 WAN interface from Internet.
 I can not access LAN services from Internet via
inbound NAT.
 How to load balance for different rate WAN
links.

Sao Thiên Vương - http://switch-router.com 42

 https://wiki.mikrotik.com/wiki/Manual:PCC
 https://blog.linitx.com/load-balancing-
multiple-internet-connections/
 MikroTik RouterOS Workshop Load Balancing
Best Practice by Warsaw, MUM Europe 2012.
 Bandwidth-based load-balancing with failover.
The easy way by Tomas Kirnak.
 Load Balancing Using by Steve Discher.

Sao Thiên Vương - http://switch-router.com 43

 THANK YOU!
Sao Thiên Vương - http://switch-router.com 44