Skip navigation

The past three weeks were no easy, I had a large setup of Mikrotik devices to setup for work, and i am going to briefly point out the issues /  glitches i had with these *damn* boxes.(RouterOS 3.13)

1) Bonding and Eoip –  Does not work. You can bond any number of antennae via EoIP but simply because one pair antennae achieved 17Mbps (TCP), bonding 4 pairs of Antennae and adding them to a bonding  /  Eoip will only give you the bandwidth of a crawling snail. The more antennae and bonding interfaces you add in the equation the worse the situation gets. I have tested it out with TTCP bandwidth tester and results were bad – oh my… so bad. The issue is, the bandwidth (of all four antennae) will be there, but to benefit from this bandwidth you will have to open several connections from a different source to a different destination all the time. (connection tracking and all other crap was disabled – these mikrotik boxes were properly configured)

So, what could  be used instead of bonding or Eoip?  OSPF you may ask? –  Nope!

2)  OSPF load balancing works properly when all interfaces are “wired” (wireless is another story)

I check the cost of the link along with the Administrative Distance settings. (load balancing depends on those), but once again –  no! load balancing over OSPF Mikrotik does not work. The boxes simply decided to pick a link from the four available and send all traffic on that one. Sometimes it even changes link to one of the available ones randomly.

So what is left? All that is required here is simply load balance over 4 wireless links, and yet it seems so difficult to achieve.

The last option is Mangle. There are different flavours of Mangle out here, It seems to be mostly utilised when once wishes to load balance over multiple ISP gateways  –  using NAT. I came across this blog which helped me…

http://mustnofee.com/story/load-balancing-mikrotik-3x

….but did not work since my situation is different. I do not need NAT since all my antennae are using static routing, and Mangle / connection / connection marking did not work since i am not using the above link . This is a LAN, and each end of the setup only sees one IP address. The source or destination.

The Mangle required here is via packet / routing marking:

ip/firewall/mangle > print detail

0   chain=prerouting action=mark-packet new-packet-mark=pktlink1
passthrough=yes in-interface=Eth-Local 1 nth=2,1

1   chain=prerouting action=mark-routing new-routing-mark=mk-link1
passthrough=no in-interface=Eth-Local packet-mark=pkttolink1

2   chain=prerouting action=mark-packet new-packet-mark=pktlink2
passthrough=yes in-interface=Eth-Local

3   chain=prerouting action=mark-routing new-routing-mark=mk-link1
passthrough=no in-interface=Eth-Local packet-mark=pktlink2

IP Route

1 A S  dst-address=192.168.10.1/28 gateway=192.168.1.1 interface=Eth2
gateway-state=reachable distance=1 scope=255 target-scope=10
routing-mark=mk-link2

2 A S  dst-address=192.168.10.1/28 gateway=192.168.2.1 interface=Eth1
gateway-state=reachable distance=1 scope=255 target-scope=10
routing-mark=mk-link1

Although the above worked, for some odd reason the antennas still seem to be giving me issues. The ptp antennae are running at 50db signals, with a registration of 54mbps/54mbps. So if the wireless peers are working correctly, why is it still necessary to open multiple connections to utilize the full capacity of my 4 links?

(The reason is that by opening multiple links, the bandwidth seems to increase for the simple reason that there are more packets bouncing around for retransmissions but the total bandwidth which can be actually utilised is minimal. Seeing large numbers does not mean that the packets are actually arriving without errors and that the total usable bandwidth is that “large figured number”.

Continued on next thead  ” Mikrotik –  Load Balancing Over Multiple Interfaces (Part 2) “

Advertisements

16 Comments

  1. Hi dear,
    Yes, OSPF is not for loadbalancing, it´s for dynamic routing… So, I´m also using Mangle (NTH) for load-balance.

    []´s
    Giselle
    from Brazil 😉

  2. nice…. post… 😀
    visit also http://mikrotik-guide.net

  3. Are you sure your antennas are not interfering with each other, causing them to drop a significant number of packets? Having four antennas close together in the same band and NOT have them interfere is no easy task.

    Try using only two antennas, cross polarized and see if your mangle load-balancing does not work better then.

    Regards, JJF.

    • Hi, Yes we had tried several scenarios and eventually it all boiled down to out of order packets. The packets were not being dropped but rather being received out of order. Mikrotik fail to reorder packets and leaves this job to the end systems. Having so many packets out of order drops drastically the actual throughput of the link. Bandwidth testing on udp works great (this already rules out interference). When we switch to TCP, the values on winbox still look good but when measured via actual file transfers only around 1/8 of the actual throughput was actually useful. 7/8 were retansmissions. (with 4 links 80 Mbps UDP was seen. When tested on TCP according to winbox it was actually 40Mbps – actual TCP file transfer was 8Mbps over 4 links) The difference between mangle and bonding is simply the fact that bonding performs a link check via arp to ensure that the link is till up. If not, it cuts such link out from the equation and performs the round robin on the remaining links. With mangle, if a link is dropped, packets are lost till manual intervention on the mangle rules is performed.

      I had personally confirmed with Mikrotik support that packet reordering is not performed by these boxes and alternative means have to be found.

      Thanks for your input though!

      Kirk

  4. i think u should try nstreme protocol if u have powerfull enough routeboard, nstreme seems to help in scenarios like that when crc error overfllow entire traffic

    my sugestion nstreme and RIP to keep it simple

    • It has been a while that i had tested this setup so i forgot the details but nstreme had increased the latency between the antennae and more packets were getting lost. I think it was due to the fact that the link was 1.5Km – nstreme works properly over shorter distances. Regarding RIP, there is no point in using it since it does not support load balancing and static routes with bonding should have done the job well. Was not the case! All routing was static over the 8 pair of antennae and the 4 load Balancers. I also had VRRP planned but since the whole project was a major flop, i didn’t even get down to test VRRP properly.

      The solution – get Cisco gear if you want a decent setup!

      As always, feedback much appreciated – i still have scars left on my body due to this project!

      Kirk

  5. hello there, just wished to say thanks alot for this post, it assisted me become aware of something I did not given a lot of attention to it previously.

  6. Hi

    Tanx a lot brother

    I am trying since last 30 days and searching on net for load balancing so finally I found your website and with the help of this i fix the problem.

    Tanx and Regards

    Engineer Rehmat ullah (Assistant Manager Nexlinx Pakistan)

    • sir rehmat bhai aap brain net sa ha ?
      4MB connection ki kiya prise ha

    • sir assalavalakum Rehmat ullah bhai mere pas mikrotik pc base router ha or ma os ma loadbalancing karna chahta hoo mere pas 2 dsl modem ha ek default getway ha 192.168.1.5 dusray ki 192.168.1.1 please aap mere madad karay

  7. Rehmat is a liar.He is not an engineer.he has done bcs from agriculture uni…………….haha if a person is searching for loading balancing for 30days then how can this tato will help you

  8. Man did f.sc from icms cannot be engineer.hahaha
    dedicated to Rehmat ullah bangash.

  9. I have gone through the entire of this blog with dismay, Mikrotik bonding works complementarily very well in all scenarios, Kindly redesign your methodology in your routing structure, mikrotik will offer you all the tools you need to perform the most extreme routing/bonding and wireless data transfer irrespective of the version of your mikrotik.

    On the contrary too much trial and errors in your mikrotik box might lead to collision of commands, when thigs go hey wire backup and reset and start all over again.

    you have many options in bonding depending on your scenario and certain limitation e.g TCP window , do you have control on both ends..

    nstreme algorithm works irrespective of the distance, short or far it is the best so far on wireless links especially backhauls.

    never put your antennas close together but if you must then make sure they are all operating in different frequencies that are non-distructive
    ( this is how mimo relies).

    We have implement extensive networks using mikrotik in universities, mostly using load balabancing, multiple ISPs, OSPF and MPLS alll working in unison. the total number of mikrotik equipment deployed were 30 APs, 10 ptp and 4 core routers X86. all our Mirotik are running versions between 3.0 to 4.XX, services are hotspot,ppoe,VOIP,EOIP,VLAN. Longest link 43 KM

    My rule No. 1. Document your structured network on paper first before hitting the road on mikrotik.

    My rule No. 2 Never allow IP addresses to roam/cruize in your links/networks, YOU MUST USE TUNNELS,VLANS etc unless it is a small household setup where security and the rest is not a factor.

    My rule No. 3 Read Mikrotik wikis before turning to blogs and googling

    • Glad that it worked for you. But it did not for us. Did you sniff your traffic and see what exactly was passing? Did you get the bandwidth you hand in mind? We did not. Support did not help us out either. No matter the months and months of sending configurations. It was clear that certain functionality was not built in at that time and was not going to be. This functionality was no rocket science but simply packet reordering over wireless as this was the issue. The design changed in many ways and no matter what, the issue boiled down to be a lack of functionality. Mikrotik can do a lot, but not everything and maybe not everything at the level of details certain firms require. Anyway, it has been a long time ago and obviously this does not apply any longer. We have abandoned Mikrotik since then with a better brand and our result have been achieved – his obviously reflected in the price tag!

  10. Dear Bro,…
    I have 3 WAN each have 8mbps for each, total 24mbps , i want to combine with RB750Up, can i ? all of the connection are PPPOE type.. can u give me the script or tips ? Please help me.. i`m newbie in mikrotik .. my email address is phokhwar.pk@gmail.com ..

  11. Look nice ………………see more http://mikrotikroutersetup.blogspot.com


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: