Building a private mesh network

Well it was bound to happen sooner or later, but working on projects at the office I have segmented networks for various things, and the stuff I host only has to talk to one specific thing out on the internet.  But things change, and flexibility is demanded as now I suddenly have half a dozen VPS’s out on the internet, that not only need to talk back to my office, but also need to talk to eachother.  And of course adding that n+1 becomes tedious, and worse when one host goes down, and you have people pointing the wrong way.  If only there was a solution…

And there is, as this ‘problem’ was solved forever ago with this magical thing called routing protocol.  Before I was doing OpenVPN with a network to network type setup, but now I want to dynamically route, and considering it’s a small network, I’m free to use any protocol, even RIP is fine, as I have about 15 networks to advertise.  In the end I went with BGP for really no real reason.  RIP/OSPF,ISIS would have worked just as well.

The first thing is that to configure OpenVPN is that I didn’t want the normal ‘shared network’ setup, nor the tun style where OpenVPN intercepts the invalid destination address.  While it does bring up links, and work for dynamic setups, I’m just bringing up static links that are point to point, and single host only.  Also BGP wants the source and destination originating address to match, so a tunnel interface that is, but expects the destination to be isn’t going to work.  So I went with the old p2p style config.

As a template I used this site right here:

First generate a static key:

openvpn --genkey --secret static.key

Next setup the config for the ‘server’ side.  In this case, each of my VPS’es is a server since they all have registered addresses, and don’t move as often.  My office is going to move soon, and in the off even that I need to travel with my office setup it’s nice to bring it online wherever, and have it connect.  At the same time if my office is down, this is why I want the VPSes to be able to talk to eachother.

Server config:

mode p2p
port 1194 #  We listen to port 1194
dev tun # possibly tun0 Linux
proto udp # protocol UDP, TCP: proto tcp-server

ifconfig #  Local IP <-> Remote IP

# Here adjust the path to point to the generated Key
secret /etc/openvpn/static.key

ping 10 # Sends every 10 seconds a ping to the remote site
ping-restart # 180 after 3 minutes without pings reconnecting the remote
ping-timer-rem # only after we've let another peer ping-restart

verb 3 # increase to debugging
mute 50 # uncomment to debugging

Client config:

mode p2p
remote 1194 # hostname / external IP of another peer, port accordingly
proto udp # protocol UDP, TCP: proto tcp-client
dev tun #  possibly tun0 Linux

ifconfig #  Local IP <-> Remote IP

# Here adjust the path to point to the generated Key
secret /etc/openvpn/static.key

ping 10 # Sends every 10 seconds a ping to the remote site
ping-restart # 180 after 3 minutes without pings reconnecting the remote
ping-timer-rem # only after we've let another peer ping-restart

verb 3 # increase to debugging
mute 50 # uncomment to debugging

Now with the config’s in place, remembering to save with the filenames ‘server.conf’ and ‘client.conf’ in the /etc/openvpn directory, I was ready to restart the OpenVPN service (service openvpn restart) and now I could ping!

tun1 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr: P-t-P: Mask:

and a quick ping…

root@Office-OpenVPN:/etc/openvpn# ping
PING ( 56(84) bytes of data.
64 bytes from icmp_req=1 ttl=64 time=1.66 ms
64 bytes from icmp_req=2 ttl=64 time=3.39 ms

Now for the routing protocol.  I went with quagga, as it is descended from zebra, and one that I’ve used before.

Configuration is pretty straight forward.  first the daemons.conf file needs to be edited for what services you are going to run.  In this case I want zebra & bgpd.


Next, the zebra.conf file.  The passwords are plaintext, I wouldn’t of course paste my real passwords:

hostname office
password zebra
enable password zebra

And next is my bgpd.conf file, for my office:

hostname office
password zebra
router bgp 8888
bgp router-id
neighbor remote-as 555
log file /var/log/quagga/bgpd.log
!log stdout

Everything is the same on the VPS, except for it’s BGP config, which is the following:

hostname vps1
password zebra
router bgp 555
bgp router-id
neighbor remote-as 8888
log file /var/log/quagga/bgpd.log
!log stdout

So now I’ve setup a p2p connection, and now defined the networks that I’m going to share from my office, in this case it’s,, and  The VPS is going to share it’s network.

Now to start up the router with a ‘service quagga restart’ and all being well I can now talk to the BGPD.

telnet localhost 2605
Trying ::1…
Connected to localhost.
Escape character is ‘^]’.

Hello, this is Quagga (version
Copyright 1996-2005 Kunihiro Ishiguro, et al.

User Access Verification


Now to check for my bgp neigbours

office> ena
office# show bgp neighbors
BGP neighbor is, remote AS 555, local AS 8888, external link
BGP version 4, remote router ID
BGP state = Established, up for 3d21h32m
Last read 00:00:33, hold time is 180, keepalive interval is 60 seconds
Neighbor capabilities:
4 Byte AS: advertised and received
Route refresh: advertised and received(old & new)
Address family IPv4 Unicast: advertised and received
Message statistics:
Inq depth is 0
Outq depth is 0
Sent Rcvd
Opens: 1 0
Notifications: 0 0
Updates: 18 1
Keepalives: 5613 5612
Route Refresh: 0 0
Capability: 0 0
Total: 5632 5613
Minimum time between advertisement runs is 30 seconds

For address family: IPv4 Unicast
Community attribute sent to this neighbor(both)
1 accepted prefixes

Connections established 1; dropped 0
Last reset never
Local host:, Local port: 179
Foreign host:, Foreign port: 35722
Nexthop global: ::
Nexthop local: ::
BGP connection: non shared network
Read thread: on Write thread: off


And how about the routes?

office# show ip bgp
BGP table version is 0, local router ID is
Status codes: s suppressed, d damped, h history, * valid, > best, i – internal,
r RIB-failure, S Stale, R Removed
Origin codes: i – IGP, e – EGP, ? – incomplete

Network Next Hop Metric LocPrf Weight Path
*> 0 32768 i
*> 0 32768 i
*> 0 0 555 i
*> 0 32768 i

Total number of prefixes 4

Well isn’t that great!  For each additional connection on my client side, I change the client port.  It doesn’t matter what number I choose as my client VPN machine isn’t exposed on the internet, only the OpenVPN servers on the VPSes.  I can also combine this with a traditional OpenVPN setup where my clients get addresses on my Office Lan, and can now access all of my VPSes at once.

All I have left to do is define the additional servers into the mesh, and add in the BGP peers.  And doing it this way two VPSes in the same data centre now have a 1ms ping, instead of traversing the internet to my office, and back up.  Plus they can stay in communication if my office, or even their internet is down, as long as the internal communications of the data centre is fine.

More fun with GCC 6.1

So after looking at the -Ofast flags in that utterly unfair GCC 1.4 vs GCC 5.1, and 6.1 , I thought I’d try to build Cockatrice III with it.  Everything went well, and I had a build in no time.

I always hated how I had to massively downsample the audio so I could at least hear things, so I thought I’d try to put them back to 44100Khz, 16bit stereo.  And while compiling, older GCC runs fine, while 6.1 throws this run error!

../SDL/audio_sdl.cpp:57:43: error: narrowing conversion of '-1404829696' from 'int' to 'uint32 {aka unsigned int}' inside { } [-Wnarrowing]
 uint32 audio_sample_rates[] = {44100 << 16};
makefile:104: recipe for target 'obj/audio_sdl.o' failed
make: *** [obj/audio_sdl.o] Error 1

Well it turns out that it’s getting truncated as the audio_sample_rates are defined as an unsigned int, but it really want’s to be a regular integer.  So I changed the type, and now I have high def audio!  While I was in there, I fixed some stupid typos in the keyboard so I can actually use vi in MacMiNT.

It’s still in 256 colors, I’m missing something fundamental as to why it’s not working but I just don’t have enough time to mess with it today.

For anyone who cares, the Win32 binary package is on sourceforge.

Stupid GCC error

So while mixing and matching GCC on Windows, I ran into this issue with stdio.h of all things:

In file included from hi.c:1:0:
\mingw\include/stdio.h:191:65: error: unknown type name ‘size_t’
_CRTIMP int __cdecl __MINGW_NOTHROW setvbuf (FILE*, char*, int, size_t);

Good grief how could stdio be all screwed up?

Well, it turns out it was my fault.  I had not rebuilt things like SDL, or copied over pcap so I figured I could incorrectly just point a -I to my old includes, and the new gcc would post-pend all use -I stuff.  NOPE it pre-pends them, meaning the old headers now take priority.  And wouldn’t you know it, things have drifted between versions.  So I just manually copied the files and libraries and all was well.

Googling around I did see other people with broken stdio.h but nobody posted the solution.  I guess it’s to embarrassing, but here we are.

DJGPP memory fouled

free: memory fouled

free: memory fouled

If you ever get this fun error while trying to exit or shell a program with DJGPP:

combinelo: memory fouled

free: memory fouled

You are mixing libraries built with a different version of the DJGPP runtime.  In my case, I built the allegro library with GCC to use the older binutils but I’m cross compiling with GCC 5, so it’s using the newer runtime.

In my case I just needed to replace the contents of the lib directory with, and I was good to go.  I guess I could re-compile allegro, but that takes the better part of forever on DOSBox, and once was good enough.

GCC 6.1.0 for MinGW

Since MS-DOS has it’s 6.1.0 I was dying to re-build stuff for 6.1.0.  As I don’t want to build it myself I did mange to find a version on

I rebuilt Cockatrice III, and that seems to be running fine.

Continuing from my GCC 1.40 on Windows, here is version 6.1.0’s scores:

Dhrystone(1.1) time for 500000000 passes = 53
This machine benchmarks at 9433962 dhrystones/second

Dhrystone(1.1) time for 500000000 passes = 18
This machine benchmarks at 27777777 dhrystones/second

Dhrystone(1.1) time for 500000000 passes = 16
This machine benchmarks at 31250000 dhrystones/second

Dhrystone(1.1) time for 500000000 passes = 13
This machine benchmarks at 38461538 dhrystones/second

As you can see the scores are actually slower than GCC 5.1.0.  Well that was kind of surprising.  Since I almost never read the manual, I did find out about -Ofast vs -O3/-O2, and yes, it is much faster!  So I guess I should recompile everything with GCC 5.1.0 … But I do prefer the diagnostic messages in GCC 6, it’s very CLANG like.

UAE 0.8.29 on Windows

Captain Blood on UAE 0.8.29

Captain Blood on UAE 0.8.29

This one should have been much easier to build, it has support for SDL built in, however the include files are a nested mess, and configure fails part of the way in the process leaving the source kinda messy.  But a few hours over a couple of days, and here we are.

This version doesn’t run at warp speed, has sound, and is great.    It wants a config file though.  You can find the specs in the readme, but something like this:


works fine.  This later (and seemingly last) branch of UAE  incorporates lots from WinUAE, except for the JIT.  It’s dated 2008, so it does include support for the 68030, 68040, and the 68881 and 68882.  It doesn’t have MMU support, so things like Linux/AMIX/NetBSD/Enforcer are out of the question.

I dumped my source tree over on sourceforge, as I’m more so interested that this builds using MinGW.