32bit / 64bit GDB collision

So I’m trying to force a break in a program to look at the call stack.  Normally pretty easy stuff as far as adding breakpoints or the random i=j/0 type thing to force a div0 exception etc.

But I’ve been going crazy with this fun error:

gdb: unknown target exception 0x4000001f at 0x453b1c

And I should have checked my path earlier, as even though I’m using the 32bit version of GCC, the TDM-GCC project bundles their 32bit GDB as gdb32.exe .  Ouch.  The other hint of course was in the banner:

GNU gdb (GDB) 7.9.1
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show copying”
and “show warranty” for details.
This GDB was configured as “x86_64-w64-mingw32”.

Yeah, further oops, in actually paying attention to how it executes:

(gdb) r
Starting program: C:\temp\dynamips\dynamips.exe -P 1700 -s 0:0:gen_eth:”\Device\NPF_{2F7A0E3F-0908-4489-8117-475A6AD299A6}” C1700-Y-.BIN -j
[New Thread 2696.0x22b0]
warning: `C:\WINDOWS\SYSTEM32\ntdll.dll’: Shared library architecture i386:x86-64 is not compatible with target architecture i386.
warning: `C:\WINDOWS\system32\wow64.dll’: Shared library architecture i386:x86-64 is not compatible with target architecture i386.
warning: `C:\WINDOWS\system32\wow64win.dll’: Shared library architecture i386:x86-64 is not compatible with target architecture i386.
warning: Could not load shared library symbols for WOW64_IMAGE_SECTION.
Do you need “set solib-search-path” or “set sysroot”?
warning: Could not load shared library symbols for WOW64_IMAGE_SECTION.
Do you need “set solib-search-path” or “set sysroot”?
warning: Could not load shared library symbols for NOT_AN_IMAGE.
Do you need “set solib-search-path” or “set sysroot”?
warning: Could not load shared library symbols for NOT_AN_IMAGE.
Do you need “set solib-search-path” or “set sysroot”?
warning: `C:\WINDOWS\system32\wow64cpu.dll’: Shared library architecture i386:x86-64 is not compatible with target architecture i386.
[New Thread 2696.0xabc]

Great.

So it’s not some exciting bug, but rather the 64bit version of gdb can kick off a 32bit exe, but it’ll get lost inside.  The proper 32bit GDB looks like this:

C:\temp\dynamips>gdb32.exe dynamips.exe
GNU gdb (GDB) 7.9.1
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show copying”
and “show warranty” for details.
This GDB was configured as “mingw32”.

And sure enough the call stack works fine.

Breakpoint 1, 0x00453b1c in mpc860_fec_set_nio ()
(gdb) bt
#0 0x00453b1c in mpc860_fec_set_nio ()
#1 0x004827e2 in dev_c1700_mb_eth_set_nio ()
#2 0x0044abdf in cisco_card_enable_all_nio ()
#3 0x0044b9df in vm_slot_enable_all_nio ()
#4 0x0044bb0e in vm_slot_init ()
#5 0x0044bb36 in vm_slot_init_all ()
#6 0x00481b5d in c1700_init_platform ()
#7 0x00481dc3 in c1700_init_instance ()
#8 0x004155b3 in vm_init_instance ()
#9 0x004125dc in main ()
(gdb)

Wasn’t that fun?!

One million packets served!

one million ICMP packets!

Success rate is 100 percent (1000000/1000000)!

So over in my work on porting Dynamips to MinGW, I’ve created a version of SLiRP that sends and receives data over UDP.  In retrospect, something I should have done a long time ago, as it makes troubleshooting it easier as now if it were to crash it’s a stand alone program, so it won’t crash the emulator.

The good news is that I’ve been able to copy files into the virtual router using HTTP.  I’ve even been able to access my OS/2 machine over FTP and load a file!

R1#copy ftp://10.12.0.14/README disk0:
Destination filename [README]?
Accessing ftp://10.12.0.14/README…
Loading README !
[OK – 76743/4096 bytes]

76743 bytes copied in 8.740 secs (8781 bytes/sec)

R1#dir disk0:README
Directory of disk0:/README

12 -rw- 76743 Sep 18 2015 09:01:08 +00:00 README

66875392 bytes total (41652224 bytes free)

Which is very cool!

Part of the ‘trick’ is that you should set your time out to be as long as possible to send a million packets.  I just set the maximum values.

line con 0

exec-timeout 35791 23
stopbits 1

line aux 0

stopbits 1

line vty 0 4

exec-timeout 35791 0
timeout login response 300
password cisco
login

!
end

In addition, a 7200 with idle performs MUCH better than a 1700 without idle.  There is something up with ptask, and only dispatching packets every so often.  I’m guessing it’s done that way for a reason.

Also one other cool IOS trick I learned today is that you can redirect to a file resource! Say you want that ‘show tech-support’ as a file on the disk? No problem!

show tech-support | redirect disk0:tech.txt

And of course the newer versions of IOS have a ‘do’ command that you can run from config mode to execute user commands.

R1(config)#do who
Line User Host(s) Idle Location
* 0 con 0 idle 00:00:00
2 vty 0 idle 15:24:11 10.0.2.2
3 vty 1 idle 14:59:56 10.0.2.2
4 vty 2 idle 13:43:44 10.0.2.2
5 vty 3 idle 11:23:44 10.0.2.2

 Take that Junos!

For anyone interested, the binary is included in the latest binary snapshot, and using it is pretty simple:

slirp_rdr.exe 20001 127.0.0.1 20000

This will listen on port 20001, and send traffic to 127.0.0.1 on port 20000.  Easy right?

Manually interfacing from the hypervisor can be the ‘fun’ part.  I haven’t tested with any of the tools, as I don’t know if they will let you leave something ‘listening’ that isn’t connected. For my tests I end up building something with their UI, then loading up my hypervisor that logs, and seeing what it is actually doing so I can inject stuff like this:

nio create_udp nio_udp99 20000 127.0.0.1 20001
ethsw add_nio S1 nio_udp99
ethsw set_access_port S1 nio_udp99 1

This creates a udp nio, and attaches it onto the virtual etherswitch S1, and puts it on VLAN 1.  As you can see it listens on UDP port 20000, which is where slirp_rdr is setup to send it’s data to, and it’ll send to 20001 where slirp_rdr is listening.

I’ve hard coded port 42323 to telnet into 10.0.2.15.  As always SLiRP is hard coded to have the following ip address schema:

Gateway 10.0.2.2
DNS 10.0.2.3
Netmask 255.255.255.0

Be sure to set your router to 10.0.2.15/24 for this to work, and add 10.0.2.2 as your default gateway.

The ONLY address that will respond to ping is 10.0.2.2 .  This is just the way SLiRP is.  HTTP and TCP based stuff works best, things like PPTP will not.  It’s really hit and miss, but the cool thing is that it doesn’t require any device drivers, it’s all user mode code!

“warning: Invalid parameter passed to C runtime function.”

So while debugging Dynamips I got this fun message under GDB.  Of course it doesn’t tell you WHAT function did it, or HOW it was trying to do it.  Fantastic.

Thankfully Dennis Yurichev’s blog gave me the hint to put a breakpoint on ‘OutputDebugStringA’ and sure enough I could see Dynamips trying to treat a socket like a stdio file handle.  Something you can’t do in Win32 world.

On the plus side, I just had to do a small re-write of some functions and I can talk to the Dynamips hypervisor!  Idle and JIT are working too!  Along with WinPcap and UDP transports.

Dynamips on MinGW

It’s always bugged me that the only way to build Dynamips for Windows was with Cygwin.

Well fear no more, I’ve mashed an old version (I would have tried newer, but of course Cmake fails spectacularly and with zero help as always!) and not only does it compile, but it can boot a 7200 version of IOS.

Dynamips on MinGW

Dynamips on MinGW

JIT is broken.  You have to telnet into the console.  And the console is a little wonkey as I’m sure it’s doing a lot more UNIX translation vs being a Win32 program but it does work enough to login, save the config, and reload.

But it’ll crash on reload.

I’m sure it’s full of bugs actually.

https://sourceforge.net/projects/dynamips-mingw/

I started with Dynamips 0.2.8-RC7-community and started commenting out stuff to get it to compile.  Luckily I found this ezwinports that includes mairix that includes some memory mapping functions, namely mmap and munmap ported to Win32 in an early glibc port. While I was trying to integrate libuuid, I got this fun error:

mingw “error: conflicting types” “UUID”

MinGW includes UUID support, since it’s a Microsoft thing.  Unfortunately libuuid doesn’t include unique names,  so I had to rename uuid_t to uu_uuid_t

//typedef unsigned char uuid_t[16];
typedef unsigned char uu_uuid_t[16];

in the uuid.h header file, along with all instances in Dynamips.

I also borrowed sendmsg/recvmsg along with the msg structures from VLC.  inetaton.c from WSHelper, and finally telnet.h from NetBSD.

After that it was a matter of making sure Winsock starts up, and fixing some linking breakage.

For those who want to try, the binary package is here.  I’ll have to setup git on this machine and upload all the changes.  It shouldn’t require any DLL’s, although I haven’t looked at the pcap stuff, as I mentioned it’s largely untested, so I have no idea if any of it works other than the telnet console.

Shoebill now has working Ethernet support!

Great news!  The excellent A/UX capable emulator Shoebill, now has working Ethernet support!  The sad news is that it only supports the TUN/TAP interface.  So Windows users are kind of left out in the fun.

Shoebill + Ethernet

Shoebill + Ethernet

Except, I’ve been here before with SIMH ages ago.  So I dusted off my source code, and injected it into Shoebill.  The first issue I had was that SLiRP was rejecting all the inputted frames, because of invalid frame length.  Even more weird is that ARP worked, and I could see the 10.0.2.2 and 10.0.2.3 virtual IP’s but TCP and UDP outbound wouldn’t work at all.

It took me longer than it should have but although this code worked great with GCC 2.7 and 3.0, 4.x breaks it.  And it’s the same reason why Shoebill originally didn’t work on Win32, the blasted packed structures!  So adding the ‘-mno-ms-bitfields’ flag to GCC is all it took, and now I could ping 10.0.2.2 for about 5-7 pings until SLiRP would crash.  I tried all kinds of stuff trying to see if there was an issue with SLiRP, but I should have payed closer attention to the debugger, with all those threads flying around.  It turns out Shoebill was trying to read & write a the same time, which caused SLiRP to crash as it is not re-entrant.  I tried to place mutex’s on every SLiRP call but that ended up having SLiRP not process any packets.  Very strange.  I then reduced it to where I read the frame out of SLiRP and pass it to Shoebill, and where Shoebill write’s a frame out the SLiRP.  And much to my amazement I can run ‘worms’ just fine!

So after a minute of worming and pinging I called it ‘good enough’ and rebuilt a production binary, and packaged up my source code.

For anyone who want’s to play, my Win32 EXE is here, and the source code I am using is here.

Shoebill ported to Windows!

Shoebill!

Shoebill!

Good news, as mentioned here, the Shoebill emulator was recently given some much needed SDL love, and ported to Linux.

Well that’s great and all, but the vast majority of people who run anything these days do it with Windows.  So I decided to try to get it to compile with MinGW to see how far I could get.

And the short version is that I got it working!

The long version is that in the first pass there is some SIGUSR2 stuff that is undefined.  And for a good reason, since it won’t work.  So I just commented them out.  The next minor problem was the lack of bzero.  Honestly I don’t know why bzero is missing from MinGW, but who knows why.

Shoebill also processes some internal macros with a perl script that for some reason was dropping in binary values into the source, making GCC mad.  I just commented out a line that was adding in more comments into the header.  This let me compile with a simple pass.

There was some issues reading the ROM file, since the 68000 is a BIG ENDIAN processor, and the 8086 is LITTLE ENDIAN, Shoebill makes extensive use of hotns and hotnl, ntohl, and ntohll.  These can be found in the winsock library, and even better they dont need any winsock initialization, they work right away.  I just have to make sure I include winsock2.h, and link against the winsock library.

However when trying to boot, the checksum was 0x00000000, not the expected value!   Luckily there was an assert to catch that and crash.  This led me to notice that in Linux files are opened in binary mode by default, while on Windows, they are opened in ASCII mode.  A quick change of all the fopen calls, and I was reading the ROM, but now crashing on the disk.

As it turns out newer versions of GCC go all crazy when it comes to structs, and try to automatically align to boundaries for quick access.  Which sound nice, until you try to read in some binary data, and expect things to be in certain locations and find out that your structure is larger than expected, and data is read in the wrong place.

The solution is to force the compiler to leave it alone with

__attribute__ ((__packed__))

HOWEVER as luck would have it, Microsoft apparently packs structures a different way, and you have to either make a macro to do a bunch of work to force it to make the structure 1:1 of what you expect, or use the CFLAG option of

-mno-ms-bitfields

And now MinGW’s GCC will build something along the lines of what it’d build on Linux.

Putting it all together, I amazingly got this!

Shoebill on Windows

Shoebill on Windows

Phew!  So for those interested, here is the source code drop(Use the updated one here!), and here is the binary.

If you ever wanted to see the “OS X” of the 1980’s, now is your chance!

In a round about way I got access to my old build machine

So I was able to get a working qemu 1.5.0 on win32. but it is so slow it really is unusable.

there is still an issue with core-routinewin32, so that has to be built with O1 optimizations, but the disk access is horrifically slow. I’ll have to see how to profile gcc code, but any write disk access spikes the CPU to 100%, and drags the whole thing down.

Oh yeah, gcc 4.6.2

Building Qemu 1.1.1-1 for Win32

I still can’t see how to build this for win64 … 🙁

The first major stumbling block on Win32 is glib, and again going back to the MinGW wiki there is a good laundry list of how to bootstrap Glib on MinGW.

Things you’ll need:

Be sure to run ming-get install for the following:

  • gcc
  • g++
  • libiconv
  • zlib
  • libz
  • gettext
  • msys
  • msys-perl
  • msys-m4

Install Python.. Nothing to fancy from what I gather Qemu still requires you have a level 2 Python, not 3 …

Build libffi

This should be the usual dance of configure & make / make install…  But sure to at least run configure like this:

./configure –prefix=/mingw

Build Glib without pkg-config

Be sure to add Python to your path..

PATH=/c/Python27:/c/Python27/DLLs:$PATH
export PATH

Glib is packed with something called ‘xz’ so hopefully you have xzcat .. Otherwise add it!

export LIBFFI_CFLAGS=’-I /mingw/lib/libffi-3.0.9/include’
export LIBFFI_LIBS=-lffi
export lt_cv_deplibs_check_method=”pass_all”
export CFLAGS=”-O0 -g -pipe -Wall -march=i486 -mms-bitfields -mthreads”
export CPPFLAGS=”-DG_ATOMIC_OP_USE_GCC_BUILTINS=1”
export LDFLAGS=”-Wl,–enable-auto-image-base”
configure –prefix=/mingw –with-pcre=internal –disable-static –disable-gtk-doc –enable-silent-rules

You may have some weird issue where when running configure it tells you it cannot create executables, or you get a bunch of weird errors trying to paste in the CFLAGS line.. For me the MinGW prompt was stripping the quotes and the leading – to the -O0 (disable optimization) bit.  I don’t know what on earth its issue was, but I had to type that line in manually.

Then do the make/make install dance. This will take a WHILE. At least with -pipe it’ll run each stage of GCC on multiple processors… But yeah.. This is intense to build.  Good thing we get to do it twice.

I also ran into some weird error in the GIO directory where it couldn’t find my Python, and was looking for python2.5 .. So I copied python.exe to python2.5.exe …

PYTHON = /usr/bin/env python2.5

pkg-config

Naturally pkg-config depends on Glib2, and pkg-config to build… Which of course is a circular problem, much like Glib2 requires pkg-config to build.   So to configure it, it goes something like this now that we’ve built a Glib2 ..

export GLIB_CFLAGS=”-I/mingw/include/glib-2.0 -I/mingw/lib/glib-2.0/include”
export GLIB_LIBS=”-lglib-2.0″
configure –prefix=/mingw

At the same time I can’t help but wonder if this version of pkg-config can use it’s own Glib with the following config:

configure –prefix=/mingw –with-internal-glib

Anyways with any luck we can now build & install pkg-config.  This only takes a few seconds..

Glib2 (again)

From this point you should open a new MinGW prompt window, as you don’t want the old CFLAGS screwing things up.  Re-eport the Python path/dll vars and now we can get on to building glib2 (again!) …

configure –prefix=/mingw;make;make install

 

SDL

This should be somewhat straightforward …

configure –prefix=/mingw;make;make install

In the old days we built zlib, but now we can just quickly add in the package (as we did way above) so we should.. now be ready for the main event!

 

And now we can finally build Qemu 1.1.1-1!!!

Now there is a few things I like to tweak, the first is in the configure script, I like to add in AdLib support.  Look for the line

audio_card_list=”ac97 es1370 sb16 hda”

and add adlib into the list

audio_card_list=”ac97 es1370 sb16 hda adlib”

Next I like to modify hw/pc.c and alter the ISA NE2000, as Qemu doesn’t like to share IRQ 9 with the card, so it is just easier to remove the 0x300/IRQ 9 definition.

static const int ne2000_io[NE2000_NB_MAX] = { 0x300, 0x320, 0x340, 0x360,
0x280, 0x380 };
static const int ne2000_irq[NE2000_NB_MAX] = { 9, 10, 11, 3, 4, 5 };

to this:

static const int ne2000_io[NE2000_NB_MAX] = { 0x320, 0x340, 0x360,
0x280, 0x380 };
static const int ne2000_irq[NE2000_NB_MAX] = { 10, 11, 3, 4, 5 };

The next tweak deals with the ability to use older qcow2 disk images… I guess converting them to RAW with an older version of Qemu, then using the new version of Qemu to convert them back into qcow2’s may be a “good idea™” but for now modifying the source is a quicker fix.

Comment out the “return -EINVAL;”  in block/qcow2.c

if (ext.len > end_offset – offset) {
error_report(“Header extension too large”);
//return -EINVAL;
}

Now one fun thing I’ve noticed is that building with the default O2 flags Qemu will crash out the moment you access a hard disk image.  It appears that coroutine-win32 is at issue (again?).  So the “easy” way I address it is to first build qemu as normal, and verify that if you attach a hard disk image (any kind) and try to access it, partition it etc, it should crash.  Next remove the file coroutine-win32.o , and edit the file config-host.mak and change the CFLAGS that specify

CFLAGS=-O2 -g

to

CFLAGS=-O1 -g

Now run make again, and it *should* just rebuild coroutine-win32.o with the lesser optimization flags, and relink all the exe’s.  If I’ve done this right, you should now have a working Qemu.

You can go ahead and strip the binaries if you so please, but that should be it.

PHEW.  For anyone who wants my build, but doesn’t want to go through this ‘exciting’ process, you can find the Win32 i386 build here.

Weird scaling in action .. Control+ALT+u kind of undoes it, but it just doesn’t look right and it is far too slow.

In preliminary testing I’ve found this version to be MUCH slower than 0.15.1 .. I think it has something to do with it wanting to scale the SDL window.  Also I’ve had issues with various network cards not initializing with the BIOS that ships with this version of Qemu so I’ve included the bios directory from 0.15.1 .  And lastly yes the disk images… I’ve had major issues with my qcow2 disks, and disk corruption with this build.  I’ve gone ahead and  included the qemu-img tool from 0.15.1 so you can convert qcow2 to raw, then use the 1.1.1 tool to take them from raw back into qcow2 … But I’d probably only do it as a test.

You may want to kick the tires on this version but 0.15.1 really blows this one away…

Qemu 1.0 binaries for Windows (Win32)

EDIT: Nobody should use these binaries, there was a critical error (well there still is) in how a core procedure is compiled, and it results in major instability, instead use Qemu 1.0.1).

Well after lots of fights, it finally built!  I booted MS-DOS and ran Quake (with sound!)…  I’ve added in my control+alt+d for a CAD, and control+alt+r for reset, and configured the sound for adlib, sound blaster & GUS.

The i386/x86_64 binaries are here.

All other systems are here.

For the most part this is completely untested, but it did boot MS-DOS….  All the DLL’s that you need should be in the archives.  7zip is what I used to keep them small.

Disk access seems to be still slower than hell, so instead of the typical

-hda bla.disk

You will want to do change it to:

 -drive file=bla.disk,if=ide,index=0,media=disk,cache=writeback

Although if it is a qcow2

 -drive file=bla.disk,if=ide,index=0,media=disk,cache=writethrough

On the way to building Qemu 1.0…

Well one of the fun things is that Qemu now relies on glib.  But in order to build glib it needs pkg-config. And of course you can believe that pkg-config requires glib in order for it to work.  Good lord, nothing like circular dependencies!!!

So what to do?  First things first libffi will be needed to build glib.  It doesn’t seem to have any crazed dependancies so that built ok.  Also something I’ve been missing in a lot of ‘native’ MinGW builds is to add:

–prefix=/mingw

to the configure strings to get things in a location where the system will find them.  For some reason MinGW doesn’t walk /usr/local .. I guess because its not UNIX. With libffi built then you can configure/build glib like this:

export LIBFFI_CFLAGS='-I /mingw/lib/libffi-3.0.10/include'
export LIBFFI_LIBS=-lffi
export lt_cv_deplibs_check_method="pass_all"
export CFLAGS='-O0 -g -pipe -Wall -march=i486 -mms-bitfields -mthreads'
export CPPFLAGS='-DG_ATOMIC_OP_USE_GCC_BUILTINS=1'
export LDFLAGS=”-Wl,--enable-auto-image-base”
./configure --prefix=/mingw --with-pcre=internal --disable-static --disable-gtk-doc --enable-silent-rules

Then it is a simple matter of building out pkg-config then rebuilding glib in a ‘sane’ fashion.  This all comes form the MinGW wiki which has great information on how to do things like bootstrap glib!

And don’t forget to install python! Even Qemu needs it now!

And add it to /etc/profile so it will be in your path…

  export PATH=”.:/usr/local/bin:/mingw/bin:/bin:/c/python27:/c/python27/dlls:$PATH”