GopherVista

Continuing with our gopher madness, next up we have GopherVista.  And this is everything I had hoped it would be when I first learned about this project.  I had joked to another friend that it’d be cool to crawl and feed the indexer data in a manner that could basically bring AltaVista back to life.  And we laughed, and I had my utzoo search and that was that.

Except it wasn’t.

However, across the internet, Ben didn’t hear any naysay about limitations or anything to get in the way, and went ahead and wrote a crawler in go, kept the results in a sane name/db order for later sanitisation in and out of AltaVista, and after an aggressive gopher port scan of the internet, he created GopherVista, an index of the gopher-verse, running on Windows 98.

No, really, you read that right, GopherVista backends on Windows 98!

Read all about the creation of GopherVista over a Ben’s blog blog.benjojo.co.uk.

GopherVista

Keep in mind that this is a search engine, not a proxy, so it’s best to use something like Internet Explorer 4, or an ancient Netscape that supports both HTTP 1.1 & Gopher.

I have to also say, that something like this is far more cooler, and better thought out than my utzoo hack, and I’m just happy to have inspired him, but now I really want to re-think my setup, and of course index all the things….

Personal AltaVista + UTZOO reloaded

Introduction

Long before websites, during the dark ages of the BBS, on the internet there was (well it’s still there!) a distributed messaging system called usenet.  There are countless topics on just about everything that was full of all kinds of incredible conversations.  Before the walled gardens, and the ease of running individual bulletin boards, the internet had prided itself on having one big global distributed messaging system.  It was a big system, and one thing that was always taken for granted was that it was too big to save, and that whatever you put out there would probably be erased as all sites had a finite amount of very expensive disk space, and they would only keep recent articles.

But it turns out that in the University of Toronto, in the zoology department they had a tape budget, and were in fact archiving everything they could.  In all they had amassed 141 tapes spanning from  February 1981 (though these are not Usenet posts, just internal netnews University stuff) all the way up to about midnight of July 01, 1991!

While the archive was made available to a few people in 2001, it was made generally available in 2009, and then in 2011 on archive.org where I downloaded a copy of it.  There is some interesting backstory over on Dogcow land, as it took quite a bit of effort to get the data from the tapes, and then slowly released out into the wild.

As mentioned on the archive.org site:

This is a collection of .TGZ files of very early USENET posted data provided by a number of driven and brave individuals, including David Wiseman, Henry Spencer, Lance Bailey, Bruce Jones, Bob Webber, Brewster Kahle, and Sue Thielen.

OK, so back a few months ago, I had setup AltaVista personal desktop search along with the UTZOO usenet archive for the purpose of using something more sophisticated than grep, but maintaining that legacy/retro feel us using outdated technology.  To recap the first challenge is that the desktop search product, is only meant to be used from the desktop of a Windows 98/NT 4.0 workstation.  It uses a super ancient version of JAVA as the webserver, and they chose to bind it to 127.0.0.1:6688 .  So the first thing to get around that was to build a stunnel tunnel allowing me to effectively connect to the webserver remotely.  And since the server assumes it’s locally I had to use Apache with mod_rewrite to setup some simple regex expressions to massage the pages into something that would be usable from a non local machine.

So with that word salad up, let’s have a brief picture!

Flow diagram

Stepping it up

On my ‘general’ hosting machine, I use haproxy to reverse proxy out multiple sites out the single address.  This is a super simple solution that allows me to have all kinds of different backends using various hosting platforms, such as Apache 1.3 on Windows NT 3.1.  So for this to work I just needed to create an altavista.superglobalmegacorp.com DNS record, and then the following in the haproxy config:

frontend named-hosts
bind 172.86.179.14:80
acl is_altavista hdr_end(host) -i altavista.superglobalmegacorp.com
use_backend altavista if is_altavista

backend altavista
balance roundrobin
option httpclose
option forwardfor
server debian8 10.0.0.18:80 check maxconn 10

So as you can see it’s really simple it looks for the string ‘altavista.superglobalmegacorp.com’ in the host header, and then sends it to the backend that has a single web server, in this case a lone Debian server, aptly named debian8 that throttles after 10 concurrent connections.

The next thing to do was generate a SSL self signed cert, which wasn’t too hard.  The stunnel installer has a profile ready to go, so it was only a matter of finding a version of OpenSSL that’ll run on NT 4.  As this isn’t public encryption I really don’t care about it using crap certs.

On the Debian server is where all the regex magic, is along with the stunnel client to connect to the NT 4.0 Workstation.

client = yes
debug = 0
cert = /etc/stunnel/stunnel.pem

[altavista]
accept = 127.0.0.1:8080
connect = 10.0.0.19:8443

Likewise on NT stunnel will need a config like this:

cert = c:\stunnel\stunnel.pem

; Some performance tunings
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1

; Some debugging stuff useful for troubleshooting
debug = 0
output = c:\stunnel\stunnel.log.txt

[altavista]
accept = 8443
connect = 127.0.0.1:6688

With the ability for the Debian box to talk to the AltaVista web server, it was now time to configure Apache.  This is the most involved part, as the html formatting by AltaVista personal search is hard coded into the java binary.  However thanks to mod_rewrite we can modify the page on the fly!  So the first thing is that I setup to virtual directories, the first one /altavista maps to the search engine, and then I added /usenet which then talks to IIS 4.0 on the Windows NT 4.0 workstation, which is just allowing read & browse to the usenet files that will need to be indexed.

#This part connect to a stunnel connection to the Altavista server
ProxyPass “/altavista” “http://localhost:8080”
ProxyPassReverse “/altavista” “http://localhost:8080”
#This connects to IIS 4.0 on the NT 4.0 machine
ProxyPass “/usenet” “http://10.0.0.19/usenet”
ProxyPassReverse “/usenet” “http://10.0.0.19/usenet”
ProxyRequests Off
RewriteEngine On

Because we mounted it on a sub directory we need to redirect the root to /altavista so I simply add:

#Redirect the root to the /altavista path.
#
RedirectMatch 301 ^/$ /altavista

To get the images to work, along with fixing the 127.0.0.1 hardcoding,  I copied them from the NT workstation onto the Apache server, then added this regex statement:

#clean up urls
Substitute “s|Copyright 1997|Copyright 2017|n”
Substitute “s|127.0.0.1:6688|altavista.superglobalmegacorp.com/altavista|n”
Substitute “s|file:///c:\Program Files\DIGITAL\AltaVista Search\My Computer\images\|/images/|n”

And now the site is starting to work.  The most involved regex is to change the links from local text files, into a path to point to the usenet shares.  This changes the text for u:\usenet\a333\comp\33.txt into a workable URL.

Substitute “s|>u:\\\\usenet.([a-z]{1,}[0-9]{3,})\\\([0-9a-z\+\-]{1,})\\\([0-9]{1,})|—><a href=\”http://utzoo.superglobalmegacorp.com/usenet/$1/$2/$3.txt\”>[$2\] Click for article|

Naturally there is a LOT of these type of statements to match various depths, and pattern types as there is A news, B news and C news archives, plus scavenged bits.

Additionally I disabled a bunch of URL’s that would either try to alter the way the engine works, or allow the search location to change, just giving you empty results, along with altering some of the branding, as digital.com doesn’t exist anymore, and various tweeks.  The finished config file for Apache is here.

Now with that in place, I can hit my personal AltaVista search.  The next insane thing was to rename all the files from the UTZOO dump adding a .txt extension, and then re-encoding them in MS-DOS CR/LF format.  I found using ‘find -type f’ to find files, and then a simple exec to rename them into a .txt extension.  Then it was only a matter of using ZIP to compress the archives, and then transferring them to Windows NT, and running UNZIP on them with the -a flag to convert them into CR/LF ASCII files on Windows.  This took a tremendous amount of time as there are about 2.1 million files in the archive.

Now with the files on Windows, now I had to run the indexer.

Indexed in under 7 hours!

While I had originally had an IIS 4.0 instance on the same NT 4.0 Workstation serving up the result files, I thought it may make more sense to just serve them from the UTZOO mirror server I have in the same collocation so it’d be much faster, so that way only the queries are relying on servers in Hong Kong, instead of being 100% located in the United States.

So here we go, my search portal for all that ancient usenet goodness:

altavista.superglobalmegacorp.com

If you are hoping for the wealth of knowledge to be gained from people posting on usenet from 1981 to 1991 then this is your ticket.  Keep in mind that usenet being usenet, there is discussions on everyone and everything, and like all other forums before you know it it’ll end with calling people Hitler, and how the Amiga is the greatest computer ever (well it was!).  A tip when searching by year, is that people commonly wrote the year as 2 digits.  However when looking for numbers like, say Battletech 3025, it will pull up files named 3025.txt.  To prevent this just add -3025.txt to stop names like 3025.txt, or if you want to find out about the movie Bladerunner from 1982, try searching for bladrunner 82 -82.txt +review +movie.  If you have any questions, there is of course the manual with a guid on how to search.

While the story of AltaVista is somewhat interesting, but much like how Digitial screwed up the Alpha market by trying to hoard high end designs, they also didn’t set the search people free to focus on search.  And the intranet stuff was crazy expensive, look at this ad from 1996 which translate to a minimum of $10,000 USD a year to run a single search engine!  But as we all know, the distributed model of google won search and AltaVista never had a chance as it was caught up in the Compaq/HP mess then spun out to be quickly absorbed by Yahoo.

Meanwhile it appears the original owners of altavista.com, AltaVista Technology, Inc. of California, are actually still in business.  If anyone cares I’ll put the installation files, and some of the config’s in this directory.

URL shortners & short domains

I needed to get some business cards, and the usual thing is to use QR codes that have a tiny URL name, that then redirect to your real web site.  Easy, right?

Well most people use ‘public’ servers like bit.ly & friends.  In china many people I do business with use 1688.com .  But this got me thinking, 1688 is a FOUR letter domain, unlike any of the three letter ones that seem to be more common.  I know all the one, two and three letter domains are all gone, but are there any four letter domains?

Turns, out YES there are.

I used this site:

Domain Name Soup .com

And I was able to hammer though their UI, and find one, and register it with my usual registrar.

*This isn’t an AD, I’m not being paid to say any of this.  I was more so surprised that I could not only find a four letter domain, but it’s the initials of my wife’s business.

The best part is that I could use YOURLS, a free PHP+Mysql app to quickly and easily manage the redirects.

Fun with regex substitutions in Apache

Continuing from my previous post, I was now able to access my AltaVista server, however from a web browser I was unable to actually view any of the documents remotely.

In the pages though I did get the MS-DOS path to the usenet article in question:

Now how do I turn that into a URL?

Well as it turns out mod_rewrite does support regex, which in turn can do variable re-ordering!

After a bit of googling I found this page on stackoverflow, on how to convert a date between UK/US formats:

s/(\d{4})-(\d{2})-(\d{2})/$1-$3-$2/

Simple, right?  So what is going on here?  The parenthesis define a variable set, and on the substitution part you can recall them with $1, $2 , $3 etc.  So using this recipe I could take something like this:

u:\b227\comp\sys\laptops\3080

and convert it into the following:

http://debian7/usenet/b227/comp/sys/laptops/3080

The code for this would look something like this:

Substitute "s|&gt;u:.([a-z]{1,}[0-9]{3,})\\\([0-9a-z]{1,})\\\([0-9]{1,})|---&gt;&lt;a href="\"http://debian7/usenet/$1/$2/$3\""]Click for article|"

Although for some reason it’s embedding the URL’s even though I specified code formatting.

Now all I had to do was install IIS 4.0 off the Option Pack CD-ROM, onto my Windows NT 4.0 workstation, and create a virtual directory of /usenet which then pointed to the U: drive where AltaVista did it’s indexing.

So to this point that gives me a config file much like this:

ServerAdmin webmaster@localhost
DocumentRoot /var/www
SSLProxyEngine On
ProxyPass "/altavista/" "https://10.12.0.16"
ProxyPassReverse "/altavista/" "https://10.12.0.16/"
ProxyRequests Off
RewriteEngine On

SetOutputFilter INFLATE;SUBSTITUTE;DEFLATE
AddOutputFilterByType SUBSTITUTE text/html
#clean up urls
Substitute "s|127.0.0.1:6688|debian7/altavista|n"
Substitute "s|file:///C:\Program Files\DIGITAL\AltaVista Search\My Computer\images\|http://debian7/images/|n"
#protect the page
Substitute "s|launch=app||n"
Substitute "s|?pg=config&amp;what=init|?pg=h|n"
#fix title
Substitute "s|&lt;IMG src=\"http://debian7/images/av_personal.gif\" alt=\"[AltaVista] \"  BORDER=0 ALIGN=middle HEIGHT=72 VSPACE=0 HSPACE=0&gt;|&lt;a href=\"http://debian7/altavista\"&gt;&lt;IMG src=\"http://debian7/images/av_personal.gif\" alt=\"[AltaVista] \"  BORDER=0 ALIGN=middle HEIGHT=72 VSPACE=0 HSPACE=0&gt;<strong>|---&gt;|n"
Substitute "s|</strong>u:.([a-z]{1,}[0-9]{3,})\\\([0-9a-z]{1,})\\\([0-9a-z]{1,})\\\([0-9a-z]{1,})\\\([0-9a-z]{1,})\\\([0-9a-z]{1,})\\\([0-9]{1,})|---&gt;&lt;a href=\"http://debian7/usenet/$1/$2/$3/$4/$5/$6/$7\"&gt;Click for article|"
Substitute "s|&gt;u:.([a-z]{1,}[0-9]{3,})\\\([0-9a-z]{1,})\\\([0-9a-z]{1,})\\\([0-9a-z]{1,})\\\([0-9a-z]{1,})\\\([0-9]{1,})|---&gt;&lt;a href=\"http://debian7/usenet/$1/$2/$3/$4/$5/$6\"&gt;Click for article|"
Substitute "s|&gt;u:.([a-z]{1,}[0-9]{3,})\\\([0-9a-z]{1,})\\\([0-9a-z]{1,})\\\([0-9a-z]{1,})\\\([0-9]{1,})|---&gt;&lt;a href=\"http://debian7/usenet/$1/$2/$3/$4/$5\"&gt;Click for article|"
Substitute "s|&gt;u:.([a-z]{1,}[0-9]{3,})\\\([0-9a-z]{1,})\\\([0-9a-z]{1,})\\\([0-9]{1,})|---&gt;&lt;a href=\"http://debian7/usenet/$1/$2/$3/$4\"&gt;Click for article|"
Substitute "s|&gt;u:.([a-z]{1,}[0-9]{3,})\\\([0-9a-z]{1,})\\\([0-9]{1,})|---&gt;&lt;a href=\"http://debian7/usenet/$1/$2/$3\"&gt;Click for article|"
# Need links for the u:\news097f1\b120\comp\society\futures\1122
Substitute "s|&gt;u:.(news[0-9]{3,}f[0-9])\\\([b0-9]{1,})\\\([a-z]{1,})\\\([a-z]{1,})\\\([a-z]{1,})\\\([a-z]{1,})\\\([a-z]{1,})\\\([0-9]{1,})|---&gt;&lt;a href=\"http://debian7/usenet/$1/$2/$3/$4/$5/$6/$7/$8\"&gt;Click for article|"
Substitute "s|&gt;u:.(news[0-9]{3,}f[0-9])\\\([b0-9]{1,})\\\([a-z]{1,})\\\([a-z]{1,})\\\([a-z]{1,})\\\([a-z]{1,})\\\([0-9]{1,})|---&gt;&lt;a href=\"http://debian7/usenet/$1/$2/$3/$4/$5/$6/$7\"&gt;Click for article|"
Substitute "s|&gt;u:.(news[0-9]{3,}f[0-9])\\\([b0-9]{1,})\\\([a-z]{1,})\\\([a-z]{1,})\\\([a-z]{1,})\\\([0-9]{1,})|---&gt;&lt;a href=\"http://debian7/usenet/$1/$2/$3/$4/$5/$6\"&gt;Click for article|"
Substitute "s|&gt;u:.(news[0-9]{3,}f[0-9])\\\([b0-9]{1,})\\\([a-z]{1,})\\\([a-z]{1,})\\\([0-9]{1,})|---&gt;&lt;a href=\"http://debian7/usenet/$1/$2/$3/$4/$5\"&gt;Click for article|"
# Need links for  u:\news002f1\b1\fa.poli-sci\8
Substitute "s|&gt;u:.(news[0-9]{3,}f[0-9])\\\([b0-9]{1,})\\\([a-z\.\-]{1,})\\\([0-9]{1,})|---&gt;&lt;a href=\"http://debian7/usenet/$1/$2/$3/$4\"&gt;Click for article|"

&lt;Location /usenet/&gt;
    ProxyPass  http://10.12.0.16/usenet/
    RewriteEngine On
    SetOutputFilter INFLATE;SUBSTITUTE;DEFLATE
    AddOutputFilterByType SUBSTITUTE text/html
&lt;/Location&gt;

bla bla rest of the 000-default crap....

Simple right?

Searching for AltaVista

Searching for AltaVista

So now I get a nicely formatted page, I can click the mountain icon, and I jump back to home, and I can click on the articles and, because I have no extensions or MIME types to intercept it’ll just download them to my PC.  I guess I need to go through them all, convert them from UNIX format to MS-DOS, and stick a .txt extension on every single one of them.

I’m still thinking this thing is far too rickety to put on the internet, but we’ll see.

Fun with Apache, (mod_proxy, mod_rewrite), stunnel, And AltaVista Personal search

As you may remember from my prior attempt at using Altavista Search I ran out of space, and found out it only serves pages on 127.0.0.1:6688 and is pretty much hardcoded to do so.  It’s a “fine” hyrid java 1.01 application, with the bulk of it being java.  I finally got around to setting up a VM, and unpacking all of the utzoo archives, and indexing them.  I should have done something about the IO because this took too long (KVM).

SIXTEEN HOURS!!!

SIXTEEN HOURS!!!

So to cheat the system, I installed stunnel as a simple https to http proxy, which let me access my search VM anywhere.  However it still embedded 127.0.0.1 in all the pages.

via stunnel

via stunnel

Enter an Apache reverse proxy to talk to stunnel to talk to AltaVista search!

First to enable a few modules:

a2enmod substitute
a2enmod proxy
a2enmod ssl
a2enmod proxy_http
a2enmod rewrite

And adding this into the config:

SSLProxyEngine On
ProxyPass “/altavista/” “https://10.12.0.16”
ProxyPassReverse “/altavista/” “https://10.12.0.16/”
ProxyRequests Off
RewriteEngine On
SetOutputFilter INFLATE;SUBSTITUTE;DEFLATE
AddOutputFilterByType SUBSTITUTE text/html
Substitute “s/1997/2016/ni”
Substitute “s/97/16/ni”
Substitute “s|127.0.0.1:6688|debian7/altavista|n”
Substitute “s|file:///C:\Program Files\DIGITAL\AltaVista Search\My Computer\images\|http://debian7/images/|n”
Substitute “s|launch=app||n”
Substitute “s|<a href=http://debian7/altavista/?pg=q&what=0&fmt=d|<!—|n”
Substitute “s|><strong>|—>|n”
Substitute “s|</strong></a>||n”
Substitute “s|>u:\|->u:\|n”

This let me redirect all of those requests into a VM called debian7 on the /altavista path.  I also copied the images to the apache server, and now I get something that looks correct!

Apache in the mix!

Apache in the mix!

I cut the results short… But here is a search of something simple:

About 16598 documents match your query.

About 16598 documents match your query.

I also killed all the ‘working URL’s that simply open a desktop application on the index ‘server’.  Naturally it was a personal service, but as a server this isn’t any good.  As such you can’t click on any search results now.  I need something else to figure out how to take the result blocks like “u:\b128\comp\databases\2852” and turn them into URL’s.

Also, as much as I want to re-index I would be best to cut off the headers, or most of them so the preview lines make sense.  Xref, Path, even From & Newsgroups don’t interest me.

I hate to leave it as ‘good enough’ but if anyone has a solution…. I’ll be glad to make this wonderful resource available!

The client needs to access the internet!

But let’s not give them access to everything.

This is a common scenario I see, where someone needs to get updates to some magical software package on the internet.  Great.  And people just give them access to ANY site, which ends up being not only the internet (the intended destination) but the rest of their internal network.  Granted a good defense in the SDN world is inbound rules as well for each VM, but everything is never 100%.

RFC1918 defines our friends, the private address ranges:

     10.0.0.0        -   10.255.255.255  (10/8 prefix)
     172.16.0.0      -   172.31.255.255  (172.16/12 prefix)
     192.168.0.0     -   192.168.255.255 (192.168/16 prefix)

However the solution to this fun filled problem is to grant them HTTP/HTTPS access to the inverse of this.  Enter the netmask command.  You can give it a range, and it’ll lay out what networks to you need to add like this:
     netmask -c 0.0.0.0:9.255.255.255
     0.0.0.0/5
     8.0.0.0/7

Now I can exclude everything right up until 10.0.0.0/8 !

It’s quite the handy tool, but I didn’t see any Windows version.  So a few minutes with MinGW, and dealing with it’s weird Makefile’s way of linking things, and here you go!

This way you can permit internet access, not give them inside, access and still have a global DENY actually work.

YAY.

And if anyone is interested here are the networks:

     0.0.0.0/5
     8.0.0.0/7
    11.0.0.0/8
    12.0.0.0/6
    16.0.0.0/4
    32.0.0.0/3
    64.0.0.0/2
   128.0.0.0/3
   160.0.0.0/5
   168.0.0.0/6
  172.0.0.0/12 
 172.32.0.0/11
 172.64.0.0/10
 172.128.0.0/9
   173.0.0.0/8
   174.0.0.0/7
   176.0.0.0/4
   192.0.0.0/9
192.128.0.0/11
192.160.0.0/13
192.169.0.0/16
192.170.0.0/15
192.172.0.0/14
192.176.0.0/12
192.192.0.0/10
   193.0.0.0/8
   194.0.0.0/7
   196.0.0.0/6
   200.0.0.0/5
   208.0.0.0/4

Yes, I know it’s a LOT of typing.

More thoughts on Minecraft, compression and encryption

So earlier, I had touched on Minecraft, and lamented on how it doesn’t compress it’s network data very well.  Well it turns out that yes, in the server.properties file, there is an option network-compression-threshold, which by default is set to 256, meaning packets larger than 256bytes are compressed

network-compression-threshold=256

So using this quick stunnel guide, I thought I’d try a quick experiment.  So I loaded up Titan City, and ran some connection experiments:

First, the Minecraft server with a setting of 256000000 which I would imagine effectively turns off compression.  I’m capturing one minutes worth of game play as it tries to render the spawn point.  Then again with the threshold set to 256:

12M 28 Apr 13:44 minecraft-nocompression.cap
1.6M 28 Apr 13:46 minecraft-256compression.cap

So, uncompressed it’s 12MB worth of data!  While with the Minecraft compression on, it’s 1.6 MB worth of data.

And now with stunnel using zlib compression, we get the following results:

2.1M 28 Apr 13:42 stunnel-nocompressioninserver.cap
1.5M 28 Apr 13:48 stunnel-256compression.cap

2.1MB worth of traffic relying on zlib in this case to do all the compression, and 1.5MB with zlib compressing the Minecraft compression.  And for the heck of it, why not compress the data again?
964K 28 Apr 13:46 minecraft-256compression.cap.gz
993K 28 Apr 13:44 minecraft-nocompression.cap.gz
938K 28 Apr 13:48 stunnel-256compression.cap.gz
1.5M 28 Apr 13:42 stunnel-nocompressioninserver.cap.gz

Well, now that is strange, why is the stunnel traffic even compressible, after it’s been encrypted?  Kind of weird to me.  At any rate, here is some more data thanks to the capinfos program:
# capinfos *cap
File name: minecraft-nocompression.cap
File type: Wireshark/tcpdump/… – pcap
File encapsulation: Ethernet
Packet size limit: file hdr: 1520 bytes
Number of packets: 14 k
File size: 12 MB
Data size: 12 MB
Capture duration: 59 seconds
Start time: Tue Apr 28 13:43:30 2015
End time: Tue Apr 28 13:44:29 2015
Data byte rate: 211 kBps
Data bit rate: 1,689 kbps
Average packet size: 844.05 bytes
Average packet rate: 250 packets/sec
SHA1: ffb5542c47da69ddc93da902136e1173d76b56e0
RIPEMD160: bc2102185a924096a8cf145c54375a05ab90e3c6
MD5: ba0e1addfcb36e7db6314764941fa6af
Strict time order: True

File name: minecraft-256compression.cap
File type: Wireshark/tcpdump/… – pcap
File encapsulation: Ethernet
Packet size limit: file hdr: 1520 bytes
Number of packets: 10 k
File size: 1,686 kB
Data size: 1,524 kB
Capture duration: 54 seconds
Start time: Tue Apr 28 13:45:28 2015
End time: Tue Apr 28 13:46:22 2015
Data byte rate: 28 kBps
Data bit rate: 226 kbps
Average packet size: 150.91 bytes
Average packet rate: 187 packets/sec
SHA1: 5b5e51f53f0716fd84a39120afd68eadbfaf9816
RIPEMD160: f2bf3839c084b1d7b31fce0a8a8ce959316643a7
MD5: dc6f56a5a1c10e642548e0eeb979629b
Strict time order: True

And now let’s look at the stunnel captures:

File name: stunnel-nocompressioninserver.cap
File type: Wireshark/tcpdump/… – pcap
File encapsulation: Ethernet
Packet size limit: file hdr: 1520 bytes
Number of packets: 9,949
File size: 2,159 kB
Data size: 1,999 kB
Capture duration: 59 seconds
Start time: Tue Apr 28 13:41:13 2015
End time: Tue Apr 28 13:42:12 2015
Data byte rate: 33 kBps
Data bit rate: 270 kbps
Average packet size: 201.02 bytes
Average packet rate: 168 packets/sec
SHA1: 418cc249c3393d85e6e59a6e02c02060b7b7ce4f
RIPEMD160: bf7f56af412734260e0e96d1a0c7d2f28be3ba95
MD5: 1b96fce1db9d38d8dbbecf9bb2278079
Strict time order: True

File name: stunnel-256compression.cap
File type: Wireshark/tcpdump/… – pcap
File encapsulation: Ethernet
Packet size limit: file hdr: 1520 bytes
Number of packets: 9,585
File size: 1,554 kB
Data size: 1,401 kB
Capture duration: 59 seconds
Start time: Tue Apr 28 13:47:35 2015
End time: Tue Apr 28 13:48:34 2015
Data byte rate: 23 kBps
Data bit rate: 189 kbps
Average packet size: 146.21 bytes
Average packet rate: 162 packets/sec
SHA1: 19b2bbfff8cd9c5c0e460d64ad0f4b966cf3a141
RIPEMD160: e31c226101daea9327a8b13a4a1012a24bea11c1
MD5: a7b4b0d5ecf3e6a472255cff13466b51
Strict time order: True

Well for me this is still interesting.  The stunnel connection sent less packets, and smaller.  I know that this is horrible to ‘measure’ like this, and yes none of the datasets are the same, making this kind of bogus. However, honestly compressing with stunnel does feel faster.

So, want to try?  I guess I can let people try if they want, but you’ll need stunnel.  I’ve read horror stories on griefers and I figure if i raise the bar to connect it’ll be somewhat distractionless.

So here is my stunnel.conf I’m using on the client side.

client = yes
compression = zlib
foreground = yes
debug = 6

[minecraft]
accept = 127.0.0.1:25565
connect = virtuallyfun.com:25566
cert = minecraft.pem

And of course, you need my certificate pair, so here is minecraft.pem:

-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEArvdyw5UGic3wCDowzGM/ZVdaCrnDFeo8SmkP1jJGIhsAzzMV
+56Ctd98raKPnw5nnEtrmV7yImBiF/A32SPbYF6nqBhNKPbrUg4u4dsr9uyo7/F4
00VMcXbcgwc+pDOU5uUCg7hZcmCGolVs+7GxC3R1m8SbPZYvlgcHHJXdnEKL8RG8
z54B2L1Ai76CYlD6Q94QnQxiJricy8DJJP6/ZuqkGg01AcoaCaTOa8mvZMLkBFkj
Ase0aX0nSDRxNkBBe/hgwkRuFWaMI1iGgTCX9J1rogLg2WfZoGIFtuYp8jnwiNQQ
dEM6TTGURm9iqb78pBd6PwWsFBqOa0GLOEzLzwIDAQABAoIBAAz8/373VBnsuLHT
qAW0JGOgfWWobov07G7Vp8BN0Rj9Ci1XbH1WQfvAUGAPXjv/dL+MdbtX6f+VShLe
2TZ8S++2dxmqXCf7VHKt7NsFSxk0bkIJmd+NGGSf3zS21/aWgao2O96NU86CzdvF
Hab9hNgF2CktCh0jRfsMIIIFugK8amQGxpKrseSXuuyWGHNW2a6TIHBJRD1L2DQA
GMVgmpb43aYQHYqeQLmgP40PkJgByazQRV40zqsap1DBxNi2TnDjLUWufKACDM64
RPltM8lZm3H2yqVQEQ2yGbzoEuwVysG5+AlDOCVcbDtQqcj5sV/gzL8X+e6DgV+0
o46pZgkCgYEA5QGN+73cApm1OkjsV1OW7RR22EmGWT9HeFFGReWSBsVoDh/dFjIJ
/KtE7PbafvMcV8JMTzbSxBElc+nPFsRk8J4W+JD3Bluw0MfU1K+AUEmPsj5bX6O6
9Mv8/n8HVhpUusIFvO7K7HutdxxNOis510AtRL8oA6NFkiUBot5KGUMCgYEAw5c1
UVT+N9nciFKopgJlzfcb60SKy60vtWd3eQOuLuRd+1QhF3VpUzTAvwbLARuYdlYM
gIaKcOb7A2+ZNKNCccwP6lTIDEwQuQ3lDyG3KXzHfHocZV4n5uT2SpKLv5NzZlF5
4ealpD5pPwN2c4zpvRDeweWznNahUVngYxma5IUCgYEAvac68eg7g3/OYZWw/WVB
kdgn0FmbxN+uDcupWguUkrz7vu7OhyorsTAZ5fFN5GLr7xX/Yn7xr+TPUp6onZ9K
RSd3uKU9nutilJVaAkXSCyvQsHoJ7DvJgiBJxm5nIfyufPhgDibosU5/yywKHQld
XpFMrClvNwwJes3g/AQB88cCgYEAvOfQ7jG5qrW3Ys765gOQ0gHlrDAyIY+ucXVy
FaYxWEbmYnSZ1W9n/54GvzlPXk2JzllDj+rh0TO1olbp0MYRyZj+kiO6Zu4chK7f
2eKFZgOHJDlILbtnrIDdQ58QbEJ8hYkRv9Yli2FgAyVUBTxHEH03uGwjMsq1Wb4F
k5FKYYUCgYBSe0wqziRUNjxhig/3g5qG6iGt0jnbHN5eTsNiEqe3QxDLKNERGTS4
pjpS0LV52bEdpX7imsIskmt81ed/cPUEKd6nbYyrLT67fUCeLuQ07Zlebc/PI8AD
ZoRiyx8x8k55z143XCIdzIyZTriQ5SkUV33wbzPRF8upD0xaygf2Tw==
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
MIIEOTCCAyGgAwIBAgIJAIOpvPCh+v5fMA0GCSqGSIb3DQEBBQUAMIGyMQswCQYD
VQQGEwJVUzERMA8GA1UECAwISWxsaW5vaXMxEDAOBgNVBAcMB0NoaWNhZ28xHDAa
BgNVBAoME1N1cGVyZ2xvYmFsbWVnYWNvcnAxFjAUBgNVBAsMDVZpcnR1YWxseSBG
dW4xGTAXBgNVBAMMEHZpcnR1YWxseWZ1bi5jb20xLTArBgkqhkiG9w0BCQEWHmpz
dGV2ZUBzdXBlcmdsb2JhbG1lZ2Fjb3JwLmNvbTAeFw0xNTA0MjgwMjU1MzJaFw0y
MDEyMDUwMjU1MzJaMIGyMQswCQYDVQQGEwJVUzERMA8GA1UECAwISWxsaW5vaXMx
EDAOBgNVBAcMB0NoaWNhZ28xHDAaBgNVBAoME1N1cGVyZ2xvYmFsbWVnYWNvcnAx
FjAUBgNVBAsMDVZpcnR1YWxseSBGdW4xGTAXBgNVBAMMEHZpcnR1YWxseWZ1bi5j
b20xLTArBgkqhkiG9w0BCQEWHmpzdGV2ZUBzdXBlcmdsb2JhbG1lZ2Fjb3JwLmNv
bTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK73csOVBonN8Ag6MMxj
P2VXWgq5wxXqPEppD9YyRiIbAM8zFfuegrXffK2ij58OZ5xLa5le8iJgYhfwN9kj
22Bep6gYTSj261IOLuHbK/bsqO/xeNNFTHF23IMHPqQzlOblAoO4WXJghqJVbPux
sQt0dZvEmz2WL5YHBxyV3ZxCi/ERvM+eAdi9QIu+gmJQ+kPeEJ0MYia4nMvAyST+
v2bqpBoNNQHKGgmkzmvJr2TC5ARZIwLHtGl9J0g0cTZAQXv4YMJEbhVmjCNYhoEw
l/Sda6IC4Nln2aBiBbbmKfI58IjUEHRDOk0xlEZvYqm+/KQXej8FrBQajmtBizhM
y88CAwEAAaNQME4wHQYDVR0OBBYEFJcb/w/SowAjTa/hvtim9oWYSarZMB8GA1Ud
IwQYMBaAFJcb/w/SowAjTa/hvtim9oWYSarZMAwGA1UdEwQFMAMBAf8wDQYJKoZI
hvcNAQEFBQADggEBAIISqlsBZKh67of21sJhsavDB4T7xrd/qC5zTUeUioScXr+j
n2aziysr+HazliIpVNa5QjicYTniG7urAZdL/zhegSyxEq6r1/BVAks0ooYxJT/f
G5EIhQurv3wQcGKSEXx6IA7+kheqYX++XcM6lAz5jPPIXsRV4NDsE7T68vVuQrr/
RYMHkbCXMqCbUq8x8k65KNN3EPJ66lH83kXuQJRazeurJmcquhmWqApjkORS17kq
+g2xRlnEolvS7umkGz9cbGP7SAWY+ySVIulKSKUKzji8qK8T/hW0dYWUPTZ6+LZx
hHnFJXiaGbnd1sEEB6uVV17XipnE15TGJ8NPT2s=
-----END CERTIFICATE-----

And I run it something like this from the client side…

# stunnel mine.conf
2015.04.28 14:25:23 LOG5[ui]: stunnel 5.16 on x86_64-apple-darwin14.3.0 platform
2015.04.28 14:25:23 LOG5[ui]: Compiled/running with OpenSSL 0.9.8zd 8 Jan 2015
2015.04.28 14:25:23 LOG5[ui]: Threading:PTHREAD Sockets:SELECT,IPv6 TLS:ENGINE,OCSP
2015.04.28 14:25:23 LOG5[ui]: Reading configuration from file mine.conf
2015.04.28 14:25:23 LOG5[ui]: UTF-8 byte order mark not detected
2015.04.28 14:25:23 LOG6[ui]: Compression enabled: 1 method(s)
2015.04.28 14:25:23 LOG6[ui]: Initializing service [minecraft]
2015.04.28 14:25:23 LOG6[ui]: Loading certificate from file: mine.pem
2015.04.28 14:25:23 LOG6[ui]: Loading key from file: mine.pem
2015.04.28 14:25:23 LOG4[ui]: Insecure file permissions on mine.pem
2015.04.28 14:25:23 LOG5[ui]: Configuration successful

Now with stunnel connected, you just have to add a server, but you connect to ‘localhost’. This will have you talking to the stunnel program which then talks to my server, which then redirects to the VM running Minecraft.

aa

Setup a server to ‘localhost’ to access stunnel

bb

Now you can connect to the stunnel server

No promises on how long it’ll be up though.

Titan City!

Titan City!

For normal clients... (shhhh!)

For normal clients… (shhhh!)

AltaVista Personal Indexer

caption

Probably not a good idea..

I never got into the whole ‘desktop search’ thing as I used to know where my stuff was.  But now we live in the future where not only can you just go out and buy terrabytes worth of storage but downloading 10 years worth of usenet is something you can accomplish in a few minutes (on a good connection) but storing it as flat files only takes 20 minutes to decompress some 2,070,332 worth of files is a trivial manner.  It’s really cool to live in the future.

Total Files Listed: 2070332 File(s) 5,429,376,673 bytes 
                    168164 Dir(s) 1,119,884,468,224 bytes free

Now what about finding something in those files?

I should be embarrassed as I was using grep.

Yes in my hunt for obscure information grep was my tool of choice.

So after Frank had mentioned it in passing, if I’d ever used AltaVista Personal Search 97 before I thought I’d give it a bit of a test.  First I unpacked some BSD source code, and had it index that.  The results were incredibly FAST.  So the next thing to do was to try the UTZOO archives.  I should have expanded my NT 4.0 VM’s disk first, but I got this far until I was down to 200MB of free disk space

Screen Shot 2014-10-29 at 9.04.27 PM

 

I should add that I’m sharing the UTZOO archvie over the network.  Not the fastest way at all.  And I only made it about 40% the way through the archive.  Even at this point the search database is only 1.2GB

So how does it run?  Well it’s a localized web service that resides on your desktop.  Of course it only works when you request from 127.0.0.1 as they sold a network searchable version of AltaVista, the Workgroup Edition.  Even this was a retail product at one point retailing for $29 to $35

Screen Shot 2014-10-29 at 9.46.46 PM

Show me the Xenix!

So you hit the web page, type in your search, and you answers like immediately.  It really is scary how fast this thing is.  Although the results can need a lot of tweaking but we are talking 800,000 files.

But needless to say there was the disastrous Compaq buyout of DEC, and the entrance of Google, and it was over.  From what I understand people are still selling the workgroup/enterprise search.  I can see why even though the 97 is rough it still has promise.

What a bargain!

What a bargain!

For anyone who cares, it’s geared to Windows 95, or Windows NT 4.0.. 2000 and beyond is at your own risk.  It uses a Win16 setup program, so Windows 7 x64 was out of the question, but you can download it here.

Exchange 5.5 OWA vs Outlook 2003

ASP 0115

ASP 0115

error ‘ASP 0115’

Unexpected error

/exchange/USA/root.asp

A trappable error occurred in an external object. The script cannot continue running.

So, call me crazy, but I’ve been running an Exchange 5.5 server a home for a while without issues.  It’s perfect for a single user, I can keep up to 16GB worth of email on there, and best of all I can use real email clients like Outlook (or is it LookOut!?).  Anyways I noticed something weird which is that Outlook 2003 always is unsure if the server is there, and I have to tell it that it’s OK to connect.  Also once the Outlook 2003 client connects, it kills OWA, giving me these weird ASP 0115 Unexpected errors.

googling around for a fix was a bit futile, and I’d largely written off OWA, as in this day & age, who really wants some ASP 3.0 app?  But for some reason, today was going to be the day to fix it, as I don’t have Outlook on my macbook air.

So with the Outlook 2003 clue in mind I finally found KB-818709, aka “Outlook Web Access stops responding when you try to access a mailbox on an Exchange 5.5 computer”.

As the cause states:

This problem occurs when you try to access a user account that was previously accessed by a client computer that is running Microsoft Office Outlook 2003.

Outlook 2003 adds a fourth entry to the PR_FREEBUSY_ENTRYIDS property. PR_FREEBUSY_ENTRYIDS is a multi-valued MAPI property that is stored on the Inbox folder. CDO expects three entries. The unexpected fourth entry causes heap corruption that causes OWA or the third-party program to stop responding.

Well how about that?

So with the hotfix in hand, and a reboot, it now works perfectly, like it did back in 1997.  And the best part is that it works great in Chrome.

And for anyone crazy like me with Exchange 5.5, remember to install SP4, and of course the KB829436 hotfix!