PHP 5.4 is now obsolete on Debian

While doing an update, I got this fun message:

[[code]]czo0NjI6XCIKcGhwNSAoNS40LjQ1LTArZGViN3UyKSB3aGVlenktc2VjdXJpdHk7IHVyZ2VuY3k9bWVkaXVtCgoqIFBIUCA1LjQgaGF7WyYqJl19cyByZWFjaGVkIGVuZC1vZi1saWZlIG9uIDE0IFNlcCAyMDE1IGFuZCBhcyBhIHJlc3VsdCB0aGVyZQp3aWxsIGJlIG5vIG1vcmUgbntbJiomXX1ldyB1cHN0cmVhbSByZWxlYXNlcy4gVGhlIHNlY3VyaXR5IHN1cHBvcnQgb2YgUEhQCjUuNCBpbiBEZWJpYW4gd2lsbCBiZSBiZXN0e1smKiZdfSBlZmZvcnQgb25seSBhbmQgeW91IGFyZSBzdHJvbmdseSBhZHZpc2VkCnRvIHVwZ3JhZGUgdG8gbGF0ZXN0IHN0YWJsZSBEZWJpYW57WyYqJl19IHJlbGVhc2UgdGhhdCBpbmNsdWRlcyBQSFAgNS42IHRoYXQKd2lsbCByZWFjaCBlbmQgb2Ygc2VjdXJpdHkgc3VwcG9ydCBvbiAyOHtbJiomXX0gQXVnIDIwMTcuCgotLSBPbmTDhWVqIFN1csODwr0gJmx0O29uZHJlakBkZWJpYW4ub3JnJmd0OyBTdW4sIDA0IE9jdCAyMDE1IDE3e1smKiZdfTowNTozNyArMDIwMApcIjt7WyYqJl19[[/code]]

Time to upgrade!

After a bit of digging around here is how I did it.  I’m moving from PHP 5.4 to PHP 5.6

[[code]]czoxNzM6XCIKIyBwaHAgLS12ZXJzaW9uClBIUCA1LjQuNDUtMCtkZWI3dTIgKGNsaSkgKGJ1aWx0OiBPY3QgMTcgMjAxNSAwOTowMTp7WyYqJl19NDgpCkNvcHlyaWdodCAoYykgMTk5Ny0yMDE0IFRoZSBQSFAgR3JvdXAKWmVuZCBFbmdpbmUgdjIuNC4wLCBDb3B5cmlnaHQgKGMpIHtbJiomXX0xOTk4LTIwMTQgWmVuZCBUZWNobm9sb2dpZXMKXCI7e1smKiZdfQ==[[/code]]

Ok, so here is the version I’m starting with on Debian 7.9 (Wheezy). During this process, I’m using unsigned packages so get used to this fine message, and packages being held back, unless I manually install them:

[[code]]czo1MzU6XCIKUmVhZGluZyBwYWNrYWdlIGxpc3RzLi4uIERvbmUKVzogR1BHIGVycm9yOiBodHRwOi8vcGFja2FnZXMuZG90ZGViLm97WyYqJl19cmcgd2hlZXp5IFJlbGVhc2U6IFRoZSBmb2xsb3dpbmcgc2lnbmF0dXJlcyBjb3VsZG5cJ3QgYmUgdmVyaWZpZWQgYmVjYXVzZSB0aGV7WyYqJl19IHB1YmxpYyBrZXkgaXMgbm90IGF2YWlsYWJsZTogTk9fUFVCS0VZIEU5Qzc0RkVFQTIwOThBNkUKVzogR1BHIGVycm9yOiBodHRwOntbJiomXX0vL3BhY2thZ2VzLmRvdGRlYi5vcmcgd2hlZXp5LXBocDU2LXp0cyBSZWxlYXNlOiBUaGUgZm9sbG93aW5nIHNpZ25hdHVyZXMgY291e1smKiZdfWxkblwndCBiZSB2ZXJpZmllZCBiZWNhdXNlIHRoZSBwdWJsaWMga2V5IGlzIG5vdCBhdmFpbGFibGU6IE5PX1BVQktFWSBFOUM3NEZFe1smKiZdfUVBMjA5OEE2RQpSZWFkaW5nIHBhY2thZ2UgbGlzdHMuLi4gRG9uZQpCdWlsZGluZyBkZXBlbmRlbmN5IHRyZWUKUmVhZGluZyBzdGF7WyYqJl19dGUgaW5mb3JtYXRpb24uLi4gRG9uZQowIHVwZ3JhZGVkLCAwIG5ld2x5IGluc3RhbGxlZCwgMCB0byByZW1vdmUgYW5kIDAgbm90IHtbJiomXX11cGdyYWRlZC4KXCI7e1smKiZdfQ==[[/code]]

But if you can read this, then it worked!

First add these to the /etc/apt/sources.lst file:

[[code]]czoxOTU6XCJkZWIgaHR0cDovL3BhY2thZ2VzLmRvdGRlYi5vcmcgd2hlZXp5IGFsbApkZWItc3JjIGh0dHA6Ly9wYWNrYWdlcy5kb3R7WyYqJl19ZGViLm9yZyB3aGVlenkgYWxsCmRlYiBodHRwOi8vcGFja2FnZXMuZG90ZGViLm9yZyB3aGVlenktcGhwNTYtenRzIGFsbApkZWItc3tbJiomXX1yYyBodHRwOi8vcGFja2FnZXMuZG90ZGViLm9yZyB3aGVlenktcGhwNTYtenRzIGFsbFwiO3tbJiomXX0=[[/code]]

Then we can run the usual apt-get update / apt-get upgrade shuffle.

On my first run I got this fun output:

[[code]]czoyNjU6XCIKVGhlIGZvbGxvd2luZyBwYWNrYWdlcyBoYXZlIGJlZW4ga2VwdCBiYWNrOgpsaWJhcGFjaGUyLW1vZC1waHA1IGxpYm17WyYqJl19eXNxbGNsaWVudDE4IG15c3FsLXNlcnZlciBwaHAtcGVhciBwaHA1IHBocDUtY2xpCnBocDUtY29tbW9uIHBocDUtZ2QgcGhwNS1teXtbJiomXX1zcWwKVGhlIGZvbGxvd2luZyBwYWNrYWdlcyB3aWxsIGJlIHVwZ3JhZGVkOgpteXNxbC1jb21tb24KMSB1cGdyYWRlZCwgMCBuZXdse1smKiZdfXkgaW5zdGFsbGVkLCAwIHRvIHJlbW92ZSBhbmQgOSBub3QgdXBncmFkZWQKXCI7e1smKiZdfQ==[[/code]]

So I went ahead and updated mysql-common.  And during that upgrade I got the new message:

[[code]]czoxMjQ6XCIKV0FSTklORzogVGhlIGZvbGxvd2luZyBwYWNrYWdlcyBjYW5ub3QgYmUgYXV0aGVudGljYXRlZCEKbXlzcWwtY29tbW97WyYqJl19bgpJbnN0YWxsIHRoZXNlIHBhY2thZ2VzIHdpdGhvdXQgdmVyaWZpY2F0aW9uIFt5L05dPyB5ClwiO3tbJiomXX0=[[/code]]

So yes these packages are all unsigned. 🙁

After this, I ran:

[[code]]czoyNDpcIgojIGFwdC1nZXQgaW5zdGFsbCBwaHA1ClwiO3tbJiomXX0=[[/code]]

And got the following scary looking output

[[code]]czo3NDA6XCIKVGhlIGZvbGxvd2luZyBleHRyYSBwYWNrYWdlcyB3aWxsIGJlIGluc3RhbGxlZDoKbGliYXBhY2hlMi1tb2QtcGhwNSB7WyYqJl19bGlidDEtNSBsaWJ2cHgxIHBocDUtY2xpIHBocDUtY29tbW9uIHBocDUtZ2QgcGhwNS1teXNxbApTdWdnZXN0ZWQgcGFja2FnZXM6CntbJiomXX1waHA1LXVzZXItY2FjaGUKUmVjb21tZW5kZWQgcGFja2FnZXM6CnBocDUtcmVhZGxpbmUKVGhlIGZvbGxvd2luZyBORVcgcGFja2Fne1smKiZdfWVzIHdpbGwgYmUgaW5zdGFsbGVkOgpsaWJ0MS01IGxpYnZweDEKVGhlIGZvbGxvd2luZyBwYWNrYWdlcyB3aWxsIGJlIHVwZ3JhZGV7WyYqJl19ZDoKbGliYXBhY2hlMi1tb2QtcGhwNSBwaHA1IHBocDUtY2xpIHBocDUtY29tbW9uIHBocDUtZ2QgcGhwNS1teXNxbAo2IHVwZ3JhZHtbJiomXX1lZCwgMiBuZXdseSBpbnN0YWxsZWQsIDAgdG8gcmVtb3ZlIGFuZCAzIG5vdCB1cGdyYWRlZC4KTmVlZCB0byBnZXQgNyw2NTkga0Ige1smKiZdfW9mIGFyY2hpdmVzLgpBZnRlciB0aGlzIG9wZXJhdGlvbiwgNSwyMjAga0Igb2YgYWRkaXRpb25hbCBkaXNrIHNwYWNlIHdpbGwgYmV7WyYqJl19IHVzZWQuCkRvIHlvdSB3YW50IHRvIGNvbnRpbnVlIFtZL25dPyB5CldBUk5JTkc6IFRoZSBmb2xsb3dpbmcgcGFja2FnZXMgY2FubntbJiomXX1vdCBiZSBhdXRoZW50aWNhdGVkIQpwaHA1LW15c3FsIHBocDUtY2xpIHBocDUtZ2QgbGliYXBhY2hlMi1tb2QtcGhwNSBwaHA1LWNve1smKiZdfW1tb24gcGhwNQpJbnN0YWxsIHRoZXNlIHBhY2thZ2VzIHdpdGhvdXQgdmVyaWZpY2F0aW9uIFt5L05dPyB5ClwiO3tbJiomXX0=[[/code]]

And then finally, after another apt-get update / apt-get upgrade I finally get this output:

[[code]]czoxMzg6XCIKVGhlIGZvbGxvd2luZyBwYWNrYWdlcyBoYXZlIGJlZW4ga2VwdCBiYWNrOgpsaWJteXNxbGNsaWVudDE4IG15c3FsLXN7WyYqJl19ZXJ2ZXIKMCB1cGdyYWRlZCwgMCBuZXdseSBpbnN0YWxsZWQsIDAgdG8gcmVtb3ZlIGFuZCAyIG5vdCB1cGdyYWRlZC4KXCI7e1smKiZdfQ==[[/code]]

So just update the held back packages and get it over with.

[[code]]czo0NzpcIgphcHQtZ2V0IGluc3RhbGwgbGlibXlzcWxjbGllbnQxOCBteXNxbC1zZXJ2ZXIKXCI7e1smKiZdfQ==[[/code]]

Ugh, it isn’t pretty.  But now we are on the newer train of PHP!

[[code]]czoyNTY6XCIKIyBwaHAgLS12ZXJzaW9uClBIUCA1LjYuMTQtMX5kb3RkZWIrenRzKzcuMSAoY2xpKSAoYnVpbHQ6IE9jdCAyIDIwMTV7WyYqJl19IDAzOjM5OjIwKQpDb3B5cmlnaHQgKGMpIDE5OTctMjAxNSBUaGUgUEhQIEdyb3VwClplbmQgRW5naW5lIHYyLjYuMCwgQ29weXJpZ3tbJiomXX1odCAoYykgMTk5OC0yMDE1IFplbmQgVGVjaG5vbG9naWVzCndpdGggWmVuZCBPUGNhY2hlIHY3LjAuNi1kZXYsIENvcHlyaWdodCAoe1smKiZdfWMpIDE5OTktMjAxNSwgYnkgWmVuZCBUZWNobm9sb2dpZXMKXCI7e1smKiZdfQ==[[/code]]

Hopefully things continue to work.

Mirroring Wikipedia

So I had an internet outage, and was thinking if I was trapped on my proverbial desert island what would I want with me?

Well wikipedia would be nice!

So I started with this extreme tech article by Sebastian Anthony, although it has since drifted out of date on a few things.

But it is enough to get you started.

I downloaded my XML dump from Brazil like he mentions.  The files I got were:

  • enwiki-20140304-pages-articles.xml.bz2 10G
  • enwiki-20140304-all-titles-in-ns0.gz 58MB
  • enwiki-20140304-interwiki.sql.gz 728Kb
  • enwiki-20140304-redirect.sql.gz 91MB
  • enwiki-20140304-protected_titles.sql.gz 887Kb

The pages-articles.xml is required.  I added in the others in the hopes of fixing some formatting issues.  I re-compressed it from 10GB using Bzip2 to 8.4GB with 7zip.  It’s still massive, but when you are on a ‘slow’ connection every saved GB matters.

Since I already have apache/php/mysql running on my Debian box, I can’t help you with a virgin install.  I would say it’s pretty much like every other LAMP install.

Although I did *NOT* install phpmyadmin.  I’ve seen too many holes in it, and I prefer the command line anyways.

First I connect to my database instance:

mysql -uroot -pMYBADPASSWORD

And then execute the following:

create database wikimirror;
create user ‘wikimirror’@’localhost’ IDENTIFIED BY ‘MYOTHERPASSWORD’;
GRANT ALL PRIVILEGES ON wikimirror.* TO ‘wikimirror’@’localhost’ WITH GRANT OPTION;
show grants for ‘wikimirror’@’localhost’;

This creates the database, adds the user and grants them permission.

Downloading and setting up mediawiki 1.22.5 is pretty straight forward.  There is one big caveat I found though.  InnoDB is incredibly slow for loading the database. I spent a good 30 minutes trying to find a good solution before going back to MyISAM with utf8 support.

With the empty site created, I do a quick backup incase I want to purge what I have.

/usr/bin/mysqldump -uwikimirror -pw1k1p3d1a wikimirror > /usr/local/wikipedia/wikimedia-1.22.5-empty.sql

This way I can quickly revert as constantly re-installing mediawiki is… a pain.  And it gets repetitive which is good for introducing errors, so it’s far easier to dump the database/user and re-create them, and reload the empty database.

When I was using InnoDB, I was getting a mere 163 inserts a second. That means it would take about 24 hours to import the entire database!!  Which simply is not good enough for someone as impatient as me.  As of this latest dump there are 14,313,024 records that need to be inserted, which would take the better part of forever to do.

So let’s make some changes to the MySQL server config.  Naturally backup your existing /etc/mysql/my.cnf to something else, then I added the following bits:

 key_buffer = 1024M
max_allowed_packet = 384M
query_cache_limit = 18M
query_cache_size = 128M

I should add that I have a lot of system RAM available.  And that my box is running Debian 7.1 x64_86.

Next you’ll want a slightly modified import program,  I used the one from Michael Tsikerdekis’s site, but I did modify it to run the ‘precommit’ portion on it’s own.  I did this because I didn’t want to decompress the massive XML file on the filesystem.  I may have the space but it just seems silly.

With the script ready we can import!  Remember to restart the mysql server, and make sure it’s running correctly.  Then you can run:

bzcat enwiki-20140304-pages-articles.xml.bz2 | perl ./mwimport2 | mysql -f -u wikimirror -pMYOTHERBADPASSWORD  –default-character-set=utf8 wikimirror

And then you’ll see the progress flying by.  While it is loading you should be able to hit a random page, and get back some wikipedia looking data.  If you get an error well obviously something is wrong…

With my slight moddifications I was getting about 1000 inserts a second, which gave me…

 14313024 pages (1041.174/s),  14313024 revisions (1041.174/s) in 13747 seconds

Which ran in just under four hours.  Not too bad!

With the load all done, I shut down mysql, and then copy back the first config.  For the fun of it I did add in the following for day to day usage:

 key_buffer = 512M
max_allowed_packet = 128M
query_cache_limit = 18M
query_cache_size = 128M

I should add that the ‘default’ small config was enough for me to withstand over 16,000 hits a day when I got listed on reddit.  So it’s not bad for small-ish databases (my wordpress is about 250MB) that see a lot of action, but wikipedia is about 41GB.

Now for the weird stuff.  There is numerous weird errors that’ll appear on the pages.  I’ve tracked the majority down to lua scripting now being enabled on the template pages of wikipedia.  So you need to enable lua on your server, and setup the lua extensions.

The two that just had to be enabled to get things looking half right are:

With this done right, you’ll see Lua as part of installed software on the version page:

mediawiki installed softwareAnd under installed extensions:

wikimedia installed extensions

I did need to put the following in the LocalSettings.php file, but it’s in the installation bits for the extensions:

$wgLuaExternalInterpreter = “/usr/bin/lua5.1″;
require_once(“$IP/extensions/Lua/Lua.php”);
$wgScribuntoEngineConf[‘luastandalone’][‘luaPath’] = ‘/usr/bin/lua5.1′;
require_once( “$IP/extensions/Scribunto/Scribunto.php” );

Now when I load a page it still has some missing bits, but it’s looking much better.

The Amiga page...

The Amiga page…

Now I know the XOWA people have a torrent setup for about 75GB worth of images.  I just have to figure out how to get those and parse them into my wikipedia mirror.

I hope this will prove useful for someone in the future.  But if it looks too daunting, just use the XOWA.  Another solution is WP-MIRROR, although it can apparently take several days to load.

ownCloud

So I was reading through a friends blog (wintellect!) and I came across this page about ownCloud…  Well I thought this was very interesting as I’ve pulled a lot of my external email mess inside (on my own Exchange 5.5 server on MS Virtual Server 2005!) .. So I like this whole idea.

I’ve got this VPS that has a few extra gigs of space, and it’d be SUPER convenient to map some drives for backups, or even back it up by copying some files..  It’s a simple AMP program setup, so I had it up and running in a few seconds.  The ‘hard’ part was mapping the drive from Vista.  Naturally it came down to reading the instructions, namely:

  1. in Services, enable the Webclient service (might be enabled already)
  2. in the Registry, change HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WebClient\Parameters\BasicAuthLevel from 1 to 2
  3. go to My Computer → Mount Network Drive
    • in the Folder field type <a href="http://address/files/webdav.php">http://ADDRESS/files/webdav.php</a>
    • check Connect using different credentials
And that is about the size of it.

Installing mediawiki on WAMP

Building on our WAMP installation, we are now going to install mediawiki.

The first thing I’d recommend to do is to move the contents of c:\wamp\www into another directory… I just shoved the terminal thing into c:\wamp\terminal .

Now mediawiki is the software that powers wikipedia. It’s a great collaboration platform, it has built in revision control, and best of all it’s free.

It’s also VERY simple to setup, well compared to other web content platforms.

The current version is 1.16, which can be downloaded here. As things change, you may be best served by just visiting the main download site.

Since most ‘AMP’ servers are Linux based, we’ll have to get gzip & tar to extract mediawiki. It’s very easy though.

Simply type this in to extract mediawiki

C:\temp>dir
Volume in drive C has no label.
Volume Serial Number is FC55-C2F4

Directory of C:\temp

12/28/2010 08:15 PM DIR .
12/28/2010 08:15 PM DIR ..
12/28/2010 08:13 PM 49,152 gzip.exe
112/28/2010 08:15 PM 12,647,934 mediawiki-1.16.0.tar.gz
12/28/2010 08:13 PM 114,688 tar.exe
3 File(s) 12,811,774 bytes
2 Dir(s) 7,073,234,944 bytes free

C:\temp>gzip -dc mediawiki-1.16.0.tar.gz| tar -xf –

C:\temp>

Ok, now with mediawiki extracted we just move the contents of c:\temp\mediawiki-1.16.0 into c:\wamp\www

Now before we go on, we are going to set a password for the MySQL process. In the off chance someone is following this on a server to deploy on the internet, it’d be crazy to leave it with no password.

So left click on the WAMP system tray icon, go to MySQL, and bring up the MySQL Console.

media1

Just hit enter for the password as there isn’t one.

Next follow this SQL statement to set the password for the root user to password. Or select your own better password.

mysql> use mysql;
Database changed
mysql> update user set password=PASSWORD(“password”) where User=’root’;
Query OK, 3 rows affected (0.05 sec)
Rows matched: 3 Changed: 3 Warnings: 0

Now restart the mysql service, by clicking on the system tray icon, then mysql, service then ‘restart service’. If you don’t do this the password change will not take effect!

With that out of the way, it’s time to configure mediawiki. Simply open up a web browser to the following location:

http://localhost

And you should see something like this:

media2

Click the setup link, and let’s walk through the options…

First is the wikiname. I’m just going to call mine ‘test wiki’. Put in your own contact email, so that mediawiki will email YOU if anything is going on… I left the language in English, and left the license alone. The next important thing to do is to select a Admin username, and password. This is all up to you. Just remember that the Username is CaSe SeNsItIvE!!!

Leave the caching off.

The next section is for the email notifications, I just left those as default.

The final thing to configure is the database.

Since we are going to keep this simple, just set the DB username to root, and put in the password you configured earlier in the MySQL Console. Next check the ‘superuser account’ box, and specify root and the password again.

You can now click the Install MediaWiki button!

You’ll see some information printed on the page, and if everything goes according to plan, you’ll get the message:

Installation successful! Move the config/LocalSettings.php file to the parent directory, then follow this link to your wiki.

You should change file permissions for LocalSettings.php as required to prevent other users on the server reading passwords and altering configuration data

So simply copy the file c:\wamp\www\config\LocalSettings.php to c:\wamp\www\

then simply click the following link to be taken to your personal wiki:

http://localhost/index.php

media4

And that should take care of it!

WAMP server 2.1

Due to a request, I figured I’d document out the joys of installing WAMP, flashterm & the flash policy server, mediawiki in a multipart article. So to start, we’ll build the foundation which is the WAMP package which the other two will rely on.

So what is WAMP, well it’s Windows Apache Mysql & PHP. If you’ve ever heard of LAMP, this is the Windows version. And I’m happy to say that PHP applications seem to be pretty portable, allowing you to freely move data & applications to & from Linux/BSD/Solaris/Windows. Or that’s been my experience.

The first thing you’ll need to do is install the WAMP package. Luckily for us, the WampServer project has packaged the whole thing up into a nice windows installer. And the new version (2.1c) even includes x64 bulids!

So for the 32bit crowd you can download WAMP here.

And for the x64 64bit crowd, download WAMP here.

I’m currently using an aging HP machine as my server so I’m using Windows Server 2003 along with the 32bit version. WAMP works best when it’s just installed with the defaults.

The first thing I do is I install the SMTP service that comes with Windows. I leave IIS off as it’s kind of silly to have two web servers, unless you are doing it for some planned reason (say Virtual Server which needs IIS and uses port 1024). If you do have IIS make sure the ‘default’ website is turned off.

All that has to be done with the SMTP server is configure it to allow 127.0.0.1 to relay emails.. As it’s nice to get notifications that could be built in with whatever PHP application you go with.

SMTP1

Now run the installer, and I’ll try to walk through the steps.

wamp1

Hit next

wamp2

Accept the GPL License.

wamp3

Let it install in the default directory, it makes things easier.

wamp4

I like both set, you may not, but it’s easier to launch.

wamp5

Verify the install settings.

wamp6

Now WAMP will copy files and install.

wamp7

Then WAMP will want to know the default browser, honestly IE is just fine.

wamp8

Next WAMP will want to know what SMTP server to use… This is kind of important, and why I installed the default MS SMTP server as a lot of things that do registration, or email alerts, well work better with an email server…

wamp9

With the install completed let it launch WAMP.

wamp10

Once the WAMP server is installed it’ll be stopped and offline.

wamp11

Left click on the WAMP tray tool, and have it start all the services.

wamp12

Then left click again on the WAMP tray tool, and now you can start it up and take it online…

wamp13

If everything has gone right your WAMP tray tool will look like this!

Now open up a web browser, and go to http://localhost/

If everything has gone right, you should see this:

wamp14

Congratulations! You’ve successfully installed the WAMP server!