Comments on: Offline Wikipedia

By: Crispin

Crispin — Wed, 17 Feb 2010 12:29:11 +0000

Thanks for the instructions, I used a very similar method a few years back but never really got it to work.

For those wanting something simpler (maybe a slightly different use case) I strongly recommend aarddict (http://aarddict.org/).

It’s a free, open-source (GPL) and extremely fast program which also provides ready built databases for all wikipedia text content. You don’t get pictures (far too large) but you do get all the text and latex (math formulae) nicely rendered with working links, tables, formatting, etc. It is available for Ubuntu, Windows, OS-X and Nokia tablet (and source obviously)

By: Reverso time « jsomers.net

Reverso time « jsomers.net — Wed, 10 Feb 2010 16:22:43 +0000

[...] posts discussing build options and compile errors and PATH variables. Examples fresh in my mind: offline Wikipedia, Metacat, and the Ruby wrapper for the Stanford Natural Language [...]

By: emc986

emc986 — Tue, 02 Feb 2010 19:30:31 +0000

– a simple stream edit will certainly not work, as it turns out. The image links are php-generated, I assume. Still, it shouldn’t be too difficult to adjust some code.

By: emc986

emc986 — Tue, 02 Feb 2010 09:52:50 +0000

Glad to see an ongoing discussion on this very excellent offline reader. Perhaps a simple stream edit on the bz2 splits would allow, if online, the retrieval of images. The current enwiki-latest-pages-articles.xml.bz2 is 5.4G, and with the generated index (another 3G!), the damn thing just won't fit on my current 8G uSD for use on a freerunner. I don't how much larger the dump would be including images but I'd guess, maybe using this great method, it wouldn't even fit on a 32G uSD card.

In any case, here's a problem:

This reader has been working beautifully excepting one thing. Pages that have the country infobox do not render the data, for instance on the Afghanistan page the infobox information exists within the file but the rendered page gives a default template value, for example:

Capital city: {{{countrycapital}}}

I'm not really sure where to start looking as I'm not very familiar with the wikimedia parser.

Does anyone else have a clue?

By: Abhay

Abhay — Mon, 02 Nov 2009 09:46:51 +0000

Sorry. the above was a silly error...the actual problem is this:

./show.pl "../wiki-splits/rec09696enwiki-20090929-pages-articles.xml.bz2" "Wikipedia" sh: php: not found

mediawiki_sa parser failed! report to woc.fslab.de

i have installed Php5.3.0 from the net.

By: Mason

Mason — Sun, 25 Oct 2009 18:36:18 +0000

I can't seem to build the reader. I've downloaded everything, built Xapian and installed Django through Mac Ports (and I already had the dev tools). "sudo make" tells me to use "make wikipedia," which gives me this error: usage: cp [-R [-H | -L | -P]] [-fi | -n] [-pvX] source_file target_file cp [-R [-H | -L | -P]] [-fi | -n] [-pvX] source_file ... target_directory make: *** [mediawiki] Error 64

I've looked through everything again, but didn't forget anything. I tried searching Google for "make Error 64" but didn't find anything, and I've tried Django with Python 2.5 and 2.6. Does anyone have idea what my problem might be? Is there any other information that I should provide?

Thanks, Mason

By: Kerry

Kerry — Sun, 25 Oct 2009 02:32:03 +0000

Worked great for me with the german version of wikipedia. The only issue is with searching using umlauts – python doesnt seem to like unicode here. Quite a major one really for German. I intend having a look some time at how to get both de and en versions running at the same time.

By: James Somers

James Somers — Wed, 14 Oct 2009 20:06:30 +0000

Glad to hear you fixed the first problem. I must have been using a different version of PHP.

There may well be a 64-bit version of the software necessary to compile texvc. Do reply again if you're able to fix that.

The dump itself does not include images, largely because they would increase the file size by orders of magnitude. If you can spare the space then you'll have to (a) download all the images separately (which I think you can do) and (b) fix all the wiki's image links so that they point to your local copies.

Note that you don't need the images to get math to display properly, since those PNGs are rendered by texvc.

Personally I get a lot of mileage out of the offline wikipedia without images, but I suppose it could be helpful to have diagrams and the like. Let us know if you're able to integrate images successfully.

By: kvaruni

kvaruni — Wed, 14 Oct 2009 15:16:43 +0000

This problem seems to have been fixed and was caused by "Namespace" being a reserved keyword in PHP 5.3.0. The solution is rather messy in that it involves checking all files in the include folder for references to Namespace:: and renaming that (as well as the class) to MyNamespace. I would advise using Smultron for this job as it can search through all the files that are open by using Advanced Find.

And then there were more problems. Splitting wnet fine, but indexing did not. The cause is a mysterious "Exception: Empty termnames aren't allowed". Ah, this tells me what I need to know. Not. Nevertheless, about half of the wikipedia dump has been indexed and I can get started. Except ... I can get texvc to compile, which would be king of necessary, since most of my research on wikipedia involves math-related subjects. And this time the error is "32-bit absolute addressing is not supported for x86-64". So basically, I cannot get this thing to compile no matter what I try since it does not support 64-bit, yet my operating system and processor are 64-bit. What a mess.

Finally a matter of debate. Why are there no images? Is this because the dump failed to parse completely or are these just omitted? In the latter case this would involve an additional download from wikimedia to make this entire idea of an offline version usable. To be honest I find the entire process to be far too hard for a encyclopedia that promotes openness.

By: kvaruni

kvaruni — Wed, 14 Oct 2009 08:09:57 +0000

I have been trying to get this to work, but I seem to have run into some kind of problem. For starters, I have done everything up to editing the show.pl file where I replaced php5 by php. The make went just fine, the files are nicely split and I can access the server. Great. But each and every page I visit, apart from the search results, tells me that the woc.fslab.de parser failed.

I took a look at the site from Thanassis and i tried to render the bonsai.html page. Tough luck since this did not work. The error I get is "Parse error: syntax error, unexpected T_NAMESPACE, expecting T_STRING in /Users/kimbauters/Documents/offline.wikipedia/mediawiki_sa/includes/Namespace.php on line 46" which coincides with the beginning of the class in the Namespace.php file. This has left me dead in my tracks since I have no idea how to resolve this. Since I am using Snow Leopard, my PHP version is the latest and greatest PHP 5.3.0. Clearly, this should be able to cope just fine with classes in PHP.

Any ideas on how to resolve this issue or am I overlooking something?