Category Archives: Software

Ordnance Survey OpenData In QGIS 3: Part 1

Recently, I’ve been preparing my UK walking maps using the Ordnance Survey’s free OpenData products, which I’ve rendered into maps using a free, open-source Geographical Information System, QGIS. I thought I’d write a little bit about that, now that I’ve got my maps looking more or less as I’d like them. For this first part, I’m going to write about using one of the OS’s free topographic datasets. Later posts will deal with adding features like contours, rivers, lakes and roads, and with adding custom data.

First, of course, you need to install QGIS. I’ve been using version 3, which at time of writing is the latest. With that done, you then need some OS data. I downloaded both versions of their Terrain 50 dataset, which provides elevation data at 50m horizontal resolution. The “Grid” format provides raster elevation data—lots of cells in a square grid, each containing a number representing the elevation in metres above sea level for that location. The “Contours” format provides a vector dataset, full of “shape files”—properly interpreted by QGIS, they’ll draw contour lines and spot heights for you. I’ll use only the Grid data for this post, though I’ll be using the ESRI version of Contours later.

OS grid squares

These downloads need to be unzipped, at which point they each produce a folder called data, full of subfolders lettered according to the OS’s 100-km grid system. To keep the Grid and Contour data separate, I unzipped them to separate folders named OS Terrain 50 (Grid) and OS Terrain 50 (ESRI). Annoyingly, each lettered grid folder contains a multitude of zipped files—anything up to a hundred, each representing a 10-km square tile of topographic data. QGIS can actually peer inside these zipped files and load tile data directly, but I find it easier to deal with multiple tiles if I do a mass unzipping for each lettered grid square that interests me, and then clear away the zip files.

Features like rivers, lakes, woodlands and buildings are added using more datasets. I’ve been using the Open Map – Local ESRI shape files. These are downloaded in packages that cover a single grid square. For my examples here I’m going to use data from the NG square. Downloaded and unzipped, that produces a folder called OS Open Map Local (ESRI Shape File) NG. (Round about this point, I started stuffing all my data folders inside one big folder called OS Data, to make them easier to find.)

The OS also provides a variety of stylesheets—instructions that tell your GIS software how to display your data. These provide a useful starting point which can be modified to produce your own custom styles. Downloading and unzipping the Terrain 50 and Open Map – Local stylesheets from the OS’s Cartographic Stylesheets page will produce another couple of folders on your hard drive called OS-Terrain-50-stylesheets-master and OS-OpenMap-Local-stylesheets-master. Into the OS Data folder they go.

With all that in place, we can open QGIS 3 and start producing a map. QGIS opens with a load of menu items and toolbars across the top, a “Browser” window at upper left, which allows you to navigate to and load data files, and a “Layers” window at lower left, which keeps track of the various layers you’re putting together to create your map. The rest of the screen will hold the map itself. Navigating to the ng subfolder of my Terrain 50 Grid data (having first unzipped all its files), I find an eyewatering array of files with different names and extensions. I want to load all the files with extension .asc, and I can call them up using the filter tool (shaped like a little funnel) in the Browser window toolbar, and typing *.asc as my filter criterion. Then I select all the .asc files, right-click and select Add Selected Layer(s) to Canvas.

This is what appears in the map window:

OS OpenData Terrain 50 NG raster tiles
Click to enlarge
Contains OS OpenData © Crown copyright and database right 2018

All the 10-km tiles containing elevation data have been loaded, and colour-coded by height, from black at the lowest to white at the highest point of each tile. The white squares are areas of no data, containing only sea. If you know your Scottish islands, it’s possible to make out the distinctive shape of the Isle of Skye, but it’s all a bit of a mess because each tile has been tinted individually.

The first thing to do is to merge all the tiles into one big square, so they can be tinted consistently. We can do that from the top menu, Raster/Miscellaneous/Merge…. For “Input Layers”, select all the tiles. Under “Merged”, save the merged data as a file with a descriptive name you can find later. I also nominate a specific “no data” value for the output, setting it to -100, so QGIS knows which areas of the merged file to make transparent.

Here’s what the merged version looks like:

OS OpenData Terrain 50 NG raster tiles merged
Click to enlarge
Contains OS OpenData © Crown copyright and database right 2018

Now it could do with a bit of colour. Double-click on the name of the layer in the “Layers” window, to open the Layer Properties dialogue box. (Choose the “Symbology” view if QGIS doesn’t take you there by default.) Switch the “Render type” to Singleband pseudocolor, and QGIS will offer you a colour ramp—a series of colours that can be used to code the height data on the map. QGIS has a wide variety of built-in ramps, but they’re hidden away. Drop down the menu for “Color ramp”, choose Create New Color Ramp… and drop down the menu in the dialogue box that follows. Catalog: cpt-city contains all sorts of useful stuff.

Finding the cpt-city catalogue in QGIS 3
Click to enlarge

I made my own ramp, based on the height tints of the classic OS tourist maps of the 1970s, and saved it (using the Style button at lower left) so I could reuse it. Here’s the set-up for the ramp:

Custom colour ramp in QGIS 3
Click to enlarge

And here’s what the map looks like with my colour ramp applied:

OS OpenData Terrain 50 NG height pseudocolour
Click to enlarge
Contains OS OpenData © Crown copyright and database right 2018

No sea colour, but that’s deliberate. Zero height in the topographic data doesn’t correspond precisely to the coastline, and the OS provides a set of tiles in Open Map – Local that will lay a precise coastline on to my topo map. From the QGIS menu, Layer/Add Layer…/Add Vector Layer… brings up a dialogue box that lets you browse to the directory in which the ESRI shape files are stored—in my case, OS Open Map Local (ESRI Shape File) NG/data. The file I’m after is NG_TidalWater.shp. Adding that as a layer on top of the topographic data produces this:

OS OpenData Terrain 50 NG tidal water
Click to enlarge
Contains OS OpenData © Crown copyright and database right 2018

Lovely coastline, shame about the colour, which has been assigned at random by QGIS. Time to use an OS stylesheet. Double-click on the new TidalWater layer to open its Layer Properties dialogue, go to “Symbology”, and use the button at lower left to open Style/Load Style. Navigate to the Open Map – Local stylesheets—in my case exhaustingly named OS-OpenMap-Local-stylesheets-master/ESRI Shapefile stylesheets/QGIS stylesheets (QML)/Full colour style. Load TidalWater.qml. Presto! Now the sea is sea-coloured, and all those tile margins have disappeared:

OS OpenData Terrain 50 NG tidal water blue
Click to enlarge
Contains OS OpenData © Crown copyright and database right 2018

The awkward white boxes can be eliminated by setting the background colour of the project to the same shade as the OS’s tidal water. Project/Properties… takes you to the dialogue box. Drop down the menu for “Background colour”—Pick Color will give you a little paint-dropper pointer that you can use to copy the tidal water colour off the map by clicking on it. The result looks like this:

OS OpenData Terrain 50 NG tidal water blue + background
Click to enlarge
Contains OS OpenData © Crown copyright and database right 2018

Starting to look pretty good, isn’t it? *

Adding a little shading will produce a more three-dimensional effect. Choose Raster/Analysis/Hillshade…. In the dialogue box, choose your topographic layer as the “Input layer”, and assign a filename to save the Hillshade result. The default settings work fine for everything else. Here’s what the hillshade layer looks like when it’s loaded on top of the topographic layer, but below the TidalWater tiles:

OS OpenData Terrain 50 NG hillshade
Click to enlarge
Contains OS OpenData © Crown copyright and database right 2018

Very pretty, but it would be nice to see the topo colours, too. Double-click on the hillshade layer to bring up its Layer Properties dialogue, choose “Transparency”, and fiddle with the “Global opacity” slider. I find an opacity of 20% looks nice:

OS OpenData Terrain 50 NG topo + hillshade + water
Click to enlarge
Contains OS OpenData © Crown copyright and database right 2018

There’s a problem, though. If you want to use these maps at large scale, you come up against the limited horizontal resolution of 50m imposed by the Ordnance Survey on its free products. Here’s what the mountain Blaven, on the Isle of Skye, looks like if we zoom in on the map above:

Blaven, Skye, tinted and shaded at 50m horizontal resolution
Click to enlarge
Contains OS OpenData © Crown copyright and database right 2018

Click or tap to enlarge the above view, and it’s glaringly obvious that the 50m squares are showing up intrusively in the map tint and shading.

Since I’m not particularly concerned with the accuracy of the height data, only in using them to generate a nice tint and shade to demonstrate the general topography, I’m happy to run an interpolation to increase the horizontal resolution and produce a smoother effect. I do that from Raster/Projections/Warp (Reproject)…. Here’s the dialogue box completed and ready to start rendering:

QGIS Warp (Reproject) settings for 10m interpolation
Click to enlarge

I’m not interested in changing the Coordinate Reference System (CRS) of my map—it stays the same (OSGB36). All I’m doing is changing the horizontal resolution to 10m. A bit of experimentation has shown me that using the Cubic spline resampling method produces the most pleasant-looking results. Here’s what I end up with, after producing the 10m-resolution layer and repeating the production of the hillshade layer, as described above:

Blaven, Skye, tinted and shaded at 10m horizontal resolution
Click to enlarge
Contains OS OpenData © Crown copyright and database right 2018

That’s better! In my next post on this topic, I’ll add some contours and other features.


* The sea colour cover isn’t perfect—depending on your graphics card, you may see occasional seams around tile margins when the view is zoomed out. I have some solutions to this, which I may write about another time.

PeakFinder

Back in 1995, a little packet of laminated cardboard diagrams fell through my letterbox. Dave Hewitt, editor of The Angry Corrie, wanted me to write a review of these items. Which I did—it appeared in TAC25, Nov ’95-Jan ’96.

They were called ViewFinder Panoramas, they’d been created by Jonathan de Ferranti, and in my opinion they were things of exquisite, minimalist beauty. Each laminated strip showed the view from the summit of some named hill, colour-coded and annotated to allow the easy identification of other hills. They had been produced from Ordnance Survey Digital Elevation Models, rendered so as to depict the curvature of the Earth and the effects of atmospheric refraction, and then carefully annotated with the names and distances of individual peaks. On occasion, magnified sections were inset to provide additional detail. And features near the horizon were subtly stretched vertically so as to bring out the detail without giving the impression of distortion.

Here’s a comparison of the view eastwards from the summit of Ben Hope, compared to the corresponding ViewFinder diagram:

View of Ben Loyal from Ben Hope
Click to enlarge
Ben Loyal view from Ben Hope ViewFinder
Click to enlarge

From the colour-coded distances, to the sector indicator at bottom left, to the bearings along the top of frame, it was a beautifully designed product. ViewFinders retailed for £1, or £1.50 for the larger, more complex products. You could order a bespoke view from the summit of your favourite hill for £16. Nowadays the entire catalogue is freely available on-line, covering worldwide views.

In 1999, Jonathan de Ferranti and I wrote an article together for the Scottish Mountaineering Club Journal, investigating whether it was possible to see any part of the Cuillin ridge in Skye from the Cairngorm plateau.* In this, we used Jonathan’s ViewFinder technology to revisit a question first raised by Guy Barlow in the same journal in 1956.Barlow had constructed a wood and paper model, and concluded that Sgurr a’Ghreadaidh, on the Cuillin ridge, would be visible from the summit of Cairn Toul in the Cairngorms, because of a fortuitous sightline down the length of Glen Shiel. Jonathan produced a rendering of the same view and discovered that, although Barlow had the alignments exactly correct, he had neglected to allow for the position of Bla Bheinn, which sits east of the main Cuillin ridge, and which neatly blocked the view of Sgurr a’Ghreadaidh. The glimpse of Bla Bheinn, 90 miles away, was minute, occupying more or less a single pixel of the ViewFinder panorama, and in reality would need a telescope, strong refraction and perfect seeing conditions to appreciate. (In the scan below, Bla Bheinn takes its Anglicized spelling, Blaven.)

ViewFinder view of Blaven from Cairn Toul

All of this is a roundabout introduction to Fabio Soldati’s excellent PeakFinder app, which is the natural successor to ViewFinder—indeed Jonathan de Ferranti is credited with providing some of the Digital Elevation Model data used by PeakFinder. With the huge leaps in processing speed and storage capacity that have occurred during the last two decades, it’s now possible to perform the necessary rendering tasks on the fly, producing annotated panoramas of pretty much anywhere in the world, on demand. Apps are available for Android and Apple phones at a cost of a few pounds, and there’s also a rather lovely on-line version. Here’s the view from Ben Hope again, compared with PeakFinder‘s on-line rendered view, and the version displayed by my rather primitive Android phone.

View of Ben Loyal from Ben Hope
Click to enlarge
On-line PeakFinder view from Ben Hope
Click to enlarge
Android PeakFinder view from Ben Hope
Click to enlarge

The on-line version provides shading, shows lochs and coastlines, and offers a few other bells and whistles, but the more basic rendering on my phone is beautifully clear. And having this software as a phone app produces multiple benefits. It can use the phone’s GPS to generate an annotated panorama for your current location, wherever that might be. And the app keeps its database locally, so it will work without a phone signal, provided you have already made the appropriate download. One quick 20MB download covers the whole of the UK, and you can add or delete additional areas as required, with most of the world available in handy chunks of data. If you have a smarter phone than I have, the app will use the phone’s orientation data to overlay its labels on the real view seen by your phone’s camera. This not only lets you easily figure out what you’re look at, but allows you to keep and share an annotated photograph. (Details on that one here.)

Apart from using your phone’s current location, you can select a different location by choosing from PeakFinder‘s extensive names database, tapping on Google Maps (you need a data connection for that), or by entering latitude and longitude coordinates.

Tapping on any of the named peaks in the displayed panorama brings up some information about that feature, and you can also flit across to look at the view from its summit. The names of all the visible peaks in the panorama can be displayed, searched, and sorted by elevation, distance or heading. Tap on one of these names, and you’re returned to the display with a handy marker pointing out that summit’s location. So with a couple of quick taps I was able to establish that the most distant feature visible from the summit of Dundee Law is Windlestraw Law, east of Peebles and a remarkable 88km away. Here’s the PeakFinder display showing me where to look for it:

Windlestraw Law from Dundee Law, PeakFinder app
Click to enlarge

The display units are configurable, and you can pop up depictions of the tracks of sun and moon across the sky, for a given date. Here’s the sun rising over the Orkneys from Ben Hope:

PeakFinder sunrise from Ben Hope
Click to enlarge

Is it accurate? It certainly seems to be, when compared to summit photographs from my collection. For me, Jonathan de Ferranti’s ViewFinder panoramas are the gold standard, so I challenged PeakFinder to show me the extremely marginal view of Bla Bheinn from Cairn Toul that Jonathan identified twenty years ago. Here it is again:

ViewFinder view of Blaven from Cairn Toul

Here’s the view from the web-based version of PeakFinder, with Bla Bheinn again occupying pretty much a single pixel:

Web-based PeakFinder view of Blaven from Cairn Toul
Click to enlarge

And here’s the view on my phone, zoomed in to its maximum extent:

Phone based PeakFinder view of Blaven from Cairn Toul
Click to enlarge

 

No Bla Bheinn. I suspect the difference comes from either the screen resolution or the processing limitations of my dumb phone.

Setting aside ridiculously exacting tests like the one above, this is an extremely impressive bit of a kit, even more so if you have a phone that will allow you to use the app’s photographic annotation mode. If you’ve ever sat by a cairn and indulged in an endless, fruitless debate about the identity of some little notch on the horizon (and which of us has not?), then you’ll certainly want to spring a few quid on this lovely little app.


* “On Seeing The Cuillin From The Cairngorms—Again” SMCJ 1999, Vol. 37 No. 190 pp 42-8.
“On The Possibility Of Seeing The Cuillin From The Cairngorms” SMCJ 1956, Vol. 26 No. 147 pp 16-24.

Which Place Gets The Most Daylight?

So this puzzle isn’t about sunshine (the amount of time the sun shines from a clear sky), or even about the intensity of sunlight (which decreases with increasing latitude), but about cumulative daylight—the length of time between sunrise and sunset in a given place, added up over the course of a year.*

It’s a surprisingly complicated little problem. I addressed it using an antique solar calculator I wrote many years ago, using Peter Duffett-Smith’s excellent books as my primary references:

Peter Duffett-Smith booksIt runs in Visual Basic 6, which means I had to open up my VirtualBox virtual XP machine to get it running again. The original program calculates the position of the sun by date and time for any given set of coordinates, and also works out the times of sunrise and sunset.

Solar calculator written in VB6You’ll see it gives sunrise and sunset times to one-second precision, which is entirely spurious—the refractive state of the atmosphere is so variable that there’s no real point in quoting these times to anything beyond the nearest minute. I just couldn’t bring myself to hide the extra column of figures.

Anyway, it was a fairly quick job to write a little routine that cycled this calculator through a full year of daylight, adding up the total and spitting out the results so that I could begin exploring the problem.

At first glance, it seems like there shouldn’t be any particular place that wins out. As the Earth moves around the sun, its north pole is alternately tilted towards the sun and away from it, at an angle of about 23.5º. If we look at a diagram of these two solstice points (which occur in June and December every year), there’s an obvious symmetry between the illuminated and unilluminated parts of the globe:

SolsticesBetween the solstices, the latitude at which the sun is overhead varies continuously from 23.5ºN (in June) to 23.5ºS (in December), and then back again:

Solar latitude by day of the year
Click to enlarge
Solar latitude by day of the year

So for every long summer day, there should be an equal and opposite long winter night. The short and long days should average out, during the course of a year, to half a day’s daylight per day—equivalent to 4280 hours in a 365-day calendar year.

And that would be true if the Earth’s orbit around the sun was precisely circular—but it isn’t. As I described in my first post about the word perihelion, the Earth is at its closest to the sun in January, and its farthest in July. Since it moves along its orbit more quickly when it’s closer to the sun, it passes through the December solstice faster than through the June solstice. This has the effect of shortening the southern summer and the northern winter. The effect isn’t immediately obvious in my diagram of solar latitudes, above, but it’s there—the sun spends just 179 days in the southern sky, but 186 days north of the equator.

This means that the total number of hours of daylight is biased towards the northern hemisphere. In the diagram below, I plot the hypothetical flat distribution of daylight hours associated with a circular orbit in purple, and compare it to the effect of Earth’s real elliptical orbit in green:

Hours of daylight by latitude (1)
Click to enlarge

So far, I’ve been treating the sun as if it were a point source of light, rising and setting in an instant of time. But the real sun has a visible disc, about half a degree across in the sky. This means that when the centre of the sun drops below the horizon, it’s only halfway through setting. Sunrise starts when the upper edge of the sun first appears; sunset finishes when the the upper edge of the sun disappears. So the extent of the solar disc slightly prolongs daylight hours, and slightly shortens the night.

At the equator the sun rises and sets vertically, and the upper half of the solar disc takes about a minute to appear or disappear. An extra minute of daylight in the morning, an extra minute of daylight in the evening—that’s more than twelve hours extra daylight during the course of a year, just because the sun is a disc and not a point.

And if we move north or south of the equator, the sun rises and sets at an angle relative to the horizon, and so takes longer to appear and disappear—adding more hours to the total daylight experienced at higher latitudes. There’s a limit to this effect, however. When we get to the polar circles, we run into the paired phenomena of the midnight sun and the polar night. There are days in summer when the sun never sets, and days in winter when the sun never rises.  The extent of the solar disc can make no difference to the length of daylight if the sun is permanently above the horizon, and it can add only a few hours to the total as the sun skims below the horizon at the start and end of polar night.  And as we move towards the poles, the midnight sun and polar night start to dominate the calendar, with only short periods around the equinoxes that have a normal day/night cycle. So although the sunrises and sunsets within the polar circles are notably prolonged, there are fewer of them.

Day and night above the Arctic Circle
Click to enlarge
Midnight sun and polar night begin to dominate over the normal day/night cycle as you go farther north (the situation is symmetrical in the southern polar circle, except with the times of polar night and midnight sun reversed)

So the prolongation of daylight caused by the rising and setting of the solar disc increases steadily with latitude until it peaks at the polar circles (around 66.5ºN and 66.5ºS), after which it declines again. Here’s a diagram of daylight hours predicted for a point-like sun (my previous green curve) with the effect of the solar disc added in red:

Hours of daylight by latitude (2)
Click to enlarge

And there’s another effect to factor in at this point—atmospheric refraction. As I described in my post discussing the shape of the low sun, light from the rising and setting sun follows a slightly curved path through the atmosphere, lifting the image of the sun by a little over half a degree above its real position. This means that when we see the sun on the horizon, its real position is actually below the horizon. This effect hastens the sunrise and delays the sunset, and it does so in a way that is identical to simply making the solar disc wider—instead of just an extra couple of minutes’ daylight at the equator, more than six minutes are added when refraction is factored in, with proportional increases at other latitudes. So here’s a graph showing the original green curve of a point-like sun, the red curve showing the effect of the solar disc, and a blue curve added to show the effect of refraction, too:

Hours of daylight by latitude (3)
Click to enlarge

The longest cumulative daylight is at the Arctic Circle, with latitude 66.7ºN experiencing 4649 hours of daylight in the year 2017. The shortest period is at the south pole, with just 4388 hours. That’s almost eleven days of a difference!

So is the answer to my original question just “the Arctic Circle”? Well, no. I have one more influence on the duration of daylight to deploy, and this time it’s a local one—altitude. The higher you go, the lower the horizon gets, making the sun rise earlier and set later. This only works if you have a clear view of a sea-level (or approximately sea-level) horizon—from an aircraft or the top of a mountain. Being on a high plateau doesn’t work, because your horizon is determined by the local terrain, rather than the distant curvature of the Earth. So although the south pole has an altitude of 2700m, it’s sitting in the middle of the vast polar plateau, and I think there will be a minimal effect from altitude on the duration of its daylight.

So we need to look for high mountains close to the Arctic Circle. A glance at the map suggests four mountainous regions that need to be investigated—the Cherskiy Range, in eastern Siberia; the Scandinavian Mountains; Greenland; and the region in Alaska where the Arctic Circle threads between the Brooks Range to the north and the Alaska Range to the south.

Arctic Circle
Click to enlarge
Source of base map

The highest point in the Cherskiy Range is Gora Pobeda (“Victory Peak”). At 65º11′N and 3003m, its summit gets 5002 hours of daylight—almost an hour a day of extra sunlight, on average.

But Pobeda is nudged out of pole position in the Cherskiy Range by an unnamed 2547m summit on the Chemalginskiy ridge, which lies almost exactly on the Arctic Circle, giving it a calculated 5006 hours of daylight.

There’s nothing over 2000m near the Arctic Circle in the Scandinavian Mountains, so we can skip past them to 3383m Mount Forel, in Greenland, at 66º56′N, which beats the Siberian mountains with 5052 hours of daylight.

Finally, the Arctic Circle passes north of Canada’s Mackenzie Mountains, and between the Brooks and Alaska Ranges. Mount Isto, the highest point in the Brooks Range, is 2736m high at 69º12′N, and comes in just behind Pobeda, with 4993 hours of daylight. Mount Igikpak, lower but nearer the Circle (2523m, 67º25′N), pushes past all the Siberian summits to hit 5010 hours. And in the Alaska Range is Denali, the highest mountain in North America. It is 6190m high, and sits at 63º04′N. It could have been a serious contender if it had been just a little farther north—but as it is it merely equals Igikpak, and falls short of Forel’s total.

So the answer to my question appears to be that the summit of Mount Forel, Greenland, sees the most daylight of any place on the planet. I confess I didn’t see that one coming when I started thinking about this.


* “A year” is a slightly slippery concept in this setting. The sun doesn’t return to exactly the same position in the sky at the end of each calendar year, and leap years obviously contain an extra day’s daylight compared to ordinary years. Ideally I should have added up my hours of daylight over a few millennia—but I’m really just interested in the proportions, and they’re not strongly influenced by the choice of calendar year. So for simplicity I ran my program to generate data for 2017 only.

What I wrote at the start of this piece, about spurious precision in rising and setting times, goes doubly for the calculations concerning altitude. These results are exquisitely sensitive to the effects of variable refraction, and my post about the shape of the low sun gives a lot more detail about how the polar regions are home to some surprising mirages that prolong sunrises and sunsets. I can’t hope to account for local miraging, or even to correctly reproduce the temperature gradient in the atmosphere from day to day. I think the best that can really be said is that some of the contenders I list will experience more daylight than anywhere else on the planet, most years, and that Mount Forel will be in with a good shot of taking the record for any given year.

Running Windows XP Under VirtualBox

As I write, it’s only another month until Microsoft’s free upgrade offer on Windows 10 expires (on 29 July 2016). I am so looking forward to that day, in the hope that it’ll mean an end to Microsoft’s intrusive little pop-up messages in the lower right corner of my monitor, and their increasingly devious attempts to trick me into accidentally upgrading. The experience is a little like having a weeping software engineer repeatedly grip your lapels, shake you gently, and sob: “But it’s cool. It’s free. Why don’t you want it? For pity’s sake, why? Why?

I don’t want it because it offers nothing I need or am even curious to see, while promising inconvenience and hassle during the upgrade process. When Microsoft are willing to pay me for the time I’ll spend on their “free” upgrade, then maybe we can talk.

But at present, I’m much cheered to be running Windows XP again, which marked the last occasion I ever felt a new Microsoft operating system actually constituted an “upgrade”. The fact that you can run a virtual XP machine in a window under Windows 7, 8 or 10 is not as well known as it should be, and it has let me continue using some old software from that era that has resisted running in “compatibility mode” under later operating systems.

Microsoft offered a Windows XP Mode for Windows 7, which could be installed and run under that operating system. The file is still available from the Microsoft Download Center (click on my link), and it contains a virtual hard drive with an XP installation—so to get XP running under Windows 8, 8.1 or 10, all you have to do is retrieve that VHD.

I downloaded WindowsXPMode_en-us.exe from my link above. Since I wanted to run XP under Windows 8.1, I didn’t run the exectuable—I tucked it away in my Downloads folder. Embedded inside two layers of compression is the virtual hard drive I needed.

I used 7-Zip to find and extract the necessary file. 7-Zip is a handy, open-source archive manipulation program, which adds a couple of options to the Windows menu you see when you right-click on a file. So I navigated to where I’d stored WindowsXPMode_en-us.exe, right clicked on the file, selected “7-Zip” and then “Open archive”. 7-Zip then gave me a view of the contents of the archive file, which includes a folder named sources. In that folder, there’s a file called xpm. This is also compressed, so I right-clicked it and opened it in 7-Zip. In the archive listing for xpm, there’s a file called VirtualXPVHD. That’s the virtual hard drive containing the XP installation. I right-clicked it, and then selected “Copy to …”, telling 7-Zip to extract and decompress VirtualXPHD. I put it in a new folder called XP.

7-Zip screenshot 1
7-Zip showing the xpm file in the sources folder (Click to enlarge)
7-Zip screenshot 2
7-Zip showing the VirtualXPVHD file inside the xpm file (Click to enlarge)

So that Windows could recognize VirtualXPVHD as a virtual hard drive, I now edited the file-name by adding a .vhd extension to it.

For an environment in which to run my new VHD, I downloaded VirtualBox, and installed it with the default options. Then I ran the program, told it I wanted to set up a new virtual XP machine using an existing virtual hard disk, and pointed it to the location of my VirtualXPVHD.vhd file.

Welcome to VirtualBox
VirtualBox, ready to go (Click to enlarge)
Create Virtual Machine
Creating the virtual machine (Click to enlarge)
Using VirtualXPVHD.vhd
Using VirtualXPVHD.vhd (Click to enlarge)

And that was that. Now you I could launch an XP machine in a window on my Windows 8.1 desktop.

New XP machine, ready to go
New XP machine, ready to go (Click to enlarge)

VirtualBox does a lot of handy things. There are a couple worth knowing about if you’re setting up your own virtual machine:
1) You can scale the window in which the XP machine runs, using the menu option View/Scale Factor. This is useful if you find yourself peering at a tiny window in the middle of your high resolution monitor.
2) At first, you’re going to need to let the XP window “capture” your mouse pointer. When you click inside the window, a dialogue box appears offering to capture the pointer. When you accept, you find your mouse is trapped inside the XP window. You can release it again by tapping a “hot key”, which defaults to the right control key on your keyboard. It’s worth checking that you actually have a right control key before you fire up XP for the first time, otherwise your mouse will be permanently trapped. If you don’t have one, you can change to a new hot key in the VirtualBox menu, File/Preferences/Input/Virtual Machine/Host Key Combination. Select the “Shortcut” box and press whichever key you want to designate as the new hot key.

I found the virtual machine was a bit crashy when XP was going through its initial set-up on my laptop (but not on my desktop). A couple of times I had to press the hot key to free up my mouse, and send a “power off” signal to the virtual machine using the VirtualBox interface. On each occasion it rebooted happily and let me proceed further with the set-up.

XP under VirtualBox
XP running at low resolution and with its horrible “Early Learning Centre” theme (Click to enlarge)

The first priority once you have the XP desktop on display is to add some additional function. Free up your mouse pointer so that you can use the VirtualBox menu at the top of the window, click on “Devices”, and then “Insert Guest Additions CD Image …” This makes the XP virtual machine think you’ve inserted a software CD, and it opens a set-up dialogue to let you install the new software. Accept this (and choose “Install Anyway” each time XP objects to the certification of the software that’s being installed).

Guest Additions CD image
Running the Guest Additions CD image (Click to enlarge)

Now you don’t need to go through the business of capturing and releasing the mouse pointer! The XP window is integrated into your desktop and behaves like any other windowed software. (On occasion, you may find that the mouse pointer behaves a little oddly in some programs—flickering, or scanning too quickly. On those occasions, capturing the mouse is useful. You can turn mouse capture on and off using Input/Mouse Integration in the VirtualBox menu at the top of the window.) I also found that installing the Guest Additions CD eliminated the crashes I’d been having during setup.

One hitch in all this was that XP soon announced that it needed activation, and demanded a Product Key. Now, back in the day, computers used to come with a copy of their operating system on a disc, and I still had the old XP installation disc for a long-defunct computer tucked away in a cupboard. I offered the Product Key from that disc, and XP was happy. In fact, I now have three virtual XP machines all happily registered with the same Product Key. Which is not unreasonable, given that Microsoft haven’t actually been supporting this operating system for a couple of years now.

XP wants to be activated
XP wants to be activated
Activating XP
Activating XP
XP successful activated
Success!
Initial tidying up of XP
Classic theme, adjustment ot resolution, disposed of some Microsoft defaults (Click to enlarge)

A couple of the programs I installed under XP are so old they’ll only run if the parent disc is in a drive for them to access. That’s not exactly convenient, so I copied the necessary discs to .iso image files using DVD to ISO , and put the .iso files into a directory on my virtual XP machine.

Then I installed the Virtual CD-ROM Control Panel from Microsoft, which lets XP mount .iso files as if they were physical discs. It’s slightly finicky to install, but the readme file talks you through the process. You need to move the .sys driver file to your Windows XP drivers directory, and then run VCdControlTool.exe to install the driver. Once the driver is installed, running VCdControlTool again lets you create an unused drive letter and then mount an .iso file to that drive. I set my .iso files up as “persistent mounts”, so they’re always available.

Mounting a couple of disc images using VCdControlTool
Mounting a couple of disc images using VCdControlTool (Click to enlarge)
Two disc images mounted as drives D: and Z:
The two disc images mounted as drives D: and Z: (Click to enlarge)

Finally, although I’m no great fan of cloud storage (sure, I’ll let a multinational corporation store my personal documents and photographs at some random location on the internet; what could possibly go wrong with that idea?) I do like to share a few program and configuration files between my various devices. The small storage capacity offered by the free version of DropBox has been more than enough for this—and anyone who cares to hack into my DropBox account isn’t going to find much that’s comprehensible to them, let alone useful. But DropBox discontinued support for XP recently, so I needed a way to transfer files from my XP virtual machine to a directory on its parent machine, where DropBox could then take over.

VirtualBox lets you set up folders that are shared between the XP virtual machine and the parent machine (go through Devices/Shared Folders) but some of my XP software is so ancient it refused to recognize the network drive on which these shared folders resided. So I tried using the free version of Tonido. Tonido synchronizes files through their server without ever storing them—I installed the server software on the parent machine, and the client software on the virtual XP machine. Presto! My shared XP files were transferred to the parent machine, where they could be DropBoxed or backed up as required. This had the advantage of being easy to set up, but the rather bizarre and deeply unsatisfying consequence of a computer transferring files to itself over the internet. It also has to be said that the Tonido file synchronization was often slow, and on occasion delayed for hours.

I found a solution in Link Shell Extension. Windows, in its various incarnations, provides various ways of tying files and folders together, so that a file that resides in one location is also accessible in another location. Traditionally, these hard links, junctions and symbolic links could only be set up from the command line prompt, but Link Shell Extension, once installed, lets you set them up from the mouse pointer with a right-click. Right-click on the origin file or folder, and select “Pick Link Source”; navigate to where you want a linked copy, right-click again, and select “Drop As …” Job done. Link Shell Extension provides all sorts of different ways to produce linked copies of folders and files—it’s well worth reading the description on their website (linked above) before you starting playing with it.

I’d already been using Link Shell Extension with DropBox, linking the DropBox copies to the parent folders my programs worked from. But in this instance what I needed was a link between a network drive and a parent folder in XP—that needs a symbolic link, which was supposedly not available under XP. But Link Shell Extension works with an XP driver written by Masatoshi Kimura, which adds symbolic link functionality to XP. The Link Shell Extension page provides download links and instructions on how to install the extra driver. Unfortunately, it’s slightly technical, requiring a one-off instruction from the command prompt. The page also says you’ll need to edit your Windows registry to make the driver permanently available. Interestingly, I found this wasn’t required on my virtual machines—the registry value HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SymLink\Start had already been set to zero when I checked, and it stayed that way—the Symbolic Link option was available immediately, each time I booted up.

Anyway, once I’d got that straightened out, I used the Devices menu in VirtualBox to make the DropBox folder on the parent machine a shared folder, ticking the boxes to make it “auto-mount” and “permanent”. It then appeared as a network drive the next time I booted the virtual XP machine. Using Link Shell Extension in XP, I could pick folders inside the DropBox network drive, and drop copies of them as symbolic links on the XP c:\ drive, where my ancient programs could see them. When the programs modify those files, the modification is then reflected in the DropBox folder on the parent machine, and that cascades off to my other devices, and their virtual XP machines. Joy!

So now, every time Microsoft offers me a free upgrade, I can sit back and enjoy my free downgrade instead.

Pennycook et al.: On the reception and detection of pseudo-profound bullshit

This from the November 2015 issue of Judgment And Decision Making. Here are links to the original paper (pdf) and its supplementary tables (pdf).

The authors seek to find a preliminary answer to the questions, “Are people able to detect blatant bullshit? Who is most likely to fall prey to bullshit and why?” Their study is therefore of the characteristics of the  bullshittee, rather than the bullshitter, or of bullshit itself.

They suggest that bullshit occupies a sort of halfway house between lie and truth. Bullshit is “something that is designed to impress but […] constructed absent direct concern for the truth.” (That is, the author of bullshit doesn’t care whether it’s true or not, in contrast to the liar, who is deliberately subverting the truth.) And “bullshit, in contrast to mere nonsense, is something that implies but does not contain adequate meaning or truth.”

I’m indebted to them for providing links to two sources of pseudo-profound bullshit, used in their study.

One, Wisdom of Chopra, uses random words taken from the Twitter feed of Deepak Chopra to construct novel sentences. Here’s an example of its output:

The unexplainable arises and subsides in the doorway to energy

The other, Seb Pearce‘s New-Age Bullshit Generator, generates an entire, beautiful page of random bullshit. Here’s one headline:

You and I are entities of the quantum matrix. By evolving, we believe

So that’s all pseudo-profound bullshit.

According to Pennycook et al., reasons you might mistake that for actual profundity include:

  • A deficiency of analytic thinking
  • Ontological confusion (confusing different categories of existence, such as the mental and the physical)
  • Epistemically suspect beliefs (such as paranormal or supernatural ideas)

Four studies are reported in the paper. They all look for correlations between the particular cognitive biases listed above with a “Bullshit Receptivity” scale—a measure of an individual’s tendency to rate randomly generated bullshit as “profound” on a five-point scale ranging from “not at all profound” to “very profound”.

I haven’t even counted the number of separate correlation measures to which the authors assign significance values; I’ll leave that as an exercise for the  Interested Reader.

But what we seem to see is that:

  • Participants tended to score random nonsense as moderately profound.
  • Participants scored selected real Deepak Chopra Tweets as a little more profound than random nonsense, but less profound than some motivational quotations.
  • Some participants scored even mundane statements like “Most people enjoy some sort of music” as having some level of profundity. These participants tended to give high profundity scores across the board.
  • To quote the authors: “Those more receptive to bullshit are less reflective, lower in cognitive ability (ie. verbal and fluid intelligence, numeracy), are more prone to ontological confusions and conspiratorial ideation, are more likely to hold religious and paranormal beliefs, and are more likely to endorse complementary and alternative medicine.”
  • Waterloo University undergraduates (or at least, those who sign up for this sort of study) are catastrophically gullible, assigning various levels of profundity to some quite astonishing twaddle (Table 1). Snake-oil salesmen are presumably converging on the campus even as I type.

So it’s good to have all that sorted out.

Book Collector

When you have more than 4000 books scattered around the house, it gets difficult to find the one you’re looking for. Especially if you’re hunting for a short story and you can’t quite remember which book you read it in. This used to happen a lot, chez Oikofuge. But not any more.

Book Collector is a book cataloguing program from Collectorz.com, and it’s the best I’ve run into. I can’t now imagine life without it.

Data entry is easy, and highly automated. If your book has an ISBN, you can type it in or scan it with a barcode reader, and the basic book data are pulled down off the Collectorz.com central database. These days, the ISBN is readily visible on the back cover, but with older books (1967 to the mid-70s) you may have to look for numbers written on the spine, or listed in the front matter. Some UK publications of that vintage have nine-digit SBNs instead of ten-digit ISBNs, but the conversion is easy—just add a zero at the left end.

Before 1967, there were no ISBNs, but Book Collector also lets you add books automatically by entering the author and title. This option will bring up all matching entries in the Collectorz.com database, so you might need to do a little poking around to find the entry that matches your specific edition.

That will get the basic data into your database, including a version of the cover art if it’s available. But the software offers a huge number of additional relevant data-fields, which you can fill in or ignore according to your wishes. (You’ll probably want to make use of the “book location” field, unless you have a memory much better than mine.) It even lets you create custom fields.

First page of data entry screen
First page of the data entry screen
Note all the additional tabs at the top

The on-line cover art comes from a variety of sources—it varies in size and quality, and can occasionally be for the wrong edition of your book. But Book Collector has an automated search facility that lets you look for more cover art on-line. It also lets you add your own art by scanning the cover. If you’re of an obsessive nature (who, me?) you may find yourself scanning a lot of book covers to get precisely the right edition.

You can view your database in various ways, usually splitting the view between some sort of overview of the books, and a detailed view of a specific volume. The overviews available are a “bookshelf” depiction of cover art (which I find useful when browsing for a specific book) and a spreadsheet-type display of multiple customizable columns. There’s also a “cover flow” option available, but the less said about that the better—it’s the sort of triumph of style over utility that could only appeal to an Apple user.

Book Collector screen capture images view
Books containing Asimov short stories, displayed using one version of the “Images” view on the left, with a “Details” view of a specific book on the right. Note the contents list.
Book Collector screen capture 2
The same short stories list, this time displayed with a “List” view using customized columns on the left, and “Details” view on the right

You can choose from a growing number of different formats for viewing your book details; or, with a little knowledge of HTML/ XML, and some digging around in the file structure, you can customize up your own view.

Searching is easy. There’s a quick-and-dirty search option that just looks for your chosen text anywhere in the book’s description. It’ll bring up false hits, but often it lets you narrow down the display enough to zero in visually on the book you want. But you can also create moderately sophisticated “filter” views, using simple Boolean logic functions, to pull out the books you want.

Boolean search for Asimov short stories
Boolean search for Asimov short stories

As someone who has a lot of short stories in my book collection, I particularly value the fact that I can enter a contents list for my books. Book Collector offers you a cut-down database and user-defined fields for each short story in a book. Unfortunately, the contents list won’t come down to you automatically from Collectorz.com—manual entry is required, which can be tedious if you have a lot of “complete works” volumes on your shelves. And I would appreciate it if Book Collector some day offered a detailed view by short story as well as by book. But that’s a minor niggle when I can choose to display anything I want in the columns of the spreadsheet view.

What else? Collectorz.com offers a cloud storage facility, so you can be sure you have the same data on all your devices. There’s a responsive Support team (I’ve only ever had to use them once) and an active users’ forum where people are happy to help out with minor queries. And there’s a free try-before-you-buy download.

If you’re in the market for book cataloguing software, do give it go.

Software: Introduction

I started computer programming on punched tape and IBM 80-column punched cards, using Fortran, back in 1974.

Punched tape
(Source)
Punched card
(Source)

One of my first teachers was a young woman who could read the program directly off the punched tape, and debug it using a hole punch and some sticky dots. I fixated on her utterly, like a baby duckling.

Later, I moved on to Teletype terminals, on which I programmed primitive astronomy calculations. Then my very own computer (who saw that coming?) the Sinclair ZX Spectrum, plugged into the television, on which I built a primitive model of the solar system using early editions of Peter Duffett-Smith’s fine astronomy programming books.Peter Duffett-Smith books

Apple IIs and PCs after that, mainly working in various dialects of Basic, and then making the switch to event-driven Visual Basic when it came along in the early 90s.

I produced and published various utility programs for the medical specialty I worked in, and even sold a couple of units of a reference-manager program I wrote as an add-on for the early scientific word processor, ChiWriter. But it was all essentially recreational until I made the mistake of writing a rostering program for the department I worked for in 1997 (out of frustration with the very poor quality of commercial rostering software at that time). That put me on a maintenance treadmill for the next fourteen years.

So I rather lost interest in developing any more complete software packages at that point! My other programming activity therefore became confined to writing little routines to get some particular piece of data-handling or calculation done, as part of some other project. I did a lot of that when I was a developer on the open-source 3D planetarium software, Celestia, and also when accessing NASA’s Shuttle Radar Topography Mission data to research mountains for Ginge Fullen’s Africa’s Highest Challenge, in which he climbed the highest point of every country in Africa.

SRTM data for Bikku Bitti
Extracted SRTM data, checking the position of the highest point in Libya, Bikku Bitti

 

Bikku Bitti photo
“Ground truth” photo of Bikku Bitti, sent to me by Ginge Fullen after he completed Africa’s Highest Challenge

I have a number of little software projects in mind for the future, some of which I may share here. But I think initially this category of blog post will be devoted to a couple of reports on software I find useful and/or fun, and want to recommend.