A Penny for Your Thoughts, a Million Tiles for Your Basemap
Basemap File Structure
Basemaps of large areas with many viewing scales can consist of millions of individual graphic files
The “basemap” is a central feature of the interactive web and mobile mapping apps that many of us use repeatedly each day. “Basemaps” are the background maps which provide the context and orienting details to which the themes and focuses of individual maps are added (think of street view, satellite view, or terrain view as examples of common basemaps). There are a variety of methods by which basemaps may be produced, but one enduring approach involves creating a cache of raster tiles consisting of thousands to millions of small graphic files which may be combined to form a map background. We are actually moving through these elegantly batched files as we zoom in and out of different viewing scales or use mouse clicks and smart phone gestures to trace hiking trails, pan across neighborhoods or hunt for happy hour.
Building a tile cache has been a very time and resource-intensive activity. Were you to look at the structure of a tile cache on a server, you’d observe a sort of pyramidal or hierarchical directory structure. Proceeding into the cache to the more “zoomed in” viewing scales, you’d witness an exponential growth in the number of individual tiles comprising the basemap. Making all those individual graphics files can humble a robust computer.
Traditionally, many organizations have established dedicated servers which they use to initially create and to maintain the tile caches for their basemaps. During tile cache generation, these servers have tended to run hot, enlisting all available processing resources, often for a period of hours or even days. But caching tends to be a periodic activity, so between cache-making events, many organizations have had powerful and relatively expensive server systems sitting idle and underutilized.
We recently did a bit of R&D using some emerging cloud technology to see if there might be a new way to address this caching conundrum. If you’re familiar with the struggles associated with tile caching, or perhaps with an analogous activity that involves intermittently intensive CPU or GPU activity, we hope you find some value in reading more about our experiment and findings related to using Windows Virtual Desktop for creating basemap tile caches.
Here Comes WVD
Microsoft has recently released Windows Virtual Desktop or “WVD” (really?). It allows end users to to access virtual desktops deployed on Azure cloud infrastructure. You can get at the desktops directly from standard web browsers, optionally running from lightweight laptop and desktop computers. You can even use a Mac or Chromebook, as the solution is agnostic with regard to the client operating system. The virtual desktops are available as Windows 7, 10 or even Windows Server editions and they support multiple user sessions. WVDs can be activated and relinquished quickly and with ease. Among their other key purported virtues are that they excel at enabling users to complete very CPU or GPU intensive tasks. Because tile caching is such a famous processor hog, we thought it might be a good candidate operation to explore the capabilities and possible applications of these cloud-based virtual desktops.
The Test - Thing 1 v Thing 2
Our test was pretty straightforward:
let’s create the same basemap, using the same source data and going to the same zoom levels (we used an Open Source Python / GDAL program to perform the tiling which went from 0 to level 17);
let’s do it once on a more traditional server;
let’s rinse and repeat on in a WVD session;
let’s clock both and compare results.
Our source data was some imagery from the U.S. National Agricultural Imagery Program (NAIP) located roughly and randomly in the Finger Lakes Region of upstate New York.
Our ‘traditional server’ (“Thing 1”) was a Virtual Machine:
deployed in Azure
(2) Xeon E5-2660 2.2GHz processors
16 GB of virtual RAM (not a super womper, but satisfactory for testing we thought).
“Thing 2” was a WVD with a bit more cowbell:
24 Xeon E5-2690 2.6GHz processors;
224 GB RAM
…we spun Thing 2 up just before the test, and put it back to sleep shortly after.
By the Dashboard Light…
So Thing 1 ran real hot for 277 minutes. The cores were pegged throughout. While it’s true that Thing 1 may have been something less than representative of the type of server people in the mapping business might devote to tiling functions, it is also true that the spatial extent and number of zoom levels (17) in our test basemap were also fairly light compared to what many tiling jobs encompass.
Thing 1 was pegged at 100% processor utilization for 277 minutes
And how about Thing 2? It ran about 10 times faster than Thing 1 and the seemed to keep its cool throughout the tiling job, according to the Performance Monitor.
We were pleased with the outcome of our baseline tile caching test. While it was a fairly basic validation effort, we think it was sufficient to demonstrate the promise that WVD holds for activities that involve intermittently intensive processing. WVD-based tiling may be worth your consideration if you have concerns such as:
The WVD kept it relatively cool. No redlining though 10x faster. Then back to bed when done.
Paying for expensive, static caching servers that are not continuously used / cost-justfied;
Running long-duration tiling processes that may be prone to disruption due, in part, to network or resource issues which may arise during the course of multi-hour or day processing.
A side note to our test — we were pleasantly surprised by the visual quality and responsiveness of the virtual desktop. As opposed to a standard “Remote Desktop” session which can be fairly dreadful if you’re stuck working in one for any length of time, the WVD session does a much better job of creating a sense of presence and obscuring the fact that one is working with computer resources that may be located in a data center hundreds of miles distant. For this and other reasons, we look forward to evaluating WVD for more graphically-focused endeavors.
If any of this has prompted questions or ideas about your own potential use of virtual desktop resource, please feel free to reach out to us. We’ll be happy to share additional information or to learn more about what you may hope to achieve with this type of technology.