Managing and migrating GIS resources in the cloud has become an increasingly regular feature of my work life over the last couple years. I think I may have gotten to a point where I’m beginning to take for granted some things that are still matters of curiosity to people who don’t have their heads in the cloud(s) so much as I do. In recent conversations with industry friends and clients, I’ve noticed interest around questions related to approach and tactics for moving computing operations, data, and applications to cloud platforms. I thought it might be helpful to share some of these details, using a recent migration project to provide a bit of context and some specifics.

Last summer, our team developed a work-back schedule to move the majority of our own remaining server and hosting resources into the Microsoft Azure platform. We had already deployed a number of our own systems in both Azure and Amazon Web Services (AWS), but a smorgasbord of on-premise and server farm-based virtualized servers, databases and applications had not yet made the journey.

As is the case with many of the clients we support with cloud deployments, our rationale for this was a blend of budget, technical currency, performance, and ease of management. From a budget point of view, it was cheaper to host virtual machines of the same specifications in Azure vs our hosting environment, and further compelling was the ability to scale resources up and down based on demand and to experience a similar elasticity in our costs — only paying for what we needed and actually used. We have also come to enjoy the granular management options offered by Azure and AWS and looked forward to the advantages of being able to oversee nearly all our resources through their respective admin dashboards.

What follows is a brief-as-possible summary of some of the strategies, tools, observations, and hurdles we encountered on our migration journey.

Our Project Plan included running dual environments for a minimum of 30 days and for up to 45 days. As a note, I really didn’t think we would need even 30 days, but we ended up needing an additional week (!).

*Tip 1: Even to the most meticulous among you, I recommend developing a detailed Migration Plan, including validation testing, and….building in some time cushioning…there’s always a lot of detail in these efforts!

Our servers (both physical and virtual) would be migrating to Azure virtual instances, but our databases would be migrated to the ‘serverless’ Azure SQL environment. Additionally, our intentions were to set up horizontal auto-scaling on 2 of the machines - which implies an additional “clone” machine ready to power up at any time to accommodate increases in traffic and/ or computing demand.

As an attempt to maintain sanity in the migration of our environment with its many years’ worth of applications, services, databases, files, and custom configurations, we began the project with a Trello board. For each server, plus the 2 additional auto-scaling machines, we created a list or ‘swim lane’. If you haven’t used or heard about Trello, I strongly recommend it as a productivity tool. Trello, in its most basic form, is a collection of lists - each with a collection of ‘cards’. You can interact with the lists and cards and supply additional media to the cards such as images, deadlines, checklists etc. If nothing else, Trello helped us to wrap our heads around the multitude of details. Cards in each list included basic access credentials, applications, database, and network configuration details.

*Tip 2: Consider using Trello or a similar task and list management tool to track, sequence, and categorize the considerable detail and information relevant to different parts of your migration effort.

Other tools that came in handy in the migration included Azure Storage Explorer, Microsoft’s Data Migration Assistant, and the Azure Dashboard. Part of our new environment entailed the use of Azure Files Storage as a “mountable” SMB-style share- in other words, a cloud disk resource that is mappable as a drive letter (to virtual machines in Azure). Azure Storage Explorer provided us with a FTP-like desktop client through which to transfer data - it’s fairly easy to use and seems to add some extra special sauce to upload/download transfer rates. This tool became very useful for moving data off old machines that couldn’t directly connect to our Azure Storage resources.

*Tip 3: For Azure-focused migrations, have a look at the Azure Storage Explorer as a means for moving data from your old systems to the new Azure target environment.

In order to migrate our databases we used the Microsoft Data Migration Assistant. This tool allows you to migrate a database from a SQL Server instance to SQL Azure and proved helpful and reliable. Crucial features missing from this tool however, IMHO, are a way to save a custom migration (included only certain tables, views, SPs etc), and an option to “refresh” or reload data during the cutover. We performed several trial runs of the migration with no problems but had to devise our own strategy and scripts for truncating tables and quickly reloading them during the cutover. Another resource I am aware of but have yet to use is the MS Azure Database Migration Service; you might also give that a roll (and let me know how it works for you;) ).

*Tip 4: With a couple noted caveats, the Microsoft Data Migration Assistant is your friend for traditional SQL to Azure SQL database migrations.

During the final week of our migration, things went relatively smoothly and our verification testing sequence was humming along with nary a hiccup. Of course, just as we were remarking on the smooth finish of the nearly-complete project… a specialized network configuration detail threw us a curve ball that we hadn’t expected and so slowed our final victory. When creating firewall rules for one of our clients whose systems were included in the migration, we hadn’t realized the extent of their own internal use of Azure - as it turned out, they were making use of Multiprotocol Label Switching (MPLS) and our machine was not receiving their connection under the IP address we had used elsewhere.

*Tip 5: Consider potential MPLS and other network-to-network protocol concerns when planning your migration. A simple re-mapping of firewall rules may not always be sufficient when moving onto a cloud platform, especially if you have other resources deployed therein.

Overall the migration was a success and we’re now entirely “perched” in the clouds. Stay tuned for an additional post describing our experiences and approaches that may be helpful for your planning.

Feel free to drop us a line if you have questions about any of these shared details or might be looking for a sherpa to help you make your way to the land of clouds.