Data Shame, Big Data, and the Need to Tell a Story

Final edits are just going in on a Database Foundation Plan for the Oregon Department of Land Conservation and Development (DLCD). We are tentatively emerging from the thickets of statutory law, plan amendment procedures, urban growth boundary expansions, appeals, hearings…. Before burrowing into the next project and it’s challenges, it’s refreshing to take a moment above ground and reflect on our work.
If you go with the marketing, ours is the age of ‘big data,’ and discovering patterns, trends, and critical relationships in behemoth and organization-straddling data sets is our grail.  In this context, I find that ‘data shame’ is a rampant if little acknowledged condition that extends into the recesses of many companies and agencies.  I don’t have statistics to share, but I know that this vexing malady is experienced in varying degrees by many people who must rely upon particular data to inform decision making and to direct actions that have real consequences.  Expectations are high, hype is unchecked, and many people endure the daily angst of working with data that doesn’t measure up to their needs.

In projects where our firm has been brought in to help people lasso their data, we have come to expect the moment of pause, then followed by apologies and looks of mild embarrassment, just before our clients reveal the uncomfortable secret of their data’s deficiency… and the workarounds, assumptions, and wild guessing that they are forced to do in order to get work done despite imperfect information.  Most people tend to think their data is the worst we will have seen.  “Can you believe this?”  “Have you ever seen anything like this?”

The answer is “yes.”  The day is yet to come when someone sits us down and shows us a perfect data system with perfect content.  The thing is, data could always be better.  And helping define and get to better is what we do.

One unique element of the DLCD engagement has been the bright light that project members were readily willing to focus on their data issues from the get-go.  This project involved designing a new data platform for the organization, and we were able to start at a jog, because there was never any need to coax or cajole information about data challenges from the stakeholders with whom we met. Perhaps this group had already been through the data shame therapy?  Or maybe they were so ready for change that they just didn’t care anymore ;-)? Whatever the reason, it was a little unusual to so promptly and directly get to identifying the problems that needed fixing.  As the end of the project has coincided with the New Year, I’ve been thinking that, for this client, this work has a sort of New Year’s resolution aspect to it.  There is a strong sense of commitment to change, and I’m gathering that it is now being accompanied by a growing sense of optimism that the changes they seek are within reach.

While the DLCD’s data may not qualify as ‘big data’ in terms of shear volume, it streams in from a broad variety of sources and has a reach extending to all the coves and corners of Oregon. As we have learned more about the work of the agency and its partners, we have come to understand that time-informed analysis of patterns and trends in land use, and exploring relationships among complex, amalgamated data is critical for the agency to be able to fulfill its mission.  One stakeholder concluded her comment about a database by saying, “we need a better way to analyze and tell the story of land use in Oregon.”

We feel good about the data platform that the DLCD will be implementing based on the design work we have done together.  The design features a rich integration of time and place. The designs resulting from this project aren’t just garden variety databases — they’re uniquely designed with sensitivity to the client’s current and envisioned needs and goals, and they include some novel elements created in response to the specific and unique challenges that stakeholders brought to light. The design will provide a great foundation from which modern, efficient, and user-focused information management tools will arise.  These folks will have the data infrastructure to capture, curate, store, search, share, analyze, and visualize their information.  This is going to be the platform to support the DLCD’s own brand and style of storytelling.

So, this has been one of those jobs where the promise of meaningful change seems real.  Things can be different.  They will be different. These folks made their resolutions and stuck to them. There’s no shame in that!

Back in School - Geostatistics

 
Yesterday was fun.  I spent part of the morning discussing GIS and geostatistical analysis with students in Professor Paul Gronke’s Intro to Statistics class at Reed College.  Enough time goes by between the occasions when I get to spend time in a classroom that I forget how much the opportunity for learning may be enjoyed by the instructor and student alike.

I got to know Paul when we worked together on a project for the Oregon Secretary of State a couple years ago. The project involved assessing the current practices of local elections officials in Oregon and developing recommendations for how GIS might be used to improve Oregon’s Central Voter Registration System.  I always find that Paul’s expertise and his infectious enthusiasm for elections and electoral behavior provide an interesting focus for discussions about mapping.  Prior to getting together in the Stats class yesterday, we had had some pre-election interaction with Stanford Political Science Professor Jonathan Rodden, who is the director of Standford’s Spatial Social Science Lab.

Jonathan and his team of data sleuths were seeking to gather some of the few and final precinct boundaries they needed to complete a national basemap for their Election Atlas. Rodden’s Atlas is very cool — it (thankfully) takes us beyond the omnipresent and simplistic state-level election maps to reveal meaningful nuances in local election geography; it is a commendable achievement, dovetailing with and supporting the rise of data-driven political analysis, and yesterday, with Paul’s interpretive skill, it provided a great reference for our discussion about geostatistics.

But discussions ranged broadly beyond electoral data… touching on how geostatistical analysis techniques may be applied to questions of social science and equity, local government, conservation, humanitarian affairs, and business intelligence.  I was impressed that the students, in less than two hours, were able to absorb basic concepts of GIS and geostatistics and, by the end of the class, pose sophisticated and interesting questions that held my attention through the afternoon. If you want to look at your work with fresh eyes, I highly recommend spending some time in the classroom trying to teach people about what you do.

Connect even when you're not connected

We tend to concentrate on the flashy, “blingy” functions within our applications, but there are some pretty darn cool functions that tend to not rise to the top of the pack. We just delivered a solution for one such function; the ability to export a project’s shapefile from our PerformaceAtlas web application to a handheld GPS device.
Our client, Dawson Geophysical, needed a way for their agents to have access to their projects while out in the field. They generally have limited internet access, so we needed to provide them with a way to download the information that they need onto a device before heading into the field. After some research, the Dawson team settled on DeLorme GPS devices.

To back up a little, we had previously created a version of our PerformanceAtlas  for the folks at Dawson in order to track their many projects. The typical user would login to the site using their web browser, and, based upon their role, would be presented with a list of projects. From here, they could do a variety of things, from checking the current status of a project to looking at the physical boundaries of a project. It’s proved to be a very useful tool that meets almost everyone’s needs.

Almost everyone.

The folks that are not served well by the flashy, web-based application are the field agents who don’t always have a reliable internet connection. They just need information on their immediately relevant project. The solution that we delivered allows these users to pull up the projects that they are directly involved with on the PerformanceAtlas (this would be done on a web browser when they have an internet connection), then export the shapefile to their GPS device. That way, they have all the information that they need when they head out into the field where there may not be any internet connectivity. The GPS device places them in the middle of their project and aids them in finding features that are essential to them in order to get their job done.

This may not be the most “flashy” function and and it may not make it into the glossy, one-page marketing material for our PerformanceAtlas, but it’s a function without which many users would be hard-put to complete their work.