Winter 2016-2017
A Newsletter from the Federal Highway Administration (FHWA)
www.gis.fhwa.dot.gov

Open Data Portal: Databases, Policy, and a New Technology

The Iowa Department of Transportation

This article by Eric Abrams reviews Iowa Department of Transportation’s (Iowa DOT) process for developing their open data portal, beginning with the development of foundational databases, to policy changes and, ultimately, the development the current portal.

Databases as the Foundation

Iowa DOT has a long history of using enterprise databases for geospatial analysis. Since 2002, Iowa Department of Transportation managed spatial data in Oracle/Oracle Spatial databases. Datasets like road centerlines, facilities, and boundaries were created to not only serve GIS but other business interfaces. Leveraging these databases in 2006, Iowa DOT found it was easy for GIS web services to pull data directly from these transactional databases. Performance was at the mercy of each transactional database. In particular, the transactional databases were not tuned and flattened in an optimal way for usage with GIS web services. Attempts were made to stand up an enterprise GIS data warehouse. Staffing resource limitations made it impossible to manage the extract, translate, load (ETL) process needed to keep the warehouse up-to-date and the warehouse concept was abandoned.

In 2015, Iowa DOT put in motion a project to create a new data warehouse residing in Oracle Spatial called “Geolibrary.” Data in Geolibrary would come from authoritative transactional databases and then tuned appropriately. Compared to the failed attempt in 2006, Iowa DOT now has commercial off-the-shelf (COTS) technology in place to extract, translate, and load (ETL) data using Safe Software’s Feature Manipulation Engine (FME) and additional contract staffing resources in place. Since 2008, the Department leveraged Esri ArcGIS Server for Representational State Transfer (REST) services from the Oracle transactional databases. Geolibrary would leverage Esri’s ST_GEOMETRY format. After speed performance benchmarks were done comparing Oracle’s SDO_GEOMETRY and Esri ST_GEOMETRY format with ArcGIS Server, it was clear that flattening data and Esri ST_GEOMETRY was the best architecture for REST services. Transactional Oracle Spatial databases were still using Oracle’s SDO_GEOMETRY format. This kept our transactional database software neutral. Plus, when the department is ready to move toward an enterprise warehousing strategy, the Geolibrary will be well positioned for migration. Building a good foundation to store data for GIS services is a key step in making your open data portal fast and reliable.

Iowa DOT Open Data Map Portal

Policies and Culture

In 2013, Iowa DOT crafted a department-wide Geospatial Policy to formalize integration of location into business systems.

The first paragraph of the policy reads:
The Department shall incorporate spatial data into Department business practices, databases, and enterprise systems in a manner that will allow integration and information sharing among staff and customers of the Department.

Iowa DOT also wrote goals into the policy to better define the policy’s intent:

• GIS data is available for on-demand mapping; and
• GIS data not defined as confidential is available to all users.

Some goals, which touched on integration like GIS standards, are aligned with Information Technology (IT) standards and GIS web services support open GIS standards.

Policy and culture is an important step. Without support from key players and well-defined policies to provide a guiding framework, the amount of data available in open data portals for staff and the public may be limited.

Iowa DOT Open Data Map Portal

REST Services

In 2009, Iowa DOT implemented Esri ArcGIS Servers for REST services. The first attempts at REST services did not meet performance expectations. In particular, services were sluggish because transactional databases were not fully tuned to deliver these services. The GIS team relied on data owners to tweak databases to accommodate REST services. For multiple datasets, data owners did not prioritize these tweaks, which made data unavailable to ArcGIS Server. The GIS team had also historically prioritized services based on the customer’s map needs, not built generically for everyone, which created duplication of services.

In 2016, Iowa DOT had an onsite ArcGIS Server system audit done. That audit gave us a better direction for reliable, predictable REST services. The GIS team restructured server infrastructure, conducted an internal REST service audit, eliminated duplicate servers and services, and put the Geolibrary in place to feed the new REST services. On December 15, 2016, IDOT officially retired its old GIS architecture.

Without predictable, reliable and authoritative GIS services, GIS teams must continuously chase data issues and performance problems, and in the end, this results in poorer quality services for everyone.

Development of Open Data Portals

REST services fuel Iowa DOT’s open data portal. In the past, the Iowa DOT GIS team would create shapefiles for people to download or point people to the cryptic Esri REST service library. Most users had no clue what to do with the Esri REST library. It was important for us to create an environment that not only allowed downloads but links to the authoritative REST services, no matter where they resided.

Since Iowa DOT already had unlimited ArcGIS Online users, an Esri open data portal was a natural choice to bring REST services together. To facilitate this transition, Iowa DOT hired an intern, David Runnels, to create the skeleton site and develop categories. Iowa DOT borrowed ideas from other open data sites, such as Utah DOT’s site, to understand best practices. David worked with the department’s graphic designers to develop a look and feel that was consistent with our web presence. When David’s internship ended, the template was in place.

In the spring of 2016, Iowa DOT hired another intern, Rachel Haukkala. Rachel was tasked with populating the open data site. She determined which services needed to be tied into the department’s open data portal, and created minimum metadata and standardized keywords. The department also developed guidelines to keep the open data portal current. For example, when adding public REST services, a determination is made if the data has value in the open data portal. Databases, policy, culture, technology, and vision all play into a successful portal. Iowa DOT moves into the next phase of open data portal by adding data and maintaining the people, processes, and procedures that will make the open data portal successful for years to come.

Extract, Translate, and Load using Feature Manipulation Engine from Safe Software

Lessons Learned

Many in the GIS community have struggled to make data available and accessible, not only to internal customers but external customers as well. I recently met with Iowa Air National Guard on how to better exchange LiDAR data during disasters. During our meeting, it came up Iowa Air National Guard leverages Iowa DOT Open data portal. Before the portal, they had multiple places to gather transportation-related information but now can get current Iowa DOT data at one stop. Increasingly, State DOTs and MPOs are working to make GIS data more open for employees and the public. Organizations become liberated when data is freely available—imagine no meetings about data access! As an organization embraces open data, it becomes natural to everyone and new innovations often arise. People inside and outside your organization have the freedom to use data in ways never imagined. Open data portals help provide that access.

From the process of developing our open data portal, Iowa DOT shares the following lessons learned and recommendations for other agencies seeking to promote an open data culture and to build an effective data portal:

• Building a good foundation to store data for GIS services is a key step in making your open data portal fast and reliable.
• Policy and culture is an important step. Without the support of key players and clear policies, the amount of data pushed to an open data portal may be limited.
• Properly managing and formatting transactional databases is vital for running efficient GIS web services.
• Without predictable, reliable, and authoritative GIS services, State DOTs will continuously encounter data issues and performance problems.
• An open data portal is not only about adding data but also about maintaining the people, processes, and procedures that will make the open data portal successful for years to come.

Datasharing with Waze

Editorial by Cliff Curtis of MaineDOT

Major travel corridors are a must to keep moving; they are the life blood of any economy. Understanding what is happening on the roadway is integral to keeping traffic moving. Crowdsourcing live data is an emerging method to help monitor conditions in real time and enhance our understanding of what is happening. Crowdsourcing provides the opportunity for the public to take an active role in reporting what they see on the road. Historically, people would call 911 to report items, burdening 911 responders with non-emergency issues. This is gradually shifting, however, with the advent of crowdsourcing mobile applications like Waze. Now, these issues can be shared amongst a community of local travelers keeping each other appraised of roadway events and conditions.

The State DOTs of Maine, New Hampshire, and Vermont, like many other State DOTs and a number of city governments, are sharing information with Waze in exchange for access to real-time data from the application. This data sharing process provides a more comprehensive view of how the transportation system is performing.

MaineDOT Map Viewer showing incident data geospatially

How MaineDOT Uses Waze Data

MaineDOT primarily compiles and analyzes Waze data using ArcGIS Server and GeoEvent Server software from ESRI. MaineDOT is able to present this information being collected by Waze users in its internal MaineDOT Map Viewer mapping application. The Waze information is presented in two forms: Alerts and Traffic. MaineDOT has a convenient summary table that quantifies types of alerts and levels of traffic delays. This information is updated every two minutes for real time reporting. This improves situational awareness as an incident develops. Tools within the internal mapping system also give operators the ability to zoom to an event location. Operators can turn on different asset and feature layers to better understand and describe the location to internal and external customers. Being in a GIS format allows Maine to layer other information and utilize simple mapping tools. This gives the ability to identify jurisdiction, asset details, and other reference information, which provides a deeper understanding of the developing situations. The power of this information is in making better informed decisions: in particular, corrective actions can be implemented faster to keep partners and the public informed and safe.

MaineDOT also shares data with Waze. We try to add more detail on alerts being reported to help travelers and partners understand event impacts. This includes long-term closures for construction, crashes, and special events. In turn, Waze shares MaineDOT information with roadway users.

Another key element to sharing information with Waze is parts of Maine are very remote. There are few drivers, and still fewer drivers using Waze. MaineDOT can help supplement the transportation common operating picture by sharing event information from MaineDOT personnel and emergency responders. This information helps keep Waze customers and others informed of critical events. The end goal is to have a common understanding of traveler impacts, so individuals can make appropriate decisions.

Data Validation

With any new data sharing tool, State DOTs need to establish business rules on how to use the data effectively. For example, defining when a reported Waze Accident is real. Based on experience, when more crowdsourcing reports appear, then a call to Public Safety is made for more information. Not every reported “Waze Accident” is an accident. Sometimes, it is just a car parked on the side of the road. So, a few steps are added before actions are implemented. Even taking a little bit of time to increase confidence in a Waze report, we may still be ahead of anyone calling 911. Some of the other business rules being updated consist of emergency response plans for traffic incident management, maintenance activities, transportation system performance, etc. With this flow of information, it gives the Department an opportunity to be more responsive.

The primary way to validate data is to rely on crowdsourcing itself. Often, when an accident or other incident alert from Waze comes in, other Waze users begin reporting traffic jams that result from the incident. This provides more certainty that an incident is happening. Other times delays are shown from Waze Traffic, which provides information on delay duration and relative backup length. Each of these provides confidence to Traffic Management Center (TMC) operators to contact MaineDOT crews and Public Safety, to see if they have any clarifying event details.

MaineDOT Map Viewer showing Waze traffic

Waze data also includes a couple attributes associated with the Waze Alert: a user rating and a report rating. Each of these attributes provides an indication of event validity. All of these tools, in conjunction with roadside Intelligent Transportation Systems, help indicate what is happening, where it is happening, and how traffic is being impacted. Either way, quicker awareness generates faster event response and reporting. MaineDOT can begin implementing the appropriate response for traffic incident management and make it scalable based on Waze’s real-time reporting.

Final Thoughts

MaineDOT’s partnership with Waze has been beneficial. This is a significant improvement in managing traffic. Whether monitoring the morning commute, watching a Friday afternoon influx of tourist travelers, or observing the impacts of a winter storm, the data from Waze has provided more highway situational awareness. This is leading to faster incident responses from MaineDOT crews. Overall, we are now aware of four times the number of incidents than before. By knowing about these incidents real-time, MaineDOT with other responders can reach the scene faster, work closely together to resolve the incident, and get the traffic back to normal speeds. In the end, this is an important partnership. It keeps travelers safe, saving time and fuel. It also allows MaineDOT to do their part in keeping Maine moving.

Other News

Upcoming Webinar on Waze Connected Citizens Program

Kali Fogel of LA Metro presented on the Regional Integration of Intelligent Transportation Systems’ use of Waze Connected Citizens data exchange program. This workshop explained the communications, transportation, and institutional architecture existing in Southern California to develop and implement the potential uses of Waze.

A summary of this webinar is forthcoming. When ready, it will be available at https://www.gis.fhwa.dot.gov/webinars.aspx

The 2017 AASHTO GIS in Transportation Symposium

The annual GIS in Transportation Symposium was held April 10-13 in Phoenix, AZ. Mark Sarmiento of FHWA and Michael Green of the Volpe Center presented on the 2016 CMM and Organizational Assessment Peer Exchange and presented their findings after six State DOTs completed the URISA CMM and reported their experience.

Upcoming Capability Maturity Model Peer Exchange

Following the 2016 CMM Peer Exchange and the 2017 CMM Workshop, attendees and peers expressed interest in followup and FHWA is responding by organizing a CMM Peer Exchange to include additional State DOTs that may not have participated in the previous event. If you’re interested in participating, please reach out to Mark Sarmiento at FHWA. Details forthcoming.

Advisory Panel

FHWA and the U.S. DOT Volpe Center are soliciting interest from GIS practitioners in serving on an advisory panel for the FHWA GIS in Transportation program. This panel would serve to 1) guide the identification of research priorities, 2) identify communications needs for stakeholders, and 3) provide recommendations to ensure that the program’s activities and products align to stakeholder needs.

If you are interested in participating or have further questions, please contact Mark Sarmiento, FHWA Office of Planning.

For more information about this newsletter, please contact Mark Sarmiento, FHWA Office of Planning, at Mark.Sarmiento@dot.gov.