Return to Reports page
PDF version (512kB)

Geospatial Information System Capability Maturity Models
Case Studies of Select Transportation Agencies

June 2017

logo of the Federal Highway Administration of the U.S. Department of Transportation

Prepared for:
Office of Planning
Federal Highway Administration
U.S. Department of Transportation

Notice

This document is disseminated under the sponsorship of the Department of Transportation in the interest of information exchange. The United States Government assumes no liability for the contents or use thereof.

The United States Government does not endorse products or manufacturers. Trade or manufacturers’ names appear herein solely because they are considered essential to the objective of this report.

REPORT DOCUMENTATION PAGE Form Approved
OMB No.
Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188), Washington, DC 20503.
1. AGENCY USE ONLY (Leave blank) 2. REPORT DATE
June 2017		 3. REPORT TYPE AND DATES COVERED
Final
4. TITLE AND SUBTITLE
Geospatial Information System Capability Maturity Models		 5a. FUNDING NUMBERS
6. AUTHOR(S)
Michael Green, Anthony Lucivero, Andrew Reovan, Drew Quinton		 5b. CONTRACT NUMBER
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)
U.S. Department of Transportation
John A Volpe National Transportation Systems Center
55 Broadway
Cambridge, MA 02142-1093 		8. PERFORMING ORGANIZATION REPORT NUMBER
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)
U.S. Department of Transportation
Federal Highway Administration
1200 New Jersey Avenue, SE
Washington, DC 20590 		10. SPONSORING/MONITORING AGENCY REPORT NUMBER
11. SUPPLEMENTARY NOTES
12a. DISTRIBUTION/AVAILABILITY STATEMENT
This document is available to the public through the National Technical Information Service, Springfield, VA 22161.		 12b. DISTRIBUTION CODE
13. ABSTRACT (Maximum 200 words)

To explore how State DOTs evaluate geospatial tool applications and services within their own agencies, particularly their experiences using capability maturity models (CMMs) such as the Urban and Regional Information Systems Association (URISA) geospatial information system (GIS) CMM, FHWA and the U.S. Department of Transportation (DOT) Volpe National Transportation Systems Center interviewed four State DOTs and developed a series of case studies focusing on their experiences:

  • Iowa Department of Transportation (IDOT)
  • Ohio Department of Transportation (Ohio DOT)
  • Tennessee Department of Transportation (TDOT)
  • Oregon Department of Transportation (Oregon DOT)
The report supports GIS practitioners and decision-makers by identifying examples of noteworthy practices, considering the advantages and disadvantages of different organizational assessment and CMM approaches, and determining the value that CMMs bring to transportation agencies.
14. SUBJECT TERMS
Geographic information systems (GIS), capability maturity model (CMM), organizational assessment		 15. NUMBER OF PAGES
33
16. PRICE CODE
17. SECURITY CLASSIFICATION OF REPORT
Unclassified		 18. SECURITY CLASSIFICATION OF THIS PAGE
Unclassified		 19. SECURITY CLASSIFICATION OF ABSTRACT
Unclassified		 20. LIMITATION OF ABSTRACT
Unlimited

NSN 7540-01-280-5500

Standard Form 298 (Rev. 2-89)
Prescribed by ANSI Std. 239-18
298-102


Contents

List of Figures

List of Tables

Executive Summary

Since its wide-scale adoption by transportation agencies, geospatial information systems (GIS) technology has provided a tremendous value to each agency and has been integrated into everyday organizational decision-making and analysis. In many cases, agencies are no longer asking themselves if they should or should not use GIS, but how well have they adopted this technology and how can they improve upon it. One way in which transportation agencies approach these questions is through a self-evaluation, otherwise known as an organizational assessment. In particular, some agencies have adopted more specific assessments known as capability maturity models (CMMs), which asses the ability of an agency to effectively and efficiently carry-out specific GIS services, rating each service using a numerical scale. The Federal Highway Administration (FHWA), in an effort to assist transportation agencies with self-assessments, has decided to identify and explore potential solutions for a uniform and effective assessment process.

To explore how State Departments of Transportation (DOTs) are assessing organizational GIS decision-making capabilities, particularly through the use of capability maturity models, FHWA and the U.S. DOT John A. Volpe National Transportation Systems Center (Volpe) interviewed the following four State DOTs and developed a series of case studies focusing on their experiences:

This report supports GIS practitioners and decision-makers by identifying examples of noteworthy practices, considering the benefits and limitations of implementing a capability maturity model approach, and determining how to identify and address factors that alter the effectiveness of a GIS division in applying geospatial tools and services. Highlights of the findings presented here include:

Back to Table of Contents
Back to Top

1. Introduction

1.1 Purpose and Methodology

The Federal Highway Administration (FHWA) has promoted Geospatial Information Systems (GIS) as a means to more effectively manage and improve transportation systems. One of the ways that FHWA has done this is through its GIS in Transportation program,1 which identifies timely and critical GIS issues and topics in transportation and connects transportation agencies with available resources and best practices. While several State Departments of Transportation (DOTs) and Metropolitan Planning Organizations (MPOs) have already implemented or are in the process of implementing GIS and geospatial tools to address complex issues, others are in the early stages of using these tools to address the same types of needs and challenges. More so, the level of success and efficiency at which these tools are applied can vary greatly from one agency to the next. While it has been common for agencies to identify or acknowledge GIS-related challenges, implementing an efficient and repeatable process for doing so has not. To address this issue, some transportation agencies have implemented what is known as an organizational assessment—a process through which an agency evaluates its strengths and weaknesses in a given area of service. An organizational assessment will generally conclude with a series of recommendations that can address existing or newly identified weaknesses.

The topic of organizational assessments is not a new area of research but its application in GIS over the last 10 to 15 years has been sparse and mostly unknown within the GIS community. The idea of assessing organizational structures and services originated from another recent topic of return on investments (ROI). In the field of GIS, return on investment studies are undertaken as a means to estimate the benefits associated with adopting a GIS application or service by quantifying outcomes such as reduced production times, lower data collection costs, and expedited project delivery timelines. An ROI study measures a change before and after the implementation of a new technology or process, which is what has motivated organizational assessments. Through an organizational assessment, transportation agencies can provide a baseline for the current state of services provided by an agency. After implementing a new application or altering a business process, a follow-up organizational assessment can demonstrate how the GIS division as a whole has changed over time, particularly in areas most closely related to the new application or business process. Assessments can also provide a snapshot of the effectiveness of current GIS practices, which do not require other baseline or follow-up evaluations. For example, an assessment can identify how data is stored across a DOT and if there are duplicative efforts for storing and processing that data.

The FHWA GIS in Transportation Program recognized the potential benefit that organizational assessments can have for State agencies' GIS divisions and, in 2016, held a peer exchange with GIS staff from seven State DOTs focused on organizational assessments. The meeting included State DOTs with varied organizational assessment experience including those whom had at least one assessment and those with no experience. The peer exchange was meant to start a discussion among interested parties where States that had assessment experience could share their lessons learned and best practices with the rest of the group. One of the outcomes of the peer exchange was that, in addition to learning how an organizational assessment can lead to cost savings and other benefits or what are common requirements when carrying out an assessment, there is no standard and efficient approach to carrying out an assessment. As a result, each State demonstrated a demand for a unified approach, which was ultimately led by the Urban and Regional Information Systems Association (URISA). URISA implements what is known as a GIS Capability Maturity Model or CMM. Generally applied to technological applications, CMMs, which are further explained in Section 1.2, are models that evaluate an organization's overall level of maturity by how well it can carry out tasks related to a certain topic. According to URISA, its GIS CMM “provides a theoretical model of a capable and mature enterprise GIS operation within a designated organization.”2

Following the 2016 peer exchange, a group of participating State DOTs agreed to test URISA's GIS CMM. In exchange, URISA was able to provide access to the online tool for testing, allowing each State DOT to create its own profile and visualize the results of the assessment. In order to best capture each State's experience using the tool, FHWA with the help of the John A. Volpe National Transportation Systems Center, interviewed four States that were able to complete a CMM and analyze its results. The questions, which are available in Appendix B: Interview Guide, include topics on the background of each State, experience with organizational assessments, the objective behind the CMM, the experience using the tool, and the results of the tool. Overall, each State's experience produced a unique case study on the use of CMMs for GIS divisions within State DOTs.

The case studies gave participating agencies the opportunity to:

FHWA and Volpe selected agencies for case studies based on their interest in self-assessments, previous experience with organizational assessments, and ability to complete URISA's GIS CMM. The participating agencies include the Iowa Department of Transportation (IDOT), Ohio Department of Transportation (Ohio DOT), Tennessee Department of Transportation (TDOT), and Oregon Department of Transportation (Oregon DOT). Appendix A: List of Case Study Participants provides details on agency participants. An interview guide, included as Appendix B: Interview Guide, provides a framework for the two telephone discussions held with each agency, held for 30-60 minutes each in January through May of 2017. Case studies were drafted based on participant responses during the two discussions. Findings from the case studies are included in the Observations section of this report.

1.2 Capability Maturity Models

The concept of CMMs originated in the late 1980s with the Software Engineering Institute. The CMM was designed to evaluate how well contractors could complete software design and development projects. Since then, CMMs have become more widespread and applied to the fields of engineering, project management, risk management, and information technology (IT) services.3 It was not until 2009 that the first GIS-specific capability maturity model was developed by the National States Geographic Information Council (NSGIC) and then later adopted as an initiative by URISA. According to one of its creators, Greg Babinski, a “capability maturity model assesses an organization's ability to accomplish defined tasks,” and “assess an organization's maturity level based on how it executes individual processes.”4

1.2.1 Defining Maturity

A mature agency can be defined as one with well-developed enabling technology and resources that are used effectively and efficiently. The URISA GIS CMM more specifically defines maturity across two broad areas of GIS operational development: enabling capability and execution ability. According to URISA, “enabling capability includes technology components, data, professional GIS staff, an appropriate organizational structure, and other resources and infrastructure.” When referring to execution ability it “is the ability of the staff to maximize the use of the available capability, relative to a normative ideal.”5 To assess an agency's execution ability, the URISA GIS CMM uses the following five-level scale:

Each level indicates the progression an organization experiences when evaluated with regards to capacity and efficiency in completing GIS-specific tasks and goals. Goals can include the use of state-of-the-art GIS technologies, high quality data, and implementing community accepted best practices for business processes.

A model such as the URISA CMM assumes there is an ideal state of managing GIS operations and performance goals. How an agency defines an ideal goal, however, can vary depending on the adopting organization. For example, an agency set a goal of piloting a new GIS staff position on a temporary basis. On the other hand, another agency might set a goal to hire a staff member for a full-time position. Additionally, agencies can leverage other models that use a more discrete definition of an ideal level of maturity. Discrete approaches can include comparing tasks to an already established ‘best’ level of capability and practices, or an estimated ‘average’ level of capability and practices as determined by an agency.

URISA developed the rating system for measuring the enabling capability of a GIS division using an existing State GIS Maturity Assessment scale, which was created by the States of Georgia and Texas. The rating system, which is used to eventually determine a level of maturity, is defined as follows:

Similar to the five-level execution ability assessment scale, each step on the enabling capability scale represents a progression towards higher levels of maturity where 1.00 is a fully implemented and mature process. Once complete, the URISA GIS CMM summarizes each level of maturity across multiple categories in a spider diagram, as seen in Figure 1 below.

Figure 1. Example URISA GIS CMM Spider Diagram

spider diagram example of URISA GIS CMM results

Source: FHWA 2017 GIS-T Workshop Presentation

After determining that geospatial tools and applications can provide measurable benefits for an agency, FHWA believes it is important to understand how effectively an agency is using those tools and applications. By implementing a CMM such as the URISA GIS CMM, State DOTs can take the necessary steps to improve upon already existing services within their GIS divisions. Because there is no standard approach for how State's should go about self-evaluations, FHWA has taken the role of convening potential stakeholders and decision-makers to identify potential approaches. The Agency believes there is a benefit in facilitating the process for organizational assessments while identifying common goals and challenges. In the end, State DOTs can identify and choose to follow a uniform assessment framework allowing them to share results and compare across similar metrics.

Back to Table of Contents
Back to Top

2. Observations

Past approaches to organizational assessments and capability maturity models varied widely for each of the four peer agencies interviewed. In some cases such as with Oregon DOT, the URISA Model is the first formal assessment the agency has been able to undertake in the recent past while in other cases, such as Iowa DOT, there is a regular history of assessments and CMMs to which the URISA CMM provides a new level of input.

Each case study participant demonstrates a strong interest in improving the respective agency's efficiency in delivering GIS services. However, each agency differs in how much time, staff, and other resources are available to take on a new task. In addition to resources, demonstrating the benefits and costs of a CMM can be challenging if there is limited support from senior management. Solving these challenges and understanding the role that CMMs can play in a transportation agency can require unique and long term approaches that are beyond the scope of this case study report. While some participants indicated plans to follow up with the URISA GIS CMM, others recognize that adopting a regular assessment approach might still be further down the line. Table 1 below, provides an overview and summary points about each of the case study examples.

Table 1. Summary of Case Study Agencies’ CMM Efforts

Agency Past Formal GIS Evaluations Centralized or Decentralized GIS Division Experience with CMMs Goals and Lessons Learned from CMMs
IDOT 2005 (contractor), 2015 (CMM), 2016 (CMM) Decentralized Extensive experience in CMMs
  • Wants a more effective method for collaborating across agency to reduce redundancies.
  • Recommends investing time and other resources towards educating individuals and departments on CMMs.
Ohio DOT Strategic Plan (2002), Enterprise Architecture Plan (2014), Data Governance Evaluation (2017) Mix - Central Office GIS team across two Divisions; Planning and Information Technology Extensive experience with assessments but limited capability maturity model experience
  • Wants a better understanding of the capability maturity process and how it can be an alternative to more formal evaluations.
  • While completing the CMM for the first time did not require more than 7 hours of work, Ohio DOT recommends take a more detailed approach that can take up to 20 hours for best results.
  • Understanding objectives beforehand can help bring the right people and players to the table, to ensure the process has an enduring impact.
TDOT None Decentralized Little to no experience
  • CMMs can further promote the need for training and more strategic funding for other identified needs.
  • TDOT recommends that multiple individuals review a CMM assessment, not just one person, and that users should gather all relevant data before beginning the CMM, particularly budget data.
  • CMM should also include a “Recommended Data Collection” guide to help users understand what data they should have on hand before using the CMM.
Oregon DOT None Centralized Little to no experience
  • Aims to use a CMM as a yard-stick for whether or not the DOT should do an in-depth contractor study, which might result in strategic and implementation plans.
  • The CMM tool was also able to demonstrate in what small areas there were varying views of maturity, which started a discussion at the State DOT.
  • The CMM tool motivated engagement between GIS stakeholders, Oregon DOT, and supervisors.

2.1 Motivations for Capability Maturity Models

Case study participants decided to explore the topic of CMMs, URISA's model in particular, not only because of the potential benefits associated with evaluating an agency from within, but also to gain insight into how other similarly organized agencies self-evaluate. CMMs allow agencies with no experience with formal assessments to start an internal discussion on implementing such a model while more experienced agencies can compare the process and results from models such as URISA's to their current standing approaches. By using the same method, multiple agencies can begin to share their results and experiences. Within that overarching goal, the agencies were motivated to pursue CMMs for one or more of these reasons:

2.2 Benefits

Transportation agencies reported a number of benefits related to using CMMs to evaluate established GIS business processes. Agencies highlighted the following benefits in the discussions:

2.3 Challenges

Agencies noted several challenges associated with starting and finishing a CMM, as well as leveraging the results to improve business processes:

2.4 Lessons Learned

The case study agencies offered the following lessons learned based on their experiences:

Back to Table of Contents
Back to Top

3. Case Studies

This section presents in-depth case studies on the transportation agencies that participated in the report interviews. Each case study starts with a background section, which provides a more in-depth review of these efforts, specifically: 1) how each agency has implemented organizational assessments in the past; and 2) motivations and goals associated with undertaking a CMM. Following the background section, each case study concludes with a summary of the benefits and challenges related to CMMs, strengths and weaknesses related to the URISA GIS CMM, a review of lessons learned and recommendations, and next steps (as applicable).

3.1 Iowa Department of Transportation (IDOT)

Background

The Iowa Department of Transportation has employed organizational assessments since 2006. Before completing the URISA GIS CMM, IDOT completed an adapted version of Slim Geographic6 Information Management called Enterprise Maturity Model.

IDOT has converted some of the older maturity models into a more appropriate format for internal use. Between 2006 and 2010, the State DOT completed different types of informal CMM assessments to assess where the organization's strengths and weaknesses lie with respect to the use of geospatial tools. Part of the success that IDOT has encountered with the use of CMMs is due to support from executive leadership. With this support, IDOT started to use more formal CMM assessments at the institutional level.

While IDOT has a core staff in the Technology Group consisting of two full-time employees, two contractors, and two interns or temporary staff, it also has 50 to 60 other employees working with geospatial data in various policy offices and divisions throughout the department. Currently, IDOT leverages geospatial tools in a number of ways including:

One way in which IDOT aims to use geospatial tools, but currently does not, is aerial photography and Light Detection and Ranging (LiDAR) to monitor changes in conditions over time. The primary hurdles to overcome in implementing new GIS technologies are the availability of both staff resources and time.

Benefits of CMMs

As IDOT engages in projects such as a single enterprise system migration and advancing data governance practices, GIS managing staff have turned to CMMs to better inform and promote the value of these projects. IDOT believes that one of the major benefits of completing CMMs is to discover institutional deficiencies in corporate culture, philosophy, and strategic orientation. For example, the results of a recent CMM assessment based on the Slim GIM model assisted IDOT to identify where the organization needed to grow by pointing out specific initiatives that needed support in order to accomplish their stated goals. In this case, there was a need to renew an existing GIS licensing agreement. Using the CMM's results, IDOT was able to obtain the support of upper management to renew the existing license.

Challenges of CMMs

The primary challenge that IDOT faces is incorporating institutional changes to structure the organization in a more effective way. The State DOT currently experiences issues with data and service duplication efforts, which are due to lack of communication and misalignment of goals across staff and State DOT departments. While a CMM can assist the GIS division in identifying communications challenges, applying solutions across multiple groups and divisions can be difficult to justify for everyone.

URISA GIS CMM Tool Strengths

From IDOT's experience, the URISA GIS CMM is more thorough than other CMMs in how it scores varying components of a GIS division. Using the URISA CMM tool, a user can visualize the model outcomes with spider diagrams. Each spider diagram, such as the one seen in Figure 1, displays a level of maturity for each predetermined component, side-by-side. Using this approach, an agency can identify stark differences between high and low levels of maturity, which in turn allow it to strategically allocate funds towards needed areas. IDOT believes the spider charts, through which the results of the assessment are communicated, are a very clear way to display the results and indeed the best approach.

The URISA CMM also helps identify if an agency is effectively managing its corporate culture and staffing requirements. For example, the assessment asks how one's agency is organized, at what level of management are GIS initiatives organized, and how effective leadership is. The questions make the user think about what staffing positions the agency is currently lacking and, through the results of the assessment, indicate what needs to be improved.

URISA GIS CMM Tool Weaknesses

One of the weaknesses of the URISA CMM tool identified by IDOT is that it is not currently optimized for State DOTs and is indirectly biased towards municipality or county frameworks. For example, many of the questions refer to elements such as cadastral data layers, utilities, and customer records, which are primarily managed at the county and municipality level. One question in particular asks if the agency manages streets and addresses, which is not relevant to departments of transportation. Similarly, there are a number of questions absent that would apply to State DOTs such as questions pertaining to centerline data, data collection methods, and LiDAR. IDOT argues the tool could be much stronger if it included questions pertaining to linear referencing, assessments, and more of an emphasis on data management. Many State DOTs have a large variety of information pertaining to the assets they manage, and questions on this topic were conspicuously absent.

Lessons Learned and Recommendations

Due to time limitations, The URISA CMM was completed by one person at IDOT. In the future, given more time to complete the assessment, IDOT recommends investing time and other resources towards educating other individuals and departments on the nature of the URISA CMM and CMMs in general. By doing so, more staff can be included in the assessment, which can result in increased input and quality of input across the agency and in turn produce more accurate and robust results.

For the results of the CMM assessment to have an effective change, IDOT would have to significantly change how the agency is organized and how they hire and manage staffing. For example, the Cartography Office and the Technology Office both use geospatial software, but because the products they produce are not necessarily coordinated, the result is less efficient than it otherwise could be.

The value of the URISA CMM tool would be further increased if a number of State DOTs could all take the same assessment and cross-reference results with each other. This would provide the advantage of making clear which agencies excel at which tasks, and which agencies would benefit from some improvement of other tasks. Furthermore, having an apples-to-apples comparison across agencies would help to facilitate peer exchanges, allowing agencies experiencing similar difficulties to work together to overcome them and leveraging the lessons learned from agencies who have encountered and overcome similar difficulties.

Next Steps

IDOT expects to continue to use and evaluate the URISA CMM. The State DOT expects the level of effort and time required to complete the CMM will be significantly lower given some of the first steps in completing the first CMM were identifying data sources and understanding definitions.

Back to Table of Contents
Back to Top

3.2 Ohio Department of Transportation (ODOT)

Background

Ohio DOT has a distributed organizational structure, with most geospatial services managed by the Office of Technical Services in collaboration with GIS, IT, and planning personnel from the 12 district offices. There are ten full time staff in the Office of Technical Services, two staff in the IT group who are GIS service administrators, and about seven full-time equivalents (FTEs) doing geospatial work across the district offices (in most districts, specialists are only partially dedicated to geospatial efforts). In addition, Ohio DOT has four on-site contractors supporting GIS-based projects.

Ohio DOT uses geospatial tools for several purposes, most notably:

Ohio DOT's Office of Technical Services has a history of conducting organizational assessments and of using the information gathered through that process to develop strategic plans. The first such GIS Strategic Plan was developed in 2002 and defined the vision and goals for the GIS program, and many of those same vision and goal statements are still used in internal communications and marketing. Ultimately, the vision and goals were used to define the Road Inventory Specialist positions at the district offices, allowing Ohio DOT to grow GIS and geospatial expertise throughout the districts.

In addition, Ohio DOT completed an enterprise architecture plan in January 2014, which had some specific GIS recommendations (amongst other enterprise data and technology management recommendations). The plan is currently in the implementation stages, with few outcomes to report at the time of this writing.

Currently, the Ohio Office of Technical Services is undertaking an organizational outreach effort and survey to understand data governance needs. Ohio DOT anticipates developing a new data governance policy, based on these findings, in the near future. The assessment began with an initial online survey for data business owners responsible for data programs. The Office of Technical Services then established a data governance steering committee, which is reviewing the survey results both for accuracy and for unexpected findings. Based on the findings, the steering committee will develop an initial data governance strategy with the goal of facilitating data sharing between different data owners and silos. Currently, the biggest challenge is time: while the survey was conducted recently (in Fall 2016), data practices are evolving so rapidly at Ohio DOT that survey responses may not be entirely accurate a few months later.

Ultimately the goal of this assessment is to get recommendations out to data owners and to management. One initial and significant recommendation is that effective data governance can help reduce the duplication of effort in capital budgeting. Maintenance of different data sets demands staff time and effort. When modifying data in a single data set, it can lead to inconsistencies with other data sets due to changes not being published authoritatively and carrying through each related dataset. Ohio DOT is managing a $2 billion capital project budget, so accuracy on all of the data for capital investments needs must be highly accurate. With improved data governance and accurate data sets, Ohio DOT would be better able to allocate that budget and plan for future investments.

Benefits of CMMs

According to Ohio DOT, CMMs can improve capital investment efforts and reduce duplication of effort. Through the data governance assessment, Ohio DOT anticipates that clear standards for data management will help reduce the lag time in processing or modifying data that was in different formats. This will improve confidence in the data and, in turn, help justify specific capital investments and prioritization.

CMMs, the State DOT claims, can also inform new job types and position descriptions. Relatedly, the geospatial assessments done in the past have informed how the Office of Technical Services and Ohio DOT's human resources team craft job descriptions. In particular, the Road Inventory Specialist position was created to increase geospatial capabilities at the district office level. In addition, Ohio DOT hopes to encourage the consideration of geospatial experience in hiring of other staff positions (e.g. planners), using the capabilities described in assessments like the URISA CMM.

Ohio DOT looks to regular assessments, such as CMMs, to evaluate its technical capabilities and its geospatial data management to help maintain a commitment to continuous organizational improvement. Ohio DOT mentioned that it regularly looks to third-party assessments as a way to identify gaps in capabilities and ways to strategically invest in improvements to how the Office of Technical Services works with its stakeholders. A CMM can also serve as a less resource-intensive alternative to this process.

Challenges of CMMs

As Ohio DOT found with its data governance assessment, quick changes in data practices are creating difficulties in getting an accurate snapshot of the data governance picture within the agency. As practices change, the data governance steering committee has to re-evaluate preferences and answers that may have changed within the six months since an initial survey was conducted. This process in turn requires more investment in time from GIS staff.

One of the biggest organizational issues at Ohio DOT is that a number of business units struggle to see how their data can contribute to the overall picture. Essentially, there are a number of “silos of excellence” with narrow mandates—groups that are excelling at their primary mandate but unaware of how their work interacts with other organizational units. Obtaining this data can be crucial to completing a CMM but may be too costly in resources to do.

URISA GIS CMM Strengths

Ohio DOT estimated that for an individual or small group the assessment should take a total of only 7 hours on average: 4 hours for the main questions, and an additional 2-3 hours to gather data and complete the metrics section. More time may be necessary for assessments involving a larger number of perspectives. This approach limits the level of detailed inputs due to the time required to collect all the data.

Ohio DOT appreciated the value of the high-level report generated by the URISA CMM; however, the organization found it limited for use in interpreting the results. Ohio DOT would recommend allowing for several reports that interpret the data in different ways, to include recommendations for each focus area or to allow users to dive into raw data. This would help champions and GIS teams tailor their recommendations to other parts of the organization and to leadership.

URISA GIS CMM Weaknesses

If Ohio DOT was to fully complete the assessment, the agency would explore collaborating with additional team members and managers throughout the State DOT. Completing the assessment in this manner would take an estimated total of 14-20 hours to conduct outreach and collect answers from data owners and other staff members throughout the State DOT. This assumes questions could be segmented to go to the relevant department or person. Currently, that segmentation would have to be completed manually. After an initial baseline, it is expected that future iterations would take less time to complete.

Ohio DOT felt that the CMM from URISA was limited in capturing details about the perspective of geospatial/GIS data customers in the organization. For a formal assessment, Ohio DOT would seek to include survey responses and other feedback from agency members (such as interviews with managers and executives). As mentioned in the interview with Ohio DOT, “You might have one idea of how well it's going, but customers might have a completely different picture.”7

Finally, the URISA model, which was developed for local agencies (i.e. URISA members), was largely adaptable for State DOTs, but some idiosyncrasies had to be overcome. In particular, at the State DOT level, service delivery to different business units or by different initiatives can vary widely: e.g. one might be doing really well while another is struggling due to factors outside of one group's control. So if competencies could be answered distinctly by initiative, project, or business unit, the analysis might help to identify specific pain points. The tool could then aggregate to the organizational level to give an average score across all initiatives.

Lessons Learned and Recommendations

From Ohio DOT's perspective, it is important and helpful to know your goals and desired outcomes before beginning the assessment process. Sometimes when doing an assessment, a user might find unexpected topics or questions. If there is no clear idea of what an agency hopes to get out of the assessment, relevant factors for questionnaire answers might be missed. Ultimately, understanding objectives beforehand can help bring the right people and players to the table, to ensure the process has an enduring impact.

Ohio DOT also considers exploring a trial run before engaging stakeholders, other organizational units, and customers. This depends heavily on how familiar an organization is with the concepts and theories of what the model is trying to do. If people are educated about maturity models and have done them in the past, an organization can dive in. Otherwise a test run might be helpful to get context before going in-depth into the process. Ohio DOT first reviewed the assessment in the Office of Technical Services, which allowed the team to go back and modify questions as they gained additional information.

Finally, Ohio DOT recommends establishing a core group of GIS CMM stakeholders that can assist in the development and promotion of State DOT GIS CMM. Establishing a standard or model that State DOTs can use can be really helpful for comparison because State DOTs can then help communicate where they are struggling and where they are succeeding as a community with their legislators and with Federal officials. Once the core group of stakeholders in the GIS CMM is assembled, that group would then be responsible for updating the URISA CMM and ensuring that it stays relevant through engagement with other peers.

Next Steps

Internally, Ohio DOT is focused on continuing to review the results of its data governance assessment and plans with the steering committee, to draft official policies about data management, and to get recommendations for a Chief Data Officer. The Office of Technical Services is looking at a parallel change management assessment process with the goals of revisiting how IT is organized and increasing skillsets in specific areas. For GIS specifically, if Ohio DOT can geospatially enable most data sets, it can address a number of issues to enable greater efficiencies in workflows.

Ohio DOT will continue to work with many of the other peer participants in this case study report, as well as with URISA, to refine and adapt a CMM model for State DOT users. The State DOT does not, however, have immediate plans to leverage the results of its initial assessment.

Back to Table of Contents
Back to Top

3.3 Tennessee Department of Transportation (TDOT)

Background

The Tennessee Department of Transportation has decentralized organizational structure when it comes to GIS and therefore no division within the State DOT dedicated to GIS activities. TDOT does have full-time employees (FTEs) that work primarily with GIS. In the Long-Range Planning Division, TDOT has 19 full time GIS staff members, and in the IT Department the State DOT has nine GIS staff members. While there are contract or part-time workers that work with TDOT, their positions are not housed within the agency.

TDOT uses geospatial tools for several purposes, most notably:

While TDOT has worked in each of these areas extensively, it believes there still exists a big gulf between engineering data and spatial data that can limit the potential of the work done. There exists a vast amount of engineering data that is not inherently or easily transferrable into a spatial format, thus limiting the applications of GIS with this data. As a way to address this issue and improve institutional capabilities, TDOT is promoting increased trainings on GIS tools to reduce the existing gulf and expand their work across the agency.

Using the URISA GIS CMM, TDOT believes it can further promote the need for training and more strategic funding for other identified needs. The URISA CMM is the first organizational assessment that TDOT has undertaken. The State DOT would like to see a common, State DOT-specific CMM to compare themselves to other DOTs as well, allowing it to easily determine effective practices.

Benefits of CMMs

Having no prior experience with CMMs, TDOT believes CMMs can provide needed support for both identifying programmatic needs across the agency and using the results of the model to demonstrate those needs to the agency's leadership. Additionally, CMMs can produce regular maturity ratings that can be traced over time. As an agency continues to use a CMM, TDOT argues it would also be expected that the cost of using the tool in terms of time and other resources will also drop making the tool easier to use and overall more accessible.

Challenges of CMMs

One of the challenges of CMMs is that the results from these models might not be revelatory or insightful. In some cases, like TDOT, knowing and understanding current limitations is the simple part while knowing how to address them is the challenge. To best use a CMM, an agency has to be able to demonstrate how the modeling tool provides a service or produces information that the agency doesn't already have. While a CMM can provide a clear and concise numerical scale rating that identifies how efficient an agency delivers a certain type of service related to GIS, if there is no buy-in from executive leadership on the importance or legitimacy of the tool, then the results will not have any significant impacts on GIS operations.

URISA GIS CMM Tool Strengths

The documentation of results was particularly useful for TDOT to see where it should focus efforts in funding and programming, how to allocate resources more efficiently, and how to self-identify GIS categories that are low on the maturity scale. TDOT also found the tool interface simple and understandable in terms of the required inputs and overall flow of the questions. One of the strengths of the tool is how it allows users to see how blank answers will affect the assessment results, which gives an indication of the importance of the question. There is also a benefit to being able to move to and from the dashboard to see what has been completed and what is missing. A potential use of this feature is allowing multiple individuals to review the assessment (instead of one), which can be especially useful for larger agencies such as State DOTs.

URISA GIS CMM Tool Weaknesses

TDOT found a number of weaknesses within the CMM tool. The budgeting questions in the CMM were difficult for TDOT to answer, as they do not have a specific GIS budget. Instead, TDOT's GIS budget is contained within the IT Department budget. It is estimated that it would take TDOT several months of prep time to tease out all the relevant information for the purpose of GIS budgeting and then sort through it to determine GIS-specific costs. Currently, there is no way to address this challenge in the URISA CMM. Additionally, TDOT does not believe the CMM addresses State DOT-specific needs as well as it could. Currently, there is only one question about transportation data layers in the CMM tool and no questions addressing Federal requirements such as the All Road Network of Linear Referenced Data (ARNOLD) and HPMS.

Lessons Learned and Recommendations

TDOT predicts that once a CMM is completed for the first time, subsequent future uses will be easier and faster to complete. The agency recommends that multiple individuals review the assessment, not just one person, and that users should gather all relevant data before beginning the CMM, particularly budget data. The CMM should also include a “Recommended Data Collection” guide to help users understand what data they should have on hand before using the CMM.

The weakness of the survey for State DOT GIS divisions shows how it needs to be adapted for applicability at the State level. TDOT does not have a central GIS division or a dedicated budget, so they received a “0” for GIS Maturity in the CMM under budget-related categories. State DOTs tend to be more complicated organizations than local governments, and some, like TDOT, have unconventional GIS staffing structures and budgets. The creation of a State DOT-specific CMM tool needs to take multiple organizational structures into consideration when writing the questions. There should also be an entire section of transportation data questions for State DOTs, including HPMS questions, which are critical to State DOT GIS work.

Next Steps

It is unclear if TDOT leadership will support the investment for a CMM to be performed on a regular basis. This is due to a new round of funding from an increased State gas tax, which means TDOT will receive a large quantity of new projects to work on, thus limiting time and resources available for another CMM to be performed. The window to assess TDOT through these kinds of measures passed by about a year. TDOT would like to perform an “apples-to-apples” comparison between their own internal requirements and Federal requirements for State DOT GIS divisions, but the plan to do this comparison is still theoretical at this time.

Back to Table of Contents
Back to Top

3.4 Oregon Department of Transportation (Oregon DOT)

Background

The Oregon Department of Transportation has a GIS division that includes 14 staff members. Unlike other technical services, including engineering and natural and cultural resource disciplines, the GIS division remains centralized along with other technical leadership centers. Oregon DOT works with different groups to populate the data and then prepare it with an Enterprise Data Management System. They have approximately 500 Esri GIS desktop software users, and approximately 30-40 super users. One of the challenges that Oregon DOT faces is knowing how many GIS software licenses it needs to successfully and efficiently provide GIS services to the entire agency. It is not clear to the State DOT whether or not all licenses are being used to the extent needed.

Oregon DOT uses geospatial tools for several purposes, most notably:

Oregon DOT wants to explore using GIS for more analysis work. When working with different groups, the State DOT wants to have the capacity to analyze various forms of geospatial data in-house. For example, the Oregon DOT would like to expand their work with safety priority index systems that takes crash record information and develop hot spots. In order to complete such a project, however, the GIS staff needs to collaborate with traffic engineers, safety engineers, and other groups. Another example of a desired project is understanding the impacts of a speed limit increase. Oregon DOT's planning section does a lot of analysis in this field but believes there is an opportunity to use more specialized tools that can provide new and more detailed perspectives. The State DOT would also like to see more reporting and integration along with expanded dashboard tool applications. Oregon DOT is exploring what more they can do with Esri Insights.

In addition to analysis, the State DOT would like work more with reporting and integration. Creating interactive graphic and ad-hoc capabilities to create report-friendly products, which will allow more users to get access to the data.

Assuming the cost of completing the URISA CMM would be low, Oregon DOT wanted to explore what kind of value could be obtained from an initial high level assessment of its GIS division. A CMM could serve as a yard-stick for whether or not the DOT should do an in-depth contractor study, which might result in strategic and implementation plans. Such plans can then be shared with stakeholders and a steering committee.

Benefits of CMMs

From Oregon DOT's perspective, organizational assessments can provide a concise summary of a GIS division's needs by category. Knowing what areas to address can also lead to a clear course of actions for addressing specific needs. At Oregon DOT, geospatial and technical managers always push to keep up with advancements in the field of GIS. Doing so can be challenging to accomplish in addition to more direct day-to-day responsibilities. A CMM can provide a way for identifying where a GIS division or staff are lacking in skills while also evaluating everyday business processes.

Challenges of CMMs

Currently, Oregon DOT's approach for managing GIS services is working well for the organization. Adopting a new method for evaluating and keeping track of their operations can potentially jeopardize their current work flow success. In addition, the State of Oregon has a hiring freeze, which also limits any additional staff to take on organizational assessment efforts. Given the hiring freeze and recent changes in the total number of staff, the State DOT is focusing its resources on cross-training employees. In order to adopt a new evaluation method, such as a CMM, the method would have to be low cost and not require additional staff.

URISA GIS CMM Strengths

Using the URISA CMM proved to be a low-resource task for Oregon DOT. Completing the URISA GIS CMM did not require many man-hours. The agency completed the CMM with four individuals in a total of 7 hours. The results from each individual's answers were compiled and averaged in order to produce one set of results on the CMM tool. Oregon DOT recognized that, in addition to taking a small amount of time, which allowed day-to-day work activities to not be disrupted, the State DOT did not have to reach out to any contractors or outside staff in order to complete the evaluation. In the end, the low time and staff requirements proved beneficial for multiple offices within Oregon DOT including the Procurement Office, which would have handled any purchases or staff requests had they been needed.

Another benefit of the URISA GIS CMM for Oregon DOT has been its support and use by other agencies. As part of a growing topic at conferences such as GIS-T through workshops and presentations, States have been able to engage in more discussions on using CMMs and these assessments can be useful. As an agency that tries to maintain open and ongoing communication with other State DOTs, Oregon DOT has been able to take advantage of other agency experiences including their experience with URISA's CMM. Finally, the State DOT noted that the tool itself is easy to obtain and complete based on the provided instructions. The questions asked in the assessment are clear and easy to understand.

URISA GIS CMM Weaknesses

One of the main weakness of the URISA GIS CMM is that the model is not State specific enough for wider adoption at Oregon DOT. There are aspects measured in the assessment that are not in the purview of the State DOT's control. For example, questions related to data backup and security are not applicable because those fields are not handled at the State DOT, but instead by contractors. Questions about application development or procurement methodologies might also not apply well to agencies such as Oregon DOT. For Oregon, these topics are dealt with agency-wide, and not just the GIS division, which means the GIS division has to compete or coordinate with other divisions and their needs when developing new software or going through procurements.

The CMM tool is also limited in its use for State DOTs through lack of questions regarding Federal reporting, requirements, and mandates related to State transportation agencies. Creating a metric for whether or not an agency is meeting a requirement, and if it is, how well it is doing so can be very useful for State DOTs. Topics such as Linear Referencing Systems and HPMS are not included in the URISA GIS CMM even though they are crucial requirements for all State DOTs.

Lessons Learned and Recommendations

When completing the URISA CMM with four staff members, there was a lot of agreement with how each question should be answered. Each member of the GIS division had an understanding of the group's GIS needs and therefore found that the CMM's results were not unexpected. Conversely, the CMM tool was also able to demonstrate in what small areas there were varying views of maturity, which started a discussion at the State DOT. The CMM tool motivated engagement between GIS stakeholders, Oregon DOT, and supervisors.

Using the tool, Oregon DOT found that its GIS division currently addresses well its service quality of assurance, legal and policy affairs management, and balances minimal privacy with maximum data usage. The tool also reported the GIS division can improve in its application development or procurement methodology as well as its resource allocation management.

Before completing an assessment, Oregon DOT recommends organizing multiple experienced GIS staff with the agency and have each member complete the assessment independently. Afterwards, the team can discuss in what areas members answers varied widely and why. Once the answers are discussed, the results can be averaged and entered into the tool for an average run. Finally, Oregon DOT believes the output spider diagrams are ideal for high-level discussions on each GIS category with GIS staff and upper management.

If there are more time and resources to complete an assessment, Oregon DOT recommends involving more internal and external stakeholder groups in the evaluation process. Along those lines, the State DOT also recommends following up with other State DOTs who have completed the CMM. By sharing results and experiences, new opportunities can arise for collaboration and sharing lessons learned.

Next Steps

The CMM self-assessment has allowed Oregon DOT to reassess what its GIS divisional weaknesses and strengths are in a clear and formal manner. Having already established a strategic plan for the upcoming six months to one year, Oregon DOT does not expected to deviate from that plan. The URISA GIS CMM has been very helpful in initiating discussions within the DOT regarding project priorities, updating its current plan, and working with GIS stakeholders. While the next steps for the agency do not include incorporating a CMM into their strategic plan, they are interested in the model's potential use in the long term.

Back to Table of Contents
Back to Top

Appendix A: List of Case Study Participants

Agency Name Title Work Phone Email
Iowa Department of Transportation Eric Abrams Geospatial Administrator 515-239-1949 Abrams@iowadot.us
Ohio Department of Transportation Ian Kidner GIMS Systems Administrator 614-752-5743 Ian.Kidner@dot.state.oh.us
Tennessee Department of Transportation Kim McDonough GIS Coordinator 615-741-4037 Kim.McDonough@tn.gov
Oregon Department of Transportation Brett Juul GIS Unit Manager 503-986-3156 Brett.A.Juul@odot.state.or.us
Philip Smith Environmental GIS Project Manager 503-986-3733 Philip.L.Smith@odot.state.or.us
U.S. DOT FHWA Headquarters Mark Sarmiento GIS Planning Specialist 202-366-4828 Mark.Sarmiento@dot.gov
U.S. DOT Volpe Center Andrew Reovan Community Planner 617-494-3843 Andrew.Reovan@dot.gov
Anthony Lucivero Community Planner 617-494-2810 Anthony.Lucivero@dot.gov
Drew Quinton Community Planner 617-494-2991 Drew.Quinton@dot.gov
Michael Green Economist 617-494-2553 Michael.Green@dot.gov

Back to Table of Contents
Back to Top

Appendix B: Interview Guide

Background

  1. Agency details:
    1. What is the size of your GIS team?
    2. Dedicated GIS staff?
    3. Contract Staff (Full-time or Part-time)?
  2. In what high-level ways does your agency currently use GIS or geospatial tools? Which areas represent the biggest ‘pain points’ or hurdles to the adoption of these tools?
    1. Are there other uses of GIS that you would like to employ but currently do not?
      1. What are they? What roadblocks exist to their use?
  3. Does your agency currently employ an Enterprise Data Management system?
  4. Does your agency use Federal funds to support any of its GIS activities? If so, what types of funds (SPR, etc.)?
  5. What initially drove the decision to conduct this current organizational assessment?
    1. What do you expect to get out of a CMM?
  6. What are your future plans for your GIS or geospatial tool development? How, if at all, has the CMM framework informed these plans? Do you anticipate it informing plans in the future?

Past Assessments (If the participant has not performed one proceed to the next section)

  1. Have you previously compared your agency against a GIS/geospatial organizational assessment (other than the URISA CMM)? Which assessment framework did you use?
    1. Did you complete the organizational assessment as an individual, or did you involve additional members of your department, management team, or leadership?
    2. Have you communicated the results of the organizational assessment to agency leadership?
      1. If not, is this something you have considered? If so, what has impeded you from doing so?
  2. What have been the biggest successes related to the use of a GIS/geospatial organizational assessment?
  3. What have been the biggest challenges of using a GIS/geospatial assessment?
  4. What 2-3 lessons or recommendations would you share with other agencies who may be considering an organizational assessment?

URISA Capability Maturity Model (CMM)

  1. Why did your organization choose to test the URISA CMM?
  2. Did you complete the CMM as an individual, or did you involve additional members of your department, management team, or leadership?
  3. How long did it take you to complete the assessment?
    1. Was there anything about your organization in particular that affected this process? (e.g. size, information is spread out, decentralized, unclear roles)
  4. If you completed the assessment again, with the most detailed information possible, how long do you estimate it would take to complete?
  5. What were the impacts (positive and negative) of completing the CMM?
    1. Who do they affect?
  6. Were the questions adequately tailored for a State-level analysis?
    1. How, if at all, can the questions be improved?
  7. Did you encounter any usability challenges when using the tool? (e.g. data limitations, time constraint, unclear instructions, tool crashing, etc.)
  8. Do you have any recommendations for improvements or changes to the tool?
  9. Do you plan on continuing to use this tool on a regular basis? If so, how often?
  10. What 2-3 lessons or recommendations would you share with other agencies who may be considering an organizational assessment?

Conclusion

  1. Would you or your agency be interested in participating in a Community of Practice that maintains communication, continues the dialogue about organizational assessments, and provides support through events and programs?
    1. If so, how could this Community best serve your needs?
    2. What types of events and programming would be most effective, in your view?

Back to Table of Contents
Back to Top

FOOTNOTES

1 See the FHWA GIS in Transportation website: http://gis.fhwa.dot.gov/.
2 URISA, GIS Management Institute, GIS Capability Maturity Model, September 2013 (PDF)
3 URISA, GIS Management Institute, GIS Capability Maturity Model, September 2013 (PDF)
4 http://www.esri.com/news/arcnews/winter1011articles/urisa-proposes.html. Accessed May 8, 2017.
5 URISA, GIS Management Institute, GIS Capability Maturity Model, September 2013 (PDF)
6 Paul Giroux, MSC, GISP http://www.slimgim.info/.
7 Interview with Ian Kidner from the Ohio Department of Transportation on May 15, 2017.