Slimgim-T: GIS Capability Maturity Model for State Departments of Transportation

Case Studies of Select Transportation Agencies

May 2018

Prepared for:
Office of Planning
Federal Highway Administration
U.S. Department of Transportation

Notice

This document is disseminated under the sponsorship of the Department of Transportation in the interest of information exchange. The United States Government assumes no liability for the contents or use thereof.

The United States Government does not endorse products or manufacturers. Trade or manufacturers’ names appear herein solely because they are considered essential to the objective of this report.

NSN 7540-01-280-5500

Standard Form 298 (Rev. 2-89)
Prescribed by ANSI Std. 239-18
298-102

Contents

• Executive Summary
• 1. Introduction
  • 1.1 Purpose
  • 1.2 Background
  • 1.3 Methodology
• 2. Case Studies
  • 2.1 Findings
    • 2.1.a Tool is user-friendly and comprehensible
    • 2.1.b CMM is geared towards State DOTs
    • 2.1.c Results and summary sheet are useful
    • 2.1.d All pilot agencies willing to share results
  • 2.2 Suggestions for Improvement
    • 2.2.a Confusion by what perspective users should take
    • 2.2.b Confusion around “Enterprise GIS”
    • 2.2.c Pilot agencies would like more information on how to improve
  • 2.3 Recommendations
    • 2.3.a Define the perspective
    • 2.3.b Define “Enterprise GIS”
    • 2.3.c Refine “Enterprise GIS Sustainability” category
    • 2.3.d Incorporate score definitions on every sheet
    • 2.3.e Refine the Results and Summary Stats sheets
• 3. Conclusion and Next Steps
• Appendix A: Case Study Participants
• Appendix B: Interview Guide

Executive Summary

Since its wide-scale adoption by transportation agencies, geographic information systems (GIS) technology has provided a tremendous value to each agency and has been integrated into everyday organizational decision making and analysis. In many cases, agencies are no longer asking themselves if they should or should not use GIS, but how well have they adopted this technology and how can they improve upon it. One way in which transportation agencies approach these questions is through a self-evaluation, otherwise known as an organizational assessment. In particular, some agencies have adopted more specific assessments known as capability maturity models (CMMs), which assess the ability of an agency to effectively and efficiently carry out specific GIS operations, rating each operation using a numerical scale.

To explore the potential of the newly developed CMM, the Federal Highway Administration (FHWA) and the U.S. DOT John A. Volpe National Transportation Systems Center (Volpe Center) reached out to a number of U.S. State and Canadian Province transportation agencies to pilot the CMM and participate in a follow-up interview to create a case study focusing on their experiences. The final list of pilot participants consisted of:

• Arizona Department of Transportation (AZDOT)
• Arkansas Department of Transportation (ARDOT)
• Manitoba Infrastructure and Transportation (Manitoba)
• Utah Department of Transportation (UDOT)
• Washington State Department of Transportation (WSDOT)

This report serves to inform those involved in the creation of the CMM of its strengths and weaknesses while providing guidance for how to improve the tool. The report is also meant to inform the wider audience of the development of the tool, how it can be used, and how to access it.

This report supports GIS practitioners and decision makers by identifying examples of noteworthy practices, considering the benefits and limitations of implementing a capability maturity model approach, and determining how to identify and address factors that alter the effectiveness of a GIS division in applying geospatial tools and services.

Back to Table of Contents
Back to Top

1. Introduction

1.1 Purpose

The Federal Highway Administration (FHWA) has promoted geographic information systems (GIS) as a means to more effectively manage and improve transportation systems. One of the ways that FHWA has done this is through its GIS in Transportation program,¹ which identifies timely and critical GIS issues and topics in transportation and connects transportation agencies with available resources and best practices. While several State Departments of Transportation (DOTs) and metropolitan planning organizations (MPOs) have already implemented or are in the process of implementing GIS and geospatial tools to address complex issues, others are in the early stages of using these tools to address the same types of needs and challenges. More so, the level of success and efficiency at which these tools are applied can vary greatly from one agency to the next. While it has been common for agencies to identify or acknowledge GIS-related challenges, implementing an efficient and repeatable process for doing so has not. To address this issue, some transportation agencies have implemented what is known as an organizational assessment—a process through which an agency evaluates its strengths and weaknesses in a given operational area. An organizational assessment will generally conclude with a series of recommendations that can address existing or newly identified weaknesses.

The topic of organizational assessments is not a new area of research but its application in GIS—particularly GIS for transportation purposes—over the last 10 to 15 years has been sparse and mostly unknown within the GIS community. The idea of assessing organizational structures and services originated from another recent topic of returns on investment (ROI). In the field of GIS, ROI studies are undertaken as a means to estimate the benefits associated with adopting a GIS application or service by quantifying outcomes such as reduced production times, lower data collection costs, and expedited project delivery timelines. An ROI study measures the change after an implementation of a new technology or process, which is what has motivated organizational assessments. Through an organizational assessment, transportation agencies can provide a baseline for the current state of services provided by an agency. After implementing a new application or altering a business process, a follow-up organizational assessment can demonstrate how the GIS division as a whole has changed over time, particularly in areas most closely related to the new application or business process. Assessments can also provide a snapshot of the effectiveness of current GIS practices, which do not require other baseline or follow-up evaluations. For example, an assessment can identify how data is stored across a DOT and if there are duplicative efforts for storing and processing that data. Identifying these inefficiencies is the first step toward eliminating them and ultimately raising the level of maturity with which an agency employs their GIS capabilities.

1.2 Background

The FHWA GIS in Transportation Program recognized the potential benefit that organizational assessments can have for State DOT GIS divisions and, in 2016, held a peer exchange with GIS staff from seven State DOTs focused on organizational assessments.² The meeting included State DOTs with varied organizational assessment experience, including those whom had at least one assessment and those with no experience. The peer exchange was meant to start a discussion among interested parties where States that had assessment experience could share their lessons learned and best practices with the rest of the group. In addition to learning how an organizational assessment can lead to cost savings and what the common requirements are when carrying one out, one of the outcomes of the peer exchange was the desire for a standard and efficient approach to completing an assessment. As a result, it was suggested that the capability maturity model (CMM) from the Urban and Regional Information Systems Association (URISA) be investigated as a possible standardized assessment tool. CMMs are models that evaluate an organization’s overall level of maturity by how well it can carry out tasks related to a certain topic. According to URISA, their GIS CMM assessment “provides a theoretical model of a capable and mature enterprise GIS operation within a designated organization.”³

In 2017 FHWA’s HEP Office sponsored a peer exchange hosted by the Ohio Department of Transportation. The goal of the exchange was to create a new CMM tailored for State DOT use. The purpose of the new CMM is two-fold: 1) to provide State DOTs with a resource to evaluate themselves in terms of efficient use of geospatial systems, determine how well they are meeting the goals and mission of the agency, assess where they need to improve and thus focus resources, and communicate performance metrics related to GIS capabilities with upper management within the agency and 2) to use those results to compare themselves to their peer agencies to create a baseline measure of their maturity using geospatial systems, identify agencies with a greater level of maturity in those areas allowing them to share best practices, and provide guidance to FHWA for how to target help. One of the most significant benefits of a CMM is establishing standardized metrics that are collected and measured in the same way by different organizations. This standardization process allows similar types of transportation agencies to more accurately compare themselves with their past selves and other transportation agencies.

1.3 Methodology

Peer exchange participants collectively decided to use the Slimgim CMM⁴ as a starting point for the creation of a CMM for State DOTs which became known as Slimgim-T. The Slimgim CMM is one of several templates of a Geospatial Information Management (GIM) framework. It was first conceived in 2012 while the founder was working to map GIS processes and to gain understanding of the organization’s impact on Enterprise GIS (EGIS). The tools were formalized in 2013-14 as a master of science dissertation titled “The Slim GIM Framework: A Pragmatic Geographic Information Management Framework for Assessing and Improving Enterprise GIS and Workflow Health.” It was shown that numerous factors including organizational leadership and culture have a significant impact on people, process, and ultimately on enterprise data quality. The objective was to develop a reusable and adaptable framework to measure EGIS health and maturity and to develop a practical transformational approach to implementing the proposed framework by GIS professionals. The framework tools and approach needed to be practical, quick, and easy to use.

To foster reusability and adaptability, the framework was shared online with the broader community via Creative Commons. The maturity model template in particular continues to be adapted and improved. Agencies that have employed the model have contributed feedback and factors that have been used to update the base model. The Slimgim CMM was suggested by the Iowa Department of Transportation, which used this assessment for about six years to evaluate the agency’s organizational maturity at a high level. In reworking the Slimgim model to fit the needs of State DOTs, the participants narrowed down to five categories from the original six. The first category, Organizational Structure and Leadership, aims to assess the maturity of the organization’s upper level management. The Corporate Culture category evaluates the organization’s practices in regards to GIS. The Organizational Capability category gauges the staff’s ability to utilize GIS programs to meet project needs. The goal of the Enterprise GIS Sustainability category is to ensure that GIS enterprise has continual financial support. The fifth category, Foundational Data and Technologies, evaluates the organization’s data maintenance practices.

Participants went through each of the criteria and selected those that appropriately assess the categories. Once the criteria were selected, the participants made necessary revisions so criteria specifically addressed State DOTs. Once Slimgim-T was ready for use, the Volpe Center and FHWA jointly identified a number of transportation agencies that had shown interest in CMM assessments, but were not involved in the creation of the pilot version of the assessment. The Volpe Center chose this approach so that completing Slimgim-T would be the first exposure to the content for all participants to help reduce bias in reporting their experience. After a participant agreed to take part in the study, the Volpe Center sent the participant a link to the Slimgim website. The website contained a download link to Slimgim-T as a spreadsheet as well as access to a Google Sheet version, which could be completed online. Follow-up interviews were also scheduled with participants about 3 to 4 weeks in advance so that they knew how long they had to complete the CMM assessment and could schedule time to do so as they were able. Participants were given limited additional direction on how to complete the assessment, including how much time to spend on it, who to involve, or how to approach it.

This approach led to each agency completing the CMM assessment in one of a number of ways. Each participant chose a method to complete the assessment that worked the best for their particular situation. This depends on the role within the agency of the individual completing the assessment, the number of employees in the agency who work closely with GIS, and the agency’s organizational structure. Some participants completed the assessment alone, as an individual, and some met with a number of colleagues as a group to complete the assessment; others had individuals throughout the agency each complete the assessment individually before reviewing their responses to produce a final version that took into account a number of viewpoints. The variety of approaches to completing the assessment also led to varied lengths of time to complete the assessment, ranging from 20 minutes to two hours.

Each participant was interviewed about their experience using the same set of questions (see Appendix B). Interviewers, however, often asked participants to elaborate on an area of particular interest. Through this forum, participants were able to recount their experience completing the assessment and express their opinion on what could be improved and what worked well or was confusing.

Below is a summary table that provides background information on the participant agencies (Table 1). The following sections summarize the key findings and recommendations.

Table 1. Case Study Participant Agency Characteristics

Most participants reported that they benefited merely from completing the assessment, even before analyzing the results. Agencies who chose to complete the assessment as one group, as well as those wherein multiple individuals completed the assessment independently reported that discussing the responses with their colleagues felt appropriate for any given question and was itself a productive activity.

Back to Table of Contents
Back to Top

2. Case Studies

This section presents an in depth discussion about agencies’ experiences with the pilot CMM.

2.1 Findings

Often, a number of participants would express a similar sentiment when it came to a particular facet of the assessment. This section of the report breaks out the participants’ feedback into themes and consolidates the individual responses into those themes.

2.1.a Tool is user-friendly and comprehensible

Most pilot participants found Slimgim-T to be easy to use, which is important because it increases efficiency, productivity, and user satisfaction, in addition to not disincentivizing participants from completing the assessment. Understandable content in a convenient format optimizes user time spent with the tool. Users are able to focus on scoring themselves rather than figuring out how to operate the tool or distinguishing what the criteria is asking for. Effective usability also increases the likelihood that users will revisit the tool for future assessments. Representatives from the Washington State Department of Transportation (WSDOT) and Arkansas Department of Transportation (ARDOT) mentioned how the spreadsheet format of the tool was simple and easily navigable. However, the Arizona Department of Transportation (AZDOT) interview respondent mentioned that some users at AZDOT felt hesitant about the spreadsheet format, citing that they thought they might break a built-in formula by accident. Participants from Utah Department of Transportation (UDOT) mentioned how the material was comprehensible and the questions were well written. Those from AZDOT and ARDOT also believed there were a reasonable number of questions and that those questions thoroughly covered the category topics.

2.1.b CMM is geared towards State DOTs

A few participants noted how the categories and questions were appropriately tailored for State DOTs. Representatives from UDOT, in particular, made note of how the questions appropriately addressed State DOTs. Existing CMMs, such as the URISA model, are designed with local MPOs in mind. Transposing these models for State DOT use is ineffective because often the State DOT is a much larger organization with complex organizational structure. Adapting a CMM for States is important because it allows for appropriate internal reflection and more accurate results.

2.1.c Results and summary sheet are useful

Upon completion of Slimgim-T, scores are generated in each category that indicated the agency’s current state of maturity and the likelihood that they will improve based on the inputs. The summary sheet creates a number of graphs for users to visualize their agency’s maturity. Participants agreed that the results and summary sheets provided a lot of useful insight on where the agency should focus on improving. The AZDOT representatives found the summary stats diagram to be especially useful. They believe being able to graphically show upper management the current state of the agency and compare it to where they would like to be will justify the need for additional resource allocation when it comes time to make the budget.

2.1.d All pilot agencies willing to share results

When asked about a publicly available national repository for all State assessment results, all the participants said they would be willing to share their results. Compiling all the results across the Nation would allow for States to see where they are falling behind. Furthermore, it will make it possible to identify those agencies that are particularly mature in a given business practice. This will allow those States with a lower maturity level to independently seek out contacts from a State with a higher level of maturity as well as allow an agency such as FHWA to do the same.

2.2 Suggestions for Improvement

2.2.a Confusion by what perspective users should take

Participants from WSDOT had a difficult time deciding how to approach the questions because they were unsure what organizational level they should be keeping in mind throughout the CMM. For example, users did not know whether they should be completing the CMM assessment by answering the questions with respect to how they, personally, handle different business practices, how their business unit does, or how the agency as a whole does. Depending on the perspective, the answer to any given question may vary widely. It is important for all users to approach the questions with the same mindset, so more accurate comparisons can be made between agencies and within the same agency from year to year.

2.2.b Confusion around “Enterprise GIS”

EGIS is a platform for delivering organization-wide geospatial capabilities while improving access to geographic information. EGIS integrates applications, data, and technology so all relevant information and functions are in the same place. EGIS is not something inherent to all agencies, including some of the pilot agencies. Washington DOT and Manitoba Infrastructure and Transportation (Manitoba) were both unsure how to score themselves on the enterprise GIS questions because neither of them operate on an EGIS system. WSDOT also mentioned that individuals who participated in filling out the CMM-defined enterprise GIS differently, and thus changed how they each scored related questions.

2.2.c Pilot agencies would like more information on how to improve

Some agencies would like guidance for improvement to be provided somehow. Participants from ARDOT would like ways to connect with other State DOTs. The CMM identifies where agencies are struggling, but it would be beneficial for users if the CMM took it a step further and provided resources for agencies that are looking to improve in those areas.

2.3 Recommendations

2.3.a Define the perspective

Participants from WSDOT and ARDOT suggested that the assessment explicitly state in the instructions at what level those completing the CMM should answer the questions (i.e., agency-level, division-level, unit-level). This will ensure that all DOTs are approaching the questions the same way and allow for better comparison.

2.3.b Define “Enterprise GIS”

Incorporating a formal definition of “Enterprise GIS” somewhere in the CMM was recommended. Participants from WSDOT reported that they had a difficult time reaching a consensus on what they believed was meant by the term “Enterprise GIS.” The differing opinions yielded different answers to the Enterprise GIS-related questions.

2.3.c Refine “Enterprise GIS Sustainability” category

The model assumes that most agencies have EGIS in place, but this is not always the case. Manitoba lacks an EGIS system, which made it difficult to answer questions. Participants from Manitoba suggested tailoring the CMM to non-users of EGIS and the use of EGIS to be a sign of maturity.

2.3.d Incorporate score definitions on every sheet

Participants from AZDOT felt that they were regularly referring back to the first tab to see the score definitions. They believed they would save a lot of time if the definitions of the scores were available on every sheet.

2.3.e Refine the Results and Summary Stats sheets

Although participants agreed that the results tab was extremely useful, some suggestions for improvements were made. UDOT would like directions on how to interpret the spider diagram to be included. Manitoba suggested future CMM results tabs to feature a mechanism for comparing the agency’s maturity across time, including a number of years into the past, so they could see where they’ve improved and when. Arkansas would like the results tab to provide some guidance on how to improve in the categories in which they scored lower levels of maturity.

Back to Table of Contents
Back to Top

3. Conclusion and Next Steps

Overall, the pilot study gave the CMM working group insight on the strengths and weaknesses of the CMM in its current state. The CMM content is comprehensible and appropriate to the target audience. Results of the CMM are presented in an effective manner that allows agencies to visualize where they are thriving and where they need to improve. Some aspects of the CMM need clarification such as what perspective the user should take when approaching the CMM and what EGIS refers to. Although users noted how potentially helpful the results and summary sheets are, guidance on how to interpret the results and summary sheets would be beneficial.

Moving forward, the working group will review these findings and recommendations and make changes where they see fit to create the final version of the model. Once the model is finalized, the working group will develop an outreach and marketing plan for broad distribution of the CMM. The working group will also explore mechanisms for consolidating, comparing, and sharing completed CMMs on a national basis.

Back to Table of Contents
Back to Top

Appendix A: Case Study Participants

Back to Table of Contents
Back to Top

Appendix B: Interview Guide

Background

1. Agency details:
   1. Approximately how many full-time, non-contract employees work for your organization?
   2. What is the size of your GIS team (full and part-time, not including contractors)?
   3. Number of State GIS staff that spend at least 50% of their time on GIS-related tasks?
   4. Number of contract staff (Full-time or Part-time) that work with your GIS team?
2. In what ways does your agency currently use GIS or geospatial tools?
3. Are there other uses of GIS that you would like to employ but currently do not?
   1. What are they?
4. What are the biggest challenges or hurdles to using existing GIS tools and/or limitations to potential not-yet-used GIS tools? (PROBES: bureaucratic hurdles, financial hurdles, staffing hurdles, time-resource hurdles, computer processing hurdles, technical skill hurdles, other)

Past Assessments

5. Have you or someone at your organization applied a Capability Maturity Model (CMM) to your organization’s GIS division in the past?

   IF YES

   1. Who led the CMM effort, you or someone else?
      

      IF YOU, proceed to b.

      IF SOMEONE ELSE, please try to answer the following questions to the best of your ability and reach out to the staff member(s) that led the CMM effort to obtain the answers you are not familiar with:

   2. What is the name of the assessment framework used? (the following are a short list of potential answers: URISA, Slimgim, Custom-in-house, etc.)
   3. How many assessments has your organization completed in the last 10 years?
   4. What year was the last assessment?
   5. How frequently does your organization complete assessments (i.e. yearly, semi-annually, quarterly, or ad hoc)?
   6. Did you complete the organizational assessment as an individual, or did you involve additional members of your department, management team, or leadership?
   7. Did you communicate the results of the organizational assessment to agency leadership?
   8. Do you have an anecdote about a major success you’ve accomplished as a direct result of completing a CMM assessment?
   9. What have been the biggest challenges to using GIS/geospatial assessments?

      IF NO OR DON’T KNOW:

   10. Have you or your organization considered using capability maturity models or assessments?
       1. If yes, what has impeded you from doing so?
       2. If not, what has been the motivation behind choosing to no do so?

Pilot CMM

The following set of questions will focus on the Capability Maturity Model you piloted in advance of this discussion.

6. Did you complete the CMM as an individual, or did you involve additional members of your department, management team, or leadership?
   1. [If others were involved]: Who was involved and why?
7. How long did it take you to complete the assessment? (estimate everyone’s time including prep time in hours)
   1. Were there any factors or issues that affected your ability to complete the CMM? (PROBES: size, information is spread out, decentralized, unclear roles, time, priority of work)
8. Were the questions adequately tailored for a State (Province)-level analysis?
   1. How, if at all, can the questions be improved? (please be as specific as possible)
9. Did you find the format of the results tab useful and informative?
10. Did you encounter any usability challenges when using the tool? (PROBES: data limitations, time constraint, unclear instructions, tool crashing, etc.)
11. Do you have any recommendations for improvements or changes to the tool?
12. If you were to complete the same assessment a second time, knowing what you know now, how would you approach it differently?
    1. How long do you estimate it would take to complete? Do you have any final thoughts you would like to share, including feedback topics we have already covered throughout this discussion?

National Comparison

One of the long-term goals of this project is to have all State DOTs complete the same assessment so that FHWA can identify strengths and weaknesses among GIS-use in State DOTs and address State needs using this data. The goal is to also measure maturity changes within and across States over time.

13. Is your organization willing to submit results to a common repository (to be determined) so that your results can be compared nationally?
14. Would such a program incentivize you to complete this assessment on an annual basis?
    1. In your opinion, what is the best mechanism or mechanisms by which the results of all DOTs should be consolidated, compared, and shared with those who participate?

Back to Table of Contents
Back to Top

FOOTNOTES