draft preliminary evaluation plan for the vii for safety ...draft preliminary evaluation plan for...

Draft Preliminary Evaluation Plan for the

VII for Safety, Mobility, and User Fee Evaluation

Prepared for Minnesota Department of Transportation Office of Traffic, Safety and Technology

Prepared by Science Applications International Corporation

Transportation Research Division

Submitted June 30, 2009

Produced Under Federal Work Order Number ITS IT08 650, State Project Number 8816-1153 by SAIC for the Minnesota Department of Transportation

Page i

TTAABBLLEE OOFF CCOONNTTEENNTTSS

Chapter 1. Introduction ................................................................................................................... 1

Evaluation Goals ......................................................................................................................... 1

Roadmap to the Remainder of this Document ............................................................................ 1

Chapter 2. Scope of the Evaluation ................................................................................................. 3

Chapter 3. Evaluation Goals and Objectives ................................................................................. 6

Documenting the Programmatic / Implementation Experience .................................................. 6

Assessing the Technical Performance of Systems and Sub-Systems ......................................... 8

Assessing Customer Satisfaction with the Applications ........................................................... 10

Investigating Safety Impacts ..................................................................................................... 13

Investigate Mobility Impacts .................................................................................................... 15

Assessing Future Potential and Overall Feasibility .................................................................. 19

Chapter 4. Design of the Pilot Study ........................................................................................... 21

Location of the Pilot Test .......................................................................................................... 21

Sampling Design ....................................................................................................................... 24

Criteria for Selection of Sites for Targeted Sampling .............................................................. 25

High-Level Criteria for Participant Selection ........................................................................... 25

Chapter 5. Determining Sample Size for the Pilot ....................................................................... 27

Sample Size for Surveys to Assess Driver Perceptions ............................................................ 27

Sample Size for Assessing Safety Benefits .............................................................................. 29

Sample Size for Assessing Effects of MBUF on Driving Behavior ......................................... 30

Sample Size for Probe Data ...................................................................................................... 31

Chapter 6. Overview of Data Collection ....................................................................................... 33

Pre-Deployment ........................................................................................................................ 33

Deployment ............................................................................................................................... 36

Participant Payment Approach ................................................................................................. 38

Chapter 7. Survey Content and Administration ........................................................................... 42

County-Wide Driver Survey ..................................................................................................... 42

Statewide Survey ...................................................................................................................... 44

Chapter 8. Focus Group / Interview Content and Procedures ..................................................... 45

General Format ......................................................................................................................... 45

Chapter 9. Roles and Responsibilities / Next Steps ..................................................................... 47

Page ii

Responsibilities for Evaluation Activities ................................................................................ 47

Next Steps ................................................................................................................................. 48

Minnesota Department of Transportation Draft Preliminary Evaluation Plan for the VII for Safety, Mobility, and User Fee Evaluation (Exhibit E)

Page 1 of 51

CHAPTER 1. INTRODUCTION This document builds upon the evaluation goals and strategies established by the Minnesota Department of Transportation (Mn/DOT) for the VII for Safety, Mobility, and User Fee Project. SAIC delivered the Draft Final Goals and Strategies for the evaluation to Mn/DOT on May 4, 2009. This document provides an overview of suggested deployment scenarios; discusses required sample sizes to look at customer acceptance, safety, and mobility; provides insights into evaluation timeframe and major activities within that timeframe; and, finally, presents preliminary approaches for the primary data collection activities, surveys, and focus groups.

Evaluation Goals Minnesota’s VII for Safety, Mobility, and User Fee Project intends to pilot three VII applications:

1. Mileage-Based User Fees

2. In-Vehicle Signing

3. Traveler Information with Probe Data

The purpose of the pilot project is to assess the value of a system that integrates a Mileage-Based User Fee (MBUF) application with other vehicle safety and mobility applications, and to provide information that will inform future public policy decisions regarding mileage-based user fees.

To accomplish these overall goals for the project, SAIC has identified six more specific goals for the evaluation as presented below. Note that each of these goals applies to all three of the VII applications with the exception of goal 4, which is focused only on the safety aspects of the in-vehicle signing application.

1. Document the programmatic / implementation experience.

2. Assess the technical performance of systems and sub-systems.

3. Assess customer satisfaction with the applications.

4. Investigate safety impacts (for the in-vehicle signing application only).

5. Investigate mobility impacts.

6. Assess future potential and overall feasibility.

Roadmap to the Remainder of this Document The remainder of this document is organized as follows: Chapter 2, Scope of the Evaluation summarizes the various implementation and execution requirements necessitated by the evaluations goals and objectives. A review of the proposed approaches for each of the evaluation goals and objectives can be found in Chapter 3, Evaluation Goals and Objectives. Readers who are already familiar with the Goals and Strategies document should skip Chapter 3.

Chapters 4 and 5, Design of the Pilot Study Scenarios, and Determining Sample Size for the Pilot, describe assumptions made regarding the problem space considered in the pilot study, including the population of drivers to sample, potential sites or locations, and various situations and contexts to investigate.

Following this introductory material, Chapters 6, 7, and 8 describe specific data collection activities:

Chapter 6, Overview of Data Collection

Chapter 7, Survey Content and Administration


Page 2 of 51

Chapter 8, Focus Group / Interview Content and Procedures

The final chapter, Chapter 9, Roles and Responsibilities / Next Steps, concludes this document and provides an overview of the major elements and considerations in conducting the evaluation.


Page 3 of 51

CHAPTER 2. SCOPE OF THE EVALUATION This section summarizes the various implementation and execution requirements necessitated by the evaluations goals and objectives developed by the project partners and presented in detail in subsequent chapters of this document. These requirements were developed in consideration of both the draft concept of operations and the anticipated extent of deployment resources. Certainly, a deployment could be proposed that exceeds these requirements – resulting in greater probabilities of detecting small impacts and expanding the potential number of insights that could be generated by the evaluation. Similarly, a deployment could be implemented that fails to meet one or more of these requirements – resulting in a diminished capability to detect before and after impacts, and limiting the potential scope of evaluation insights. The requirements listed here are over and above those requirements necessitated by the underlying concept of operations being developed for MnDOT by Mixon/Hill (e.g., ability to receive and present traveler information, capability to record vehicle miles traveled (VMT), ability to transmit probe reports).

Table 1 provides a summary of the evaluation requirements that will be described in subsequent chapters. Table 1.Implications for the Deployment.

Evaluation Requirement Purpose

Concept of Operations Considerations

Include one larger, randomly selected panel of drivers. Provides a representative sample of driver opinions, perceptions, and actions.

Include one smaller panel of drivers targeted around a single origin or destination (e.g. employer, school, etc.).

Facilitates the measurement of safety surrogates (e.g., speed reduction in a targeted school zone) and potential testing of the efficacy of vehicle probes (e.g., may ensure sufficient density of probe data to generate travel times for a limited network).

Technology Considerations

System should be capable of recording and archiving VMT by time of day, jurisdiction, and roadway classification.

Facilitates assessment of impacts of MBUF on travel decisions.

System should be capable of recording and archiving geo-specific speed profiles with sufficient fidelity to determine braking actions or other speed reductions proximate to speed warning locations (e.g. school zones, etc.).

Facilitates enhanced assessment of safety impacts. May be limited to a sub-set of drivers (smaller, targeted panel described above).

If geo-coding is used for the safety portion of the system, the system should include speed warnings for an entire county and/or within a predetermined radius (e.g., a 10-mile radius) around a targeted origin or destination.

Facilitates enhanced assessment of safety impacts. May be limited to a sub-set of drivers (smaller, targeted panel described above).


Page 4 of 51


Sample Size Considerations

Option #1: Total of 475 participants. Distributed as follows:

• 400 randomly selected participants from a single suburban / urban county.

Required to ensure representative sample of suburban drivers with respect to:

• Attitudes and perceptions of system.

• Reported changes in behavior (e.g. travel behavior, speed reductions for safety).

Allows evaluation to statistically detect changes in individual VMT larger than 2.5% (assuming controls for seasonal variations).

• 50 drivers targeted around a single origin or destination (e.g., major employer, school).

Required sample size to statistically detect reductions in average speed (due to speed warning) of 3.5 mph.

Required sample size to derive 1- or 2-minute estimates of travel times (using system vehicles as probes) on targeted arterial.

• 15 drivers in control group (assuming adoption of nine month test duration).

Serves as a control group for VMT impacts.

• 10 rural drivers. Provides informed inputs (through use of devices and focus groups) on potential issues affecting rural users.

Option #2: Total of 865 participants. Distributed as follows:

• 400 randomly selected participants from a single suburban / urban county.

Required to ensure representative sample of suburban / urban drivers with respect to:


• Reported changes in behavior (e.g., travel behavior, speed reductions for safety).


• 400 randomly selected participants from a single rural county.

Required to ensure representative sample of rural drivers with respect to:


• Reported changes in behavior (e.g., travel behavior, speed reductions for safety).



Page 5 of 51


• 50 drivers targeted around a single origin or destination (e.g., major employer, school). This could be in a rural or urban area depending on Mn/DOT’s interest.

Required sample size to statistically detect reductions in average speed (due to speed warning) of 3.5 mph.

Required sample size to derive one-minute estimates of travel times (using system vehicles as probes) on targeted arterial.

• 15 drivers in control group (assuming adoption of 9 month test duration).

Serves as a control group for VMT impacts.

Schedule Considerations


Option #1: 9 month deployment – no cohorts. Provides for a 3 month baseline and 6 months of system experience.

May negatively impact ability to detect changes in individual VMT patterns. Can be mitigated somewhat with use of control group. If individual variability of drivers’ VMT cannot be controlled for, will only be able to detect changes in VMT greater than 20%.

Option #2: 15 month deployment (three cohorts) Provides for a 3 month baseline and 6 months of system experience.

However, drivers deploy in three cohorts such that one group begins its baseline as the preceding group enters the system usage phase. Aids in controlling for seasonal variations.


Page 6 of 51

CHAPTER 3. EVALUATION GOALS AND OBJECTIVES The goals of the evaluation are as follows:

1. Document the programmatic / implementation experience

2. Assess the technical performance of systems and sub-systems

3. Assess customer satisfaction with the applications

4. Investigate safety impacts (for the in-vehicle signing application only)

5. Investigate mobility impacts

6. Assess future potential and overall feasibility

The following sections present a summary of the approach for each goal as presented in the Draft Final Evaluation Goals and Strategies document submitted to Mn/DOT on May 4, 2009. Readers already familiar with the Goals and Strategies document should skip this chapter.

Documenting the Programmatic / Implementation Experience The first evaluation goal is to document the overall implementation experience. Note that this applies to all three VII applications. The primary goal here will be to document whether the overall programmatic approach or business model that Mn/DOT took to VII, and in particular to MBUF, was successful in the pilot demonstration. This will be important in considering alternate means to the gas tax in the future in the state of Minnesota, and perhaps on the national stage. Few studies have been conducted to date looking at alternatives to the gas tax, so the Mn/DOT study will be an important step towards addressing this national challenge.

This area of the evaluation will also include documenting costs associated with development and deployment and documenting the challenges and limitations encountered by the participating organizations and stakeholders.

Documenting the implementation experience will be important in presenting the project findings, as it will give the reader an understanding of the context within which the findings were generated. It will also be important to the evaluation team in gathering and interpreting findings. This aspect of the evaluation will also be important for developing insight into the success factors and pitfalls that may be encountered in any subsequent system expansion.

Financial costs to document will include both capital expenditures and operations and maintenance costs. Some of these costs will be easy to quantify since they are concrete expenses; however, many of these costs are difficult to track. Organizations often have inconsistent record-keeping, record-keeping which is not transparent, or lack the willingness to provide data for evaluation purposes. In order to address these issues, the evaluation team will work to identify liaisons within each organization accruing costs, will develop a format for tracking expenses, and will train these liaisons to record or log the necessary data. The evaluation team will also interview staff to develop a more detailed understanding of the cost data.

Challenges could include a wide range of issues, including those that are administrative, regulatory, technical, and institutional. Some examples of likely challenges that may be encountered are shown in Table 2.


Page 7 of 51

Table 2. Example Challenges to Implementing MBUF that will be Explored in the Pilot Evaluation.

Administrative Ensuring privacy of individual travel data

Allowing for repudiation

Ensuring regular and ongoing participation of participants

Configuration management of safety database (safety)

Data ownership (probes and MBUF)

Regulatory Enforcing MBUF tampering restrictions

Operating within existing legislative and policy framework

Establishing a policy of MBUF for hybrid and electric vehicles

Technical Developing techniques to avoid MBUF evasion

Integrating system and vehicle components

Integrating with fuel tax point of sale (POS) data

Developing a system that allows for repudiation

Allowing for multi-state recording of MBUF

Institutional Gaining buy-in from revenue staff regarding collection of MBUF

Gaining buy-in from fuel companies and franchise owners (if applicable)

Gaining acceptance among privacy advocates and lobbyists

Impacts on various stakeholders

Other Liability (safety)

Privacy (probe data and MBUF)

In identifying challenges, the evaluation team will primarily rely on anecdotal information obtained through interviews with individuals or from focus group meetings. In addition to merely identifying challenges faced, the evaluation team will also target questions to identify success strategies (what worked) and mitigation strategies (what was done to overcome challenges). The evaluation team will, in particular look for learning moments or “lessons learned.” Table 3 presents measures and possible methods for this goal.


Page 8 of 51

Table 3. Approach for Documenting the Programmatic / Implementation Experience.

Goals Measures Data Sources Possible Methods Considerations

Document whether the overall programmatic approach or business model that Mn/DOT took to MBUF was successful in the pilot demonstration.

Project costs Participating organizations and stakeholders including Mn/DOT, the TPM Team, the developer team, technicians, and the Minnesota Department of Revenue

Survey of capital expenditures

Diary or log of maintenance costs in labor and parts

Interviews with key staff to understand costs

Access to staff

Buy-in and responsiveness of staff

Inconsistent record-keeping

Comparisons to baseline data (e.g., costs associated with gas tax compared to MBUF costs)

Identification of project challenges including administrative, regulatory, technical, and institutional.

Participating organizations and stakeholders: including Mn/DOT, the TPM team, the developer team, technicians, and drivers

Observations during regularly scheduled team meetings

Anecdotes

Interviews

Focus Groups

Access to staff

Identification of negative, positive, and mitigating factors

Assessing the Technical Performance of Systems and Sub-Systems The second evaluation goal is to assess the technical performance of systems and subsystems. Note that this applies to all three VII applications. As with the previous goal area, this will be important in presenting the project findings, as it will give the reader an understanding of the context within which the findings were generated. It will also be important to the evaluation team in gathering and interpreting the findings. As an example, if the evaluation team is hearing negative reactions from participants, it will be important to know whether there are particular technology challenges that the participants may be facing which could be causing these feelings (rather than a general lack of acceptance of the policy around MBUF, for example), and the evaluators can then probe on these particular issues.

An important step in the deployment will be usability testing, an activity that occurs in parallel with development, implementation, and testing. A well-considered and cost-conscious deployment plan will integrate evaluation components at every stage from prototyping through the months following system release. Usability activities should occur before, during, and after system design and development.1 Implementing usability engineering has been demonstrated to reduce development time and costs, reduce maintenance costs, and decrease training and support costs.2 An investment in usability engineering can produce a return on investment in the range of 3:1 to 100:1.3

1 Nielsen, J. (1992). The Usability Engineering Life Cycle. IEEE Computer 25, 3 (March), 12-22. 2 http://usability.gov/basics/usasaves.html (Accessed 03/09/2009). 3 Karat, C. Cost-benefit Analysis of Usability Engineering Techniques. Proceedings of the Human Factors Society 34th Annual Meeting. Santa Monica, CA: Human Factors Society (1990).

http://usability.gov/basics/usasaves.html


Page 9 of 51

Table 4. Approach for Assessing the Technical Performance of Systems and Sub-Systems


Assess system reliability (percentage of time system functions as expected)

Uptime of overall system and subsystems

Uptime of various system functions (e.g., percent of time that safety warnings are provided when vehicle enters a speed zone)

System

Drivers

Maintenance logs

System logs

Secondary data collection (e.g., cameras, redundant GPS devices)

Driver logs of suspected failures

Access to staff

Buy-in and responsiveness of staff and drivers

Ability of system to log data

Ability of secondary tools to collect data

Assess system accuracy and precision

Accuracy of various system functions (e.g., for MBUF, calculating miles driven in various jurisdictions)

For In-Vehicle Signing, timing and location of warnings relative to speed zones

System

Drivers

Maintenance logs

System logs

Secondary data collection (e.g., cameras. GPS devices)

Driver logs of suspected failures

Access to staff




Assess opportunity for evasion and tampering

Tamper rate (if applicable)

Impacts of tampering (uptime and downtime, lost revenue costs, repair costs, enforcement costs)

System

Drivers

Technical Subject Matter Experts (SMEs)

Interviews with SMEs

Maintenance logs

System logs

Secondary data collection (e.g., cameras, redundant GPS devices)

Survey of drivers

Need to investigate potential tamper methods with SMEs

Access to staff


Driver honesty and experimental bias




Page 10 of 51

Assessing Customer Satisfaction with the Applications The third evaluation goal is to assess customer satisfaction and acceptance of the system. Note that this applies to all three VII applications. User acceptance of the system is critical as the entire business model could fail if the users of the system—that is, the customers, who are the driving public—do not accept it. If customers do not accept either the technology or the MBUF policies of the Mn/DOT model, it will be important to know why people do not accept it and how or what it might take to overcome this lack of acceptance. Therefore, the customer satisfaction and customer acceptance portion of the evaluation will focus on both technical issues as well as policy issues, both of which are central to this pilot program.

For this portion of the evaluation, the evaluation team will be looking primarily at self-reports of comprehension, utility or usefulness, usability, acceptance, and barriers to success; that is, drivers’ perceptions of the system.

An important input into this portion of the evaluation is the results of the usability testing. Usability tests will provide direct insight into drivers’ use of and reaction to the system (i.e., whether they use the system properly and whether they are able to make sense of system messages).

To complement the findings of the usability study the evaluation team anticipates conducting a combination of surveys, interviews, and focus group meetings to gather feedback from study participants. Table 5 presents measures and possible methods for this goal.

Table 5. Approach for Assessing Customer Satisfaction with the Applications.


Assess driver comprehension of the system

Demonstrated understanding of:

- Overall system functionality

- System limitations (e.g., the work zone system alerts drivers when they enter a work zone, but is not able to differentiate between whether a worker is present or not)

- Interface (e.g., meanings of alerts, interpretations of travel time information)

Drivers Usability testing

Interviews

Focus groups

Surveys

Effects of demographics (e.g., age, trip type, vehicle type)

Integration of testing into the development cycle

Access to participants on an ongoing basis and compensation costs

Ability to compare survey data with behavioral data (i.e., how does opinion data correspond with driving behavior)

Assess perceived utility of the system

Perceived usefulness of the system elements including MBUF mileage reports and school zone alerts


Interviews

Focus groups

Surveys



Access to participants on an


Page 11 of 51


ongoing basis and compensation costs

Assess driver confidence in the system

Perceived accuracy and reliability of in-vehicle signing (both safety and mobility information) and of MBUF system

Perceived fairness of MBUF system compared to gas tax

Perceived fairness of driver’s MBUF system compared that of other drivers

Perceived ability to evade or sabotage MBUF system

Perceived ability to repudiate faulty MBUF assessments

Perceived invasions of privacy


Interviews

Focus groups

Surveys




Assess perceived system quality

Perceived ease of use

Perceived alert or warning quality (e.g., annoying, difficult to notice)

Perceived alert or warning frequency (e.g., too often)

Perceived alert or warning timing (e.g., too late)

Perceived quality of MBUF messages or logs


Interviews

Focus groups

Surveys



Ability to separate drivers’ perceptions of system characteristics from other system aspects (e.g., functionality and purpose, reliability, etc.)



Page 12 of 51


Assess perceived system impacts

Perceived effects on speed

Perceived effects on route choice or route avoidance

Perceived effects on mileage

Perceived effects on driving distraction

Perceived effects on stress

Perceived improvement in driving (safety and mobility)


Interviews

Focus groups

Surveys



Ability to separate drivers’ perceptions of system characteristics from other system aspects (e.g., functionality and purpose, reliability, etc.)


Assess system value and overall driver acceptance

Ratings of desire to use system in the future (or some component of the system)

Willingness to participate in future pilot programs

Identifications of barriers to acceptance

Acceptance of “opt-in” scenario (e.g., willingness to participate in opt-in, perceived fairness of opt-in)

Drivers

Usability testing

Interviews

Focus groups

Surveys



Ability to separate drivers’ perceptions of system characteristics from other system aspects (functionality and purpose, reliability, etc.)



Page 13 of 51

Investigating Safety Impacts Background

The fourth evaluation goal is to investigate the safety impacts of the VII safety applications. Note that this applies to the in-vehicle signing application only. A limited number of school zones, work zones, curves, and speed zones will be identified for the pilot, and the system will convey messages to motorists via their in-vehicle device as they approach these locations, most likely via GPS coordinates where possible and with dedicated short-range communications (DSRC) where communication is not possible (e.g., in rural areas). The nature of the message (i.e., visual, audible, a combination?) is not yet known.

Site Selection Criteria

Just as the participant selection is critical to gathering useful information about customer satisfaction and acceptance, so is having an appropriate set of selection criteria for identifying roadway locations for the pilot study. It would be expected that curves would have the greatest potential for safety impacts. Therefore, the evaluation team would recommend identifying dangerous curves as a starting point for the study if feasible. Ideally the pilot would focus on those curves that are considered to be the most “at risk”.4 After identifying these curves, the team would then look for other roadway locations nearby (e.g., school zones, work zones) which may be appropriate for inclusion in the pilot.

Despite the fact that it is unlikely that the study will show a statistical reduction in crashes, crash data can still be helpful in directing site selection for the pilot. Crash data show that the most “at risk” curves on county roads are those curves with the following characteristics:

A curve radius between 700 and 1,500 feet.

An intersection within the curve.

A “visual trap” within the curve.

The traffic volumes on the curve are within a certain range of volumes (specific to the county or Mn/DOT District).5

Another source of information that may be useful in identifying appropriate sites for inclusion in the pilot are individuals with “on the ground knowledge” of local roads, such as city or county traffic engineers. These individuals are often quite knowledgeable about the trouble spots on their local system. As an example, local staff may be familiar with which school zones have repeat offenders when it comes to speeding.

Measures

In a perfect world, assessing the impacts of any new safety technology would involve a study of crash data before and after the system is introduced. However, crashes are rare events and in many cases it is not possible to detect a statistical reduction in crashes within a reasonable amount of time. That is the case here, where even amongst some of the most dangerous curves in the State of Minnesota, there are

4 It is important to note that this may not be possible due to geographic constraints. The location of the safety applications will need to be considered in the context of other practicalities such as the best location for MBUF, the best location for recruitment of participants, and the best location relative to the deployment team technicians for repairs to the in-vehicle systems. 5 This finding is based on a preliminary analysis of crashes on curves in four Minnesota counties.


Page 14 of 51

typically fewer than one or two crashes per year.6 School zones are perhaps even less likely to have crashes due to the low posted speeds and/or the presence of enforcement. While it will certainly be important to look at any incidents that occur (including both crashes and reported near-misses), it is unlikely that there will be a meaningful number of crashes.

Therefore, instead of focusing on crashes, the team will make use of safety surrogate measures. If possible the team will assess observed driver behavior through areas of interest (i.e., a school zone, work zone, curve, or speed zone). This would be done by looking at surrogates such as braking behavior on the approach to the areas of interest, average speed [through the areas of interest], and speed limit compliance [through the areas of interest]. This level of analysis, however, requires detailed data for all participants (or at least for a select group of participants). This would include the speed profile for each “driving incident” where the participant drives through an area of interest. This information could be obtained either from the system itself if it accommodates this level of data collection and archiving, or possibly through a secondary data recording system that the evaluation team could put in select vehicles.

An important challenge associated with this is privacy and liability concerns. Some participants may be concerned with this level of monitoring either for privacy reasons or due to concerns that their data could be made available in the event of an accident. The evaluation team will work with Mn/DOT to develop strategies to address this. As an example, a select number of the study participants could participate in this portion of the study, or the pilot could have an “opt-out” program whereby participants would have exclusive rights to their logs initially, giving them the opportunity to review their logs on a periodic basis (perhaps monthly) and to remove small chunks of data that they believe may pose a problem for them.

Table 6 presents measures and possible methods for investigating safety impacts of the system.

To this point it has been assumed that the MBUF/VII pilot will likely have positive effects on safety. For instance, drivers may slow down more frequently as a result of an in-vehicle warning, resulting in fewer or less severe collisions. However, any time new information is provided to the driver, there is an opportunity for a negative effect on safety either through communicating poorly understood information or through creating distraction. These are areas that should be considered through usability testing in the software and hardware development phases of the pilot. Further, the evaluation team will be investigating reported comprehension and distraction, as described in the “Customer Satisfaction and Acceptance” section of this document.

An example of an issue that highlights this continuum of positive and negative safety impacts is the area of providing traveler information to the driver. The evaluation team will seek to assess traveler acceptance of the safety information (e.g., do they understand it, do they like the way in which it is presented, do they find it to be useful, do they find it to be distracting). Measures and possible methods are presented below. Additional details on approaches for this were discussed in the “Customer Satisfaction and Acceptance” section of this chapter. As discussed previously, the output of the usability testing will be extremely important to this aspect of the evaluation.

6 An analysis of the most dangerous 50 curves in each of 4 counties in Minnesota showed that the average number of crashes per curve was less than 0.5 per year. Approximately half of the curves had no crashes over a 5-year period while only one or two curves in each county had an average of one crash per year.


Page 15 of 51

Table 6. Approach for Investigating Safety Impacts.


Investigate the impact of the system on driving behavior on approach to areas of interest (e.g., school zones, work zones, curves, speed zones)

Average speed through area of interest with and without system warnings

Level of speed limit compliance through the area of interest

Braking maneuver on approach to the area of interest (e.g., deceleration, “hard stops”)

System

Drivers

Vehicle logs (i.e., system or secondary data recording)

Usability testing

Experiments

Privacy and liability concerns (which may require opt-out or data cleaning functions)

Use of redundant or secondary data collection device (which may require additional access to vehicle, reliance on driver to perform some maintenance activities, and additional compensation)

Investigate the impact of the system (if any) on route choice

Avoidance of zone areas

Vehicles/Drivers Vehicle logs (i.e., system or secondary data recording)

Usability testing

Experiments

Privacy and liability concerns (which may require opt-out or data cleaning functions)

Use of redundant or secondary data collection device (which may require additional access to vehicle, reliance on driver to perform some maintenance activities, and additional compensation)

Assess the perceived utility of the safety information

Perceived usefulness of safety information


Surveys, interviews, and/or focus groups


Access to participants

Investigate Mobility Impacts The next evaluation goal is to investigate the mobility impacts of the system. Note that this applies to all three VII applications.

Mileage-Based User Fees

First, depending on the way in which MBUF is implemented in this pilot, the system itself could affect mobility by impacting the participants’ travel patterns, such as when they drive, how much they drive,


Page 16 of 51

and where they drive. The way in which it will impact travel behavior depends on the policy decisions that are made such as:

How the system is implemented (i.e., if fees are tallied via a real-time system that collects and displays information to the driver or whether fees are simply assessed through an odometer reading).

How much the fee is relative to how much that motorist might otherwise pay under the gas tax.

What the fees are based on (e.g., the fee could vary based on time of day, jurisdiction, and/or facility type).

The presentation of the fee to the driver and how this effects their perception of cost (e.g., costs indicated on their gas pump receipts may be more or less salient than an annual registration fee).

Even in the absence of any of these factors, simply paying a fee based on miles driven rather than based on gallons of gas used may impact driving behavior by making people more aware of their behavior. The recent Portland mileage-based user fees study found that those participants assessed a different fee based on zone and time of day made changes in their travel patterns including: driving less often during the peak periods (and more often during the “shoulder” periods immediately before and after the peak periods, shifting to other modes such as public transit, driving outside of the “congestion zone” to avoid higher fees, and deciding to take fewer trips. The Portland study also found that the group charged a flat per-mile fee that was what they would have paid under the gas tax model actually showed a 12 percent reduction in total miles driven.7

Depending on which aspects of MBUF Mn/DOT is interested in studying, the pilot could be designed to assess only the technical viability of one or more of these factors or it could be designed to assess the impacts of one or more of these factors on motorists’ travel decisions. This decision will need to be made before the evaluation plan is finalized.

Assumptions or requirements in terms of assessing the impact of MBUF on mobility are:

The evaluation team would ideally have access to detailed travel data for all participants (or at least for a select group of participants). Although it is not expected that all of the data of interest will be available, the detailed travel data would ideally include the following data for each trip: origin, destination, general route information, departure time, arrival time, number of miles driven in total for trip, number of miles driven in each jurisdiction (as applicable), and number of miles driven on various facility types (as applicable). Some participants may be concerned with this level of monitoring either for privacy reasons or due to concerns that their data could be made available in the event of an accident. The evaluation team will work with Mn/DOT to develop strategies to address this. As an example, a select number of the study participants could participate in this portion of the study, or pilot could have an “opt-out” program whereby participants would have exclusive rights to their logs initially, giving them the opportunity to review their logs on a periodic basis (perhaps monthly) and to remove small chunks of data that they believe may pose a problem for them.

The team would ideally have an opportunity to collect this detailed information from participants during a “baseline” period prior to when the “MBUF-charging” period would begin. This will allow the team an opportunity to compare individual driver behavior prior to the addition of MBUF (or even with various scenarios of MBUF).

7 Oregon Department of Transportation, Oregon’s Mileage Fee Concept and Road User Fee Pilot Program, Final Report, (Salem, OR: November 2007). http://www.oregon.gov/ODOT/HWY/RUFPP/docs/RUFPP_finalreport.pdf

http://www.oregon.gov/ODOT/HWY/RUFPP/docs/RUFPP_finalreport.pdf


Page 17 of 51

The team would ideally establish a control group to account for external factors that can influence travel (such as seasonal variation and changes in gas prices). This control group would include the same monitoring devices as the MBUF group, but would pay the gas tax as usual rather than the MBUF fee. Impacts due to seasonal variation and changes in gas prices can also be assessed to some degree through interviews and focus groups.

Table 7 presents measures and possible methods for investing the mobility impacts of MBUF. Table 7. Approach for Investigating Mobility Impacts of MBUF.


Investigate the impact of MBUF on travel choices

Miles driven

Time of day of trips

Route choice

Number of trips per day / trip chaining

Mode choice

Vehicles

Drivers

System logs


“Modeling” of mobility impacts with inputs from survey taken together with background travel times.



System must log

In-Vehicle Signing

Another aspect of the project that could impact mobility relates to the in-vehicle signing component of the project. At this point is it unknown whether participants in the pilot will be provided with real-time information about travel times via their in-vehicle device. If this is the case, the evaluation team may seek to assess traveler usability and acceptance of the information (e.g., do they understand it, do they like the way in which it is presented, do they find it to be useful, do they find it to be distracting). Measures and possible methods for investigating the impact of the mobility components of the VII system are presented below. Additional details on possible approaches for this were discussed in the “Customer Satisfaction and Acceptance” section of this document.

Table 8 presents measures and possible methods for investing the mobility impacts of the traveler information.

Table 8. Approach for Investigating Mobility Impacts of Traveler Information.


Investigate the perceived utility of the traveler information

Perceived usefulness of en-route information about travel times



“Modeling” of mobility impacts with inputs from survey taken together with background travel times.




Page 18 of 51

Traveler Information with Probe Data

Regardless of whether travel time information is presented to motorists, the MBUF/VII system could be used to collect probe data from the vehicles participating in the pilot, and therefore has great potential to impact mobility in the future. Mn/DOT currently has good sensor data for freeways but lacks coverage on arterial streets. The agency has expressed interest in using the system to accomplish an additional objective of collecting probe data that could be used to calculate travel times on both arterials and freeways. The concept of using vehicles as probes to estimate travel times has been a common practice in the transportation community for a long time, beginning with researchers gathering information about travel times on a route by conducting “floating car runs” (i.e., driving vehicles in such a way that they pace traffic along a route of interest). This use of manual probes, while effective, was expensive and not suitable for widespread collection of travel time information for traveler information.

With the recent advent of methods for remotely identifying vehicles (e.g., using license plate recognition, using toll tag transponders, using blue tooth signals from cell phones) and in-vehicle navigation systems, two new approaches for using probe vehicles to measure travel times became possible:

One can identify vehicles that pass a starting point of a travel time segment and record the time at which the vehicle passed that location. Then, one can monitor vehicles passing the end point of that segment until the identified vehicle is located. The time difference between when the vehicle passed the ending and starting locations is the travel time for that vehicle.

One can build into in-vehicle navigation systems methods for tracking travel times along pre-specified segments and the in-vehicle system can either (a) transmit information about recent travel times when the vehicle passes a data collection “hot spot” or (b) transmit its current location, speed, and heading information on a periodic basis.

Either of these approaches may be feasible with the type of hardware that will be used to support Mn/DOT’s MBUF/VII system.

With the first approach, DSRC transmitters would be placed at the start and end points of select travel time segments. Equipped vehicles passing these locations would be identified by IDs associated with the in-vehicle DSRC units. These time-stamped ID measurements would be transmitted to a central processor,8 which would match measured IDs for the start and end nodes of travel time segments to produce travel time estimates.

With the second approach, the in-vehicle system would monitor vehicle movements and note when a vehicle made an appropriate trip along pre-defined travel time segments; i.e., a trip that did not include any diversions off of the segment or parking along the segment. When the in-vehicle system detected completion of an appropriate trip, it would store the travel time measurement. When the vehicle passed a DSRC unit designated for collecting travel time information (or if wireless communications are used instead of DSRC, whenever the system is within range to transfer information), the in-vehicle system would send the latest travel time measurements to a central server.

8 A distributed processing model is also possible in which the IDs collected at one node would be transmitted to processors located at adjoining nodes at the end of travel time segments originating from the initial node. These “smart nodes” could compute travel times from the data received from adjoining nodes and the data collected at that node and could transmit the resulting travel time estimate to a central server. An advantage of this approach is that it scales naturally with the number of nodes deployed—each node contains its own processing power, so addition server processing power is not required. A disadvantage is that each node must include a processor capable of performing travel time estimates.


Page 19 of 51

To some extent, the general feasibility of these two methods for gathering travel time information has been proven. The first approach was demonstrated during FHWA-sponsored Model Deployments in San Antonio and Orlando, with toll tag transponders used to identify vehicles. The iFlorida evaluation report, in particular, includes considerable information on the use of toll tag readers for traveler information, including a list of causes of failures and analyses of penetration rates needed to support real-time travel time operations. Some form of the second approach has been used by DASH in the United States and by TomTom in Europe. The ADVANCE project in the Chicago area in the 1990s also attempted this, including analysis of needed numbers of equipped vehicles. It has also been used extensively in the form of GPS data loggers for collecting probe vehicle data.

If the evaluation team is tasked with helping to demonstrating the technical feasibility of using the system for gathering probe data that can be used to generate accurate travel times, the team would consider the measures and methods presented in Table 9 below.

Table 9. Approach for Investigating Mobility Impacts of Probe Data.


Investigate the feasibility of gathering probe data and determining travel times based on probe data

Technical ability to send probe data from the system

System

Vehicle logs

Developer testing

Access to developer testing documents

Assessing Future Potential and Overall Feasibility In addition to examining the effects of the system in its proposed state, the evaluation team will also address the potential of the system in terms of expanded use and functionality. For example:

Is the system scalable at a state-wide or national level?

Does the Mn/DOT business model support phased or partial-implementation side-by-side with the gas tax or license tab fees?

Does the Mn/DOT business model support tracking of usage by jurisdiction (i.e., to facilitate distribution of funds according to jurisdictional boundaries)?

Is the model expandable (i.e., to other VII applications or to commercial vehicle operators)?

Is the Mn/DOT system interoperable with other systems (e.g., toll systems)

What are the costs associated with full-scale deployment (versus a pilot implementation)?

How does the Mn/DOT MBUF implementation fit into the national vision for a Federal and multi-state MBUF picture?

How does the Mn/DOT approach to VII fit into the national vision for VII?

The table below presents the team’s initial thoughts on measures and possible methods for this portion of the evaluation. The issues that may arise in this section of the evaluation are difficult to specify without a precise understanding of the pilot system design, functionality, and limitations as well as understanding the pilot results in terms of costs and customer satisfaction. Thus, the future potential of the system will be defined in part by the final plans for the pilot study.

Table 10 presents measures and possible methods for this goal.


Table 10. Approach for Assessing Future Potential and Overall Feasibility.


Assess whether the system is scalable

Degree of technical functionality across state lines and at a national level

Degree of acceptance by drivers beyond pilot participants

Understanding of costs, success factors, challenges and barriers to broadening scale

Drivers

SMEs

Developers

State DOT representatives

State revenue officials

Usability testing

Interviews

Focus groups

Surveys

Population and sampling

Understanding of national issues

Impact of phased and full deployments

Assess whether the system is interoperable

Technical compatibility with other systems (e.g., toll systems)

Ability to function technically for vehicle to vehicle communications

Assess whether the system is expandable

Ability to succeed with different roadway situations

Ability to succeed with different user groups (e.g., CVO)

Page 20 of 51


Page 21 of 51

CHAPTER 4. DESIGN OF THE PILOT STUDY This chapter presents options for the design of the pilot study including criteria for site selection and sampling design. In order to carry out a successful evaluation it will be necessary to design a pilot study that is both representative of real world users and conditions and manageable and cost-effective in terms of size and schedule. The key to ensuring a valid and efficient approach is to carefully construct a study that makes the best use of what is known about the subject population and possible driving conditions to sample drivers, roadways, and situations (e.g., school zones) which will be rich in relevant observations.

Location of the Pilot Test A successful pilot requires a carefully thought-out approach that takes into account the goals of the project along with practicality and project constraints. At one extreme, Mn/DOT could conduct a statewide pilot study in which data and participants come from all over the State of Minnesota. At the other extreme, Mn/DOT could conduct the pilot study in only one city or town in Minnesota. The first approach emphasizes representativeness, but would be extremely time consuming and expensive to conduct (e.g., it would not be efficient to conduct installation or any necessary maintenance on the devices if they are spread out around the state). The latter approach emphasizes efficiency at the cost of limiting the study’s representativeness. The approach proposed by the evaluation team ignores these extremes and considers the practicalities of collecting data while utilizing a relevant and broad, yet realistic, sample of Minnesota drivers and driving situations.

There are several project goals that impact the selection of a general location for the pilot:

The project aims to obtain opinions and feedback from both the rural and urban residents of Minnesota. The reason for this is that implementing MBUF statewide will impact all Minnesota residents, and the State of Minnesota has a quite diverse population. Nearly half of the state’s population resides in the Twin Cities Metro Area. However, much of the rest of the state lives in rural areas, where opinions of MBUF will likely differ substantially from those of the state’s urban residents.

The project aims to test the impacts of congestion pricing on the travel behavior of individual travelers (e.g., overall miles traveled, time of day of travel, number of trips, etc.). Congestion is not a concern in rural areas of the state so congestion pricing will need to be tested in an urban area. In order to assess the usability and value of the traveler information application, travel time reliability should be fairly low in the selected region (or at least on select corridors) so that traveler information would be of interest to the participants.

The project aims to demonstrate the capability to distinguish miles driven by jurisdiction. This means that it is critical that the test include one or more jurisdictional boundaries that are frequently crossed (such as a county line) in order to test this capability.

The project aims to convey travel time information to motorists where travel time information already exists and additionally to demonstrate that technologies not designed to serve as probes can generate data that is accurate enough to serve as probe data. To do this, travel information data sources will need to exist in the selected region so that traveler information can be provided.


Page 22 of 51

Taking these project goals into account, Table 11 presents the benefits and drawbacks of different study locations. The focus is on counties as county lines seem a logical boundary to demonstrate the capability of the system to distinguish miles driven by jurisdiction.9

Table 11. Benefits and Drawbacks of Various Study Locations.

Type of County Benefits Drawbacks

Rural County - Provides indicator of rural opinions.

- No indication of urban opinions.

- Limited jurisdictional crossings.

- Limited availability and utility of traveler information.

- Limited opportunity for congestion pricing.

Suburban County - Jurisdictional crossings.

- Provides indicator of urban opinions.

- Availability and utility of traveler information.

- Opportunity for congestion pricing.

- No indication of rural opinions.

Urban County - Provides indicator of urban opinions.

- Availability and utility of traveler information.

- Opportunity for congestion pricing.

- No indication of rural opinions.

- Limited jurisdictional crossings.

Taking these benefits and drawbacks into consideration, the evaluation team proposes the following options:

• Option 1: Suburban County Pilot Study (One County). Focus the pilot on a suburban county as this will allow for all evaluation goals to be achieved with the exception of obtaining feedback from rural drivers. Although a statistically representative sample of rural drivers would not be possible under this option, the team proposes gathering some perceptions from this group by recruiting a group of 12-15 additional individuals from a rural county in Minnesota. These participants would also receive units and would provide input through surveys and a focus group.

• Option 2: Two County Study. If Mn/DOT is interested in obtaining a statically representative sample of rural drivers, then the evaluation team recommends focusing one study on an urban or suburban county and one study on a rural county. This option would double the sample size required as is discussed in the following chapter.

The evaluation team proposes focusing the study on the greater Twin Cities Metro Area. Its high population density and corresponding level of traffic congestion as well as the size of its population10 all

9 Note that other boundaries such as major roadways can be used instead of county lines if desired. 10 Metropolitan Council. http://www.metrocouncil.org/Census/KeyFacts/7-county.htm (Accessed May 28, 2009).

http://www.metrocouncil.org/Census/KeyFacts/7-county.htm


Page 23 of 51

make it a good choice for the pilot when taking the project goals into account. In considering where in the region to focus the study, it is important to consider commute patterns. It is desirable to select a county that will provide sufficient opportunity to obtain data on commute trips into the Metro Area that requires crossing county lines. Figure 1 provides a depiction of the commute patterns in the Twin Cities area.

As shown in the figure, among the surrounding counties, Wright County has the largest number of residents commuting into the Metro Area and would therefore make a good choice for the pilot. In fact, approximately 49 percent of the working population of the county commutes into the Metro Area. The population of Wright County is approximately 120,000 and the county covers approximately 660 square miles (136.1 persons per square mile). 11 Males and females are equally divided in the county. Approximately 80 percent of workers drive to work alone while 12.6 percent of the working population carpool. Nearly 5 percent of the population works from home. In 2000 the average commute time of residents was 29.1 minutes, the fifth longest average commute in the state.12

In the event that option 2 is pursued, it would be desirable to select a county that is located in close proximity to the Metro Area to facilitate study activities (e.g., equipment servicing). Taking this into account, Goodhue County would be good choice as it is the “least connected” rural county13 (in terms of the percentage of residents who commute into the Metro Area) located in the vicinity of the Twin Cities Metro Area. Goodhue County has a population of 45,835 and covers approximately 760 square miles (58.2 persons per square mile). Males and females are equally divided in the county.14 Approximately 80 percent of workers drive to work alone while 9.8% percent of the working population carpool. Nearly6 percent of the population works from home. In 2000 the average commute time of residents was 21.3minutes.15

11 United States Census. http://quickfacts.census.gov/qfd/states/27/27053.html. (Accessed May 28, 2009). 12 Minnesota State Demographics Center, Population Notes Minnesota, OSD-03-104, April 2003. http://www.demography.state.mn.us/DownloadFiles/CommuterPopnote.pdf (Accessed May 28, 2009). 13 Metropolitan Council. http://www.metrocouncil.org/doing_business/WMBE_SBRA/RuralCounties.pdf (Accessed June 17, 2009). 14 United States Census. http://quickfacts.census.gov/qfd/states/27/27049.html (Accessed June 16, 2009). 15 Minnesota State Demographics Center, Population Notes Minnesota, OSD-03-104, April 2003. http://www.demography.state.mn.us/DownloadFiles/CommuterPopnote.pdf (Accessed May 28, 2009).

http://quickfacts.census.gov/qfd/states/27/27053.html

http://www.demography.state.mn.us/DownloadFiles/CommuterPopnote.pdf

http://www.metrocouncil.org/doing_business/WMBE_SBRA/RuralCounties.pdf

http://quickfacts.census.gov/qfd/states/27/27049.html

http://www.demography.state.mn.us/DownloadFiles/CommuterPopnote.pdf


Figure 1. Number of Drivers Commuting from Adjacent Counties into the Metro Area.

Sampling Design When sampling an entire county it would be impossible to randomly recruit drivers to participate in the study who represent the composition of the county but who also drive frequently through specific “areas of interest” (i.e., school zones, work zones, and curves to be included in the study). Therefore, regardless of the counties that are selected for the pilot, the evaluation team proposes to conduct two parallel data collection activities:

1. County-wide Sampling. This sample would be taken from the population of the entire county and would examine all measures of focus in this study with the exception of the in-vehicle safety system, which warrants a more focused study. This county-wide sample would be used primarily to collect perception data and general driving data (e.g., miles traveled). A representative sample of drivers from the county would be selected for participation in the field study and the associated survey and focus groups/interviews.

2. Targeted Sampling. This sample would be established primarily to examine perceptions and impacts of the in-vehicle safety system. A targeted sample will also allow for analysis of the capabilities of the probe aspects of the system. The goal will be to target individuals who frequently travel through “areas of interest” to generate a large enough sample size to assess the safety applications. Areas of interest would include school zones, work zones, and curves that are included in the study (note that even if all curves and school zones in the selected county are included in the pilot, only a select number of curves and school zones may be selected for inclusion in the analysis), but would also include select major roadways for the probe study. At least one major arterial will be identified for study as well as at least one stretch of freeway that Mn/DOT currently has travel times for to allow for comparison purposes (i.e., to determine the accuracy of the data versus existing Mn/DOT data). The method of recruitment for this targeted sample will likely be a particular employer or school.

The pilot test will be used to assess the value of incorporating the MBUF application with in-vehicle signing and traveler information from probes. The intention is to evaluate this system as a whole containing three integrated applications. Therefore, a comparison of application conditions has not been proposed. Each application serves a unique purpose and a direct comparison between the three is not

Page 24 of 51


Page 25 of 51

meaningful. However, it might be interesting to compare MBUF alone versus MBUF integrated with existing technologies. This is because there may be carry-over effects associated with the integration itself. For instance, if drivers highly dislike the in-vehicle signing application, this may negatively influence their perceptions of MBUF. Regardless of the sampling approach selected, the evaluation team recommends selecting a sub-sample of drivers who will use only the MBUF system without the additional two applications.

Criteria for Selection of Sites for Targeted Sampling To identify specific locations for study of the safety applications, it will be necessary to understand the incidence of collisions associated with curves, school zones, and work zones throughout Minnesota, and if possible, to understand surrogate measures at these locations such as average speeds. Once sites are selected it will be important to understand the roadway characteristics (e.g., traffic volume, speed limit) that define the driving environment at the selected locations.

General requirements for site selection are as follows:

Work zones selected for study shall be located on major commuter routes to increase the expected sample size (i.e., study participants are more likely to drive on the higher volume roadways in the region) and should be locations where speeding is expected or already documented.

School zones selected for study shall be on major commuter routes to increase the expected sample size (i.e., study participants are more likely to drive on the higher volume roadways in the region).

Curves selected for study shall have a high incidence of speeding and/or a poor safety record.

High-Level Criteria for Participant Selection General requirements for participant selection are presented in Table 12.

Table 12. High-Level Requirements for Participants.

Both Samples County-Wide Sample Targeted Sample

Drivers should be representative of the drivers across the county in terms of age and gender.

Drivers shall be the primary driver of the equipped study vehicle and approximately 95 percent of the miles driven shall be by primary driver.

Participants should currently drive during peak congestion periods and commute patterns shall be appropriate for congestion pricing.

Commute times shall be long enough that drivers will likely be sensitive to mileage and congestion pricing.

Participants shall frequently travel through “points of interest” including the school zone(s), work zone(s), and curve(s) selected for study in the pilot.

The evaluation team will work with Mn/DOT to further refine or identify population segments of interest beyond those indicated here, such as:

Age. For example, do older, middle-aged, and younger drivers differ?

Experience (years since obtaining license). For example, do novice drivers differ from experienced drivers?

Gender (male, female).


Page 26 of 51

Annual Mileage. For example, do “heavy” drivers (greater than10,000 miles per year) differ from “light” drivers (less than 5,000 miles per year)?

Type of Vehicle (compact car, sedan, sport utility vehicle, light truck).

Type of Power (hybrid, gasoline).

Type of Driver. Are there categories of drivers that differ significantly from others (e.g., commuters, commercial drivers, local area drivers)?

Level of technology sophistication or comfort with technology (e.g., early and late adopters).

Level of risk acceptance. E.g., “risk-aversive” and “risk-accepting” drivers.

It should be noted that it will not likely be possible to include all of the target driver segments in the full pilot. First, it might be difficult to identify a large enough sample of some of these population segments who are willing to participate in the pilot, and it may be difficult and time-consuming to pre-select for some of these factors. Second, there is a great deal of correlation among some of these variables. For example, older drivers might be the most experienced and the least supportive of new policies or technologies. Third, the more segments that are included in the study, the larger the overall sample size required and the greater the resources, time, and cost required to complete the study.

As a result, the selection of these segments will need to be carefully considered. It will probably be more possible to look at a variety of segments in the statewide survey which is relatively low-cost compared to the county-wide pilot study. Focus groups also may be useful to drill down and understand the perspectives of a particular population. These issues will be considered further in the following chapter describing sample size requirements.


Page 27 of 51

CHAPTER 5. DETERMINING SAMPLE SIZE FOR THE PILOT A major consideration for any evaluation is to ensure that findings are representative of the project stakeholders and that results are statistically significant and not simply a matter of the normal variability in our complex transportation system. This chapter discusses the required sample size to ensure that the behaviors and perceptions reported by pilot participants are representative of a wider group of stakeholders. It documents the number of drivers that will need to be recruited to determine statistically significant impacts on vehicle miles driven and to detect reductions in speeds in critical safety areas. Finally, it addresses an approach and the number of vehicles that would be required to facilitate a small test of vehicle probes.

Sample Size for Surveys to Assess Driver Perceptions In conducting surveys it is important that the results be representative of the population being sampled – in this case the residents of Wright County (and potentially Goodhue County if Site Selection Option #2 is implemented). As shown in Table 13, in order to obtain a 95 percent confidence level with a confidence interval of ±5, approximately 380 respondents would be required for each county. Despite the large difference in county populations, these numbers are similar because as populations get very large there is very little to be gained from increasing sample size. Even assessing statewide perspective will require approximately the same number of participants.

Thus to recruit participants for the driver pilot program, and recognizing that some participants may drop out of the test or experience equipment problems that preclude the provision of data, it is recommended that approximately 20 additional participants be recruited for each county, bringing the total number of drivers to 400 (or 800 if both counties are included in the pilot). Additionally, at least 400 drivers will be surveyed in a statewide survey but will not participate in the pilot program. This is consistent with the way that Mn/DOT conducts their Omnibus Survey, phone surveying 400 Metro Area residents and 400 Greater Minnesota residents.16

Table 13. Required Sample Sizes for County-wide Surveys, 95% Confidence Level and Confidence Interval of +/-5%.

County Population1

Sample Size2

Confidence Level = 95 Percent /

Confidence Interval = ±5

Wright County 120,599 384

Goodhue County 45,835 381

1Population Division, U.S. Census Bureau. Annual Estimates of the Population for Counties of Minnesota: April 1, 2000 to July 1, 2007 (CO-EST2007-01-27). 2Sample sizes were calculated using the sample size calculator at: www.surveysystem.com/sscalc.htm#two (accessed May 20, 2009)

16 Minnesota Department of Transportation. (2007). Minnesota Department of Transportation Statewide Omnibus Study 2006. Prepared for Mn/DOT Market Research by Market Line Research, Inc.

http://www.surveysystem.com/sscalc.htm#two


Page 28 of 51

It should be noted that while it will be possible to ascertain representative opinions and reported behaviors of the population of each county as a whole, it may not be possible to obtain representative responses for individual segments (e.g., by age group) within a given county. Segmenting the population would require a larger sample size as that same number of participants (380) would be required of each unique population segment. For instance, if Mn/DOT decides that it is important to compare older and younger drivers’ opinions of MBUF, it might be necessary to survey approximately 760 drivers per county (i.e., approximately 380 older drivers and 380 younger drivers). In this way, the number of segments that are investigated will cause the sample size to grow exponentially. For example, fully investigating a simple 3x3 experimental design of 9 population segments would require about 3,400 survey respondents. The following table provides an example of this. In the case of the survey, we may be interested in how drivers of varying ages feel about MBUF and whether these feelings are related to the type of vehicle they drive.

Clearly, such large samples are unrealistic with respect to equipping pilot vehicles. Assuming a pilot vehicle study of about 400 within a county, the evaluation team will need to rely on detecting only the largest effects and assessing patterns and trends with respect to different driver characteristics or segments. However, Mn/DOT may want to consider a broader sample for conducting the statewide survey of drivers not participating in the pilot program. However, increasing the segments to be considered will increase the cost of the survey in terms of the time to recruit and pre-screen respondents and to actually conduct the survey.

Beyond ensuring a representative and random sample for this pilot test, it is also important to have sufficient statistical power to facilitate the comparison of survey results in the before and after cases. That is, the evaluation team will compare behavior and perceptions between the Baseline (system “off”) and Deployment (system “on”) periods. Determining statistically significant changes in these responses necessitates different sample sizes than were required to ensure representation. For example, Table 14 indicates the number of subjects needed to make a significant comparison of the results of a before and after survey question that investigates whether or not drivers like the MBUF concept. This is an example of the type of preference questions that might be asked of survey respondents. The actual process of survey development will require thoughtful question construction and a sophisticated understanding of the population and the timing of the surveys.

Table 14. Example Sample Sizes for Survey Comparisons.

Acceptance (Percentage of Drivers Who Like the MBUF

Concept) Power

Required Sample

Size Before After 60% 75% 0.80 61 60% 70% 0.80 142 60% 65% 0.80 583

2 Sample sizes were calculated using the sample size calculator at http://www.dssresearch.com/toolkit/sscalc/size_p2.asp

In this scenario 40 percent of participants report that they do not like the MBUF concept before using the system (only 60 percent like the concept). If it is expected that after using the system, this value will increase by between 5 and 15 percentage points, then the required sample size would be between 61 and 583. The smaller the change that might be detected, the larger the number of required participants will be. The size of the expected or desired improvement will determine how big the sample will need to be in order to detect a significant difference. Given that the evaluation team has thus far recommended that the pilot driver sample size be approximately 400 participants (or 800 for two counties) to ensure a representative sample, the sample may not be large enough to reveal very small changes in driver

http://www.dssresearch.com/toolkit/sscalc/size_p2.asp


perceptions. To observe small changes in driver perceptions or opinions might require closer to 600 vehicles.

Sample Size for Assessing Safety Benefits The number of collisions on a roadway segment is the most direct measure of safety. However, looking at impacts on the number of collisions can be difficult to assess. Crashes are rare events and in many cases it is not possible to detect a statistical reduction in crashes within a reasonable amount of time. That is the case here, where even amongst some of the most dangerous curves in the State of Minnesota, there are typically fewer than one or two crashes per year.17 School zones are perhaps even less likely to have crashes due to the low posted speeds and/or the presence of enforcement.

Nonetheless, crashes may occur during the course of the evaluation. If they do, the evaluation team will study any crash data to determine the role of the system in the crash—either as ineffective in slowing the driver, as “not preventable” in the context of the system, or as evidence of the need for additional Mn/DOT system safety features. This data, while not useful from a quantitative measurement perspective, may still prove to be quite rich.

In lieu of statistically significant collision data, the evaluation team will look to surrogate safety measures – specifically, before and after changes in driver speeds through critical safety zones and acceleration/deceleration behavior. This will require high-quality data about the

Vehicle and driver performance (e.g., speed)

Environment (e.g., GPS data about the location of a zone to determine when the vehicle enters a zone)

System performance (e.g., when a warning is given to the driver)

The evaluation will seek to improve the probability of detecting statistically significant changes by employing the targeted recruitment of a sub-set of participants who regularly travel through one or more of these safety zones (e.g. work zone, dangerous curve, or school zone). Table 15 provides insights into how many subjects would need to be recruited for this group to determine the effect of in-vehicle signing in a typical school zone on driving speed.

Table 15. Required Sample Sizes for Assessing Safety Benefits of In-Vehicle Signing Depending on Average Speed and Speed Variability.

Speed Range - 95% of all participant

vehicles

Expected Reduction in Mean Speed

Required Sample Size1

20 - 30 mph 2 mph 198 5 mph 33 10 mph 10

1 Calculated using one-sample (paired test) calculator at calculator at: www.stat.uiowa.edu/~rlenth/Power/ (accessed May 20, 2009)

Page 29 of 51

17 An analysis of the most dangerous 50 curves in each of 4 counties in Minnesota showed that the average number of crashes per curve was less than 0.5 per year. Approximately half of the curves had no crashes over a 5-year period while only one or two curves in each county had an average of one crash per year.

http://www.stat.uiowa.edu/%7Erlenth/Power/


Page 30 of 51

ity) is ing

Anderson et al (1997) estimated that the probability of a pedestrian dying in a collision increased six-fold from about 10 percent to about 60 percent when vehicle impact speed increased from 23 mph to 28 mph.18 Thus, being able to detect a 5 mph change is a good starting point in determining the type of change that must be detected in this evaluation. However, the range of initial speeds (their variabilalso important to consider in detecting a change. If the speeds of 95 percent of all participants travelthrough the school zone ranged from 20 to 30 mph (assuming a speed zone limit of 25mph) and this group (on average) reduced their mean speeds by 5 mph, then 33 drivers would be required to provide confidence that the observed reduction was statistically unlikely to be the result of random variance (and therefore actually due to the system). However, calculations of sample size that result in small samples may be oriented to labs where variability in the data would be expected to be less than in a field study. Therefore using only very small numbers of drivers in the target group could be risky (it may be difficult to detect an effect such as a reduction in speed). Given the range of values presented here, the evaluation team is recommending the recruitment of 50 vehicles for the targeted group.

Sample Size for Assessing Effects of MBUF on Driving Behavior In addition to assessing user acceptance and behavior related to safety, it is expected that the system will impact drivers’ travel choices. In particular, it is expected that drivers may decrease the number of miles they drive or the degree to which they “chain” trips together. Unfortunately, there is so much variability in vehicle miles traveled (VMT) per person that it is difficult to separately identify any changes caused by MBUF from this underlying variability. Minnesota’s mean VMT per capita in 2001 was reported as 14,507 19 However, it is difficult to assess the variability among drivers from year to year and between drivers. It may be possible to assess such behavioral changes by comparing the before and after VMT data of the project participants. Table demonstrates the number of participants needed in order to detect significant effects of MBUF using this approach of comparing the before and after experiences of project participants. One central assumption in these calculations is the underlying variability in VMT for each individual from year to year – something for which very little (if any) data currently exist. However, if it is reasonably assumed that this variability will range from +/- 2000 miles per year to +/- 5000 miles per year for any given driver, then the test should be able to detect quite small changes in VMT with the numbers of participants (400 per county) already identified to support a representative sample size for the survey effort.

Table 6. Example Sample Sizes for Assessing the Effects of MBUF on Mileage.

Change in Mileage (Average = 14, 507) Percentage Reduction Variability of VMT

(95% of all participants) Sample Size

-290 2% 10507 - 18507 5975 -725 5% 10508 - 18507 968 -1451 10% 10509 - 18507 241

-290 2% 12507 - 16507 1496 -725 5% 12508 - 16507 241 -1451 10% 12509 - 16507 62

18 Anderson, R.W.G., McLean, A.J., Farmer, M.J.B., Lee, B.H., & Brooks, C.G. (1997). Vehicle travel speeds and the incidence of fatal pedestrian crashes. Accident Analysis and Prevention 29 (5), 667-674. 19 Derived from US DOT’s 2001 National Household Travel Survey (NHTS) on-line analysis tool at http://nhts.ornl.gov/tools.shtml (accessed May 20, 2009).

http://nhts.ornl.gov/tools.shtml


Page 31 of 51

The recommended full number of pilot participants (400) should be sufficient to detect changes in VMT that are modest (e.g., a 5 percent decline in VMT that ranges between 12,000 and 16, 000). If there is a great deal of variability among the pilot program drivers, it may be difficult to detect changes in VMT. However, in order to support these calculations and determine these impacts, it will also be necessary to control and account for the underlying weekly, monthly, and seasonal variation. Unless participants are enrolled for multiple years (infeasible), a control group must be established. Two ways to accomplish this were discussed earlier in Chapter 4 (with the recruitment of additional “baseline” participants and the use of staggered participant start dates).

Sample Size for Probe Data Early in the development of the con-ops for this pilot test it was recognized that using the pilot vehicles as probes would not likely support wide-scale determination of travel times. However, it may be possible to leverage the proposed targeted sample group being established for the safety study to demonstrate the efficacy of vehicle probes on a limited section of roadway(s). For example, by selecting a small group of drivers associated with a single origin or destination (such as a school or major employer) it may be possible to calculate travel times during peak periods for the roadways immediately surrounding the origin / destination. An evaluation of the European Union’s OPTIS Field Operational Test suggested that the levels of market penetration presented in Table 16 are sufficient.

Table 16. Levels of Market Penetration Required to Achieve Various Goals.

Goal Required Market Penetration

To provide 5 to 10 minute updates of travel times on major arterials under normal conditions

0.5%

To provide minute by minute updates of travel times on major arterials under normal conditions

3%

To detect major incidents 5-10%

Taking a simple assumption that the average peak period flow of an arterial surrounding a targeted origin or destination is 1,500 vehicles per hour (vph) and assuming that all equipped participants arrive during the peak period, Table 18 presents the number of vehicles that would need to be operating on the targeted arterial as vehicle probes.

Table 17. Number of Vehicles Needed to Achieve Various Goals.

Goal Required Market Penetration

To provide 5 to 10 minute updates of travel times on major arterials under normal conditions

8 vehicles

To provide minute by minute updates of travel times on major arterials under normal conditions

45 vehicles

To detect major incidents 78-150 vehicles


Page 32 of 51

Based on these calculations, it is reasonable to assume that a simplified vehicle probe demonstration maybe possible using the 50 vehicles recruited for the targeted safety study.20

20 Gunnar, Lind (2006). Optimised Traffic in Sweden, Draft Synthesis Report. Prepared for the European Commission Directorate General, Energy and Transport, September 2006.


Page 33 of 51

CHAPTER 6. OVERVIEW OF DATA COLLECTION To address the goals of the evaluation as presented in Chapter 2, the evaluation team will implement a combination of field data collection, surveys, focus groups and interviews, and other methods. This chapter presents some additional details on how data will be collected in support of the evaluation.

The exact schedule for data collection will be finalized based on the stability of the system applications. Figure 2 shows an example schedule.

2010 2011

JAN

FEB

MA

R

APR

MA

Y

JUN

JUL

AU

G

SEP

OC

T

NO

V

DE

C

JAN

FEB

MA

R

APR

MA

Y

JUN

Pre-Deployment Deployment Post- Deployment

Field Data Collection

Field Driver Survey

Statewide Survey

Focus Groups & Interviews

Figure 2. Example Evaluation Timeline.

The evaluation will be broken into Pre-Deployment, Deployment, and Post-Deployment Phases. For each aspect of the evaluation shown in the figure above, certain activities will be required, such as participant recruitment and training, installation of the system, and development of surveys and interview scripts. For the purposes of this document, some assumptions have been made about the system and its deployment. These will be further specified as system development and deployment are finalized.

At the end of this chapter, we present a description of how to approach the payment and incentive scheme for participant recruitment and retention over the various deployment phases.

Pre-Deployment There will be several pre-deployment activities. These include:

Usability Testing.

Participant Recruitment.

Participant Training.

System Installation and Verification.

UUssaabbiilliittyy TTeessttiinngg The majority of usability testing will occur during development and will fall within the purview of the developer(s). There is much that can be learned about the design of a system using simple laboratory and simulation techniques prior to field testing. While laboratory and simulation tests lack some representativeness and may not fully address all issues, they are important first steps in designing a good pilot system.


Page 34 of 51

In addition to evaluating the usability of the system itself, the developer(s) will need to evaluate the usability of any other interfaces that will require interaction with the pilot participants. For example, the participants may need to interface with a website in order to learn how to install the system or to view their account balance. If this is the case, this website will serve an important component of the system as both an instructional resource for participants (e.g., for installation of the system) and a tool (e.g., for checking MBUF balances) and will therefore require usability testing.

Some of the goals of usability testing will be to assess:

Comprehension of the system (understanding of overall functionality, system limitations, and interface alerts and information).

Learning (degree of training required to install and operate the system).

Utility of the system (perceived usefulness).

Confidence in the system (perceived accuracy, reliability, fairness, ability to evade, ability to repudiate, and perceived invasion of privacy).

Quality of the system (ease of use, alert quality, frequency, and timing, quality of messages).

Impacts of the system (on speed and braking, route choice, distraction, stress, perceived safety and mobility).

Value of the system and overall acceptance (desire to use the system, willingness to participate in a pilot program, barriers to acceptance at the outset).

PPaarrttiicciippaanntt RReeccrruuiittmmeenntt Volunteer participants will be recruited for participation in the field study and in the statewide survey. Participants will be required to sign an IRB-approved Informed Consent letter that will have the following key components:

1. Overview of the Mn/DOT system and applications.

2. Description of the evaluation.

3. Responsibilities of the participant.

4. Responsibilities of the evaluation team.

5. Confidentiality policy.

6. Withdrawal procedures.

Following the informed consent process, participants will be answer a demographic questionnaire. This questionnaire will contain questions about the participant, the vehicle(s) they will be using during the pilot, and general travel behavior. For example:

Name, address, phone number, e-mail address, best methods of contact.

Date of birth, gender, education level.

Vehicle make and model, year of purchase, current mileage, mileage at purchase.

Other drivers in household, frequency of vehicle use.

Work schedule, average commute (origin, destination, mileage, time of day).

Information on other common trips (e.g., driving for business, driving kids to school, driving to the gym) including trip origin, destination, and frequency.


Page 35 of 51

PPaarrttiicciippaanntt TTrraaiinniinngg Following the initial introduction to the evaluation study, participants will be instructed on a) how to install and use the system and b) what their role as a participant will involve.

For the former, the evaluation team suggests that participants receive only as much training as would be expected in the actual deployment of the system. It is assumed that in actual deployment, Mn/DOT would have drivers receive and install the system themselves with the option of driving to an installation location. Thus, this approach is recommended for the pilot as this will allow the evaluation team to evaluate the extent of the instructional material needed in the future to facilitate proper installation by the general public. It is assumed that the developer will provide instructional materials. As there is a high likelihood that study volunteers will be more technologically savvy than some potential system users, the results will have to be assessed in that context.

The participants will also need to understand their role as a participant, the tasks they will need to perform, and the deadlines they will be asked to meet, as well as the responsibilities of the research team. To accomplish this, the evaluation team expects to conduct a briefing session with participants as well as to set up a project website that will serve as a central source of information for participants about the project. Participants will be able to access the site for information and instructions; to read frequently asked questions (FAQs); to enter information, such as a change of address and maintenance requests; and to ask questions. The evaluation team may also develop and post a webcast briefing for participants on their duties during the study which participants can watch when convenient, including:

System installation.

Survey participation.

When and how to turn the system on.

Where to find instructional material about the operation of the system applications (MBUF, safety, and traveler information).21

Incident reporting.

Obtaining help.

Using the MBUF website.22

SSyysstteemm IInnssttaallllaattiioonn aanndd VVeerriiffiiccaattiioonn If drivers install the system themselves, it will be important to verify that the installation has been done properly for the purposes of data collection. Additionally, if secondary data collection devices are needed, such as GPS or cameras, the evaluation team will need hands-on access to the vehicle. Thus, it is expected that the evaluation team or deployment team will need to hold garage events, during which drivers will bring their vehicles to a garage for system installation verification and for installation of secondary devices if required.

It is assumed that the developer(s) or system integrator will have developed instructional materials to assist drivers in installing the device in the vehicle. After the driver has installed the device, the driver

21 The evaluation team makes the assumption that this material (produced by the developer(s) and system integrator) will be in written, visual, or audio form and will be simple and brief. 22 The project website may also be used as a driver tool for examining MBUF data such as mileage over a given period and by time-of-day and day-of-week, fees charged, and a comparison of flat pricing versus congestion pricing versus fuel tax pricing.


Page 36 of 51

will bring in the vehicle for an installation check. Here, the developer will work with SAIC to complete “acceptance testing” to assure that each device is installed in a consistent fashion across different vehicle makes and models and that the system is functioning similarly for all drivers

Initially, the system will appear to drivers to be in the off position. However, the system will actually be capable of recording driver behavior and vehicle performance. The developer will provide a mechanism for turning the system on at a later date (e.g., they might e-mail participants a system authorization code).

Deployment After participants have been recruited, and the system has been installed, the Deployment Phase of the Evaluation will begin, which will include three phases:

Baseline

Phase I (Early Deployment)

Phase II (Late Deployment)

After participants have been recruited, their driving behavior will be recorded for a period of 3 months. The purpose of this Baseline period will be to document the typical driving behavior of each participant so that changes due to the Mn/DOT system can be assessed after the system is turned on. It is possible that drivers will drive differently after the system is installed than they normally would due to experimental bias. For instance, drivers might drive slower or be more cautious, because they feel that they are being watched or evaluated. However, a Baseline of 3 months should be sufficient for driving behavior to return to normal. During this period, the evaluation team will

Collect baseline field data.

Conduct a baseline driver survey with study participants.

Conduct baseline focus groups and interviews with drivers and stakeholders.

Subsequently, Phase I (Early Deployment) will begin. After collecting 3 months of Baseline data, the system in each vehicle will be turned from “off” to “on.” At this point, drivers will have access to the MBUF and safety and traveler information applications. The participants will use the project website as a “help system” when needed.

The goal of this early data collection phase is to allow drivers to acclimate to the system and to overcome any strong initial experimental bias associated with being observed during the study. It also allows the evaluation team to assess system learning focusing primarily on:

Compliance with using the system and participating in the study.

Frequency and type of help requests.

Accuracy and reliability of system performance (e.g., false alarms).

System calibration (e.g., if available, how do drivers customize the system such as setting the volume and tone of an alert).

Initial driving behavior with respect to the system (e.g., speed adjustment in response to an alert, traveler information requests, access of MBUF information in-vehicle and on-line).

During this period, the evaluation team will

Collect field data with the system turned on to observe initial system performance and driver behavior

Conduct focus groups and interviews with drivers and stakeholders to assess initial impressions


Page 37 of 51

In Phase II (Late Deployment), the evaluation team will assume that the novelty of the system has worn-off and that drivers are performing with less awareness of the system and experimental observation. While it is unlikely this bias will disappear entirely, the team expects that drivers will have integrated the system into the vehicle to some degree and see it as just another in-vehicle device. It is in this phase that the evaluation team will focus mostly on the behavior or success of the system and the impact of the system on driver performance. During this period, the evaluation team will:

Collect field data.

Conduct a second driver survey with study participants.

Conduct focus groups and interviews with drivers and stakeholders.

Conduct a statewide survey.

A challenge associated with data collection is selecting the proper timeframe and duration of data collection. First it will be important to consider how certain seasonal and environmental characteristics may significantly impact the logistics of the field study. For instance, if the Baseline period occurs between January and April and the deployment period occurs during the summer months, there may be driving differences in commute patterns due to weather, vacations and holidays, school schedules (i.e., driver behavior through school zones will change when school is out of session), and Mn/DOT construction/maintenance work (i.e., work zone activity typically increases during the warmer weather). Also severe winter weather can significantly affect driving behavior and the number of trips taken. The evaluation team will need to gather information about these factors, particularly in the selected study locations, prior to the deployment.

Next it will be important to assure that the Baseline and “after” data differ only in the operation of the Mn/DOT system and not in respect to seasonal or time variations. In order to account for seasonal variations and other external factors that could affect travel behavior (e.g., changing gas prices), the evaluation team proposes one of two options:

Option 1: To establish a small control group to serve throughout the entire duration of the pilot and collecting data from the control subjects in parallel with the study participants.

Option 2: To divide the participants into four groups and implement a staggered approach to the study as shown in Figure 3 whereby each group serves a control for the subsequent group.

There are benefits and drawbacks to each option are presented in Table 18. Table 18. Benefits and Drawbacks to Options for Controlling for Seasonal Variation.

Variable Option 1 – Control Group Option 2 – Staggered Approach

Timeline The project can be completed more quickly (following a 3-month baseline, data can be collected over a 6- or 9-month period for all participants).

Requires a longer study length (for example, if data is collected for 6 months following a 3-month baseline, the study length would be 18 months as compared to 9 months under option 1).

Interaction with the participants

All activities involving interaction between participants and the team will occur simultaneously.

All activities involving interaction between participants and the team will need to occur four separate times.

Sample size / cost implications

An additional 10-20 participants will need to be recruited for the control group.

No impact on sample size but additional project resources will be required to accommodate the additional interactions with participants.

Effectiveness at achieving Chance that the data resulting from this small Very effective - 100 participants will be acting as


Page 38 of 51

Variable Option 1 – Control Group Option 2 – Staggered Approach

goal of controlling for seasonal variability

control group may not be sufficient to explain all findings.

a control to the other 300 participants at any given time.

Figure 3 depicts the “Staggered Approach” of Option 2. While such a design may mean that different groups of participants have different experiences with respect to the Baseline and Deployment Phases I and II, there will always be a control in place to account for outside influences:

Groups B and C provide the baseline for A. Groups C and D provide the baseline for B. Groups D and A provide the baseline for C (with A providing data on the previous May through

June). Groups A and B provide the baseline for D (from the previous May through October).

MA

Y

JUN

JUL

AU

G

SEP

OC

T

NO

V

DE

C

JAN

FEB

MA

R

APR

MA

Y

JUN

JUL

AU

G

SEPT

OC

T

A Baseline / Control Phases I / II

B Baseline / Control Phases I / II

C Baseline / Control Phases I / II

D Baseline / Control Phases I / II

Figure 3. Proposed Option 2 - Staggered Approach to Control for Seasonal Variation in VMT.

Participant Payment Approach Selecting an approach to recruit, pay, and retain participation by drivers is a very important aspect of the pilot study. There are three important aspects of this decision:

1. The business model that the state of Minnesota chooses to implement with respect to fees levied on drivers.

2. The impact of driver reimbursement for “extra” expenditures as a result of their participation, specifically paying a mileage based fee in addition to the regular gas tax at the pump.

3. The need to incentivize retention of participants over the long term course of the pilot study.

The options for each of these factors are described below, followed by recommendations for developing an effective Participant Payment Approach.

BBuussiinneessss MMooddeell Minnesota might consider two different business models for a statewide MBUF program:

Replace the gas tax.

Supplement the gas tax.

In the first case, the gas tax would be fully replaced by a mileage-based fee. For example, drivers would pay for gas, minus any state tax, and the state would collect the additional revenue through some other


Page 39 of 51

mechanism such as manual reporting of mileage at the time of registration renewal. These fees are likely to be substantial to account for the loss (and current shortfall) of the state gas tax.

Conversely, the gas tax might be supplemented by a mileage-based fee. Thus, drivers would continue to pay the gas tax at the pump and would also report mileage to the state at pre-determined point(s) throughout the year (either manually or through the opt-in MBUF in-vehicle system). These fees are likely to be smaller since the gas tax would still be in place. Hypothetically, such a fee might range from a tenth of a cent to 5 cents per mile as indicated in Table 19.

Table 19. Example Mileage Based User Fee Based on Miles Driven Annually.

Hypothetical Supplemental Fee

(per mile)

Annual Cost Based on Average Annual Mileage for Minnesota Residents

of 14,507

$0.001 $14.51

$0.005 $72.54

$0.01 $145.07

$0.02 $290.14

Drivers who choose to forego manual mileage reporting would opt-in to use the MBUF in-vehicle system. These drivers would pay a discounted rate during peak congested times, thus operating under a different fee schedule than those who manually report mileage.

This business model is important as the size of the annual fee would impact customer acceptance and may also impact driver behavior.

RReeiimmbbuurrsseemmeenntt Participants should not be required to pay as part of their participation in this study. Three ways of dealing with this are:

To reimburse the gas tax paid by participants (they pay the fee instead).

To not charge a fee to pilot participants at all (they pay the gas tax as they normally would).

To reimburse the fee that pilot participants pay (they pay the gas tax as they normally would but also pay a mileage-fee that they would be reimbursed for).

It would be challenging to reimburse drivers for the gas tax. This would require detailed tracking of expenditures and state fees and taxes, and would overly burden the pilot and the participants. It would also be very prone to error. For this reason, the evaluation team would not recommend reimbursing the gas tax (the first option).

Since pilot participants will continue to pay the gas tax as will the rest of Minnesota drivers, they should not incur a fee unless they are reimbursed for that fee. However, one aspect of this study is the investigation of whether MBUF impacts driver behavior, specifically the number of miles driven overall – or at peak congestion times, as might be dictated by congestion pricing. If participants do not pay a fee during the pilot or are fully reimbursed, there is no incentive for them to adjust the number of miles that they drive. This issue of mileage behavior is important to understanding the potential impact of the MBUF application on congestion, the environment, and the true revenue potential for the MBUF model in the future.


Page 40 of 51

IInncceennttiivviizzaattiioonn Participants will be asked to participate in this study for approximately one year which is a significant length of time. Additionally, the study could be perceived as intrusive, requiring access to their private vehicle and requiring them to perform certain actions on an on-going basis. For this reason, we foresee a need to incentivize their participation; that is, not pay them in one lump sum, but instead to tie payment to the performance of certain tasks. It also involves a graduated payment scheme in which the longer that a participant stays in the study, the greater the payoff. The largest payments will only become available late in the study.

The following table shows how such a payment plan might look. Table 20. Example Incentive Schedule.

Activity Percentage of Total Payment

Sample Total Payment Possible

$300 $400 $500

Install System 10% $30 $40 $50

Survey #1 15% $45 $60 $75

Maintenance Check 10% $30 $40 $50

Survey # 2 15% $45 $60 $75

Survey # 3 20% $60 $80 $100

Uninstall System 30% $90 $120 $150

Under this incentive approach, participants would receive 50 percent of payment at the end of the study. Participants might also be offered payment for participation in additional activities, such as focus groups or interviews available on a first-come first-serve basis and for additional payment.

RReeccoommmmeennddeedd AApppprrooaacchh Looking at the issues (Mn/DOT’s business model, the impact of reimbursement, and incentivization), the evaluation team envisions two options for paying participants. In both cases the participants would pay the gas tax as usual and the participants would receive payment according to some type of incentivization schedule like that outlined in Table 20.

Option 1. No Fee. In this case, participants could track their potential fee but would not actually be charged. Payment would likely be based on the sample size, budget, and attractiveness of the study to participants. The consequences of this option, however, are that drivers’ perceptions and behaviors may not be driven by the financial aspects of the MBUF. It will be a fee in theory only, and thus there is not real motivation for them to change their behavior, especially with respect to miles driven.

Option 2. Fee Reimbursed. In this case, we propose charging drivers a fee that will be taken out of their overall payment. Thus, they may be motivated to keep their fee (and mileage) low. This will provide conservative estimates of how much drivers might change their behavior due to MBUF. In order to promote a sense of fairness between participants (who will by design each drive a different number of miles from the next participant), the evaluation team might establish a fee range, guaranteeing all participants a minimum payment, but allowing participants to receive a greater fee (reimbursement) depending on how much they drive. For instance, if the total participant incentivized payment is $300, and the fee range is capped at $100, all participants will be assured of being paid $200. The more miles


Page 41 of 51

they drive (especially during peak hours), the larger the fee assessed (up to $100) and the smaller the overall payment.

Table 21. Benefits and Drawbacks of Two Options for Paying Pilot Participants.

Option Benefits Drawbacks

1. No Fee Consistent with either Mn/DOT business model by showing participants what their fee would be.

Reimbursement is not necessary.

Fee itself is likely to lose meaning such that driver perception of the fee might also not be meaningful.

In particular, it is unlikely that a change in driver mileage behavior would be seen as drivers would not be motivated to adjust their mileage to reduce their fee.

2. Fee Reimbursed Consistent with Mn/DOT’s supplemental fee business model, which would be a nominal amount in addition to the gas tax.

Drivers are more likely to be motivated to minimize their fees and optimize payment.

Participants are paid different amounts which could be perceived as unfair.

Paying participants to keep their mileage low is representative of the real MBUF scenario; however, it is not the same as merely assessing a fee.

SAIC recommends the second option as it would provide for much more realistic and richer data collection opportunities. The precise fee and payment calculations will need to be determined based on the potential fee structure set by MnDOT and by the budget for the evaluation. However, a challenge will be assuring that the fee cap and payment are well balanced, so that the minimum payment is not too high leading participants to be indifferent to the cost of their mileage fees. For instance, if participants were guaranteed $500 and the fee cap was $50 (or 10 percent extra payment), participants might be less inclined to reduce their mileage than in the first scenario where the fee cap of $100 is 33% of their potential payment. These options will be discussed with Mn/DOT and decisions will be made during Phase II.


Page 42 of 51

CHAPTER 7. SURVEY CONTENT AND ADMINISTRATION Surveys will be used to assess driver perception related to acceptance, customer satisfaction, and usability. The first survey will be used to assess the perceptions amongst drivers in the County-Wide Sample field study, and it will be administered twice. The second survey will be administered across the state to assess scalability of the findings beyond the county or counties examined in the field study.

County-Wide Driver Survey Drivers participating in the Field Study will also participate in a driver survey administered twice during the study – once to assess their initial opinions regarding the device and once during Phase II or later deployment. The goals of the survey will be to assess perceptions regarding:

Opportunities for evasion and tampering.

Comprehension of the system.

Utility of the system.

Confidence in the system.

Quality of the system.

Reliability of the system (uptime).

Accuracy of system.

Impact of the system on their driving behavior on approach to zones of interest (e.g., braking).

Impact of the system on their choice of routes (e.g., choice of an alternate route in the case of the traveler information component).

Impact of the system on their overall travel choices (e.g., time of day of travel, trip chaining, overall mileage).

The goals of this first survey will be to assess different attitudes toward and expectations about the system and its three applications as well as to gather information on typical driving behavior. The goals of the later survey will be to assess changes in perceptions and expectations, changes in driving behavior, and system usability (as the drivers will have had sufficient experience with the system at this point in the study). Table 22 depicts these goals.


Page 43 of 51

Table 22. Topics to be Covered in the Driver Surveys.

Goals

Surv

ey 1

Surv

ey 2

Topics

Driving behavior ♦ ♦ Route choice, trip types and planning, mileage, speeding

Attitudes and expectations: MBUF ♦ ♦ Fairness, privacy, opportunities for evasion and tampering

Attitudes and expectations: Traveler information ♦ ♦ Utility

Attitudes and expectations: Safety warnings ♦ ♦ Accuracy, reliability, importance

System usability ♦ Comprehension, ease of use, confidence, usefulness, quality

The survey will likely be administered through the project website. Each participant will log-in to answer survey questions. Prior to this session, they will already have provided demographic data such as age, sex, vehicle type, etc. However, they will be given a change to update this information prior to taking the survey. The survey will be adaptive, asking follow-up questions which are dependent on the previous responses. Examples of survey questions follow:

The government will be replacing the fuel tax with a fee associated with the amount of miles traveled by each vehicle. In the future, the gas tax won’t be associated with the number of gallons you pump, but with the number of miles you drive. Are you in favor of this?

o 5 - Strongly in favor

o 4

o 3

o 2

o 1 – Strongly opposed

Why do you support this? Why are you opposed?

How long is your typical commute to work?

o 0-15 minutes

o 15-30 minutes

o 30-45 minutes

o 45-60 minutes

o Between 1 and 2 hours

o Greater than 2 hours

o My commute varies immensely

Is your commute congested?


Page 44 of 51

o Yes, very congested. Sometimes my commute doubles in length.

o Yes, my trip varies by ____minutes each day.

o Yes, sometimes.

o No, I rarely face congestion.

The government is creating a system that would provide you with information about current travel times and expected delays. Would such a traveler information system be valuable to you in reducing your travel time?

o Yes, I could adjust my route to avoid congestion.

o Yes, I could adjust my schedule to leave earlier or later.

o No, I would not find this valuable.

Statewide Survey The evaluation team will conduct a statewide survey of opinions about the proposed system. This survey will help determine where study results are in the context of opinions from across the state.

The statewide survey will have some overlap with the driver survey described above. However, the statewide survey will focus more on issues of citizen acceptance of the system and of the MBUF concept. The team proposes that this survey take place late in the deployment cycle when much has been learned from the Field Study, the driver survey, focus groups, and other data collection activities. This survey will focus on putting forward the best and most convincing message possible and assessing its ability to convince drivers to a) accept the premise of the system and its three applications and b) utilize the system.


Page 45 of 51

CHAPTER 8. FOCUS GROUP / INTERVIEW CONTENT AND PROCEDURES Focus groups and interviews will be used to gather more detailed insight into the issues associated with implementing the Mn/DOT system. The focus groups and interviews will be conducted with two sets of participants – drivers from the field study and stakeholders (e.g., MN/DOT representatives).

The surveys and focus groups will complement one other. An initial focus group with drivers may assist the evaluation team to shape the survey questions in terms of topics, language, and tone. However, subsequent surveys may provide results that may be used to shape the development of focus group topics. That is, the focus groups will be useful to refine our understanding of information obtained through the surveys.

General Format The types of questions posed during focus groups and interviews may be very similar. These methods vary based on the number of participants and the degree of facilitation and probing that will occur.

FFooccuuss GGrroouuppss Focus groups will involve 8-10 participants and be led by a facilitator. Each focus group will last approximately 90 minutes and participants will receive a stipend for participating, if appropriate (Mn/DOT participants may not be eligible for a stipend).

Each group will be focused around 5-10 key questions or topics. These topics will involve the goals relevant to each group (stakeholders or drivers).

A focus group protocol will be developed outlining these topics, detailing probing or follow-up questions, and specifying when ratings or votes need to be performed. In these sessions, it will not be important for consensus to be reached. Instead, the goal of the session is to obtain various and divergent opinions and perceptions and to obtain details about each.

At least three focus groups will be conducted with groups of pilot study drivers. These will be conducted early during the Field Study; during the Baseline Phase, right after the system is turned on; and during Phase II (Late Deployment):

1. The initial focus groups will focus on expectations regarding the system.

2. The second focus group will focus on initial opinions and system learning.

3. The third focus group will focus on perspectives of daily use and impact of the system.

The overall goals of the focus groups interviews with pilot study drivers will be to achieve an understanding of their acceptance of the system, the barriers to acceptance, and strategies for overcoming these. Additionally, the evaluation team will attempt to identify lessons learned for both the conduct of the pilot study and the deployment of the system on a broader scale.

IInntteerrvviieewwss Interviews will be one-on-one interviews or dyads in which two people participate in an interview session. Interviewees may be recruited from willingness expressed on surveys or though volunteer requests. The interviews will be conducted by an interviewer and each will last approximately 60 minutes. Interviewees will receive a stipend if appropriate.

Interviews with drivers will occur on an as-needed basis, and will likely focus on select drivers who feel extremely positive or negative about the system as an attempt to identify the reasons for and strategies to manage their feelings.


Page 46 of 51

Each interview will be focused around 5-10 key issues involving the goals relevant to each group (stakeholders or drivers).

An interview script will be developed and customized for each interviewee or group of interviewees. This script will serve as a guide for the interviewer. However, the interviews will be flexible and primarily utilize open-ended questions.


Page 47 of 51

CHAPTER 9. ROLES AND RESPONSIBILITIES / NEXT STEPS The preceding chapters detail the data collection activities that will occur as part of the evaluation. This chapter discusses roles and responsibilities and next steps.

Responsibilities for Evaluation Activities The scope of the evaluation will be defined by its goals and associated measures and metrics, as well as the resources available to carry-out evaluation tasks both in terms of pilot participants and stakeholders and staff. For the purposes of this document, the various elements of the evaluation will be conducted by the “evaluation team.” It is important to note, however, that for certain activities described in this document, the “evaluation team” may include some combination of SAIC, the TPM team, Mn/DOT, and developer/integrator staff. It will be critical that these roles be determined prior to the start of Phase II.

Table 23 outlines the key evaluation activities associated with each goal, and presents the probable lead(s) for each along with the associated resources and responsibilities.

Table 23. Responsibilities for Key Evaluation Activities.

Evaluation Activity Lead(s) Resource(s) Documents(s)

Usability testing Developer(s) Test participants

Prototypes (paper, real)

Test plans

Technical memorandum of findings

IRB Approval SAIC

Mn/DOT

Test plans

Informed consent letters

IRB Application

Participant Recruitment Mn/DOT

SAIC

Informed consent letters (completed)

Schedule

Demographic questionnaire

Training/instructional materials Developer(s)

System Integrator

Mn/DOT

Instructions for participant drivers

Instructions for maintenance staff

Instructions for Mn/DOT and government staff

Garage Events Developer(s)

System Integrator

SAIC

Mn/DOT

Participants and participant vehicles

Systems

Maintenance staff

System checklist


Page 48 of 51

Evaluation Activity Lead(s) Resource(s) Documents(s)

Project Website: MBUF Site Developer(s)

System Integrator

Mn/DOT

Website content

Project Website: Participant Site SAIC

Mn/DOT

Website content

Field Study SAIC Participants

System data output and analysis

Database

Technical memorandum

Driver Survey / Statewide Survey / Focus Groups and Interviews

SAIC Participants Database

Technical memorandum

Next Steps This draft preliminary evaluation plan will serve a central role for the entire evaluation process and will serve to help shape the system development and deployment. The next step in this process will be for Mn/DOT to review this updated document and for the evaluation team to receive feedback on the document. The evaluation team will then revise the document and submit the Final Preliminary Evaluation Plan. The evaluation plan will be revisited in Phase II once the deployment team has been selected.