Developing an Equity Evaluation Framework for Place-Based, Community-Scale Mobility Investments

Contact

I.      Objective

The objective of this project is to develop an evaluation framework and identify performance metrics to assess the synergistic equity impacts of place-based, community-scale mobility investments during and after project implementation. The resulting framework should be adaptable for diverse mobility project types and minimize the data collection burden for grantees while also supporting a robust quantitative evaluation of the real-world outcomes relevant to equity-related program objectives. It should incorporate local community context and measure the synergistic effects of multiple complementary mobility investments. This project will build on existing evaluation methods by incorporating new academic research and lessons learned from relevant pilot projects.

II.     Background

The Clean Energy and Pollution Reduction Act of 2015 (Senate Bill 350) directed state agencies to identify and assess barriers limiting zero-emission and near-zero-emission transportation options for low-income residents, including in disadvantaged communities. Pursuant to this directive, CARB (in consultation with the California Energy Commission and in collaboration with many other state agencies, local jurisdictions, and community groups) developed “The Low Income Barriers Study, Part B: Overcoming Barriers to Clean Transportation Access for Low-Income Residents” in 2018. The report outlined several priority recommendations, the first of which is to ensure low-income residents play an active role in clean transportation and mobility project planning. This report has informed the approach of several CARB programs addressing transportation and climate priorities at the local scale.

Climate equity programs funded under California Climate Investments such as the Sustainable Transportation Equity Project (STEP), launched in 2019, take a place-based, community-scale investment approach by funding a suite of complementary mobility-enhancing projects that address community-identified needs within one disadvantaged community. Some of the project types include active transportation infrastructure, such as bike lanes; mobility subsidies, such as prepaid transit debit cards; neighborhood design features that promote active transportation, such as urban forestry; and carsharing programs featuring zero-emission vehicles. A more extensive list of project types can be found in the table of eligible project types (Table 2) of the document “Draft Planning and Capacity Building, CMIS, and STEP Requirements and Criteria.” The Strategic Growth Council’s Transformative Climate Communities Program (TCC), authorized by Assembly Bill 2722 in 2016, demonstrates the transformative potential of place-based, community-driven climate investments, as detailed in the UCLA Luskin Center for Innovations progress reports. However, the progress reports also identify some of the challenges in quantitative evaluation of program outcomes for these types of community-scale investments.

Place-based, community-scale projects are tailored to the local context and community priorities, which can complicate standardized evaluation and comparison between diverse projects. Much of the extant mobility equity evaluation literature is focused on project selection and projected or modeled impacts rather than metrics for the real-world impact of the investment during and after implementation. Policy makers need methods to evaluate and compare the real-world outcomes of different mixes of community-scale mobility project types during and after implementation to better understand the impacts of mobility programs and make evidence-based decisions about prioritizing future investments. Additionally, programs such as STEP that are funded as a part of California Climate Investments are required to report on investment outcomes in accordance with the 2018 California Climate Investments Funding Guidelines. In 2020, the California State Auditor issued a report titled “Improved Program Measurement Would Help California Work More Strategically to Meet Its Climate Change Goals,” based on an audit of CARB’s transportation programs targeting greenhouse gas reduction. A key recommendation in the report is to develop metrics that directly measure outcomes tied to each of a program’s stated objectives.

Under the current evaluation method, the impacts of each project component are assessed separately and added to calculate the cumulative impact of the overall investment. This evaluation method fails to capture any potential synergistic effects of coordinated investments and does not support assessment of whether this place-based funding approach should be prioritized over a piecemeal approach to funding individual projects. Similarly, it fails to assess what investment types or characteristics maximize the benefits achieved and the effectiveness of public dollars.

One challenge in assessing the impacts of community-scale investments is collecting assessment data at an appropriate scale. Census data may not detect the impacts of small-scale investments in one priority community because of the infrequency of census data collection relative to grant terms and the broad geographic scale of reported metrics. To capture smaller-scale impacts, the current assessment methodology relies on user surveys. However, this approach misses information about community members who are not accessing the funded project and any barriers preventing them from accessing it. For instance, surveying the users of a new shuttle service about their satisfaction with the service will be biased towards positive feedback and will not reflect the needs of any community members facing barriers to accessing the service. Survey data are also more costly and labor-intensive for grantees to collect and potentially more prone to sampling bias and errors compared to other evaluation metrics, such as telematic information about trip origins and destinations.

This contract will address the need for a consistent framework to evaluate and prioritize investments in diverse community-scale, place-based project types supporting mobility equity. The contractor will propose a framework and new equity performance metrics that support robust project evaluation while minimizing the data collection burden for grantees and users of these services.

III.    Scope of Work

The draft scope of work below outlines CARB’s goals, expected outcomes, and potential deliverables for this project. The pre-proposal should address at a high level how the proposers will approach these tasks. Proposers are not required to stay within these specific parameters and are encouraged to add detail or suggest alternative tasks and methods for realizing project goals based on their own expertise and understanding of the project and budget. The final contract and scope of work will be developed through an iterative process with CARB and the selected contractor and will include a detailed timeline as well as guidance on deliverables (e.g., white paper contents).

The outcome of this contract will be a white paper describing a new framework for the evaluation of the equity impacts of community-scale, place-based mobility investments in disadvantaged communities, including a set of performance metrics. The specific equity-related objectives to be evaluated will be developed in collaboration with the CARB contract contract manager but may include program objectives such as increasing access to key destinations for low-income communities and communities disproportionately impacted by air pollution who lack access to housing, jobs, and services. The new framework will help CARB staff prioritize mobility investments as more projects are implemented and evaluated. The framework should be sensitive to the constraints and challenges identified in the background section and to community priorities for these types of investments. While the contractor will not be expected to answer the questions below as part of this scope of work, the framework and metrics should support CARB’s ongoing investigation of the following priority policy questions:

  1. Do multiple coordinated place-based mobility investments in one community work synergistically to advance equity objectives more effectively than individual projects implemented in a piecemeal approach?
  2. Which combinations of project types most effectively advance mobility equity priorities? (note: specific priorities to be evaluated will be decided in collaboration with the CARB contract manager)
  3. Is there a threshold of investment or project scale at which a given project type has a significant impact? (e.g., a density of trees along a sidewalk that increases walking)
  4. Are some project types or combinations of project types more effective than others or better suited to a particular community context?

In developing the framework, the contractor should complete the following six tasks and any others identified in collaboration with the CARB contract manager.

Task 1 – Review Existing Grants to Inform New Framework

Concurrent with the development of the annotated bibliography in Task 2 and in collaboration with the CARB contract manager, the contractor should select between two and four CARB grant projects to review in depth as case studies of the types of grants to be evaluated using the new framework to be developed in Task 4. CARB will provide existing data and information about the selected projects and relevant project-specific documents, such as status reports. The purpose of this review is for the contractor to become familiar with the types of projects funded; the place-based, community-scale structure of the grants; and the types of data that have been collected for evaluation so far. The contractor should summarize key insights in a brief written update to the CARB project manager. The review should identify evaluation challenges and opportunities for each project.

Task 1 Deliverables:

  1. Brief written summary of key insights from project review

Task 2 – Prepare Annotated Bibliography of Mobility Equity Project Evaluation, Place-Based Investment Evaluation, and Relevant Pilot Projects

This project will build upon existing project evaluation methodologies by incorporating new insights from academic and gray literature. The contractor will prepare an annotated bibliography to support the new evaluation framework to be developed in Task 4. The contractor should review literature on equity evaluation of mobility projects during and after project implementation. Selection criteria should prioritize studies of community-scale mobility equity pilot projects, particularly those including multiple coordinated investments in one community, but may include larger scale, more traditional transportation projects depending on the available literature. Given that the literature on evaluating community-scale, place-based mobility projects is limited, the review should also include relevant sources that discuss evaluation of the synergistic impacts of other types of place-based investments, with an emphasis on project types that are most similar to the mobility equity projects funded by CARB.

Task 2 Deliverables:

  1. Draft of annotated bibliography
  2. Final annotated bibliography

Task 3 – Develop Table of Mobility Equity Performance Metrics

By synthesizing findings from Tasks 1 and 2 and, the contractor should develop a table of existing and proposed new equity performance metrics, divided by project type. In consultation with the CARB contract manager, the contractor should select a minimum of five mobility project types from the list of project types in Table 2 of the document “Draft Planning and Capacity Building, CMIS, and STEP Requirements and Criteria.” The table may include details like community context and related example pilot projects if appropriate.

Task 3 Deliverables:

  1. Template for equity performance metrics summary table, identifying parameters to include
  2. Draft of equity performance metrics summary table
  3. Final draft equity performance metrics summary table

Task 4 – Develop Recommendations for a New Evaluation Methodology for Equity Impacts of Community-Scale, Place-Based Mobility Investments in White Paper

In collaboration with the CARB contract manager, the contractor should synthesize findings from Tasks 1-3 to develop recommendations for a new equity evaluation framework for community-scale, place-based mobility investments. The new methodology should be summarized in a white paper and include the table of performance metrics developed in Task 3. The methodology should support evaluation of a diverse range of mobility projects, while minimizing the data collection burden for grantees. It should be designed to support CARB’s ongoing investigation of the four priority policy questions identified at the beginning of Section III Scope of Work. The white paper should also include future areas of investigation and recommendations for any further work needed to develop the framework.

Task 4 Deliverables

    1. Draft white paper “New Equity Evaluation Framework for Place-Based, Community-Scale Mobility Investments”
    2. Final white paper “New Equity Evaluation Framework for Place-Based, Community-Scale Mobility Investments”

Task 5 – Summarize Key Findings in Policy Brief and Seminar Presentation

After developing the white paper in Task 4, the contractor should summarize key insights in a policy brief for non-technical readers and users, which may include policy makers or elected officials and community members. The policy brief will include a summary of the issue-of-focus for the white paper, key findings from Tasks 1-4, recommended next steps, and a summary of future research directions, as identified by the contractor in earlier tasks. The contractor will also present these key project findings at a one-hour seminar, to be held via webinar or webcast.

Task 5 Deliverables

  1. Draft policy brief “New Equity Evaluation Framework for Place-Based, Community-Scale Mobility Investments”
  2. Final policy brief “New Equity Evaluation Framework for Place-Based, Community-Scale Mobility Investments”
  3. Draft slide deck presentation “New Equity Evaluation Framework for Place-Based, Community-Scale Mobility Investments”
  4. Final slide deck for presentation “New Equity Evaluation Framework for Place-Based, Community-Scale Mobility Investments”
  5. Seminar to present findings

Task 6 – Prepare Initial Kickoff Meeting Agenda, Monthly Progress Meeting Agendas, and Quarterly Progress Reports

The contractor will have an initial kickoff meeting with CARB staff via conference call or other virtual platform (e.g., Teams, Zoom, etc.). Before work on the agreement begins, the contractor’s key personnel, such as the project manager and technical staff, will meet with the CARB Project Manager and other CARB staff to discuss the overall plan, details of performing the tasks, the project schedule, items related to personnel or changes in personnel, and any issues that should be resolved before work may begin.

The contractor will meet monthly with CARB’s contract manager and additionally as needed throughout the contract term. The contractor will develop the agendas for monthly meetings and prepare quarterly progress reports for the contract manager and applicable staff. Meetings may include other key stakeholders—such as other CARB staff and State agency experts—to discuss project progress and solicit input on key project milestones and decision points.

Task 6 Deliverables:

  1. Initial Kickoff Meeting Agenda
  2. Monthly Meeting Agendas
  3. Quarterly Progress Reports

IV.    Deliverables

The project pre-proposal must include but is not limited to the following deliverables:

During Active Contract Period

  1. Work with CARB staff at the beginning of the project to create a plain-language outreach deliverable for the public describing the project’s goals, process, and planned deliverables (available in multiple languages, template will be provided).
  2. Agendas for monthly progress update meetings with CARB contract manager.
  3. Quarterly Progress Reports including public-facing updates to be posted to CARB’s website.

Prior to Contract Close

  1. Brief written summary of key insights from review of CARB grants (Task 1)
  2. Annotated bibliography “Equity Evaluation of Place-Based, Community-Scale Mobility Investments” (Task 2)
    1. Draft annotated bibliography
    2. Final annotated bibliography
  3. New equity performance metrics summary table (Task 3)
    1. Template of table, identifying parameters to include
    2. Draft of performance metrics summary table
    3. Final performance metrics summary table
  4. White paper describing new evaluation framework and performance metrics “New Equity Evaluation Framework for Place-Based, Community-Scale Mobility Investments” (Task 4)
    1. Draft white paper
    2. Final draft of white paper
  5. Policy brief and presentation summarizing findings (Task 5)
    1. Draft policy brief
    2. Final policy brief
    3. Draft presentation
    4. Final slide deck for presentation
    5. Seminar to present findings

NOTE: The contractor will be responsible for ensuring their documents comply with the American with Disabilities Act.

Additional deliverables may be determined in consultation with CARB staff.

V.     Timeline

It is anticipated this project will be completed in 18 months from the start date (start date is estimated to be in Spring 2024). The estimated budget for this project is up to $150,000.

Scoring Criteria

1. Responsiveness to the goals and objectives outlined in the proposal solicitation (30 points)

The proposal should explain how the proposed project satisfies the project objectives. In particular, the proposal should describe how the proposers will approach incorporating the following key priorities into the development of the new review framework and performance metrics: (1) capturing the synergistic effects of a place-based investment approach, (2) identifying data sources that meaningfully measure real-world equity impacts at the community scale, (3) minimizing data collection burden on grantees, and (4) balancing the need for metrics that are specific to certain project types and community contexts with the need for standardized evaluation and comparison of effectiveness.

2. Work experience and subject matter expertise (30 points)

The proposers should demonstrate that they have the work experience and subject matter expertise required to successfully carry out the proposed project as described. In particular, the proposers should describe their commitment to equity and their expertise and experience in the areas of project evaluation frameworks and metrics, community-scale mobility projects, and place-based investments. The strongest proposers will also demonstrate a commitment to collaboration with community groups.

3. Explanation of technical or methodological approach (20 points)

The proposal should clearly explain the logic and feasibility of the project’s methodology, spell out the sequence and relationships of major tasks, and explain methods for performing the work. The proposal should include a clear description and plan for how each task will be completed. The strongest proposals will use the scope of work described above as a starting point to develop a more detailed and specific plan for achieving the project outcomes. The proposers are encouraged to add specific interim deliverables or propose additional or alternative tasks or subtasks.

4. Budget and timeline detail and feasibility (15 points)

The proposal should describe how time and resources will be allocated and demonstrate how this allocation ensures the project’s success. Proposal reviewers will evaluate, for example: if the objectives of the project can be met given this allocation, if there is adequate supervision and oversight to ensure that the project will remain on schedule, and if time and cost are appropriately divvied up across different project tasks and stages.

5. Expanding expertise (5 points)

The proposal should explain how the project team expands expertise such as by incorporating multidisciplinary expertise or perspectives; including members from various public universities, non-academic institutions, or community-based organizations; or providing opportunities to build skills and expertise for individuals from underrepresented groups. Reviewers will consider if key personnel contributing significantly to the project (i.e., a principal investigator, co-principal investigator or co-investigator, contributing 25 percent or more of their time to the project) have not worked with CARB in the past five years.