California Air Resources Board

I.      Objective

Per Senate Bill 375, the California Air Resources Board (CARB) is responsible for accepting or rejecting the greenhouse gas emission reduction determinations provided in sustainable communities strategies (SCS) prepared by metropolitan planning organizations (MPO). As part of each determination, CARB provides a detailed staff report supporting its findings and conclusions, with recommendations for addressing identified risks to implementation. Conducting the assessment takes substantial staff time from both CARB and the MPOs. CARB will be engaging in a public process to consider concepts for streamlining this review process, which will include concepts for evaluating and measuring regional VMT reduction strategies and the likelihood of implementation. The research outlined in this contract will provide CARB with recommendations which will assist CARB in this work.

II.     Background

In 2008, the state Legislature passed SB 375. This bill requires MPOs to adopt an SCS as part of its regional transportation plan, to achieve reduction of greenhouse gas emissions (GHG) from automobiles and light trucks in the region. The bill requires CARB, working in consultation with the MPOs, to provide each affected region with GHG reduction targets for 2020 and 2035. The bill requires MPOs to submit the SCS to CARB and requires CARB to accept or reject the MPO’s determination that the plan would achieve the GHG reduction targets for the region. By 2015, every MPO had adopted its first SCS, and many are now on their third SCS.

Since the program’s inception, CARB has provided research and guidelines to support regions in SCS development. This information has been periodically updated as the program and associated legislation evolved over time. In 2019, CARB published the Sustainable Communities Strategy Program and Evaluation Guidelines, which outlines how CARB evaluates SCSs and outlines methodologies MPOs may use to quantify their SCS strategies. In 2020, CARB published an SCS Submittal Guide and Template that supports the 2019 Evaluation Guidelines.

Under Senate Bill 150, CARB is also required to evaluate the progress of SB 375 and report to the Legislature every four years. CARB completed the first progress report in 2018. A 2022 Progress Report is also now available and highlights that SCS strategies face significant implementation challenges and are falling behind. Data show that not only are regional greenhouse gas emissions from passenger travel not going down as much as needed to meet state climate goals, but they are also trending in the wrong direction.

One of the outcomes of the progress report is the need to focus on plan implementation. There are many facets to this. One way CARB can support this is to improve the SCS evaluation process to foster stronger plans. CARB will begin a public process towards updating the evaluation process. This process will help further define the goals and desired outcomes for updating the evaluation process. However, CARB’s initial goals for updating CARB’s SCS evaluation process are:

1. Simplify and streamline the review process so that CARB can make a determination quicker, allowing MPOs to focus on implementation sooner.
2. Create a transparent line between when a plan should be accepted or rejected so that CARB’s expectations on SCS strategies are clear to everyone. This includes considering whether the target would be met if the plan were fully implemented and the likelihood that strategies in the plan will be implemented.

The purpose of CARB’s evaluation is to accept or reject the SCS based on whether region-specific strategies to reduce vehicle miles traveled (VMT) are likely to achieve the GHG reduction target. As such, a critical component of the SCS determination process is evaluating the forecasted benefits of these strategies and the likelihood of their implementation. Under the current SCS evaluation process, CARB staff analyze whether SCS strategies for meeting the GHG emission reduction targets are supported by regional/local actions (e.g., key policies, investments, and other commitments) to advance their implementation. CARB staff’s analysis is organized across four broad SCS strategy categories: (1) land use and housing, (2) transportation infrastructure and network, (3) local/regional pricing, and (4) electric vehicle and new mobility.^[1] In general, across all strategy categories, CARB staff looked for:

• Data that show the SCS is moving in a direction consistent with the planned outcomes.
• Supportive, specific, and measurable regional/local actions that correspond to each individual SCS strategy.
• Regional investment commitments in the RTP/SCS project list that support the strategies and the GHG reductions.
• Analysis on whether the region is currently on track for achieving the target and, if needed, what corrective actions are being taken.

Updates to the evaluation process need to focus on how CARB can continue to assess these factors in a simplified and transparent way. 

To help inform potential evaluation process changes, CARB would like to gather information on examples of other potentially applicable evaluation processes. This research would provide information that will be considered as part of a future update to the evaluation process that will involve input and discussion by the public and interested parties.

III.    Scope of Work

CARB is exploring new evaluation processes and concepts to assess how MPOs intend to meet their targets for 2035 through various land use, transportation, pricing, and new mobility strategies. CARB will conduct a public process that would consider various approaches to evaluate whether an SCS is likely to achieve its GHG reduction goals. CARB would like to gather research on different approaches to evaluating planning documents, including but not limited to, approaches such as a scoring rubric, points-based evaluation methodology, or other methods. CARB would also like research informed recommendations on specific components of the evaluation process, including but not limited to:

• What quantitative measures should be used to evaluate specific VMT reduction strategies and supporting actions to advance implementation?
• How to connect assumptions about future performance to on-the-ground progress to assess whether a region’s plan is on track for achieving its GHG reduction target by 2035? 
• How to evaluate specific VMT reduction strategies and supporting actions when implementation relies on the actions and authority of many state, local, and community partners? For example, should a discount be applied if there is not statutory authority for the measure?
• Given the complexity of travel demand models, the influence of exogenous factors in the implementation and performance of a plan, and the reality that many different agencies or partners are responsible for taking actions that would achieve the planned outcomes, what are some recommendations for creating a clear, transparent standard for determining whether an SCS would meet the GHG target? 
• How to be responsive to the different sizes and geographies of regions, as well as differing capacities of MPO staff, while also creating some standardization in the expectations and review process?
• What key information to require as part of the evaluation process?

The recommendations should result in an evaluation process that reduces necessary State agency and MPO staff time and resources, recognizing the need to meet statutory deadlines and that State agency staff can have multiple evaluations occurring at the same time.  

The research and recommendations being requested in this proposal will help CARB advance evaluation concepts for the public and stakeholder process.

Task 1 – Process Evaluation and Recommendations

Task 1 is focused on evaluating long-range plans and/or other climate, transportation, and land use documents, including CARB’s current process, and making recommendations on how to improve CARB’s current evaluation process to address the challenges and questions outlined above.

This task includes evaluating processes used by other agencies to evaluate long-range climate, transportation, or land use plans. This task also includes a review of CARB’s evaluation guidelines, materials and reports including but not limited to SB 150 reports detailing the challenges of the program, the AB 285 report, existing law, and existing methods for quantifying estimated and actual outcomes from plans. This task potentially includes interviews with staff from other agencies to understand more about their experience with evaluation processes. Which agencies to be interviewed will be determined and agreed upon by CARB and the selected contractor. Based on the evaluations, review of materials, and potential interviews, the consultant should make recommendations on improvements to CARB’s overall evaluation process and quantification methods. The recommendations should include examples of how the changes could be implemented. These recommendations should consider quantifying performance including any data or metrics to consider, measuring progress, who is responsible for implementing the strategies, creating bright lines between acceptance and rejection, regional variation, and what elements should be required in the review process.

This task also includes regular meetings and presentations with CARB staff. This task should include the opportunity for two rounds of review and comment on the draft by CARB before delivering the final report.

Deliverables:

• “Evaluation Summary” – Documentation of evaluation processes. This should include a brief summary of the evaluation processes reviewed. The report should include context, who reviews, and detailed information about the evaluation framework and scoring criteria, including any weighting, variations, special considerations, and potential pitfalls with the methodology. It should aim to include at least two examples of efforts to compare plan forecasts to measurable outcomes and strengthen the accuracy of forecasts.
• “Evaluation Recommendations” – Report of recommendations. The report should include recommendations for changes to CARB’s evaluation process and potential quantification methods. Documentation should include a discussion of how the recommendations will support stronger plans that better match future outcomes in support of the state’s goals, how they streamline the process, and how they increase the transparency of whether a plan should be accepted or rejected.

IV.    Deliverables

The project pre-proposal must include but is not limited to the following deliverables:

During Active Contract Period

1. Quarterly Progress Meetings.
2. Informal monthly progress update meetings with CARB contract manager and other key staff (if applicable).

Prior to Contract Close

3. Draft white paper /interim report “Evaluation Summary”
4. Final white paper /final report “Evaluation Summary”
5. Draft white paper/interim report “Evaluation Recommendations”
6. Final white paper/final report “Evaluation Recommendations”
7. Public presentations summarizing findings (if applicable)

NOTE: contractor will be responsible for ensuring their documents comply with the American with Disabilities Act.

Additional deliverables to be determined in consultation with CARB staff.

V.     Timeline

The timeline for this project is anticipated to begin in May 2024 and end by December 2025. Early research and draft deliverables are assumed to be delivered around December 2024 with specific deliverables and timelines to be negotiated upon project award. The budget for this project is not to exceed $200,000.

Scoring Criteria

1. Responsiveness to the goals and objectives outlined in the proposal solicitation (20 points)

The proposal should explain—in adequate detail and clear, understandable language—how the proposed project satisfies the project objectives.

2. Work experience and subject matter expertise (20 points)

The proposal should demonstrate that the proposers have the work experience or subject matter expertise required to successfully carry out the proposed project as described. Additionally, the proposal should describe how the project will build upon previous relevant work that was funded by CARB, other regional, state, and federal agencies.

3. Expanding expertise (10 points)

The proposal should explain how the project team expands expertise such as by incorporating multidisciplinary expertise or perspectives, including members from various public universities, non-academic institutions, or community-based organizations, or providing opportunities to build skills and expertise for individuals from underrepresented groups. Reviewers will consider if key personnel contributing significantly to the project (i.e., a principal investigator, co-principal investigator or co-investigator, contributing 25 percent or more of their time to the project) have not worked with CARB in the past five years.

4. Explanation of technical or methodological approach (20 points)

The proposal should clearly explain the logic and feasibility of the project’s methodology, spell out the sequence and relationships of major tasks, and explain methods for performing the work. The proposal should include a clear description and plan for how each task will be completed.  

5. Level and quality of effort and cost effectiveness (15 points)

The proposal should describe how time and resources will be allocated and demonstrate how this allocation ensures the project’s success. Proposal reviewers will evaluate if the objectives of the project can be met given this allocation, if there is adequate supervision and oversight to ensure that the project will remain on schedule, and if time and cost are appropriately divvied up across different project tasks and stages.

[1] In this context, “SCS strategies” are the high-level strategies used in the SCSs to achieve the GHG reduction target. These generally fall into these four categories. Each SCS often also includes a list of activities or actions that the MPO and/or other agencies will do to implement the strategies.