California Air Resources Board

I. Objective

The California Air Resources Board (CARB) is responsible for developing guidance for agencies administering programs funded by the Greenhouse Gas Reduction Fund (GGRF) referred to as California Climate Investments. The goal of this contract is to develop a comprehensive evaluation framework for California Climate Investments programs and projects. The contractor will produce a guidebook for State agencies that administer California Climate Investments programs on how to integrate evaluations into program planning and design, conduct evaluations during the operational phase of a project, and analyze data collected during the evaluation activities. This framework will serve as a new standard for California Climate Investments program and project evaluation.

II. Background

California Climate Investments refers to the portfolio of programs and projects that the State invests in using Cap-and-Trade auction proceeds with the goal of reducing greenhouse gas emissions, strengthening the economy, improving public health and the environment, and providing meaningful benefits to the most disadvantaged communities and low-income communities and households. The California Air Resources Board (CARB) provides guidance and support to over 80 programs administered by more than 20 California State agencies (administering agencies).

Senate Bill 862 (Committee on Budget and Fiscal Review, Chapter 36, Statutes of 2014) requires CARB to develop guidance on reporting and quantification methods for agencies receiving an appropriation from the GGRF. To date, CARB has developed methodologies for quantifying greenhouse gas (GHG) emission reductions and co-benefits across programs such as water savings, vehicle miles traveled reductions, utility cost savings to consumers, air pollutant reductions, and soil health and conservation, among others. Estimated program achievements and co-benefits are reported to CARB by the administering agencies using the California Climate Investments Reporting and Tracking System (CCIRTS). The database currently includes more than 580,000 projects, incentives, rebates, plans, and grants with funding information, project details, and estimated benefits. CARB administers CCIRTS and analyzes, consolidates, and presents data for public consumption through an annual report, interactive project map, online data dashboard, and other materials.

CARB recognizes the importance of assessing outcomes for California Climate Investments projects to gain a better understanding of their environmental, social, and economic impact. While CCIRTS serves as an effective mechanism for tracking and reporting on the prospective benefits of California Climate Investments projects, it lacks the capacity to measure outcomes for operational projects.

CARB developed a Project Outcome Reporting phase for California Climate Investments to collect primary data from grantees and program administering agencies during the period following project implementation to gain insights into project results. Over the last decade, various external initiatives have evaluated programs funded by California Climate Investments. These initiatives have analyzed program investments and produced recommendations for enhancing the impact of California Climate Investments programs to continue to better serve communities. In 2020, CARB engaged with the University of California, Davis Policy Institute for Energy, Environment, and the Economy to review project outcomes reporting within California Climate Investments and identify opportunities to enhance its effectiveness as an evaluation tool. While this effort resulted in recommendations to support improved data collection, analysis, transparency, and evaluation for individual programs and the California Climate Investments portfolio, there is need for a more comprehensive evaluation framework to guide administering agencies in integrating evaluation practices into new and existing programs and projects.

III. Scope of Work

Task 1 - Review current evaluation efforts from administering agencies and external initiatives evaluating California Climate Investments programs and projects.

Task 1 will include the review of existing evaluation efforts performed on California Climate Investments programs and/or projects by administering agencies. The contractor will conduct an analysis of a selection of materials from the California Climate Investments portfolio: program guidelines and/or solicitation materials, program and/or project award and implementation guidelines, past evaluations, reporting templates, materials that identify community benefits from projects, and program materials to determine how and to what extent evaluations are conducted. The review of current evaluation efforts must consider, but should not be limited to, how agencies evaluate the following outcomes: benefits to disadvantaged communities and low-income communities and households, economic impact, social equity, public health, energy security, innovation and technology development, environmental sustainability, resilience and adaptation, capacity building, education and awareness, and community engagement. This task will require coordination with administering agencies that have been identified as having expertise in designing and conducting evaluations in consultation with CARB staff.

This task will also include review of the aforementioned Project Outcome Reporting Data Collection and Assessment Report that previously evaluated California Climate Investments project outcomes and review of other assessment initiatives for California Climate Investments programs and/or projects by external organizations as determined by CARB, including, but not limited to: A Call to Invest in Community Power: Lessons from 10 Years of California Climate Investments for the State and the Nation, Evaluation of the California Climate Investments Technical Assistance Program and UCLA Luskin Center for Innovation Evaluation for California Climate Investments.

Task 2 - Develop an evaluation guidebook for California Climate Investments programs and projects

Task 2 involves developing an evaluation framework informed by Task 1 research and coordination with administering agencies on best practices, challenges, and lessons learned, and other methods for programs and projects evaluation. The framework will provide guidance on a variety of evaluation processes, selected in consultation with CARB, that are applicable to programs and projects that reflect the impact of California Climate Investments programs on communities, the environment, environmental justice, social equity, public health, and the economy. This framework will introduce comprehensive criteria for evaluating both programs and projects from multiple perspectives and will be developed into a guidebook for administering agencies that outlines the necessary steps to incorporate evaluations into program planning and design, best practices for conducting program and project evaluations and monitoring during the operational phase of a project, and guidance to support evaluation data analysis.

Importantly, the guidebook will provide the following, at minimum:

• Strategies for incorporating evaluation procedures and language into new or existing programs at the design stage, including key processes such as solicitations, grant agreements, and project implementation.
• Practical examples and best practices for selecting criteria and metrics, both quantitative and qualitative, for evaluations derived from case studies.
• Guidance on aligning programs with community-defined needs and evaluating the effectiveness of a program in addressing these needs through appropriate data collection and operations assessments.

Task 3 - Pilot the evaluation framework in existing California Climate Investments programs and projects

This task involves selecting a subset of California Climate Investments programs and/or projects to pilot the evaluation framework being developed in Task 2, securing buyoff from administering agencies for later involvement, and producing case study material(s) summarizing the findings. The contractor, in consultation with CARB, will identify California Climate Investments programs or projects with varying levels of evaluation procedures in place that are at different stages of implementation. The contractor will work with CARB to identify at least one of the following to pilot the framework within the contract period: 1) a program lacking established evaluation procedures and language, 2) a program that partially implements evaluation measures, and 3) a program that comprehensively implements project outcomes evaluation. These programs or projects may be in such program areas as transportation, nature-based solutions, clean energy, affordable housing, or others that are suitable to pilot the framework.

The contractor will work with CARB to select which projects and/or programs to pilot the framework and will conduct the pilot by undertaking such activities as advising on best practices and actionable steps. The contractors will work with relevant administering agencies to incorporate evaluation guidance into new or revised program guidelines, engage in an ongoing evaluation, assist in analyzing collected data, and advise on issuing a final evaluation. The contractor will then work with the administering agency to document the process and lessons learned through case studies or other materials. The goal of this pilot is to refine the evaluation framework and finalize it based on the lessons learned from its application. The results of the pilot will be included as an appendix in the evaluation framework developed for Task 2. To implement the pilot evaluation framework, there is an opportunity to extend the contract timeline beyond the two-year proposed period.

Task 4 - Produce the final evaluation Guidebook

The contractor will develop a draft guidebook, in consultation with CARB, which will include the evaluation framework with pilot case studies. The contractor shall develop outreach materials, in consultation with CARB, to facilitate sharing the guidebook. The draft guidebook will be available for public comment. The project team will also share results with administering agencies during a California Climate Investments multi-agency meeting organized by CARB. The contractor will then develop a final guidebook, in consultation with CARB.

IV. Deliverables

The project pre-proposal must include but is not limited to the following deliverables:

During Active Contract Period

• Work with CARB at the beginning of the project to create a 1-page plain language outreach deliverable for the public describing the project’s goals, process, and planned deliverables (available in multiple languages, template will be provided).
• Quarterly Progress Meetings.
• Informal monthly progress update meetings with CARB contract manager.
• Draft evaluation framework.*
• Draft Guidebook*
• Participation in one California Climate Investments multi-agency meeting organized by CARB staff. The contractor is expected to present the draft evaluation framework developed in Task 2 to administering agencies and incorporate feedback as applicable.

Prior to Contract Close

• Final evaluation framework.
• Final Guidebook.
• Participation in a second California Climate Investments multi-agency meeting organized by CARB staff. Following the completion of the final guidebook, the contractor will present the guidebook to administering agencies.

* CARB will provide up to 2 revisions on draft versions of this deliverable.

NOTE: contractor will be responsible for ensuring their documents comply with accessibility standards in the Web Content Accessibility Guidelines 2.0 or subsequent version.

Additional deliverables to be determined in consultation with CARB staff.

V. Timeline

It is anticipated this project will be completed in 24 months from the start date (start date is estimated to be in early 2025). The estimated budget for this project is up to $350,000. There is an opportunity to extend the contract timeline beyond the two-year proposed period.

Scoring Criteria

1. RESPONSIVENESS TO THE GOALS AND OBJECTIVES OUTLINED IN THE PROPOSAL SOLICITATION (25 POINTS)

The proposal should explain how the proposed project satisfies the project objectives. In particular, the proposal should describe how the proposers will approach incorporating the following key priorities into the development of the comprehensive evaluation framework and guidebook for State agencies that administer California Climate Investments programs: 1) how to integrate evaluations into program planning and design, 2) how to conduct evaluations during the operational phase of a project, and 3) how to analyze data collected during the evaluation activities.

2. WORK EXPERIENCE AND SUBJECT MATTER EXPERTISE (20 POINTS)

The proposal should demonstrate that the contractor has work experience or subject matter expertise in designing and implementing evaluation methodologies to assess the performance of climate investments programs and projects. This includes developing metrics and indicators, conducting both formative and summative evaluations, and using findings to make informed recommendations required to successfully carry out the evaluation of climate investments. Additionally, the proposal should describe how the project will build upon previous relevant work that was funded by CARB, other regional, state, and federal agencies.

3. EXPANDING EXPERTISE (15 POINTS)

The proposal should explain how the project team expands expertise such as by incorporating multidisciplinary expertise or perspectives, including members from various public universities, non-academic institutions, or community-based organizations, or providing opportunities to build skills and expertise for individuals from underrepresented groups.

4. EXPLANATION OF TECHNICAL OR METHODOLOGICAL APPROACH (25 POINTS)

The proposal should clearly explain the logic and feasibility of the methodology, spell out the sequence and relationships of major tasks, and explain methods for performing the work. The proposal should include a clear description and plan for how each task will be completed and how the proposer will approach collecting information that may not be found in published works (i.e., how the proposer will identify and engage with administering agencies to learn about their evaluation efforts).

5. LEVEL AND QUALITY OF EFFORT AND COST EFFECTIVENESS (15 POINTS)

The proposal should describe how time and resources will be allocated and demonstrate how this allocation ensures the project’s success. Proposal reviewers will evaluate, for example: if the objectives of the project can be met given this allocation, if there is adequate supervision and oversight to ensure that the project will remain on schedule, if time and cost are appropriately divided up across different project tasks and stages.