EPIC Final Report 090817 Compressed

Electric Program Investment
Charge (EPIC) Evaluation
May 1, 2015
Final Report
September 8, 2017

Table of Contents
LIST OF KEY TERMS................................................................................................................... VI
ACRONYMS AND ABBREVIATIONS.....................................................................................IX
1 EXECUTIVE SUMMARY ..................................................................................................... 1-1
1.1 EVALUATION OBJECTIVES AND APPROACH ..................................................................... 1-1
1.2 KEY FINDINGS AND RECOMMENDATIONS ........................................................................ 1-3
1.2.1 Program Requirements .................................................................................................... 1-3
1.2.2 EPIC Project Impacts ........................................................................................................ 1-3
1.2.3 Optimizing EPIC’s Portfolio ........................................................................................... 1-5
1.2.4 Effectively Engaging Stakeholders ................................................................................ 1-5
1.2.5 Independent Coordination Body ................................................................................... 1-7
1.2.6 IOUs in Compliance, but Not Meeting Intent of All Requirements ......................... 1-8
2 PROGRAM AND POLICY BACKGROUND ................................................................... 2-1
2.1 PROGRAM BACKGROUND .................................................................................................. 2-1
2.2 CALIFORNIA ENERGY POLICY ........................................................................................... 2-3
2.2.1 Greenhouse Gas Reduction Legislation and the Renewable Portfolio Standard .... 2-3
2.2.2 Grid Infrastructure ........................................................................................................... 2-4
2.2.3 Energy Efficiency and the Loading Order .................................................................... 2-5
2.2.4 Distributed Energy Resources ........................................................................................ 2-5
2.2.5 Governor Brown’s Clean Energy Jobs Plan .................................................................. 2-7
2.2.6 Diversity and Disadvantaged Communities ................................................................ 2-8
3 EVALUATION APPROACH ............................................................................................... 3-1
3.1 EVALUATION OBJECTIVES AND RESEARCH QUESTIONS ................................................... 3-1
3.2 RESEARCH APPROACH ...................................................................................................... 3-3
3.3 PROGRAM DOCUMENT REVIEW ........................................................................................ 3-4
3.4 PROGRAM ADMINISTRATOR INTERVIEWS ......................................................................... 3-4
3.5 EPIC PROJECT DATA REVIEW ........................................................................................... 3-5
3.6 PROJECT SAMPLE DATA REVIEW....................................................................................... 3-5
3.7 PROJECT INTERVIEWS ........................................................................................................ 3-7
3.8 STAKEHOLDER INTERVIEWS .............................................................................................. 3-8
3.9 CEC SOLICITATION AND IOU OUTSOURCING PROCESS REVIEW .................................. 3-10
3.10 BEST PRACTICES ASSESSMENT ......................................................................................... 3-11
3.11 NETWORK ANALYSIS ....................................................................................................... 3-15
4 PROGRAM THEORY AND LOGIC MODEL .................................................................. 4-1
Evergreen Economics Page i

4.1 OVERVIEW OF LOGIC MODEL APPROACH ........................................................................ 4-1
4.2 METRIC DEVELOPMENT .................................................................................................... 4-2
4.3 OVERARCHING EPIC LOGIC MODEL ................................................................................ 4-3
5 PORTFOLIO CHARACTERIZATION .............................................................................. 5-1
6 PROGRAM ADMINISTRATION ...................................................................................... 6-1
6.1 COMPLIANCE WITH PROGRAM REQUIREMENTS ............................................................... 6-2
6.2 PROGRAM MANAGEMENT ................................................................................................ 6-9
6.2.1 CEC .................................................................................................................................. 6-9
6.2.2 IOUs ................................................................................................................................ 6-10
6.3 CONTRACTING ................................................................................................................. 6-12
6.3.1 CEC ................................................................................................................................ 6-12
6.3.2 IOUs ................................................................................................................................ 6-14
6.3.3 Non-Competitive Bidding ............................................................................................. 6-14
6.4 BEST PRACTICES COMPARISON ....................................................................................... 6-18
6.4.1 Program Administration and Context......................................................................... 6-19
6.4.2 Program Management ................................................................................................... 6-19
6.4.3 Contracting ...................................................................................................................... 6-20
6.4.4 Comparison with EPIC .................................................................................................. 6-20
7 INVESTMENT PLANNING PROCESS ............................................................................ 7-1
7.1 INVESTMENT PLAN SCOPING ............................................................................................ 7-1
7.1.1 CEC Plan Scoping ............................................................................................................. 7-2
7.1.2 IOU Plan Scoping ............................................................................................................. 7-8
7.2 STAKEHOLDER OUTREACH ............................................................................................. 7-14
7.2.1 Stakeholder Workshops ................................................................................................. 7-15
7.2.2 Idea Exchange ................................................................................................................. 7-19
7.2.3 Stakeholder Interviews .................................................................................................. 7-25
7.3.1 Policy Alignment ............................................................................................................ 7-26
7.3.2 Investment Planning ...................................................................................................... 7-26
8 PROJECT SELECTION PROCESS ..................................................................................... 8-1
8.1 PROJECT SELECTION PROCESS ........................................................................................... 8-1
8.1.1 CEC .................................................................................................................................. 8-1
8.1.2 IOUs ................................................................................................................................ 8-13
8.2 PROJECT SCOPING PROCESS ............................................................................................ 8-16
8.2.1 CEC ................................................................................................................................ 8-18
8.2.2 IOUs ................................................................................................................................ 8-20
8.3 MATCH FUNDING ............................................................................................................ 8-22
Evergreen Economics Page ii

8.4 PLAN CONSISTENCY ........................................................................................................ 8-25
8.5 COORDINATION ............................................................................................................... 8-28
8.5.1 Coordination Across Administrators .......................................................................... 8-28
8.5.2 Coordination with Other Energy Innovation Efforts ................................................ 8-31
8.6.1 Match Funding ................................................................................................................ 8-33
9 PROJECT ASSESSMENT PROCESS ................................................................................. 9-1
9.1 PROJECT STATUS ASSESSMENT .......................................................................................... 9-1
9.1.1 CEC .................................................................................................................................. 9-4
9.1.2 IOUs .................................................................................................................................. 9-6
9.1.3 Current Project Status ...................................................................................................... 9-6
9.2 BENEFITS QUANTIFICATION .............................................................................................. 9-7
9.2.1 CEC .................................................................................................................................. 9-8
9.2.2 IOUs .................................................................................................................................. 9-9
9.3 RESULTS DISSEMINATION ................................................................................................ 9-12
9.3.1 Knowledge Dissemination Activities .......................................................................... 9-14
9.3.2 Network Composition and Activities .......................................................................... 9-21
10 POLICY ALIGNMENT AND PROJECT IMPACTS ..................................................... 10-1
10.1 POLICY ALIGNMENT ........................................................................................................ 10-2
10.2 PROJECT DIVERSITY ......................................................................................................... 10-4
10.2.1 Technology and Study Characterization .................................................................... 10-4
10.2.2 Commercialization Status ............................................................................................. 10-6
10.2.3 Schedule of Project Results ......................................................................................... 10-10
10.3 PUBLIC INTEREST FOCUS ............................................................................................... 10-12
10.3.1 Broad Public Benefits .................................................................................................. 10-12
10.3.2 Net Public Benefits....................................................................................................... 10-13
10.4 PROGRAM EQUITY ......................................................................................................... 10-16
10.5 INDICATORS OF PROJECT PERFORMANCE ..................................................................... 10-19
10.5.1 Applied R&D Projects ................................................................................................. 10-19
10.5.2 TD&D Projects .............................................................................................................. 10-20
10.5.3 Market Facilitation Projects ........................................................................................ 10-22
11 FINDINGS AND RECOMMENDATIONS .................................................................... 11-1
11.1 PROGRAM ADMINISTRATION .......................................................................................... 11-5
11.2 INVESTMENT PLANNING PROCESS .................................................................................. 11-7
11.2.1 Administrator Investment Planning Processes ......................................................... 11-7
Evergreen Economics Page iii

11.2.2 Portfolio Optimization .................................................................................................. 11-7
11.2.3 Stakeholder Engagement .............................................................................................. 11-9
11.3 PROJECT SELECTION PROCESS ....................................................................................... 11-10
11.3.1 Administrator Project Selection Processes ............................................................... 11-10
11.3.2 Administrator Coordination ...................................................................................... 11-11
11.3.3 Match Funding ............................................................................................................. 11-12
11.3.4 Intellectual Property Terms ........................................................................................ 11-12
11.3.5 Flexibility ....................................................................................................................... 11-13
11.4 PROJECT ASSESSMENT PROCESS .................................................................................... 11-14
11.4.1 Project Status Reports .................................................................................................. 11-14
11.4.2 Benefits Quantification ................................................................................................ 11-15
11.4.3 Results Dissemination ................................................................................................. 11-16
11.4.4 Project Networks .......................................................................................................... 11-17
11.5 PROJECT IMPACTS AND POLICY ALIGNMENT ............................................................... 11-18
11.6 OVERARCHING COORDINATION AND COLLABORATION ............................................. 11-20
11.7 ON-GOING PROGRAM EVALUATION ............................................................................ 11-21
12 APPENDIX A: CPUC DECISION DETAIL ..................................................................... 12-1
1. CPUC DECISION 13-11-025 ATTACHMENT 4 (LIST OF POTENTIAL EVALUATION
AREAS) ............................................................................................................................. 12-1
2. CPUC DECISION 12-05-037 STATUTORY GUIDANCE .................................................... 12-5
13 APPENDIX B: LOGIC MODELS ...................................................................................... 13-1
13.1 ADMINISTRATION LOGIC MODEL ................................................................................... 13-1
13.2 APPLIED R&D LOGIC MODEL ......................................................................................... 13-5
13.3 TD&D LOGIC MODEL ................................................................................................... 13-13
13.4 MARKET FACILITATION LOGIC MODEL ........................................................................ 13-21
13.4.1 Foster Commercialization ........................................................................................... 13-26
13.4.2 Facilitate Procurement ................................................................................................ 13-26
13.4.3 Enhance Permitting ..................................................................................................... 13-27
13.4.4 Conduct Market and Technical Analysis ................................................................. 13-28
13.4.5 Mid-Term & Long-Term Outcomes P ....................................................................... 13-28
13.4.6 External Influences ...................................................................................................... 13-28
13.5 SELECTED REFERENCES FOR APPENDIX B ..................................................................... 13-29
14 APPENDIX C: BEST PRACTICES ASSESSMENT RESULTS MEMORANDUM . 14-1
14.1 LITERATURE REVIEW ....................................................................................................... 14-1
14.1.1 Introduction .................................................................................................................... 14-1
14.1.2 Summary of Findings .................................................................................................... 14-2
14.1.3 Conclusions, Lessons Learned, and Recommendations .......................................... 14-9
14.1.4 Appendix A: Programs and Documents Included in the Review ........................ 14-10
Evergreen Economics Page iv

14.2 IN-DEPTH INTERVIEWS .................................................................................................. 14-17
14.2.1 Introduction .................................................................................................................. 14-17
14.2.2 Summary of Findings .................................................................................................. 14-18
14.2.3 Conclusions ................................................................................................................... 14-24
15 APPENDIX D: EXAMPLE OF PROJECT TRACKING RECOMMENDATION ..... 15-1
16 APPENDIX E: SUPPLEMENTAL INFORMATION ON EPIC PROJECT
PERFORMANCE ........................................................................................................................ 16-1
16.1 APPLIED RESEARCH AND DEVELOPMENT PROJECTS ...................................................... 16-1
16.1.1 Knowledge Creation and Knowledge Utilization (Logic Model Boxes H, I,
N, T) ................................................................................................................................ 16-2
16.1.2 Private and Public Investment (Logic Model Box V) ............................................... 16-5
16.1.3 Follow-on Applied R&D (Logic Model Box P) .......................................................... 16-7
16.1.4 Technology Demonstrations (Logic Model Box U)................................................... 16-7
16.1.5 Technology Adoption (Logic Model Boxes Q, R, S, and T) ..................................... 16-8
16.2 TECHNOLOGY DEMONSTRATION AND DEPLOYMENT PROJECTS ................................... 16-8
16.2.1 Knowledge Creation and Knowledge Utilization (Logic Model Boxes F, H,
N and S)........................................................................................................................... 16-9
16.2.2 Private Investment (Logic Model Boxes M and O) ................................................. 16-13
16.2.3 Development of Partnerships and Investors (Logic Model Box O)...................... 16-15
16.3 MARKET FACILITATION PROJECTS ................................................................................ 16-18
16.3.1 Background and Overview ........................................................................................ 16-18
16.3.2 Funding (Logic Model Box C) .................................................................................... 16-19
16.3.3 Knowledge Creation and Knowledge Utilization (Logic Model Boxes E, H,
K, N, F, I, L and O)....................................................................................................... 16-19
16.3.4 Project Scalability (Replicability) (Logic Model Boxes F, I, L and O) ................... 16-24
16.3.5 Follow-On Research (Logic Model Boxes F, I, L and O) ........................................ 16-24
17 APPENDIX F: SELECTED REFERENCES REGARDING THEORY-DRIVEN
EVALUATION ............................................................................................................................ 17-1
Evergreen Economics Page v

List of Key Terms
Term Description
Administrators The four entities that the CPUC authorized to administer EPIC: the
California Energy Commission (CEC), Pacific Gas and Electric Company
(PG&E), San Diego Gas & Electric (SDG&E), and Southern California
Edison (SCE).
Investor-Owned Utilities The three regulated electric utilities, which are EPIC administrators:
PG&E, SDG&E and SCE.
The Legislature The California State Legislature
Triennial Investment Plans Required investment planning document to be completed by each
(or Investment Plans) administrator on a three year cycle.
Annual Reports Reports that the CPUC requires the administrators to develop and
distribute at the end of each program year, which are due each
February 28.
Investment Areas/Program The defined categories in which the CPUC authorized the
Areas/Funding Categories administrators to conduct EPIC projects: Applied Research and
Development (Applied R&D), Technology Demonstration and
Deployment (TD&D) and Market Facilitation. The IOUs’ role is limited
to the area of TD&D.
Innovation A product innovation is defined as the introduction of a good or service
that is new or significantly improved with respect to its characteristic
or intended uses. Product innovations can utilize new knowledge or
technologies or be based on new uses or combinations of existing
knowledge or technologies.
Program Theory and Logic A narrative (program theory) and accompanying diagram(s) that
Model describe Program inputs and activities and how they combine to
produce expected outputs which, in turn, may produce expected short-
term, mid-term and long-term outcomes. Each pathway or linkage in
the logic model describes a hypothesized cause and effect relationship.
Theory-Based Evaluation An evaluation approach and accompanying framework that is often used
Framework for complex programs such as EPIC whose benefits are intended to be
realized over a long time period. The approach relies on the
development of logic models and associated indicators of progress to
identify plausible causal mechanisms and test related hypotheses in
order to assess the extent to which major components of the Program
are successfully implemented and build a case for attribution. For a list
of references that provide detail on theory-driven evaluation, please see
Section 17: Appendix F.
Applied Research and Includes activities supporting pre-commercial technologies and
Development (Applied R&D) approaches that are designed to solve specific problems in the
Evergreen Economics Page vi

Term Description
electricity sector.
Technology demonstration Installation and operation of pre-commercial technologies or strategies
and deployment (TD&D) at a scale reflective of anticipated operating environments to enable
appraisal of the operational and performance characteristics and
financial risks.
Market Facilitation Refers to a range of activities including program tracking, market
research, education and outreach, regulatory assistance and
streamlining and workforce development to support clean energy
technology and strategy deployment.
Electric System Value Chain Consisting of:
Elements • Grid operations/market design
• Generation
• Transmission
• Distribution
• Demand side management
IOU Investment Framework A framework the IOUs developed to structure their Investment Plans,
Elements which includes four investment areas (all within the TD&D area) that
the IOUs have identified as critical areas on which to focus in order to
modernize the grid:
• Renewables and Distributed Energy Resources (DER) Integration –
focuses on renewables and distributed energy resources integration,
and supports the state’s Renewable Portfolio Standard, greenhouse
gas emission reduction and energy storage goals.
• Advanced Asset Management and Optimization – focuses on grid
modernization and optimization, and addresses SB 17 and smart
grid planning and implementation.
• Customer Products/Service Enablement and Integration – focuses
on the integration of demand side management with the smart grid,
and supports Zero Net Energy policies.
• Cross-Cutting/Foundational Strategies and Technologies – focuses
on smart grid architecture, cybersecurity, telecommunications and
standards development.
Grant Funding Opportunity The CEC has issued these two types of solicitations for EPIC.
and Program Opportunity
Notice Solicitations
Notice of Proposed Award The CEC issues a NOPA to announce the winning bidder.
(NOPA)
Grantee The bidder who receives a grant from the CEC to conduct EPIC
projects under the direction of the CEC’s Commission Agreement
Manager (CAM).
Evergreen Economics Page vii

Term Description
Vendor The contractor who receives a contract from the IOUs to conduct a
component of an EPIC project under the direction of the IOUs.
Program Manager For CEC projects, the person designated by the bidder to oversee the
project and to serve as the main point of contact from the grantee’s
organization. For IOU projects, the IOU staff member who serves as
the main point of contact for an EPIC project.
Technology/Knowledge A plan that the CEC requires grantees to complete and implement that
Transfer Plan describes how they will disseminate project-related knowledge and
results through channels such as published articles, presentations at
conferences and workshops, and dissemination of information on
grantee websites or via social media platforms.
The Evergreen team The evaluation team composed of Evergreen Economics, NMR Group,
Inc., Ridge & Associates, Jai J. Mitchell Analytics and Advanced Survey
Design
Peer RD&D programs Other Research, Development and Demonstration programs that we
reviewed during the course of the best practices assessment, to which
we compare EPIC.
Smart Grid An electricity supply network that uses digital communications
technology to detect and react to local changes in usage.
Regional Energy Innovation Provides key and coordinated assistance, resources, and infrastructure
Cluster needed by entrepreneurs and researchers in the region to successfully
bring to market energy innovations that can benefit IOU electric
ratepayers. Composed of a concentration of interconnected companies,
universities, investors, business incubators and business accelerators
that stimulate innovative activity by promoting intensive interaction and
collaboration, sharing of facilities, competition and promotion of
entrepreneurship.
Comprehensive evaluation An evaluation that includes elements of a process evaluation, a
formative evaluation, and an impact evaluation.
Process evaluation A form of program evaluation designed to determine whether the
program is delivered as intended to the target recipients, also known as
implementation assessment.
Formative evaluation Evaluation activities undertaken to furnish information that will guide
program improvement.
Impact evaluation An evaluation study that answers questions about program outcomes
and the conditions it is intended to ameliorate. Also known as outcome
evaluation.
Evergreen Economics Page viii

Acronyms and Abbreviations
ACE The New York State Energy Research and Development Authority's (NYSERDA's)
Program Advanced Clean Energy Exploratory Research Program
ARPA-E The Department of Energy’s Advanced Research Projects Agency-Energy Program
CAISO California Independent System Operator
CalSEED California Sustainable Energy Entrepreneur Development Initiative
Commission Agreement Manager, individual designated by the California Energy
CAM Commission to oversee the performance of EPIC contracts resulting from a solicitation
and to serve as the main point of contact for the recipient.
CEC California Energy Commission
CEDMC The California Efficiency + Demand Management Council
CPUC California Public Utilities Commission
CVD Commercialization Valley of Death
DER Distributed Energy Resources
DG Distributed Generation
DOE Department of Energy
DRP Distributed Resource Plan
DSM Demand Side Management
DVBE Disabled Veteran Business Enterprise
Electric Program Investment Charge, the source of funding for the projects awarded
EPIC
under this solicitation.
EPRI Electric Power Research Institute
Energy Research & Development (ERD) Division – the CEC manages EPIC within its
ERD
ERD Division
EV Electric Vehicle
GFO Grant Funding Opportunity
GHG Greenhouse Gas
IOU Investor-Owned Utility, including PG&E, SDG&E and SCE
IP Intellectual property
IRP Integrated Resource Plan
NOPA Notice of Proposed Awards
NYSERDA New York State Energy Research and Development Authority
Evergreen Economics Page ix

OECD Organization for Economic Co-operation and Development
PAC Policy Advisory Committee
PG&E Pacific Gas and Electric Company
PIER Public Interest Energy Research program (EPIC’s predecessor)
PON Program Opportunity Notice
RD&D Research Development and Demonstration
RFP Request for Proposals
RPS Renewable Portfolio Standard
SBIR Department of Energy’s Small Business Innovation Research Program
SCE Southern California Edison
SDG&E San Diego Gas & Electric
STTR The Department of Energy’s Small Business Technology Transfer Program
T&MD Technology and Market Development – a NYSERDA RD&D program
TAC Technical Advisory Committee
TRL Technology Readiness Level
TVD Technological Valley of Death
ZEV Zero Emission Vehicle
ZNE Zero Net Energy
Evergreen Economics Page x

1 Executive Summary
This report documents the results of an evaluation of the Electric Program Investment
Charge (EPIC, or the Program), which is a research, development and demonstration
(RD&D) program that funds a broad portfolio of innovations1 that seeks to advance the
frontiers of energy science and technology. The California Public Utilities Commission
(CPUC) established and oversees EPIC, which is administered by four entities: the
California Energy Commission (CEC), Pacific Gas and Electric
Company (PG&E), San Diego Gas & Electric (SDG&E) and By the end of 2016, a
Southern California Edison (SCE). PG&E, SDG&E and SCE are total of 19 EPIC
the three electric California investor-owned utilities (IOUs). projects were
CEC projects cover three program areas: 1) Applied Research completed and 250
and Development (Applied R&D), 2) Technology were in progress.
Demonstration and Development (TD&D) and 3) Market
Facilitation, while the IOU projects are focused only on TD&D.
Table 1 shows the total program budget for the first two Triennial Investment Plans
covering program years 2012–2017.
Table 1: EPIC Program Budget (2012–2017)

Budget Percent of
Administrator (2012–2017) Budget
CEC $696,804,500 80%
PG&E $89,000,000 10%
SCE $72,754,534 8%
SDG&E $15,540,000 2%
Total $874,099,034 100%
Source: Administrators’ 2016 Annual Reports
1.1 Evaluation Objectives and Approach

The objective of the study was to conduct a comprehensive evaluation of EPIC to identify
opportunities to improve program management and effectiveness. To this end, specific
objectives included the following:
1A product innovation is defined by the Organization for Economic Co-operation and Development (OECD)
as “. . . the introduction of a good or service that is new or significantly improved with respect to its
characteristic or intended uses. Product innovations can utilize new knowledge or technologies or be based
on new uses or combinations of existing knowledge or technologies" (Tinguely 2013).
Evergreen Economics Page 1-1

• Determining if the Program is being implemented in a manner consistent with the
program objectives, requirements and intent of the CPUC and the California
Legislature as set forth in a series of CPUC Decisions;2
• Assessing the extent to which the Program supports key energy policies and public
research code sections;
• Identifying best practices in research administration;
• Assessing the extent to which the Program is on track, thus far, in meeting its
objectives to provide ratepayer benefits, advance energy innovation and support
California’s energy policy goals; and
• Providing recommendations for improvements to program requirements and
practices.
The evaluation results were intended to support and inform the CPUC’s consideration of
EPIC within the next triennial application proceeding, which was initiated in the spring of
2017 with the administrators’ 2018-2020 Investment Plan application filings.
For the purposes of focusing this evaluation on the most fundamental program
characteristics, CPUC staff established EPIC’s core, overarching goals (or core values).
1. Electricity IOU Ratepayer Benefits: Providing electric The evaluation

ratepayer benefits was established as the mandatory focused on EPIC’s core
guiding principle of EPIC, which the CPUC defined as values of:
promoting greater reliability, lower costs and increased • providing ratepayer
safety.
benefits;
2. Energy Innovation: The CPUC established a range of • advancing energy
administration and program requirements intended to innovation; and
ensure that EPIC funds Research Development and • supporting
Demonstration (RD&D) activities that assist the California’s energy
emergence of innovative energy technologies and policy goals.
services, for the benefit of California’s electricity IOU
ratepayers and the public interest. Overall, advancing
true, productive energy innovation is a core purpose of the Program. This goal
should be supported not only in EPIC’s technical program areas, but also in EPIC’s
administrative structure. Therefore, all program decisions and activities—from the
setting of research priorities to the treatment of intellectual property concerns and
the administrative balancing of due process with flexibility and responsiveness—
should support, embody and advance true, productive energy innovation.
3. California’s Energy Policy Goals: As the CPUC’s EPIC decisions make clear,
supporting California’s key energy, climate and economic policy goals is one of the
2 D.11-12-035, D.12-05-037, D.13-11-025, D.15-04-025, and D.15-09-005.

core justifications of the Program that drives much of EPIC’s governance and
requirements. Each EPIC funding decision and activity must be made and
implemented in the context of these policy goals.
Evergreen Economics, along with NMR Group, Inc., Ridge & Associates, Jai J. Mitchell
Analytics and Advanced Survey Design, employed a theory-driven evaluation framework
within which we assessed the Program’s effectiveness, guided by a series of logic models
that we developed. We conducted several research activities including review of program
documents and data and in-depth telephone interviews with program administrators,
project managers, contractors and stakeholders. We also conducted a best practices
assessment that included a literature review and in-depth telephone interviews, as well as
a network analysis.
1.2 Key Findings and Recommendations

Below, we summarize the highest priority findings and recommendations, in order of
priority. A complete set of recommendations is provided in Section 11: Findings and
Recommendations, in which we number recommendations sequentially by section (which
we have retained in the Executive Summary).
1.2.1 Program Requirements

The Program is guided by a series of CPUC Decisions that establish the purpose and
governance for EPIC and provide a policy rationale for continuing public interest funding
in the energy area where private capital is unlikely to provide adequate support. To
ensure that the Program is administered consistently with its core values, the Decisions
establish the requirements and administrative procedures for EPIC.
Based on our review of program administrative procedures

Key finding: The
(including program document review and interviews with
administrators are in
administrators), we find that the four administrators are in
compliance with the
compliance with program requirements. However, we
letter of EPIC
identified areas where the administrators are technically
program
compliant but could better fulfill the spirit or intent of the
requirements, but
requirements (such as stakeholder engagement, coordination
could better fulfill the
and information sharing). Likewise, we identified other cases
spirit of some
where the minimum requirements are not sufficient to ensure
requirements.
best-in-class program administration. We discuss these issues
below.
1.2.2 EPIC Project Impacts

The Evergreen team employed a theory-driven evaluation framework within which we
assessed the Program’s effectiveness, guided by a series of logic models that we developed
to support the theory-based research design. For a complex program such as EPIC, our

evaluation team developed performance metrics for each activity, output and outcome to
assess the extent to which major activities of EPIC have been and are being successfully
implemented and whether these activities had led to or are likely to lead eventually to the
expected short-, mid- and long-term outcomes. We also assessed the extent to which the
projects are aligned with the Program’s objectives.
Overall, the EPIC portfolio appears to be on track, thus far, in

meeting its objectives to provide ratepayer benefits, advance Key finding: Each
energy innovation and support California’s energy policy project in the EPIC
goals. Collectively, EPIC is both broad and deep, and project portfolio is
administrators take steps to integrate projects into the broader meeting its objectives,
innovation and policy landscape. To that extent, projects but it is unclear if the
appear to be consistent with the Program’s objectives and portfolio as a whole is
core values. However, as a portfolio, EPIC may not be fully optimized.
optimized to best support energy policy and innovation, which
we describe in the next sub-section.
Since 2012, the EPIC administrators have funded hundreds of projects and will continue to
do so for the foreseeable future. The combined efforts of the past, present and future
projects is expected to move EPIC incrementally closer to achieving California’s important
public policy goals. Given the on-going nature of EPIC and its long-term objectives, we
recommend that:
• 7a) Using the theory-driven framework developed for this evaluation, monitor and
report key performance metrics on an on-going basis and conduct a comprehensive
evaluation every three to four years. All of these evaluation activities should be
conducted by an independent evaluator in close collaboration with the four
administrators to avoid any duplication of efforts and to ensure that the results
will be useful to all stakeholders (e.g., the CPUC, state legislators, and the four
administrators and other stakeholders). While this evaluation report documents
what is working and what could be improved, the Program is still very young and
should undergo ongoing independent assessment to ensure it remains on track and
addresses the issues we have noted. Moreover, most projects have yet to be
completed, and independent review is needed in the future to assess project
benefits as the Program matures. Conducting independent program evaluations is
consistent with the best practice of peer RD&D programs.
• 7b) The administrators create a single, centralized database containing all relevant
information on active and completed EPIC projects along with monitoring and
quarterly reporting of key performance metrics, in order to support the on-going
evaluation of the Program.

1.2.3 Optimizing EPIC’s Portfolio
Given the many policy areas EPIC is attempting to address,
Key finding: There is
we have identified a need to prioritize the guiding
a need to prioritize
principles, policies and strategic objectives, and
among EPIC’s many
operationalize what it means for a portfolio to be
objectives.
optimized. There is no clear set of priorities EPIC is seeking
to address, or prioritization of research gaps or needs. For
example, we confirmed that every EPIC project may be likely to provide ratepayer
benefits, but there is variation in how broad and/or direct those benefits are. We also
confirmed that every EPIC project supports at least one energy policy area, but there are
many relevant policy areas that lack clear prioritization. Projects are not categorized or
tracked by technology or policy area, making it difficult to assess the effectiveness of EPIC
on advancing key policy. We also observed that EPIC focuses less on commercialization
than peer RD&D programs, though we note that EPIC’s objectives are much broader.
As stated above, we can determine that program administrative procedures and EPIC
projects meet regulatory requirements, but without clear priorities for the Program, we
cannot determine whether the portfolio of projects collectively is truly optimized. To
ensure that EPIC is generating the optimal mix of projects that maximize ratepayer
benefits, lead to energy innovation and support the state’s key policy goals, we
recommend that:
• 2a) The CPUC establish priorities among its current policy goals and funding
criteria to better guide the administrators in their
investment planning. In Section 1.2.5, we
• 2b) The administrators collaborate in categorizing and recommend that the
summarizing projects (such as by technology type administrators convene
and/or policy area) and review projects by topic areas an independent body
to ensure that the portfolio of projects effectively to support an effort to
supports key policy goals. categorize projects.
1.2.4 Effectively Engaging Stakeholders

Our analysis of knowledge dissemination activities and relational networks suggests that
project teams are actively engaged in developing networks of stakeholders and other
market actors and are well positioned to disseminate knowledge widely once projects are
completed. However, EPIC's four-administrator model has limitations compared to peer
RD&D programs that are administered by a single organization. For EPIC, there is no
single administrator that can effectively convey information to stakeholders nor is there a
program-wide communications mechanism or central repository of project information.
As a result, information dissemination and stakeholder engagement is less than optimal.

Further contributing to the inherent limitations associated with the four-administrator
model is that each administrator relies primarily on an Annual Report to disseminate
information on project status and results. However, this does not optimally engage
stakeholders or support effective CPUC oversight. While the administrators are in
compliance with program reporting requirements, there is room for improvement by
following the best practices of peer RD&D programs.
In order to ensure that the Program is generating a set of projects that most effectively
advance energy innovation, and that the administrators more effectively engage
stakeholders and facilitate CPUC oversight, we recommend that the administrators
share project information more frequently and do so in a more coordinated and topical
manner. We recommend that:
• 4a) The administrators share information while projects are in progress with the
CPUC and the public on a more frequent basis, such as quarterly. The Annual
Reports, on their own, are not the most effective way to disseminate information
about EPIC projects. These quarterly reports should:
o Be distributed via a single EPIC website and listserv, so that the CPUC and
stakeholders can more easily obtain information about all projects without
having to review four separate reports. Included on this website would be a
downloadable centralized Excel spreadsheet that contains key information for
all EPIC projects. This would ensure that stakeholders have an easy way to
obtain all relevant information about EPIC projects that supports their particular
areas of interest.
o Categorize projects (as recommended previously) In Section 1.2.5, we
by technology and/or policy, with sort / filter recommend that the
capability, so the CPUC and stakeholders can administrators convene
more easily obtain information about projects in a an independent body to
particular category without having to search support such
through long lists of individual projects. coordination efforts.
o Include current information about project
outcomes such as recent or upcoming presentations, publications and interim
knowledge dissemination. See Appendix D for an example of a quarterly status
report that we developed that provides a recommended starting point for such a
report.
• 4b) The administrators collaborate and jointly convene a quarterly workshop to

share results about project status and lessons to-date on a topical basis, with
engagement from stakeholders on topics that are of interest. Such workshops
should be publicized in advance along with the topic or topics to be covered. All
EPIC projects that fall under the announced topic should be discussed and
organized topically. In general, EPIC stakeholder workshops are not organized

topically, and IOU projects are typically presented in an ad hoc fashion, which is
not an effective way to engage stakeholders.
We also have identified a need for the administrators to coordinate on compiling and
jointly reporting on project benefits. There is no central place where project results and
benefits are summarized. Reports that document project results are included in the
administrators’ Annual Reports and posted on websites, and additional dissemination
plans are developed on a project-by-project basis. A policymaker, legislator or other
stakeholder would have a difficult time determining the aggregate impact of the Program.
We recommend that:
• 4d) The administrators develop a process to jointly report on EPIC’s short-, mid-
and long-term project benefits across the portfolio on a routine basis (e.g., annually)
to the CPUC, relevant stakeholders and the general public.
1.2.5 Independent Coordination Body

Our evaluation has identified a critical need for improving administrative coordination
and stakeholder engagement that the administrators are not currently addressing due to
limitations associated with the administrative model and their reliance on minimum
project reporting procedures. We have identified a need to explicitly supplement the
existing administrative structure by convening an
independent body that provides coordination and Key finding: There is a
facilitation support to the administrators and compiles and need to supplement
helps disseminate information. Such efforts would increase the administrative
transparency and stakeholder engagement and ensure the structure by
Program is most effectively directing EPIC funds toward convening an
energy innovation that meets the highest priority state policy independent body to
goals, as identified by the CPUC. These efforts would also coordinate, facilitate
address the deficiencies we outlined in the previous two sets and lend technical
of findings. We recommend that: expertise.
• 6a) The CPUC and/or the administrators fund and

convene an independent body to coordinate, facilitate and lend technical expertise.
The highest priority areas such a body could support include:
o Convening and engaging stakeholders earlier in the investment planning
process;
o Engaging stakeholders and ensuring any input that would lead to greater
ratepayer and state policy benefits is considered by the administrators in their
Investment Plans;
o Supporting administrator and CPUC efforts to track and prioritize policy goals
and funding criteria, and periodically revisiting priorities as policy goals change
and EPIC matures;

o Supporting administrator and CPUC efforts to ensure that those priorities are
effectively addressed in the administrators’ Investment Plans;
o Supporting administrator efforts to categorize projects by technology and/or
policy areas, to facilitate easier access of EPIC project information for interested
stakeholders;
o Reviewing administrator project research plans and quarterly status reports,
such as by policy and/or technology areas; tracking related developments in
CPUC proceedings and engaging relevant stakeholders in projects of interest;
and helping to identify issues or concerns;
o Planning and facilitating a quarterly meeting devoted to a particular topic of
interest to stakeholders, including publicizing the meetings to stakeholders and
addressing their needs;
o Coordinating an effort to develop a centralized EPIC website, database and
listserv; helping to identify interested parties; and ensuring that those parties are
linked to relevant information on projects and topic areas of interest; and
o Identifying interested stakeholders and appropriate forums for administrators
to more broadly disseminate their results.
1.2.6 IOUs in Compliance, but Not Meeting Intent of All Requirements

The IOUs are technically in compliance with EPIC program requirements, but many of
their administrative practices are inconsistent with best practices we identified among
peer RD&D programs. We examined the numerous CPUC requirements that fell into the
following nine categories.
• Statutory guidance
• Investment Plans
• Limitations on projects
• Contracts
• Stakeholder engagement
• Quantifying benefits/metrics
• Budget
• Annual reports
• Miscellaneous
To verify compliance with these requirements, we relied on a combination of sources,

including program filings (e.g., Annual Reports and Investment Plans), the sample of
projects for which we had more detailed information (supplemented by interviews with
grantees, vendors and IOU/CEC project managers) and the sample of CEC
solicitations/bids and IOU request for proposals (RFPs) and vendor bids. We then

compared the IOU administrative practices to other peer RD&D program practices and
observed a number of areas where performance could be improved. For example, project
selection, transparency and stakeholder engagement could be improved. By adopting
some or all of the best practices we identify below, EPIC would be more effective at
producing energy innovation that is more explicitly supportive of state energy policies.
Table 2 summarizes key differences (and a few similarities) in CEC and IOU
administrative processes as compared to other peer RD&D programs. A check mark
indicates that the administrative process is consistent with peer programs.

Table 2: Comparison of EPIC Administrative Processes to PEER RD&D Programs
Administrative
Process CEC IOUs
Program Awards grants to external organizations Conduct their research using internal
Management and that conduct their research. staff with use of vendors.
Administration ✓
Administration is managed by a core team with RD&D program expertise,
with technical support provided by both internal and external experts.
Four-administrator model.
Investment Identifies a series of strategic objectives Develop Investment Plan priorities
Planning with strong and transparent linkages internally, predominantly relying on
to state policy goals. their own technical experts and
✓ management to identify and prioritize
research areas, with linkages to policy
less transparent.
Relies mostly on input from multiple Rely mostly on external input from a
external stakeholders; develops its single utility-focused stakeholder;
Investment Plans transparently and insufficient transparency in
engages external stakeholders developing Investment Plans.
throughout the process.
✓
Project Selection Uses a transparent and public Use a less transparent, internal
process for selecting projects and shares process for selecting projects and do
project scopes of work in a timely not share project scopes of work in a
manner. timely manner.
✓
Due diligence is being done to identify projects that, absent EPIC funding, would not
move forward or would move forward more slowly.
✓
Project Shares information about projects while they are being implemented but less
Assessment frequently than optimal.
Uses a robust process for collecting Lack a robust process for collecting
the necessary quantitative data needed to the necessary data needed to
comprehensively report on project comprehensively report on project
benefits and disseminate results. benefits and disseminate results.
✓
= consistent with peer RD&D program practices

Note that we also examined each of the four administrators’ processes and projects. When
we compared across the four administrators, the main finding was the difference in how
the CEC approaches program administration as compared to the IOUs. However, we did
note some differences in processes across the three IOUs in this report, which we discuss
in the report results and findings sections. Note that those comparisons were limited by
the size of the project sample for each IOU (which reflects the relatively smaller portion of
the EPIC budget that the IOUs receive as compared to the CEC).
While each IOU project is related to at least one area of the state’s energy policy, the IOUs’
internal needs, rather than energy policy, most often determine which projects are
implemented. There is obviously overlap between the IOUs’ internal needs and state
energy policy, but the mix of projects the IOUs select may not be the most optimal to
advance the highest priority state policy areas. Once priorities are set, it will be easier to
assess how optimal the set of IOU projects are in advancing a particular energy policy.
The IOUs’ TD&D portfolio focuses on a much narrower set of investment areas as
compared to the CEC and peer RD&D programs. Moreover, the IOUs have a much
narrower stakeholder group from which they solicit project ideas as compared to the CEC
and peer RD&D programs. As stated above, the projects the IOUs implement are each
related to one or more energy policy areas, but the framework and stakeholder input that
shapes their projects leads to a relatively narrow focus. At this time, we are not able to
comment on whether internally driven and narrowly focused investment planning and
project selection are problematic, but the intent of the program requirements related to
stakeholder engagement and transparency are not being met fully with current IOU
investment planning and project selection processes.
In order to improve the transparency and comprehensiveness of IOU investment planning

information sharing and linkages to state policy priorities, we recommend that:
• 2d & 2e) The IOUs engage more stakeholders earlier in the investment planning
process and provide more comprehensive information about their plans prior to
and at investment planning workshops to allow time for more meaningful
engagement. Currently, the IOUs present incomplete
information to stakeholders, and mostly rely on Key finding: The
internal staff expertise along with the Electric Power IOUs, while
Research Institute's (EPRI’s) input to shape their plans. technically in
Above, we recommended that the administrators compliance with
and/or the CPUC form a new coordination body, program
which could also support the shaping of the IOUs’ requirements, could
Investment Plan priorities to ensure they complement improve upon
the CEC’s Investment Plan and are responsive to the information sharing
priorities established by the CPUC. and stakeholder
engagement.

There is a lack of transparency in the IOUs’ project selection criteria, project selection and
research planning processes. In addition, the IOUs are not effectively tracking and
reporting on benefits metrics.
Consistent with the issues identified above, while the IOUs are meeting the minimum
requirements, they are not operating in a transparent and inclusive manner that would
maximize the value that stakeholders could offer. Since the CPUC typically only has access
to the same minimal information that stakeholders do, except on an ad hoc basis when
staff make inquiries, increasing the completeness and frequency of information sharing
would also improve oversight of the Program and increase its support of the highest
priority energy policy areas. Our previous recommendation to categorize and report on
projects by topic areas of interest would greatly improve the usefulness of project
information for both stakeholders and the CPUC.
To improve the transparency of the IOUs’ project selection processes, we recommend that:
• 3a) The IOUs develop more transparent project selection criteria, which determine
the project areas that are described in their Investment Plans as well as the specific
projects that are eventually implemented. Once the CPUC establishes priorities,
these criteria could be reviewed and revised over time to ensure an appropriate
focus on the highest priority areas for advancing state energy policy.
• 3b) The IOUs share project research plans and budgets with the CPUC and the
public, at least one month prior to launch. The coordination body we recommend
establishing could support efforts to classify projects and disseminate such
information in a coordinated manner across administrators so the information
would be more readily obtained by interested parties, increasing the benefits
generated by EPIC projects.
We offer additional recommendations in Section 11 that focus on program administration

details.

2 Program and Policy Background
2.1 Program Background
The Electric Program Investment Charge (EPIC, or the Program) is an innovation funding
program that seeks to advance the frontiers of energy science and technology. The
California Public Utilities Commission (CPUC) established and oversees EPIC. The
Program is administered by four entities: the California Energy Commission (CEC), Pacific
Gas and Electric Company (PG&E), San Diego Gas & Electric (SDG&E), and Southern
California Edison (SCE). PG&E, SDG&E and SCE are the three electric California investor-
owned utilities (IOUs). The CEC administers 80 percent of the EPIC budget, and the three
electric IOUs administer the remaining 20 percent.3
There are four primary CPUC Decisions that together established the requirements and
administrative procedures for EPIC:
1. Decision 12-05-037 (May 24, 2012), which established the purpose and governance
for EPIC and funding collections for 2013-2020 as Phase 2 Decision of Rulemaking
11-10-003.
2. Decision 13-11-025 (November 14, 2013), known as the “EPIC Decision”, approved
applications for the first program investment period of 2012-2014 (which is referred
to as “EPIC 1”) from the four EPIC administrators.
3. Decision 15-04-020 (April 9, 2015), approved applications for the second program
investment period of 2015-2017 (which is referred to as “EPIC 2”) from the four
EPIC administrators.
4. Decision 15-09-005 (September 17, 2015), addressed new EPIC projects that are
introduced by the IOUs between triennial funding cycles, requiring them to submit
Tier 3 advice letters.
The first Decision (12-05-037) established the purpose and governance for EPIC and
funding collections for 2013-2020, providing a policy rationale for continuing public
interest funding in the energy area where private capital is unlikely to provide adequate
support. Providing electric ratepayer benefits was established as the mandatory guiding
principle of EPIC, which the CPUC defined as promoting greater reliability, lower costs
and increased safety. In addition, complementary guiding principles were designed to
guide investment decisions, which include:
• Providing societal benefits;
3Of the 20 percent share of the EPIC budget allocated to the IOUs, PG&E administers 50.1 percent of the IOU
share, SDG&E administers 8.8 percent, and SCE administers 41.1 percent.

• Reducing greenhouse gas (GHG) emissions in the electricity sector at the lowest
possible cost;
• Supporting California’s loading order to meet energy needs, first with energy
efficiency and demand response, then with renewable energy (distributed
generation and utility scale), and third with clean conventional electricity supply;
• Supporting low-emission vehicles and transportation;
• Providing economic development; and
• Using ratepayer funds efficiently.
Program funding is approved in the following defined investment areas: Applied

Research and Development (Applied R&D), Technology Demonstration and Deployment
(TD&D) and Market Facilitation. The IOUs’ role is limited to the area of TD&D.
• Applied Research and Development (Applied R&D) – Applied R&D activities

support pre-commercial technologies and approaches that are designed to solve
specific problems in the electricity sector.
• Technology Demonstration and Deployment (TD&D) – TD&D addresses the
installation and operation of pre-commercial technologies or strategies at a scale
sufficiently large enough and in conditions sufficiently reflective of anticipated
actual operating environments to enable appraisal of operational and performance
characteristics and financial risks.
• Market Facilitation – Market Facilitation refers to a range of activities including
program tracking, market research, education and outreach, regulatory assistance
and streamlining and workforce development to support clean energy technology
and strategy deployment.
In addition to the three investment areas, projects must be mapped to at least one of the
different elements of the electricity system value chain, which consists of:
• Grid operations/market design

• Generation
• Transmission
• Distribution
Section 6 provides more information about EPIC program requirements and how the
Program is administered by the CEC and the IOUs.

2.2 California Energy Policy
EPIC is intended to support the state’s energy policy priorities. In this subsection, we
provide a brief summary of some of the most relevant energy policy areas to provide
context for the remainder of this section. The Evergreen team drew from the
administrators’ Investment Plans and secondary research on energy policy to develop
these summaries.
2.2.1 Greenhouse Gas Reduction Legislation and the Renewable Portfolio

Standard
On October 7, 2015, California State Senate Bill 350: Clean Energy and Pollution Reduction
Act4 (SB 350) was signed into law, establishing new clean energy, clean air and GHG
reduction goals for 2030 and beyond. SB 350 is considered to be the most significant
climate and clean energy legislation since the 2006 passage of Assembly Bill 32: California
Global Warming Solutions Act5 (AB 32) that set the statewide goal of reducing GHG
emissions to 1990 levels by 2020. In addition, AB 32 directed and authorized various state
agencies to engage in actions necessary to achieve this goal. Building off of AB 32, SB 350
established California’s 2030 GHG reduction target of 40 percent below 1990 levels. To
achieve this goal, SB 350 sets ambitious 2030 targets for energy efficiency and renewable
electricity, among other actions aimed at reducing GHG emissions. SB 350 is intended to
greatly enhance the state’s ability to meet its long-term climate goal of reducing GHG
emissions to 80 percent below 1990 levels by 2050.
SB 350 increases California’s renewable electricity procurement goal from 33 percent by

2020 to 50 percent by 2030. This will increase the use of Renewable Portfolio Standard6
(RPS) eligible resources, including solar, wind, biomass and geothermal. In addition, SB
350 requires the state to double statewide energy efficiency savings in electricity and
natural gas end uses by 2030. To help ensure these goals are met and the GHG emission
reductions are realized, large utilities will be required to develop and submit Integrated
Resource Plans (IRPs) to the CPUC. These IRPs will detail how each entity will meet its
customers’ resource needs, reduce GHG emissions and ramp up the deployment of clean
energy resources.
Significant technology advancements and cost savings must be achieved in order to meet
SB 350’s requirements, and EPIC is well situated to help foster these changes.
4 De León, Chapter 547, Statutes of 2015.
http://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=201520160SB350
5 Nunez, Chapter 488, Statutes of 2006. http://www.leginfo.ca.gov/pub/05-06/bill/asm/ab_0001-
0050/ab_32_bill_20060927_chaptered.pdf
6 California’s aggressive RPS requires all electricity retailers, including IOUs, to serve 33 percent of their
retail sales with renewable energy procurement. The RPS is mandated under Public Resources Code
399.11.22.

2.2.2 Grid Infrastructure
In order to implement the RPS successfully, the CEC identified that it is necessary to
upgrade the state’s existing transmission and distribution systems. To achieve this, the
CEC supported legislation such as Senate Bill 17,7 which mandates implementing and
planning a smart grid to improve the efficiency, reliability, economics and sustainability of
electricity services.
The California Independent System Operator (CAISO) developed a smart grid roadmap
and associated architecture8 in 2010, which offers strategies to transition the grid to be
responsive to wind and solar energy and smart devices in response to the state’s energy
and environmental policy goals. Based on the roadmap, “the 'smart grid' is the application
of technologies to all aspects of the energy transmission and delivery system that provide
better monitoring, control and efficient use of the system.”9 The roadmap identifies
research needs that may be used by stakeholders to develop business models and policies.
The CEC, CPUC and CAISO are also collaborating on the development of a Roadmap for
the Commercialization of Microgrids in California,10 which will be completed by the end
of 2017. A CEC-commissioned study released in July of 201511 provided an assessment of
microgrids in California and offered recommendations for future research and
development investments.
Additionally, the Distribution Resources Plan Proceeding (R. 14-08-013) requires electrical
corporations to file distribution resources plans, which may include demonstration
projects, related to grid modernization and distributed energy resources (DER).12
In the first EPIC Decision (12-05-037), the CPUC required the IOUs to address the
applicability of the relevant public utility codes, which includes Public Utility Code 8360
statutory guidance regarding the smart grid. The code stipulates that: “It is the policy of
the state to modernize the state's electrical transmission and distribution system to
maintain safe, reliable, efficient, and secure electrical service, with infrastructure that can
7 Padilla. SB-17 Electricity: smart grid systems, Chapter 327. 2009.

8 California ISO. Smart Grid Roadmap and Architecture. 2010.
https://www.caiso.com/Documents/SmartGridRoadmapandArchitecture.pdf
9 ibid, p. 5.
10 California Energy Commission. "California Microgrid Roadmap."
http://www.energy.ca.gov/research/microgrid/
11 DVN GL. Microgrid Assessment and Recommendation(s) to Guide Future Investment. Prepared for the
California Energy Commission. 2015. http://www.energy.ca.gov/2015publications/CEC-500-2015-

071/CEC-500-2015-071.pdf
12 California Public Utilities Commission. Distribution Resources Plan (R.14-08-013).
http://www.cpuc.ca.gov/General.aspx?id=5071

meet future growth in demand and achieve all of the following, which together
characterize a smart grid.”13 See Appendix A for more detail.
2.2.3 Energy Efficiency and the Loading Order

The CEC and CPUC, along with the California Consumer Power and Conservation
Financing Authority, established an energy resource loading order to guide their energy
decisions in the 2003 Energy Action Plan14 (and the CEC’s Integrated Energy Policy
Report, which is described later in this section). The loading order consists of decreasing
electricity demand through energy efficiency and demand response, and meeting new
generation needs first with renewable and distributed generation resources, and second
with clean fossil-fueled generation. The loading order provides a foundation for energy
policies and decisions.
On September 18, 2008, the CPUC adopted California’s first Long Term Energy Efficiency
Strategic Plan,15 presenting a single roadmap to achieve maximum energy savings across
all major groups and sectors in California. The plan provides an integrated framework of
goals and strategies for saving energy, covering government, utility and private sector
actions, and holds energy efficiency to its role as the highest priority resource in meeting
California’s energy needs.
While supporting the loading order continues to be an official EPIC supporting principle,
and the loading order is still California policy broadly speaking, we note that SB 350 and
other current CPUC proceedings are investigating alternative approaches. For example,
the IRP process required by SB 350 mandates that energy procurement planning decisions
will be optimized for each electricity provider based on a wide range of constraints—with
GHG benefits a primary goal. In other words, energy efficiency may not always be placed
first, if another resource enables a better outcome (for example, if increased load from
electric vehicles in a specific location enables better integration of renewables). Other
proceedings, such as the Distribution Resources Planning Proceeding, are investigating the
varied situations in which different types of resources hold different value or are pursued
in different order.
2.2.4 Distributed Energy Resources

SB 350 required the CPUC to implement an integrated resource plan process to identify
optimal portfolios of resources to achieve the state’s GHG goals and meet the challenge of
13 California Code, Public Utilities Code 8360.

14 CEC, CPUC and the Consumer Power and Conservation Financing Authority. State of California Energy
Action Plan. 2003. http://www.energy.ca.gov/energy_action_plan/index.html

15 CPUC. Long Term Energy Efficiency Strategic Plan. 2008, updated in 2011.
http://www.cpuc.ca.gov/General.aspx?id=4125

renewables integration. In response, the CPUC issued a Distributed Energy Resources
Action Plan16 in 2016 that is intended to serve as a roadmap for decision-makers, staff and
stakeholders and to guide DER policies. The scope of the plan encompasses three groups
of related proceedings and initiatives: rates and tariffs; distributed grid infrastructure,
planning, interconnection and procurement; and wholesale DER market integration and
interconnection. The primary focus is on DER strategies that are sensitive to time and
location. The following subsections provide more information on distributed energy
resources.
Electric Vehicles
In a 2012 Executive Order (B-16-2012),17 California Governor Jerry Brown established
expectations for agencies to expedite zero emission vehicle (ZEV) commercialization. The
order directed California to “encourage the development and success of ZEVs to protect
the environment, stimulate economic growth and improve the quality of life in the state,”
with a long-term target of reaching 1.5 million ZEVs in the state by 2025. Subsequently, the
Governor developed a 2013 ZEV Action Plan18 that identifies specific strategies and actions
that state agencies will take to meet milestones of the Executive Order.
In response, the CEC and the CPUC, along with the ISO, are developing a vehicle to grid
integration road map19 to explore how electric vehicles (EVs) can provide grid services
while managing charging levels and implementing two way interactions between the
vehicles and the grid, and how to leverage smart grid technologies to support reliable grid
management. The CPUC has also initiated proceedings in response to California State
Senate Bill 626 Electrical Infrastructure: plug-in hybrid and electric vehicles,20 which
requires the CPUC to develop rules to overcome barriers to widespread use of private EVs
in the state.
The CPUC also works to support widespread transportation electrification through

implementing requirements set forth in SB 350, which directs the IOUs to file Applications
16 CPUC. California's Distributed Energy Resources Action Plan: Aligning Vision and Action. 2016.
http://cpuc.ca.gov/uploadedFiles/CPUC_Public_Website/Content/About_Us/Organization/Commission
ers/Michael_J._Picker/2016%20DER%20Action%20Plan%20FINAL.pdf
17 Office of Governor Edmund G. Brown Jr. Executive Order B-16-2012. 2012.
https://www.gov.ca.gov/news.php?id=17472
18 Governor's Interagency Working Group on Zero-emission Vehicles. 2013 ZEV Action Plan. 2013.
https://www.opr.ca.gov/docs/Governor's_Office_ZEV_Action_Plan_(02-13).pdf
19 California ISO. "Vehicle to grid integration roadmap."
http://www.caiso.com/informed/Pages/CleanGrid/Vehicle-GridIntegrationRoadmap.aspx
20 Kehoe, Chapter 355, Statutes of 2009.
http://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=200920100SB626

for programs that “accelerate widespread transportation electrification.” Three of the IOUs
have since filed applications for transportation electrification projects.21
Energy Storage
In response to AB 2514, the CPUC released a decision in 2013 establishing a target of 1,325
megawatts (MW) of energy storage to be procured by 2020 and installed by the end of
2024.22 In December of 2014, the CEC and the CPUC, along with the ISO, developed an
energy storage roadmap23 that identifies policy, technology and process changes to
address challenges faced by the energy storage sector. The comprehensive roadmap
assesses the current market environment and regulatory policies for connecting new
energy storage technology to the state’s power grid. It is the result of collaboration by the
three organizations and input from more than 400 stakeholders including utilities,
technology companies, environmental groups and interested parties. The roadmap focuses
on activities that address three critical categories of challenges:
• Expanding revenue opportunities for energy storage providers;

• Reducing costs of integrating and connecting to the power grid; and
• Streamlining and defining policies and processes to increase the certainty of
expected benefits of energy storage systems.
In 2017, the CPUC released a Proposed Decision, which considered developments from
the roadmap and sets forth a process for utilities to propose programs and investments of
an additional 500 MW of energy storage, per AB 2868.24
2.2.5 Governor Brown’s Clean Energy Jobs Plan

Governor Brown set ambitious goals at the start of his most recent term in 2015 to produce
20,000 new MW of renewable energy to accelerate the development of energy storage
capacity and strengthen energy efficiency by 2020. Some specifics of the plan include
installing 8,000 MW of renewable central station capacity, 12,000 MW of renewable
distributed generation by 2020 and adding 6,500 MW of combined heat and power
21 CPUC. "Transportation Electrification Activities Pursuant to Senate Bill 350."

http://www.cpuc.ca.gov/sb350te/
22 CPUC. Decision Adopting Energy Storage Procurement Framework and Design Program. 2013.
http://docs.cpuc.ca.gov/PublishedDocs/Published/G000/M079/K533/79533378.pdf
23 California ISO. Advancing and Maximizing the Value of Energy Storage Technology. 2014.
https://www.caiso.com/Documents/Advancing-
MaximizingValueofEnergyStorageTechnology_CaliforniaRoadmap.pdf
24 CPUC. Decision on Track 2 Energy Storage Issues. 2017.
http://docs.cpuc.ca.gov/PublishedDocs/Published/G000/M185/K070/185070054.PDF

systems over a twenty-year period.25 Governor Brown has stated that these efforts will
produce a half a million jobs over a ten-year period.
2.2.6 Diversity and Disadvantaged Communities

The CEC issued a formal Diversity Policy Resolution in April of 2015 to state its policy to
“improve fair and equal opportunities for small businesses; women-, disabled veteran-,
minority- and LGBT-owned business enterprises; and economically disadvantaged and
underserved communities to participate in and benefit from Commission programs.”
Subsequently, Assembly Bill 86526 directs the CEC to establish a diversity task force to
consider and make recommendations about diversity in the energy industry. In response,
the CEC developed a plan comprised of four main areas:
• Outreach to raise awareness about EPIC and opportunities for participating;

• Geographic targeting of regions for projects (such as economically depressed
communities);
• Efforts to address energy-related challenges and opportunities in economically
depressed communities; and
• A tracking system to monitor and report on participation among the groups
mentioned above.
The CEC has conducted specific outreach activities and has implemented a tracking
system to routinely report on participation in EPIC among targeted under-served groups
and communities.
The CEC released Part A of a Low-Income Barriers Study mandated by SB 350 in

December of 2016.27 The study presents findings and recommendations related to limited
access to clean energy for low income customers including policy and program barriers
and structural barriers along with challenges that exist for small businesses located in
disadvantaged communities. The study recommended the CEC dedicate 25 percent of its
EPIC TD&D funding to investments within or benefiting disadvantaged communities, and
made similar recommendations for the IOUs’ EPIC funds.
25 Governor Edmund G. Brown Jr. Clean Energy Jobs Plan.

https://www.gov.ca.gov/docs/Clean_Energy_Plan.pdf
26 Alejo, Chapter 583, Statutes of 2015.
http://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=201520160AB865
27 CEC. SB 350 Low-Income Barriers Study, Part A – Commission Final Report. 2016.
http://docketpublic.energy.ca.gov/PublicDocuments/16-OIR-
02/TN214830_20161215T184655_SB_350_LowIncome_Barriers_Study_Part_A__Commission_Final_Report.p
df

3 Evaluation Approach
This section describes the objectives and research questions that guided the evaluation
design, and then presents details on the methods the Evergreen team used to collect the
data and information necessary to conduct the EPIC evaluation.
3.1 Evaluation Objectives and Research Questions

The objective of the study was to conduct a comprehensive evaluation of EPIC to identify
opportunities to improve program management and effectiveness. To this end, specific
objectives included the following:
• Determining if the program is being implemented in a manner consistent with the

program objectives, requirements and intent of the CPUC and the Legislature;28
• Assessing the extent to which the program supports key energy policies and public
research code sections;
• Identifying best practices in research administration;
• Assessing the extent to which the Program is on track, thus far, in meeting its
objectives to provide ratepayer benefits, advance energy innovation and support
California’s energy policy goals; and
• Providing recommendations for improvements to program requirements and
practices.
The evaluation results are intended to support and inform the CPUC’s consideration of
EPIC within the next triennial application proceedings, which were initiated in the spring
of 2017 with the administrators’ 2018-2020 Investment Plan application filings.29
For the purposes of focusing this evaluation on the most fundamental program
characteristics, CPUC staff established EPIC’s core, overarching goals (or core values).
These core values are (1) to provide electricity IOU ratepayer benefits by (2) advancing
energy innovation that (3) supports California’s energy policy goals. These core values are
defined in more detail below.
28As set forth in D.11-12-035, D.12-05-037, D.13-11-025, D.15-04-025, and D.15-09-005.

29The applications are:
PG&E: A.17-04-028
(https://apps.cpuc.ca.gov/apex/f?p=401:56:0::NO:RP,57,RIR:P5_PROCEEDING_SELECT:A1704028),
CEC: A.17-05-003
(https://apps.cpuc.ca.gov/apex/f?p=401:56:0::NO:RP,57,RIR:P5_PROCEEDING_SELECT:A1705003),
SCE: A.17-05-005
(https://apps.cpuc.ca.gov/apex/f?p=401:56:0::NO:RP,57,RIR:P5_PROCEEDING_SELECT:A1705005) and
SDG&E:
A.17-05-009
(https://apps.cpuc.ca.gov/apex/f?p=401:56:0::NO:RP,57,RIR:P5_PROCEEDING_SELECT:A1705009).

Electricity IOU Ratepayer Benefits: As mentioned in Section 2.1, providing electric
ratepayer benefits was established as the mandatory guiding principle of EPIC, which the
CPUC defined as promoting greater reliability, lower costs and increased safety.
Energy Innovation: The CPUC established a range of administration and program

requirements intended to ensure that EPIC funds RD&D activities that assist the
emergence of innovative energy technologies and services, for the benefit of California’s
electricity IOU ratepayers and the public interest. Overall, advancing true, productive
energy innovation is a core purpose of the program. This goal should be supported not
only in EPIC’s technical program areas, but also in EPIC’s administrative structure.
Therefore, all Program decisions and activities—from the setting of research priorities, to
the treatment of intellectual property concerns, to the administrative balancing of due
process with flexibility and responsiveness—should support, embody and advance true,
productive energy innovation.
California’s Energy Policy Goals: As the CPUC’s EPIC decisions make clear, supporting
California’s key energy, climate and economic policy goals is one of the core justifications
of the Program, and a purpose that drives much of EPIC’s governance and requirements.
Each EPIC funding decision and other activity must be made and considered in the
context of these policy goals.
To achieve these overarching goals, the evaluation was designed to address a series of
specific research questions organized by topic area:
1. Program Management and Administration - Are the administrators effectively

complying with program requirements? Beyond mere compliance, are
administrators functioning as world-class energy innovation program managers?
2. Investment Planning Process - Is the triennial investment planning process
effectively identifying a broad range of potential energy RD&D objectives,
evaluating those objectives according to sensible criteria, and ultimately producing
investment plans with a high likelihood of producing benefits for California
ratepayers and achieving other EPIC goals?
3. Project Selection Process - Is the project selection process resulting in funds going
to projects that are consistent with EPIC policy objectives and planning processes,
in an open, effective and efficient manner?
4. Project Assessment Process - What is the status of EPIC investments? Do the
administrators do everything possible to track the progress of funded work? Are
ongoing projects showing reasonable indicators of success? Are processes in place
to determine project viability over time and disseminate project results to
stakeholders?
5. Policy Alignment and Project Impacts - Looking beyond project- and
administrator-specific considerations, what impacts does the Program overall have

in a wider context? How is EPIC situated in the broader innovation and policy
landscape?
To investigate these questions, the CPUC selected a team of consultants (the Evergreen
team) to conduct this evaluation. Evergreen Economics was the prime contractor and
managed all evaluation activities. Evergreen was assisted by the NMR Group, Inc., Ridge
& Associates, Jai J. Mitchell Analytics and Advanced Survey Design.
3.2 Research Approach

The Evergreen team employed a theory-driven evaluation framework within which we
assessed the Program’s effectiveness. Guided by a series of logic models that we
developed to support the theory-based design, we identified plausible causal mechanisms
and tested related hypotheses that the successful implementation of key program activities
involving multiple actors will lead to the expected outputs and that these in turn will
eventually lead to the achievement of the short-, mid- and long-term benefits. Absent a
logic model, much that can and should be measured in assessing a program’s efficacy
would be missed.
The logic models, summarized in Section 4 and with more detail provided in Appendix B,
show the program activities and expected outputs and outcomes and serve as a guide for
describing the underlying program theory and for developing researchable questions and
metrics. This theory-driven approach30 relies on a mixed methods approach involving the
collection and analysis of both quantitative and qualitative data covering program inputs,
activities, outputs and outcomes. We conducted several research activities to collect data
on the logic model metrics and to support the assessment of the program effectiveness
research questions:
1. Review of relevant CPUC Decisions, EPIC Investment Plans and Annual Reports;
2. Telephone interviews with program administrators (several rounds of interviews
with key contacts at the CEC, PG&E, SCE and SDG&E involved with administering
the Program);
3. Development and review of data on all EPIC projects (for EPIC 1 and EPIC 2, the
first two triennial cycles that cover program years 2012–2017);
4. Development of program theory, logic models and metrics;
5. Detailed review of data for a sample of EPIC projects (54 projects);
30Ruegg and Feller, 2003; Chen, 1990; Rogers, 2000, 2008; Rogers et al., 2000; Weiss, 1995, 1997; Coryn, 2011,
and consistent with the Emerging Technologies Protocol in the California Energy Efficiency Evaluation
Protocols (http://www.cpuc.ca.gov/General.aspx?id=5399). See Section 17: Appendix F for reference
information on the theory-driven evaluation approach.

6. In-depth telephone interviews with CEC CAMs, CEC grantees, and IOU project
managers and vendors associated with the sampled projects (90 interviews);
7. In-depth telephone interviews with stakeholders (9 interviews with representatives
from organizations related to but not necessarily directly involved in CPUC EPIC
projects including state agencies, California engineering and technical experts, third
party firms that help evaluate EPIC proposals, and incubators/accelerators);
8. Review of a sample of CEC solicitations (5 solicitations, 93 bids submitted) and IOU
vendor bids (6 projects, 46 bids submitted);
9. A best practices assessment involving a review of the literature that focused on
innovative energy RD&D programs (38 programs) and in-depth interviews with
RD&D program administrators (7 interviews); and
10. Network analysis to assess the exchange of project-related information between
EPIC project teams and other individuals and organizations sharing similar
interests.
The remainder of this section describes how we conducted each data collection activity.
3.3 Program Document Review

One of the first evaluation tasks was to gather and review all relevant documents related
to EPIC, including CPUC Decisions and administrators’ Investment Plans and Annual
Reports. First, the Evergreen team reviewed the CPUC Decisions that together establish
the requirements and administrative procedures for EPIC.
Next, we reviewed each administrator’s EPIC 1 (2012–2014) and EPIC 2 (2015–2017)

Investment Plans, and their Annual Reports for Program Years 2012–2016. To provide an
update on EPIC projects, we relied on administrators’ 2016 Annual Reports (which were
submitted to the CPUC on February 28, 2017), which were the most recent comprehensive
status updates of all EPIC projects, covering EPIC 1 and EPIC 2.
We also reviewed each administrator’s EPIC website and reviewed relevant California
legislation and policies.
3.4 Program Administrator Interviews

Another early evaluation task was to conduct an initial interview with the key contacts
from each EPIC administrator to further inform our understanding of how the Program
operates. The Evergreen team worked with the CPUC evaluation manager and the
administrators to determine the appropriate contacts. We conducted one-hour interviews
with each administrator (for a total of four interviews) in October of 2016, and a follow-up
round of interviews in April of 2017 to discuss some topics that we had not covered in the
first round. We discussed a range of topics, and organized the interview guides by the five
research topic areas presented previously in Section 3.1.

3.5 EPIC Project Data Review
The Evergreen team developed a database of all EPIC 1 and EPIC 2 projects, based on data
and information provided by the administrators in their Investment Plans and Annual
Reports, supplemented by a data request we submitted to them in September of 2016.31
The database included basic information about each project, including project type, status,
technology type and information on the metrics that would be tracked and the types of
benefits the project was intended to generate. The CPUC required that the administrators
identify metrics against which each project’s success may be evaluated, and they provided
a list in Decision 13-11-025 (See Appendix A).
We used the database to generate a sample frame from which to draw a project sample
(discussed next) to gather more detailed information. We also used the database to
characterize EPIC projects (see Section 5).
3.6 Project Sample Data Review

The study team, in consultation with the CPUC project manager for the evaluation
contract with Evergreen, decided to develop a project sample to explore in more detail
since it would have been cost- and time-prohibitive to review all projects in detail. We
wanted to draw a random (or close to random) sample so that when we examined a
smaller number of projects, the findings could be generalized to the program level. First,
we developed a sample frame of all active and completed projects based on data from the
administrators’ 2015 Annual Reports supplemented by responses to evaluation data
requests. The sample frame (248 total projects32) is shown in Table 3, by administrator,
EPIC phase and status (i.e., complete or active), at the time we developed the sample.
31We also submitted several follow-up requests to obtain additional data as the evaluation progressed.
32Note that one of the early evaluation steps was to draw the project sample, and we relied on the
administrators’ 2015 Annual Reports supplemented with data from the Investment Plans. Later evaluation
tasks that were conducted after the 2016 Annual Reports were released on February 28, 2017 made use of the
additional data provided in those reports and include a higher number of projects. Additionally, during the
evaluation period, additional projects reached completion.

Table 3: Sample Frame by Status, Administrator and EPIC Phase*
Complete Active Total
Administrator EPIC 1 EPIC 2 EPIC 1 EPIC 2 EPIC 1 EPIC 2 Total
CEC 3 0 171 18 174 18 192
SCE 3 0 11 0 14 0 14
SDG&E 0 0 5 6 5 6 11
PG&E 5 0 12 14 17 14 31
Total 11 0 199 38 210 38 248
* The sample frame table represents the status of projects when the evaluation team drew the sample in
November of 2016.
We selected a sample size of 54 projects to study more closely targeted at the 90/10 level of
confidence and precision across all projects.33 Next, we selected all completed projects for
the sample (a total of 9 projects).34 We then randomly selected 43 of the active projects,
stratified by TD&D and Market Facilitation. For TD&D projects, we further stratified by
administrator. We oversampled the Market Facilitation and IOU projects to make sure that
there were enough projects in each category to support our analyses. We then made
further adjustments to ensure that the sample represented the number of projects in each
phase (EPIC 1 and EPIC 2) and included projects specifically named in Decisions or given
special guidance. This sample design was a mix of random and purposive and allowed us
to address key research questions more effectively.35 Table 4 and Table 5 describe the final
sample allocation.
33 90/10 is a standard level of confidence and precision used in energy efficiency evaluation. The 90 (percent)
represents the level of statistical confidence we have that the true, but unknown, parameter lies within an
interval that is +/- 10 percent of the value of the estimated parameter.
34 The number of projects classified as "complete" changed from 11 to 9 from the time we developed the
sample frame to when we pulled the sample. Three of the CEC projects listed as completed in our sample
frame (Table 3) had completed the Applied RD&D phase at the time we selected the sample. Each of these
projects was transferred to the Market Facilitation phase and awarded EPIC funding and is now classified as
active. An additional PG&E project was completed.
35 Nick Emmel, Sampling and Choosing Cases in Qualitative Research: A Realist Approach. Thousand Oaks, CA:
SAGE Publications, 2013.

Table 4: Project Sample by Status, Administrator and EPIC Phase*
Complete Active Total
EPIC 1 EPIC 2 EPIC 1 EPIC 2 EPIC 1 EPIC 2 Total
CEC 0 0 30 3 30 3 33
SCE 0 3 4 0 4 3 7
SDG&E 0 0 0 4 0 4 4
PG&E 6 0 1 3 7 3 10
Total 6 3 35 10 41 13 54
*The project sample table represents the status of projects at the time of the evaluation report completion.
One sampled PG&E project was completed in the interim. The three CEC projects shown as completed in the
sample frame table (Table 3) were transferred to the Market Facilitation stage and were classified as active
EPIC 2 projects, between the time we developed the sample frame and obtained the project sample from the
administrators. An additional PG&E project was completed.
Table 5: Project Sample by Project Type and Administrator

Applied Market
R&D TD&D Facilitation
CEC 9 17 7
SCE - 7 -
SDG&E - 4 -
PG&E - 10 -
Total 9 38 7
From Table 4 and Table 5, we can see that, in general, the sample size within any given cell
is quite small, thus reducing the level of confidence and precision in each. This is
particularly true for the IOUs. One should, therefore, given this increased uncertainty be
cautious in interpreting any observed differences in the project sample across the three
IOUs. In addition, the fact that about 58 percent of the IOU projects are still active suggests
that any of these observed differences could change over time as projects approach
completion.
3.7 Project Interviews

We conducted telephone interviews with managers associated with each of the 54 projects
in our sample in order to obtain a deeper understanding of EPIC processes and project
outcomes (both those already realized and potential future impacts). First, we contacted
the administrator project manager (e.g., the CEC or IOUs) and conducted an interview.
Next, for the CEC, we conducted an interview with the grantee, which is the organization
that was awarded the project. We also interviewed any vendors that were implementing

key parts of IOU projects. Often, we conducted more than one interview per respondent or
group of respondents to cover all topics. Counting interviews as a single respondent or
group of respondents (not double counting follow-up interviews), we conducted a total of
90 project-level interviews associated with the 54 sampled projects, as shown in Table 6.
Table 6: Project Interviews by Interview Type and Administrator

Admin Grantee/Vendor Total
CEC 32 32 64
IOUs 21 5 26
Total 53 37 90
We developed interview guides for the project interviews based on the logic models and
associated metrics, which are discussed in Section 4. Topics included:
• Successes and challenges of the project;

• Feedback on the funding process;
• Information on partners that have collaborated on the project;
• Feedback on the EPIC administrator management processes; and
• Suggestions for program improvement.
The evaluation team conducted the interviews by phone from January through April of
2017.
3.8 Stakeholder Interviews

We conducted a total of nine telephone interviews from February through May of 2017
with representatives from organizations related to but not necessarily directly involved in
EPIC projects. These included state agencies, national labs, California engineering and
technical experts, and third party firms that help evaluate EPIC proposals. Stakeholder
input was critical to inform our understanding of how administrators draw on expert
opinion to develop their plans and guide their projects. Likewise, we sought to gauge
stakeholder and expert engagement with the Program and identify ways to improve how
the administrators tap experts’ knowledge and networks.
The team initially sought to conduct interviews with 15 stakeholder organizations, but
could not reach this interview target, primarily due to lack of respondent knowledge
about EPIC or insufficient cooperation with interview requests. The team made repeated
attempts to reach appropriate contacts from a total sample of 19 organizations that were
identified as potential stakeholder interview candidates after a review of attendee lists in
various EPIC meetings and consultation with CPUC staff. With these requests, the team

also inquired about other individuals within their organization who might be
knowledgeable about the Program. From the initial sample, three potential interviewees
indicated that they did not know enough about EPIC to provide an informed response to
our request for an interview. The remaining contacts did not respond to multiple requests
for an interview about the Program; their lack of response may indicate that they also were
not very knowledgeable—or did not have strong opinions—about EPIC. Table 7 below
lists organizations where stakeholders declined or failed to respond to multiple requests
for an interview; the status for the former is labeled as “declined” while the latter is
termed “incomplete.”
Table 7: Incomplete Interviews by Organization Type

Organization Type Organization Status
California Air Resources Board Declined*
California EPA Declined
CA state agency/organization California Energy Efficiency Industry Council Incomplete
California Department of Resources Recycling and
Incomplete
Recovery
National Lab National Renewable Energy Laboratory Incomplete
Sunpower Incomplete
Private company
Solar City Incomplete
Industry organization CALSTART Incomplete
The Nature Conservancy Incomplete
Environmental organization
Clean Coalition Incomplete
Center for Energy Efficiency and Renewable
Non-profit/Advocacy Declined
Technology
Organization
The Utility Reform Network (TURN) Incomplete
* Stakeholder agreed to an interview, but did not follow through and failed to respond to attempts to
re-schedule.
While the number of completed interviews is lower than initially planned, the team feels
confident that the interviewees reflect a range of experiences generally representative of
program stakeholders. The team completed interviews with at least one individual from
five of the seven targeted organization types (Table 8). We were not able to conduct
interviews with representatives from private companies or nonprofit organizations.

Table 8: Completed Interviews by Organization Type
Number of
Organization Type Interviewees
CA state agency/organization 3
National Lab 3
Private company 0
Academic organization 1
Industry organization 1
Environmental organization 1
Non-profit/Advocacy Organizations 0
Total 9
Note: Individual organizations are not listed because the
evaluation team assured interviewees that their identities and
individual responses would be kept confidential.
Each interview lasted approximately an hour and was conducted by senior professional
staff by phone. We developed interview guides for the project interviews based on the
logic models and associated metrics. Topics included:
• Identifying the individuals’ role and information about their organization

• Engagement with EPIC
• Networks and relationship building
• Information sources and knowledge exchange
• Program Administration
• Market Facilitation
3.9 CEC Solicitation and IOU Outsourcing Process Review

Each administrator implements a specific project scoping process to move from approved
proposals (once its Investment Plan is approved by the CPUC) to reality. The CEC’s EPIC
1 and EPIC 2 Investment Plans propose broad objectives and strategic initiatives within
them, and the CEC develops solicitations (either Grant Funding Opportunities [GFOs] or
Program Opportunity Notices [PONs]) that are aligned with each initiative and issues
them throughout the year and awards one or more projects per solicitation. The award
status is documented and posted publicly on the CEC’s EPIC website via a Notice of
Proposed Award.

We developed a sample frame of solicitations and bids (including accepted and rejected)
and drew a sample of five solicitations. The sample was drawn to include almost all of our
sampled projects and to cover a range of project types and funding levels. The sample
included 93 bids. We requested documentation from the CEC for the sample, including the
actual bids submitted, solicitation requirements and criteria, and scores. We reviewed the
sample of bids to ensure that the bids were reviewed consistently with CPUC
requirements and that bid results (e.g., top-scoring winning bidders) matched what was
posted to the CEC website and distributed to bidders. We also reviewed the bids to assess
how well the awarded bids aligned with the strategic objectives the solicitations were
designed to achieve.
The IOU administrators’ Investment Plans include the specific projects they plan to
implement in the plan period. The IOUs are required to use contractors (which they refer
to as vendors) for all projects, for which they either use a direct award or a competitive bid
process. To confirm the competitive bid processes were consistent with CPUC
requirements and aligned with the project scoring and selection criteria outlined in the
relevant CPUC Decisions, we conducted a detailed review of project bids for a sample of
six IOU projects. We drew a sample of six projects—four PG&E projects, one SCE project
and one SDG&E project—from the project sample described in Section 3.6. The sample
included 46 vendor bids across the six projects. We requested documentation from the
IOUs for the sample, including the actual bids submitted, project proposal requirements,
scoring criteria and scores. We also assessed and documented the cases where the IOUs
used a direct award (or sole source) process to award a contract to a vendor.
3.10 Best Practices Assessment

The best practices assessment was used to identify effective practices in other research
programs that are comparable to EPIC. Although EPIC is unique in its size, scope and
breadth, there are still lessons to be learned by examining the experiences and best
practices developed in other research organizations even if they are not directly
comparable. The best practices assessment consisted of a literature review combined with
telephone interviews with RD&D program administrators and evaluators.
The literature review included reports and documents relevant to the planning, design
and implementation of innovative energy RD&D programs. We identified a selection of
programs similar to EPIC and reviewed publicly available evaluation reports and other
documents about these programs. The team reviewed a total of 38 resources including
reports, white papers and webpages. In general, program-related documents draw on
publicly available reports, evaluations and other resources related to the peer RD&D
programs listed above. We primarily tried to cite practices supported by evaluation
results, but some programs either did not have publicly available evaluation reports or the
reports did not cover the topics of interest for this review. The team employed a systematic
approach to this review, documenting whether elements relevant to the evaluation were

present in each document. The research dimensions covered a range of topics including
best practices related to investment planning, implementation, program management and
tracking; technology transfer mechanisms; market development mechanisms; and key
findings from program- or project-level research/evaluation efforts.
The initial literature review identified several programs similar to EPIC. Through the in-
depth interviews and further review of program documents, the team determined that
four of the seven programs initially identified were very similar to EPIC in terms of their
objectives and mission. These primary peer RD&D programs include:
• The U.S. Department of Energy’s (DOE) Small Business Innovation Research (SBIR)
Program
• The DOE’s Small Business Technology Transfer (STTR) Program
• The DOE’s Advanced Research Projects Agency-Energy Program (ARPA-E)
• The New York State Energy Research and Development Authority's (NYSERDA’s)
Technology and Market Development (T&MD) Program
The literature review also identified three other peer RD&D programs that, upon closer
investigation, were revealed to be not as similar to EPIC as the programs listed above.
Although these three programs support innovative R&D efforts, they do not have an
explicit goal related to commercialization, which is a key feature of EPIC. These secondary
peer RD&D programs include:
• The New Mexico Small Business Assistance (NMSBA) Program

• The Washington State Clean Energy Fund
• The Connecticut Green Bank
Table 9 provides a description of each of the peer RD&D programs included in the best
practices assessment.

Table 9: Peer RD&D Programs Included in the Best Practices Assessment
Program Description
DOE Small Business The SBIR Program awards Federal Research/Research and Development
Innovation Research (R/R&D) grants to small businesses through various federal agencies, including
(SBIR) Program the Department of Energy (DOE). Both the SBIR and STTR Programs
(described below) have four program goals: 1) stimulate technological
innovation; 2) use small business to meet Federal R/R&D needs; 3) foster and
http://science.energy.gov/
encourage participation in innovation and entrepreneurship by socially and
sbir/
economically disadvantaged persons; and 4) increase private sector
commercialization of innovations derived from Federal R/R&D. The SBIR and
STTR Programs offer participation in three phases. Phase I awards grants to
fund up to $225,000 over nine months for feasibility studies. Phase II grants,
which are up to $1,500,000 over two years, support more extensive R&D to
develop scientific and technical merit. Only Phase I awardees may compete
for Phase II funding. Phase III is the period during which Phase II innovation
moves into the marketplace with non-SBIR/STTR Program funding, and the
small businesses are expected to acquire additional funds to cover these
efforts.
DOE Small Business Like the SBIR Program, the STTR Program is administered by the DOE’s
Technology Transfer Office of Investments and Innovation. While the SBIR Program is focused on
(STTR) Program having small businesses engage in federal R/R&D, the STTR Program facilitates
R&D cooperation between small businesses and research institutions. As a
result, there are two key differences between the two programs: First, with
http://science.energy.gov/
an SBIR Program award, the principal investigator (PI) must be primarily
sbir/
employed by the small business concern (SBC) while the PI of an STTR
Program project can be employed by either the SBC or research institution.
Second, with the STTR Program, using a research partner with a Phase I and
Phase II award is required, and the minimum level-of-effort expended by the
SBC must not be less than 40 percent in both Phase I and Phase II. With the
SBIR Program, the SBC is not required to partner with a research institution;
however, if using a research partner, the minimum level-of-effort expended by
the SBC must not be less than 60 percent.
DOE Advanced Research ARPA-E focuses on advancing energy technologies designed to reduce the
Projects Agency-Energy dependence on energy imports, reduce energy related emissions, improve
Program (ARPA-E) energy efficiency across all sectors of the economy, and ensure that the US
remains competitive in developing and deploying transformational
technologies. The program provides funding for technologies that are too
https://arpa-e.energy.gov/
early for private sector investment but have the potential to lead to new ways
to generate, store and use energy. Projects receive direct commercialization
support through the Agency’s Technology-to-Market program. This support
equips projects with a clear understanding of market needs to guide technical
development and help projects succeed in the marketplace.

Program Description
NYSERDA Technology The mission of the T&MD Program is to “test, develop, and introduce new
and Market Development technologies, strategies, and practices that build the statewide market
(T&MD) Program infrastructure to reliably deliver clean energy to New Yorkers.” The specific
objectives include moving new and under-used technologies and services into
the marketplace to help achieve the goals for the Energy Efficiency Portfolio
http://www.nyserda.ny.go
Standard (EEPS) and Renewable Portfolio Standard (RPS) Programs; validating
v/All-Programs
emerging energy efficiency, renewable and smart grid technologies/strategies
and accelerate market readiness; stimulating technology and business
innovation to provide more clean energy options and lower cost solutions,
while growing the state’s clean energy economy; and spurring actions and
investments to achieve results distinct from incentive-based programs. The
T&MD portfolio is designed to support these objectives by funding nine
initiatives in a range of areas, including power supply and delivery, building
systems and clean energy infrastructure.
New Mexico Small The NMSBA Program fosters collaboration between New Mexico small
Business Assistance businesses and the Los Alamos and Sandia national laboratories. Small
(NMSBA) Program businesses can receive assistance from lab scientists or engineers for projects
that require testing, design consultation and access to equipment or facilities
that are not available in the private sector. The NMSBA Program offers three
http://www.nmsbaprogra
types of projects: individual, leveraged and contract. Individual projects involve
m.org/
a single small business tackling a problem with national laboratory expertise.
Requests for individual projects are accepted year-round until funding is
exhausted. Leveraged projects include multiple small businesses with shared
technical challenges. Proposals for leveraged projects are reviewed twice, and
awards range from $20,000 to $100,000 per laboratory. Contract projects
allow small businesses to contract for services typically not available in the
private sector at a considerably reduced cost (such as courses on renewable
energy development).
Washington State Clean The Washington State Clean Energy Fund invests in clean energy development
Energy Fund by supporting the “…development, demonstration, and deployment of clean
energy technologies that save energy and reduce energy costs, reduce harmful
air emission, or otherwise increase energy independence for the state.” The
http://www.commerce.w
fund provides funding for a range of projects, including grants to electric
a.gov/growing-the-
utilities for smart grid projects, grants to leverage support for research and
economy/energy/clean-
development, financing opportunities for renewable energy manufacturing,
energy-fund/
and grants to nonprofit lenders that provide capital to residential and
commercial consumers who install renewable energy systems and make other
energy-efficient upgrades.

Program Description
Connecticut Green Bank The Connecticut Green Bank furthers the adoption of clean and renewable
energy solutions by making financing available to homeowners, businesses,
municipalities and capital providers. The focus of the Green Bank is to attract
http://www.ctgreenbank.c
and deploy capital to fill the investment gap needed to support the successful
om/
implementation of the state’s clean energy policy goals. The Green Bank is
structured to address four consumer sectors: residential (single and
multifamily properties), commercial and industrial, institutional (state,
municipal, universities, schools and hospitals) and infrastructure (grid-tied
projects as well as statutorily required programs such as the Residential Solar
Investment Program and the Anaerobic Digester Pilot Program).
We conducted interviews with the seven RD&D program administrators in February

through March of 2017. The interviews were designed to gather information about
program design, program management and coordination, successes and challenges, and
information on other stakeholders to interview and resources to review.
Each interview lasted approximately one hour and was conducted by an experienced
interviewer from NMR by telephone. Topics included:
• Background on their RD&D program

• The project selection process
• Program administration
• Support for commercialization
• Program management, tracking and reporting
• Knowledge dissemination and technology transfer
• Indicators of success
3.11 Network Analysis

For programs such as EPIC to be successful, they must collaborate to some extent with
other experts and transfer the knowledge gained from their investigations to other
stakeholders. The diffusion of such information is critical if others are to adopt these
technologies and tools and/or to conduct further research to improve upon them.
The goal of the network analysis is to clarify the knowledge pathways formed when
information moves through the relational channels of market actors in the California
economy and expert communities, stemming from the outreach, collaboration, marketing
and information dissemination activities of the CEC and IOU administrators, CEC project
managers, CEC grantees, IOU project managers and IOU contractors. The network
analysis was conducted at two levels, the project level and the program level. The project-

level analysis, covering the Applied R&D, TD&D and Market Facilitation funding areas,
was based on the results of 90 in-depth interviews covering over 250 hours with the CEC
project managers, the CEC grantees, IOU project managers and their vendors. Areas
covered in these interviews included the sharing of information and resources with other
organizations and individuals during the planning and implementation of their projects
and, once the projects were completed, the dissemination of the final reports. Questions
touched on such topics as the effectiveness of the technical advisory committees,
relationships with other organizations, workshop and conference presentations, articles
published, fact sheets prepared, and websites and list serves used to disseminate
information.
The program-level network analysis was based on in-depth interviews with and data
collected from CEC and IOU administrators regarding program-level interactions with
other organizations and individuals during the planning and implementation of each EPIC
funding cycle. Questions covered such topics including:
• Coordination with other administrators on electric vehicles, storage or micro grids

and with other R&D programs and research organizations both inside and outside
California (e.g., DOE programs, National Labs);
• Coordination of innovation clusters;
• Creation of an information exchange for facility owners, design professionals, and
skilled labor working in facilities construction, operation, and maintenance trades
to share integrated DSM, ZNE and other information and experiences based on
demonstration deployment results; and
• Coordination of projects across the three program areas.

4 Program Theory and Logic Model
The foundation of a theory-based evaluation is the development of a program logic model.
This is critically important when evaluating an RD&D program that consists of a complex
network of activities, outputs and outcomes that combine to produce over time a number
of mid- and long-term benefits. Absent a logic model, much that can and should be
measured in assessing a program’s efficacy would be missed. This section discusses how
the logic models for EPIC were developed, with additional details provided in Appendix
B.
4.1 Overview of Logic Model Approach

At a high level, logic models describe inputs and activities and how they combine to
produce expected outputs which, in turn, may produce expected short-term, mid-term and
long-term outcomes (see Figure 1 later in this section for an illustration). Each pathway or
linkage in the logic model describes a hypothesized cause and effect relationship.
The key elements of any logic model are the inputs, activities, outputs and outcomes.
Inputs can include human, financial, organizational, community or systems resources in
any combination. These inputs serve as the catalyst for program activities, which reflect the
processes, events, technologies and other devices that are intentional in the program. The
direct results of program activities are the outputs, which can include the production or
availability of different types of program assistance. Outcomes are about change and
indicate what ideally will occur as a result of the program activities and outputs. Common
outcomes include specific changes in awareness, knowledge, skill and behavior.
Outcomes are often categorized by time to indicate which are expected in the short,
intermediate, and long term. The evaluation team also used these logic models as guides
to identify and operationalize specific metrics to be measured along the various paths
from inputs to activities and then outputs and outcomes.
We have prepared five logic models that when taken together provide a comprehensive
view of the Program. The first logic model, referred to as the overarching EPIC logic
model, describes EPIC at a high level including its four key program areas:
Administration, Applied R&D, Technology Demonstration and Deployment (TD&D), and
Market Facilitation. While this overarching logic model is useful in discussing the entire
Program, we developed a sub-logic model for each of these four key program areas in
order to reflect the unique logic and contributions of each. It was these other four more
detailed logic models that provided the framework for identifying performance metrics
and data collection activities we describe in the following subsection. This framework was
also used to integrate and communicate the results of all of the planned evaluation
activities. Note that the Administration logic model encompasses the crosscutting program
activities and expected outputs and outcomes associated with the administrative activities,
including processes used to develop Investment Plans, solicit bids for projects and
implement projects. In each logic model, each activity, output and outcome is assigned a

letter and each link, representing a hypothesized cause-and-effect relationship, is assigned
a number.
These logic models and the underlying social/economic theories are intended to serve as a
single source of reference for the foundational components of EPIC including the
activities; outputs; short-term, mid-term and long-term outcomes; and the market barriers
associated with the Program. The logic models identify the various strategies that are
designed to achieve the intended program objectives. They also describe the various
positive and negative external factors that might influence the design, delivery and
expected outcomes and the relationship of the Program to the activities being carried out
by the other organizations and market actors. Each logic model is followed by a list of
potential indicators that could be used in testing hypotheses regarding key cause-and-
effect relationships. The indicators were the main source for developing research
instruments.
In developing these logic models, the following activities were performed:
• Document reviews;
• Discussion with the CPUC study manager and staff from the four administrators to
help define the logic model elements (these included identification of key program
inputs, activities, market actors, outputs, outcomes, potential external influences
and other program interactions);
• Logic model diagram construction—entailing transposition of key logic model
elements into a series of boxes or circles and arrows to identify preliminary logical
relationships among the elements;
• Identification of market barriers and context development; and
• Identification of potential program measurement indicators.
The initial logic models were completed in October 2016 and were used as a guide for the
study research. Some of the models were revised based on updated information we
gathered as the evaluation progressed, and we include the final logic models in this
section and Appendix B.
4.2 Metric Development

For each of the logic models, there are a set of activities and expected outputs and
outcomes. We developed performance metrics for each activity, output and short-term
outcome that guided our data collection plan. The metrics are in turn linked to a data
collection source or sources, and documented in a data collection table.
The format for each of the data collection tables is the same. For every program activity,
each related program output and outcome is included in a table. For each output and
outcome, specific metrics are created that will provide an indication of whether the
underlying program logic is succeeding in practice. Each metric is then linked to specific

data collection and analysis activities. In this way, all metrics are covered by data
collection activities, and all data collection and analysis activities are explicitly linked to
underlying elements of the program logic models. An example of such a data collection
table that we developed to inform the evaluations is shown below in Table 10.
Table 10: Example Metrics and Data Collection Table:

Outputs H and I (from Logic Model Activities D and E)
Outputs Metrics Data Source
Number of projects that meet technical targets;
Results in the form of
Number of databases, software tools, fact sheets, D, IDI-P
data/software/fact sheets/articles
and articles
Frequency of progress reports IDI-P
Project progress reports
Usefulness of progress reports IDI-P, IDI-A
Number of promising technologies, tools and
Project final reports D, IDI-P
strategies identified
Number of patent applications filed D, IDI-P
Number of patents issued D, IDI-P
Patents/copyrights filed/issued
Number of project results copyrighted D, IDI-P
Number of project results licensed D, IDI-P
Data source key: IDI-A=Administrator in-depth interviews, IDI-P=Project-level in-depth interviews,
D=Project data.
Green text indicates main source, orange text indicates supporting source.
Once the metrics were developed for all outputs and short-term outcomes, we used the
metrics for each data source to develop the associated data collection instrument. We
tracked the metrics in the research instruments, enabling us to develop an analysis plan
organized by metrics.
4.3 Overarching EPIC Logic Model

The overarching EPIC logic model in Figure 1 uses the goals and principles of the program
as ultimate outcomes and shows pathways to these outcomes in the three project type
areas: Applied Research and Development (R&D), Technology Demonstration and
Deployment (TD&D) and Market Facilitation. For each area, the activities are expected to
lead to specific outputs that, in turn, are expected to lead to specific short-term outcomes.
The combined short-term outcomes from these three areas are in turn expected to lead to
the expected mid-term and long-term outcomes. Note that this logic model also shows a
number of external influences (e.g., changing national and international R&D policies for
energy efficiency/renewables/EV/grid integration, decoupled ownership of power
generation and delivery infrastructure, low investment in grid infrastructure, actions of

third-party providers, etc.) that might also contribute to the observed outcomes in addition
to EPIC activities and outputs.
As noted earlier, the more detailed and complex Administration, Applied R&D, TD&D
and Market Facilitation logic models in Appendix B are the ones that guided our
identification of multiple performance metrics that informed our data collection plan.

Figure 1: EPIC Program Logic Model
technologies
decoupled ownership of power generation and delivery infrastructure; low investment in grid infrastructure; actions of third-party providers; risk aversion for new
External Influences: California energy polcy; Electricity prices; changing national and international R&D policies for energy efficiency/renewables/EV/grid integration;
Inputs A: EPIC Funds, matched funds of partners; expertise, data from past RD&D
1
E: Identify the specific
research, demonstration, or D: Investment plan B: Coordination with Other Research,
Administration deployment objectives for
3
development
2 C: Stakeholder input
Development and Demonstration Efforts
the solicitation
Applied R&D Technology Demonstration & Deployment Market Facilitation

4
H: K: N: Conduct initiatives that streamline
Contractors conduct lab-scale testing Select technologies/tools for regulatory processes and project
F: Opportunity and pilot-scale testing permitting, help develop the state’s
Activities notices posted on Disseminate results
demonstration
Demonstration sites implemented clean energy workforce, fund education
CEC website Engage with other research Disseminate results and outreach programs, and fund
6 organizations & individuals Engage other research organizations program tracking activities.
Seek additional funding and individuals
5 13
7 10
O
I L
Strategies for streamline regulatory
G: Projects awarded Publish data/reports/fact sheets/ Customer/stakeholder visits
processes and project permitting
articles Patents/copyrights filed/granted
Tools and resources that connect the
Patents/copyrights filed/granted Publish data/reports/fact sheets/articles
clean energy industry to the labor
Results delivered through various Results delivered through various channels
Outputs 19
channels Interorganizational/interpersonal networks
market
9 Improved program-tracking database
Interorganizational/interpersonal formed 12 & clearinghouse
networks formed
S: Annual reports Publish data/reports/fact sheets/
Meetings/proposals to seek additional
articles
funding
Market insights for selecting new
11 technologies
8
14
J M P
Increased awareness and knowledge Increased awareness and knowledge Vendors find & serve customers at
regarding grid integration, efficiency among targeted population regarding
lower cost, risk & time
renewables and clean transportation technologies/toosl in real-life situations
Short-Term Use in planning, management Increased adoption of clean generation & Accelerate the commercial
Follow-on RD&D smart-grid technologies viability of high-priority
Outcomes Venture capital investment Deploy new efficient technologies to technologies and strategies in IOU
(1-4 Years) Deploy new technologies to efficiency utility and third-party programs service territories
programs Knowledge used in planning & mgmt. Publish data/reports/fact sheets/
Clean generation adopted Increased number of producers of articles
Smart grid technologies/tools adopted demonstrated technologies
15 16 17
Mid-Term & W:
Long-Term - Cleaner generation - Greater energy reliability - Lower grid operating costs -Increased safety - Environmental sustainability
Outcomes - Greater efficiency - Enhanced renewable technologies - Greater uptake of Evs - New codes and standards - Benefits to ratepayers
- Support high penetration of renewables - Economic value to California grid - Public health benefits - Clean jobs
(5 - 10 Years)

5 Portfolio Characterization
This section presents summary information about the projects that make up the current
EPIC project portfolio, spanning EPIC 1 and EPIC 2 (the first two triennial plans from
2012–2017). More detail on the projects may be found in Section 11, which presents results
on EPIC project impacts.
Figure 2 below provides a summary of the Program through the end of 2016. There are 296
projects across the two portfolio periods, shown by project type and administrator. As
explained previously, the CEC is responsible for 80 percent of the EPIC budget and is the
only administrator that may conduct Applied R&D and Market Facilitation projects. The
IOUs are restricted to TD&D projects and jointly administer 20 percent of the budget.
The first EPIC portfolio period (2012-2014) is further along than the second, especially for
CEC projects. The CPUC approved the administrators’ first EPIC Investment Plans in
November of 2013, with most projects launched in 2014 and 2015. The CEC typically has a
longer period before launching projects since it utilizes a wider range of stakeholders for
project development, issues solicitations and then awards projects after a lengthy bid
review process. Project launch for IOU administrators may be delayed due to internal
resource availability.
Figure 2: Number of Projects by Administrator and Project Type (through 2016)
120
107
100
87
80
SDG&E
60 55 SCE
PG&E
40
CEC
22 18
20
7
0
MF R&D TD&D MF R&D TD&D
EPIC 1 EPIC 2

Figure 3 shows the projects by status, as of the end of 2016, with 250 active projects, 19
completed projects, 23 on hold and 7 cancelled.36 The bar to the right of the pie chart
shows how the 19 completed projects are distributed by administrator. The CEC has fewer
completed projects due to its lengthier implementation process.
Figure 3: Projects by Status (through 2016)
Cancelled - 7 (2%)
CEC - 1
On-hold -
23 (8%)
PG&E - 14
Complete -
19 (6%)
Ac#ve - 250 (84%)
SCE - 4
36Figure 3 includes three projects that were listed in the Investment Plans but were cancelled before any
funds were spent. These three projects are not shown in Figure 2. Figure 3 includes all seven cancelled
projects—including four that spent some funds before being cancelled and are also shown in Figure 2.

Figure 4 below shows the total budget ($874,099,034) that is authorized for projects for the
first two EPIC investment periods (2012–2017), with the breakdowns by administrator.
Figure 4: Total Authorized Budget (through 2016)
$15,540,000
$72,754,534
$89,000,000
CEC
PG&E
SCE
$696,804,500
SDG&E

Figure 5 shows the total budget that has been allocated to projects by administrator (a total
of $578,242,270). For the IOUs, most of their budget has been allocated (or “encumbered”
as the IOUs refer to assigning budget to a project) to EPIC 1 and 2 projects. The CEC had
not yet issued all the planned solicitations for EPIC 1 and 2 as of the end of 2016.
Figure 5: Budget Allocated to Projects (through 2016)
$15,540,000
$76,939,659
CEC
PG&E
$90,199,248
SCE
$395,563,363 SDG&E
Source: Administrators’ 2016 Annual Reports (based on encumbered funds)

Figure 6 shows the project budget by EPIC period, project type and administrator.
Figure 6: Project Budget by Administrator and Project Type (through 2016)

$200,000,000
$150,000,000
CEC
$100,000,000
PG&E
SCE
SDG&E
$50,000,000
$0
MF R&D TD&D MF R&D TD&D
EPIC 1 EPIC 2


Figure 7 shows the budget that has been spent through the end of 2016, by project type
and administrator. Figure 8 shows the budget spent as a percentage of allocated funds.
Figure 7: Projects by Budget Spent (through 2016)
IOUs - TD&D $64,393,635
CEC - TD&D $27,247,046
CEC - R&D $27,084,156
CEC - MF $2,594,751
$- $30,000,000 $60,000,000 $90,000,000
Funds Spent
Figure 8: Percent of Allocated Funds Spent (through 2016)
34%
30%
% of Budget
20%
4%
0% 10% 20% 30% 40%
IOUs - TD&D CEC - TD&D CEC - R&D CEC - MF
Source: Administrators’ 2016 Annual Reports (based on expenditures as a fraction of encumbered

funds)

The CPUC requires the administrators to map projects to the different elements of the
electricity system value chain, which consists of:
• Grid operations/market design

• Generation
• Transmission
• Distribution
Figure 9 shows the percentage of current EPIC projects that map to each element. The
demand side management category maps to the largest fraction of projects. These projects
generally promote new technologies or operational practices that will produce efficiencies
in end-use consumption of electricity. Other projects primarily administered by the IOUs
look at further understanding and management of end-use consumption, load profiles or
other technology developments (i.e. smart-charging platforms of plug-in electric vehicles).
A given project may map to multiple categories. For example, a project that will result in
the increased utilization of smart-meter monitoring may also be coupled with increased
distributed-energy resource penetration studies, affecting both generation and
distribution. These values are assigned by administrators as reported in their Annual
Reports.
Figure 9: Percentage of Projects by the Electric System Value Chain by Administrator

and Project Type (through 2016)
100%
80%
60% CEC - MF
CEC - R&D
40% CEC - TD&D
IOUs - TD&D
20%
0%
Grid Operations and Generation Demand-Side Distribution Transmission
Market Design Management

6 Program Administration
In this section, we discuss the effectiveness of the Program’s administrative processes and
how they meet the program requirements, as described in the EPIC Administration logic
model (see Section 13.1 for more details), which informed our data collection efforts.
Program administration is a crosscutting activity in the causal chain that is expected to
ultimately lead to the achievement of EPIC’s mid- and longer-term outcomes. For our
assessment, we conducted interviews with the management teams from each
administrator (a total of nine interviews), reviewed program documents such as the
Investment Plans and Annual Reports, conducted 90 project-level interviews associated
with our sample of 54 projects, and conducted a best practices assessment.
The CEC and the three electric IOUs (PG&E, SDG&E and SCE) administer EPIC under the
oversight and control of the CPUC. There are four primary CPUC Decisions that together
established the requirements and administrative procedures for EPIC. The first Decision
(12-05-037) established the purpose and governance for EPIC and funding collections for
2013-2020, providing a policy rationale for continuing public interest funding in the energy
area where private capital is unlikely to provide adequate support. The Decision also
established electricity ratepayer benefits as a mandatory guiding principle, defined as
promoting greater reliability, lower costs and increased safety.
The IOUs’ role was defined to be limited to TD&D programs, and they were prohibited
from investing in generation-only projects using EPIC funds. The Decision (12-05-037)
described the rationale as:
“For activities that are completely pre-commercial in nature (applied research and
technology development), a state agency with public interest objectives [the CEC] is
ideally suited to administer those activities. For activities that are more related to
technology demonstration and deployment on the grid, as technologies and
approaches move toward commercialization, utilities may be better suited to
administer the funding … since they own the infrastructure on which or through
which the technologies will be tested. They also may ultimately become the consumers
of technologies or processes that are designed to improve utility systems, so it will
behoove them to invest in and test some new ideas. Other TD&D activities may be best
suited to a state agency that does not have a business interest in any particular
company or solution.”37
The Decision also ordered that EPIC would be the primary venue for IOUs’ RD&D
expenditures other than RD&D proposed by IOUs as part of their budget applications for
37CPUC. Phase 2 Decision Establishing Purposes and Governance for Electric Program Investment Charge and
Establishing Funding Collections for 2013-2020. 2012.
http://docs.cpuc.ca.gov/word_pdf/FINAL_DECISION/167664.pdf

energy efficiency and demand response. The IOUs could request separate funding for
electric RD&D in their energy efficiency and demand response budget applications, but
they are required to justify why such expenditures could not have been considered within
EPIC.
Subsequent Decisions (13-11-025 and 15-04-020) approved the administrators' 2012-2014

(EPIC 1) and 2015-2017 (EPIC 2) Investment Plans and established additional requirements
such as for Annual Reports, quantifying plan benefits and holding stakeholder workshops.
Decision 15-09-005 clarified the process the administrators should use to introduce new
projects between cycles.
6.1 Compliance with Program Requirements

EPIC was authorized by the CPUC, and a series of Decisions provides the requirements to
which the administrators must adhere. Below in a series of tables, we present each
requirement along with its source, and indicate whether the Program is in compliance
based on our review. The tables are organized by the following categories:
• Contracts
• Quantifying benefits / metrics
• Budget
• Annual reports
• Miscellaneous
We used a combination of sources to conduct the assessment, including program filings

(e.g., Annual Reports and Investment Plans), the sample of projects for which we had
more detailed information (supplemented by interviews with grantees, vendors and
IOU/CEC project managers) and the sample of CEC solicitations/bids and IOU request
for proposals (RFPs) and vendor bids.
Table 11 below addresses the requirements based on Public Utility Code Statutory
Guidance that was referenced in the first EPIC Decision (12-05-037). Based on our review
of our project sample, we found that the Program is in compliance with the statutory
guidance in Public Utility Codes 740.1 and 8360 (see Appendix A).

Table 11: EPIC Requirements Checklist - Statutory Guidance
Requirement Source Compliant? Comments
Guidelines for evaluating D. 12-05-037 / Public Yes Documented in
projects proposed Utility Code 740.1 Investment Plans
Statutory Guidance and confirmed in
review of sampled
projects
State policy to modernize D. 12-05-037 / Public Yes Several projects in
T&D system / smart grid Utility Code 8360 our sample of the
criteria Statutory Guidance EPIC portfolio
address smart grid
and T&D
modernization
Table 12 addresses the requirements for the administrator Investment Plans from the first
EPIC Decision (12-05-037). Based on our review, we can confirm that all these
requirements have been meet for the first two Investment Plans.
Table 12: EPIC Requirements Checklist - Investment Plans

Map projects to electricity D. 12-05-037 / Yes Documented in
system value chain Ordering Paragraph Investment Plans
12
Document funds allocated to D. 12-05-037 / Yes Documented in
each project type category Ordering Paragraph Investment Plans
12
IOUs provide a summary of D. 12-05-037 / Yes Documented in
their EE/DR portfolio R&D and Ordering Paragraph IOU Investment
demonstration activities 12 Plans

Table 13 addresses limitations on projects from the first and second EPIC Decisions (12-05-
037 and 13-11-025). The first three (of four) pertain only to the IOUs, and the last pertains
to all administrators. Based on our review of our project sample, the Program is in
compliance with all the requirements that place limitations on projects.
Table 13: EPIC Requirements Checklist - Limitations on Projects

Source of
Requirement Requirement Compliant? Comments
IOUs’ role limited to TD&D D. 12-05-037 Yes All sampled IOU
projects administered projects are
classified as TD&D
IOUs cannot use funds for EE D. 13-11-025 Yes None of our sampled
projects or Market Facilitation projects involve IOU-
activities administered energy
efficiency or Market
Facilitation activities
EPIC funds cannot be used to D. 12-05-037 / Yes None of our sampled
fund generation-only projects for Ordering projects involved
IOUs Paragraph 13 generation-only efforts for
IOUs
Funds cannot be used for market D. 12-05-037 Yes None of our sampled
support activities or general projects used EPIC funds
education and outreach on basic for general market
value of renewables support, education, or
outreach on the basic
value of renewables
Table 14 below addresses contract requirements from the first and second EPIC Decisions
(12-05-037 and 13-11-025). We reviewed the administrators’ plans for contracting in their
Investment Plans and the summary of their contracting in the project status report that
goes with their Annual Reports, and we also reviewed the sample of CEC solicitations/
bids and IOU RFPs and vendor bids for compliance. We note some issues with the use of
non-competitive bidding, which we explore more fully in Section 6.3.3.

Table 14: EPIC Requirements Checklist - Contracts
Source of
Requirement Requirement Compliant? Comments
Projects must be competitively D. 12-05-037 Yes, Nearly all CEC projects in
bid unless justified in technically, the sample we reviewed
Investment Plans but issues were competitively bid with
with IOU the exception of three
excessive use projects sole sourced
of direct through interagency
award agreement. 18 IOU projects
(of 96 total) were sole
sourced. These cases were
identified in the
administrators’ Annual
Reports. (SCE identified
general cases for the use of
sole source in its Investment
Plan.)
The IOUs may not use EPIC D. 13-11-025 / Yes The IOUs indicate in their
funds for in-house activities Ordering Investment Plans, and
where they are conducting all Paragraph 12 corroborated by the 2016
of the work using its own staff Annual Reports, intent to use
and facilities vendors for all projects
Identify type of funding D. 12-05-037 / Yes Identified in Investment Plans
mechanisms to be used Ordering
Paragraph 12
Eligibility criteria for award of D. 12-05-037 / Yes Documented in IOU
funds Ordering Investment Plans and for
Paragraph 12 CEC, in solicitations
Adhere to IOU grant D. 13-11-025 Yes Documented in Investment
solicitation guidelines Plans and confirmed through
audit of sample projects for
CEC and IOUs
IOUs must report on funds that D. 13-11-025 Yes Documented in Annual
they award to contractors, and Reports, project status
they cannot include those costs report
in their Admin category
Sole sourcing is allowed, but D. 13-11-025 Yes Documented in Annual
must be reported on in Annual Reports, project status
Reports report

Table 15 below addresses requirements for engaging with stakeholders from the first and
third EPIC Decisions (12-05-037 and 15-04-020). As shown, all requirements concerning
stakeholders have been met based on our review of Investment Plans, meeting notices and
workshop presentations. However, we note that we identify room for improvement in the
effectiveness of how stakeholders are engaged in investment planning and project
selection in subsequent report sections.
Table 15: EPIC Requirements Checklist - Stakeholders

Summarize stakeholder D. 12-05-037 / Ordering Yes Comments
comments and administrators’ Paragraph 12 summarized in
response in Investment Plans Investment Plans
Solicit stakeholder input and D. 12-05-037 Yes Input solicited via
expertise at least twice/year and in-person meetings,
notify parties on service list and webinars, and
related proceedings online forums
Certain types of stakeholders D. 12-05-037 Yes All types of
must be consulted (specific list stakeholders are
included in Decision) consulted
Must coordinate with input from D. 15-04-020 Yes Workshops held
CPUC on an annual EPIC on December 3,
Innovation Symposium starting in 2015 and
2015 (counts as one of two December 1, 2016
required workshops)
Table 16 below addresses requirements related to quantifying benefits/metrics from the

first and second EPIC Decisions (12-05-037 and 13-11-025). Based on our review of the
Investment Plans and Annual Reports, the administrators are in compliance with the
initial step of identifying appropriate metrics for each project they propose and
implement. However, not all administrators have plans in place to quantify the metrics. At
the time of our evaluation, few projects had been completed and the in-progress projects
had only been operational for a short time. This precluded a comprehensive assessment of
how the administrators are quantifying project benefits. We did review the processes the
administrators had in place at the time of the evaluation and any activities that had been
conducted on completed projects. See Section 9.2 for an assessment of benefits
quantification.

Table 16: EPIC Requirements Checklist - Quantifying Benefits / Metrics
Indicate which metrics D. 12-05-037 / Yes Documented in IOU
project’s success will be Ordering Paragraph 12 Investment Plans and
measured against in for the CEC, in Annual
Investment Plans Reports
Establish a measurement plan D. 12-05-037 / Unclear The CEC has a process
to quantify benefits based on Ordering Paragraph 12 in place to quantify
metrics in Investment Plans metrics (via project
benefit questionnaires).
The IOUs do not have
plans to systematically
quantify and report on
project benefit metrics.
Must identify metrics for each D. 13-11-025 Yes List included in each
project in Annual Report Annual Report
Table 17 below addresses administrator budget requirements from the first EPIC Decision
(12-05-037). As shown, based on our review of the administrators’ Annual Reports, all
requirements have been met.
Table 17: EPIC Requirements Checklist - Budget

Admin. expenses cap set at 10% D. 12-05-037 Yes All administrators
below cap for EPIC 1
and EPIC 2 (average
across both: CEC
10%, PG&E 8%, SCE
6%, SDG&E 10%)
5% limit for fund shifting across D. 12-05-037 Yes The CEC is in
project type categories compliance per
Annual Reports (IOUs
effectively have
unlimited fund shifting
authority since they
administer only one
program area)

Table 18 below addresses administrator Annual Reporting requirements from the first and
second EPIC Decisions (12-05-037 and 13-11-025), which have all been met.
Table 18: EPIC Requirements Checklist - Annual Reports

Must file by February 28 each D. 12-05-037 Yes IOUs filed 2015 reports
year on Feb. 29, 2015
Include project-level info on # D. 13-11-025 Yes Included in project status
bidders passing the initial reports
screening and ordinal rank of
selected bidder (if not #1,
provide explanation)
Reports must follow a specific D. 13-11-025 Yes The CEC does not match
outline including for each exact numbering of
project’s status report sections, but content is
consistent with outline
Include description of each D. 13-11-025 Yes Included in project status
project including winning bidder reports
Table 19 below addresses miscellaneous requirements from the first, third and fourth EPIC
Decisions (12-05-037, 15-04-020 and 15-09-005). As shown, these requirements have been
met.
Table 19: EPIC Requirements Checklist - Miscellaneous

Independent Evaluation: Requires D. 12-05-037 Yes Evergreen team selected
at least one independent and contact initiated in
evaluation, consultant to be hired 2016
in 2016
Waiver of Program Requirements: D. 15-04-020 NA PG&E submitted an
Use Tier 3 Advice Letters to Advice Letter as required
request a waiver of program to request a waiver of
requirements program requirements
New EPIC Project Approval: Use D. 15-09-005 Yes PG&E used this process38
Tier 3 Advice Letters for new
EPIC projects between triennial
cycles, or for material changes to
existing approved projects
38Note that in Section 8.4, we discuss Investment Plan consistency over the plan period, which includes a
discussion of the Tier 3 Advice Letter requirement and the process for seeking approval for changes.

6.2 Program Management
Based on our review of program reports and information provided during interviews with
the EPIC program managers at each administrator, we determined that the administrators
have a very different approach to how they administer program funds and conduct
projects. In essence, the CEC develops a set of initiatives as part of its EPIC triennial
investment planning process, issues Grant Funding Opportunity notices to solicit project
proposals, selects projects from the proposals and oversees the efforts of the grantees, who
manage the day-to-day project activities. As such, the IOU subject matter experts work
together to create a set of initiatives (using the Joint IOU EPIC Framework), develop
projects that demonstrate technologies or solutions to address current and future grid
issues, and manage vendors, who are hired to conduct specific work or provide equipment
to support the projects.
In this section and throughout the report, we have tended to report on results for the CEC
and IOUs (combined) due to the difference in administrative approach that we noted at
the initial stages of the evaluation. However, we examined each of the four administrators’
processes and projects, and where we noted differences across the three IOUs, we discuss
those differences. Note that those comparisons were limited by the size of the project
sample for each IOU (which reflects the relatively smaller portion of the EPIC budget that
the IOUs receive as compared to the CEC).
6.2.1 CEC
Based on a series of interviews with the CEC’s EPIC program management team and
corroborated by review of the CEC’s Investment Plans and Annual Reports, we learned
that the CEC manages EPIC within its Energy Research & Development (ERD) Division,
which is comprised of four offices (Energy Efficiency, Generation, Systems Integration and
Market Facilitation). The Division employs around 80 staff, mostly engineers and
scientists. The ERD Division regularly coordinates with other CEC divisions, particularly
those focused on energy efficiency and renewables.39
The CEC primarily uses a grant-based process for its projects, whereby grantees propose
specific projects within the confines of a competitive solicitation. This approach is
consistent with the CPUC’s acknowledgement in Decision 12-05-037 that the CEC as a
public agency is suited to a role that stimulates innovation and creates opportunities for
customers to pilot new technologies for the advancement of energy policy. EPIC projects
are conducted under the oversight of CEC Commission Agreement Managers (CAMs).
The CAMs oversee grantees in the development of a scope of work with milestones and
regular reporting for each project, and review project progress reports and invoices and
follow project developments. Each project convenes a technical advisory committee (TAC)
39The ERD Division also coordinates with the Fuels and Transportation Division on Vehicle Grid
Integration, and the Energy Assessment Division/Siting on renewable integration/DERs.

that is made up of both internal and external experts and stakeholders that provide
guidance. In addition, CEC staff and technical leads also stay engaged on project
developments as they relate to specific CEC program efforts or technology and policy
roadmaps. In limited cases, CEC projects may convene a policy advisory committee (PAC),
where a project or set of projects could influence future policy.40
Based on interviews we conducted with the CEC’s CAMs and grantees associated with a
sample of 33 projects, we determined that the grantees drive technical aspects of projects
with input from the CEC. CEC staff learn from projects to inform program areas and
manage scope, contract and budgeting issues. While not necessarily experts on the
technologies being investigated, CAMs are technical staff with an understanding of the
technology and technical issues, internal technical resources, and a structure that allows
them to manage the grants. CAMs are supported by senior staff who are technology
experts.
Nevertheless, some of the 32 grantees that we interviewed commented on various

challenges they have encountered related to project administration. These include:
• Slow payments and administrative processing by the CEC, which can be a

challenge when working with smaller subcontractors who are used to faster
payment cycles.
• The perception that CAMs were sometimes either inflexible or did not have the
authority to be sufficiently flexible when circumstances required non-technical
project adjustments. Specifically, 19 of 32 grantees we interviewed spoke of some
inflexibility from the CEC, often dealing with challenges in reallocating project
funds to employees or tasks.
• Turnover or reassignment among CAMs, which results in some discontinuity but
generally does not have any detrimental effect on projects.
6.2.2 IOUs
Based on a series of interviews with each IOU’s EPIC program management team,
interviews with IOU project managers (21 projects), and our review of IOU Investment
Plans and Annual Reports, we learned that the IOUs implement their projects in-house
and use vendors to conduct a specific scope of a project, such as to provide and maintain
software needed to carry out the demonstration project. Earlier in this section, we
described how the CPUC authorized funding for the IOUs to conduct TD&D projects, as
well as the requirement that they must outsource some portion of each project to vendors.
The IOUs provide oversight of their EPIC projects via internal stakeholder and
40 At the time we conducted our research, the CEC had convened one PAC (Long Term (2050) Energy
Scenarios), which is advising three projects related to GHG reduction goals and the impacts of a changing
climate on the state’s energy systems.

management groups. A project manager is responsible for day-to-day management of the
project in adherence to a project management plan that is developed for each project, and
they regularly report to the internal sponsors who weigh in on any needed project
modifications (or sometimes termination).
• PG&E’s Emerging Technology Group41 includes EPIC along with two other related
technology demonstration programs (21st Century Energy Systems R&D and Smart
Grid Pilot). The number of staff who work on EPIC projects varies, which includes
staff who conduct projects (a mix of engineers, data scientists and product
managers) and staff who manage projects (handling administration and contract
management). Like the other IOUs, they also coordinate with many other internal
departments and technical experts. PG&E has a three-person program management
team that oversees EPIC: an overall program manager (100% time), a regulatory
affairs manager (20% time) and the manager of the Emerging Technology Group
within which EPIC is housed (25-33% time). The team handles overall project
management, regulatory compliance, coordination across their internal groups and
portfolio of projects, and administrative functions. The team is cross functional and
brings different types of expertise based on their function—regulatory knowledge,
administrative and project management experience, and technical background.
• SCE’s Advanced Technology Group manages EPIC in coordination with other
internal technical groups and experts, with nine engineering managers, each of
whom works about half time on EPIC, with another 80 engineers that support the
group. SCE has a two-person management team that is involved on a day-to-day
basis managing EPIC. The team consists of an overall EPIC program manager who
spends roughly 40 percent of his time on EPIC and a technical lead who oversees
the engineers who manage EPIC projects (50% time). A regulatory affairs staff
person also supports the program (25% time).
• SDG&E’s Technology and Systems Demonstrations Group, which is housed within
the Distributed Energy Resources Group, is where EPIC is managed; it also
coordinates extensively with other internal departments and technical experts (such
as T&D and IT departments). It has about 10 staff members working on EPIC at or
near full-time capacity. SDG&E has a three-person management team: one staff
person dedicated to regulatory affairs (10%), another serving as the dedicated
project management lead (has ramped up to 100% recently) and an administrative
support person (around 80% time). The program manager coordinates project
technical leads and other technical staff that conduct the projects, as well as
coordinates with other groups at SDG&E that sponsor the projects (the research
stakeholders).
41Not to be confused with the IOUs’ Emerging Technology Program (ETP), which each IOU implements
separately from EPIC. The ETP focuses exclusively on energy efficiency technologies.

The IOUs’ vendor contracts are fairly straightforward to achieve a narrower scope than
CEC grants, so much of the project management and coordination occurs internally using
the IOU’s standard business practices. We conducted interviews with vendors that the
IOUs used (five vendors), and these vendors mentioned being involved in internal
discussions involving project intent and scope—such as discussions concerning
technology use cases—but were generally not involved in any broader coordination with
stakeholders or external experts as the CEC grantees are.
We offer some observations related to the dual public agency/private utility

administration of EPIC, which are related to the differences just described in how each
administrator approaches projects. First, because CEC projects must go through a
competitive solicitation process with outside entities proposing projects, they take longer
to launch than IOU projects, which are competitively bid far less often. Once launched,
CEC projects take longer to complete. This is due to the fact that the CEC, as a public
agency, must implement its projects in a transparent manner, with multiple opportunities
for public review and input. The IOU projects, on the other hand, are designed to serve the
needs of their internal stakeholders and, as a result, are conducted in a less transparent
manner.
6.3 Contracting
As mentioned previously, the CEC contracts out all of its projects, while the IOUs
implement their projects in-house with the assistance of vendors to accomplish a specific
component of the funded projects.
Before we present the results of our review of grantees and vendors’ opinions with respect
to contracting for EPIC, we offer observations about the different relationships that the
CEC and IOUs have with their contractors. The CEC CAMs oversee the grantees as they
perform the technical project scope of work. (CAMs also lend their expertise by
developing solicitations and sharpening the scope of work; however, the grantee conducts
the research.) Grantees view the Program as a funding source, and the CAMs provide
oversight that those funds are spent judiciously and the project goals are met on schedule
and within budget. IOU vendors conduct a much more limited number of EPIC projects;
they may have a business relationship with the IOU and therefore view the IOU as their
client, from whom they hope to get future business. We think it is important to note this
distinction before we present results from CEC grantees and IOU vendor interviews. That
is, the results are not directly comparable in the sub-sections that follow (6.3.1 and 6.3.2).
6.3.1 CEC
The CEC has an established competitive solicitation and contracting process, which is the
primary method by which it awards funds for EPIC projects. After selecting winning
bidders from its solicitations, the CEC develops a detailed scope of work for all of its

projects that reflects the winning bid along with any necessary clarifications or
modifications. The CPUC requires the administrators to competitively bid EPIC project
work, and when they do not competitively bid a contract, they must report such cases and
justify their use in their EPIC regulatory reporting.42
Our review of CEC grant documents associated with our sample of 32 CEC projects
focused on grant request forms that document project details and expectations of grantees.
Grant request forms are consistently in place and follow a standard format for CEC-
administered projects. These grant request forms describe the following:
• Project awardee general information

• Business meeting information
• California Environmental Quality Act (CEQA) compliance
• List of subcontractors (major and minor) and equipment vendors
• List of all key partners
• Recipient’s administrator/officer & project manager
• Selection process used
• Scope of work
• Budget detail
• CEC 105, questionnaire for identifying conflicts
• Recipient resolution
• CEQA documentation
We explored contracting experiences from the perspective of the 32 CEC grantees that we
interviewed. Some experienced research firms put contracting with the CEC in context,
indicating that the CEC contracting process seems relatively standard and resembles that
of other funding sources that they are used to working with (such as IOU or CPUC
research projects). Nevertheless, in grantee interviews, we did hear of occasional
contracting issues that seemed moderate but noteworthy. Examples include:
• Projects that got a late start due to administrative delays; and

• Inflexibility with budget details. For example, the CEC has a policy that locks pay
scales for a contract period.
The CEC is also required to notify the Joint Legislative Budget Committee in accordance with Public
42
Resources Code section 25711.5(g).

For the eight CEC-administered projects for which we have consistent data on the length
of the contracting period, the median contracting time was three months. The range was
one to seven months.
Overall project funding, on the other hand, has been nearly uniformly identified as
sufficient, even if “more is always better.” No objections to awards were noted by the
administrators.
6.3.2 IOUs
The IOUs are required to contract with vendors for all of their projects. We reviewed their
2016 Annual Reports and Investment Plans to confirm the use of vendors, and we
validated that they have plans to use vendors for all 93 projects from the first two
Investment Plans.
We reviewed the six projects in our 21-project IOU sample that already had a vendor
contract in place, and conducted interviews with five of those six vendors to gather
information about the IOUs’ contracting process and seek vendor input. The contracts for
these projects describe the vendor’s task and terms of the work, but they may not capture
the overall project scope of work or plan when an IOU manages the overall technical effort
internally. That scope of work is summarized in the IOUs' Annual Reports (in the project
status report). However, they do not share a more detailed version like the CEC’s grant
request documents that are shared publicly once projects are awarded and the scope of
work has been developed.
For the IOU-administered projects, vendors said the contracting process was relatively
straightforward, and they did not recall any major issues. However, a couple of minor
issues that were discussed include:
• Irrelevant amendments included in the contract for “Non-American” companies

that prolonged the contracting process; and
• Intellectual property (IP) logistics with result-sharing across IOUs, including one
vendor that noted that additional internal discussions were required because the
administrator wanted to share the vendor’s results with other IOUs.
6.3.3 Non-Competitive Bidding

As noted in the compliance tables presented previously, CEC projects are consistently bid
out competitively, with few non-competitive awards (the CEC sole sourced three projects
through interagency agreements with the University of California).43 The IOUs use non-
competitive bid or sole source contracts much more often. The CPUC has allowed for the
use of sole sourcing on a limited basis. This issue was discussed previously in Section 6.1,
43 These awards were leveraged to help secure a water center in California, along with federal funding.

where we identified that the IOUs are using sole source contracting more frequently than
we believe the CPUC intended, and that these cases are not justified as required by the
CPUC.
Of the 93 active IOU projects that were described in their 2016 Annual Reports, 30 were
competitively bid, 23 were sole sourced and 40 had not yet been determined. Note that
there could be more cases of sole sourcing as the administrators continue to award funds
for EPIC 1 and 2 projects, since 40 of the 93 IOU projects have not yet been contracted and
the CEC has not yet awarded all of its EPIC 1 and 2 funds.
Table 20 summarizes the use of sole sourcing by administrator and Investment Plan
period, as reported in the 2016 Annual Reports. SDG&E provided necessary information
that allowed us to estimate the dollar and percent of project funds being sole sourced,
which was $695,152, equal to 4 percent of the SDG&E’s total project budget over the first
two Investment Plan periods. The CEC sole sourced a total of $2.5 million, which was less
than 1 percent of project funds that were awarded through 2016.
PG&E and SCE did not provide the necessary information that allowed us to estimate the
dollar amount or percent of project funds that are being sole sourced.

Table 20: Use of Non-Competitive Bidding (EPIC 1 and EPIC 2)
Number of % of Sole Source
Total Cases of Projects Project Funding/%
Number of Sole with Sole of Total Project
Projects Source Source Funds
EPIC Phase 1
PG&E 25 7 28% Unknown
SCE 12 9 75% Unknown
SDG&E 5 0 0% NA
CEC 174 2 1% $1,370,000
Total Phase 1 216 18 8%
EPIC Phase 2
SCE 15 0 0% Unknown
SDG&E 6 1 17% $695,152
CEC 28 1 4% $1,130,000
Total Phase 2 79 8 10%
EPIC Phases
1-2
SCE 27 9 33% Unknown
SDG&E 11 1 9% $695,152 / 4%
CEC 202 3 1% $2,500,00 / <1%
Total Phases
291 26 9%
1-2
Source: Administrator 2016 Annual Reports
Notes:
One SCE project is funded across both EPIC Phase 1 and EPIC Phase 2. This project is only included in EPIC
Phase 1 in this table to avoid double counting.
One of the six SDG&E projects used a competitive interview process rater than a competitive RFP process.
CPUC Decision 12-05-037 allowed for some circumstances where administrators could use
non-competitive bidding:
“Finally, on the issue of competitive bidding, this is generally our selection process of
choice in all areas. However, there may be limited and unique circumstances where it
is not possible or desirable. In each Investment Plan, the administrators may propose a

limited authorization for non-competitive bidding for particular purposes. An
example, as suggested by the Efficiency Council in their comments on the proposed
decision, could be continuation of funding for successful projects. These exceptions to
competitive bidding should be justified separately and clearly for a specific purpose.
During consideration of the first set of investment plans, we will also consider whether
there should be a separate approval process required for any contract or grant not
awarded through a competitive bidding process, to set a higher standard for the use of
a non-competitive process.”
SCE’s EPIC 2 Investment Plan identified where it might use non-competitive bidding in
general cases.
“SCE plans to comply with the Commission’s [the CPUC's] requirements for
competitive bidding, but will continue to use non-competitive awards in limited
instances such as when:
• Material or services required are available from only one reliable source and no
other supplier will satisfy utility requirements;
• Specialized knowledge, skill, experience or expertise is needed for the work and
only one supplier is determined to have what is needed;
• Bidding is cost prohibitive relative to the cost of materials or services needed;
• An opportunity exists to develop Diverse Business Enterprise suppliers;
• The procurement provides special discounts, rates, or terms and conditions (i.e.,
cost share) that are not available under normal competitive conditions; or
• Equipment, materials, or services are obtained for trial testing, research or
experimental work.”
We did not see any comments from PG&E in its second Investment Plan about the use of
sole sourcing; however, in its Annual Report, PG&E offered a justification for each use of
sole source projects.
We did not find any comments from SDG&E in its first Investment Plan about the use of
sole sourcing. SDG&E indicated the use of sole sourcing in its Annual Report, but did not
offer any justification.
We reviewed the use of sole-sourced contracting by IOUs by analyzing the reported

justifications provided by PG&E to determine applicability. We also assessed the potential
for justification within SCE's and SDG&E’s portfolio, though explicit statements of
justification were not provided in their Annual Reports. SCE only had the general set of
exceptions it provided in its Investment Plan. As indicated in the excerpt above, the CPUC
Decision stated that “exceptions to competitive bidding should be justified separately and
clearly for a specific purpose.” We determined that the PG&E justifications resulted from

program changes to licensed equipment, platforms or software that could only adequately
be provided by the vendors responsible for these systems. The types of projects that these
sole-sourced vendors engaged in were, with the exception of one case, operating on utility
monitoring, display or management of grid operations that relied on these unique
tools. While SCE and SDG&E did not provide justifications in their reports, the similarity
of project types indicates in our estimation that similar sole-sourced vendor-supplied
services were also required for successful project implementation.
It is not clear at this time what weight the value of sole-sourced contracts has within the
entire project non-administrative budget. Based on our review of the sample of IOU
projects that had contracted with vendors, only a small portion of the overall project
budget was applied to sole-sourced project work. The requirement to report the existence
of sole-sourced contracts does not specify this level of reporting, and within current
reporting protocols, the existence of any sole-sourced work, no matter how small,
identifies the entire project as 'sole-sourced'.
Our assessment is that the CPUC was expecting a more specific justification to be given in
the Investment Plan (on a project by project basis). The Annual Reports identify the cases
of sole sourcing (which projects and firms), but SCE’s and SDG&E’s do not offer a specific
justification, while PG&E’s does. SDG&E provides estimated budget amounts to be
allocated to project vendors; however, PG&E and SCE do not provide this information.
The CEC provides this information for all projects, since all the project budget goes to the
grantee. Such budget information could help the CPUC determine the extent to which
funding is being awarded in these cases as a percentage of total IOU EPIC project budgets.
6.4 Best Practices Comparison

Our review of best practices examined issues related to administration of effective RD&D
programs. This section summarizes best practices pertaining to program management,
contracting and match funding. The primary peer RD&D programs44 that we included in
our assessment are:
• The Department of Energy’s (DOE's) Small Business Innovation Research (SBIR)

Program – awards Federal Research/Research and Development (R/R&D) grants
to small businesses through various federal agencies, including the DOE.
• The DOE’s Small Business Technology Transfer (STTR) Program – facilitates R&D
cooperation between small businesses and research institutions.
• The DOE’s Advanced Research Projects Agency-Energy Program (ARPA-E) –
provides funding for technologies that are too early for private sector investment
but have the potential to lead to new ways to generate, store and use energy.
44 Fuller descriptions of the primary and secondary peer RD&D programs are included in Section 3.10.

• The New York State Energy Research and Development Authority's (NYSERDA’s)
Technology and Market Development (T&MD) Program – tests, develops and
introduces new technologies, strategies and practices that build the statewide
market infrastructure to reliably deliver clean energy to New Yorkers.
6.4.1 Program Administration and Context

Although the selection of the peer RD&D programs for the best practices review focused
on key similarities between these programs and EPIC, it is important to note unique
characteristics of EPIC. For example, EPIC is administered by four separate entities while
most of the peer RD&D programs are overseen by one main department or agency. EPIC
is also funded by ratepayers, authorized by the state’s public utilities regulatory
commission, and requires legislative spending approval (for the CEC). As a result, EPIC
has a central focus on ratepayer benefits. These features of EPIC are important for
contextualizing the findings from the best practices assessment.
6.4.2 Program Management

Findings from the literature review and in-depth interviews with peer RD&D programs
show that effective RD&D programs have a core internal staff that oversees design and
implementation. The program administration team facilitates relationships with investors,
government agencies, small and large companies, and other organizations. Because these
programs support new and emerging technologies, the peer RD&D program
administrators also rely on other experts both within and outside of their respective
agencies who may have specialized knowledge of specific technologies or concepts. For
example, when developing a funding opportunity, the program administrators will
convene industry experts to help inform the initiative. Interviewees also spoke about
coordinating with other state or federal agencies and industry groups to stay current on
topics affecting their work.
The interviewees from all of the peer RD&D programs cited the depth of knowledge and
expertise of the program administrators, consultants and other stakeholders that
contribute to effective program implementation. Each of the RD&D programs employs
staff with strong technical expertise. In general, program staff are also responsible for
ensuring that the Program’s overall goals and objectives are reflected in the portfolio of
projects.
Commercialization Supports
Program staff also coordinate with internal and external experts to support projects.
Program staff and other institutional supports play an important role in facilitating
technology transfer and market development mechanisms for projects. For example, two
of the four primary peer RD&D programs offer support and resources to promote
commercialization activities.

• ARPA-E’s Technology-to-Market Team provides direct business-related technical
support and assistance to grantees. In addition to providing guidance on the project
team’s Technology-to-Market Plan, the Technology-to-Market advisors assist
projects in a variety of ways, including helping them conduct market assessments,
identify potential investors, and develop and deliver a business pitch. ARPA-E also
holds an Annual Energy Summit that convenes individuals and organizations from
industry, academia and government and provides an opportunity for project
managers to meet with potential partners, collaborators or investors.
• Projects funded through NYSERDA’s T&MD Program can also access
commercialization-related resources available through the Entrepreneurs-in-
Residence (EIR) Program, which is very much like ARPA-E’s Tech-to-Market Team.
In addition, NYSERDA funds six business incubators across the state and supports
a Proof-of-Concept Center where businesses can access existing intellectual
property and work on commercializing it.
Helping Projects Respond to Unexpected Changes

All four of the primary peer RD&D programs have structures in place to help projects
identify and capitalize on opportunities to change course, when needed, to maximize the
projects’ success. Because the technology innovation process is not linear, it often requires
program administrators, grantees and others involved in the RD&D process to be adept at
recognizing critical opportunities in order to maximize success. The SBIR Program allows
projects to be able to reallocate up to 10 percent of their budget without prior approval.
Because ARPA-E and NYSERDA’s T&MD Program both have an explicit focus on
commercialization, advisors for their respective programs actively work with project
teams to identify appropriate pivots, and may help facilitate these course corrections.
6.4.3 Contracting
Contracting times vary substantially across the different peer RD&D programs. One of the
secondary peer RD&D programs has the shortest turnaround of one week. On the other
end of the spectrum, one of the primary peer RD&D programs takes roughly three to six
months on average. Another primary peer RD&D program, which reportedly takes six
weeks, allows the project team to begin working on the project at-risk up to 90 days before
the contract is finalized.
6.4.4 Comparison with EPIC

In general, with one exception, we found that EPIC compares favorably with the peer
RD&D programs in the best practices assessment. Like the peer RD&D programs, EPIC
utilizes a core internal staff that coordinates with internal and external stakeholders. We
also found that the contracting time for EPIC was within the range of the peer RD&D
programs. However, compared to peer RD&D programs, EPIC has only a nominal focus
on support for commercialization, limited to a requirement for a project-based
technology/knowledge transfer plan and a production readiness plan (as needed).

The Program does fund commercialization assistance through Market Facilitation projects,
but these supports do not directly benefit EPIC projects. The Market Facilitation projects
help entrepreneurs and researchers overcome non-technical barriers to accelerate the
commercial viability of their technologies. The commercialization supports include
training or technical assistance related to conducting market research and analyses;
assistance with developing tools to help transition research and innovations from the lab
to the market; and formal networks that bring together entrepreneurs, researchers and
investors to exchange ideas, best practices and information on market opportunities. As
noted above, two of the four primary peer RD&D programs—ARPA-E and NYSERDA’s
T&MD Program—similarly offer commercialization resources activities, but these
supports directly benefit their grantees. Table 21 summarizes the broad categories of
effective practices that were referenced during interviews with the peer RD&D programs.
Table 21: Best Practices Comparison of EPIC and Peer RD&D Programs
Peer RD&D Programs Program Current EPIC
Management Practice Comments
Core internal staff that coordinates with The CEC does this; the The CEC engages external experts but
internal and external stakeholders IOUs rely mostly on a manages the process internally.
single stakeholder. The IOUs work with the Electric
Power Research Institute (EPRI) to
determine research priorities, but
manage the process internally.
Offer support and resources to promote The CEC does this The CEC funds Market Facilitation
commercialization activities (two of four through Market projects, which currently do not target
primary peer RD&D programs) Facilitation program. Applied R&D or TD&D projects.
Explicit requirement for project-based The CEC does this. Process appears to be sufficient.
knowledge or technology plan
Help projects change course, when EPIC does this. Process appears to be sufficient.
needed, to maximize the projects’ success

7 Investment Planning Process
In this section, we discuss the effectiveness of the Program’s investment planning, as
described in the EPIC Administration logic model (see Section 13.1 for more details),
which informed our data collection efforts. Investment planning is a critical first step in the
causal chain that is expected to ultimately lead to the achievement of EPIC’s mid- and
longer-term outcomes. For our assessment, we conducted interviews with the
management teams from each administrator (a total of nine interviews), reviewed
program documents such as the Investment Plans and Annual Reports, conducted 90
project-level interviews associated with our sample of 54 projects, conducted nine
interviews with EPIC stakeholders, reviewed comments submitted to the EPIC Idea
Exchange and conducted a best practices assessment.
The administrators are required to file triennial Investment Plans, and to-date plans for the
first three triennial program period plans have been filed. The first set of plans covering
2012–2014 was approved in November of 2013 in CPUC Decision 13-11-025, and the
second set of plans covering 2015–2017 was approved in April of 2015 in CPUC Decision
15-04-020. The administrators recently filed their third triennial plans for 2018–2020, which
they were required to do by May 1, 2017. The first EPIC Decision (12-05-037) that
authorized the Program identified specific requirements for the Investment Plans,
including:
• Screening and scoring criteria for evaluating funding proposals (which were
established in the next Decision, 13-11-025);
• Metrics against which success will be measured (which were established in the next
Decision, 13-11-025) (See Appendix A);
• Addressing how Public Utility Code 740.1 and 8360 statutory guidance are applied
to program plans (See Appendix A);
• A summary of stakeholder comments received during the development of each
Investment Plan and the administrators' responses to the comments; and
• Mapping of planned investments to the electricity system value chain.
7.1 Investment Plan Scoping

As discussed throughout this report, the CEC and IOUs have different approaches for
administering EPIC. The CEC administers research grants while the IOUs conduct the
research internally. This fundamental difference in administration is exemplified in their
investment planning approaches. Each administrator is required to address a number of
elements required by the CPUC, which we confirmed to have been addressed in their
EPIC 1 and EPIC 2 Investment Plans.

As mentioned previously in Section 6, throughout the report we have tended to present
results for the CEC and all three IOUs combined due to the difference in their
administrative approach that we noted at the initial stages of the evaluation. However, we
examined each of the four administrators’ processes and projects, and where we noted
differences across the three IOUs, we discuss those differences. Note that those
comparisons were limited by the size of the project sample for each IOU, which reflects the
relatively smaller portion of the EPIC budget that the IOUs receive as compared to the
CEC.
7.1.1 CEC Plan Scoping

Based on a series of interviews we conducted with the CEC’s EPIC program management
team, we learned that the CEC approaches investment planning from two perspectives.
First, it identifies strategic topical issues facing the state (e.g., water conservation and tree
mortality for prior Investment Plans) along with the state’s clean energy policy priorities
for a top-down view of identifying a series of objectives that EPIC research projects should
address during the next funding cycle. Next, they take a bottom-up approach with input
from their in-house technical staff who conduct energy research (either in the R&D
Division’s energy research office or in other divisions such as Energy Efficiency and/or
Renewables) who identify gaps in the available research. Often, publicly available
interagency (e.g., CEC, CPUC, CARB [California Air Resources Board] and CAISO
[California Independent System Operator]) research road maps inform CEC staff
assessments of research gaps. The CEC’s program management team also described how
they orient the top-down and bottom-up processes to EPIC’s guiding principles. They do
this to ensure that any strategic objectives or areas on which they decide to focus their
planning are in alignment with Program requirements. The guiding principles also serve
as screening criteria, since any topics that might be pursued that are inconsistent with the
Program’s guiding principles would be screened out. They also said that they coordinated
with the other administrators to prevent redundancies (such as with the IOUs’ Emerging
Technology Programs). Similar to the IOUs, the CEC also solicits input from experts,
stakeholders and the public (which is explored in the next subsection, 7.2), to identify gaps
and to prevent redundancies with other efforts.
The CEC’s Investment Plan document identifies a set of strategic objectives that are
intended to result in a project portfolio that selects the most promising technology
solutions that align with energy policy. The plan is organized by program research area
(or project type, i.e., Applied R&D, TD&D and Market Facilitation), identifying the
strategic investment objectives along with a number of funding initiatives that address
research gaps, as determined by the CEC and corroborated by stakeholders, for each
technology area. Each objective is discussed in detail, including barriers and challenges
and how any prior Investment Plan investments have already addressed some of those.
For each objective, a set of initiatives is presented that focuses on a particular area of
research, along with a discussion of each that includes the purpose of the initiative,

relevant stakeholders of the research, and background on the research area including
research findings to date on the subject and relevant policy and other developments. The
plan also indicates alignment with EPIC guiding principles and key policy drivers.
We conducted analysis on the administrators’ project portfolios spanning EPIC 1 and EPIC
2 to attempt to independently validate the effectiveness of their investment planning
processes and frameworks in leading to a set of projects that meets the Program’s goals.
First, we reviewed their frameworks to identify how project funding was awarded to each
element in the framework (i.e., CEC strategic objectives). Next, we assessed each project to
independently validate the extent to which projects align with EPIC’s guiding principles
and address the state’s clean energy policies. We conducted this analysis based on a
review of the administrators’ Investment Plans and Annual Reports. We discuss the
results of both of these analyses below.
CEC Framework Review

The results of the review of CEC’s framework are shown in Table 22. The CEC has a total
funding amount of $395,563,363 for EPIC 1 and EPIC 2 that was allocated across 20 unique
strategic objectives.45 All but two of the strategic objectives resulted in one or more projects
being funded. The two initiatives that were not yet awarded projects are currently in
progress and will be included in future solicitations. This result indicates that the CEC was
successful in attracting viable bids for projects for nearly all the initiatives it identified in
its Investment Plan as being required to meet its Investment Plan goals across the first two
funding cycles. The table indicates that two of the strategic objectives that are devoted to
energy efficiency (Strategic Objective 1 (S1): Improve Energy Efficiency Technologies and
Strategies in California's Building, Industrial, Agriculture, and Water Sectors and Strategic
Objective 12 (S12): Demonstrate and Evaluate the Technical and Economic Performance of
Emerging Energy Efficiency and Demand-Side Management Technologies and Strategies)
together account for 30 percent of awarded funds. The other strategic objectives have
relatively lower numbers of projects and funding amounts. The CPUC has not indicated
which of the areas of policy should be prioritized, so we are not able to say whether a
focus on energy efficiency (and less of a focus on other topics such as renewables and
smart grid) is optimal or problematic. This type of analysis provides a useful starting point
to assess the portfolio and which areas are receiving the most funding.
45Strategic Objective 18 was modified from EPIC 1 to EPIC 2, as shown in the table. Strategic Objective 15
was addressed in a later solicitation that occurred after we conducted our research, and the CEC had
released a Request for Comments to get further stakeholder input on the solicitation scope of Strategic
Objective 19. The Request for Comments was released on February 27, 2017, with comments due on March
13, 2017. The CEC indicated it planned to release the solicitation by August 2017.

Table 22: CEC’s EPIC Project Funding By Strategic Objective (EPIC 1 and EPIC 2)
Proportion of
Strategic Objective $ Amount N Budget
S1: Improve EE Technologies $66,253,005 38 17%
S2: Enable Cost-Effective Demand Response $28,136,304 9 7%
S3: Develop Innovative Solutions $16,508,646 11 4%
S4: Develop Utility Scale Renewable Technologies $10,869,219 8 3%
S5: Reduce the Environmental and Public Health Impacts $19,154,881 34 5%
S6: Advance Smart Inverters to Manage Areas with High Penetrations of PV $5,401,868 5 1%
S7: Develop Distribution Modeling Tools for the Smart Grid $1,690,055 2 0%
S8: Advance Customer Systems to Coordinate with Utility Communication Systems $8,870,498 6 2%
S9: Advance EV Infrastructure to Provide Electricity System Benefits $6,681,669 5 2%
S10: Leverage CA Clean Energy Technologies & Companies $19,435,655 6 5%
S10_2: Advance the Early Development of Breakthrough Energy Concepts $20,211,957 1 5%
S11: Provide Federal Cost Share for Applied Research Awards $750,000 1 0%
S12: Evaluate the Technical Performance of EE Technologies and Strategies $49,981,350 22 13%
S13: Evaluate Clean Energy Generation Technologies and Strategies $42,334,079 13 11%
S14: Demonstrate Integration of EE Demand-side Resources $40,257,789 12 10%
S16: Enhance Current Regulatory Assistance and Permit Streamlining Efforts $17,437.354 12 4%
S17: Strengthen the Clean Energy Workforce $8,908,107 2 2%
S18 EPIC 1: Guide EPIC Investments Through Outreach Activities $18,101,945 8 5%
S18 EPIC 2: Foster the Development of Energy Technologies into Businesses $12,091,373 5 3%
S20: Accelerate the Deployment of Energy Technologies in IOU Territories $2,487,609 2 1%
Total $395,563,363 202 100%

CEC Assessment of Program Goals
Next, we reviewed information provided in the Investment Plans, Annual Reports and
solicitations, and used our expert judgment to make an assessment of whether each project
in the CEC’s EPIC 1 and EPIC 2 portfolios meets each of the Program’s guiding principles
and whether it supports key energy policies.
The CPUC established ratepayer benefits as the Program’s mandatory guiding principle;
as such, we would expect that all projects would provide ratepayer benefits. The CPUC
also directed the administrators to address complementary guiding principles in their
Investment Plans:
• Providing societal benefits;

• Reducing GHG emissions in the electricity sector at the lowest possible cost;
• Supporting California’s loading order to meet energy needs, first with energy
efficiency and demand response, second with renewable energy (distributed
generation and utility scale), and third with clean conventional electricity supply;
• Supporting low-emission vehicles and transportation;
• Providing economic development; and
• Using ratepayer funds efficiently.
Many of the complementary guiding principles are broad and may also overlap, and one
would expect a high proportion of projects to meet many of them (such as providing
societal benefits, supporting the loading order and reducing GHG emissions). Others have
a narrower focus, and one would expect only a small subset to meet the guiding principles
(e.g., supporting low-emission vehicles).
Table 23 shows the results of our assessment, with the number of projects and proportion
of funds the CEC awarded to projects in its EPIC 1 and 2 portfolios that address each EPIC
guiding principle. Based on the above analysis, all of the CEC’s EPIC 1 and EPIC 2 projects
are designed to provide ratepayer benefits, as mandated by the CPUC. When we reviewed
the types of benefits that would accrue to ratepayers, we found that the majority (63%) of
CEC projects are expected to directly benefit the entire population of ratepayers, by
leading to increased reliability of the electric power grid, increased grid operational safety
and reduction in ratepayer costs. The remainder of projects will initially generate benefits
for a subset of ratepayers, but will eventually benefit everyone. For example, biogas
capture and microgrid activities for dairy farm operators, or wastewater capture and
purification process developments for specific and limited-use industry processes, are
expected to initially benefit a smaller subset of ratepayers but over time will lead to more
sustainable water use and lower energy usage, generating broader benefits. This mix of
projects is what one might want to see, with sufficient diversity across projects to address
both the broader needs that benefit all ratepayers and the unique needs that immediately
impact only a subset of ratepayers, but still will benefit everyone eventually.

A very high percentage of CEC projects meet the broad complementary guiding
principles, such as providing societal benefits and supporting the loading order, which is
what we would want to see. A small proportion of projects meet the principle related to
supporting low-emission vehicles, which is expected and indicates an appropriate focus
on a more narrowly technology-focused program goal. While we cannot say whether 25
percent is the right number, we would not want a high proportion of projects being
focused on one particular policy and technology area, in order to ensure a diverse
portfolio that meets many policy objectives.
Table 23: CEC’s EPIC Project Funding by Guiding Principles (EPIC 1 and EPIC 2)*
(N=202)
Proportion of
EPIC Guiding Principle N $ Amount Budget
Ratepayer Benefits 202 $395,563,363 100%
Broad ratepayer benefits 122 $248,619,809 63%
Narrow ratepayer benefits 80 $146,943,554 37%
Societal Benefits 180 $361,378,599 91%
GHG Mitigation 165 $348,372,424 88%
Health and Safety 170 $346,271,648 88%
Supporting the Loading Order 163 $335,679,461 85%
Operating Efficiency and Reliability 159 $328,168,535 83%
Economic Development 118 $276,694,575 70%
Supporting Low-Emission Vehicles 40 $99,930,329 25%
*A given project along with its budget can appear multiple times in this table.
Next, we examined how the CEC’s EPIC 1 and EPIC 2 projects address the following key
energy policy areas (which are described in more detail in Section 2.2):
• Greenhouse gas (GHG) reduction – the state’s ambitious goals to reduce GHG
emissions (AB 32 that set a goal of reducing GHG emissions to 1990 levels by 2020,
and SB 350’s goals for 2030 to reduce GHG emissions 40 percent below 1990 levels,
and 80 percent below 1990 levels by 2050).
• Renewable Portfolio Standard (RPS) – related to GHG reduction goals, the state’s
requirement for electricity retailers to serve 33 percent of their retail sales with
renewable energy procurement, and SB 350 increased that to achieving 50 percent
by 2030.

• Smart grid – upgrades to the state’s existing transmission and distribution system
to improve efficiency, reliability, economics and sustainability of electricity services
• Energy efficiency – supporting the state’s Long Term Energy Efficiency Strategic
Plan, which seeks to achieve maximum energy savings across all major groups and
sectors in the state.
• The loading order – the state’s energy resource guide, which is to decrease
electricity demand by increasing energy efficiency and demand response, and meet
new generation needs first with renewable and distributed generation (DG)
resources, and second with clean fossil-fueled generation.
• Distributed energy resources (DER) – integrated resource planning efforts that will
identify an optimal portfolio of resources to achieve the state’s GHG goals and
meet the challenge of renewable integration.
• Electric vehicles – efforts to expedite zero emission vehicle commercialization,
including the development of a vehicle-to-grid integration road map
• Energy storage – collaborative efforts to meet the target for 1,325 MW of energy
storage to be procured by 2020 and installed by 2024.
• Clean jobs – job creation goals associated with the acceleration of renewable energy
production, energy efficiency and energy storage.
Table 24 shows the proportion of EPIC funds the CEC awarded to projects in its EPIC 1
and 2 portfolios that address each California policy goal based on our independent
assessment of each project. The table shows that a very high percentage of projects meet
the broad policy goals of distributed energy resources, supporting the loading order,
energy efficiency and generating clean energy jobs. (Many of these broad policy goals in
fact overlap, so a project focused on energy storage would also support distributed energy
resources and the RPS, and would likely generate clean jobs.) A smaller proportion of
funding is allocated to projects that will support meeting more narrowly-focused policy
goals of energy storage, electric vehicles and the smart grid. In the next subsection, we
present this same analysis for the IOU projects, where we find they have allocated a higher
proportion of funding toward projects that support polices related to the smart grid and
RPS in particular. As we will discuss in that section, we found that the administrators
complement each other with the projects proposed for RPS, covering demonstrations to
support upgrades to the grid to accommodate increases in renewables (IOUs), and
Applied R&D and Market Facilitation projects sponsored by the CEC that address policy
and research needs to support meeting RPS goals.

Table 24: CEC’s EPIC Project Funding by California Policy Goals (EPIC 1 and EPIC 2)*
(N=202)
Proportion of
California Policy Goals N $ Amount Budget
Distributed Energy Resources 189 $380,090,360 96%
GHG Reduction 165 $348,372,424 88%
Loading Order 164 $337,244,861 85%
Energy Efficiency 134 $296,498,362 75%
Clean Energy Jobs 118 $276,694,575 70%
Energy Storage Adoption 55 $138,219,867 35%
Renewable Portfolio Standard 53 $129,748,555 33%
Electric Vehicles 40 $99,930,329 25%
Smart Grid 57 $92,372,281 23%
Next, we present the same set of analyses for the IOUs. We note that these analyses offer
just one possible way of assessing whether administrators’ planning processes are
effective. Based on this way of looking at the CEC’s projects, we independently verified
that every project meets the required guiding principle of providing ratepayer benefits and
supports at least one key energy policy area in a meaningful way. The CEC’s current
portfolio is diverse in terms of the types of ratepayer benefits it offers (with projects
focused on both broad benefits and certain subsets of customers with specific needs) and
how it addresses the broad and technology-specific policy areas. In both those cases, we
found a higher proportion of funding directed where one would expect (where the policy
objectives are broad and overlapping) and less where one would expect (where the
policies are more narrowly focused). In addition, our analysis makes clear the challenges
of assessing planning processes that address 20 strategic objectives and several guiding
principles all nested within nine or more California policy objectives. Since there have not
been any priorities set among the guiding principles and policy areas, we cannot
determine whether the current investment planning processes lead to an optimal portfolio.
However, this analysis approach provides a starting point for further review of the
portfolio and development of priorities.
7.1.2 IOU Plan Scoping

Based on a series of interviews we conducted with the IOUs’ EPIC program management
teams and review of their Investment Plans and Annual Reports, we learned that the IOUs
developed their own joint framework to guide their investment planning to ensure they
offer the right mix of projects in their portfolios. The framework includes four investment
areas (all within the TD&D area) that the IOUs have identified as critical areas on which to

focus in order to modernize the grid. In their Investment Plan documents, the IOUs
identify demonstration objectives for each of the four investment areas, which are aligned
with a description of intended project benefits, EPIC guiding principles and key policy
drivers and objectives. Each investment area is also mapped to the electric system value
chain as indicated:
• Renewables and Distributed Energy Resources (DER) Integration – maps to grid

operations/market design, focuses on renewables and distributed energy resources
integration, and supports the state’s RPS, GHG emission reduction and energy
storage goals.
• Advanced Asset Management and Optimization – maps to T&D, focuses on grid
modernization and optimization, and addresses SB 17 and smart grid planning and
implementation.
• Customer Products/Service Enablement and Integration – maps to DSM, focuses on
the integration of DSM with the smart grid, and supports ZNE policies.
• Cross-Cutting/Foundational Strategies and Technologies – maps across the electric
value chain, and focuses on smart grid architecture, cybersecurity,
telecommunications and standards development.
The IOUs use very similar processes to develop their Investment Plans. They each use a
process to identify potential project areas that is focused internally, guided by their
technical staff whom they refer to as their stakeholders. Each IOU EPIC program
management team consults with their internal technical stakeholders to identify the most
critical needs that justify a demonstration project. The IOUs, unlike the CEC, are better
able to draw on the specific technical expertise of internal staff who work directly on these
issues to help inform and guide Investment Plan scoping. The management team filters
each potential demonstration project by whether it is market-ready and deployable, or if it
has already been deployed, could it be tested in a new and novel way. They consult with
their internal management to provide input on their draft plans as they develop from
initial ideas to the draft Investment Plan. The IOUs, along with the CEC, jointly meet with
the Electric Power Research Institute (EPRI), which conducts RD&D projects focused on
electricity generation, delivery and use in collaboration with the electricity sector. This
semi-annual EPRI advisory committee meeting46 is a one-day feedback session to examine
gaps, risks and potential duplication of research. The IOUs and EPRI told us during
interviews that EPRI provides insight into the latest advances in energy-related
technologies and research within California as well as nationally and globally. While the
IOUs solicit other stakeholders besides EPRI to weigh in on their plans, their plans are
very far along by the time they hold stakeholder workshops, and input provided at these
Funders of specific programs meet to define work and review work on those programs. A committee
46
meeting is for funders of a specific program, which is open to IOUs and the CEC.

forums does not appear to shape their plans substantially. (We discuss stakeholder
outreach in the next section more thoroughly.)
Once they have a set of potential projects based on input from their internal stakeholders
and EPRI, the IOUs indicated that they ensure that projects are relevant to state energy
policies and meet all EPIC program requirements. Thus, our assessment is that these last
two criteria are used as driving factors (along with their program budget) applied to their
working list of projects. The main source of project origination, based on the IOUs’
description of their processes and also verified by review of their projects and how they
engage stakeholders, is their internal IOU stakeholders. However, because many of those
internal stakeholders are often working on state energy policies, the internal needs are
often complementary with state policy needs.
Next, the IOUs described to us how they use their joint framework to structure their
written Investment Plans and as a final check to ensure that projects are balanced across
the four investment areas. The IOUs described how they discuss their initial plans with the
other IOUs during their biweekly coordination calls47 and their meeting with EPRI to
ensure no duplication, or to coordinate on a project if appropriate. We also corroborated
that there is little duplication in the first two Investment Plans as a result of coordination
efforts (we discuss coordination more thoroughly in Section 8.5.) However, we did note
some potential areas of duplication in the IOUs' EPIC 3 plans, which suggests the need for
more thorough review of IOU project plans (in the Investment Plans and throughout
project implementation).
As mentioned in the previous section that focused on the CEC’s investment planning
processes, we conducted analysis on the administrators’ project portfolios spanning EPIC
1 and EPIC 2 to attempt to independently validate the effectiveness of their investment
planning processes and frameworks in leading to a set of projects that meets the Program’s
goals. Below, we present the results of the analysis of the IOUs’ framework and
assessment of program goals.
IOU Framework Review

The results of the review of the IOUs’ framework are shown in Table 25. The IOUs have a
total funding amount of $182,678,907 allocated across their four unique research
categories. The IOUs fund projects fairly evenly across their four investment areas,
indicating that they followed through and launched projects in each area that they
identified in their Investment Plans. The Crosscutting/Foundational Strategies and
Technologies investment framework element has the largest percentage of projects funded
(33%).
47All four administrators meet biweekly; and on the off week, the IOU administrators meet.

Table 25: EPIC Project Funding By IOU Framework (EPIC 1 and EPIC 2)
Proportion of
IOU Framework $ Amount N Budget
Cross-Cutting/Foundational Strategies and
$60,093,566 8
Technologies 33%
Customer Products/Service Enablement
$44,707,865 16
and Integration 24%
Advanced Asset Management and
$44,543,686 29
Optimization 24%
Renewables and Distributed Energy
$33,333,790 17
Resources Integration 18%
Total $182,678,907 71 100%
Note: Twenty-four projects are on-hold at the time of this evaluation, and the IOUs have not allocated
funding for these projects. These projects are excluded from this analysis. They include 1 Cross-Cutting/
Foundational Strategies and Technologies, 5 Customer Products/Service Enablement and Integration, 14
Advanced Asset Management and Optimization, and 4 Renewables and DER Integration projects.
IOU Assessment of Program Goals

Using the same approach described above for the CEC, we independently assessed
whether each project in the IOUs’ EPIC 1 and EPIC 2 portfolios meets each of the
Program’s guiding principles and whether it supports key energy policies. Table 26 shows
the results of our assessment, with the number of projects and proportion of funds the
IOUs awarded to projects in its EPIC 1 and 2 portfolios that address each EPIC guiding
principle.
All of the IOUs’ EPIC 1 and EPIC 2 projects are designed to provide ratepayer benefits, the
first category of guiding principles in Table 26, as mandated by the CPUC. When we
reviewed the types of benefits that would accrue to ratepayers, we found that the vast
majority (85%) of IOU projects are expected to directly benefit the entire population of
ratepayers (broad ratepayer benefits in the table), increasing the reliability of the electric
power grid, improving grid operational safety, and reducing ratepayer costs. These IOU
projects are designed to achieve these broad benefits by addressing grid communications
and interactivity and grid maintenance, optimization, planning and management. The
remainder of the IOU projects, such as microgrid and vehicle-to-grid applications, and
various types of market research, are expected to initially benefit a smaller subset of the
ratepayers such as electric vehicle owners and grid-system operators (narrow ratepayer
benefits in the table). However, if successfully demonstrated, such projects are expected to
eventually lead to further system or regulatory advances that may provide greater societal
benefits. This mix of projects is what policy makers might want to see, with sufficient
diversity across projects to address both the broader needs that benefit all ratepayers and
the unique needs that immediately impact only a subset of ratepayers, but still will benefit
everyone eventually.

Looking at the rest of the guiding principles that follow the ratepayer benefits guiding
principle category in Table 26, we found that a very high percentage of IOU projects meet
each of the complementary guiding principles such as operating efficiency reliably and
providing health and safety benefits. A smaller fraction of projects meet the last two
guiding principles in the table, which are more narrowly focused. Sixteen percent of
projects meet the principle related to supporting low-emission vehicles, which is expected
and indicates an appropriate focus on a more narrowly technology-focused program goal.
While we cannot say whether the optimal percent of projects that support low-emission
vehicles is 16 percent, we do believe that, in order to ensure a diverse portfolio that meets
the many policy objectives, a high proportion of projects should not be focused on one
particular policy and technology area. Similarly, we cannot comment on whether the
optimal fraction of projects that support GHG mitigation is 68 percent, but we would
expect to see (and we do see) a high percentage.
While there is a clear tendency for the IOUs to focus more narrowly on the specific needs
of their own systems (that still generate benefits for all customers), the CEC has a much
broader focus (as evidenced by 70 percent of CEC projects supporting economic
development versus only 35 percent of IOU projects). Perhaps this is what one should
expect from the utilities given they are confronted every day with a set of very specific
challenges related to their grids and a general need to focus on increasing grid reliability.
Note that while the work on improving grid reliability will involve IOU employees,
reducing the number of unscheduled outages increases overall productivity. Whether IOU
TD&D portfolios should resemble the CEC TD&D portfolio with respect to the proportion
of projects that support each guiding principle is a policy issue.
Table 26: IOUs’ EPIC Funding by Guiding Principle (EPIC 1 and EPIC 2)* (N=71)
Proportion of
EPIC Guiding Principle N $ Amount Budget
Ratepayer Benefits 71 $182,678,907 100%
Broad ratepayer benefits 61 $154,799,682 85%
Narrow ratepayer benefits 10 $27,879,225 15%
Operating Efficiency Reliability 68 $176,878,907 97%
Health and Safety 38 $126,602,554 69%
GHG Mitigation 39 $123,596,716 68%
Supporting the Loading Order 33 $116,331,489 64%
Economic Development 15 $63,556,934 35%
Supporting Low-Emission Vehicles 12 $29,519,538 16%

Next, we examined how the IOUs’ EPIC 1 and EPIC 2 projects address key energy policy
areas. Table 27 shows the proportion of EPIC funds the IOUs awarded to projects in its
EPIC 1 and 2 portfolios that address each California policy goal (using a similar process as
described above for the CEC). The table shows that a high percentage of projects meet the
broad policy goals of distributed energy resources and supporting the loading order. The
IOUs direct a much higher proportion of EPIC funding toward projects that support the
smart grid compared to the CEC, and we would expect that based on the IOUs’
framework and its focus on grid research areas. The IOUs devote only a small percentage
of EPIC funding toward energy efficiency projects, though we note that they have their
Emerging Technology Program, which focuses on new technology development and
research for energy efficiency. Like the CEC, a smaller proportion of funding is allocated
to projects that will support meeting more narrowly focused policy areas of energy storage
and electric vehicles.
With respect to the Renewable Portfolio Standard policy area, the CEC and IOUs approach
this topic very differently (with the IOUs directing more than half their project funds
toward projects that support that policy versus 33 percent for the CEC). The IOUs have
several types of demonstration projects that support RPS—covering planning, controls,
vehicle-to-grid, integration of DG, and energy storage. The IOUs are conducting
demonstration projects in these areas to help them better predict how the increase in
renewables and DG will impact their grid and to test potential solutions (storage, controls)
to be able to adapt. The CEC has fewer projects in these areas, mostly Applied R&D and
Market Facilitation projects. CEC projects are focused on policy and research, with only a
small number of demonstration projects (biomass, ZNE demonstrations.) Jointly, the
projects are progressing the research on RPS, and this area may demonstrate an effective
example of how having the IOUs and the CEC administer the Program can generate a
complementary and comprehensive set of projects.
Table 27: IOUs’ EPIC Funding by California Policy Goals (EPIC 1 and EPIC 2)* (N=71)
Proportion of
California Policy Goals N $ Amount Budget
Distributed Energy Resources 64 $165,729,682 91%
Smart Grid 46 $133,692,928 73%
Loading Order 34 $117,347,491 64%
GHG Reduction 34 $106,836,716 58%
Renewable Portfolio Standard 26 $99,423,298 54%
Energy Storage Adoption 24 $88,008,243 48%
Electric Vehicles 15 $66,680,120 37%
Clean Energy Jobs 15 $56,576,934 31%
Energy Efficiency 10 $20,281,727 11%

Based on this way of looking at the IOUs’ projects, we independently verified that every
project meets the required guiding principle of providing ratepayer benefits and supports
at least one key energy policy area in a meaningful way. The IOUs’ current portfolio is
more narrowly focused on grid projects as compared to the CEC’s. We confirmed that
every IOU project supports at least one key policy area, with a particular focus on
supporting distributed energy resource and smart grid policies. Similar to the CEC
analysis, we note the challenges of assessing planning processes that attempt to reconcile
the IOUs’ investment planning framework along with several guiding principles all nested
within nine or more California policy objectives. Since there have not been any priorities
set among the guiding principles and policy areas, we cannot determine whether the
current investment planning processes lead to an optimal portfolio. However, this analysis
approach provides a starting point for further review of the portfolio and development of
priorities.
7.2 Stakeholder Outreach

Decision 12-05-037 requires the administrators to seek stakeholder input at least twice per
year and to notify parties on the EPIC service list about related proceedings, to consult
certain types of stakeholders, and to summarize stakeholder comments and their response
in their Investment Plans. Stakeholders involved in these outreach efforts include:
• Members of the Legislature (to the extent their participation is not incompatible
with their legislative positions);
• Government representatives, including state and local agencies;
• Utilities;
• Investors;
• The California Independent System Operator (ISO);
• Consumer groups;
• Environmental organizations;
• Agricultural organizations;
• Academics;
• The business community;
• The energy efficiency community;
• The clean energy industry and/or associations; and
• Other industry associations.
In a subsequent Decision (15-04-020), the CPUC required the administrators to coordinate

on an annual EPIC Innovation Symposium starting in 2015, which counts toward the two
stakeholder forums per year.

The administrators also each maintain an EPIC webpage that provides information about
EPIC and related energy policies and proceedings, including previous and upcoming
workshops and opportunities for public comments. The CEC has a specific webpage
dedicated to requesting comments to inform selected solicitation development. The CEC
also holds a number of different types of public workshops (described in more detail
below) to assess R&D needs and opportunities around specific topics, solicit input on the
CEC’s administration of EPIC, and to increase awareness across the state regarding EPIC
funding opportunities. The IOUs conduct a gaps workshop with EPRI as described earlier
to identify priority investment areas, and they also meet with EPRI on a semi-annual basis
to exchange ideas and information in order to stay abreast of emerging technologies and
innovations.
7.2.1 Stakeholder Workshops

EPIC hosts public workshops to solicit input on the Triennial Investment Plans. We
reviewed workshop materials and attended 2018-2020 Triennial Investment (or EPIC 3)
Planning workshops48 to observe the administrators’ procedures and stakeholder
engagement. Prior to the workshops, the administrators distribute a workshop notice,
which includes background on EPIC, a brief description of the upcoming workshop, and
instructions on how to participate remotely as well as how to provide public comment in
general. During the workshops, the EPIC administrators provide a more in-depth
overview of the EPIC decision and the draft Investment Plans. Each workshop has a
slightly different focus. The workshops provide an opportunity for stakeholders to share
their expertise and provide feedback on emerging technologies and future research needs.
The CEC and IOUs held a scoping workshop via webinar on February 3, 2017, during
which the administrators offered an overview of proposed topic areas and concepts, and
provided guidance on how stakeholders should submit proposed funding initiatives for
the 2018-2020 Investment Plans. The CEC posted their draft funding initiatives on March
10 and presented detailed information about the initiatives on March 14, and at that
workshop, asked questions of the audience to ensure the research is most impactful for
their organization or sector. They also asked for challenges that are not being addressed by
the draft funding initiatives. The CEC required stakeholder comments back on March 20,
but considered comments until the next CEC business meeting on April 27. The CEC held
an additional five topic-specific workshops between March 13 and April 11 on Distributed
Energy Resources, Research on Climate Change for the Electricity and Natural Gas Sector,
Incorporating Community Focused Equity in Research, Incorporating Community
Focused Equity in Research, and Customers of Climate Science Research.
48Including a CEC and IOU scoping workshop held on February 3, 2017, a CEC workshop held on March 14,
2017, and IOU workshops held on March 9 and 24, 2017. (Note that the CEC and IOUs were only required to
hold one joint public workshop).

The IOUs provided information about some of their project plans at the March 24
Investment Planning workshop, with a written comment deadline of March 31. (At prior
workshops, they presented very general information about the Program and their
planning framework.) According to stakeholders, the turnaround time for comments for
both the IOUs and the CEC is relatively short, although it does appear that the
administrators make an effort to consider and respond to them. The IOUs provided a
high-level policy overview to garner stakeholder feedback and ultimately shared a draft
plan on March 24, which incorporated such feedback. The content of what the IOUs
provided at the earlier workshop on March 9 was not sufficient for stakeholders to provide
meaningful input on the IOUs’ plans. In addition, the information prepared for the March
24 workshop was too high level to provide much value for workshop participants. Some
CEC staff even asked for more detail because they were concerned about duplicating
efforts, but the IOUs did not supply more information on their plans.
Workshop Attendees
The findings below are summarized from workshops held to inform the EPIC 2 plans.49
Workshop attendees came from a broad range of organizations (Table 28). The private
sector, industry organizations, and academic institutions represented more than three-
fourths of stakeholders who attended the workshops. The predominance of private sector
and academic workshop attendees was consistent with the types of projects awarded
through EPIC.
Table 28: Comments from Workshop Attendees by Organization Type (n=101)

Type of Organization % of Organizations
Private Sector 38%
Energy Policy and Research Organization* 27%
Academic 14%
Advocacy 8%
State Agency/Organization 8%
National Lab 3%
Independent Contractor 1%
Source: Public comments shared through 2015-2017 Triennial Investment Planning workshops
* Note: Energy Policy and Research organizations include private and public sector companies
focused on energy-related research and policy (e.g., EPRI, CALSTART, CEDMC). Private sector
includes energy-focused companies, which provide commercial products and services (e.g., Solar
City, Sunpower). They may also be focused on energy policy, but it is not their primary focus.
49At the time of this evaluation, planning for the EPIC 3 Triennial Investment Plan was underway and
related comments were not yet publicly available. As a result, our analysis focuses on comments from the
EPIC 2 Triennial Investment Plan, which includes a CEC-led scoping workshop held on February 7, 2014, a
joint IOU workshop held on February 21, 2014, coordinated workshops held on March 17 and 21, 2014, and
other public comments through other means.

Workshop Comments
Stakeholders’ comments addressed a wide array of topics presented during the workshops
(Table 29 and Table 30). Their substantive comments reflected strong engagement among
stakeholders and administrators. The CEC and IOUs documented all of the comments and
provided a response to the remarks in each of the Triennial Investment Plans. During the
CEC-led workshop, attendees’ comments regarding Applied R&D most frequently
referred to issues related to DSM and energy storage. Comments regarding TD&D were
most commonly focused on bioenergy and generation. Of the 23 comments regarding
Market Facilitation, nearly half (10) were regarding commercialization assistance. These
topics were also prevalent in the other workshops held by the CEC and IOUs.
The EPIC administrators documented and responded to stakeholders’ comments in their

Investment Plans as required by the CPUC. However, there does not appear to be many
instances where comments resulted in a significant change in the Investment Plans. For
example, one CEC workshop participant suggested that staff include “gateways” among
the examples of DSM technologies. The administrators made a non-substantive change to
the list of examples by adding “home automation network devices and systems,” noting
that “gateways” fell within the broader categories of technologies, but did not consider
including the more narrow specification that the participant suggested. In most cases, the
administrators acknowledged and reiterated relevance to existing funding opportunities
or priority investment areas; on occasion, the administrators noted and explained why
individual comments were not within the scope or focus of the Program.

Table 29: Scoping Workshop and Questionnaire Comments (CEC only)
Applied R&D (54 comments)
Demand-side management 12
Energy storage 9
Building envelope materials 7
Modeling/Tools 7
Environment and public health 4
Building controls 3
Grid operations/Market design 3
Indoor air quality 2
Fuel cells 2
ZNE 1
Plug-load 1
Water heating 1
Ground Source Heat Pumps 1
Goods movement 1
TD&D (30 comments)
Bioenergy 12
Other generation 7
Vehicle-grid integration 5
Offshore generation technologies 4
Microgrids 2
Clean energy and transportation 2
Ventilation and air conditioner
1
precooling
Buoyant atmospheric PV systems 1
Market Facilitation (23 comments)
Commercialization assistance 10
Data and analytics 5
Open-source standards 3
Programmatic Environmental Impact
2
Report

Regulatory and Permitting 2
Workforce Development 1
General Comments / Other Topics (6 comments)
Data and analytics 3
Nuclear energy 2
New solar homes partnership 1
Source: CEC-led scoping workshop held on February 7, 2014
Table 30: Other Workshops Comments (CEC and IOUs)

Topic CEC IOUs
Generation 14 4
Demand-Side Management 10 6
Market Facilitation/
8 2
Commercialization Assistance
Smart Grid Enabling Clean Energy 6 1
Siting 3 0
General Comments / Other Topics 3 10
Source: Public comments shared through 2015-2017 Triennial Investment Planning workshops
7.2.2 Idea Exchange

The EPIC Idea Exchange Docket is an electronic forum for stakeholders to provide public
comments on the CEC’s RD&D and Market Facilitation initiatives and on the Program in
general. The Idea Exchange provides an ongoing opportunity for stakeholders to submit
ideas to the CEC. These can include ideas for technology areas, administrative practices,
and outreach strategies. Stakeholders submit comments to the Idea Exchange in response
to various public workshops held by the CEC. The IOUs do not have a comparable process
for soliciting stakeholder feedback. We note that the CPUC does not require the
administrators to facilitate the Idea Exchange. Although the IOUs do not have a formal
electronic forum, they do solicit input throughout the year via their individual websites
and by a centralized email for the respective IOUs.
Since it was launched on July 8, 2016, over 200 comments have been submitted via the Idea
Exchange. Idea Exchange topics are based on the guiding questions posted by the CEC.
For example, following the September 22, 2016 public workshop focused on increasing
private sector participation in EPIC, workshop participants were invited to engage further
by submitting written remarks through the Idea Exchange. The Evergreen team reviewed

comments submitted in response to this workshop to illustrate this aspect of stakeholder
outreach. Stakeholders responded to four guiding questions related to increasing private
sector participation:
1. What are some concerns and challenges the private sector, including small
businesses and entrepreneurs, face when considering applying for grant funding
opportunities?
2. How can the California Energy Commission better increase awareness of the
research programs to California private sector companies?
3. What are some ideas to encourage private sector companies to apply for research
funding?
4. Besides grant funding, what else can the California Energy Commission do to help
California private sector companies to be successful?
Twelve individuals submitted written comments on this topic via the EPIC Idea Exchange.
Respondents represented a range of perspectives including six representatives from the
private sector, three from industry organizations, one from an academic institution, and
one from a National Lab. Representation from these groups is fairly consistent with the
types of grants awarded through EPIC.
According to the Idea Exchange respondents, the EPIC proposal requirements and the
cost-share obligation were the most common challenges for small private sector firms
interested in applying for grant funding. For both topics, five of the twelve respondents,
including two of the six private sector firms, referenced concerns with these aspects of the
EPIC grant requirements.
• Five of the 12 respondents stated that the EPIC proposal requirements are
challenging. Specific concerns related to breadth of the application, including detail
requested in the narrative as well as the number of attachments. A few respondents
also commented that the time and resources required of an EPIC proposal were
challenging for small businesses, start-up companies, entrepreneurs and other
organizations that do not have a lot of experience with producing the level of detail
needed for a successful application. This result is also corroborated by interviews
with grantees.
• Five of the 12 respondents also indicated that the cost-share requirement is a
burden, particularly for small businesses and entrepreneurs. A couple of
respondents thought that the substantial amount of the cost-share50 is difficult to
50Match funding is only required for TD&D projects. The applicant must match at least 20 percent of the
requested funds. Applied R&D and Market Facilitation projects typically do not require match funding, but

secure, and indicated that the Program should offer opportunities for smaller grants
and match-funding obligations.51
The second most commonly cited challenge for private sector companies was related to
Intellectual Property (IP) terms and conditions.
• Three respondents stated that the IP terms and conditions are a significant barrier to
applying to or participating in EPIC. In general, they thought that the terms and
conditions were “onerous” and rather limiting for potential grantees. Two of these
individuals stated that the IP provisions very likely inhibit or prevent small
business and entrepreneurs from even applying for grant funding. While these
respondents mentioned this barrier, only one respondent offered a solution, which
is to consider adopting IP language developed for the federal Small Business
Innovation Research (SBIR) Program.52 Whether the CPUC decides to consider the
IP template from SBIR or not, it appears that the Program should review existing
terms and conditions, since potential grantees perceive them to be a significant
barrier.
Other comments mentioned challenges related to competition between large research

firms and small businesses and variable or inconsistent solicitation timelines.
Table 31 below shows the full range of responses from the Idea Exchange regarding
challenges to applying for grant funding.
applicants can receive extra points on their proposal for such funding. Up to 10 percent of EPIC funding for
Applied R&D and TD&D can be used as federal cost share.
51 The Program funds a project, California Sustainable Energy Entrepreneur Development (CalSEED)
Initiative, which provides up to $600,000 in grant funding for early-stage research and development for
emerging technologies.
52 The DOE's Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR)
Programs have developed a “Model Agreement for Property and Commercialization Rights,” which is a
template designed to help in the develop of an agreement for allocating between small business concerns
and research institutions' intellectual property rights and rights, if any, to carry out follow-on research,
development, or commercialization.
https://science.energy.gov/~/media/sbir/pdf/files/manageapp/doe_model_agreement_for_property_an
d_commercialization_rights.pdf

Table 31: Challenges to Applying for Grant Funding
Response Count
Proposal requirements 5
Cost-share obligation 5
IP terms and conditions 3
Competition between research institutions and
2
small businesses
Variable/Inconsistent solicitation timelines 1
Source: EPIC Idea Exchange

Six of the 12 respondents thought that the CEC could increase awareness of research
programs through targeted outreach and dissemination efforts.
• Examples of targeted efforts mentioned by respondents included conducting

outreach to trade shows and conferences, leveraging relationships with industry
associations to promote upcoming solicitations, and potentially having industry
associations co-sponsor some solicitations to encourage members to apply for
funding.
• In addition to working with established partners, one respondent suggested that
the CEC should generally target the private sector prior to the release of a
solicitation, and another stated that more advance notice of upcoming solicitations
would be helpful.
• Two respondents thought that dissemination efforts that focused on successes,
lessons learned, and project accomplishments would help to promote research
funding opportunities.
Table 32 summarizes the outreach and dissemination strategies that the Idea Exchange
respondents offered.

Table 32: Suggested Strategies for Outreach and Dissemination
Outreach Dissemination
• Leverage industry associations to promote • Publish successes and lessons learned

funding opportunities • Require projects to share findings as part
• Have industry associations co-sponsor of the grant
solicitations
• Conduct outreach to trade shows,
conferences
• Engage private sector prior to release of
solicitation
• Provide more notice of upcoming
solicitations
The most common suggestions regarding how to encourage private sector applications
include comments related to funding priorities, the cost-share obligation, and the
application process.
• Seven of the 12 respondents remarked on specific areas that the CEC should fund.
In some cases, these areas are already funded (for example, funding to support
testing and validation, proof of concept projects, and demonstration sites). There
also was a contrasting perspective where one respondent thought that the CEC
should fund projects with a wide range of Technology Readiness Levels (TRLs),
while another thought that there should be greater opportunity for disruptive
technologies. Overall, there was not a consistent theme among respondents’
remarks. Most comments referred to existing priority areas, indicating that some
stakeholders may not be very aware of the range of technologies and opportunities
funded through EPIC.
• Five of the 12 respondents provided suggestions regarding cost-share requirements.
One respondent thought that this obligation should be eliminated entirely. Another
respondent reported that the cost-share should be optional, and that small
businesses that secure match funding should receive weighted or bonus scores
toward their applications since it is harder for them to procure such funding. Two
respondents suggested the CEC should facilitate matches with other funding
opportunities. One respondent proposed that the CEC provide an option for the
cost-share to be offered as a loan.
• Five of the 12 respondents mentioned that the application process should be
simplified. Only a few respondents provided specific recommendations, including
having more of the proposal narrative be focused on the technology being

proposed, providing more clarity on the intent of specific solicitations, and
reviewing details of solicitations when they are released.
• Other less common suggestions included offering an open solicitation (4), allowing
for a two-stage proposal (3), and helping to facilitate teaming partnerships (2).
Table 33 shows the full range of topics that respondents shared.
Table 33: Suggestions Regarding How to Encourage Private Sector Applications

Response Count
Change/expand funding priorities 7
Change/modify cost-share obligation 5
Simplify application process 5
Offer an open solicitation 4
Offer a two-stage proposal 3
Facilitate partnerships 2
Encourage universities to serve as primes 1
Help businesses find grant writing firms (similar to
1
partnerships)
Help facilitate disseminating project findings to industry 1
Host specific subject-matter workshops; engage various
industry stakeholders in Investment Plans (consortia, 1
incubators, accelerators, investors)
Remove common metrics from scoring and rely on CEC to
1
track and monitor

When asked what else the CEC could do to help private sector companies be successful,
respondents generally referred to facilitating partnerships with other companies,
including supporting demonstration sites, as well as with the investment community.
• Suggestions regarding partnerships included providing prescriptive partnering

guidelines, offering proposal scoring incentives for targeted private sector entities,
and maintaining a database of lab or research facilities that companies could team
with on a proposal.
• Respondents offered a few general comments related to facilitating relationships
with private funders. One respondent suggested that the CEC should promote
disseminating project findings to the investment community. Another
recommended creating a venture capital fund to help support commercialization

activities. Respondents did not provide a lot of detail about how these suggestions
should be implemented.
7.2.3 Stakeholder Interviews

As described previously, the team conducted telephone interviews with nine program
stakeholders to understand how EPIC administrators engage experts in developing the
Triennial Investment Plans and to help administrators guide their projects. The
stakeholders represented a range of organizations and perspectives, including state
government; national labs; and industry, research and academic organizations.
In general, stakeholders thought that the EPIC administrators do a fairly good job of
providing opportunities for participation in the planning process. Interviewees reported
that the public nature of the workshops, meetings and conference calls allows for a high
degree of transparency. In addition, stakeholders thought that the process was very
inclusive since the administrators seek to engage a wide range of stakeholders. Although
interviewees generally thought that these efforts were adequate, a minority of respondents
raised a couple of concerns.
One stakeholder thought that while there was value in having transparency by using a
public forum to solicit input, stakeholders engaged in this process may be worried about
being politically correct in sharing concerns and might not share as freely as in a closed
meeting setting, potentially resulting in missing needed EPIC funding opportunities. This
individual had experience with the Public Interest Energy Research program (PIER)53 and
believed that the advisory committee process, which held closed meetings rather than
forums open to the public, helped foster a level of comfort among all participants where
they shared information more freely.
A few stakeholders that we interviewed offered suggestions regarding the process of

developing the Triennial Investment Plans. A couple of them thought that there needs to
be more lead time for providing comments on the Triennial Investment Plans. One
stakeholder mentioned that the California ISO should have more direct involvement in the
investment planning process. Comparing the EPIC and PIER management structures, one
stakeholder noted that PIER used to have a standing committee that effectively served as a
board of directors that, along with a supporting organization, was effective in identifying
research needs. He suggested that the lack of such a board of directors is a weakness in the
way EPIC is structured. The repeated reference to PIER’s management structure may
indicate an issue for EPIC to investigate further. Remnants from PIER such as the advisory
committee process contributed to the program’s success and may be worth examining
further for EPIC.
53PIER was the predecessor to EPIC. The CEC is still administering funding for public interest natural gas
research and development projects.

The best practices assessment examined effective practices related to investment planning,
including how peer RD&D programs conduct market assessments, identify gaps and
opportunities, and determine program or portfolio-level investment priorities. This section
summarizes the key takeaways from our review of the literature and interviews with peer
RD&D programs.
7.3.1 Policy Alignment

Interviewees from all seven peer RD&D programs indicated that their initiatives are
explicitly aligned with federal or state local policy goals. For example, as the primary
author for the New York State Energy Plan, The New York State Energy Research and
Development Authority (NYSERDA) played a key role in defining energy policy goals and
objectives. As a result, the Technology and Market Development (T&MD) Program is
closely coordinated with the Energy Plan recommendations and helps contribute to the
state’s energy goals. Since the SBIR and STTR Programs are administered across multiple
federal agencies, the program administrators do not establish the technology areas that
they fund. Instead, the program administrators work closely with the relevant R&D
program offices at the Department of Energy (DOE), which typically develop their
research priority areas, often driven by roadmaps or other investment planning
mechanisms, that are aligned with their mission. The SBIR and STTR Programs help
facilitate RD&D in these priority areas by stimulating technology innovation among small
businesses.
While the DOE’s Advanced Research Projects Agency-Energy Program (ARPA-E) is

mission driven, it is the only peer RD&D program that intentionally is not bound by policy
roadmaps and other similar directives. As a result, the program takes into consideration
such documents, but specifically seeks to fund the “disruptive technologies and solutions”
that may not necessarily be identified in these investment planning mechanisms. As the
Strategic Vision for ARPA-E states, the program is designed to advance “high-potential,
high-impact energy technologies that are too early for private sector or other DOE applied
research and development investment.”54
7.3.2 Investment Planning

All seven of the peer RD&D programs are informed in some way by external stakeholders;
interviewees from the four primary peer RD&D programs noted that they have formal
mechanisms to help identify and develop funding opportunities. As noted above, the peer
RD&D programs are directly influenced by policy directives, which involves engagement
with a variety of external stakeholders including federal and state government agencies,
54Advanced Research Projects Agency-Energy Program (ARPA-E). "ARPA-E Strategic Vision 2013.” October
2013. https://arpa-e.energy.gov/sites/default/files/ARPA-E_Strategic_Vision_Report_101713.pdf

legislators and legislative committees, businesses and other interest groups. The primary
peer RD&D programs also formally engage external stakeholders in identifying
investment areas. Staff from NYSERDA’s T&MD Program worked with an external
consulting firm to determine the program’s investment priority areas. The DOE convenes
planning workshops and other meetings that help inform funding priorities for individual
research offices. These investment areas support research funding, and a portion is
allocated to small business initiatives like the SBIR and STTR Programs. ARPA-E uses a
different approach. ARPA-E rotating Program Directors have a considerable degree of
autonomy in defining funding opportunities. ARPA-E’s use of rotating Program Directors
helps the program bring in fresh ideas based on their prior experience and expertise. Also,
because they have a three-year tenure with ARPA-E, they typically are working within a
relatively short timeframe and are attempting to make a significant impact during their
time with the program. These staff members conceptualize focused technical funding
opportunities, which are informed by meetings with external stakeholders. The Program
Director takes the lead on soliciting input from external stakeholders as well as other
Program Directors and leadership staff within the program, and ultimately has a great
degree of ownership in developing funding opportunities and selecting projects.
Six of the seven peer RD&D programs offer focused and clearly-defined funding
opportunities; two of the four primary peer RD&D programs—ARPA-E and NYSERDA’s
T&MD Program—additionally offer an open funding opportunity to encourage ideas that
are not covered by focused program areas. In general, the focused funding opportunities
are developed by program staff and are designed to address the mission and goals of the
program. These opportunities offer funding to specific technology areas such as battery
storage, building efficiency and grid modernization. To foster innovative approaches and
solutions, both ARPA-E and NYSERDA’s T&MD Program also offer funding beyond the
traditional focused opportunities. ARPA-E offers a funding opportunity called Innovative
Development in Energy-Related Applied Science (IDEAS), which accepts, on a rolling
basis, proposals for single-phase projects of up to 12 months and $500,000 of funding.55
NYSERDA similarly offers funding through The Advanced Clean Energy Exploratory
Research Program (ACE Program); proposals are accepted on a rolling basis, and project
funding and timelines vary.56

EPIC, like all of the peer RD&D programs included in the best practices assessment, is
explicitly aligned with state policy energy goals. In addition, EPIC directly engages
internal and external stakeholders to help inform funding and research priorities.
55 More detail on the IDEAS funding opportunity is available at ARPA-E's Funding Opportunity Exchange,
at https://arpa-e-foa.energy.gov/#FoaIda8bdd9ec-2cb7-4349-8184-4dde00c77663
56 More information on ACE Program funding is available at NYSERDA's funding opportunity portal, at
https://portal.nyserda.ny.gov/CORE_Solicitation_Detail_Page?SolicitationId=a0rt0000000QnqdAAC

However, unlike the peer RD&D programs that tend to have a tight focus and clear policy
directives, EPIC has a broader set of policy goals resulting in a more diffuse focus and
increased complexity in managing the total portfolio and ensuring that it is optimized to
achieve those goals.
Six of the seven peer RD&D programs offer focused funding opportunities, just as EPIC
does. However, two peer RD&D programs also offer open solicitations to encourage
cutting edge research, technology and approaches. To date, we believe that EPIC has only
offered two open solicitations, by the CEC only.
Current EPIC
Peer RD&D Programs Practice Comments
Policy Alignment
Help define state energy policy goals EPIC does not do this. This is currently beyond the purview
of EPIC.
Prioritize research areas, driven by EPIC has a process for EPIC process appears to be
roadmaps prioritizing research sufficient.
areas.
Focus on white spaces not addressed by EPIC does not do this. EPIC should consider doing this.
other programs
Investment Planning
Planning workshops and other meetings that EPIC mostly does this. The CEC does this more fully than
help inform funding IOUs.
Rotating Program Directors helps the EPIC does not do this. Likely not applicable or relevant to
program bring in fresh ideas EPIC
External consulting firm helps determine EPIC does not do this. EPRI serves this function for the
investment priority areas IOUs.
The CEC engages external experts
but manages the process internally.
Offer focused and clearly-defined funding EPIC does this (CEC). The CEC does this more fully than
opportunities to address program goals the IOUs.
Offer single-phase, open funding opportunity EPIC (CEC) has done EPIC should continue, and perhaps
to encourage ideas that are not covered by this. consider expanding, this practice.
focused program areas

8 Project Selection Process
In this section, we discuss the effectiveness of the Program’s project selection processes, as
which informed our data collection efforts. Project selection is a critical second step in the
interviews with EPIC stakeholders and conducted a best practices assessment.
8.1 Project Selection Process

The CEC and IOUs take a different approach to selecting projects for their EPIC portfolio.
As mentioned previously, throughout the report we have tended to present results for the
CEC and all three IOUs combined due to the difference in their administrative approach
that we noted at the initial stages of the evaluation. However, we examined each of the
four administrators’ processes and projects, and we identify any differences in the report.
Note that those comparisons were limited by the size of the project sample for each IOU,
which reflects the relatively smaller portion of the EPIC budget that the IOUs receive as
compared to the CEC.
8.1.1 CEC
Based on interviews we conducted with the CEC’s EPIC program management team and
review of their EPIC reports, we learned that the CEC outsources EPIC projects by issuing
solicitations to fund the initiatives outlined in its Investment Plans. For example, the CEC
issued 17 solicitations in 2015 and 11 solicitations in 2016, from both the EPIC 1 and EPIC 2
Investment Plans. Each solicitation is directly linked to one or more strategic objectives
and funding initiatives outlined in the Investment Plans. The solicitations also group
strategic objectives and funding initiatives by program area and Investment Plan period.
The CEC follows an established process that includes the following steps for each
solicitation:
1. Posting the solicitation (either a Grant Funding Opportunity [GFO] or Program

Opportunity Notice [PON]) on the CEC website;
2. Notifying interested parties through the Opportunity listserv (serving broader
stakeholders engaged with the CEC, not just on EPIC), the EPIC listserv (dedicated
to EPIC, issuing EPIC funding opportunity and award notices) and the LinkedIn
Networking Hub;
3. Holding at least one public workshop;

4. Providing an opportunity to submit questions about the solicitation, posting
responses on the CEC’s EPIC webpage and sending out to the Opportunity listserv;
5. Posting a notice of proposed awards (or NOPA) on the CEC website; and
6. Providing an opportunity for a debrief with losing bidders.
The CEC also ensures that any stakeholders who have provided input on a solicitation
(outside of a public forum) are not allowed to bid, including the utilities, state agencies
and universities.
For each solicitation, the CEC publishes criteria by which it will score each bid in a
solicitation manual, which contains eligibility requirements and submission instructions,
and outlines the evaluation and award process. There is a two stage screening process,
whereby applications (bids) that fail to comply with the screening criteria are rejected.
Those that pass the first stage are submitted to an evaluation committee for review and
scoring based on the criteria listed in the manual. The scores for each application are the
average of the combined scores of all committee members. A minimum score of 70 points
(out of a maximum of 100) is required to be eligible for funding, along with a minimum of
49 points for the first four criteria (out of eight total). The criteria categories are:
1. Technical merit and need (20 points)

2. Technical approach (20 points)
3. Impacts and benefits for California ratepayers (20 points)
4. Team qualifications, capabilities and resources (10 points)
5. Budget and cost-effectiveness (10 points)
6. EPIC funds spent in California (15 points)
7. Ratio of direct labor and fringe benefit rates to loaded labor rates (5 points)
8. Match funding (optional 5 points) - required only for Technology Demonstration
and Deployment applications. Match funding is not required for Applied R&D or
Market Facilitation projects, but applicants can receive extra points on their project
for match funding.
Applications are then ranked by score, and the top projects are awarded funds. The results
are posted in the NOPA.
Table 35 summarizes each CEC solicitation in EPIC 1 and EPIC 2, and includes the number
of applications received and awarded for each solicitation, as well as funding by research
area. Note that the CEC awarded projects to all strategic objectives identified in its first
two Investment Plans, with the exception of two, which have not yet been included in a
solicitation:

• S15: Demonstrate Advanced Energy Storage Interconnection Systems to Lower
Costs, Facilitate Market and Improve Grid Reliability (in the TD&D area) – which
was included in a solicitation released in November 2016 with $26 million in
potential funding (which also included strategic objectives 3, 4 and 6).
• S19: Facilitate Inclusion of Emerging Clean Energy Technologies into Large-Scale
Procurement Processes (in the Market Facilitation area) – not yet included in a
solicitation to date.57
57The CEC released a Request for Comments to get further stakeholder input on the scope of the upcoming
solicitation. The Request for Comments was released on February 27, 2017, with comments due on March 13,
2017. The solicitation will be released in the summer of 2017.

Table 35: EPIC 1 and 2 CEC Solicitations and Awards
Funding for Funding for Funding for
Applied R&D TD&D Market
EPIC Strategic. Number of Number Projects Projects Facilitation Total Funding
Solicitation Phase Objective Applications Awarded Awarded Awarded Projects Awarded Awarded
GFO-15-302 1 S17 4 2 - - $8,908,107 $8,908,107

GFO-15-303 1 S5 13 4 $1,899,929 - - $1,899,929
GFO-15-306 1&2 S10; S18 12 6 $8,000,000 - $6,980,000 $14,980,000
S1; S12;
GFO-15-308 1&2 38 15 $16,042,170 $14,748,290 - $30,790,460
S14
GFO-15-309 1 S1; S5 24 20 $9,539,187 - - $9,539,187
GFO-15-310 1 S1 19 9 $10,270,943 $4,819,805 - $15,090,748
GFO-15-311 1&2 S2 12 9 $24,137,717 $2,007,875 - $26,145,592
GFO-15-312 1&2 S12; S16 28 13 - - $18,924,963 $18,924,963
GFO-15-313 1 S6; S7 29 7 $7,091,923 - - $7,091,923
S1; S12;
GFO-15-317 1 34 14 $5,867,521 $11,367,282 $1,000,000 $18,234,803
S20
GFO-15-321 1 S18 2 1 - - $4,999,247 $4,999,247
GFO-15-323 1 S12 8 2 - $3,565,362 - $3,565,362
GFO-16-301 1 S3; S4 7 1 $873,387 - - $873,387
PON-13-301 1 S1 48 13 $33,205,581 - - $33,205,581
PON-13-302 1 S8 38 5 $8,673,198 - - $8,673,198
PON-13-303 1 S4 18 11 $15,814,772 - - $15,814,772
PON-14-301 1 S14 41 9 - $29,641,646 - $29,641,646
PON-14-303 1 S3 27 8 $12,689,706 - - $12,689,706
PON-14-304 1 S12; S13 13 13 - $41,928,723 - $41,928,723

Funding for Funding for Funding for
Applied R&D TD&D Market
EPIC Strategic. Number of Number Projects Projects Facilitation Total Funding
Solicitation Phase Objective Applications Awarded Awarded Awarded Projects Awarded Awarded
PON-14-305 1 S13 23 4 - $13,799,386 - $13,799,386

PON-14-306 1 S18 12 5 - - $2,138,713 $2,138,713
PON-14-307 1 S13 22 6 $399,818 $12,430,226 - $12,830,044
PON-14-308 2 S11, S17 3 1 $750,000 - - $750,000
PON-14-309 1 S5 14 8 $6,143,247 - - $6,143,247
PON-14-310 1 S9 25 5 $6,681,669 - - $6,681,669
Other* 1&2 Multiple 11 11 $34,147,612 - $16,075,358 $50,222,970
Total 525 202 $202,228,380 $134,308,595 $59,026,388 $395,563,363
* 'Other' includes three non-competitive grants and eight grants issued through Requests for Proposals (RFPs).

Figure 10 on the next page shows the timeline for each CEC solicitation. Red sections of
each project timeline represent the application period. The blue sections represent the
scoring and selection period. Lastly, the green sections represent the projected project
implementation period, which extends through the first quarter of 2022. The first
solicitation round was released in the second quarter of 2014, and projects began in the
second quarter of 2015. The last solicitation round was released in the second quarter of
2016, and projects began in the first quarter of 2017.

Figure 10: EPIC 1 and 2 CEC Solicitation Timeline
Investment EPIC Number of Number Total
ID Research Area 2014 2014 2014 2014 2015 2015 2015 2015 2016 2016 2016 2016 2017 2017 2017 2017 2018 2018 2018 2018 2019 2019 2019 2019 2020 2020 2020 2020 2021 2021 2021 2021 2022
Area Round Applications Awarded Funding Q1 Q2 Q2 Q3 Q1 Q2 Q2 Q3 Q1 Q2 Q2 Q3 Q1 Q2 Q2 Q3 Q1 Q2 Q2 Q3 Q1 Q2 Q2 Q3 Q1 Q2 Q2 Q3 Q1 Q2 Q2 Q3 Q1
$33,205,581
PON-13-301 R&D EE Technology 1 48 13 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
Utility Coordination
$8,673,198
PON-13-302 R&D Systems 1 38 5 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
Utility Scale
$15,814,772
PON-13-303 R&D Renewables 1 18 11 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
$29,641,646
PON-14-301 TD&D EE Demonstration 1 41 9 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
$12,689,706
PON-14-303 R&D Innovative Solutions 1 27 8 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
EE and
$41,928,723
PON-14-305 TD&D GenerationTechnology 1 13 13 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
Renewable Generation
$13,799,386
PON-14-304 TD&D Technology 1 23 4 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
Social Cultural
$2,138,713
PON-14-306 MF Behavioural Aspects of 1 12 5 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
Renewable Generation
$12,830,044
PON-14-307 TD&D Technology 1 22 6 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
Applied Research with
$750,000
PON-14-308 R&D/TD&D Federal Cost Share 2 3 1 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
$6,143,247
PON-14-310 R&D EV Infrastructure 1 14 8 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
$6,681,669
PON-14-309 R&D IDSM and ZNE 1 25 5 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
Vulnerability and
$8,908,107
GFO-15-303 R&D Adaptation Studies 1 4 2 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
EE Workforce
$1,899,929
GFO-15-302 MF Development 1 13 4 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
$14,980,000
GFO-15-306 MF/R&D Innovation Cllusters 1 & 2 12 6 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
$30,790,460
GFO-15-308 R&D/TD&D IDSM and ZNE 1 & 2 38 15 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
$9,539,187
GFO-15-310 R&D EE Technology 1 24 20 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
$15,090,748
GFO-15-309 R&D EE Technology 1 19 9 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
$26,145,592
GFO-15-317 R&D/TD&D EE Technology 1 & 2 12 9 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
Advanced Energy
$18,924,963
GFO-15-312 MF Communities 1 & 2 28 13 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
Smart Grid
$7,091,923
GFO-15-313 R&D Development 1 29 7 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
Demand Response
$18,234,803
GFO-15-311 R&D/TD&D Technology 1 34 14 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
$4,999,247
GFO-15-321 MF Innovation Cllusters 1 2 1 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
Water and EE
$3,565,362
GFO-15-323 TD&D Demonstrations 1 8 2 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
Utility Scale
Renewables $873,387 ### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ####
GFO-16-301 R&D 1 7 1
$394,040,772
Total 510 190
Application Period
Selection Period
Project Period

To evaluate the CEC’s selection process, we drew a sample of five solicitations to
determine how well the awarded bids align with the strategic objectives covered by the
solicitations. We also reviewed all the bids submitted for the sampled solicitations (93
bids) to ensure that bids were scored based on the published criteria and that the results
posted publicly (via the NOPA) were accurate. Our sample represents 18 percent of the 28
solicitations, 29 percent of total solicitation funding, and 26 percent of the 201 awarded
bids, as shown in Table 36 on the next page.

Table 36: CEC Solicitation Sample
Funding for
Funding for Market
Applied R&D Funding for Facilitation
Investment EPIC Number Number Projects TD&D Projects Projects Total Funding
Solicitation Area Round of Bids Awarded Awarded Awarded Awarded Awarded
GFO-15-311 R&D/TD&D 1&2 12 9 $24,137,717 $2,007,875 - $26,145,592
GFO-15-312 MF 1&2 28 13 - - $18,924,963 $18,924,963
PON-13-303 R&D 1 18 11 $15,814,772 - - $15,814,772
PON-14-304 TD&D 1 13 13 - $41,928,723 - $41,928,723
PON-14-307 TD&D 1 22 6 $399,818 $12,430,226 - $12,830,044
Total 93 52 $410,352,307 $56,366,824 $18,924,963 $115,644,094

To determine if funding is awarded to projects consistent with EPIC policy objectives and
planning processes, and project selection is guided by the Investment Plans, we reviewed
the planning documentation, including the application manuals and attachments, for each
of the five solicitations. Based on this review, we found each solicitation package linked
directly to a strategic objective in either EPIC 1 or EPIC 2 Investment Plans. We also found
each solicitation package to provide a clear and thorough outline of the requirements to
prospective applicants. Table 37 presents the sampled solicitations, applicable Investment
Plans and the strategic objectives each solicitation is designed to address.
Table 37: Solicitation Alignment with Strategic Objectives in Investment Plans

EPIC
Solicitation Solicitation Name Round Strategic Objective
GFO-15-311 Advancing Solutions that Allow 1&2 S2: Develop New Technologies and Applications that
Customers to Manage Their Enable Cost-Beneficial Customer-Side-of-the-Meter
Energy Demand Energy Choices
GFO-15-312 The EPIC Challenge: 1&2 S16: Collaborate with local jurisdictions and
Accelerating the Deployment of stakeholder groups in IOU territories to establish
Advanced Energy Communities strategies for enhancing current regulatory assistance
and permit streamlining efforts that facilitate
coordinated investments and widespread deployment
of clean energy infrastructure
S12: Overcome Barriers to Emerging Energy
Efficiency and Demand-Side Management Solutions
through Demonstrations in New and Existing
Buildings
PON-13-303 Advancing Utility-Scale Clean 1 Strategic Objective S4: Develop Emerging Utility-
Energy Generation Scale Renewable Energy Generation Technologies
and Strategies to Improve Power Plant Performance,
Reduce Costs, and Expand the Resource Base
PON-14-304 Bringing Energy Efficiency 1 S12: Demonstrate and Evaluate the Technical and
Solutions to California's Economic Performance of Emerging Efficiency and
Industrial, Agriculture and Demand-Side Management Technologies and
Water Sectors Strategies in Major End-Use Sectors
S13: Demonstrate and Evaluate Emerging Clean
Energy Generation Technologies and Deployment
Strategies
PON-14-307 Demonstrating Clean Energy 1 S13: Demonstrate and Evaluate Emerging Clean
Solutions That Support Energy Generation Technologies and Deployment
California's Industries, the Strategies
Environment, and the Electrical
Grid
After confirming that each solicitation was guided by an Investment Plan, we reviewed
project-level documentation for each approved project to confirm that the project
description and planned outcomes aligned with the strategic objectives of the solicitation.

Figure 11 summarizes this review. We found that of 52 approved projects, 47 align directly
with the strategic plan objective(s) in the solicitation documentation. The remaining five
projects align with other strategic objectives not specifically targeted by that particular
solicitation. Based on this review, we found that projects were largely guided by an
Investment Plan, and selected based in part on their potential to fulfill a strategic objective
outlined in an Investment Plan.
Figure 11: Project Alignment with Strategic Objectives in Investment Plans
GFO-15-311 8 1
GFO-15-312 12 1
PON-13-303 10 1
PON-14-304 13
PON-14-307 4 2
0% 100%
Aligned With Strategic Objective Not Aligned With Strategic Objective
The evaluation team reviewed all documentation from the CEC for the sample, including
the actual bids submitted, solicitation requirements and criteria, and scores, to confirm
that the administrator selected projects according to the process outlined above, with
sufficient consideration and documentation of each project’s goals and objectives. The first
task the evaluation team engaged in was an audit of project sample application
documentation. This audit found that for four of the five solicitations sampled (GFO-15-
311, GFO-15-312, PON-13-303 and PON-14-307), applications were very complete, with
only one of 39 project applications not having a complete application package. The one
anomaly was missing one item. PON-14-304 had less complete application packages, with
only one in thirteen having all documentation. However, all approved projects had the
required documentation, the application and a project summary.
The second task was an assessment of the review and scoring process for each application.
Committee members scored each project based on eight criteria. The scoring and selection
process occurred in two phases. In phase one, the committee selected only projects that
had an average score of over 49 on the first four selection criteria. In phase two, the
committee selected projects with an average total score of above 70. Funding is allocated to
these projects in order from highest to lowest score until funding is exhausted. The

following box-and-whisker plots present the median and range of scores for approved
projects across the five sampled solicitations. The first plot shows the median and range of
scores for the first four criteria across the five solicitation rounds reviewed. The second
plot shows the median and range of scores across all eight criteria across the five
solicitation rounds. The boxplots show the range of scores with the median score falling
between the green and blue portions of each bar, and the second and third quartiles as the
lower edge of each of the green boxes and the upper edges of the blue boxes, respectively.
The end points of the whiskers represent the minimum and maximum scores. The
evaluation team found that all approved projects scored greater than 70 overall and
greater than 49 points for the first four criteria.
Figure 12: Box and Whisker Plots – Distribution of Scores for Sample Solicitations

Lastly, the evaluation team reviewed and compared all scores across all committee
members to identify any anomalous scoring, indication of errors, favoritism or other
issues. Based on this review, we found that:
• All project scores were calculated correctly.

• Scores deviated across reviewers for each project which we would expect, but there
is no indication that one scorer had extreme scoring that would skew a project
toward passing or failing.
• Scorers on average tended to rank projects in a similar order, although as we would
expect, there was not always agreement between scorers across projects in terms of
ranking.
• There was no indication of anomalous scoring or mathematical errors in calculating
scores.
In conclusion, the evaluation team's review of CEC project selection confirms that the
administrator aligned solicitations with the Triennial Investment Plans and adopted
projects that addressed the specific strategic objectives and funding initiatives outlined in
each solicitation. Projects selection was conducted in a clear and transparent fashion,
adhering to the project selection criteria outlined by the CEC.
8.1.2 IOUs
The process the IOUs follow to select EPIC projects is significantly different than the one
used by the CEC. Based on a series of interviews we conducted with the IOUs’ EPIC
program management teams and review of their EPIC reports, we learned that all three
IOUs identify their project ideas during their internal investment planning processes, and
their Investment Plans provide a general description (with no budgets identified) of the
projects they may implement. However, SDG&E provides an estimated funding allocation
for all projects.
Once their plans are approved, the IOUs have internal processes to scope the project,
allocate budget and develop a project management plan that they use to guide the project.
They coordinate internally with technical experts, and hold periodic status meetings with
their management. They may defer on launching a project that is described in their
Investment Plan and place it on hold (sometimes indefinitely) until they decide it is a
priority at that point in time, based on consultation with their technical teams and
management and/or response to external changes in public policies and markets. For
PG&E and SCE, this process and criteria used to select which projects to launch, how
much budget to allocate and the project scope of work is kept internal and is not
transparent to the CPUC, stakeholders or the public until the IOUs file their Annual
Reports each February 28 (covering the prior year). SDG&E provides a budget estimation
table for each EPIC project in its Investment Plan. This table does not provide specifics;

however, it does provide total estimated funding allocation to internal SDG&E employees,
outside consultants, and equipment and contractors funds.
When the IOUs seek a vendor through a competitive bidding process, they are required to
adhere to specific EPIC guidelines and a general process. Each IOU is also permitted to
follow its individual corporate procurement, supply management and affiliate compliance
rules. RFPs are sent to pre-screened suppliers with the appropriate corporate
qualifications, including certified Diverse Business Enterprises (DBE), and/or posted on
IOU websites for response by all interested parties. Where competitive bids are used,
utility communications, including bidder conferences and answers to RFP questions, are
offered equally to all bidders under consideration. IOUs may also use a Request for
Information (RFI) to request feedback from vendors about a project scope.
Potential vendors may be pre-qualified through RFI or Request for Qualifications (RFQ)
process initiated prior to issuing an RFP. The RFIs and RFQs consider relevant factors such
as (but not limited to):
• Potential suppliers’ individual capabilities;

• Product and/or service offerings; and
• Past performance (e.g., technical work and financial status).
For each RFP, IOUs publish a document outlining the criteria by which the IOU will score
and judge each bid, including eligibility requirements, submission instructions and an
outline of the evaluation and award process. The IOU scoring and selection process
follows a two-stage process that differs from the CEC process in the second round. The
IOU administrators follow a comparable screening process as the CEC uses for an initial
pass/fail round. Projects that pass the first stage are judged based on a variety of factors
that are specifically chosen to meet EPIC objectives and address the project work needed.
Factors are to be scored on a numeric scale and given weight based on their comparative
importance to the project. Possible factors include:
• Overall technical merits of a proposed approach

• Bidder capability, skills and related experience
• Diversity
• Budget
• Proposal feasibility

The IOU administrators differ in the funding mechanisms used for vendor compensation.
SCE and SDG&E use pay-for-performance contracts while PG&E uses a combination of
time-and-materials and pay-for-performance contracts.58
To evaluate the IOU project selection processes, we reviewed a sample of six IOU projects
to confirm that the projects aligned with the strategic objectives and electric system value
chain mapping outlined in the IOU Triennial Investment Plans.59 We reviewed project
documentation and scoring documents to confirm that projects aligned with the
Investment Plans and responses to RFPs were scored and awarded based on the published
criteria. We also reviewed the 46 bids associated with the six RFPs.
Table 38: Solicitation Sample for Detailed Review

Number
of Total Funding
IOU and EPIC Number of Awarded Awarded to Total Project
Project Number Round Bids Vendors Vendors Funding
PG&E-1.08 1 7 1 $585,000 $2,800,000
PG&E-1.09A 1 4 1 $84,600 $1,100,000
PG&E-1.09B 1 5 2 $75,500 $550,000
PG&E-2.1 2 3 1 $1,919,735 $2,420,000
SCE-13-0158 2 22 1 $231,000 $3,995,462
SDG&E Project 7 2 5 1 $1,166,000 $1,200,000
Total 46 7 $4,061835 $12,065,462
The second task was an assessment of the IOU review and scoring process for each
application. In contrast to the CEC where there is a specific set of scoring requirements,
each IOU defines its scoring criteria according to key factors related to each specific
project. For each IOU project, a committee of scorers consisting of key project staff from
the IOU scored applications according to the unique criteria of each project.60 The IOUs
also do not have the same minimum score requirements. The IOUs select projects based on
the highest scoring bidder according to the unique project scoring criteria, and in the case
of a tie, choose the lowest cost bid.
58 Recently, PG&E has moved to executing primarily pay-for-performance contracts.

59 As described in the methodology section, we drew this sample of six projects from the sample of projects
selected for interviews and documentation analysis.
60 The composition of scoring committees is determined by project, and does not adhere to any specific
guidelines.

We reviewed the scoring criteria for each project, as well as the final scoring for each
vendor bid. We confirmed that the scoring criteria are clearly defined and the scoring
process was well documented for each project and each vendor bid. All bids and scoring
results that we reviewed complied with the IOU grant solicitation guidelines described in
D. 13-11-025.
8.2 Project Scoping Process

We reviewed how the administrators develop scopes of work for projects once they select
them based on interviews with the EPIC teams' management and review of project sample
documents. Once projects are selected, administrators document key project components
such as project scope, any identifiable market deficiency or knowledge gaps, defined
project goals, key objectives, and individual metrics. As shown in Table 39, the manner in
which these key components are documented and incorporated into project planning
varies across administrators. CEC projects include overarching scopes of work and grant
request forms, while IOU projects tend to convey the information through business and
implementation plans that are developed with key project team members. The audience
for the documents and whether they are made available to stakeholders or the public also
varies. The IOUs typically create their planning documents for their internal use only,
versus the CEC, which posts the grant request form (with project scope of work) publicly.

Table 39: Project Documentation Sources by Administrator
Location in Which the Following Items are Documented

Identified Problem / Key
Admini- Knowledge Gap Defined Project Objectives / Intended
strator Project Scope Addressed Goals Targets Metrics Audience
CEC Scope of Scope of work/Grant Scope of Scope of Scope of work/Grant CEC, grantee, TAC
work/Grant Request Form* work/Grant work/Grant Request Form* and PAC (for
Request Form* Request Form Request Form* /Implementation plan projects that have a
TAC)
PG&E Business Plan Business Plan Business Plan Business Plan Business Plan IOU project
management team
and IOU internal
stakeholders
SCE Project Fact Project Fact Project Fact Sheet/ Implementation Scope of IOU project
Sheet Sheet/Implementation Implementation Plans/ work/Implementation management team
Plan Plan Investment Plans and IOU internal
plans* stakeholders
SDG&E Project Plans Project Plans Project Plans Project Plans Project IOU project
Plan/Demonstration management team
Plan and IOU internal
stakeholders
*Publically shared

8.2.1 CEC
Project Scope
For sampled CEC projects that we reviewed, the project scope was either provided as a
standalone scope of work document or as a key component of the grant request form. As
outlined above, the scope of work documents included a short, relatively broad
description of the general scope of the project and delineation of tasks to be completed as
part of the grant. Figure 13 provides an illustrative example of a task list and subsequent
paragraph on the overarching project scope.
Figure 13: Example Task Outline from Project EPC-14-055
Note: CPR refers to the scheduled Critical Project Review meetings for the project, with an “X” indicating the task is a
primary focus of the CPR meetings. (The footnote next to the CPR column heading indicates that full descriptions of
the CPR can be found in Part III of the Scope of Work.)
The purpose of this Agreement is to install and demonstrate battery storage and a microgrid energy
management system using existing photovoltaic (PV) systems. This agreement will fund the
evaluation of the next generation of microgrid energy management systems based on a concept of an
“Internet of Energy” using a standards based Energy Operating System with Energy Management
Applications that can connect and communicate with any type of energy asset and can connect and
communicate outside the local facility microgrid to the programs and systems of the local energy
provider and regional energy coordinators. The results of this project will be described in a
Microgrid Blueprint that will provide educational institutions the means to evaluate, plan and
implement the addition of energy storage with smart microgrid energy management controls to
their existing solar photovoltaic (PV) and other renewable energy sources.
Knowledge Gap
For sampled CEC projects, the knowledge gap was primarily presented in the “Problem”
or “Solution” sections of the scope of work. In general, the knowledge gap was more
indirectly referenced in comparison to other project components such as objectives and
goals. Below is an example of the “Problem” and “Solution” sections for a component of

project 300-15-009 (from the scope of work), a CEC project that examined microgrid
feasibility in California.
Problem
Microgrids tend to be customized aggregations and optimizations of diverse distributed energy

resources (DERs). Therefore, microgrids often require expensive customized engineering
solutions incorporating emerging technologies such as advanced energy storage often requiring
government subsidy. However, there are microgrids that are moving forward without
significant government subsidies. The Energy Commission would benefit from understanding
the technologies, business models, scale, and vendor landscape supporting microgrids that
appear to be economically viable without significant government subsidy. However there is
limited public documentation about these microgrids to support Energy Commission’s needs.
Solution
The deliverables resulting from this work authorization will help fill the information gap and
inform how best to promote and accelerate microgrid adoption by learning about how successful
projects have been developed without government support. Drawing from existing examples
throughout California, North America and the world will help build a broader understanding of
how universal some proposed microgrid solutions may be and whether regulatory structures,
incentives and differing vendor approaches to market development can all inform the Energy
Commission about best practices for economic viability.
Objectives
Within the scope of work documentation, the CEC projects generally included only high-
level objectives that serve more as a broad overview of what the project seeks to achieve
than a detailed account of what each component of the research is focused on. Below is an
example from the same CEC microgrid project 300-15-009:
The objectives of this WA [work authorization] are to:
• Create a common language and definition of what a microgrid is in order to sharpen broad
market understanding among stakeholders in order to pinpoint market drivers and barriers
to future microgrid development.
• Profile successful microgrid projects developed in California, North America and the rest of
the world.
• Better understand the business case for microgrids across different markets by comparing
case studies of market participants, from around the world, and identifying common
features, synergies, and business models that result in successful microgrid projects.
• Develop reports, based on reviewed case studies, to inform the Microgrid Research Roadmap
with recommendations for growing the market for microgrids through public and private
sector actions, and meeting California’s energy supply, resiliency, and climate change goals.

• Inform the Energy Commission’s future EPIC funding objectives for microgrids and DERs
in general.
Project Goals
For the sampled CEC projects, the project goals were generally outlined in the grant
request or scope of work documents. In comparison to the objectives, the CEC provided a
more detailed look at the desired technical goals and benefits of the proposed EPIC
project. For example, as shown below, CEC project EPC-14-055 outlined how it planned to
develop and disseminate information regarding its microgrid research:
• Develop and Disseminate a Microgrid Blueprint that can be used by educational

institutions statewide to evaluate, plan and install a smart microgrid that will manage and
coordinate the output of their existing renewable energy assets using energy storage systems
with the ability to provide benefits to the local utility grid though automating demand
response and other energy services.
• Demonstrate the benefits to customers, utility companies and CAISO of an “Internet of
Energy” concept using Institute of Electrical and Electronic Engineers (IEEE) and
American National Standards Institute (ANSI) standards based Energy Operating System
and standardized Energy Management Applications to control and coordinate local energy
assets and enable coordination with utility programs and controls.
• Collaborate with Pacific Gas and Electric (PG&E) to gain a better understanding of the
benefits and risks and practical realities of using behind the meter energy assets to provide
grid and/or market services using real world operating data.
Metrics
In general, the metrics were the most varied project documentation element across each
individual project. For example, while all CEC projects followed similar templates for
presenting objectives and goals, the metrics varied across projects, as some were more
measureable technical metrics and others appeared to be more qualitative in nature. See
Section 9.2 for more information about the processes to track and quantify project benefits.
8.2.2 IOUs
Project Scope
PG&E and SCE project scope documents that we reviewed contained high-level
descriptions for the projects with an overview of the market the project was targeting and
the project itself. The level of detail on project implementation varied, but provided less
detail than most CEC project scopes. For example, one business plan divided the project
into tasks with identified budgets and deliverables, but the task descriptions could be as
short as a few words or three or four bullet points for a $500,000 task. SDG&E project plans
provide more detail, including a description of the task, objective, a detailed approach that
they will take to obtain the objective and the output/deliverables that will be produced

from the task. To illustrate the level of detail and type of information included in many of
the IOU plans, we provide examples from a sampled PG&E project below.
Below is a description that PG&E provided for project #1.09A on Close Proximity
Switching that is illustrative of other IOU project descriptions.
Phase 1 of this project will focus on demonstration of a portable remote controlled switch
operator tool for sub surface Load Break Oil Rotary (LBOR) switches. This tool is a field
installed device operated remotely by an individual on an underground switch location. This
tool is a portable field installed unit which is used, and then returned to the vehicle after job has
been completed. It can be used to open and close the rotary operated oil-filled switch, a mounting
hardware to securely mount the tool to different sub-surface switch manufacturers and a remote
control to operate the tool. The tool will open and close the switch at a safe distance enhancing
worker and public safety.
The approach to the manufacturing requirements during this demonstration project will be to
award up to three different vendors a contract to build this tool in parallel. Once the prototypes
have been developed and delivered by these three firms, PG&E will be an optimal position to
choose the best solution of the three to carry forward.
Knowledge Gap
IOUs follow a similar process as the CEC, with the knowledge gap primarily presented as
the “Problem” or “Solution” sections of the scope of work. We present the manner in
which PG&E documented the knowledge gap for its project #1.09A below.
PG&E has over 2,000 legacy sub-surface oil filled switches that have no oil level indicators (pre
1976) and has had 255 reports of failed oil switches since 2000. Of these 255 reported failed oil
switches, 57 were significant failures with risk to employee and public safety. Thirty-five of the
failures were on switches manufactured in the 1970s or 1980s. Due to the aging distribution
infrastructure, there is a higher risk of significant failure when operating older sub-surface oil-
filled switches potentially endangering workers and affecting public safety. Older sub-surface
oil-filled switches are operated manually in close proximity to the switch.
Objectives
In general, the IOUs present project objectives in a similar format as the CEC. However, in
some cases, the objectives were seemingly presented to show how they correspond to
EPIC program goals at large. For example, for PG&E #1.09A on Close Proximity
Switching, the “primary objective” on worker safety was presented as it related to EPIC’s
overarching goal of increase safety:
This project’s primary objective is to enhance worker’s safety by using a remote-controlled tool
when operating rotary operated sub surface switches. The tool will also potentially provide

additional benefit in reducing outage duration by eliminating the number of potential incidents
caused by a significant failure when operating sub surface switches.
Project Goals
For the IOU projects, the project objectives were presented in a much more general sense
as compared to the CEC, while the objectives corresponded with specific targets of each
individual project. For example, the PG&E project business plans contain project goals as
they pertain to PG&E’s “core values” on public safety, employee safety, reliability,
operational excellence, customer focus, delighted customers, and environmental
leadership. For PG&E project #1.09A, project goals are described as:
This project aligns with three PG&E goals: Public Safety, Employee Safety and Reliability.
Metrics
IOU projects also included a range of metric types. In the business plan for PG&E
project #2.24, the project team identified “High Level Success Criteria” for different project
deliverables. For example, in the planning phase—consisting of creating the business and
project plans—the key metric for success was creating “viable table-top proof of project
feasibility,” while the building and testing stage included more technically focused metrics
based on their project models, such as “Expected test and operating results achieved” for
the residential Smart AC data loggers.
However, while these test metrics directly correspond with technical components of the
project, the metrics themselves remain high level and in certain cases do not further
identify the specifics of the quantitative “results” alluded to in the metrics portion of the
program documentation.
Similarly, SCE includes a list of metrics with its description of upcoming projects in its
Investment Plan. These lists comprise generic types of metrics to be collected and reported
at the conclusion of the projects. They span technical, economic and market issues and
include such varied metrics as “the number and total nameplate capacity of distributed
generation facilities,” “non-energy economic benefits,” and “forecast accuracy
improvement” associated with a project, but without further discussion or detail. SDG&E
also refers to metrics in a general way in its Investment Plans, but defers their
development to project-specific plans.
8.3 Match Funding

The CEC explicitly seeks match funding from bidders who respond to EPIC solicitations (it
is required for TD&D projects; for the other project types, applicants receive bonus points
for offering it). We compiled project data from the 2016 Annual Reports to assess the
extent to which CEC projects have match funding. Figure 14 shows how the reported total
sum of $195,064,775 in match funding (through 2016) was distributed by project type in
their EPIC portfolio.

Figure 14: CEC Match Funding by Project Type (through 2016)
R&D $85,170,503
TD&D $64,598,152
MF $45,296,120
Figure 15 and Figure 16 show the ratio of match funding (equal to match funding divided
by the sum of match funding and project budget), by project and grantee type. The overall
ratio is 32 percent across all of the CEC’s projects. As shown, Market Facilitation projects
have the highest match funding ratio. Among grantees, trade organizations have the
highest match funding ratio.
Figure 15: Match Funding Ratio by Project Type (CEC Only, through 2016)
MF 43%
TD&D 32%
R&D 30%
0% 10% 20% 30% 40% 50%

Figure 16: Match Funding Ratio by Grantee Type (CEC Only, through 2016)
Trade Organization 42%
Private Business 38%
Non-Profit Agency 34%
Investor-Owned Utility 27%
Government 25%
Academic 21%
0% 10% 20% 30% 40% 50%

We discussed match funding with the IOU EPIC program management teams during a
series of telephone interviews. The IOUs do not require match funding (what they also
refer to as cost sharing or in kind support). The IOUs instead seek cost sharing indirectly
by engaging in a number of practices, which they typically employ for all their
subcontracting. First, they negotiate with vendors on their rates when setting up master
contracts, upon initiation and at renewal. They also encourage vendors to offer discounts
or even to offer services at no cost (we confirmed at least two cases of this in our sample of
six vendor bids from our project sample.) They do not currently track cost sharing or in
kind support.
One IOU program manager acknowledged that the CEC is better about instituting a
formal process to get bidders to agree to provide match funding, but felt for the IOUs, it
might be better to have an informal process for which they use their standard negotiation
procedures, which they think of as due diligence on all their contracts. One IOU felt that if
they were required to seek match funding, vendors may just mark up their project cost
rates.
SDG&E gave some examples of projects planned for the third investment planning period
where they may get match funding, based on the type of bidder (e.g., a lab and a
government agency). But these potential project examples might be more on an ad hoc
basis, since IOU project partners are often from the private sector, whereas less than half of
CEC projects have been awarded to private sector firms.
Five of the 12 Idea Exchange respondents indicated that the CEC’s match funding
requirement is a burden, particularly for small businesses and entrepreneurs. A couple of

respondents felt the substantial amount of the cost share61 is difficult to secure, and felt
that EPIC should offer opportunities for smaller grants and match-funding obligations.62
One respondent felt that this obligation should be eliminated entirely. Another respondent
felt that the cost share should be optional, and that small businesses that secure match
funding should receive weighted or bonus scores toward their applications since it is
harder for them to procure such funding. Two respondents suggested the CEC should
facilitate matches with other funding opportunities. One respondent proposed that the
CEC provide an option for the cost share to be offered as a loan. These concerns regarding
match funding indicate that this requirement may be a specific barrier for small business
applicants. The CEC might consider exploring different options to determine what might
work best.
8.4 Plan Consistency

We compared the current status of our sample of projects versus what the administrators
had filed in previous Investment Plans and Annual Reports to ensure consistency over
time. We also referred to the latest project status reports (from the 2016 Annual Reports,
filed on February 28, 2017) to provide current information about projects. Across all
administrators, we found good consistency in the reporting of interim development and
projected milestone achievements. These reported values aligned well across the spectrum
of reports, and projected future developments were well represented in successive reports.
No inconsistency was found in the most recent project status report as compared with
previously reported values within the sample projects.
Note that the administrators may add, modify or cancel a project, so the current status of
projects is different from what is filed in their Investment Plans. Per CPUC Decision 15-09-
005 (September 17, 2015), they are required to file Tier 3 Advice Letters for new EPIC
projects or for material changes to existing approved Investment Plans.
PG&E filed an Advice Letter on February 7, 2017, to request approval for several new
EPIC projects. Specially, PG&E requested approval for six new projects with an estimated
budget of between $8 million and $10 million out of its 2015-2017 EPIC budget of $51
million. As of the time we conducted the second round of administrator interviews, the
CPUC had not yet responded to PG&E’s Advice Letter.
61 Match funding is only required for TD&D projects. The applicant must match at least 20 percent of the
requested funds. Applied R&D projects do not require match funding, but applicants can receive extra
points on their proposal for such funding. Up to 10 percent of EPIC funding for Applied R&D and TD&D
can be used as federal cost share.
62 EPIC funds a project, the California Sustainable Energy Entrepreneur Development (CalSEED) Initiative,
which provides up to $600,000 in grant funding for early-stage research and development for emerging
technologies.

PG&E justified its request because of changes in the policy landscape and advances in
technologies which require the company to adapt its priorities. PG&E put some projects on
hold in order to dedicate budget for the new projects. PG&E argued in its Advice Letter
that had it waited until its next Investment Plan (filed on May 1, 2017, for the program
period 2018-2020), that would have delayed the ability to use the knowledge and benefits
gained. The new projects PG&E requested to add are described below:
1. Aggregated Behind-The-Meter Storage Market / Retail Optimization: This project

will demonstrate how aggregated behind-the-meter energy storage systems that are
operated by a third party dispatcher may address wholesale market needs, while
also operating as a customer resource to reduce customers’ retail electric bills.
2. Service Issue Identification Leveraging Momentary Outage Information: This
project will demonstrate an approach to proactively identify potential service issue
problems related to locations with frequent momentary outages, which may be
caused by imminent failures of conductors, insulators, transformers and/or
vegetation contact.
3. Predictive Risk Identification with Radio Frequency (RF) Added to Line Sensors:
This project will demonstrate an approach to integrate real-time radio frequency
(RF) monitoring technologies into Line Sensor devices to potentially improve
outage prediction and identify areas for grid reliability improvement.
4. Call Center Staffing Optimization: This project will create and demonstrate an
algorithm to optimize call center staffing to better match call volumes (including for
major events) through predictive modeling, incorporating data from historical
volumes correlated with data such as general news, PG&E announcements,
regulatory proceedings, rate schedule seasons, weather information, restoration
times, and/or other data sources.
5. Electric Load Management for Ridesharing Electrification: This project will be
implemented in order to understand and demonstrate grid impacts from Electric
Vehicle (EV) charging used for ridesharing fleet applications and assess the ability
to manage the resulting electric load using active demand management.
6. Dynamic Rate Design Tool: This project will develop and demonstrate new
analytical solutions and modeling to bring increased flexibility and speed to
designing more dynamic rates while understanding the impact on customer bills, as
well as better understanding potential customer load changes as a result of different
rates.
In our review of these new projects, all but the Call Center Staffing Optimization project
seem to be justified and added to address new technological developments. For example,
Behind-The-Meter storage aggregation and dispatch is a new concept that is being
explored as a product of the recent developments in the lower-cost personal home use
battery system market (i.e. Powerwall and Sonnenbatterie systems). The utilization of this
new product through aggregate dispatch will be a low-cost way to help the IOUs achieve

their mandated 2020 energy storage portfolio mandates, through the leverage of private
investment. However, it should be noted that project #2.01, Evaluate Storage on the
Distribution Grid, was placed on hold to free up budget for this project. To the extent that
energy storage needs assessment and siting will be required to facilitate improved grid
reliability within a high DER system, it is likely that this action represents a shift in
prioritization toward achieving storage portfolio standard requirements, coupled with a
perception that rooftop DER build outs will continue to slow, as they have since the end of
the California Solar Initiative program. It is also likely that energy storage siting, as a grid-
optimization tool, is relatively straightforward and that more granular-scale storage build
outs will not be cost-effective, can be better met through other adaption strategies (i.e.
increased switchgear automation) or cannot be adequately performed during this period
of rapid customer-driven DER adoption.
Similarly, the extremely rapid rate of technological development of autonomous electric

vehicle (AEV) ridesharing platforms (currently scheduled to be in operation by 2021), the
need for the development of new accompanying rate schedules for both AEV/EV
operations and the continuing installation of distributed rooftop solar and accompanying
battery storage systems collectively warrant a shift in the research/demonstration
activities to facilitate their smooth integration into the PG&E distribution grid.
The Call Center project, however, does not appear to be something that is being added to
address external developments (such as market, policy or technology changes). The project
seems to be a routine optimization effort that could have been planned in advance or
proposed as part of the next EPIC portfolio period.
We examined the projects to determine if they could have been fit into existing Investment
Plan project areas, since the descriptions are general and no budgets are assigned, lending
flexibility to introduce new project concepts mid-cycle. We found that three of the six
projects, though justified as new project concepts, could have been addressed by existing
project descriptions from their Investment Plans (i.e., not requiring an Advice Letter). Two
of the new projects (#5 and #6 from the list above), also justified as new project concepts,
would not fit within existing Investment Plan descriptions and are justifiable new projects
that warrant being included in the Advice Letter request. However, one of the projects (#4)
does not seem like a justifiable project to introduce mid-cycle, nor does it fit within the
guiding principles of the EPIC portfolio. It could contain duplication of current call center
activities currently underway within internal PG&E processes. This project is an example
of an internal IOU need that does not clearly align with a state strategic policy need.
The other two IOUs also indicated they would like the opportunity to make mid-course
changes in projects without having to go through the formal Tier 3 Advice Letter process,
which requires approval by CPUC resolution and can take several months or longer to
address. They reported that they have to anticipate the specific projects they will

implement over a three-year period, and that constrains them from fully being able to
respond to market and technology changes.
However, in our review of the Investment Plans and the processes the IOUs use, we found
that there is significant flexibility to modify and introduce new projects, and to place
projects introduced in an Investment Plan indefinitely on hold. In PG&E’s request to add
projects, two of the six were new projects that did not fit well under their existing
Investment Plan project descriptions, so there are limited cases where new projects would
require the Advice Letter process. But generally, there is extensive flexibility built into the
system and based on how the IOUs interpret the requirements and administer their
program.
8.5 Coordination
We assessed how the administrators coordinate with each other, and also with other
energy innovation efforts across the country.
8.5.1 Coordination Across Administrators

We begin by noting that the IOUs and the CEC have agreed to pursue the following
principles for cooperating and collaborating for EPIC-funded projects:
1. Information sharing and coordinated planning;

2. Leveraging funding and avoiding duplication of projects;
3. Consistent evaluation, measurement and verification of RD&D results;
4. Coordinated input and advice from stakeholders; and
5. Intellectual Property.
During the administrator interviews, we asked the administrators about how they
coordinate program activities. The EPIC administrators described how they hold regular
coordination meetings biweekly at minimum (all four administrators meet biweekly, and
on the weeks in between, the IOU administrators meet). They also described how they
coordinate program activities at various stages including investment planning, project
selection, project implementation and results dissemination. For example, they coordinate
to plan EPIC stakeholder meetings and workshops, jointly developing meeting agenda
and deciding on meeting logistics. The four administrators also coordinate on projects that
the CPUC mandates that they implement jointly. Since the CPUC set a requirement for an
annual EPIC Symposium in Decision 15-04-020, they also coordinate to plan and host that
event. They share project results with each other informally (e.g., during coordination
meetings) and formally (by attending project presentations). However, based on these
interviews and document reviews, in appears that the CEC and IOUs focus their
coordination activities more on program administration activities and less on individual
projects.

The IOUs coordinate more with each other on individual EPIC projects, since they may
directly benefit from each others’ project results, and they want to use program funds
judiciously and limit duplication of efforts. There are cross-IOU technology teams that
coordinate, and the Program provides a unifying framework for them to test technologies
before they go to production scale. These technology teams also have routine working
relationships with each other where they share information about project scopes, project
progress and completed projects to ensure coordination and avoid duplication. However,
each IOUs’ grid is somewhat unique along with their customer bases. We asked the IOU
EPIC program managers about the extent of duplication with the other IOU
administrators. All the IOUs indicated there is some duplication, but little to none of this
duplication is unnecessary; they described the Electric Power Research Institute's (EPRI’s)
involvement in their investment planning activities as the external vetting that confirms
this. The IOUs' technical teams also meet on an ad hoc basis to discuss projects that have
recently closed to provide lessons learned as another measure to prevent unnecessary
duplication of EPIC projects. Likewise, these technical teams are in contact with external
stakeholders, which is another measure they take to avoid duplication with utility projects
outside the state. No administrator could think of concrete examples of coordination
activities that led to significant program changes between IOUs to facilitate synergies
between portfolios.
One stakeholder who we interviewed suggested that there has been duplication of TD&D
projects across IOUs. The stakeholder thought that the recurrence of similar TD&D
projects across the IOUs stemmed from a few different factors including lack of
transparency related to funding priorities and decisions, combined with limited awareness
of project accomplishments and follow-on funding. The other stakeholders had not
observed any duplication.
We conducted our own assessment to determine whether we could identify unnecessary

duplication across IOU projects. We conducted a cross-platform analysis of the portfolio of
IOU projects (covering EPIC 1 and EPIC 2, through the end of 2016) based on the IOUs’
project status reports along with information from the Annual Reports and Investment
Plans. We assigned tracking metrics based on project descriptions, which we compared
between individual projects that demonstrated similar technologies and processes. We
selected projects that scored a high potential for duplication as a result of this first-level
analysis and conducted an in-depth documentation review to determine the level of
duplication (if any) between these projects.
For example, the IOU project portfolio included six projects that addressed the potential
for increased integration of grid-scale energy storage technologies. Two projects, one
administered by PG&E and the second by SCE, evaluated market benefits, cost-
effectiveness and processes for the identification of key distribution grid locations for
potential storage systems. Duplication between these projects may be observed due to the
similarity of technologies and cost-assessment analysis methodologies. However, the

inherent differences between distribution grid operations, including the physical
distribution of generation resources, regional-scale load profiles and subsequent economic
benefits of regionally-sourced storage, significantly reduced duplication impacts to
portfolio efficiencies.
We also conducted a review of the IOU third triennial Investment Plans to identify
potential duplication of IOU activities going forward. Similar results were found that
warranted some level of necessary duplication to address the unique differences between
service territories. However, other potential project duplications exist where specific and
shared technologies are being demonstrated (for example, the integration of EV batteries
acquired through ‘second-life’ purchases and configured as an additional source of grid-
scale storage potential). Other potential duplications exist in methodologies of automated
demand response aggregation and price-based controls. The results of this review suggest
the need for an earlier external review of IOU projects before they appear in the
Investment Plans to ensure no unnecessary duplication. Otherwise, the CPUC gets an
opportunity to review the Investment Plans by reading each IOUs’ project descriptions,
and it may be difficult at that late stage with only general project descriptions to identify
potential duplication.
There are opportunities for the IOUs to coordinate with the CEC on its R&D projects,
which could possibly lead to demonstration (TD&D) projects.63 However, the CEC has
completed very few of its projects, while the IOUs are well into the process of planning
their third investment portfolio, which specifies projects through 2020. Thus, the
opportunity for intra-agency coordination and project throughput is limited at this time.
There is likely the opportunity for future coordination once the CEC completes more
projects, but the IOUs indicated there is not a set process in place, so any coordination
would be ad hoc.
Stakeholders interviewed for this evaluation expressed mixed opinions regarding

coordination between the CEC and IOUs. Because most stakeholders indicated that they
worked with either the CEC or one or more of the IOUs but typically did not work with
both the CEC and IOUs, they were not very aware of the extent to which administrators
across all four entities coordinate their efforts. Two stakeholders generally thought that
work was well coordinated, and one of these individuals also thought that coordination
among the IOUs had improved over time. On the other hand, two other stakeholders
thought that the CEC and IOUs came together during key times related to investment
planning, but believed that much of this coordination was limited to specific investment
planning events and was not sustained outside of these times.
63The IOUs are not allowed to bid on CEC solicitations where they provided input per CPUC Decision 15-
04-020.

8.5.2 Coordination with Other Energy Innovation Efforts
In addition to coordinating across administrators, the CEC and IOUs also attempt to
coordinate with R&D efforts outside the Program to exchange information. In this section,
we describe the processes the administrators use as well as feedback from stakeholders.
Later, in Section 9.3.2, we present a network analysis that independently validates and
reports on quantitative measures of EPIC networking.
The CEC project teams formally engage technical experts through the TAC that is formed
for each project (and in limited cases, a policy advisory committee [PAC]). For both IOU
and CEC projects, EPIC project teams (e.g., the Commission Agreement Manager [CAM],
IOU administrator, grantees and members of the TACs) make formal and informal
connections with a wide range of other organizations beyond the project team to share
knowledge and experience. Such organizations as government research laboratories,
private research and consulting companies, universities, utility associations and
manufacturers are included in this network (see Sections 7 and 9.3 for more details
regarding coordination). Both the CEC and the IOUs also have internal subject matter
experts who routinely coordinate with other individuals and entities in order to stay
current. Both the administrators described this type of coordination and information
exchange, which was also corroborated by project-level interviews.
The CEC has in-house technical experts on whom it relies to coordinate with other
innovation efforts. Furthermore, the CEC works on related issues outside of EPIC and is
already engaged with stakeholders for selected technologies, both at a policy level and in
developing roadmaps. At the program level, the CEC also participates in the Association
of State Energy Research & Technology Transfer Institutions (ASERTTI), which is a
nonprofit organization whose mission is to increase the effectiveness of energy research
efforts in contribution to economic growth, environmental quality and energy security,
through:
• Collaboration on research projects with state, federal and private partners; and
• Sharing technical and operational information among members and associates
(including the DOE [Department of Energy] and Lawrence Berkeley National
Laboratory [LBNL]).
The IOUs have formalized the seeking of input from EPRI in order to minimize the
chances of duplicating projects being conducted by other utilities worldwide. This
feedback and IOU response are documented in their Investment Plans and Annual
Reports. The IOUs also rely on their in-house technical experts, who engage with
stakeholders (such as universities, academic institutions, national labs) during the course
of addressing other CPUC proceedings.
In general, stakeholders that we interviewed had little insight into the extent to which
EPIC investments are coordinated with other efforts led by statewide, federal, private and

other research entities. One interviewee said that the Program needs more coordinated
statewide or western region planning. Another stakeholder thought that EPIC and
California energy policy planners in general could do more to coordinate with federal
programs and initiatives. In both instances, the interviewees did not offer any concrete
suggestions or strategies.
Stakeholders reported that EPIC supports California’s policy goals in principle. Limited
results regarding program impact makes it difficult for them to assess the extent to which
the Program as a whole or individual projects are advancing various aspects of key
policies such as GHG reduction and renewable energy generation. In addition, as noted
above, a few stakeholders felt that the Program could be better coordinated statewide and
with other regional and federal efforts, which would help identify both gaps and solutions
on which the state should focus resources. Citing examples of research to improve grid
resiliency and cybersecurity, one stakeholder thought that EPIC is bypassing or
overlooking opportunities to play a strategic role in the work being done in these areas
nationally. This stakeholder thought this oversight might be an outcome of an EPIC focus
on the state-mandated goals.

The best practices assessment examined how effective practices related to project selection,
including alignment with program goals and objectives, selection criteria, the review
process and match funding. This section summarizes key findings from the literature
review and interviews with administrators of seven peer RD&D programs.
The literature review showed that rigorous selection criteria, strong alignment with overall
goals and objectives, and a peer review process are fairly common among RD&D
programs. All six of the peer RD&D programs that offer grants (or, in the case of the
DOE's Small Business Innovation Research [SBIR] and Small Business Technology Transfer
[STTR] Programs, cooperative agreements) provide transparent information about RD&D
funding opportunities, proposal review criteria and the selection process. Each of these
programs includes information about funding opportunities on their website. In general,
the funding announcements describe technology-related parameters, funding amount and
duration, due date and award date. The SBIR and STTR Programs and ARPA-E also have
webinar archives that provide background on the program and criteria for eligibility, as
well as information on how applications are reviewed and selected for funding.
All four of the primary peer RD&D programs require applicants to provide a preliminary
research description prior to developing and submitting a full proposal. The SBIR and
STTR Programs require a letter of intent while ARPA-E and the New York State Energy
Research and Development Authority's (NYSERDA’S) T&MD Program require a concept
paper. While the specific requirements are different for each program, this preliminary
description serves a few purposes. It allows program administrators to get a sense of the

technology areas that are likely to be addressed in response to a particular solicitation,
which helps them determine the types of external reviewers that will be needed for
proposal review. It also allows the applicant to describe their proposed research with
relatively modest commitment in terms of time and resources. Based on a review of the
preliminary description, the programs provide applicants with a notice of encouragement
or discouragement to submit a full proposal.
8.6.1 Match Funding

Three of the seven peer RD&D programs, including two primary peer RD&D programs,
require projects to obtain match funding; while they do not directly assist with finding the
match funding, to the extent that they can, they try to help address any barriers the project
team may face. Washington State’s Clean Energy Fund requires a 50 percent cost share,
NYSERDA’s T&MD Program requires 25 percent or 50 percent cost share depending on
the size of the project, and the DOE’s Advanced Research Projects Agency-Energy
Program's (ARPA-E’s) cost share varies by specific funding opportunity. While the SBIR
and STTR Programs do not mandate match funding, it is encouraged for projects that fall
under their Phase II funding, where the programs place increased emphasis on
commercialization.

The CEC’s model with respect to transparency about funding opportunities, proposal
review criteria, and the overall selection process generally is consistent with the peer
RD&D programs. The four primary peer RD&D programs use a two-stage application
process where interested applicants submit a concept paper or letter of intent prior to
developing a full proposal. According to interviewees, this benefits both program
administrators and applicants. Program administrators receive a preview of anticipated
submissions, including research and technology areas, and can start planning staffing
resources accordingly. Applicants can explore a funding opportunity without committing
extensive staff resources. To date, the CEC has offered four two-stage solicitations and
plans to offer more of these funding opportunities under EPIC 3. Given other programs’
success with this approach, this appears to be a promising development for EPIC, as it
might make the program more accessible to small businesses. The two-stage process also
helps them to assess if it is worth the relatively sizeable effort they would need to commit
to for a full-fledged application to EPIC.

Current EPIC
Selection Process
Provide information about The CEC has an open The existing CEC process appears to be
funding opportunities, including and transparent process; sufficient.
webinars and FAQs the IOU process is more The IOUs are more constrained by their
limited vendor selection. focus on internal system needs.
Utilize external reviewers The CEC does this, but The existing CEC process appears to be
the IOUs do not. sufficient.
The scope of the IOUs’ solicitations tends
to be more limited to vendor selection
for specific tasks, which may make this
unnecessary.
Two-stage application process The CEC does not The CEC should consider expanding this
regularly conduct a two- practice. However, we note that this adds
stage application process, time to the process (90 days or more,
but the IOUs do not per the CEC).
follow this practice at all. The scope of the IOUs’ solicitations tends
unnecessary.
Match Funding
Require match funding (three Match funding is required The CEC should explore different
of seven peer RD&D for CEC TD&D projects options, including removing the match
programs, including two and encouraged for funding requirement for TD&D or making
primary peer RD&D Applied R&D projects. it optional.
programs)* The IOUs do not do this The scope of the IOUs’ solicitations tends
at all. to be more limited to vendor selection
unnecessary.
Assist with identifying match The CEC does this to a The CEC should explore the possibility of
funding opportunities limited extent. providing this assistance, particularly for
small business applicants.
The scope of the IOUs’ solicitations tends
unnecessary.
* The SBIR and STTR Programs (programs focused on small businesses) do not mandate match funding, but they do
encourage it for Phase II projects.

9 Project Assessment Process
In this section, we discuss the effectiveness of the Program’s assessment processes, as
which informed our data collection efforts. Project assessment is a critical third step in the
interviews with EPIC stakeholders and conducted a best practices assessment.
9.1 Project Status Assessment

EPIC administrators report on existing projects in a variety of ways and formats. Based on
what we learned from our interviews with EPIC program management teams (several
interviews with each Administrator’s management team) and project managers (21 IOU
project managers and 32 CEC grantees), the common thread and most consistent reporting
mechanism for on-going projects is the administrators’ annual report, which is required by
the CPUC per Decision 13-11-025.
The administrators are required to provide an overview of the overarching components,

background information and regulatory processes of the Program as a whole. The reports
also include a project status report that is in a table format, which is consistent across
administrators per CPUC requirements. These tables present the following information
about each active project and approved EPIC project awards:
• Project name and description;

• Date of the award and timeline;
• Budget, spending, and matching funds;
• Project partners;
• Intellectual property and project results;
• Applicable metrics;
• Grant awarding process and summary of bidders/selection process; and
• Barriers addressed by project and technological advancements.
Figure 17 illustrates a project description from an annual report.

Figure 17: Example Project Description from an Annual Report

In addition, some administrators also report on any deliverables that have been
completed, such as modeling systems, interim reports or presentations, and demonstration
site updates.

Furthermore, information about some on-going projects is shared in public venues and
meetings that the administrators hold on at least an annual basis, as required by the
CPUC. The administrators hold a variety of symposiums and topic-specific meetings at
which information about on-going projects is shared among the CEC and IOUs. In general,
all administrators participate at these meetings, although some specified meetings may be
held by administrators individually and include project results and updates that expand
beyond EPIC. Examples include:
• EPIC Symposium: In 2015 and 2016, the CEC, in conjunction with PG&E, SCE and
SDG&E, hosted the EPIC Symposium to help highlight key findings from selected
projects. For example, PG&E’s Close Proximity Switching EPIC project was
presented at the 2016 symposium, focusing on project background and scope,
technical issues, and the key findings and recommendations for their three vendors
creating remote access proximity switches.
• Some project teams also participated in program-area meetings and workshops
focused on specific research topics within the Program. For example, in September
of 2016, the CEC held a workshop on microgrids where administrators presented
their EPIC-related research. Presenters included the project team behind the
Borrego Springs microgrid, who provided an overview of how they are making
their microgrid more flexible and automated during power outages.
• In addition, some CEC grantees and IOUs present results of their on-going work at
conferences and in peer-reviewed journals. For example, members of a grantee
team looking into the use of sulfur for thermal energy storage presented aspects of
their work at the American Society of Mechanical Engineers’ 2016 Power and
Energy Conference, and PG&E technical staff have attended the Energy Storage
North America Annual Conference on numerous occasions.
Later in this section (in 9.3), we more closely examine how the administrators share project
results. The remainder of this subsection focuses on how the administrators track and
report on projects while they are being implemented.
As mentioned previously, throughout the report we have tended to present results for the
CEC and all three IOUs combined due to the difference in their administrative approach
that we noted at the initial stages of the evaluation. However, we examined each of the
four administrators’ processes and projects, and where we noted differences across the
three IOUs, we discuss those differences. Note that those comparisons were limited by the
size of the project sample for each IOU, which reflects the relatively smaller portion of the
EPIC budget that the IOUs receive as compared to the CEC.
9.1.1 CEC
At the project level, the CEC Commission Agreement Managers (CAMs) generally appear
to be well informed about the status and progress of individual projects based on the

combination of formal project reporting and informal day-to-day interaction. The CEC
contracts out its projects, and it requires grantees to track and report on information at
least quarterly and often on a monthly basis in progress reports that accompany the
invoices. These progress reports include:
• Planned and actual accomplishments during the prior reporting period;

• Planned accomplishments during the next reporting period;
• A comparison of progress to the implementation plan;
• Evidence of progress;
• Current and cumulative expenditures compared to the budget; and
• Status of milestones and products.
CAMs told us that they find these reports to be a useful way to follow project progress.
Our review suggests that they do provide a good snapshot of project progress compared
to the scope of work—sometimes with helpful graphics, pictures and interim results.
We noted some normal variation in the degree to which CAMs focus on the technical
details of the projects and the frequency with which they interact with project teams
between regular progress reports. Furthermore, we noted a few instances in which
reassignments or personal leaves have resulted in mid-project transitions of CAM
responsibilities to other CEC staff. In these instances, CEC staff sometimes appeared to
have lesser familiarity with project details.
As mentioned previously, grantees are required to form a technical advisory committee

(TAC) for each CEC project to provide technical expertise and advise the CAM, such as
during a critical project review when lab tests results are available. The process of forming
and convening a TAC takes time and may not occur immediately at the start of a project.
At the time of our interviews, there were 25 projects with TACs formed (of the 32 projects
in our sample), five projects with no TAC, and two projects that were in the process. Of the
25 project teams that had formed their TACs, 12 had not held a formal meeting at the time
of our interview, eight project teams had met once, three had met twice and two project
teams had met three or more times. In limited cases, the CEC may convene a policy
advisory committee (PAC), where a project or set of projects could influence future
policy.64
64At the time we conducted our research, the CEC had convened one PAC (the Long Term Energy Scenarios
PAC), which is advising three projects related to GHG reduction goals and the impacts of a changing climate
on the state’s energy systems.

9.1.2 IOUs
The IOUs internally administer their projects in accordance with established
companywide and EPIC requirements. They have EPIC leads and internal project
sponsors for each project that track the necessary project data that go into the Annual
Report project status report. Their involvement on the project teams and updates to the
internal project sponsor keeps them informed about project progress to supplement
monthly progress reports. The IOU internal project team periodically reviews project
status (typically monthly), and they have higher levels of review and reporting that
happens quarterly and annually that involve management. The IOUs described how they
hold periodic meetings (usually quarterly) with their management to discuss projects and
any issues that have arisen. At that time, management weighs in on any decisions such as
to end a project or put a project on hold. A project might be ended if early results
suggested it was not productive to continue, or a project put on hold because of an
external development that made that project no longer a high priority (or if it was
duplicative with something else being done). Detailed processes for internal reporting,
go/no-go decisions, risk management and other project management and oversight
functions are utility-specific and vary accordingly.
9.1.3 Current Project Status

As of the end of 2016, 250 projects were currently in progress. A total of 19 projects have
been completed (1 CEC project and 18 IOU projects), 7 projects have been cancelled and 23
projects are on hold, based on the project status reports included with the administrators’
2016 Annual Reports. We talked to CEC CAMs and IOU project managers during the
course of project interviews about how occasionally projects are revised or rescoped to
adjust to unforeseen circumstances or project results. Administrators may cancel a project
if during the review they determine it is not productive to continue. But often, there is still
knowledge created that is worth sharing. The IOUs may also decide not to launch a project
if the market or technology question has shifted priorities or is no longer needed, and/or
they want to reserve budget to prioritize other projects.
Among the projects we sampled for closer review, CEC CAMs and IOU project managers
reported that 11 had been revised in some way. SCE was much more likely to have revised
their projects (three out of seven). Typical adjustments included adjustments to timeline,
technology upgrades (i.e. software updates, metering equipment), adding project partners,
reshaping scope to address policy questions or adjusting the bidding process (i.e. sole
sourcing versus competitive bidding). The adjustments were triggered by consultation
from TAC and PAC members, critical project reviews and discussions between the
administrators and grantees. Table 41 below shows the breakdown between CEC and IOU
administered projects.

Table 41: Project Revisions to Scope by Administrator (n=53)
Number of Number of Percent of
Sampled Projects Sampled Projects
Projects65 Revised Revised
CEC 32 6 19%
SCE 7 3 43%
SDG&E 4 0 0%
PG&E 10 2 20%
Total 53 11 21%
Stakeholders we interviewed for this evaluation (n=9) were not very aware of project-
related results. While they indicated interest in learning about project successes, it does not
appear that the EPIC administrators are sharing these findings widely to the general
stakeholders that we interviewed, which includes policymakers, state agencies and
research organizations. However, as we present later in this section, results are shared
with a significant number of organizations that are part of project-specific networks.
Although project results across the EPIC portfolio are limited and very preliminary at this
point, there seems to be some interest from the stakeholders we interviewed in receiving
periodic updates from the CEC and IOUs about EPIC project accomplishments. There also
appears to be a desire among stakeholders for greater transparency about how to access
project results to ensure that appropriate stakeholder groups, particularly those at the
CPUC, are informed about projects’ progress. This feedback loop is critical to helping
stakeholders fully understand why the CEC or one of the IOUs might conduct additional
research or deploy technology in a particular area.
9.2 Benefits Quantification

As mentioned previously, the administrators are required to identify metrics by which
each project’s success will be measured. CPUC Decision 12-05-037 requires the
administrators to propose metrics and criteria for awarding EPIC funding in individual
areas in their Investment Plans, against which the Investment Plan’s success should be
judged. The Decision provided a comprehensive list of metrics that administrators could
consider covering a range of benefits including:
• Potential energy and cost savings

• Job creation
65One sampled CEC project did not complete this question set of the interview because we discovered the
project had not yet started during the time of the scheduled interview.

• Economic benefits
• Environmental benefits
Decision 13-11-025 further states that:
“The Administrators should have the flexibility to choose metrics on a project-by-

project basis, and because the list of proposed metrics and potential areas of
measurement is not exhaustive, the Administrators should be able to use additional
metrics where appropriate. However, the Administrators must identify those metrics
in the EPIC annual report for each project.”
9.2.1 CEC
The CEC has a structured and transparent process in place for tracking project benefits,
based on our review of documents and grantee and CAM interviews associated with the
CEC project sample (n=32).
Grantees are required to complete three project benefit questionnaires—one in association

with the project kick-off meeting, one during the project, and one at the end of the project.
The questionnaires are designed to allow for increased customization of the benefit
assessment with subsequent questionnaires. Most EPIC projects are still ongoing, so the
majority has resulted in the completion of only the kick-off benefits questionnaire. In these
questionnaires, project benefits appear to be sufficiently and reasonably documented
either quantitatively or qualitatively based on our review of the project sample.
The actual estimates originate from the grantee and appear to be based on professional
judgment or an educated guess. If the project benefits cannot be reasonably quantified at
the current point in the project, the questionnaire allows for ranges of quantitative
estimates to be provided or a qualitative explanation of intended benefits. Table 42 lists the
benefits that are routinely tracked in the 24 questionnaires we were able to review from
our sample of projects.
Table 42: Project Benefits Tracked by CEC for EPIC Grants (n=24)
# with Quantified # with Qualitative
Benefit Estimates estimates
Potential Energy and Cost Savings 5 6
Job Creation 0 1
Economic Benefits 6 13
Environmental Benefits 9 5
Safety, Power, Quality, and Reliability
3 4
(Equipment, Electric System)

Once projects are complete, the CEC includes this information in the report it is required
to produce and include in its Annual Reports. We reviewed the CEC’s project close out
report for the single completed project in our sample, and we confirmed that the report
includes information about initial project benefits.
The CEC’s process appears to be well thought out and thorough, and addresses the
CPUC’s requirements to measure and report on project benefits. However, we could not
determine whether the data being collected and reported are accurate and effective since
most CEC projects are just getting started.
9.2.2 IOUs
The IOUs report on project benefits in their project close out reports. Their internal project
teams, with support from internal IOU technical staff, gather and summarize data and
information related to project benefits and metrics in the reports. They initially identify the
types of metrics that are relevant in their Investment Plans.
We reviewed all IOU projects that were complete at the time we drew our sample (since
we included all completed projects in our sample). At that time, PG&E had completed six
projects and SCE had completed three, while SDG&E had not completed any projects.66
The project close out reports included the following sections:
• Project objective
• Project scope
• Major task summary
• Results
• Issues addressed or knowledge gaps filled
• The value proposition for the project
• How the project meets EPIC program metrics
All nine of the IOU project close out reports included information about potential project
benefits. PG&E’s reports contained a subsection called “Project Metrics” within a Key
Accomplishments and Recommendations section,67 while SCE had a section called
“Metrics” that followed Project Results. The potential benefits were broadly characterized
and sometimes cross-reference another section of the report. A typical example of
66 Note that we checked in June of 2017 to see if SDG&E had released any project close out reports, and did
not find any completed projects or close out reports to review for this evaluation.
67 PG&E also has a plan for capturing its corporate benefits (referred to as core values) via a Preliminary
Project Benefits Plan, which is included in the Business Plan for each project. PG&E’s core values include
public safety, employee safety, reliability, operational excellence, customer focus, delighted customers, and
environmental leadership

economic benefits is as follows (from PG&E project 1.01 – Energy Storage for Market
Operations):
• Potential energy and cost savings

o Nameplate capacity (MW) of grid-connected energy storage
• Economic benefits
o Maintain/Reduce operations and maintenance costs
• Other Metrics
o CAISO Non-Generator Resource financial statements
• Identification of barriers or issues resolved that prevented widespread deployment
of technology or strategy
o Description of the issues, project(s), and the results or outcomes
o Increased use of cost-effective digital information and control technology to
improve reliability, security and efficiency of the electric grid
o Dynamic optimization of grid operations and resources, including appropriate
consideration for asset management and utilization of related grid operations
and resources, and services
o Identification and lowering of unreasonable or unnecessary barriers to adoption
of smart grid technologies, practices, and services
• Effectiveness of information dissemination
o Number of information sharing forums held
• Adoption of EPIC technology, strategy, and research data/results by others
o EPIC project results referenced in regulatory proceedings and policy reports
For each metric category, there is a cross-reference to the project results that provides more
information. For example, for “potential energy and cost savings”, there is a cross-
reference to Tables 4-1 and 4-2. Table 4-1 from the project close out report is shown below.

Table 43: Excerpt from PG&E Project 1.01 Results

SCE addresses metrics in its closeout reports in a similar fashion. Project metrics are
acknowledged and discussed, but not always quantified. For example, SCE’s closeout
report for its Outage Management and Customer Voltage Analytics project (from EPIC 1),
discusses the project’s technical details in specifics, but presents EPIC metrics in purely
qualitative terms. Overall, we found details on metrics to range from a single sentence to
quantification for a variety of core metrics relevant to a project.
As shown, the report includes detailed technical information on project results. However,
it stops short from quantifying benefits associated with the project. It provides useful
inputs that one could possibly use to quantify the benefits, but the reports are more
focused on sharing project results and lessons learned, which would be useful for their
own utility and other utilities and stakeholders. However, the CPUC’s reason for requiring
that administrators identify and track metrics was to “propose metrics… against which the
investment plan’s success should be judged.”68 It is clear from our review that the IOUs
are not taking the necessary steps to use the data and information they have gathered, and
possibly supplement it by gathering additional data, to report on the success and benefits
associated with their projects in the broader realm of how these projects are contributing
to the over-arching Investment Plan goals.
9.3 Results Dissemination

The following section presents two related topics, how EPIC tracks project level results,
and how results and other project information and knowledge are disseminated to
relevant stakeholders and market actors. In this section, we primarily report on results for
the CEC and IOUs (combined) due to the difference in administrative approach we noted
at the initial stages of the evaluation. However, we examined each of the four
administrators’ processes and projects, and where we noted differences across the three
IOUs, we discuss those differences. Note that these comparisons were limited by the size
of the project sample for each IOU (which reflects the relatively smaller portion of the
EPIC budget that the IOUs receive as compared to the CEC).69
The success of research, development and deployment programs such as EPIC depends
partly on the extent to which the program participants are able to transfer knowledge to a
wider audience that can use or advance that knowledge. This includes communicating
research results and developing relationships with the appropriate audiences and
stakeholders, including industry experts and other potential technology adopters. The
diffusion of project information is critical if stakeholders beyond the projects are to adopt
technologies and tools developed and take innovations from being a nascent technology to
commercialization. There are two key components to successful knowledge transfer:
68 Decision 12-05-037, p. 8., 2012

69 The project sample included two PG&E projects, seven SCE projects, and four SDG&E projects.

• Identifying a variety of information sharing channels and activities to successfully
transfer knowledge to the appropriate audience; and
• Developing relationships and networks that include industry experts and
influential knowledge disseminators who can create pathways for knowledge
dissemination.
To describe and assess EPIC project efforts to disseminate knowledge and develop
appropriate relationships and networks, the evaluation team conducted a network
analysis. The goal of the network analysis is to clarify the composition of relational
networks and knowledge pathways developed through the Program. These networks are
created as a result of the outreach, collaboration, marketing and information dissemination
activities of the CEC and IOU administrators, CEC CAMs, CEC grantees, IOU project
managers and IOU contractors. To capture all of these nuanced relationships, the network
analysis was conducted at both the project level and the program level. We also reviewed
the processes the administrators use to disseminate results.
This section is organized around the following topics, linked to specific activity and output
boxes in the research area logic models (see Table 44).
Table 44: Network Analysis Approach and Linkage to EPIC Logic Model
Logic Model Box
Topic Text Metrics
Knowledge Disseminate Results • Number of CEC projects with knowledge transfer plan
Dissemination developed or funded
Activities Results delivered • Number of projects presenting at workshops
through websites,
workshops, • Number of projects presenting at conferences
conference • Number of projects with published peer reviewed journal
presentations, & articles
professional • Number of projects with other published articles or collateral
networks
• Number of unique organizations and individuals to which
results were disseminated via listservs
• Number of websites used with project results or information
Network Inter-organizational • Diversity of organizations in networks
Composition and interpersonal • Overall network characteristics (cohesion and reciprocity)
and Activities networks developed
• Number of formal agreements to share information and
resources
• Number of informal agreements to share information and
resources
• Types of information and resources shared

9.3.1 Knowledge Dissemination Activities
In this subsection, we describe the processes the administrators use to disseminate project
knowledge. Project teams engage in a variety of knowledge exchange activities. In
addition to EPIC requiring each team to provide a final close out report and present
interim and/or final results at the EPIC symposia, project teams may disseminate project
results in other formal and informal ways. Examples of this include presenting at
workshops and conferences, publishing articles in peer reviewed journals, publishing
articles in trade journals and other publications, sending project results to stakeholders via
listservs, publishing results or updates on websites, and through formal and informal
communication arrangements with other organizations.
CEC
The CEC has an extensive process intended to disseminate results that includes:
Formal project reporting:
• Projects require final reports, which the CEC publishes.

• The CEC provides summaries of projects in its Annual Reports and features projects
in the CEC Innovation Showcase.
• Projects require the completion of a knowledge transfer plan, which the grantees
are expected to implement to disseminate project-related knowledge and results
through channels such as published articles, presentations at conferences and
workshops, and dissemination of information on grantee websites or via social
media platforms.
EPIC-wide information sharing:
• The CEC and IOU administrators collaborate at the program and technology level,
and these discussions include project-specific information sharing.
• The Media and Public Communications Office conducts general outreach about
EPIC and its value to California via social media platforms, with additional
placement in technical journals and traditional news media.
Grantee and TAC information dissemination:
• Grantees may conduct additional outreach independent of the grant, as they are
often experts on their topic areas, deeply involved in their fields, and publish
papers and present at technical events on their work.
• Similarly, members of project TACs are often well placed in their fields and
sometimes chosen to facilitate diffusion of project-related knowledge; hence, for
some projects, they may become conduits for information sharing as well.

Knowledge Transfer Plans
To determine if project teams are adhering or planning to adhere to the requirement to
develop knowledge transfer plans, we asked the CEC CAMs and CEC grantees if they
have completed, are preparing, or have funding set aside to complete a knowledge
transfer plan.
Table 45 summarizes the status of the CEC project knowledge transfer plans. Overall, five
projects have completed the knowledge transfer plan, including the one completed CEC
project. The remaining projects either have set aside funding for the plan or are preparing
the plan. This suggests that all project teams are adhering to this requirement.
Table 45: CEC Project Knowledge Transfer Plan Development (n=32)

CEC
Applied
R&D MF TD&D Total
Knowledge Transfer Plan Complete 2 2 1 5
Knowledge Transfer Plan In Progress 0 0 4 4
Knowledge Transfer Plan Funded - Not Started 6 5 12 23
Total 8 7 17 32
* One CEC project is excluded because the interview was incomplete.
Technical Advisory Committees

For the CEC EPIC projects, the TACs provide an important source of project guidance and
connection to technical and market expertise, and also provide an avenue for knowledge
dissemination. For the 32 CEC projects in our sample, we requested the list of TACs for
projects that have reached the stage where these groups have been formed. In total, we
received information on TAC composition from 15 projects. Across the 15 projects, there
were 178 TAC members, an average of just under 12 members per project TAC. The TACs
range in size from four members at the smallest to 24 members at the largest. As we would
expect, the TACs are made up of a variety of individuals selected for their specific
expertise germane to the project, and TAC members come from a diverse range of
organizations from government, research institutes, technology manufacturers,
universities, utilities and industry organizations. The TAC members are engaged in the
project on a voluntary basis. Administrators and grantees from these projects were
generally positive about their interactions with the TAC members when we discussed
them during interviews, although some project teams did note that they can have
difficulty scheduling meetings with TAC members, especially for larger teams. Some
teams also noted that the voluntary nature of the role also made it challenging to engage
TAC members at times.

CEC Media Office
The CEC Media and Public Communications Office (the Media Office) also plays an active
role in informing the public about EPIC and the Program’s benefits, using individual CEC
projects as examples of the Program’s role in providing societal and ratepayer benefits. We
asked the project CAMs associated with the 32 CEC projects in our sample if they have
utilized the Media Office and, if so, how they reported the helpfulness of their
involvement on a five-point scale from not at all (1) to extremely (5). As shown in Table 46,
75 percent of projects have used the Media Office, and the average helpfulness score was
3.6.
To date, the Media Office has focused on communicating about EPIC and been less
involved in project-specific outreach to stakeholders and technical audiences, however.
Examples of Media Office involvement from CAMs tended to emphasize assistance in
holding project-related events, such as ribbon-cuttings or other events associated with
projects in which public officials or citizens might be interested.
Table 46: Usage and Helpfulness of the CEC’s Media and Public Communications
Office (n=32)
Have Have Not Five-Point
Used Used Total Rating Average
Applied R&D 7 1 8 3.2
TD&D 11 6 17 3.7
Market Facilitation 6 1 7 4
Total 24 8 32 3.6
IOUs
The IOUs also report on how they plan to disseminate project results in their project close
out reports. Similar to how the IOUs treat project benefits, they rely on their internal
project teams with support from internal IOU technical staff to determine and document
how they plan to share project results.
As mentioned previously, we reviewed all IOU projects that were complete at the time we
drew our sample, for a total of nine project close out reports. PG&E’s reports contained a
subsection called “Technology Transfer Plan for Applying Results into Practice” within a
Key Accomplishments and Recommendations section, while SCE has a subsection called
“Technology / Knowledge Transfer Plan” that is part of a Project Results section.
Technology Transfer Plans

Seven of the nine reports indicated where information about a project had been or would
be presented. Five of six PG&E reports had such information in the Technology Transfer
Plan subsection:

• Project 1.01 identified a total of six presentations, three conferences, one meeting
and two EPIC workshops.70
• Project 1.09 A identified two conferences.
• Project 1.09 B identified three conferences.
• Project 1.18 identified two conferences.
• Project 1.24 identified five conferences, including three that were pending.
To illustrate a typical PG&E technology transfer plan, for Project 1.18, which evaluated the
ability of commercial vendors to use PG&E smart meter data to disaggregate residential
customer energy usage, we list the conferences at which PG&E either presented or
planned to present the results:
• Peak Load Management Alliance (PLMA) in San Francisco, CA - April 19, 2016
• Demand Response and Smart Grid National Town Meeting (abstract pending
approval) in Washington D.C. - July 11-13, 2016
• Western Load Research Association (WLRA) (abstract pending approval) in
Chicago, IL - Fall 2016
• Association of Energy Services Professionals (AESP) (abstract pending approval)
Summer Conference in Chicago, IL - August 16-18, 2016
• National Conference - February 13-16, 2017
Two of the three SCE reports had such information:
• Project 13-086 identified two EPIC workshops.

• Project 14-028 identified two presentations, one at SCE and another at Duke Energy.
While the number of projects we examined is small, we note that at this stage at least,
PG&E is sharing its project results at significantly more conferences as compared to SCE.
SDG&E has not yet completed a project so has not released a project close out report.
Analysis of Project Dissemination

We also assessed how well the administrators disseminate information about projects
based on interviews we conducted with project management teams, as well as the
administrators’ technology and knowledge transfer plans (including the IOUs’ as
documented in project close out reports). We conducted analysis to report on quantitative
measures of EPIC project dissemination.
70The EPIC workshops are required by the CPUC (though the administrators are not required to report on
every project at those workshops).

We start with looking at the number of projects for which project teams have presented
information. It is important to note that not all projects have reached the point where they
have sufficient project results to present. Only one CEC project (out of 32) in our sample
was completed, for example, and only 9 of the 21 IOU sampled projects were completed at
the time we conducted our analysis.
Project Information Dissemination

Table 47 presents the number and percent of projects with teams that reported presenting
project information or results at workshops, conferences or through publications. As it is a
required activity, a large proportion of projects have been presented in some form at an
EPIC Symposium, with 53 percent of total CEC projects represented and 86 percent of total
IOU projects. In addition to the required EPIC symposia, approximately 40 percent of CEC
projects and 50 percent of IOU projects have been presented at outside workshops.
Workshops mentioned include National Electric Energy Technology and Application
Research Committee (NEETRAC), the Demand Response and Smart Grid National Town
Meeting Forum, and the Non-Intrusive Load Monitoring International Workshop. Fewer
CEC projects have been presented at conferences (19%) in comparison to IOU projects
(52%), which in part is due to CEC projects being launched later than the IOUs’ projects
(with 8 of 21 IOU sampled projects completed). Conferences mentioned include the
Electric Power Research Institute (EPRI) and the Institute of Electrical and Electronics
Engineers conferences, the Energy Storage North America Conference, the Electric Vehicle
Conference, the Advanced Research Projects Agency-Energy Program (ARPA-E) Annual
Conference, and the Western Load Research Association Conference among others.
Overall, 66 percent of projects have been presented at workshops or conferences or have

been published, which is extensive since so few projects have been completed. Information
about a significant number of projects has been shared at workshops or conferences
besides those required by EPIC.

Table 47: Proportion of Projects Engaged in Dissemination Activities
(Workshops, Conferences and Publications) (n=53)
CEC IOU*
Applied CEC
R&D MF TD&D Total TD&D Total
(n=8) (n=7) (n=17) (n=32) (n=21) (n=53)
Workshops
EPIC Workshops / Symposia 3 (38%) 3 (43%) 11 (65%) 17 (53%) 18 (86%) 35 (66%)
Other Outside Workshops 3 (38%) 1 (14%) 9 (53%) 13 (41%) 11 (52%) 24 (45%)
Conferences 2 (25%) 1 (14%) 3 (18%) 6 (19%) 11(52%) 17 (32%)
Journal Articles 2 (25%) 0 3 (18%) 6 (19%) 2 (9%) 7 (13%)
Articles in Other Publications 1 (13%) 1 (14%) 2 (12%) 4 (13%) 4 (19%) 8 (15%)
Total 3 (38%) 3 (43%) 11 (65%) 17 (53%) 18 (86%) 35 (66%)
* There was no substantial or significant difference in dissemination activity participation between the
individual IOUs.
Websites and Listservs

In addition to in-person or traditional print forms of knowledge dissemination,
information is also disseminated by some projects via email listservs and through
publication of project information on CEC, IOU and other websites. Listservs in particular
are a very basic first step for disseminating project and program information. The CEC
and IOU administrators maintain extensive listservs with a large number of subscribers
that receive program level information including information about program solicitations
and annual reports, as well as information on specific projects when appropriate.
Table 48 presents the number and percent of projects with teams that reported presenting
project information through listservs and websites. This could include sending
information through websites and listservs maintained by an IOU or the CEC, or else
using those managed externally by project team members. All projects are maintaining
some project information on either the CEC’s EPIC website (the Energy Innovation
Showcase at innovation.energy.ca.gov/) or the respective IOU administrator website. In
addition, approximately 53 percent of CEC projects maintain information on an external
website—typically on the grantee’s website (for example, the Lawrence Berkeley National
Laboratory). The interviewees reported that the IOU projects do not have an online
presence beyond the IOU websites.
More than half of all projects have disseminated results via a listserv or website. With only
10 of 53 projects were completed (20%), these data indicate that the administrators have
shared information about many projects even before they are completed. Within the IOU

projects, PG&E projects made up the majority of projects with 6 of 10 sampled projects
disseminating information via the IOU listserv, while SDG&E had one project and SCE
had no projects with information disseminated via a listserv. This is in part due to PG&E
having more completed projects than SCE or SDG&E.
Table 48: Projects That Have Disseminated Results via Listserv or Website (n=53)
CEC IOU
RD&D MF TD&D CEC Total TD&D Total
(n=8) (n=7) (n=17) (n=32) (n=21) (n=53)
CEC Listservs 2 (25%) 6 (86%) 6 (35%) 14 (44%) 3 (14%) 17 (32%)
IOU Listservs - - - - 7 (33%) 7 (13%)
External Listservs - 2 (29%) 3 (18%) 13 (41%) - 13 (25%)
CEC EPIC Website 8 (100%) 7 (100%) 17 (100%) 32 (100%) - 32 (60%)
IOU Website - - - - 21 (100%) 21 (40%)
Other Websites 3 (38%) 7(100%) 7(41%) 17 (53%) - 17 (32%)
Total 8 (100%) 7 (100%) 17 (100%) 17 (53%) 21 (100%) 35 (66%)
Listserv Analysis
As noted, an important avenue of knowledge dissemination is through the CEC and IOU
listservs. The CEC and IOU administrators deliver annual reports, grant solicitations and
other information via internal listservs. In addition, some projects also use these listservs
to disseminate written information about their projects (44 percent of CEC projects and 33
percent of IOU projects reported delivering information though these listservs). There are
as many as 80 listservs between the CEC and IOU administrators, but four CEC listservs
and one aggregated listserv for each IOU are used most frequently. We obtained the
domain names from each subscriber from these listservs from the administrators. From the
CEC, we obtained the opportunity listserv that disseminates information about EPIC grant
opportunities, the EPIC listserv that disseminates general information about the Program,
the news release listserv that delivers project and program-specific news releases from the
CEC Media and Public Communications Office, and the research results listserv that is
used to deliver project-specific results, including interim findings and final close out
reports. From the IOUs, we obtained each of their aggregate listservs that they use to
disseminate the Annual Reports.
Details of these listservs are presented in Table 49. We are able to present counts of total
subscribers and unique organizations; however, for confidentiality reasons, we were
unable to obtain full email addresses so cannot provide more detailed analysis of listservs.

The EPIC listservs disseminate information to a very wide array of individuals and
organizations with several thousand recipients. The IOU listservs are more limited, with
subscribers numbering in the hundreds. Both sets of listservs consist of a diverse range of
recipients—including government officials; utility staff; investors; California ISO staff; and
individuals from the business community, environmental groups, the clean energy
industry and other industry associations—and there is a large amount of crossover
between the IOU and EPIC listservs.
Table 49: CEC Annual Report Listserv Summary

Total Number of Total Number of
Subscribers Unique Organizations
CEC
EPIC 1,355 672
Opportunity 4,926 2,228
News Releases 1,446 810
Research Results 2,275 970
IOU
SDG&E 337 148
PG&E 346 121
SCE 395 134
Based on the evaluation team assessment, the four-administrator model for EPIC may
cause some barriers and limitations to information dissemination, as there is no program-
wide communications vehicle or central repository of project information. While the
administrators do collaborate on the required EPIC symposia, outreach is conducted
separately even though there is possible overlap in stakeholder interest across
technologies investigated by the administrators. Similarly, the CEC Media Office focuses
on CEC projects and does not disseminate any information about IOU projects. Each
administrator uses its own website and listserv(s) to distribute information. While this is
appropriate, it highlights some limitations to the four-administrator model.
9.3.2 Network Composition and Activities

This subsection focuses on the organizations and individuals who may be receiving
information about EPIC projects. In addition to the above information dissemination
activities and avenues, the second key component to successful knowledge transfer is
development of relational networks with appropriate audiences and stakeholders. To this
end, the EPIC project teams may engage with organizations and individuals outside their
project teams to exchange information. The CEC project teams formally engage technical

experts through the TAC that is formed for each project (and in limited cases, a PAC). For
both IOU and CEC projects, EPIC project teams (e.g., the CAM, IOU administrator,
grantees and members of the TACs) make formal and informal connections with other
organizations beyond the project team to share knowledge and experience. We conducted
network analysis to independently validate and report on quantitative measures of EPIC
networking.
These information exchanges can occur in both directions, sending information from the
project to outside organizations and individuals, and receiving information from outside
organizations and individuals. These relationships can be formal or informal, and the
knowledge exchanged can be either directly related to the EPIC project or related to other
projects but with information germane to the EPIC project. To understand the magnitude
and composition of networks of knowledge sharing organizations and individuals, as well
as the types of information being shared and the level of formality of information sharing,
we asked project teams for the following, for the CEC and IOU project sample:
• A list of up to 15 of the most important organizations both inside and outside

California with whom they are sharing information, collaborating, or sharing
resources (expertise, equipment, etc.) during the implementation of an EPIC project;
• For CEC projects, a list of the TAC members;
• The level of frequency with which information is shared;
• If information shared is related to the EPIC project, related to other projects with
similar topics, or both; and
• If the relationship is formal or informal.
We received networking information for 36 of the 53 sampled IOU and CEC projects (25
CEC projects and 11 IOU projects). Many of the remaining 17 projects are not far enough
advanced for extensive networks or information sharing to be established. The following
exhibits provide a description of the composition of the external networks, the level and
frequency of information sharing, and the formality of the information sharing
arrangements.
Project Networks
Table 50 presents the total number and types of organizations reported by each of the 36
project teams that responded. We investigated each organization mentioned and
categorized it as either a public or private organization, and as one of ten organization
types. Note that the government category includes military and civilian government
agencies, as well as the California regulatory agencies (the CEC and the CPUC). The
manufacturer category includes both manufacturers of hardware and software
technologies.

On average, the CEC engages with 19 organizations per project, while the IOUs engage
seven. The types of organizations the CEC and IOUs engage with differ to a great extent.
Ninety percent of organizations that the IOUs engage with are private compared with 49
percent for CEC TD&D projects. This distinction may explain why the stakeholders we
interviewed had not heard anything about completed EPIC projects (which at the time of
our interviews, consisted mostly of IOU projects). The IOUs’ networks consist mainly of
manufacturers, private companies and other utilities, with very few government/policy-
making organizations included.
Among the CEC research funding areas, there was also some difference between the
public and private organization composition of networks, with TD&D projects more likely
to engage with public entities than Market Facilitation or Applied R&D projects. On
average, CEC projects engage with more organizations (18) than IOU projects (7). Note
that this includes CEC engagement with the project TACs, each of which includes on
average eight organizations.
We also investigated differences across the IOU administrators. There were differences in
the number of organizations that projects have engaged with across the IOUs in our
analysis. SDG&E engaged with the most outside organizations per project having, on
average, ten organizations per project in their network, compared with an average of six
organizations for PG&E and four for SCE. Note that the sample of projects for each IOU is
small, so this may not be reflective of the entire portfolio of IOU projects.

Table 50: Types of Organizations Involved in Project Networks
(n=35 Projects, 338 Organizations)
CEC IOU
Applied
R&D MF TD&D Total
Projects (n=7) (n=8) (n=9) (n=24) IOU (n=11)
Total Organizations 132 137 167 436 80
Public vs Private Organizations
Private 76 (58%) 80 (58%) 68 (41%) 224 (51%) 72 (90%)
Public 56 (42%) 57 (42%) 99 (59%) 212 (49%) 8 (10%)
By Type
Government 22 (16%) 25 (18%) 63 (38%) 110 (25%) 7 (9%)
Government Research Labs 9 (7%) 4 (3%) 8 (5%) 21 (5%) 1 (1%)
Implementer 7 (5%) 5 (4%) 8 (5%) 20 (5%) 1 (1%)
Innovation Incubation
9 (7%) 5 (4%) - 14 (3%) -
Organization
Manufacturer 10 (7%) 13 (9%) 12 (7%) 35 (8%) 20 (25%)
Non-Profit Industry Association 28 (20%) 30 (22%) 9 (5%) 67 (15%) 4 (5%)
Other Business Customer 8 (6%) 5 (4%) 15 (9%) 28 (6%) 4 (5%)
Private Research & Consulting
18 (13%) 17 (12%) 19 (11%) 54 (12%) 25 (31%)
Company
University 13 (9%) 17 (12%) 9 (5%) 39 (8%) -
Utility/CAISO/Utility
18 (13%) 16 (12%) 25 (15%) 59 (14%) 18 (23%)
Associations
Mean Number of Organizations
19 17 19 18 7
per Project
Figure 18 below presents the proportion of private versus public network members for
each administrator by project type. This figure illustrates the major difference between the
CEC and the IOUs that was introduced in the previous exhibit.

Figure 18: Project Network Composition by Administrator
CEC$.$Applied$R&D$ 58%$ 42%$
CEC$.$MF$ 58%$ 42%$

Private$
Public$
CEC$.$TD&D$ 41%$ 59%$
IOU$.$TD&D$ 90%$ 10%$
0%$ 20%$ 40%$ 60%$ 80%$ 100%$
Frequency of Information Exchange

Frequency of information exchange between project teams and the network of outside
organizations also varies between IOU and CEC projects, as reported during project
interviews (Figure 19). The reported frequency of information exchange suggests that CEC
TD&D projects tend to exchange information more frequently with outside entities (57%
reporting always or often) than IOU TD&D projects (35% reporting always or often). We
found no significant or substantial difference between IOU administrators in terms of
frequency of information exchange with their network.

Figure 19: Frequency of Information Exchange between Projects and Network
Organizations (CEC n=24; IOUs n=11)
IOU
TD&D 25% 40% 21% 14%
Never
TD&D 15% 10% 28% 35% 12%
Rarely
Occasional
ly
CEC
Applied R&D 8% 11% 59% 18%

Often
MF 18% 18% 32% 32%
0% 100%
Network Relationships
Lastly, we examined the formality of relationship agreements between the EPIC project
teams and their network of outside organizations. We asked the CEC CAMs and IOU
project managers to tell us which projects in our sample had formal information sharing
agreements with the external organizations, which we defined as having a written contract
agreeing to share information. As shown in Table 51, approximately 20 percent of outside
organizations engaged by IOU projects have such formal agreements in place, while 57
percent of outside organizations (including TACs) engaged by the CEC have formal
agreements in place. CEC projects have, on average, a higher number of organizations per
project with a formal information sharing agreement in place than IOU projects (9.7
organizations on average versus 2.0 organizations, respectively). We compared the
formality of relationships across the three IOUs and we found no significant difference.

Table 51: Formal Versus Informal Relationship Agreements
(n=35 projects, 338 organizations)
CEC IOU
Applied
R&D MF TD&D Total IOU
(n=6*) (n=8) (n=9) (n=23) (n=11)
Total Organizations 125 136 166 394 80
Mean Number of
Organizations per Project
Formal Relationship 11.6 6.8 11 9.7 2.0
Informal Relationship 9.1 11.6 7.5 9.3 6.1
Proportion of Total Organizations By Type
Formal Relationship 70 (56%) 55 (40%) 98 (59%) 223 (57%) 16 (20%)
Informal Relationship 55 (44%) 81 (60%) 68 (41%) 171 (43%) 61 (76%)
No Response 0 0 1 1 3
*One RD&D project in this sub-sample was in a very early stage, and the organizations in the project
network were unable to report on the type of sharing that was occurring or would occur.
While most projects are still in the early to mid stages with only 19 of 177 project
completed, this analysis of knowledge dissemination activities and the relational networks
of each of the sampled interview projects suggests that projects are actively engaged in
developing networks of stakeholders and other market actors. Furthermore, the wide
range of entities that is already engaged in projects suggests that the projects are well
positioned to lead to wide knowledge dissemination once they are completed. While
projects appear to be on a good trajectory to lead to knowledge dissemination, it is too
early in the life of the program to collect more instances of concrete knowledge benefits
such as patents, journal articles and existence of follow-on research. (We discuss these
potential EPIC project benefits in Section 10.) It is also premature at this time to assess if
these activities will be sufficient to encourage further research and technology adoption
after the EPIC projects are completed.

The best practices literature recognized the importance of identifying metrics and systems
for tracking progress. Six of the seven peer RD&D programs use formal metrics to track
project performance. Although the metrics of the peer RD&D programs vary based on
their stated focus and technology, the interviewees generally reported that they do track
such indicators on a regular basis. Among the primary peer RD&D programs, common
project-based metrics include patents, copyrights and publications. Metrics tracked across
program portfolios typically include indicators such as progress toward

commercialization, identifying potential funders and investors, conducting sales pitches,
sales revenues and leveraged funding.
All seven of the peer RD&D programs stated that projects are required to report regularly
on their progress. For example, projects funded through the New York State Energy
Research and Development Authority's (NYSERDA’s) T&MD Program provided monthly
or quarterly performance reports, which the assigned Project Manager uses to track
progress on milestones identified in the Statement of Work. Many T&MD projects also
receive support through the Entrepreneurs-in-Residence (EIR) program, a resource
available to companies engaged in commercialization activities (described above under
Program Management and Administration). Through these interactions, the main points
of contact between the EIR program and the project often develop close relationships, and
the EIR advisor is often privy to project status updates on an ongoing basis. Interviewees
representing other peer RD&D programs similarly reported regular interaction between
program administrators and grantees through written reports (monthly, quarterly or
annual), on-site visits or phone calls.
Five of the seven peer RD&D programs, including all four of the primary peer RD&D
programs, use an external evaluator to assess the program’s performance. In addition to
internal tracking reporting mechanisms, interviewees from the primary peer RD&D
programs stated that they have used third-party evaluations to aggregate, assess and
report on program-wide outcomes. The National Academies of Sciences recently
published assessments of the DOE's Small Business Innovation Research (SBIR) and Small
Business Technology Transfer (STTR) Programs.71,72 NYSERDA also has a separate
evaluation group, which works with a third-party to evaluate the T&MD Program’s
effectiveness.73 ARPA-E recently completed an independent evaluation, which is
forthcoming. Interviewees generally indicated that these external efforts help provide
objective insights regarding the programs' performance and are useful for reporting to
external program stakeholders.
71 National Academies of Sciences, Engineering, and Medicine. STTR: An Assessment of the Small Business
Technology Transfer Program. Washington, DC: The National Academies Press, 2016.
https://www.nap.edu/catalog/21826/sttr-an-assessment-of-the-small-business-technology-transfer-
program
72 National Academies of Sciences, Engineering, and Medicine. SBIR/STTR at the Department of Energy.
Washington, DC: The National Academies Press, 2016. https://www.nap.edu/catalog/23406/sbirsttr-at-

the-department-of-energy
73 New York State Energy Research and Development Authority. “NYSERDA Technology and Market
Development Program Semiannual Report through June 30, 2016, Final Report,” 2016.
https://www.nyserda.ny.gov/-/media/Files/Publications/PPSER/NYSERDA/tmd-report-2016jun.pdf

In addition to providing monthly reports to their CAM, EPIC grantees are required to
submit annual and final reports. This level of reporting is similar to the peer RD&D
programs. Like the peer RD&D programs, the metrics reported by EPIC grantees depend
on the focus of the individual projects, which vary by technology, research area and other
factors. To date, EPIC has not provided a summary of findings across the entire portfolio,
due, in part, to the wide-ranging research and technology areas. This evaluation is the first
program-wide assessment, and since so few projects have been completed, we were not
able to make a definitive assessment of project results and benefits and effectiveness of
processes to measure and disseminate them. Table 52 below provides a comparison in
these areas between EPIC and the peer RD&D programs.
Current
EPIC
Formally track program-wide metrics EPIC does not EPIC should develop a set of
do this. common, high-level metrics
based on technology type and/or
policy area.
Require regular progress reports (monthly or EPIC does this. EPIC currently requires regular
quarterly) project status reports.
Use third-party evaluation to assess program EPIC does this. The CPUC has commissioned
impact its first evaluation of EPIC.

10 Policy Alignment and Project Impacts
This section examines the wider impact of the EPIC projects beyond the program
administration, investment planning, project selection, and project assessment topics
discussed in previous sections. Specifically, this section addresses the fifth and final
research topic area:
• Policy Alignment and Project Impacts: Looking beyond project- and

administrator-specific considerations, what impacts does the program overall have
in a wider context? This category seeks to evaluate EPIC’s place in the broader
innovation and policy landscape.” We base these findings on the results from
interviews with sampled project administrators, grantees and vendors, as well as
detailed reviews of project proposals, benefits questionnaires, close-out reports and
other documentation.
This section assesses the performance of Applied R&D, TD&D and Market Facilitation
projects across five important big-picture areas that are sub-categories of the project
impacts and policy alignment research topic area:
1. Policy Alignment – we explored how well EPIC projects are integrated into the
broader innovation74 and policy landscape and support the key overarching policy
goals of the State of California.
2. Portfolio Diversity – we examined projects by research area to determine how
diverse the EPIC portfolio is in terms of the technologies it is addressing, the extent
to which those technologies are expected to become commercialized, and the timing
of project completion. These measures of diversity are related to the assessment of
the portfolio’s alignment with the state’s energy policy objectives.
3. Public Interest Focus – we reviewed the extent to which EPIC projects meet the
mandatory guiding principle to provide public benefits and benefits to electricity
ratepayers.
4. Program Equity – we assessed the extent to which EPIC funds have been invested
in disadvantaged communities, which is one of the policy areas on which the CEC
has explicitly focused.
5. Indicators of Project Performance – based on our review of project data and
interviews with CEC CAMs and grantees and IOU project managers and vendors,
74A product innovation is defined by the Organization for Economic Co-operation and Development
(OECD) as “. . . the introduction of a good or service that is new or significantly improved with respect to its
characteristic or intended uses. Product innovations can utilize new knowledge or technologies or be based
on new uses or combinations of existing knowledge or technologies." X. Tinguely. The New Geography of
Innovation: Clusters, Competitiveness and Theory. 2013.

we provided an early assessment of whether projects either are or are likely to
produce their expected outcomes and impacts.
As mentioned previously in Section 6, throughout the report we have tended to present

results for the CEC and all three IOUs combined due to the difference in administrative
approach that we noted at the initial stages of the evaluation. However, we examined each
of the four administrators’ processes and projects, and found no significant or substantial
differences across the three IOUs in the topics addressed in this section. Note that those
comparisons were limited by the size of the project sample for each IOU, which reflects the
relatively smaller portion of the EPIC budget that the IOUs receive as compared to the
CEC.
10.1 Policy Alignment

EPIC is a key element of the State of California’s efforts to meet its energy policy goals.
Specifically, the Program is designed to support innovation in clean energy technologies
and strategies through public interest investments that will benefit California ratepayers,
and drive California toward meeting its ambitious clean energy and climate policy goals
while maintaining an affordable, safe and reliable energy supply. According to the
administrators’ EPIC 2 Investment Plans, the Program supports the following policy areas:
• Greenhouse Gas (GHG) Reduction Legislation (SB 350) establishing clean energy,
clean air and GHG reduction goals for 2030 and beyond;
• The Renewable Portfolio Standard requiring retail sellers and publicly owned
utilities to procure 50 percent of their electricity from eligible renewable energy
resources by 2030;
• Grid Infrastructure (Smart Grid) upgrades necessary to the state’s existing
transmission and distribution systems. SB17 mandates implementing and planning
a smart grid to improve efficiency, reliability, economics and sustainability of
electricity services;
• Energy Efficiency and the Loading Order that provides a foundation for energy
resource procurement policies, and consists of decreasing demand through energy
efficiency and demand response, and meeting new generation first with renewable
and distributed generation resources;
• Optimal portfolio integration of Distributed Energy Resources to achieve the
state’s GHG goals and meet the challenge of renewable integration; and
• Governor Brown’s Clean Energy Jobs Plan to produce a half a million clean energy
jobs in the next ten years.
In Section 7, we provided a portfolio-level analysis of project alignment with the above

policy goals indicating that to a large extent, the EPIC program portfolio addresses each of
the California policy goals. At the program-area level, we see similar trends across the

policy areas, as shown in Figure 20. TD&D project funding is largely aligned with
addressing issues that support the integration of distributed energy resources as well as
the loading order and GHG reduction policies. We do see a differing emphasis in project
type between CEC and IOU TD&D projects, with CEC TD&D projects more focused on the
loading order, GHG reductions and energy efficiency, while IOU projects have a greater
emphasis on grid infrastructure (Smart Grid), and meeting the Renewable Portfolio
Standard. Applied R&D has a similar alignment, with somewhat more funding allocated
to these areas. Market Facilitation projects most closely align with distributed energy
resources, loading order, GHG reductions, energy efficiency and clean energy jobs.
Relatively fewer Market Facilitation projects address the Smart Grid research area.
Figure 20: Research Area Alignment with California Policies
Distributed Energy Resources
Loading Order
GHG Reductions
Energy Efficiency
Clean Energy Jobs
Smart Grid
Renewable Portfolio Standard
Energy Storage
Electric Vehicles
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
IOU - TD&D CEC - TD&D CEC - R&D CEC - MF
Our analysis shows that many projects effectively address more than one of these
principles and policy goals. In particular, projects that aimed to increase end-user
adoption of energy efficiency measures or overcome technical barriers to the end-user
adoption of distributed generation also addressed other areas such as reducing GHG
emissions and meeting requirements of the Renewable Portfolio Standard (RPS).
Additionally, the Regional Energy Innovation Cluster projects align with numerous policy
objectives including GHG reduction, the loading order and clean energy jobs. However,
other projects, such as a study of the effective placement of fast-charging stations, may
have strong GHG reduction, health and safety benefits (associated with air pollution
reduction) and other environmental benefits, but they do not necessarily address other
policy areas such as energy use reductions or the loading order.

It is not surprising to find some differences between the CEC and the IOUs with respect to
their TD&D projects. For example, the IOUs have a greater focus on projects that address
smart grid, RPS, energy storage and electric vehicle (EV) policies. This makes sense since
these types of projects are the ones that would have the greatest impact on the electric
systems of the IOUs.
In Section 11, we recommend that the CPUC establish priorities among the many energy
policy goals. This analysis serves to provide a current snapshot of how EPIC projects are
supporting policy. We also recommend that the administrators categorize projects by
technology areas and policy goals to support future analyses and project reporting to
ensure priorities are being adequately addressed.
10.2 Project Diversity

To achieve EPIC’s objectives and support numerous energy policy goals requires a diverse
portfolio. Below, we characterize the diversity of the EPIC portfolio in terms of the types of
technologies and studies, their commercialization status and the timing of projects. This
type of examination is useful to inform how well the EPIC project portfolio is aligned with
the highest priority policy objectives and how it can be used to support the development
of priorities and inform adjustments to future investment plan strategies.
10.2.1 Technology and Study Characterization

Through examining project-level publications and analyzing administrator and grantee
interviews, we developed an assessment of the technology types and study topics that
each program area serves. This categorization identifies the primary technology types that
are directly impacted through successful project implementation. However, not all projects
include development of discrete technologies or processes, with many projects instead
focusing on sociological or grid-operation studies. We split the 296 projects into two
groups:
1. Projects that produce discrete identifiable technologies or develop market

identification or market pairing opportunities for discrete technologies; and
2. Projects with broader focus on groups of technologies, or that develop
methodologies, or sociological or grid-operation studies.
Figure 21 summarizes the first group (n=162), by technology, administrator and program
area.

Figure 21: Project Outputs - Technology by Administrator and Program Area (n=162)
Market"Iden9fica9on"&"Market"Pairing" 43"
Microgrid"&"V2G" 29"
Water"Energy"Nexus" 19"
Biomass"&"Biogas" 16"
Green"Building" 16"
Grid"Scale"Storage" 11"
Industrial"Management"&"Opera9ons" 9"
HVAC" 7"
PV"Smart"Inverter" 6"
Ligh9ng" 3"
Concentrated"Solar"Power" 3"
0" 5" 10" 15" 20" 25" 30" 35" 40" 45" 50"
CEC"O"MF" CEC"O"R&D" CEC"O"TD&D" IOUs"O"TD&D"
Of these projects, about 27 percent of the projects are Market Identification & Market Pairing
types of projects that focus on sociological or grid-operation studies while the remaining
73 percent of the projects focus on discrete identifiable technologies. One can also see that
the IOU TD&D portfolio in this group is comprised entirely of Market Identification &
Market Pairing and Microgrid & V2G projects while the CEC TD&D portfolio is much
broader. Only the CPUC can decide whether these distributions are optimal.
Figure 22 summarizes the second group (n=133), by general project type, administrator
and program area.
Figure 22: Project Outputs - General Project Type, by Administrator and Program Area
(n=133)
Electric"Grid"Maintenance"
Grid"Communica:on" 17"
Environmental"and"Public"Health"Studies" 13"
Energy"Efficiency"Modeling"and"Demand"Reduciton" 11"
Climate"Impact"Assessments" 9"
Popula:on"and"Economic"Studies" 5"
Workforce"Developments" 2"
0" 20" 40" 60" 80" 100"

CEC"K"MF" CEC"K"R&D" CEC"K"TD&D" IOUs"K"TD&D"

Of these projects, about 70 percent focus on electric grid maintenance and grid
communication types of projects, of which the IOUs are responsible for the vast majority.
Again, only the CPUC can decide whether these distributions are optimal.
Table 53 provides additional detail regarding the types of Market Facilitation projects, all
of which are active.
Table 53: Types and Frequency of Market Facilitation Projects

Types of Projects Frequency
Fostering Commercialization 4
Facilitating Procurement 0
Enhancing Permitting 14
Market & Technical Analysis 9
Other* 3
Total 30
*The evaluation team included one project funded through

Applied R&D, but included it in this area since its activities
are more aligned with the Market Facilitation strategic
goals.
These Market Facilitation projects are clearly consistent with the aim of addressing non-
technical barriers. As one can see, projects aimed at enhancing permitting and market and
technical analysis predominate, thus far. Also note that while there are no projects that are
aimed primarily at facilitating procurement, there are projects that include procurement as
a component (e.g., EPC 15-065: Berkeley Energy Assurance Transformation (BEAT)
Project).
Similar to the policy area analysis presented earlier in this section, analysis of projects by
technology can inform an assessment of how well EPIC projects are supporting Program
and state policy goals.
10.2.2 Commercialization Status

Another way to characterize EPIC project portfolio diversity is by assessing the Applied
R&D and TD&D projects with respect to their commercialization status: intent to
commercialize versus no intent to commercialize.75 Assigning each project to one of these
75Commercialization is not an issue for Market Facilitation projects since only two grantees indicated that
they did not plan to commercialize their product.

two categories is more challenging that the earlier assignment of projects to technology
and study types.
We begin by noting that in the CEC Triennial Investment Plan, the mission of Applied
R&D and TD&D is to address gaps in the funding necessary to help innovative energy
technologies and approaches bridge the “Technological Valley of Death” (TVD) and
“Commercialization Valley of Death,” (CVD) respectively (Jenkins and Mansur 2011).
• The TVD occurs early in the development of a technology, as breakthrough research

and technological concepts aim to achieve commercial proof-of-concept. At this
stage, innovators and entrepreneurs conducting basic and applied research need
further capital to undergo a process of developing, testing and refining their
technologies in order to prove to private funders that these technologies will be
viable in markets beyond initial success in the laboratory. However, private
investors are typically reluctant to fund such early-stage research and product
development, largely due to the high technical, market and management execution
related risks and long development horizons associated with as-yet-unproven
technological concepts. As a result, many entrepreneurial start-up firms and
research laboratories fail to accumulate the necessary capital to see their innovative
research concepts translated into commercial products and ventures.
• The CVD exists between the pilot/demonstration and commercialization phases of
the technological development cycle and aligns with a gap between the traditional
role of venture capital and the later stage investments of project finance and
debt/equity investors. Demonstrations of multiple, integrated demand-side
management technologies are required to document the synergies, overall
economics and other benefits of combining technologies that would result in the
greatest ratepayer benefits. These demonstrations are especially necessary to
establish the right mix of technologies for particular applications, document
technical and economic feasibility, and minimize risk to building
owners/operators. Public funding for demonstrations to bridge the CVD is
essential. The private sector does not typically conduct applied research and is risk-
averse regarding new, unproven technologies, often lacking the resources to
analyze and evaluate various technologies. Frequently, new technologies are
developed in academic communities that do not have the funding for large-scale
demonstrations. Typically, the private sector only offers funding after a successful
field demonstration.
This orientation suggests that the goal of all projects is eventual commercialization of one
or more technologies. However, the diversity of EPIC Applied R&D and TD&D projects is
more complex and nuanced than this mission suggests. Based on project-level data
provided in the Annual Reports, and interviews with grantees and administrators, we

assigned each project in the Applied R&D portfolio to one of two categories described
below:76
1. Commercialization: Projects that either:

a. Develop discrete new technologies or tools for which the goal is commercialization;
or
b. Investigate a combination of commercially available technologies within a discrete
service or innovative platform that qualifies as a unique and innovative product for
commercialization.
In the case of a., developing discrete new technologies, the path to commercialization
entails the successful development of the technology or technologies (with EPIC support)
leading to the eventual availability of the technology or technologies in the market as
either a business-to-business or business-to-consumer product or service. For example,
EPC-15-057 “Customer-controlled Price-mediated, Automated Demand Response for
Commercial Buildings” (an Applied R&D project) is developing an end-user tool to
manage small commercial energy consumption, for eventual sale to customers. Another
example is EPC-14-065 “Demonstration of Forward Osmosis to Produce Juice Concentrate,
Purify and Reuse Wastewater and Reduce Energy Use” (a TD&D project) which is
demonstrating a new technology innovation to reduce the energy, chemicals and
maintenance required for food and beverage processing and waste concentration.
In the case of b., combining commercially available technologies, the path to

commercialization entails project teams combining existing technologies in an innovative
new way that constitutes a new product, and developing this product (with EPIC support)
to the point it is available in the market as either a business-to-business or business-to-
consumer product or service. An example of this type of project is EPC-15-074 “Meeting
Customer and Supply-side Market Needs with Electrical and Thermal Storage, Solar,
Energy Efficiency and Integrated Load Management Systems” (an Applied R&D Project),
which involves the packaging of efficiency products into a services platform for small-
commercial retrofits or the development of a packaged combined energy
storage/Distributed Energy Resources (DER) platform for end-use consumers. Another
example is EPC-15-042 (a TD&D project), the ZERO-CA demonstration project that will
serve as proof of concept for large-scale deployment of Zero Net Energy (ZNE) single-
family homes in California. The product, in this case, that is being commercialized is the
ZNE home.
76While others might develop a somewhat different classification scheme, we believe that the basic
conclusion would remain the same: the EPIC portfolio is quite diverse with respect to its commercialization status.
Of course, an even more complex typology could have been developed, but this relatively simple typology
adequately conveys the basic message.

2. Non-Commercialization: Projects can include innovative methodologies, tools or other
processes that open new opportunities for deploying existing technology, or help users
understand the potential applications and risks of deploying existing technology in new
ways. For example, EPC-14-069 “Develop Analytical Tools and Technologies to Plan for
and Minimize the Impacts of Climate Change on the Electricity System” (an Applied R&D
project) will advance the understanding of key parameters of long-term energy scenarios
and GHG abatement options in the California energy system. This project will achieve this
by further developing detailed scenarios and modeling capabilities of the California
electricity sector, as well as interactions between the electricity sector and other sectors,
and by exploring the implications of particular policy choices on the electricity system in
the medium (2020-2030) and long term (2050). Wider adoption of this public-domain
software by other planning organizations in other jurisdictions is the goal rather than
commercialization. Another example is SCE Project 7 (a TD&D project) which will
demonstrate a Substation Level Voltage Control (SLVC) unit working with a transmission
control center Supervisory Central Voltage Coordinator (SCVC) unit to monitor and
control substation voltage. The technical details of how this project was designed and
implemented will be made available in the public domain so that other utilities can adapt
it to their own systems.
In Table 54, we show the results of our analysis for Applied R&D projects.
Table 54: Commercialization Status for Applied R&D Projects

Commercialization Status CEC
Goal: Commercialization 49
Goal: No Commercialization 72
Total 121
This analysis shows that 60 percent of the Applied R&D projects are being developed for
the public domain rather than for commercialization. Their goal of such projects is not
profit but the wider adoption of existing and new technologies and tools to solve problems
in order to advance key policy objectives.
In Table 55, we show the results of our analysis for TD&D projects.
Table 55: Commercialization Status for TD&D Projects

Commercialization Status CEC PG&E SCE SDG&E Total
Goal: Commercialization 44 1 2 0 47
Goal: No Commercialization 4 33 23 11 71
Total 48 34 25 11 118

One obvious pattern is that about 92 percent of the CEC projects have commercialization
as a goal, while only 4 percent of the IOU projects have this as a goal. Traditionally, public
funding for demonstrations is described as essential for bridging the CVD. However, as
was also the case for the Applied R&D program area, we see that public support for
TD&D projects should not be understood as strictly devoted to the commercialization of
new technologies but also to the wider adoption of existing and new technologies and
tools that can produce important benefits such as improved planning and grid reliability.
This diversity of projects with respect to technologies and studies and commercialization
is consistent with the CPUC’s directive to support clean energy technologies and
approaches for the benefit of (IOU) ratepayers, with projects directly developing new clean
energy technologies; packaging existing technologies into innovative bundles to address a
new problem; or developing new processes, tools or methods to promote existing
technology adoption in direct or indirect ways that will be added to the public domain.
The question is whether this current EPIC portfolio contains the optimal mix of projects
with respect to technology and study types and commercialization. However, determining
whether this is the optimal mix would require that the CPUC prioritize its policy
objectives. The CPUC can then compare the policies supported by the EPIC portfolio to
determine the extent to which they align with these priorities in terms of budgets and
expected benefits. To the extent it does not align well, the CPUC can then recommend
changes in the types of technologies and studies and the proportion of projects that are
intended to be commercialized.
10.2.3 Schedule of Project Results

We analyzed the EPIC 1 and EPIC 2 portfolio (collectively spanning years 2012–2017) in
terms of the average time required to complete a project, the number of active and
completed projects, and the number of active projects expected to be completed in future
years. The timing of project completion may be an important consideration when project
results are critical to supporting time sensitive energy policy decisions. Table 56 presents
the results for Applied R&D and Market Facilitation projects. Note that the majority will
be completed in 2019 or later.
Table 56: Project Status for Applied R&D and Market Facilitation Projects
Mean Time to
Completion Due End Due End Due
Program Area (yrs.) Completed of 2017 of 2018 2019+
Applied R&D 3.3 1 7 35 82
Market Facilitation 2.9 0 2 17 10

On average, Applied R&D projects take an average of 3.3 years to complete from the time
they are launched. There is one completed Applied R&D project, and seven that have a
projected completion date at the end of 2017, while an additional 35 projects have a
scheduled completion date at the end of 2018. Market Facilitation projects are expected to
take an average of 2.9 years to complete. Through the end of 2017, only two Market
Facilitation projects are projected to be completed, while an additional 17 are scheduled to
be completed by the end of 2018.
Table 57 presents the results for TD&D projects. The majority of IOU projects are either
completed or scheduled to be completed by the end of 2017, while the majority of CEC
TD&D projects will be completed in 2019 or later.
Table 57: Project Status for TD&D Projects

Mean Time Due
to Completion End of Due End Due
Administrator (yrs.) Completed 2017 of 2018 2019+
CEC 3.5 0 0 16 32
PG&E* 2.6 14 10 9 1
SDG&E 2.4 0 11 0 0
SCE 4.1 4 10 6 2
* There are an additional 21 PG&E projects that are “on-hold.”
On average, CEC TD&D projects are expected to take an average of 3.5 years to complete,
slightly more time than Applied R&D and Market Facilitation projects. The IOU TD&D
projects have taken or are expected to take an average of 2.7 years to complete, slightly less
time than Applied R&D and Market Facilitation projects.
Of the 48 active CEC TD&D projects, none are projected to be completed by the end of
2017. CEC projects on average start later than IOU projects since the CEC uses a
solicitation process (which introduces additional time), and they take about nine months
longer to complete, once launched.
What emerges from this analysis is that the diversity of projects with respect to
technologies, commercialization and timing of results in the EPIC portfolio that has been
formed by the four administrators adequately addresses the multiple policy goals.
However, whether each administrator is expending the right level of effort to address each
of these policy goals, delivering project results quickly enough and bringing important
innovations to market or the public domain cannot be determined until these policies are
prioritized. It is also clear that portfolio optimization is an on-going process that must

respond to changing priorities and budgets in a rapidly changing technological and
environmental landscape.
10.3 Public Interest Focus

One of EPIC’s guiding principles, specified in the foundational EPIC program Decisions, is
to provide public benefits to electricity ratepayers in the form of greater reliability, lower
costs and increased safety.77 To address this public interest focus, the evaluation team
reviewed the portfolio of projects and closely examined the sample of 54 projects to
ascertain:
• To what extent were projects selected to provide broader public interest benefits?
• To what extent did projects propose work that could not be or was not being
conducted by the competitive market at the time it was funded?
10.3.1 Broad Public Benefits

Using the same approach described in Section 7.1 to assess program alignment with
guiding principles and policy goals, we assessed the potential for ratepayer benefits by
investment area. All EPIC projects are designed and selected to provide ratepayer benefits,
as mandated by the CPUC. We reviewed the types of benefits that would accrue to
ratepayers, and categorized them as either providing direct benefits to all ratepayers
(broad benefits) or initial benefits to a smaller subset of ratepayers such as EV owners or
grid-system operators, with greater societal benefits in the longer term (narrow benefits).
Figure 23 shows the results of our assessment. A high percentage of Applied R&D, Market
Facilitation and IOU TD&D projects meet the criteria for broad benefits such as providing
societal benefits and supporting the loading order. A smaller fraction of CEC TD&D
projects meet the criteria for broad benefits.
77 D.12-05-037

Figure 23: EPIC Funding by Ratepayer Benefit, by Program Area
85%#
IOU#/#TD&D#
15%#
57%#
CEC#/#TD&D#
43%#
68%#
CEC#/#Applied#R&D#
32%#
67%#
CEC#/#MF#
33%#
0%# 10%# 20%# 30%# 40%# 50%# 60%# 70%# 80%# 90%#
Broad#Ratepayer#Benefits# Narrow#Ratepayer#Benefits#
We cannot say whether each administrator is expending the right level of effort addressing
broad versus narrow ratepayer benefits. The CPUC should explore more deeply the cost
and benefits of broad versus narrow benefits and the optimal proportions of each. This
type of project-level analysis is similar to the policy alignment and project diversity
analyses we presented previously. These analyses provide a current snapshot of how
projects are fulfilling various Program and policy goals and the diversity of the portfolio
with respect to technologies addressed and timing of project results. These analyses may
be revisited at a later date to support future investment planning efforts and the setting
and revising of priorities.
10.3.2 Net Public Benefits

It is critical that EPIC produces benefits that would not have been realized otherwise, i.e.,
to some extent, these benefits must be attributable to EPIC. Establishing a causal link
between EPIC-funded activities and measureable changes in metrics associated with the
mid- and long-term outcomes begins with how projects were selected for funding. The
prudent expenditure of EPIC funds requires that they be targeted at technologies and tools
that, absent EPIC funding, would not survive the TVD or the CVD or have their progress
to market significantly slowed, and at studies that produce new knowledge. To assess the
extent to which this is the case, we asked project personnel whether, absent EPIC funding,
their projects would not have been done or would at least have been delayed. Across the
three program areas, there is good evidence that EPIC is investing in sound projects that

have no other source(s) of available funding. Below, we provide more details regarding
our results.
We begin by repeating that approximately 40 percent of the grantees of the CEC Applied
R&D portfolio and 92 percent of the grantees of the CEC TD&D projects intend to
commercialize their products, while about 4 percent of IOU project managers of TD&D
projects intend to commercialize their products. It is for these projects that the TVD and
CVD pose a threat. For projects in which the goal is commercialization, we interviewed a
sample of Applied R&D and TD&D grantees as well as IOU project managers of TD&D
projects and asked whether, absent EPIC funding, their project would have fallen into the
TVD or the CVD. For the Market Facilitation projects, for which there is almost never any
intent to commercialize, we asked project managers whether the results expected from
their projects are currently available from other sources, i.e., whether their research
unnecessarily duplicates existing research.
With few exceptions, Applied R&D and TD&D grantees and IOU TD&D project managers
indicated that their projects would have fallen into one of the two Valleys of Death or their
progress toward commercialization would be slowed. Nearly all of the Market Facilitation
grantees indicated that the expected results of their projects were not currently available.
Clearly, CEC grantees and IOU project managers felt that EPIC funding is necessary,
although not sufficient, for their projects to be commercialized.
The issues of whether, absent EPIC funding, a technology or tool would likely fall into the
TVD or CVD or whether the expected results are already available from other sources
were also assessed as part of the investment planning process and during the proposal
review process.
During the investment planning process, the CEC develops expert- and stakeholder-
driven documents, referred to as research roadmaps, which provide strategic guidance on
prioritizing funding initiatives. These roadmaps summarize current research, data gaps,
connections to state policy, potential impact by cost, urgency and timeliness of outcomes,
and potential partnerships with other funding entities. As part of the 2015-2017 EPIC
Investment Plan development process, the CEC used the numerous research roadmaps as
well as U.S. Department of Energy (DOE) roadmaps to identify gaps and funding
opportunities. For example, the gaps analysis in the Plug-in Hybrid Electric Vehicle
Research Roadmap (CEC-500-2010-039) found an abundance of basic chemical and battery
formatting research conducted by battery manufacturers but minimal research into the
second use of batteries after the primary vehicle application.
In addition, the CEC reviewers (some of whom are third-party reviewers) require bidders
to justify their projects based on market forecasts and penetration estimates, and provide a
compelling argument why EPIC funds are needed (i.e., bidders need to justify the need for
EPIC funding, including an explanation of why the proposed work is not adequately

supported by competitive or regulated markets). Bidders who cannot justify the need for
EPIC funds are much less likely to be successful in obtaining an EPIC grant.
In addition, the framework established by the CPUC within which the IOUs operate helps
to ensure that IOU TD&D projects would not be done absent EPIC funding:
"For activities that are more related to technology demonstration and deployment
on the grid, as technologies and approaches move toward commercialization,
utilities may be better suited to administer the funding, as they point out, since they
own the infrastructure on which or through which the technologies will be tested.
They also may ultimately become the consumers of technologies or processes that
are designed to improve utility systems, so it will behoove them to invest in and
test some new ideas."78
It is in the interest of each IOU to find new technologies or combinations of existing

technologies that address the unique needs of their electric system (and the needs of other
utilities both inside and outside California that could possibly adapt them). Each IOU is
also motivated to assess whether the technologies and tools that address their particular
problem currently exist. If they do not exist, then the IOUs are far more likely than
anybody else to investigate these new technologies and tools (i.e., in the current
framework, absent EPIC funding, it is unlikely that these TD&D projects will be
conducted). The Decision also ordered that EPIC would be the primary venue for the
IOUs’ RD&D expenditures other than RD&D proposed by IOUs as part of their budget
applications for energy efficiency and demand response. The IOUs can request separate
funding for electric RD&D in their energy efficiency and demand response budget
applications, but they are required to justify why such expenditures could not have been
considered within EPIC. To date, separate funding has never been requested.
In summary, the various planning documents and CPUC Decisions combined with the
comments from the CEC administrators and TD&D grantees as well as the IOU
administrators and TD&D project managers support the conclusion that due diligence is
being done to identify projects that, absent EPIC funding, would not move forward or
move forward more slowly. Again, this is a necessary but not sufficient condition for
establishing a causal connection between EPIC-funding activities and any measureable
changes in metrics associated with the mid- and long-term outcomes. The probability that
these outcomes will eventually be achieved will increase substantially only through the
combination of successfully designed and implemented EPIC projects and the diffusion of
78CPUC. Phase 2 Decision Establishing Purposes and Governance for Electric Program Investment Charge and
Establishing Funding Collections for 2013-2020. 2012.
http://docs.cpuc.ca.gov/word_pdf/FINAL_DECISION/167664.pdf, p. 27

these technologies and tools over time in their targeted markets. Of course, a more
complete case for attribution must be built on an on-going, theory-driven evaluation that,
based on the preponderance of the evidence, will assess the extent to which these mid- and
long-term outcomes have been achieved and the extent to which EPIC is responsible for
any of these achievements, and whether the preponderance of the evidence suggests that
the benefits of EPIC exceed its costs.
10.4 Program Equity

We reviewed program equity in terms of EPIC projects providing benefits or services to
disadvantaged communities. The CEC has an explicit goal of assisting disadvantaged
communities as stated in its formal Diversity Policy Resolution adopted in April of 2015.
Figure 24 below shows the proportion of each CEC investment area that serves
disadvantaged communities in some capacity. Seventeen percent of CEC TD&D projects
serve disadvantaged communities to some extent, followed by 8 percent of Applied R&D
projects and 3 percent of Market Facilitation projects.
Figure 24: Percentage of CEC Projects that Serve Disadvantaged Communities
CEC - TD&D
Disadvantaged Community
CEC - R&D
CEC - MF
0% 5% 10% 15% 20%
Figure 25 below shows the proportion of each CEC research area that serves
disadvantaged communities by California policy goals. All Applied R&D, TD&D, and
Market Facilitation projects that are serving disadvantaged communities are also aligned
with meeting GHG reduction and the loading order policy goals. A small percentage of
projects serve disadvantaged communities within the Electric Vehicle and Energy Storage
California policy goals, with 4 percent of Applied R&D projects and 2 percent of TD&D
projects including these communities in some capacity.

Figure 25: Percentage of CEC Projects that Serve Disadvantaged Communities by
California Policy Goals
GHG Reduction
Loading Order
Distributed Energy Resources
Energy Efficiency
Clean Energy Jobs
Smart Grid
Renewable Portfolio Standard
Energy Storage
Electric Vehicle
0% 10% 20%
CEC - TD&D CEC - R&D CEC - MF
CEC programs can have both direct and indirect benefits to disadvantaged communities.
We reviewed annual reports, program documentation and interviewed administrators and
grantees to understand how specific projects were targeted to directly or indirectly benefit
disadvantaged communities and whether they have thus far been successful. Below, we
provide two examples of projects that are intended to provide benefits to disadvantaged
communities among the 54 sampled projects:
• The stated goals of EPC-15-010 were to recruit workers from disadvantaged

communities into an apprenticeship program, and provide them with
comprehensive classroom and on-the-job training on the installation and
maintenance of AutoDR communications equipment. The project aims to pilot a
new California Advanced Lighting Controls Training Program course focused on
installation and maintenance of AutoDR communications equipment for lighting
applications and will recruit small and medium buildings located in disadvantaged
communities to serve as on-the-job training sites for the apprentices enrolled in the
program. While the project is at an early stage, the administrator and grantee are
pleased with the progress of the project and noted successes with early recruitment
of apprentices and developing the training curriculum. In addition, the project

serves communities designated as disadvantaged communities using
CalEnviroScreen.79
• EPIC-14-029 intends to convert dairy manure into biogas and store the biogas under
an inflatable cover. The biogas will be converted into renewable electricity
anticipated for sale and delivered to the PG&E distribution grid through a
Bioenergy Feed-in Tariff. Furthermore, dairy biogas systems qualify for
participation in the CPUC's Assembly Bill (AB) 2514 electricity storage program. In
a future phase, the biogas system may compete for an energy storage contract.
Among other benefits, this project is expected to reduce GHG and improve the
quality of groundwater, both of which affect disadvantaged communities. The
project also serves communities designated as disadvantaged communities using
CalEnviroScreen.
• EPIC-14-027 is demonstrating a pre-commercial flow battery storage control system
at a Regional Wastewater Treatment Plant located in a disadvantaged community
as designated using CalEnviroScreen. The expected benefits of the project align
with policies around energy storage, GHG reduction and clean energy job provision
all of which, if realized, can provide tangible benefits to disadvantaged
communities.
Although the IOUs do not have an explicit goal of assisting disadvantaged communities,
their projects nevertheless benefit them. When we asked the IOU program management
teams about whether they may add a focus on disadvantaged communities in future EPIC
Investment Plans, they indicated that the nature of their TD&D projects does not lend itself
to a focus on a specific community. For example, because their projects are usually
conducted in-house, they do not lead to localized job creation which could generate
benefits to a disadvantaged community. Rather, their projects are intended to benefit their
whole service territory, including disadvantaged communities, through improvements to
the grid that make it easier to incorporate distributed energy resources. Based on our
examination of the IOUs’ projects, we agree with their assessment. The IOUs added that
some of their other (non-EPIC) initiatives, such as the ratepayer-funded Local Government
Partnerships, focus on providing services to disadvantaged communities.
We have provided some initial, baseline measurements of how well the CEC projects are
supporting disadvantaged communities, one of the state's key policy goals. Such
measurements can be repeated to assess progress over time. Moreover, the CEC intends to
focus even more on this issue in the future. Given this, the CPUC might choose to set a
goal with respect to serving disadvantaged communities.
79CalEnviroScreen is a tool developed by CalEPA to designate California communities as disadvantaged

pursuant to Senate Bill 535. The tool was used to help identify California communities that are
disproportionately burdened by multiple sources of pollution.

10.5 Indicators of Project Performance
Based on a detailed review of our sampled projects (including project data and telephone
interviews), this sub-section summarizes the performance of each of the three program
areas. Guided by the logic models we developed (see Sections 13.2, 13,3, and 13.4 for more
details), we collected and analyzed data related to the project activities, outputs and short-
term outcomes to assess, using the preponderance of evidence approach, the extent to
which the Program is on track to achieve its mid- and long-term outcomes. In Sections
13.2, 13.3 and 13.4, we explicitly link our more detailed results to the specific boxes that
describe the activities, outputs and outcomes of each logic model so that one can better
assess the extent to which the collection of hypothesized causal linkages is functioning as
expected (e.g., is a given activity leading to the expected output and are the expected
outputs leading thus far to the expected outcomes). As C. H. Weiss notes: “If the
evaluation can show a series of micro-steps that lead from inputs to outcomes, then causal
attribution for all practical purposes seems to be within reach.”80 In other words, is EPIC
on track to produce benefits beyond what would have happened anyway? Please see
Appendix E for more detailed results from our review of project performance.
10.5.1 Applied R&D Projects

Our review of a sample of Applied R&D projects suggests that this part of EPIC’s portfolio
is progressing toward the ultimate goal of furthering the adoption of innovation and
technology advancement related to clean energy. Existing projects appear headed toward
successful completion or are on track to investigating the research questions at hand.
However, we do note that progress is taking time, as most EPIC Applied R&D projects are
multi-year research efforts, and none are yet completed. Hence, even as we hear that
technology is changing rapidly, advancement of EPIC-supported technology is measured
in years, not months.
It is also apparent that EPIC-supported research does not always match the program logic
assumption that technologies start as an Applied R&D project, proceed to TD&D, and,
with the support of Market Facilitation projects, eventually make it to market. While many
EPIC-supported research efforts do involve privately developed technologies that are
intended to be commercialized, some will result in tools or information that will be
available in the public domain and others are supportive of existing technologies. Hence,
the path toward technology adoption varies by project. Furthermore, even among the
commercialized technologies, the path toward commercialization is not always linear, and
some projects will require additional research while others appear poised for
commercialization even without additional EPIC involvement.
80 C.H. Weiss, “Theory-based Evaluation: Past, Present and Future,” in Progress and Future Directions in
Evaluation: Perspectives on Theory, Practice and Methods, eds. D.J. Rog & D. Fournier (1997), 41-55.

Overall, when viewed as a portfolio, EPIC’s Applied R&D projects do show movement
toward knowledge and awareness among potential users, follow-on research and
development, technology demonstrations, potential private investment, patents and early
adoption. However, some of these projects are unable to show such progress since they are
in their early stages.
10.5.2 TD&D Projects

Since 78 percent of the TD&D projects are still active, our primary focus was on the
outputs and short-term outcomes, which are considered to be leading indicators of
whether the TD&D projects are on track to contribute to the achievement of the mid- and
long-term objectives.
The data collected from our sample of the nine completed IOU projects indicate that they
have met their technical targets, verified their expected benefits, gained knowledge, and
increased their confidence in the performance of the technology. For the active IOU
projects and for many of the CEC projects, nearly all of the project teams felt that thus far
they are on track to be as successful as the completed projects. For the active and
completed projects, the knowledge that has been gained is being reported in mostly
monthly progress reports or final reports and is being used by the Commission Agreement
Managers (CAMs), members of the TACs, IOU project managers, and other organizations
apart from the EPIC project teams. However, while patents, copyrights and licenses are
important indicators that knowledge is being produced, none were reported. It should be
noted that many of these projects were in mid-cycle development, and it was determined
that any patents, copyrights and licenses, if they were to occur, would likely take place
near the end of a project and prior to technology dissemination activities.
For those projects for which commercialization is the goal, most project teams felt that,
absent EPIC funding, their technologies would have fallen into the CVD. Some reported
that while it would not have fallen into the CVD, EPIC funding was able to accelerate its
development, which is still an important benefit. Respondents also indicated some level of
understanding of their respective markets, while many of the CEC CAMs indicated that it
was too soon to tell. In general, respondents felt that their projects were scalable, cost-
effective at commercial scale, and commercially viable. Nevertheless, very few grantees
have met with interested parties to form partnerships, and only one has actually added a
partner. In addition, none of the grantees reported that they had discussed possible
acquisition of their company with any manufacturers or venture capital firms. Of the 22
projects with the goal of eventual commercialization, six project teams indicated that new
manufacturers had entered the market, which is a strong indicator of potential demand
and profitability.
In general, the TD&D portfolio appears to be on track, thus far, in meeting its short-, mid-
and long-terms objectives. However, we also observed some interesting differences
between the CEC and IOU TD&D projects that merit mentioning. More specifically, there

are substantial differences with respect to whether there is an intent to commercialize the
technology being investigated, the types of technologies being investigated, and follow-on
TD&D research.
Differences by Administrator
In this section, we present the differences we observed between the CEC and IOU projects.
There are substantial differences with respect to whether there is intent to commercialize
the technology being investigated. Of the CEC projects, 95 percent have commercialization
as the goal, while only 4 percent of the IOU projects have this as a goal. For those projects
for which commercialization is the goal, most felt that, absent EPIC funding, they would
have fallen into the CVD. Others reported that while it would not have fallen into the
CVD, EPIC funding was able to accelerate its development, which is still an important
benefit. Respondents also indicated some level of understanding of their respective
markets while many of the CEC CAMs indicated that it was too soon to tell. In general,
respondents felt that their projects were scalable, cost-effective at commercial scale, and
commercially viable. Nevertheless, very few grantees have met with interested parties to
form partnerships, and only one has actually added a partner. In addition, none of the
grantees reported that they had discussed possible acquisition of their company with any
manufacturers or venture capital firms. More information about the performance of the
technology might be needed before talking to potential partners and investors. That these
projects are new might also explain to some extent why none of the projects reported any
activity with respect to patent, copyrights and licenses.
Another interesting difference is that about 83 percent of the IOU projects address grid
communications and interactivity and grid maintenance, optimization, planning and management,
while none of the CEC projects address these topics. While ideas for projects originate
from multiple sources including their own staff, the staff of the other IOUs, EPIC,
manufacturers, government laboratories such as LBNL, the DOE and universities, these
ideas are organized by the IOU joint framework that includes four investment areas (all
within the TD&D area) that the IOUs have mapped to the electric system value chain as
follows:
• Cross-Cutting/Foundational Strategies and Technologies – maps across the electric

value chain.
• Customer Service and Enablement – maps to demand-side management (DSM)
• Grid Modernization and Optimization – maps to transmission and distribution
(T&D)
• Renewables and Distributed Energy Resources (DER) Integration – maps to grid
operations/market design
This framework is what one should expect from the utilities that are confronted every day
with a set of very specific challenges related to their grids, while the CEC has a much

broader focus. When prompted, each IOU noted that, while their projects are narrowly
targeted toward the unique needs of their systems, they are certainly relevant to the needs
of the other utilities both inside and outside California. We have no comment on whether
the IOU TD&D portfolios should resemble the CEC TD&D portfolio with respect to
technology types (and commercialization) since this is a CPUC policy issue.
Finally, when comparing the active CEC projects with the completed IOU projects, more
than half of the CEC project teams indicated that they plan to conduct additional TD&D
research based on the results of their projects thus far, while nearly all of the IOU project
teams indicated that they do not intend to conduct any additional TD&D research. At least
some of these differences might be due to the fact that the IOU projects are more narrowly
defined for a very specific application and entail less risk since they are often based on a
combination of existing, commercially available components.
10.5.3 Market Facilitation Projects

Based on our review, we found that EPIC's Market Facilitation projects are clearly
consistent with the aim of addressing non-technical barriers with projects aimed at
enhancing permitting and market and technical analysis predominating, thus far. Also
note that while there are no projects that are aimed primarily at facilitating procurement,
there are projects that include procurement as a component of their project (e.g., EPC 15-
065: Berkeley Energy Assurance Transformation (BEAT) Project). In addition, both the
CAMs and grantees reported that their projects generally are on track to achieve their
research objectives and expected benefits. Many also reported that these same benefits
could be achieved if these projects were replicated in other states and jurisdictions.
We reviewed benefit questionnaires from four of seven Market Facilitation projects to

identify the types of projected benefits. Three of the four projected economic benefits, and
two projected energy and cost savings, environmental, and electrical system safety and
reliability benefits.
However, the coordination between the Market Facilitation program area and the Applied
R&D and TD&D program areas appears to be less than ideal. As mentioned earlier,
interviewees were not as likely to cite the CEC Applied R&D grantees or TD&D grantees
as primary audiences. Grantees were even less likely than CAMs to do so. While the
reasons for this are largely a function of the timing of the EPIC portfolio, in the future, a
more systematic assessment of the needs of those conducting projects that are upstream
from the Market Facilitation projects should be undertaken. The CEC administrators are
aware of this gap and report that they are committed to addressing it.
.

11 Findings and Recommendations
In this section, we present our findings and recommendations that are based on the
preponderance of evidence gathered from various sources throughout this evaluation and
organized using the logic models presented in Sections 13.2, 13.3 and 13.4.
As discussed throughout this report, the CEC and IOUs have different approaches for
administering EPIC. Table 58 provides a comparison of how the administrators approach
various administrative processes, highlighting key differences. We also identify whether
EPIC administrative practices are consistent with other peer RD&D programs. A check
mark indicates that the administrative process is consistent with peer programs.
Table 58: Comparison of EPIC Administrative Processes to Peer RD&D Programs

Administrative
Process CEC IOUs
Program Awards grants to external Conduct their research using
Management and organizations that conduct their internal staff with use of vendors.
Administration research.
✓
Administration is managed by a core team with RD&D program expertise,
with technical support provided by both internal and external experts.
Four-administrator model
Investment Planning Identifies a series of strategic objectives Develop Investment Plan priorities
with strong and transparent internally, predominantly relying on
linkages to state policy goals. their own technical experts and
✓ management to identify and prioritize
research areas, with linkages to policy
less transparent.
Relies mostly on input from multiple Rely mostly on external input from a
external stakeholders; develops its single utility-focused stakeholder;
Investment Plans transparently and insufficient transparency in
engages external stakeholders developing Investment Plans.
throughout the process.
✓
Project Selection Uses a transparent and public Use a less transparent, internal
process for selecting projects and process for selecting projects and do
shares project scopes of work in a not share project scopes of work in a
timely manner. timely manner.
✓
Due diligence is being done to identify projects that, absent EPIC funding, would
not move forward or would move forward more slowly.
✓

Administrative
Process CEC IOUs
Project Assessment Shares information about projects while they are being implemented but less
frequently than optimal.
Uses a robust process for collecting Lack a robust process for
the necessary quantitative data needed collecting the necessary data needed
to comprehensively report on project to comprehensively report on project
benefits and disseminate results. benefits and disseminate results.
✓
= consistent with peer RD&D program practices
Throughout the evaluation, we examined each administrators’ processes and projects.

When we compared across the four administrators, the main finding was the difference in
how the CEC approaches program administration as compared to the IOUs. However, we
did note some differences in processes across the three IOUs in this report, which we
summarize below in Table 59. Note that those comparisons were limited by the size of the
project sample for each IOU (which reflects the relatively smaller portion of the EPIC
budget that the IOUs receive as compared to the CEC).
As shown, program management and administration is fairly similar, with only minor
differences in administrative spending across IOUs. PG&E was the only IOU to submit a
request to the CPUC for new and revised projects. With respect to Investment Plans, the
IOUs provide a similar level of detail describing their plans; however, SDG&E is the only
IOU to include project budget allocations. We identify the need for the IOUs to share more
information about projects later in this section.
All three IOUs are deficient (as is the CEC) regarding justifying their use of non-
competitive bidding. We discuss this issue in Section 11.1 below.
PG&E and SDG&E have slightly larger organizational networks than SCE with which they
engage and share project results, and PG&E has shared project results more frequently
than SCE.81 Note that with a small fraction of projects completed at the time we conducted
our research, the sample sizes are small. Once more projects are completed, this analysis
could be repeated with larger sample sizes to identify trends and facilitate comparisons
more robustly. Finally, nearly all of the IOUs’ projects (in EPIC 1 and 2) are not intended to
be commercialized as shown in the last category of the table. We discuss this issue in
Section 11.5.
81 SDG&E had not yet completed project at the time of our research.

Table 59: Differences in EPIC Program Administration by IOU
Administrative Process PG&E SCE SDG&E
Program Management Administrative spending 8% 6% 10%
and Administration
Submitting a waiver of program Submitted an Advice Letter Has not sought to waive Has not sought to waive
requirements / for new or revised program requirements program requirements
projects
Investment Planning Level of project detail provided Does not provide budget Does not provide budget Provides project budget
information information estimate
Project Selection Use of non-competitive bidding 24% of EPIC 1&2 projects 33% of EPIC 1&2 9% of EPIC 1&2 projects
projects
Justifying use of non-competitive Offered justification for Offered general Only indicated the use of
bidding each case in their Annual justification for use of non-competitive bid, did
Report non-competitive bidding not offer justification for
in their Investment Plan each case in their Annual
(but did not offer Report
justification for specific
cases)
Providing vendor budget Not provided Not provided Provided in the Annual
information for competitive Report
bidding
Project Assessment Revisions to projects 20% of projects revised 43% of projects revised 0% of projects revised
(EPIC 1&2) (EPIC 1&2) (EPIC 1&2)
Project Results Dissemination PG&E has disseminated SCE has not disseminated NA (no projects
project results to listservs any information via completed at the time of
for 60% of sampled listservs for sampled our evaluation)
projects (n=10) projects (n=7)
Project Networks PG&E has an average of 6 SCE has an average of 4 SDG&E has an average
organizations per project organizations per project of 10 organizations per
network (n=31 network (n=8 project network (n=41

Administrative Process PG&E SCE SDG&E
organizations associated organizations associated organizations associated
with project sample) with project sample) with project sample)
Project Schedule Mean time for project Mean time for project Mean time for project
completion is 2.6 years completion is 4.1 years completion is 2.4 years
71% of EPIC 1&2 projects 64% of EPIC 1&2 100% of EPIC 1&2
expected to be completed projects expected to be projects expected to be
by end of 2017 completed by end of completed by end of
2017 2017
Project Impacts and Commercialization of projects 3% of EPIC 1&2 projects 8% of EPIC 1&2 projects 0% of EPIC 1&2 projects
Policy Alignment expected to be expected to be expected to be
commercialized commercialized commercialized

Key study findings and recommendations are discussed below, organized by study
research area. Recommendations are numbered by section (which also correspond to the
five evaluation research topics, plus two additional sets of recommendations that are
cross-cutting).
11.1 Program Administration

The key research question that we sought to address under this study research area is:
Are the administrators effectively complying with program requirements?
Based on our review of program administrative procedures, we find that the four
administrators are in compliance with program requirements.
We examined the numerous CPUC requirements that fell into the following nine
categories.
• Contracts
• Quantifying benefits/metrics
• Budget
• Annual reports
• Miscellaneous
To verify compliance with these requirements, we relied on a combination of sources,

including program filings (e.g., Annual Reports and Investment Plans), the sample of
projects for which we had more detailed information (supplemented by interviews with
grantees, vendors and IOU/CEC project managers) and the sample of CEC
solicitations/bids and IOU request for proposals (RFPs) and vendor bids. We then
compared the IOU administrative practices to other peer RD&D program practices and
observed a number of areas where performance could be improved.
Based on our review, we find that the EPIC administrators have sufficient administrative
and technical expertise and capabilities and devote adequate staffing to EPIC-related
positions. Administration is managed by a core team with RD&D program expertise that
handles design and implementation, with technical expertise provided by internal experts
as well as by external experts, which is consistent with other peer RD&D programs.

Likewise, administrative roles and responsibilities are well defined. The administrative
project management teams are staffed sufficiently with cross-functional teams, consistent
with other peer RD&D programs, fulfilling the range of required duties: administrative,
technical and regulatory. They have effective processes in place to continually track and
comply with EPIC requirements.
However, we identified two specific cases where the administrators are technically
compliant, but they are not meeting the intent of the requirements.
In particular, there are two areas where we determined that the administrators are not
meeting the intent of EPIC program requirements: non-competitive bidding practices (all
administrators) and benefits tracking (the IOUs). With respect to competitive bidding, we
recommend that:
• 1a) The administrators provide more detailed justification for non-competitive

bidding in their Annual Reports. The current administrative processes do not
provide enough information to allow for appropriate oversight.
• 1b) The CPUC consider requiring a review of the non-competitive bidding cases
before they are contracted, since they are not being presented in the Investment
Plans, where the CPUC could review these cases before contracts are awarded. By
the time the administrators report on such cases in their Annual Reports, it is too
late for review. (The CPUC intended that administrators indicate their non-
competitive bidding plans in their Investment Plans, but typically, the CEC and the
IOUs are not far enough along with project plans to provide the determination and
possible justifications in the Investment Plans.) We note that if there is a lengthy
review period, there is the potential risk of delaying a project.
• 1c) The CPUC require the IOUs to specify the funding amount for the non-
competitive award to make it easier to assess the fraction of funding that is being
directly awarded. (SDG&E provides budget information for vendors in its
Investment Plans, so such information is available for SDG&E. However, budgets
may change once projects are implemented and vendors are selected, so the actual
budget amount being sole sourced should be confirmed.) Such information would
be useful to determine how much project funding is being directly awarded versus
competitively bid.
We discuss deficiencies in how the IOUs quantify project benefits in Section 11.4.2.
We also identified additional areas where the minimum requirements are not sufficient
to ensure best-in-class program administration (such as stakeholder engagement,
coordination and information sharing.) In these cases, the administrators are technically
compliant but could better fulfill the spirit or intent of the requirements. We discuss these
items throughout the rest of this section.

11.2 Investment Planning Process
The key investment planning research question we sought to address is:
Is the triennial investment planning process effectively identifying a broad range of potential
energy RD&D objectives, evaluating those objectives according to sensible criteria, and
ultimately producing Investment Plans with a high likelihood of producing benefits for
California ratepayers and achieving other EPIC goals?
We address this research question in the following three sub-sections.
11.2.1 Administrator Investment Planning Processes

The CEC’s investment planning process produces plans that have a high likelihood of
producing benefits for California ratepayers and achieving other EPIC goals.
While each IOU project is related to at least one area of the state’s energy policy, their
TD&D portfolios focus on a much narrower set of investment areas compared to the
CEC.
The CEC’s administrative model is consistent with other peer RD&D programs in that
there is strong explicit alignment of program initiatives with relevant energy policy goals
and transparency in investment planning.
IOU project ideas predominantly originate from internal IOU staff, and are organized by
the IOU joint framework that includes four investment framework elements (all within the
TD&D area), each of which the IOUs have mapped to the electric system value chain. As a
result, the IOU projects are more narrowly focused primarily on grid communications and
interactivity and grid maintenance, optimization, planning and management. The IOUs
noted that, while their projects are narrowly targeted toward the unique needs of their
respective systems, they are certainly relevant to the needs of the other utilities both inside
and outside California and benefit all ratepayers.
• We have no recommendations related to whether the IOU TD&D portfolios should

resemble the CEC’s broader focus since this is a CPUC policy decision.
11.2.2 Portfolio Optimization

EPIC’s objectives are broad, but, based on our best-practices review, we did not find
that a broad approach to project selection was a problem. However, given the many
policy areas EPIC is attempting to address, we have identified a need to prioritize
among these broad objectives.
Based on our assessment, the administrators’ investment planning processes result in a

collection of projects that together meet all the various EPIC program requirements,

support state policy and address research gaps. For example, we confirmed that every
EPIC project provides ratepayer benefits, but there is variation in how broad and/or direct
those benefits are. Likewise, we identified that a high percentage of CEC projects focus on
energy efficiency, but since the CPUC has not established priorities, it is not possible to
determine if the administrators’ investment planning frameworks are effective in creating
a portfolio that has the optimal mix of projects.
EPIC is a broadly focused program compared to other RD&D or emerging technology

programs that have much more narrowly focused mandates. The EPIC portfolio of
projects must strike a balance between maintaining its broad focus and concentrating on
the state’s highest policy priorities.
We have identified a need to prioritize the principles, policies and strategic objectives and
operationalize what it means for a portfolio to be optimized, though we acknowledge that
such prioritization must also balance the desire to “allow a thousand flowers to bloom” to
ensure sufficient opportunities to sow broad innovation. Once prioritized, the
administrators could then assess the extent to which each project addresses these
principles, policies and strategic objectives. Doing this would assist decision makers in
gaining a better understanding of what it means for a portfolio to be integrated into the
broader innovation and policy landscape and the extent to which the EPIC portfolio meets
that standard. A more refined assessment of the policy alignment then could be conducted
to determine whether the portfolio is optimized.
We recommend that:
• 2a) The CPUC establish priorities among its current policy goals and funding
criteria to better guide the administrators in their investment planning. Each
project is vetted, and they all meet the various criteria. However, funding is finite,
and allocating that funding across too many policy goals and funding criteria runs
the risk of diluting EPIC’s impact.
• 2b) The administrators collaborate in categorizing and summarizing projects (such
as by technology type and/or policy area) and review projects by topic areas to
ensure that the portfolio of projects effectively supports key policy goals.
• 2c) The administrators’ Investment Plans are closely reviewed to ensure they not
only meet program requirements, but that they are also effective in advancing the
energy policy priorities that the CPUC identifies. Such a review could focus on
ensuring the CEC’s strategic objectives are in line with state priorities and are not
overly responsive to priorities that may be temporary (such as tree mortality).The
review could ensure that IOU projects are effectively advancing state policy,
beyond just being related to policy and proceedings. Later in this section, we
recommend that EPIC be independently evaluated regularly, providing an
opportunity for on-going assessment of Program planning and implementation.

11.2.3 Stakeholder Engagement
The CEC provides comprehensive information about its Investment Plans to a broad
array of stakeholders.
The IOUs have a much narrower stakeholder group on which they typically rely for
input.
Stakeholders (besides the Electric Power Research Institute [EPRI]) are engaged
relatively late in both the CEC’s and IOUs’ investment planning processes, and plans
have not changed significantly as a result of stakeholder input.
The CEC provides comprehensive information to stakeholders about its Investment Plans,
and its processes are consistent with other peer RD&D programs.
The internal IOU stakeholders and EPRI (an electric utility-focused organization) are the
main sources of expertise on which the IOUs rely to determine their EPIC investment
priorities. EPRI identifies gaps and any redundancies with other utility efforts nationwide.
The EPIC administrators hold stakeholder workshops, document public comments and
respond to these remarks in each of their Investment Plans, as required by the CPUC.
However, the IOUs do not provide comprehensive information about their draft plans
when they conduct stakeholder workshops, and, according to stakeholders, allow little
time for input. The CEC also allows little time for input, though it offers more information
and gives more time for input than the IOUs. Compared to other peer RD&D programs,
the EPIC administrators appear to rely more on their own internal technical experts (and
for the IOUs, EPRI), seeking input from external stakeholders after investment planning
goals are established. However, we note that the administrators’ internal staff routinely
collaborate with other external subject matter experts. The peer RD&D programs we
reviewed engaged industry experts to help shape the focus of their initiatives. We
recommend that:
• 2d) The administrators engage more stakeholders earlier in the investment planning
process, and
• 2e) The IOUs provide more comprehensive information, to allow time for more
meaningful engagement. With the current IOU approach, the Investment Plans are
so close to final that stakeholder input at workshops does not materially change
their plans. This issue is exacerbated by the fact that once the plan is approved, little
information is shared with the public until the projects are described in the IOUs’
Annual Reports.

11.3 Project Selection Process
The key research question related to project selection by the administrators that we sought
to address is:
Is the project selection process being conducted in an open, effective, and efficient manner and
resulting in funds going to projects that are consistent with EPIC policy objectives and
planning processes?
We address this research question in the next five sub-sections.
11.3.1 Administrator Project Selection Processes

The CEC has established rigorous selection criteria, strong alignment with overall
program goals and objectives, and a peer review process for selecting and awarding
project grants.
There is a lack of transparency in the IOUs’ project selection and research planning
processes. The IOU project selection criteria could also be more transparent.
The CEC’s project selection processes are rigorous and transparent, and are consistent
with other peer RD&D programs.
Based on our review of IOU processes, we find that the IOUs’ project selection criteria are
not as robust as other peer RD&D programs or the CEC’s, which identifies a series of
strategic objectives around which initiatives are organized. While the IOUs are technically
in compliance with investment planning, project selection and annual reporting
requirements, they are not as robust and transparent as other peer RD&D programs. We
recommend that:
• 3a) The IOUs develop more transparent project selection criteria, which determine
the project areas that are described in their Investment Plans as well as the specific
projects that are eventually implemented. Once the CPUC establishes priorities,
these criteria could be reviewed and revised over time to ensure an appropriate
focus on the highest priority areas for advancing state energy policy.
There is typically a substantial lag between the time when the IOUs decide whether or not
to launch a project once their Investment Plans are approved and when they share
information about each project in their Annual Reports, including such information as the
budget, overview of project scope, and its current status (active, completed, cancelled or
on hold). We do note that SDG&E provides project budget information (including a
breakdown of SDG&E versus vendor budget) in its Investment Plans.
The IOUs do not share their detailed project research plans publicly or with the CPUC, so
there is less transparency as compared to CEC projects (where detailed scopes of work are

posted publicly once they are developed). The brief description in the Investment Plans
and Annual Reports are all that is available to the CPUC and the public (with the
exception of projects that are featured in presentations at EPIC workshops and the annual
symposium or ad hoc communications). The IOUs may not conduct all projects that are
described in the Investment Plans, so the Annual Reports are the best sources for
determining which projects are being implemented. Though the IOUs comply with EPIC
program requirements, we have identified the need for more timely reporting on projects
after the Investment Plans are approved to increase transparency and ensure more
effective CPUC oversight. We recommend that:
• 3b) The IOUs share project research plans and budgets with the CPUC and the
public, at least one month prior to launch.
11.3.2 Administrator Coordination

All four EPIC administrators coordinate program administrative activities on a regular
basis, and the IOUs coordinate on individual projects; however, there is less
coordination between the IOUs and the CEC at the project planning level. We have
identified the potential for future increased coordination between the IOUs and the
CEC once Applied R&D projects begin to mature.
While the IOUs have effectively coordinated on the EPIC 1 and EPIC 2 project
portfolios, we found some cases of apparent duplication in the IOUs’ EPIC 3 plans.
The administrators share project results with each other informally (e.g., during their
routine biweekly coordination meetings) and formally (by attending project
presentations). There is the potential for more project coordination once the CEC’s
Applied R&D projects begin to mature. (We offer a related recommendation in Section
11.5.)
To minimize duplication, the IOUs coordinate with each other on project plans. There are
cross-IOU technology teams that coordinate, and the Program provides a unifying
framework for them to test technologies before they go to production scale.
We reviewed the IOUs’ three Investment Plans. In the first two plans, we did not identify
any unnecessary duplication. However, we did note some cases of apparent duplication in
the IOUs’ EPIC 3 plans. Once the projects are launched and more detailed research plans
are developed, the IOUs should make those plans available (similar to what the CEC does)
to allow for more vetting of projects (which we recommended above). We recommend
that:
• 3c) The CPUC review the IOUs’ project research plans (which we have
recommended that they make public as they are developed) to ensure that there is
no unnecessary duplication in their EPIC 3 projects.

11.3.3 Match Funding
The CEC explicitly seeks match funding (consistent with three of the seven peer RD&D
programs), while the IOUs use informal means for attracting cost sharing from their
vendors.
Some stakeholders indicated the CEC’s match funding requirements may be too
onerous, especially for small companies that lack the resources or track record to attract
such funding.82 Moreover, our review of best practices of other peer RD&D programs
found that two primary peer programs which focus on small businesses, the SBIR and
STTR Programs, do not mandate match funding during the projects’ Phase I
implementation, but it is encouraged during Phase II when these projects are more
established and better positioned to secure such funding. We recommend that:
• 3d) The CEC consider modifying the match funding requirement for TD&D projects
and make it optional, giving bonus points to their scores like they typically do for
Applied R&D and Market Facilitation projects, to ensure they are not rejecting good
projects that are unable to secure funding (such as those from small businesses).
This might encourage more bids, especially from small businesses that in general
have fewer options for securing match funding.
11.3.4 Intellectual Property Terms

Some stakeholders reported that the Program’s Intellectual Property (IP) terms were a
barrier to participation. We observed that peer RD&D programs have more flexibility, but
EPIC is very different from its peers because of the legislatively mandated ratepayer
benefit requirement, which effectively constrains how IP may be treated for EPIC-funded
projects. However, it is possible that there may be more flexibility than what the CPUC has
communicated in EPIC Decisions. We recommend that:
• 3e) The CPUC review IP rules or guidance developed for the Department of Energy’s
Small Business Innovation Research (SBIR) Program to explore possible
opportunities for easing IP requirements. Regardless of the outcome of any such
efforts, the CPUC should ensure that IP requirements are communicated effectively.
82 This barrier was mentioned by five of the 12 respondents who submitted comments to the Idea Exchange
which followed the September 22, 2016 public workshop on increasing private sector participation. The 12
respondents included six representatives from the private sector, three from industry organizations, one
from an academic institution, and one from a National Lab. Although we did not explicitly address this issue
during the interviews with 17 TD&D project grantees and administrators, none of them volunteered that this
is a barrier to applying to EPIC.

11.3.5 Flexibility
Despite the IOUs' assertion that the need to submit an Advice Letter to add, modify or
cancel a project is onerous, the evaluation team found that the IOUs have sufficient
flexibility to make changes to projects that have been described in their Investment
Plans as well as to put projects on hold indefinitely.83
The CEC differs from other peer RD&D programs in that it does not always have the
necessary flexibility to adjust a project scope of work to respond to rapid changes in
technologies and markets.
The administrators are permitted to add, modify or cancel a project by filing a Tier 3
Advice Letter with the CPUC. The IOUs indicated this is an onerous requirement since the
approval process takes several months. They would need to submit an Advice Letter to
add a completely new project that is not covered by one of the existing general
descriptions in their Investment Plans. In such cases (we identified two out of six projects
that PG&E included in their Tier 3 Advice Letter, with justifiable reasons for adding them
mid-cycle), the lengthy review period could be a problem. However, before the CPUC
considers any changes to this process, we recommend that the IOUs address the issues we
have identified in this report related to transparency and information sharing. We
recommend that:
• 3f) The administrators should use the Advice Letter process only for requesting
substantive changes to projects or adding new projects that are not covered by one
of the existing general descriptions in their Investment Plans. Recently, PG&E
submitted an Advice Letter that included three projects that could have been
implemented within the existing approved project descriptions in its Investment
Plan. The administrators should only introduce new projects in between Investment
Plans (necessitating a lengthy Advice Letter approach) in response to new market
or technology developments. Other projects, such as a call center-related project
submitted by PG&E, that are not new projects being introduced mid-cycle in
response to market or technological developments, could be better anticipated and
included in the investment planning process. The Tier 3 Advice Letter approach
takes time, and should be limited only to projects when it is necessary.
As a public agency, the CEC does not always have the necessary flexibility to adjust a
project scope of work to respond to rapid changes in technologies and markets. Other peer
RD&D programs have structures in place to help projects identify and capitalize on
opportunities to change course, when needed, to maximize a projects’ success (e.g., the
SBIR Program allows projects to be able to reallocate up to 10 percent of their budget
83in Section 8.4, we discussed Investment Plan consistency over the plan period, which includes a discussion
of the Tier 3 Advice Letter requirement and the process for seeking approval for changes.

without prior approval). Because the technology innovation process is not linear, it often
requires program administrators, grantees and others involved in the RD&D process to be
able to quickly modify their scopes of work to maximize benefits. We recommend that:
• 3g) The CEC explore how and whether it could add more flexibility to its grant
request forms and/or research planning process to be able to respond to market and
technology changes that occur between the time the project is proposed and the
project is launched.
11.4 Project Assessment Process

The first key research question related to the assessment of EPIC projects that we sought to
address is:
What is the status of EPIC investments?
As of the end of 2016, based on the project status reports included in the administrators’
2016 Annual Reports, 250 projects were active and 19 projects have been completed (1 CEC
project and 18 IOU projects). Of active projects, about half are scheduled to be completed
by the end of 2018 and the remainder in 2019 or later.
11.4.1 Project Status Reports

The second key research question related to the assessment of EPIC projects that we
sought to address is:
Do the administrators do everything possible to track the progress of funded work?
The administrators each have adequate processes in place to internally track the
progress of projects and ensure effective project implementation.
All administrators could improve upon the frequency, usefulness and transparency of
project status reports.
The administrators internally track project progress frequently, and their processes are
consistent with other peer RD&D programs. While the administrators are in compliance
with EPIC program requirements, we identify the need for more frequent and effective
project-status reporting (such as on a quarterly basis, with projects categorized by policy
and/or technology area). With a four-administrator model, it is more difficult to classify
and summarize projects across the whole EPIC portfolio. The Annual Reports, the
primary way that the CPUC and stakeholders monitor project status, consists of four
different reports, posted to four different websites and distributed to different listservs,
without any categorization of projects (beyond by investment period and investment
area), such as by policy or technology area.

This process is not effective and impedes the ability of stakeholders and the public to fully
engage with the Program. Providing key project information on a more frequent basis is
consistent with the peer RD&D programs we reviewed. We recommend that:
• 4a) The administrators share information while projects are in progress with the
CPUC and the public on a more frequent basis, such as quarterly.
• The administrators collaborate in categorizing and summarizing projects, as
previously recommended (2b), (such as by technology type and/or policy area) so
that interested parties can more easily obtain pertinent information on a given topic
area. See Appendix D for an example of a quarterly status report that we developed
that provides a foundation for such a report.
• 4b) The administrators collaborate and jointly convene a quarterly workshop to
share results about project status and lessons to-date on a topical basis, with
engagement from stakeholders on topics that are of interest. Such workshops
should be publicized in advance along with the topic or topics to be covered. All
EPIC projects that fall under the announced topic should be discussed and
organized topically. This process will ensure that: 1) information about EPIC
projects is conveyed to the appropriate audiences, and 2) stakeholders can better
anticipate the types of information that will be shared at EPIC workshops and thus
be better prepared to participate in discussions about future research needs and
EPIC investment areas.
11.4.2 Benefits Quantification

The third key research question related to the assessment of EPIC projects that we sought
to address is:
Are processes in place to assess project viability over time and disseminate project results to
stakeholders?
The CEC has an effective, structured and transparent process in place for tracking
project benefits.
The IOUs are not effectively tracking and reporting on benefits metrics.
We identified a need for the administrators to coordinate on compiling and jointly

reporting on project benefits.
The CEC’s process for tracking project benefits consists of three project benefits
questionnaires that grantees are required to complete. These processes are consistent with
other peer RD&D programs. Though the CEC’s projects are not very far along, the first set
of questionnaires that were prepared when projects were initiated appear to be sufficiently
and reasonably well documented quantitatively and/or qualitatively.

The IOUs track and report on project results and provide some information related to
project benefit metrics in their project closeout reports, as required by the CPUC.
However, there is room for improvement with respect to measuring project success and
reporting on the metrics. They may need to conduct additional data and information
gathering in order to estimate and report project benefits, which is a consistent practice
among other peer RD&D programs. We recommend that:
• 4c) The IOUs develop more detailed processes to quantify benefits associated with
their projects, including what types of data would be necessary and how they will
collect these data, as well as a reporting structure and process that would document
and report those benefits to all relevant stakeholders. The IOUs should include a
plan to collect and report on project benefits metrics in their project scopes of work,
and analyze and report on benefits in their project closeout reports and follow-up
reports as necessary (since some benefits may take more time after project
completion before they can be quantified).
We have identified a need for the administrators to coordinate on compiling and reporting
on project benefits. Such reporting should be shared with the CPUC, key stakeholders and
the general public to widely publicize the Program’s collective benefits, consistent with
peer RD&D program practices. Unlike EPIC, peer RD&D programs have a single
administrator, making it much easier to produce a single report publicizing program
benefits. It is more challenging for EPIC to categorize and summarize project benefits
across the four administrators. We recommend that:
• 4d) The administrators develop a process to jointly report on EPIC's short-, mid-
and long-term project benefits across the portfolio on a routine basis (e.g., annually)
to the CPUC, relevant stakeholders and the general public.
11.4.3 Results Dissemination

The CEC has a robust process in place to disseminate project results and track
knowledge gained.
PG&E has disseminated project results widely, while SCE has not. (SDG&E had not yet
completed projects at the time of our research.)
The four-administrator model for EPIC may create some barriers and limitations to
information dissemination since there is no program-wide communications mechanism
or central repository of project information.
The CEC’s robust results dissemination processes are consistent with peer RD&D
programs. However, since it is premature to determine whether the CEC’s project
dissemination tracking processes are effective at achieving their mid- and long-term
objectives, we recommend that:

• 4e) The CEC’s project benefits quantification processes be reviewed again once
more projects are completed.
The IOUs document the results of their dissemination plans in their project closeout
reports by identifying presentations they have made or plan to make. SDG&E had not yet
submitted any closeout reports at the time of our assessment, but PG&E appeared to
identify several external conferences to share results, whereas SCE reported far fewer
information sharing plans. These results are consistent with analysis we conducted on
information dissemination by IOU, with SCE sharing information about projects much less
frequently compared to PG&E and SDG&E. SCE should make additional efforts to
disseminate information about its EPIC projects to ensure project results are widely
shared. We recommend that:
• 4f) SCE share its project results more widely with interested stakeholders,
including delivering presentations at conferences and workshops.
• 4g) SDG&E’s project closeout reports be reviewed once projects are completed to
ensure results are being widely disseminated.
While the administrators do collaborate on EPIC symposia, outreach is conducted

separately even though there is possible overlap in stakeholder interest across
technologies investigated by the administrators. Similarly, while the CEC Media Office is
an additional source by which the public learns about CEC projects, the Office does not
disseminate any information about IOU projects. Each administrator uses its own website
and listserv(s) to distribute information. Given these inefficiencies of the four-
administrator model, we recommend that:
• 4h) The administrators jointly develop a single EPIC website and listserv to post
and distribute project information (including quarterly project status reports,
project closeout reports and any additional documents on project benefits and
knowledge gains) for both the CEC and IOU projects. Included on this website
would be a single, downloadable Excel spreadsheet that contains key information
for all EPIC projects. This would ensure that stakeholders have an easy way to
obtain all relevant information about EPIC projects that support their particular
areas of interest. More stakeholders might engage with the Program if it is easier for
them to receive and organize all relevant information about the performance of
individual projects as well as the entire EPIC portfolio.
11.4.4 Project Networks

The administrators have been effective even at this early stage of the Program’s
implementation in developing networks and disseminating knowledge.

The CEC engages with more organizations than the IOUs, with much more focus on
public organizations, corroborating previous findings in this report about the much
narrower focus of IOU projects and the lack of transparency of IOU projects compared
to the CEC.
Our analysis of knowledge dissemination activities and the relational networks of each of
the sampled interview projects suggests that projects are developing broad networks of
stakeholders and other market actors. The wide range of entities that is already engaged in
projects suggests that the projects are well positioned to lead to wide dissemination of
knowledge once projects are completed.
On average, the CEC engages with 18 organizations per project during implementation,
while IOUs engage with 7. The type of organizations with which the CEC and IOUs
engage also differs to a great extent. Ninety percent of organizations that the IOUs engage
are private compared to 36 percent for CEC TD&D projects. The IOUs’ networks consist
mainly of manufacturers, private companies and other utilities, with very few
government/policy-making organizations included. The CEC engages more often with its
project networks as compared to the IOUs, which may be due to having fewer formal
relationships with external organizations (20 percent formal agreements in place for the
IOUs versus 57 percent for the CEC).
• We have no recommendation on whether the IOUs should engage a higher

percentage of public agencies in line with the CEC’s approach, since the program
rules regarding the dissemination of information do not distinguish between public
versus private organizations.
11.5 Project Impacts and Policy Alignment

The key research questions related to project impacts and how they align with policy that
we sought to address are:
Are ongoing projects showing reasonable indicators of success? Looking beyond project- and
administrator-specific considerations, what impacts does the Program overall have in a wider
context?
Overall, the EPIC portfolio appears to be on track, thus far, in meeting its short-, mid-
and long-terms objectives. Collectively, EPIC is both broad and deep, and
administrators take steps to integrate projects into the broader innovation and policy
landscape. To that extent, projects appear to be consistent with the Program’s objectives
and core values.
The Program as a whole is not consistent with other peer RD&D programs, which have
a much greater focus on support for commercialization. However, we note that EPIC

reflects the diversity of projects encouraged by CPUC Decisions, which contain a more
broadly defined set of objectives.
We reiterate the need to prioritize the principles, policies and strategic objectives and
operationalize what it means for a portfolio to be optimized.
Overall, when viewed as a portfolio, EPIC’s Applied R&D projects show some progress
with respect to knowledge and awareness among potential users, follow-on research and
development, technology demonstrations, potential private investment, patents and early
adoption. In general, the TD&D portfolio also appears to be on track, thus far, in meeting
its short-, mid- and long-terms objectives.
EPIC’s Market Facilitation projects are clearly consistent with the goal of addressing non-
technical barriers with projects aimed at enhancing permitting and market and technical
analysis predominating, thus far. We found that projects generally are on track to achieve
their research objectives and expected benefits. Many also reported that these same
benefits could be achieved if these projects were replicated in other states and
jurisdictions. We did note that there is a disconnect between the Market Facilitation
projects and the Applied R&D and TD&D projects that are upstream from them. This
disconnect is primarily due to the fact that all three program areas were launched more or
less at the same time, making it extremely difficult to develop projects that are aimed to
meet the needs of Applied R&D and TD&D projects. The CEC has recognized this gap and
reported that it is committed to working more closely with those responsible for the
upstream projects to better meet their needs.
We observed that that the EPIC portfolio of projects is highly diverse, which is consistent
with CPUC Decisions, which contain a broadly defined set of objectives. We characterized
this diversity in terms of the types of technologies and studies and their commercialization
status. We found that the diversity of projects with respect to technologies and
commercialization in the EPIC portfolio adequately addresses EPIC’s multiple policy
goals. However, whether each administrator is expending the right level of effort to
address each of these policy goals, delivering project results quickly enough and bringing
important innovations to market or the public domain cannot be determined until these
policies are prioritized. It is also clear that portfolio optimization is an on-going process
that must respond to changing priorities and budgets in a rapidly changing technological
and environmental landscape.
We recommend that:
• 5a) The CPUC consider using our characterization of the EPIC portfolio in terms of
the types of technologies and studies and their commercialization status as
baselines against which to compare future iterations of EPIC.

• 5b) The CPUC regularly evaluate EPIC to confirm that the CEC is ensuring the
Market Facilitation projects are effectively connected to and serving the needs of the
Applied R&D and TD&D projects.
We identified future opportunities for more coordination at the project level between
the CEC and the IOUs, with the IOUs considering demonstration projects that could build
on CEC Applied R&D projects, once those are further along. Given the lack of extensive
coordination between the IOUs and the CEC at the project level, such opportunities may
not happen unless a more formal process is introduced. We recommend that:
• 5c) EPIC administrators establish a process to ensure that once Applied R&D
projects are completed by the CEC, the results are considered and potential TD&D
projects are identified. Such a process would ensure that projects that have evaded
the Technological Valley of Death do not subsequently fall into the
Commercialization Valley of Death. Such a view is consistent with the underlying
theory that supports the three phases of the administrator’s innovation pipeline.
11.6 Overarching Coordination and Collaboration

Our evaluation has identified a critical need for improving administrative coordination
and stakeholder engagement that the administrators are not currently addressing due to
limitations associated with the administrative model and their reliance on minimum
project reporting procedures. We have identified a need to explicitly supplement the
existing administrative structure by convening an independent body that provides
coordination and facilitation support to the administrators and compiles and helps
disseminate information. Such efforts would increase transparency and stakeholder
engagement and ensure the Program is most effectively directing EPIC funds toward
energy innovation that meets the highest priority state policy goals, as identified by the
CPUC. These efforts would also address the deficiencies we outlined in the previous two
sets of findings. We recommend that:
• 6a) The CPUC and/or the administrators fund and convene an independent body to
coordinate, facilitate and lend technical expertise. The responsibilities of such a
body could include:
o Supporting administrator and CPUC efforts to categorize projects by technology
and/or policy areas, in order to facilitate easier access of EPIC project
information for interested stakeholders;
o Convening and engaging stakeholders earlier in the investment planning
process;
o Supporting administrators in collecting data regarding key performance metrics
in a consistent manner.

o Supporting administrator and CPUC efforts to track and prioritize policy goals
and funding criteria, and periodically revisiting priorities as policy goals change
and EPIC matures;
o Supporting administrator and CPUC efforts to ensure that those priorities are
effectively addressed in the administrators’ Investment Plans;
o Supporting administrator and CPUC efforts to review Investment Plans and
project research plans to ensure there is no unnecessary duplication (particularly
for the IOUs’ TD&D projects);
o Engaging stakeholders and ensuring any input that would lead to greater
ratepayer and state policy benefits is considered by the administrators in their
Investment Plans;
o Reviewing administrator quarterly status reports, such as by policy and/or
technology areas, engaging relevant stakeholders in projects of interest, and
helping to identify issues or concerns;
o Planning and facilitating a quarterly meeting devoted to a particular topic of
interest to stakeholders, including publicizing the meetings to stakeholders and
addressing their needs;
o Coordinating an effort to develop a centralized EPIC website and listserv,
helping to identify interested parties and ensuring that those parties are linked
to relevant information on projects and topic areas of interest;
o Reviewing project research plans and ensuring consistency with the Investment
Plans, and supporting the engagement of relevant stakeholders in reviewing
research plans;
o Identifying interested stakeholders and appropriate forums for administrators
to more broadly disseminate their results; and
o Facilitating coordination at the project planning level between the CEC and
IOUs once the CEC’s Applied R&D projects are further along.
11.7 On-Going Program Evaluation

We have identified the need for on-going assessment of the program and project
benefits.
The various planning documents and CPUC Decisions combined with the comments from
EPIC project teams support the conclusion that due diligence is being done to identify
projects that, absent EPIC funding, would not move forward or move forward more
slowly. Again, this is a necessary but not sufficient condition for establishing a causal
connection between EPIC-funding activities and any measureable changes in metrics
associated with the mid- and long-term outcomes. The probability that these outcomes
will eventually be achieved will increase substantially only through the combination of
successfully designed and implemented projects and the diffusion of these technologies

and tools over time in their targeted markets. While projects appear to be on a good
trajectory to disseminate knowledge, it is too early in the program life to determine the full
extent of more concrete knowledge benefits such as patents, journal articles and existence
of follow-on research. It is also premature at this time to assess if these activities will be
sufficient to encourage further research and technology adoption after the EPIC projects
are completed. A more complete case for attribution must be built on an on-going, theory-
driven evaluation that, based on the preponderance of the evidence, will assess the extent
to which these mid- and long-term outcomes have been achieved and the extent to which
EPIC is responsible for any of these achievements. We therefore recommend that:
• 7a) Using the theory-driven framework developed for this evaluation, monitor and
report key performance metrics on an on-going basis and conduct a comprehensive
evaluation every three to four years. All of these evaluation activities should be
conducted by an independent evaluator in close collaboration with the four
administrators to avoid any duplication of efforts and to ensure that the results
will be useful to all stakeholders (e.g., the CPUC, state legislators, and the four
administrators and other stakeholders). While this evaluation report documents
what is working and what could be improved, the Program is still very young and
should undergo ongoing independent assessment to ensure it remains on track and
addresses the issues we have noted. Moreover, most projects have yet to be
completed, and independent review is needed in the future to assess project
benefits as the Program matures. Conducting independent program evaluations is
consistent with the best practice of peer RD&D programs.
• 7b) The administrators create a single, centralized database containing all relevant
information on active and completed EPIC projects along with monitoring and
quarterly reporting of key performance metrics, in order to support the on-going
evaluation of the Program.
We also have identified a need to better characterize the Program to support future
assessment efforts.
Our initial understanding of EPIC was based on the Triennial Investment Plans and other
CPUC documents. However, through our interviews, document reviews and analysis of
the EPIC portfolio, we now understand that the Program is far more complex. This theme
of greater complexity is woven throughout this report. Below, we highlight the key areas
of complexity.
EPIC projects do not follow a linear process. EPIC is not characterized by a linear
progression from Applied R&D projects to TD&D projects and on to Market Facilitation
projects, as we expected based on our review of the CEC’s Investment Plans. The best
practices literature noted that the technology innovation process is not linear; it requires
program stakeholders, including administrators, implementers and other partners to be
adept at identifying and capitalizing on opportunities as they arise. For example, Chiavari

and Tam note that feedback from the market and technology users during
commercialization and diffusion phases can influence additional RD&D and promote
continuous innovation.84 The literature also indicated that monitoring of early-phase
products or pilots can help inform program design and implementation.85 Mazzucato also
notes that:
"The causation that occurs in the steps taken from basic science to large-scale R&D, to
applications, and finally to diffusing innovations is not linear. Rather, innovation
networks are full of feedback loops existing between markets and technology,
applications and science. In the linear model of innovation, the R&D system is seen as
the main source of innovation, reinforcing economists’ use of R&D stats to understand
growth. In this more non-linear view, the roles of education, training, design, quality
control and effective demand are just as important." 86
We understand why the linear model remains so appealing.
"Although the linear model does not represent the reality and complexity of the
innovation process, it has been particularly influential and remains widely used
because it offers a(n) (over)simplified description of the innovation process and allows
the spokesmen for the economic and scientific community to communicate their
thoughts to the general public and policy makers in an understandable (though
flawed) way." 87
While the administrators fully appreciate the complexity of the EPIC Program, they should
develop logic models (building on the ones we developed) and associated performance
metrics that better reflect a shared understanding of this complexity among themselves,
the CPUC, and key stakeholders. Such logic models can provide an effective framework
for identifying the underlying assumptions and theories as well as the potential
performance metrics that can better test these assumptions and theories and track progress
towards the short-, mid- and long-term outcomes. This framework can also assist all
stakeholders in interpreting the large amount of program performance data that has been
and will continue to be collected throughout the life of the program. In collaboration with
the CPUC and their independent evaluators, the administrators should also develop a
84 Joana Chiavari and Cecilia Tam, Good Practice Policy Framework for Energy Technology Research, Development
and Demonstration (RD&D). International Energy Agency, 2011.
https://www.iea.org/publications/freepublications/publication/good_practice_policy.pdf
85 Bhavya Lal, Nayanee Gupta, and Christopher L. Weber, “Innovation Pipeline Management: Lessons
Learned from the Federal Government and the Private Sector” (Washington, DC: IDA Science & Technology
Policy Institute, 2012). https://www.ida.org/idamedia/Corporate/Files/Publications/STPIPubs/2014/D-
5367.ashx; Chiavari and Tam, Good Practice Policy Framework.
86 Mariana Mazzucato, The Entrepreneurial State (New York: Public Affairs, 2015), 43.
87 D. Edgerton, “The Liner Model Did Not Exist,” in K. Grandin, N. Worms and S. Widmalm (eds). The
Science Industry Nexus: History, Policy, Implications (New York: Science History Publication, 2004).

more condensed set of metrics that the administrators could track and report annually to
provide more timely feedback on high priority areas.
As an example of this nonlinearity, we noted previously in this section that there is a

disconnect between the Market Facilitation projects and the Applied R&D and TD&D
projects that are upstream from them. In the future, one would expect that the Market
Facilitation Commission Agreement Managers (CAMs) will better assess the needs of the
Applied R&D and TD&D CAMs as well as the IOU TD&D project managers, conduct
relevant research to meet these needs and provide feedback that might trigger changes in
the emphasis or direction of the project-level research and strategies for commercialization
or wider adoption in the public domain of EPIC innovations. Such needs assessments and
feedback loops need to be recognized and their effectiveness tested periodically.
The sources of proposed Applied R&D and TD&D projects are diverse. The CEC’s
Triennial Investment Plan stated that once Applied R&D projects are completed, they
continue through the pipeline as a TD&D project and, once finished, their market-related
research questions are then addressed through Market Facilitation research projects. In
fact, TD&D project proposals come from many sources including internal staff, EPRI and
manufacturers, as well as public and private R&D laboratories. None of the TD&D projects
thus far have originated in the EPIC Applied R&D portfolio. This is not surprising since
none of the Applied R&D projects were completed at the time of our research. Over time,
more and more TD&D projects are expected to originate in the Applied R&D portfolio.
The question of the expected proportion of EPIC TD&D projects that should originate in
the EPIC Applied R&D portfolio should be addressed by the CPUC.
The types of projects are diverse. The Triennial Investment Plan stated that two of EPIC’s
primary objectives are to avoid the Technological Valley of Death (for Applied R&D
projects) and/or the Commercialization Valley of Death (for TD&D projects), implying
that the primary objective of every EPIC project is commercialization. As indicated in
Section 10.5.1, about 60 percent of the Applied R&D projects do not have
commercialization as an objective, while about 95 percent of the CEC TD&D projects and 4
percent of the IOU TD&D projects have commercialization as an objective. Even within
these two general categories of intend to commercialize and do not intend to commercialize,
projects are diverse, with some involving a combination of commercially available
technologies applied to an emerging problem while others are developing products such
as forecasting software that are intended for the public domain.
Given this, we recommend that the CPUC work with the administrators with the support
of the recommended coordination body (if created) to:
• 7c) Modify (and continually update as needed) the characterization of the Program
to more accurately reflect its complexity.

• 7d) Modify (and continually update as needed) the EPIC program theory and logic
models to better reflect the more complex character of the Program.
• 7e) Revisit the key performance metrics that should be tracked and the frequency
with which they should be tracked and reported.

12 Appendix A: CPUC Decision Detail
1. CPUC Decision 13-11-025 Attachment 4 (List of Potential
Evaluation Areas)
List of Proposed Metrics and Potential Areas of Measurement (as applicable to a
specific project or investment area in applied research, technology demonstration, and
market facilitation)
1. Potential energy and cost savings
a. Number and total nameplate capacity of distributed generation facilities
b. Total electricity deliveries from grid-connected distributed generation facilities
c. Avoided procurement and generation costs
d. Number and percentage of customers on time variant or dynamic pricing tariffs
e. Peak load reduction (MW) from summer and winter programs
f. Avoided customer energy use (kWh saved)
g. Percentage of demand response enabled by automated demand response technology

(e.g. Auto DR)
h. Customer bill savings (dollars saved)
i. Nameplate capacity (MW) of grid-connected energy storage
2. Job creation
a. Hours worked in California and money spent in California for each project
3. Economic benefits.
a. Maintain / Reduce operations and maintenance costs
b. Maintain / Reduce capital costs
c. Reduction in electrical losses in the transmission and distribution system
d. Number of operations of various existing equipment types (such as voltage

regulation) before and after adoption of a new smart grid component, as an indicator
of possible equipment life extensions from reduced wear and tear

e. Non-energy economic benefits
f. Improvements in system operation efficiencies stemming from increased utility

dispatchability of customer demand side management
g. Co-benefits and co-products (e.g. feed, soil amendment, lithium extraction)
h. Energy Security (reduced energy and energy-related material imports)
4. Environmental benefits
a. GHG emissions reductions (MMTCO2e)
b. Criteria air pollution emission reductions
c. Water savings
d. Water quality improvement
e. Waste reductions
f. Habitat area disturbance reductions
g. Wildlife fatality reductions (electrocutions, collisions)
5. Safety, Power Quality, and Reliability (Equipment, Electricity System)
a. Outage number, frequency and duration reductions
b. Electric system power flow congestion reduction
c. Forecast accuracy improvement
d. Public safety improvement and hazard exposure reduction
e. Utility worker safety improvement and hazard exposure reduction
f. Reduced flicker and other power quality differences
h. Reduction in system harmonics
i. Increase in the number of nodes in the power system at monitoring points
6. Other Metrics (to be developed based on specific projects through ongoing

administrator coordination and development of competitive solicitations)

7. Identification of barriers or issues resolved that prevented widespread deployment of
technology or strategy
a. Description of the issues, project(s), and the results or outcomes
b. Increased use of cost-effective digital information and control technology to improve

reliability, security, and efficiency of the electric grid (PU Code § 8360)
c. Dynamic optimization of grid operations and resources, including appropriate

consideration for asset management and utilization of related grid operations and
resources, with cost-effective full cyber security (PU Code § 8360)
d. Deployment and integration of cost-effective distributed resources and generation,

including renewable resources (PU Code § 8360)
e. Development and incorporation of cost-effective demand response, demand-side

resources, and energy-efficient resources (PU Code § 8360)
f. Deployment of cost-effective smart technologies, including real time, automated,

interactive technologies that optimize the physical operation of appliances and
consumer devices for metering, communications concerning grid operations and
status, and distribution automation (PU Code § 8360)
g. Integration of cost-effective smart appliances and consumer devices (PU Code §

8360)
h. Deployment and integration of cost-effective advanced electricity storage and peak-

shaving technologies, including plug-in electric and hybrid electric vehicles, and
thermal-storage air-conditioning (PU Code § 8360)
j. Provide consumers with timely information and control options (PU Code § 8360)
k. Develop standards for communication and interoperability of appliances and

equipment connected to the electric grid, including the infrastructure serving the
grid (PU Code § 8360)
l. Identification and lowering of unreasonable or unnecessary barriers to adoption of

smart grid technologies, practices, and services (PU Code § 8360)
8. Effectiveness of information dissemination

a. Web-based surveys of people viewing materials or participating in program reviews
b. Number of reports and fact sheets published online
c. Number of times reports are cited in scientific journals and trade publications for
selected projects
d. Number of information sharing forums held
e. Stakeholders attendance at workshops
f. Technology transfer
9. Adoption of EPIC technology, strategy, and research data/results by others
a. Description/documentation of projects that progress deployment, such as

Commission approval of utility proposals for widespread deployment or
technologies included in adopted building standards
b. Number of technologies eligible to participate in utility energy efficiency, demand

response or distributed energy resource rebate programs
c. EPIC project results referenced in regulatory proceedings and policy reports
d. Successful project outcomes ready for use in California IOU grid (Path to market)
e. Technologies available for sale in the market place (when known)
10. Reduced ratepayer project costs through external funding or contributions for EPIC-
funded research on technologies or strategies
a. Description or documentation of funding or contributions committed by others
b. Co-funding provided for solicitations
c. Dollar value of funding or contributions committed by others

2. CPUC Decision 12-05-037 Statutory Guidance
Public Utility Code 740.1 Statutory Guidance
The commission shall consider the following guidelines in evaluating the research,
development, and demonstration projects proposed by electrical and gas corporations:
(a) Projects should offer a reasonable probability of providing benefits to ratepayers.

(b) Expenditures on projects which have a low probability for success should be
minimized.
(c) Projects should be consistent with the corporation’s resource plan.
(d) Projects should not unnecessarily duplicate research currently, previously, or
imminently undertaken by other electrical or gas corporations or research
organizations.
(e) Each project should also support one or more of the following objectives:
1. Environmental improvement.
2. Public and employee safety.
3. Conservation by efficient resource use or by reducing or shifting system load.
4. Development of new resources and processes, particularly renewables resources and
processes which further supply technologies.
5. Improve operating efficiency and reliability or otherwise reduce operating costs.
Public Utility Code 8360 Statutory Guidance
It is the policy of the state to modernize the state's electrical transmission and distribution
system to maintain safe, reliable, efficient, and secure electrical service, with infrastructure
that can meet future growth in demand and achieve all of the following, which together
characterize a smart grid:
(a) Increased use of cost-effective digital information and control technology to improve
reliability, security, and efficiency of the electric grid.
(b) Dynamic optimization of grid operations and resources, including appropriate
consideration for asset management and utilization of related grid operations and
resources, with cost-effective full cyber security.
(c) Deployment and integration of cost-effective distributed resources and generation,
including renewable resources.

(d) Development and incorporation of cost-effective demand response, demand-side
resources, and energy-efficient resources.
(e) Deployment of cost-effective smart technologies, including real time, automated,
interactive technologies that optimize the physical operation of appliances and
consumer devices for metering, communications concerning grid operations and
status, and distribution automation.
(f) Integration of cost-effective smart appliances and consumer devices.
(g) Deployment and integration of cost-effective advanced electricity storage and peak-
shaving technologies, including plug-in electric and hybrid electric vehicles, and
thermal-storage air-conditioning.
(h) Provide consumers with timely information and control options.
(i) Develop standards for communication and interoperability of appliances and
equipment connected to the electric grid, including the infrastructure serving the grid.
(j) Identification and lowering of unreasonable or unnecessary barriers to adoption of
smart grid technologies, practices, and services.

13 Appendix B: Logic Models
This section presents the more detailed and complex Administration, Applied R&D,
TD&D and Market Facilitation logic models that guided our identification of multiple
performance metrics that informed our data collection plan.
13.1 Administration Logic Model

Numerous administrative activities support the Program’s Applied R&D, TD&D and
Market Facilitation investment areas. Comprehensive program administration by the CEC
and the IOUs is required to select and fund strategic research projects, monitor and report
on project progress and outcomes to a variety of audiences, and meet program legal
requirements. Administrative activities are the “oil in the machine” that allows the
Program to start, operate and conclude effectively each program cycle.
The following logic model diagram (Figure 26) provides a visual representation of the
Program’s key administrative activities, expected outputs and outcomes.

Figure 26: Administration Logic Model
aversion for new technologies

decoupled ownership of power generation and delivery infrastructure; low investment in grid infrastructure; actions of third-party providers; risk
External Influences: Electricity prices: changing national and international R&D policies for energy efficiency/renewables/EV/grid integration;
Inputs A: EPIC funds, program staff, CEC roadmap
Investment Planning Project Selection – CEC only Project Assessment Management and Administration
B: D: G: J:
1. Identify, evaluate and prioritize a 1. Fund projects consistent with EPIC 1. Implement Effective Project Tracking 1. Adhere to Program Requirements
broad range of potential energy RD&D goals
Activities objectives 2. Conduct selection process openly
2. Implement Effective Project Reporting 2. Implement Effective Program Management
3. Effectively Coordinate across Administrators
2. Produce coordinated investment and effectively
plans 3. Contract with grantees fairly and
effectively
4 6
1 2
C E H K
Relevant stakeholders identified PONs on CEC website List of potential benefits and metrics Program manuals, tracking databases
and contacted about needs to Final screening and scoring criteria associated with investment initiatives Training sessions/meetings with staff on
identify research topics developed, posted and applied (CEC) and projects (IOUs) in Investment programs requirements
Surveys and workshops conducted, Projects ranked with pass/fail Plans Organizational chart/ documentation of
meetings with internal experts, & status, projects receive NOPAs Periodic status reports management structure
IOU meetings with in-house Coordination meetings w/IOUs Meetings and communications w/ Admin and tech. expert time spent on
technical experts during project selection stakeholders program
Investment Plan: Proposed Posted selection results Final Project Reports Internal meetings/communication
Outputs Meetings, workshops, publications, Final audit reports
initiatives for each funding area Grant agreements and contracts;
(CEC)/proposed projects (IOUs) non-competitive (direct) awards articles, fact sheets, reports, Meetings, emails and phone calls with
Coordinated Investment Plans administrators websites, stakeholder other administrators
across administrators list serve Project reports, presentations to
administrators
3
5
7
F I L
Stakeholders aware of project Projects effectively monitored to Administrators aware and knowledgeable
selection criteria and process ensure they meet stated objectives about program requirements
Coordinated set of projects across Stakeholders aware and Program meeting its CPUC and legal
Short-Term administrators knowledgeable about project requirements
Timely and effective project evaluation results Administrators effectively tapping staff
Outcomes contracts in place and technical resources
(1-4 Years) Effective program administrative process
Maximized budget for funding projects/
minimized overhead expenses
Administrators learning from each others’
projects

Inputs A: EPIC program funding and administrator staff resources are the two key inputs
to ensure that projects are solicited, selected fairly, monitored and implemented according
to plans/contracts, and that they provide useful results (which get disseminated). In
addition, CEC roadmaps provide strategic direction to the Program, as well as guidelines
that administrators must follow.
Activities B (Investment Planning): Under the broad category of Investment Planning are
two sub-categories of activities: prioritizing a range of R&D objectives and producing
coordinated Investment Plans. These initial activities are critical to ensure that each
administrator, and the Program as a whole, has a firm set of research and project
objectives in mind prior to soliciting specific proposals.
Outputs C: Investment planning activities lead to a range of short-term outputs, including

public solicitations for current and important research topics, workshops and surveys to
gather stakeholder and expert input, and ultimately strategic and coordinated Investment
Plans to guide the next cycle of project selection and funding.
Activities D (Project Selection): Under the broad category of Project Selection are three
subcategories of activities: conducting transparent selection processes, funding projects
that are expected to meet program goals, and executing contracts with selected grantees.
These activities are critical to ensure that the strategic program objectives and current
research needs are translated into actual projects. Note that these activities only occur for
the CEC; the IOUs select projects to fund during the earlier investment planning stage.
Outputs E: The project selection process should yield several short-term outputs,
including issuance of multiple Public Opportunity Notices (PONs), the development and
posting of transparent selection criteria, administrator meetings to rank and select projects,
posting of the final grantees, and contracts/funding awarded to grantees.
Outcomes F: As the Project Selection process unfolds, stakeholders become more aware
and knowledgeable of the process and expectations for selected projects, and can
potentially submit higher quality bids going forward. At the conclusion of the process,
there should be a coordinated set of selected projects across administrators, as well as
contracts in place to initiate those projects.
Activities G (Project Assessment): The broad activity of Project Assessment requires

effective project tracking and reporting. At a high level, these activities are critical to
ensure that projects move forward (and implementation barriers are addressed),
stakeholders are informed of project status, and that project results are disseminated in a
wide range of formats.
Outputs H: During the Project Assessment phase, multiple outputs are produced by the
program administrators, including periodic status reports, communications with

stakeholders, and final project results in a variety of formats (e.g., formal reports, websites,
workshops, publications, list serves, etc.).
Outcomes I: As a result of the Project Assessment activities, stakeholders remain informed

about the status of specific projects, the projects are effectively monitored to ensure that
they meet the contracted research objectives, and a wide range of stakeholders have access
to the final project results.
Activities J (Management and Administration): A last set of administrative activities is

broadly categorized as Management and Administration. As opposed to the other kinds of
activities that require more interaction with the general public, project grantees and
research stakeholders, this group of activities is more internally focused. Key sub-activities
include implementing effective project/grantee management, coordination with other
administrators, and adhering to program requirements.
Outputs K: Management and Administration activities result in a broad set of short-term

outputs, including staff allocation and training, program manuals, project monitoring,
project tracking databases, internal audit reports, communications with other EPIC
administrators, and internal project reporting.
Outcomes L: As a result of ongoing management and administration activities, program

staffs become well-versed about program requirements, staff resources and program
budgets are used effectively, program administrators are informed about each others’
projects, and the Program meets its CPUC and legal requirements.

13.2 Applied R&D Logic Model
In the 2012-2014 EPIC Investment Plan (EPIC 1), the CEC provided $158.70 million for
Applied R&D funding for development of new technologies, methods and approaches
from early bench scaleup to pilot scale prototype demonstration. For the 2015-2017 EPIC
Investment Plan (EPIC 2), the CEC provided $151.63 million to support the same activities.
The funding areas include energy efficiency and demand response (DR), clean generation,
smart grid enabling clean energy and cross-cutting projects. Each strategic objective
outlines a set of initiatives focused on a particular area of proposed research (Table 60).

Table 60: Proposed Strategic Objectives for the Applied Research and Development
Program Area
Funding Area, by Strategic Objective
Energy Efficiency and Demand Response
S1 Strategic Objective: Improve Energy Efficiency Technologies and Strategies in

California’s Building, Industrial, Agriculture, and Water Sectors.
S2 Strategic Objective: Enable Cost-Effective Demand Response for California
IOU Electricity Customers.
Clean Generation
S3 Strategic Objective: Develop Innovative Solutions to Increase the Market

Penetration of Distributed Renewable and Advanced Generation.
S4 Strategic Objective: Improve Power Plant Performance, Reduce Cost, and
Accelerate Market Acceptance of Existing and Emerging Utility-Scale Renewable Energy
Generation Systems.
S5 Strategic Objective: Reduce the Environmental and Public Health Impacts of
Electricity Generation and Make the Electricity System Less Vulnerable to Climate
Impacts.
Smart Grid Enabling Clean Energy
S6 Strategic Objective: Advance the Use of Smart Inverters as a Tool to Manage

Areas with High Penetrations of PV.
S7 Strategic Objective: Develop Advanced Distribution Modeling Tools for the
Future Smart Grid.
S8 Strategic Objective: Advance Customer Systems to Coordinate with
Utility Communication Systems.
S9 Strategic Objective: Advance Electric Vehicle Infrastructure to Provide
Electricity System Benefits.
Cross-Cutting
S10 Strategic Objective: Advance the Early Development of Breakthrough

Energy Concepts.
S11 Strategic Objective: Provide Federal Cost Share for Applied Research Awards.
Applied Research and Development Program Area Total $151.63 million
Source: California Energy Commission

While the key barrier that the Applied R&D program area addresses is the Technological
Valley of Death (or TVD, described later), there are many other barriers such as legal and
regulatory, the higher cost of new technologies, performance uncertainty, split incentives,
risk avoidance, information-search costs, and a wide variety of technical barriers. This
means that the number and types of barriers addressed varies to some extent by project.
Details regarding interrelated program activities engaged in by CEC staff and grantees
and how they combine to produce the hypothesized outputs, short-term outcomes, mid-
term outcomes and long-term outcomes88 are provided in Figure 27. Each input, activity,
output and outcome in the logic model is assigned a letter and each link is assigned a
number. These letters and numbers are for ease of reference and do not indicate a
sequence of activities.
Through a review of key documents and an in-depth discussion with CEC staff, we
developed an initial understanding of the set of inputs and activities that would lead to
certain outputs and outcomes. In addition, literature in the following areas was reviewed:
diffusion of innovation, communication through organizational and interpersonal
networks, organization decision-making, engineering research methods and software
development methods, and evaluations of similar programs. Theories that were found to
be especially relevant are discussed below, with the theory discussion broken out by the
key program activities presented in the logic model diagram.
The discussion below is organized by program activities and their causal links to outputs,
short-term, and mid-term/long-term outcomes. In this way, the reader can focus on
specific program intervention activities and whether comparative program findings
and/or theory literature provide support for the program theory or suggest concerns that
might need to be tested in evaluation or addressed in future program refinement activities.
88Note that we have not distinguished in the model between mid- and long-term outcomes, since it is
difficult to determine which desired outcomes would happen at specific intervals of time. Once the Program
has matured, it may be possible to distinguish between mid- and long-term outcomes and indicate
associated time horizons (e.g., 5-10 years, 10 years +).

Figure 27: Applied R&D Logic Model
decoupled ownership of power generation and delivery infrastructure; low investment in grid infrastructure; actions of third-party providers
External Influences: California energy policy; Electricity prices; changing national and international R&D policies for energy efficiency/renewables/EV/grid integration;
E: Track and assess

3 6
progress
B. Selected D: Contractors implement R&D plans by conducting lab-
contractors propose scale testing and pilot-scale testing in the areas of:
• Energy efficiency & demand response 5
research hypotheses
and appropriate • Clean generation 4
Activities methods to test • Smart grid enabling clean energy
them • Cross cutting F: The CEC and G: Contractors
contractors engage seek private
J: Disseminate
other research investment/
results
7 8 organizations and partners/grants
2 individuals
1
10 12
9 11
I: Rresults in the form K: Results delivered
M: Meetings N: Progress
H: Patents/ of data/reports/fact through Websites,
L: Inter- with reports to
copyrights filed/ sheets/articles workshops,
organizational potential CPUC,
C: R&D plans granted conference 15 and private program
Outputs presentations, & 16
interpersonal investors/ administrator
professional
networks partners & s and other
networks
formed proposals stakeholders
13 14 submitted
26
O: Increased awareness and knowledge among targeted
population regarding energy efficiency, DR, clean generation,
17
and smart-grids leading to an increase in private investments
and funding and levels of production 25
18
21
Short-Term 19 20 22 23 24
Outcomes
Q: Clean S: Smart grid T: Knowledge used in V: Additional
(1-4 Years) generation R: Energy efficiency and funding from
technologies/ planning and
P: Follow- technologies and DR technologies adopted private investors,
tools management by U: Technology
on R&D tools adopted by by ee programs and partnerships
adopted by program administrators Demonstrations
electricity customers inside & formed, grants
grid inside & outside
generators outside California awarded
operators California
27 28 29 30
31
Mid-Term & W:
Long-Term • Cleaner generation • Greater energy reliability • Lower grid operating costs • Increased safety • Environmental sustainability
Outcomes • Greater efficiency • Enhanced renewable technologies • Greater uptake of Evs • New codes and standards • Benefits to ratepayers
• Support high penetration of renewables • Economic value to California grid • Public health benefits • Clean jobs
(5 - 10 Years)

Inputs A: Capital and expertise are the two key inputs into any EPIC Applied R&D
project. These come in the form of EPIC funds as well as the matched funds from partners.
Expertise provided by the CEC, IOUs, and other interested stakeholders are also essential
for these funds to be prudently spent. These experts will also rely on any past research to
guide their decisions.
Activities B: The first activity is that contractors are selected by EPIC administrators. In
general, it involves reviewing and then selecting proposals for projects in energy efficiency
and demand response, clean generation, and smart grid enabling, as well as cross-cutting
projects. Proposals are evaluated on a number of administrative and technical criteria,
which includes 1) technical merit and need, 2) technical approach, 3) impacts and benefits
to California IOU ratepayers, 4) team qualifications, capabilities and resources, 5) budget
and cost-effectiveness, 6) funds spent in California and 7) the ratio of direct labor and
fringe benefit rates to loaded labor rates. More details regarding this activity are presented
in the Administration logic model in Section 13.1.
The awarding of grants to winning bidders is crucial to the mission of Applied R&D,
which is to address gaps in the funding necessary to help innovative energy technologies
and approaches bridge the TVD.89 The TVD occurs early in the development of a
technology, as breakthrough research and technological concepts aim to achieve
commercial proof-of-concept. At this stage, innovators and entrepreneurs conducting basic
and applied research need further capital to undergo a process of developing, testing and
refining their technologies in order to prove to private funders that these technologies will
be viable in markets beyond initial success in the laboratory. However, investors are
typically reluctant to fund such early-stage research and product development, largely
due to the high technical, market and management execution related risks and long
development horizons associated with as-yet-unproven technological concepts. As a
result, many entrepreneurial start-up firms and research laboratories fail to accumulate the
necessary capital to see their innovative research concepts translated into commercial
products and ventures.
This early-stage TVD, while endemic to the development of most innovative technologies,
is particularly acute in the energy sector. In this sector, the process of developing
technologies is both capital- and time-intensive, and new innovations must quickly
compete with well-entrenched and commoditized conventional energy technologies. The
early stage expenses necessary for nascent advanced energy technologies to demonstrate
market validity, including prototyping and laboratory costs, are significantly higher than
for many other sectors. In the “garage culture“ of Internet startups, for example, it takes
89J. Jenkins and S. Mansur, "Bridging the Clean Energy Valleys of Death: Helping American Entrepreneurs
Meet the Nation’s Energy Innovation Imperative." Breakthrough Institute, 2011.
http://thebreakthrough.org/blog/Valleys_of_Death.pdf

comparatively little capital or time to advance an innovative research idea or product
concept into a provable business plan. In contrast, bringing innovative energy research to
its pre-deployment phase requires significant capital and as much as 10 to 15 years.
Venture capitalists usually expect a shorter time frame for exit from an investment (often
just three to five years), and the long investment payoff periods typical to new energy
technologies only serve to further discourage investors. Alternatively, angel investors, who
tolerate high-risk projects, often provide funding for startups in exchange for ownership
equity. However, these entities only provide financing on the order of $1–2 million, not
nearly enough to bring these capital-intensive projects across the TVD.
Outputs C: Once bidders are selected, they prepare a final research plan in close
collaboration with the appropriate subject matter experts at the CEC. This plan should
incorporate best research practices which are a necessary condition for a successful project.
Activity D: The research plan is then well managed and faithfully implemented by the
project team. However, we recognize that deviations from the original plan will by
necessity sometimes occur.
Activity E: Each project is tracked to assess its progress. As the interim results become
available, it is possible that grantees will need to make mid-course corrections in the
research design and methods to keep the project on track. Interim results might also
indicate that the technology/tool is infeasible and, in close collaboration with CEC staff,
conclude that the project should be terminated.
Output N: These progress reports are also delivered to the CPUC, the program
administrators and other stakeholders.
Activity F: This activity is concerned with building an essential part of the results-
dissemination infrastructure, i.e., a network of organizations and individuals with which
to share research results to decrease the uncertainty of new technologies/tools and move
them closer to commercialization and deployment.
Activity G: Contractors seek private investment, partners and grants. Grantees are
encouraged to identify other sources of funding to help defray the costs to IOU ratepayers
and share the inherent risks associated with emerging technologies/tools. They might
meet with venture capitalists to present their technology/tool and its clear path to
profitability. They could also seek organizations and individuals with whom to partner in
order to share the risk or seek public and private grants.
Output M: Meetings are held with potential private investors and partners, and grant
proposals are submitted.

Outputs H and I: The results in the form of databases, fact sheets and articles are
produced and, in some cases, patents and copyrights are granted.
Activity J: Any new knowledge must be effectively disseminated if it is to change behavior

and advance the new technologies, methods and approaches. To disseminate this new
knowledge, the CEC, CPUC, and IOUs rely on a variety of paths to reach their target
audiences including websites, workshop presentations and conference presentations.
While journal articles and other professional publications reach their subscribers, the CEC,
CPUC, and IOUs attempt to broaden their reach by making them available on the websites
and at workshops and conferences.
Output K: Results are posted on the CEC, CPUC, and IOU websites, workshops are held
for interested parties, and presentations are made at professional conferences and
delivered to a variety of organizations and individuals that comprise their professional
network.
Output L: Another way to reach the various target audiences is through a network of
individuals and organizations with which to share research results in order to decrease the
uncertainty and cost of new technologies/tools and move them closer to
commercialization and deployment. While networks can form naturally, the CEC and the
IOUs have intentionally sought out other individuals and organizations that have
expressed an interest in EPIC (Activity F). That these individuals and organizations have a
shared interest is important: “A fundamental principle of human communication is that
the exchange of ideas occurs most frequently between individuals and organizations who
are alike, or homophilous. Homophily is the degree to which a pair of individuals who
communicate are similar.”90
As a result of Activity F, complex inter-organizational and interpersonal networks are

formed. The nature of each network depends on the interests of each member. For
example, those interested in efficient technologies might belong to one network with sub-
groups formed around a particular end use such as lighting or HVAC. Other networks
might form among those interested in public policy surrounding energy innovations such
as the CPUC, CEC and IOUs. Interactions among network members can vary with respect
to frequency, reciprocity, connectedness,91 and the types of information and resources
shared.92
90 Everett M. Rogers, Diffusion of innovations (New York: The Free Press, 2003), 305.
91 Connectedness refers to the extent to which the actors are able to connect to each other through the
network. If there is no path from one actor to the other actor, then the two actors are disconnected.
92 Martin Kilduff and Wenpin Tsai, Social Networks and Organizations (Los Angeles: SAGE Publications, 2009).

There could be multiple networks: ones established by the CEC, ones established by the
grantees, and ones established by the IOUs. There could also be some considerable overlap
in network membership. The efficacy of building and maintaining such dissemination
networks as a way of communicating important information is highlighted in the diffusion
of innovation literature.93 When cooperation is high among members of a network with
similar interests, one can think of the network communications as an exercise in group
problem solving.
Outcome O: The Outputs H, I, K and L combine to increase awareness and knowledge

among targeted population regarding energy efficiency, demand response, clean
generation and smart grids. This, in turn, leads to a reduction in key barriers to the
commercialization and adoption of these new technologies and tools.
Short-Term Outcomes P, Q, R, S, T, U and V: Changes in behavior follow from Outcome

O including 1) the conduct of additional research (follow-on R&D), 2) the adoption of
clean generation technologies and tools by electricity providers, 3) the adoption of energy
efficiency and demand response technologies by energy efficiency and demand response
programs, 4) the adoption of smart grid technologies and tools by grid operators, 5) the
use of new knowledge in planning and management by program administrators, and 6)
the formation of partnerships and the obtaining of additional grants and funding from
private investors. It is also possible that some completed R&D projects could be selected
for demonstration in a lab or at customer sites.
Mid-Term & Long-Term Outcomes W: As a result of the short-term outcomes, over time,
the adoption and use of these technologies, methods and approaches will increase in the
targeted populations, eventually leading to a broad set of mid-term and long-term large-
scale outcomes such as cleaner electricity generation, greater energy reliability and
increased safety.
External Influences: There are many external factors that can also influence the Applied
R&D program area at all levels and time frames. These factors include:
• California energy policy

• Electricity prices
• Changing national and international R&D policies for energy
efficiency/renewables/EV/grid integration
• Decoupled ownership of power generation and delivery infrastructure
• Low investment in grid infrastructure
Rogers, Diffusion of Innovations; Jesse H. Ausubel and H. Dale Langford, Technological Trajectories and the
93
Human Environment (Washington DC: National Academy Press, 1997).

• Actions of third-party providers
• Risk aversion for new technologies
13.3 TD&D Logic Model

In the 2012-2014 EPIC Investment Plan (EPIC 1), the CEC provided $129.9 million to test
new technologies in conditions that approximate real-world applications. In the 2015-2017
EPIC Investment Plan (EPIC 2), the CEC provided $145.02 million to support the same
activities. The EPIC 2 funding areas vary slightly from the EPIC 1 funding areas (see Table
61 and Table 62). While EPIC 1 addresses grid reliability (S14), EPIC 2 is focused
specifically on three areas of improving grid reliability (S14, S15 and S16). In addition,
while EPIC 1 addresses clean energy generation (S13), EPIC 2 focuses specifically on
Biomass-to-Energy Conversion Systems (S13). Because relatively few EPIC 2 projects have
launched, this evaluation will focus primarily on EPIC 1 projects.
Table 61: EPIC 1 Proposed Funding Allocation for the Technology Demonstration and
Deployment Program Area by Strategic Objective
Funding Area
S12 Strategic Objective: Demonstrate and Evaluate the Technical and Economic Performance of
Emerging Energy Efficiency and Demand-Side Management Technologies and Strategies
S13 Strategic Objective: Demonstrate and Evaluate Emerging Clean Energy Generation Technologies
and Deployment Strategies
S14 Strategic Objective: Demonstrate the Reliable Integration of Energy Efficient Demand-side
Resources, Distributed Clean Energy Generation, and Smart Grid Components to Enable Energy-
Smart Community Development
S15 Strategic Objective: Provide Cost Share for Federal Awards

Table 62: EPIC 2 Proposed Funding Allocation for the Technology Demonstration and
Deployment Program Area by Strategic Objective
Funding Area
S12 Strategic Objective: Overcome Barriers to Emerging Energy Efficiency and Demand-Side
Management Solutions Through Demonstrations in New and Existing Buildings
S13 Strategic Objective: Demonstrate and Evaluate Biomass-to-Energy Conversion Systems, Enabling
Tools, and Deployment Strategies
S14 Strategic Objective: Take Microgrids to the Next Level: Maximize the Value to Customers
S15 Strategic Objective: Demonstrate Advanced Energy Storage Interconnection Systems to Lower
Costs, Facilitate Market and Improve Grid Reliability
S16 Strategic Objective: Expand Smart Charging and Vehicle-to-Grid Power Transfer for Electric
Vehicles
S17 Strategic Objective: Provide Federal Cost Share for Technology Demonstration and Deployment
Awards
While the key barrier that the TD&D program area addresses is the Commercialization
Valley of Death (or CVD, described later), there are many other barriers such as legal and
regulatory, the higher cost of new technologies, performance uncertainty, split incentives,
risk avoidance, information-search costs and a wide variety of technical barriers. This
means that the number and types of barriers addressed varies to some extent by project.
term outcomes and long-term outcomes94 are provided in the logic model. Each input,
activity, output and outcome in the logic model is assigned a letter and each link is
assigned a number. These letters and numbers are for ease of reference and do not indicate
a sequence of activities. Finally, the potential performance indicators for each activity in
this logic model are presented.

The discussion below is organized by program activities and their causal links to outputs
and short-term and mid-term/long-term outcomes. In this way, the reader can focus on

Figure 28: TD&D Logic Model
technologies
decoupled ownership of power generation and delivery infrastructure; low investment in grid infrastructure; actions of third-party providers; risk aversion for new
External Influences: California energy policy; Electricity prices; changing national and international R&D policies for energy efficiency/renewables/EV/grid integration;
C: Proposals for B: Proposals for TD&D

TD&D projects projects submitted to
submitted to CEC IOUs 4 J: Track and assess progress
2 1 5
Activities 6
E: TD&D research K: Program administrators
plans developed and 9 engage other research
implemented 11 I: Disseminate results organizations, investors and
individuals
12 14
3 7 8
L: Results delivered N: Progress reports to

M: Inter-
H: Results in through websites, CPUC, program
F: Patents/ organizational
D: TD&D G: Customer/ the form of workshops, conference 13 administrators and
copyrights and interpersonal
presentations, & other stakeholders
Outputs Projects
filed/
stakeholder 10 data/reports/ networks formed
selected visits fact sheets/ professional networks
granted
articles
16 17
15
O: Increased awareness and knowledge among targeted population
regarding technology/tool in real-life situations leading to an increase 18
in private investment and levels of production.
Short-Term
Outcomes 19 20 21 22 23 24
(1-4 Years)
P: Demonstrated Q: Demonstrated energy
clean generation efficiency and DR technologies R: Smart grid S: Knowledge used in T: Increased number of
planning and management producers of V: Follow-on TD&D
technologies and adopted by ee programs and technologies/tools
by program administrators demonstrated Research
tools adopted by customers inside & outside adopted by grid
electricity California operators inside & outside California technologies
generators
26 27 28
25 29
Mid-Term & U:
Long-Term • Cleaner generation • Greater energy reliability • Lower grid operating costs • Increased safety • Environmental sustainability
Outcomes • Greater efficiency • Enhanced renewable technologies • Greater uptake of Evs • New codes and standards • Benefits to ratepayers
• Support high penetration of renewables • Economic value to California grid • Public health benefits • Clean jobs
(5 - 10 Years)

Inputs A: Capital and expertise are the two key inputs into any EPIC TD&D project. These
come in the form of EPIC funds as well as the matched funds from partners. Expertise
provided by the CEC, IOUs, and other interested stakeholders are also essential for these
funds to be spent prudently. These experts will also rely on any past research to guide
their decisions.
Activities B and C: Once technologies have been successfully tested in bench scale
systems and meet pre defined performance targets, the technologies must be fully
demonstrated and deployed in actual commercial applications to document the benefits
and savings in real world conditions. Demonstrations and large-scale deployments are
necessary in real-world conditions to independently document technical feasibility;
validate energy, water and cost savings, and environmental benefits; resolve regulatory
barriers; and determine overall life-cycle economics. Without an independent assessment
of technical and economic viability, these technologies and strategies lack a solid value
proposition to potential customers and often do not make it past the CVD, which exists
between the pilot/demonstration and commercialization phases of the technological
development cycle and aligns with a gap between the traditional role of venture capital
and the later stage investments of project finance and debt/equity investors.95
Demonstrations of multiple, integrated demand-side management technologies are

required to document the synergies, overall economics and other benefits of combining
technologies that would result in the greatest ratepayer benefits. These demonstrations are
especially necessary to establish the right mix of technologies for particular applications,
document technical and economic feasibility, and minimize risk to building
owners/operators. Key features of a demonstration include that it is open to the public or
to an interest group, that many viewers are encouraged to visit, and that it may highlight a
systems approach rather than an individual measure.
Public funding for demonstrations to bridge the CVD is essential. The private sector does
not typically conduct applied research and is risk averse regarding new, unproven
technologies, often lacking the resources to analyze and evaluate various technologies.
Frequently, new technologies are developed in academic communities that do not have the
funding for large scale demonstrations. Typically, the private sector only offers funding
after a successful field demonstration.
95Jenkins, J., and Mansur, S. 2011. "Bridging the Clean Energy Valleys of Death: Helping American
Entrepreneurs Meet the Nation’s Energy Innovation Imperative." Breakthrough Institute.
http://thebreakthrough.org/blog/Valleys_of_Death.pdf

The importance of demonstration sites in reducing uncertainty among key stakeholders
was highlighted in the diffusion of innovations literature.96 Rogers and Shoemaker97 also
provides us with a list of five factors of the product or service that influence the rate of
diffusion. These five factors are the following:
1. Relative advantage: The perceived relative advantage compared to the previous

product/service, including economic, social prestige, convenience and satisfaction.
2. Compatibility: The degree to which the product/service is perceived to be consistent
with existing values, past experiences and needs.
3. Complexity: The degree of difficulty of understanding the product/service—the
more difficult it is, the longer it takes for acceptance/adoption.
4. Trialability: The degree to which the new product can be tried on an “installment
plan” basis.
5. Observability: The degree to which the product can be observed in use fulfilling
similar needs for others.
While the TD&D program area attempts to address all five of the attributes through the
development of technology demonstrations, relative advantage and observability have
been found to be most important.98 However, they go on to point out that, while customers
can be invited to buildings where a given technology has been installed, it remains
difficult to observe reduced energy use and the complexity of new innovations. The TD&D
program area attempts to overcome this problem by monitoring each technology in a
laboratory or field setting to verify its energy and demand savings and providing very
detailed project descriptions that provide information beyond what can be seen.
The CEC and the three IOUs consider proposals for TD&D projects from a variety of
sources.
Output D: The TD&D projects are selected that have the greatest potential energy and
non-energy benefits and have the highest risk of falling into the CVD.
Output E: A final demonstration project research plan is developed by the CEC grantees
and the IOU project managers in close collaboration with the appropriate subject matter
experts. This plan should incorporate best research practices which are a necessary
condition for a successful project. The demonstration site is then launched, and the
96 Rogers, Diffusion of innovations.

97 Everett M. Rogers with F. Floyd Shoemaker, Communication of innovations: A cross-cultural approach (New
York: Free Press, 1971), 137-157.
98 John H. Reed and Nicholas P. Hall, "Methods for measuring market transformation." Proceedings of the
International Energy Evaluation Conference, 1997, 177-184.

research plan is faithfully implemented by the project team. However, we recognize that
deviations from the original plan will by necessity sometimes occur.
Output F: At some point during and/or after the completion of a given project, patents
might be filed and granted. In addition, the results of some projects might be copyrighted
or licensed.
Output G: Customers and other stakeholders, including potential investors, visit the
customer site or laboratory where the technology is being demonstrated.
Outputs H: The results in the form of final reports, databases, fact sheets, journal articles
and trade magazine articles are produced.
Activity I: Any new knowledge must be effectively disseminated if it is to change behavior

and advance the new technologies, methods and approaches. To disseminate this new
knowledge, the CEC, CPUC and the IOUs rely on a variety of paths to reach their target
audiences including websites, workshop presentations and conference presentations.
While the journal articles and other professional publications reach their subscribers, the
CEC, CPUC and IOUs attempt to broaden their reach by making them available on their
websites and at workshops and conferences.
Output L: Results are posted on the CEC, CPUC and IOU websites, workshops are held
for interested parties, and presentations are made at professional conferences and
delivered to a variety of organizations and individuals that comprise their professional
network.
Activity J: Each project is tracked to assess its progress. As the interim results become
available, it is possible that grantees will need to make mid-course corrections in the
research design and methods to keep the project on track. Interim results might also
indicate that the technology/tool is infeasible and, in close collaboration with CEC staff,
conclude that the demonstration project should be terminated.
Output N: These progress reports are also delivered to the CPUC, the program
administrators and other stakeholders.
Activity K: This activity is concerned with building an essential part of the results-
dissemination and collaboration infrastructure, i.e., a network of potential investors,
technology manufacturers, research organizations and individuals with whom to share
ideas, resources and research results to decrease the uncertainty of new technologies/tools
and move them closer to commercialization and deployment.
Output M: Another way to reach the various target audiences is through a network of
potential investors, technology manufacturers, research organizations, and individuals
with whom to share ideas, resources and research results in order to decrease the

uncertainty and cost of new technologies/tools and move them closer to
commercialization and deployment. While networks can form naturally, the CEC and the
IOUs have intentionally sought out other individuals and organizations that have
expressed an interest in EPIC. That these individuals and organizations have a shared
interest is important: “A fundamental principle of human communication is that the
exchange of ideas occurs most frequently between individuals and organizations who are
alike, or homophilous. Homophily is the degree to which a pair of individuals who
communicate are similar.”99
As a result of Output M, complex interorganizational and interpersonal networks are

formed. The nature of each network depends on the interests of each member. For
example, those interested in efficient technologies might belong to one network with sub-
groups formed around a particular end use such as lighting or HVAC. Other networks
might form among those interested in public policy surrounding energy innovations such
as the CPUC, the CEC and the IOUs. Interactions among network members can vary with
respect to frequency, reciprocity, connectedness100 and the types information and
resources shared.101 Of course, private investors are a key audience for every
demonstration since additional capital is needed to increase level of production.
There could be multiple networks, ones established by the CEC, ones established by the
grantees, and ones established by the IOUs. There could also be some considerable overlap
in network membership. The efficacy of building and maintaining such dissemination
networks as a way of communicating important information is highlighted in the diffusion
of innovation literature.102 When cooperation is high among members of a network with
similar interests, one can think of the network communications as an exercise in group
problem solving.
Outcome O: The Outputs I, J, K, O and Q combine to increase awareness and knowledge

among targeted populations regarding real technologies/tools in real life situations and a
subsequent reduction in key market barriers. This in turn leads to an increase in private
sector investments and levels of production.
Short-Term Outcomes P, Q, R, S, T and V: Changes in behavior follow from Outcome O

including: 1) the adoption of clean generation technologies and tools by electricity
providers, 2) the adoption of energy efficiency and DR technologies by energy efficiency
and DR programs, 3) the adoption of smart grid technologies and tools by grid operators,
4) the use of new knowledge in planning and management by program administrators, 5)
99 Rogers, Diffusion of innovations, 305.

100 Connectedness refers to the extent to which the actors are able to connect to each other through the
network. If there is no path from one actor to the other actor, then the two actors are disconnected.
101 Kilduff and Tsai, Social Networks and Innovations.
102 Rogers, Diffusion of innovations; Ausubel and Langford, Technological Trajectories.

an increase in the number of producers of demonstrated technologies, and 6) follow-on
TD&D research.
Mid-Term & Long-Term Outcomes U: As a result of the short-term outcomes, over time,
the adoption and use of these technologies, methods and approaches will increase in the
targeted populations, leading eventually to a broad set of mid-term and long-term large-
scale outcomes such as cleaner electricity generation, greater energy reliability and
increased safety.
External Influences: There is a wide variety of external factors that can also influence the
TD&D area at all levels and time frames. These factors include:
• California energy policy;

• Electricity prices;
efficiency/renewables/EV/grid integration;
• Decoupled ownership of power generation and delivery infrastructure;
• Low investment in grid infrastructure;
• Actions of third-party providers; and
• Risk aversion for new technologies.
The research designs should attempt to control for these external factors so that any
impacts of the TD&D program area can be observed.
13.4 Market Facilitation Logic Model

For the 2015-2017 EPIC Investment Plan, the CEC has provided $53.27 million for Market
Facilitation funding which encompasses a range of activities that include program
tracking, market research, education and outreach, regulatory assistance and streamlining,
and workforce development to support clean energy technology and strategy deployment.
The key barrier that the Market Facilitation program area addresses is the
"Commercialization Valley of Death.” As Jenkins and Mansur observe in their monograph,
A Clean Energy Deployment Administration, “many energy technologies fall prey to this
Commercialization Valley of Death [CVD], having exhausted the (comparatively) small
investments of venture capitalists yet remaining too risky to be attractive to traditional
debt or equity finance.” The CVD also includes a variety of non-technical and structural
barriers embedded in the existing market environment. These barriers to accelerating the
commercial viability of high-priority technologies and strategies may generally be
classified as follows:

1. Supply-side barriers that impede entrepreneurs from successfully bringing their
innovations to market;
2. Demand-side barriers that inhibit customers from purchasing and adopting clean
energy technology innovations;
3. Regulatory barriers that block or delay the market implementation of the
innovations; and
4. Information barriers that hinder complete knowledge and rational decisions with
respect to clean energy technologies.
The strategic objectives for Market Facilitation guide the development of initiatives and
activities that address such non-technical and structural barriers. Accordingly, Market
Facilitation funding areas include activities associated with fostering commercialization,
facilitating procurement, enhancing permitting, and conducting market and technical
analysis. Each strategic objective has a set of associated initiatives/activities (Table 63).
Table 63: Strategic Objectives for the Market Facilitation Program Area

Foster the Development of the Most Promising Energy Technologies into
Successful Businesses.
S1 Facilitate a Commercialization Assistance Network to Foster Successful Clean Energy
Entrepreneurship.
S2 Integrate Market Insight into the Selection and Management of EPIC Funded
Technologies and Strategies.
S3 Provide Support for Entrepreneurs to Test, Verify, and Certify their Innovations.
Facilitate Inclusion of Emerging Clean Energy Technologies into Large-Scale

Procurement Processes
S4 Develop Tools and Strategies to Encourage Large-Scale Purchasers to Adopt Emerging

Energy Technologies.
S5 Facilitate Innovative Procurement Strategies to Reduce Costs for Clean Energy

Technologies.
Accelerate the Deployment of Energy Technologies in IOU Territories
Through Innovative Local Planning and Permitting Approaches
S6 Develop Innovative Approaches to Integrate Utility and Local Government Planning for
Emerging Technology Deployment.
S7 Develop Innovative Strategies to Streamline the Permitting Process for Zero Net
Energy Buildings.

Inform Investments and Decision-Making Through Market and Technical
Analysis
S8 Conduct Analyses on Different Technology Options and Strategies for the
Electricity System.
S9 Develop a Clearinghouse for Advanced Energy Technologies, Strategies and Tools.
S10 Measure and Verify the Ratepayer Benefits of EPIC-funded Innovations
Applied Research and Development Program Area Total $53.27 million

term outcomes and long-term outcomes103 are provided in the logic model. Each input, set
of activities, set of outputs and set of outcomes in the logic model is assigned a letter and
each link is assigned a number. These letters and numbers are for ease of reference and do
not indicate a sequence of activities. Finally, the potential performance indicators for each
activity in this logic model are presented.
The discussion below is organized by program activities and their causal links to outputs
and short-term and mid-term/long-term outcomes. In this way, the reader can focus on

Input A: Capital and expertise are the two key inputs into any EPIC Market Facilitation
project. These come in the form of EPIC funds as well as the matched funds from partners.
Expertise provided by the CEC, IOUs, and other interested stakeholders are also essential
for these funds to be prudently spent. These experts will also rely on any past research to
guide their decisions.
Activity B: Opportunity notices posted on CEC website.
Activities C: Grantees become aware of funding opportunity and submit proposal;

projects are awarded and contracts signed.

Figure 29: Market Facilitation Logic Model
power generation and delivery infrastructure; low investment in grid infrastructure; actions of third-party providers; risk aversion for new technologies
External Influences: Electricity prices; changing national and international R&D policies for energy efficiency/renewables/EV/grid integration; decoupled ownership of
power generation and delivery infrastructure; low investment in grid infrastructure; actions of third-party providers; risk aversion for new technologies
External Influences: Electricity prices; changing national and international R&D policies for energy efficiency/renewables/EV/grid integration; decoupled ownership of
Inputs A: EPIC Funds, matched funds of partners; expertise
A: EPIC Funds, matched funds of partners; expertise
Foster Commercialization
Foster Commercialization Facilitate Procurement
Facilitate Procurement Enhance Permitting
Enhance Permitting Market & Technical Analysis
Market & Technical Analysis
DD GG J J M
M
Activities 1. Facilitate a Commercialization
1. Facilitate a Commercialization 1. Develop tools & strategies
1. Develop tools & strategies 1. Innovate Local Govt. Planning
1. Innovate Local Govt. Planning 1. Analyze Technology Options
1. Analyze Technology Options
BB Assistance Network that fosters clean
Assistance Network that fosters clean to encourage large purchasers
to encourage large purchasers for Emerging Technologies
for Emerging Technologies and Strategies
and Strategies
Opportunity notices
Opportunity notices energy entrepreneurship
energy entrepreneurship to adopt emerging energy
to adopt emerging energy 2. Streamline Permitting for ZNE
2. Streamline Permitting for ZNE 2. Develop a Clearinghouse for
2. Develop a Clearinghouse for
posted on CEC website
posted on CEC website 2. Integrate Market Insight into
2. Integrate Market Insight into technologies
technologies Buildings
Buildings Advanced Energy Technologies
Advanced Energy Technologies
management of EPIC technologies
management of EPIC technologies 2. Facilitate innovative
2. Facilitate innovative 3. M&V of EPIC-funded
3. M&V of EPIC-funded
3. Support entrepreneurs to test,
3. Support entrepreneurs to test, procurement and financing
procurement and financing 9 Innovations
Innovations
2 verify, and certify their innovations
verify, and certify their innovations strategies
strategies
12
C KK
- Grantees become 3 6 - Resources for local governments
- Resources for local governments NN
aware of funding to clean energy technologies and
to clean energy technologies and - Market and scientific
- Market and scientific
opportunity, submit EE HH ZNE-ready communities
ZNE-ready communities research studies and
6 research studies and
proposal; - Resources to help entrepreneurs - Tools and strategies that help - Information for local
- Information for local evaluation reports
- Resources to help entrepreneurs - Tools and strategies that help evaluation reports
- Project awarded, commercialize innovations large-scale purchasers adopt government planning for
government planning for - Clean energy investment
commercialize innovations large-scale purchasers adopt - Clean energy investment
Outputs contracted - Entrepreneurs engage with investors
- Entrepreneurs engage with investors emerging energy technologies
emerging energy technologies
electricity system resilience and
electricity system resilience and strategies and business cases
strategies and business cases
and customers - Innovative procurement and reliability
reliability - Online tool and information
and customers - Innovative procurement and - Online tool and information
- Energy Commission (EC) successfully financing strategies that mitigate - Resources for communities to
- Resources for communities to exchange for clean energy
- Energy Commission (EC) successfully financing strategies that mitigate exchange for clean energy
administers EPIC costs for clean energy implement builder agreements
implement builder agreements technologies, integrated DSM,
administers EPIC costs for clean energy technologies, integrated DSM,
- High priority technologies are certified technologies and permitting strategies for ZNE-
and permitting strategies for ZNE- and ZNE
- High priority technologies are certified technologies and ZNE
- The results (e.g., reports, fact sheets - The results (e.g., reports, fact ready communities
ready communities - The results (e.g., reports, fact
- The results (e.g., reports, fact sheets - The results (e.g., reports, fact - The results (e.g., reports, fact
and articles) are disseminated. sheets and articles) are - The results (e.g., reports, fact
- The results (e.g., reports, fact sheets and articles) are
and articles) are disseminated. sheets and articles) are sheets and articles) are
disseminated. sheets and articles) are
sheets and articles) are disseminated
disseminated. disseminated
disseminated.
disseminated.
4 7 13
10
FF II LL OO
Entrepreneurs:
Entrepreneurs: -!
!- Resources help large-scale
Resources help large-scale - Local governments: plan for and
- Local governments: plan for and - Increased stakeholders'
- Increased stakeholders'
Short-Term - Develop successful commercialization
- Develop successful commercialization purchasers reduce uncertainty
purchasers reduce uncertainty promote clean energy
promote clean energy awareness and knowledge
awareness and knowledge
plans and strategies - Large-scale purchasers adopt
- Large-scale purchasers adopt technologies; streamline
technologies; streamline sharing
sharing
Outcomes plans and strategies
- Connect with other stakeholders to more clean energy technologies
more clean energy technologies permitting and reduce costs for
permitting and reduce costs for - Successful emerging
- Successful emerging
- Connect with other stakeholders to
(1-4 Years) facilitate commercialization
facilitate commercialization into procurement practices
into procurement practices ZNE projects
ZNE projects technologies are adopted
technologies are adopted
- Successfully transition into
- Successfully transition into - Large-scale purchasers reduce
- Large-scale purchasers reduce - Progress towards CA goal of all
- Progress towards CA goal of all - EC assesses EPIC progress
- EC assesses EPIC progress
commercially viable businesses
commercially viable businesses clean energy financing and
clean energy financing and ZNE new construction by 2020
ZNE new construction by 2020 toward goals
toward goals
- Can easily test, verify, and certify their
- Can easily test, verify, and certify their purchase costs
purchase costs - Follow-on research
- Follow-on research -Follow-on research
-Follow-on research
Innovations
Innovations - Follow-on research
- Follow-on research
5 8 11 14
Mid-Term &
P:
Long-Term - Cleaner generation - Greater energy reliability - Lower grid operating costs -Increased safety - Environmental sustainability
Outcomes - Greater efficiency - Enhanced renewable technologies - Greater uptake of Evs - New codes and standards - Benefits to ratepayers
(5 - 10 Years) - Support high penetration of renewables - Economic value to California grid - Public health benefits - Clean jobs

13.4.1 Foster Commercialization
Activities D: The activities to foster commercialization include: 1) Facilitate a
Commercialization Assistance Network that fosters clean energy entrepreneurship, 2)
Integrate market insight into management of EPIC technologies, and 3) Support
entrepreneurs to test, verify, and certify their innovations. In their monograph, A Clean
Energy Deployment Administration, Jenkins and Mansur also note that, “The
Commercialization Valley of Death represents an institutional barrier, as no particular set
of private actors or institutions is well equipped or willing to bear the risks or invest the
capital on their own to help an energy technology cross the bridge between development
and deployment.” A critical element of this barrier is that investors need assurance of the
technical viability and marketability of the innovations. The activities to foster
commercialization are intended to address this barrier by helping entrepreneurs make
business connections, gain market knowledge, and certify their technologies.
Outputs E: The activities to foster commercialization will result in resources that will help
entrepreneurs commercialize innovations such as business networks, information and
commercialization tools, market research, and access to testing and certification. Results
such as final reports, fact sheets and articles are published and disseminated to target
audiences.
Short-Term Outcomes F: The resulting outcomes will be that entrepreneurs use the
information and commercialization tools and market research to develop effective
commercialization plans and strategies, leverage the business networks to connect with
other stakeholders, and access testing and certification to certify their clean energy
technologies. All of these will serve to streamline the path of entrepreneurs to the
successful commercialization of their innovations.
13.4.2 Facilitate Procurement

Activities G: The activities to facilitate procurement include: 1) Develop tools and
strategies to encourage large purchasers to adopt emerging energy technologies, and 2)
Facilitate innovative procurement and financing strategies. In his book, Crossing the Chasm,
Geoffrey Moore posits that a key approach to bridging the CVD is to cultivate
“champions” among early adopters.104 In the context of the EPIC Market Facilitation
initiatives, large procurers such as military bases, government facilities, the University of
California and building developers are targeted as these prospective champions and early
adopters. Adoption of the technologies by such large customers can help create the early
market pull that provides entrepreneurs with the volume of sales needed to scale up
production and establish a foothold in the market.
104 Geoffrey A. Moore, Crossing the Chasm (New York: Harper Business Essentials, 2014).

Outputs H: The activities to facilitate procurement will result in tools and strategies that
reduce risks associated with purchasing decisions for clean energy—thereby increasing
incorporation of emerging energy technologies into large-scale purchasers’ procurement
strategies—and reduce the costs for large-scale purchasing of clean energy technologies
through collaborative purchasing arrangements and innovative financing mechanisms.
Results such as final reports, fact sheets and articles are published and disseminated to
target audiences.
Short-Term Outcomes I: The resulting outcomes will be that large-scale purchasers will
have resources that reduce the technical and financial risks of adopting new clean energy
technologies, increase the incorporation of the technologies into their procurement
practices, and adopt more clean energy technologies.
13.4.3 Enhance Permitting

Activities J: The activities to enhance permitting include: 1) Helping local governments
make innovations and changes to their information and planning for emerging clean
energy technologies, and 2) Helping local governments streamline their permitting
processes for Zero Net Energy (ZNE) buildings and communities. The environment for
distributed energy resources and emerging clean energy technologies is undergoing
significant changes. Energy end-users are increasingly taking advantage of opportunities
to become ZNE users and energy providers. Inadequate funding also has slowed the
response of local governments to improve preparedness for extreme and manmade
emergency events. Local government regulations, permitting and land use requirements
may be challenged in trying to keep up with these changes. Accordingly, these activities
are aimed at aiding local governments by providing needed information and tools and
helping streamline their regulations and processes to facilitate the timely deployment of
applicable technologies. Results such as final reports, fact sheets and articles are published
and disseminated to target audiences.
Outputs K: The activities to enhance permitting will result in the development of

resources for local governments to upgrade regulations for clean energy technologies and
ZNE-ready communities, information to help local government planning for resilience and
reliability of electricity systems, and resources for communities to implement builder
agreements and permitting strategies for ZNE-ready communities.
Short-Term Outcomes L: The resulting outcomes will be that local governments plan for
and promote clean energy technologies, streamline permitting and reduce costs for
implementation of ZNE projects. This, in turn, will further progress towards achievement
of the goal of 100 percent ZNE new construction in California by 2020.

13.4.4 Conduct Market and Technical Analysis
Activities M: The market and technical analysis activities include: 1) Analyses of
technology options and strategies, 2) Development of a clearinghouse for advanced energy
technologies, and 3) Measurement and verification of EPIC-funded innovations. The clean
energy market is in a state of flux and rapid change. In such an environment, the barriers
to adoption of clean energy technologies may not be readily obvious. Additionally, the
California clean energy goals set by the recent SB 350 has intensified the need for
implementation of clean energy technologies, further underscoring the importance of
ensuring the optimal allocation of EPIC funds in technologies that provide the greatest
ratepayer benefits. All of these trends make it essential to conduct research on gap
analyses, scenario assessments and other decision-making tools. Accordingly, these
activities are aimed at supporting market and technical analyses to provide guidance on
available technology options and ensuring that EPIC funding is achieving program goals.
Outputs N: The market and technical analysis activities will result in the production of
market and scientific research studies and evaluation reports, clean energy investment
strategies and business cases, and an online tool and information exchange for clean
energy technologies, integrated DSM, and ZNE. Results such as final reports, fact sheets
and articles are published and disseminated to target audiences.
Short-Term Outcomes O: The resulting outcomes will be that stakeholders have higher
awareness of clean energy technologies and there is greater knowledge sharing and
successful adoption of the technologies; the CPUC will also make a credible assessment of
progress towards EPIC goals.
13.4.5 Mid-Term & Long-Term Outcomes P

As a result of the short-term outcomes, over time, the adoption and use of these
technologies, methods and approaches will increase in the targeted populations leading
eventually to a broad set of mid-term and long-term large-scale outcomes such as cleaner
electricity generation, greater energy reliability and increased safety.
13.4.6 External Influences

There is a wide variety of external factors that can also influence the Market Facilitation
program area at all levels and time frames. These factors include:
• California energy policy;

• Electricity prices;
efficiency/renewables/EV/grid integration;
• Decoupled ownership of power generation and delivery infrastructure;
• Low investment in grid infrastructure;

• Actions of third-party providers; and
• Risk aversion for new technologies.
The research designs should attempt to control for these external factors so that any
Program impacts may be observed.
13.5 Selected References for Appendix B

Atkinson, Rob, Netra Chhetri, Joshua Freed, Isabel Galiana, Christopher Green, Steven
Hayward, Jesse Jenkins, Elizabeth Malone, Ted Nordhaus, Roger Pielke Jr., Gwyn
Prins, Steve Rayner, Daniel Sarewitz, and Michael Shellenberger. “Climate
Pragmatism.” The Hartwell Group, July 2011.
Ausubel, Jesse H. and H. Dale Langford. Technological Trajectories and the Human
Environment. Washington D. C.: National Academy Press, 1997.
California Energy Commission. The Electric Program Investment Charge: Proposed 2012-2014
Triennial Investment Plan, Staff Report. Publication Number CEC-500-2012-082-CMF,
2012.
California Energy Commission. The Electric Program Investment Charge: Proposed 2015-2017
Triennial Investment Plan. California Energy Commission. Publication Number: CEC‐
500‐ 2014‐038‐CMF, 2014.
Jenkins, J., and S. Mansur. A Clean Energy Deployment Administration: Unlocking Advanced
Energy Innovation and Commercialization. Breakthrough Institute, 2011.
http://thebreakthrough.org/blog/CEDA.pdf
Jenkins, J., and S. Mansur. Bridging the Clean Energy Valleys of Death: Helping American
Entrepreneurs Meet the Nation’s Energy Innovation Imperative. Breakthrough Institute,
2011. http://thebreakthrough.org/blog/Valleys_of_Death.pdf
Kilduff, Martin and Wenpin Tsai. Social Networks and Organizations. Los Angeles,
California: SAGE Publication, 2009.
Moore, Geoffrey A. Crossing the Chasm. New York, NY: Harper Business Essentials, 2014.
Reed, John H. and Nicholas P. Hall. "Methods for measuring market transformation."
Proceedings of the International Energy Evaluation Conference, August 1997, 177-184.
Rogers, Everett M. Diffusion of innovations. New York: The Free Press, 2003.
Rogers, Everett M., with F. Floyd Shoemaker. Communication of innovations: A cross-cultural

approach. New York: Free Press, 1971.

14 Appendix C: Best Practices Assessment Results
Memorandum
This section presents the findings from two research tasks associated with the Best
Practices Assessment: a literature review and in-depth interviews. These results were
developed and delivered to the CPUC as an interim memo before the evaluation analyses
were completed and this report was drafted.
14.1 Literature Review

This section summarizes findings from a literature review of reports and documents
relevant to the planning, design, and implementation of innovative energy research,
development, and demonstration (RD&D) programs. The team reviewed a total of 38
resources, including reports and white papers, evaluation reports, and webpages. This
review will help inform the evaluation of the EPIC program by identifying effective
practices summarized in the literature.
14.1.1 Introduction
This review compiles the key learnings from reports and analyses of programs that are
designed to increase coordinated investment in new and emerging energy solutions. In
particular, this review focuses on key topics of relevance to the EPIC program.
Objectives
The objective of this best practices review is to identify effective practices that have been
used to support RD&D programs. This research will help the EPIC program identify
improvement opportunities by developing lessons learned and comparing the
performance of the EPIC projects to other RD&D programs.
Methodology
To accomplish the objectives of this effort, the Evergreen team conducted a literature
review to identify best practices and lessons learned regarding the planning, design,
implementation, and evaluation of RD&D programs. In doing so, we identified a selection
of programs similar to the EPIC program and reviewed publicly-available evaluation
reports and other documents about these programs. These peer programs include:
• Department of Energy’s Small Business Innovation Research Program (SBIR)

• Department of Energy’s Small Business Technology Transfer Program (STTR)
• Department of Energy’s Advanced Research Projects Agency-Energy Program
(ARPA-E)
• Department of Energy’s Clean Energy Manufacturing Initiative (CEMI)
• New Mexico Small Business Assistance (NMSBA) Program

• NYSERDA Technology and Market Development Program
• Washington State Clean Energy Fund
• Connecticut Green Bank
In general, program-related documents draw on publically-available reports, evaluations,
and other resources related to the peer programs listed above. We primarily tried to cite
practices supported by evaluation results, but some programs either did not have
publically-available evaluation reports or the reports did not cover the topics of interest for
this review. Overall, we documented the evidence as it was available, and plan to further
corroborate this information through follow up interviews with program stakeholders. In
addition to reviewing reports and documents from these peer programs, we reviewed
existing best practices literature that addressed the issues related to the design and
implementation of RD&D programs in general (see 14.1.4 for a full listing of programs and
related documents). The team plans to take into consideration these various sources to
determine the extent to which a particular practice is effective and relevant to the EPIC
program.
The team employed a systematic approach to this review, documenting whether a set of
elements relevant to the evaluation was present in each document. The research
dimensions covered a range of topics, including best practices related to investment
planning, implementation, program management and tracking; technology transfer
mechanisms; market development mechanisms; and key findings from program- or
project-level research/evaluation efforts. See below for the full list of research dimensions.
14.1.2 Summary of Findings

We have distilled the findings from the literature review to correspond with the five
research categories that the EPIC evaluation will cover: 1) Investment Planning, 2) Project
Selection, 3) Project Assessment, 4) Program Management and Administration, and 5)
Contextualized Considerations. Table 64 below presents a general overview of how the
reports that we reviewed align with the research dimensions. Overall, documents
reviewed for the peer programs offered some background information on factors related
to investment planning, and, to some extent, implementation. There was not as much
specific information related to topics related to project management, technology transfer
mechanisms, or market development mechanisms. The best practices literature covered
many of the topics of interest, but in varying levels of detail.

Table 64: Best Practices Reports and Related Dimensions
Programs Best Practices Literature
WA State Clean Energy Fund
Lal, Gupta, & Weber, 2012
Murphy & Edwards, 2003
Townsend & Smith, 2016

Jenkins & Mansur, 2011
Chiavari & Tam, 2011
Hughes et al., 2011

NYSERDA TMD1
CT Green Bank
Wessner, 2013
DOE APRA-E
USDOT, 2011
WHCC, 2013
DOE STTR1
DOE SBIR1
IEA, 2010
NMSBA
CEMI
l = supporting documents cover topic in-depth

¢ = supporting documents mentioned topic, but do not provide the level of
Research Dimension detail relevant to this review
Investment Planning (internal and external inputs—stakeholders, experts, etc.)
Federal/state/local
l l l l l l l l l ¢
energy context
Setting goals and
¢ ¢ ¢ ¢ ¢ ¢ ¢ l ¢
priorities
Market
assessment/identificatio
l l l l
n of gaps and
opportunities
Plan development
¢
processes
Implementation
Project identification
l l l ¢ ¢ ¢
and selection processes
Organizational structure
(internal and external ¢ ¢ ¢ ¢ ¢ ¢ ¢ ¢ ¢ ¢
coordination)
Contracting processes
Intellectual Property
Identification and
Protection
Market barriers ¢ ¢ ¢ ¢ l l
Project Management, Tracking, and Reporting
During implementation
(metrics tracked,
¢ ¢ ¢ ¢ ¢ ¢ ¢ l ¢
reporting frequency,
other mechanisms)
After implementation
(metrics tracked,
reporting frequency, ¢ ¢ ¢ l
dissemination of results,
other mechanisms)


Hughes et al., 2011

NYSERDA TMD1
CT Green Bank
Wessner, 2013
DOE APRA-E
USDOT, 2011
WHCC, 2013
DOE STTR1
DOE SBIR1
IEA, 2010
NMSBA
CEMI
Technology Transfer Mechanisms
Licensing (license
¢ l
agreements, start-ups)
Cooperative R&D
(CRADAs and
Cooperative Research
l l ¢ l ¢ l ¢
Agreements,
government-industry
partnership programs)
Technical Assistance
(user agreements, work
¢ l
for others, commercial
test agreements)
Information
Dissemination and
¢ ¢ l l l
Exchanges (formal and
informal)
Other technology
transfer mechanisms
(consortia, research l ¢ ¢ l ¢ l
parks, innovation hubs,
advisory committees)
Market Development Mechanisms
Market-readiness
facilitation (regulatory
environment
adjustments, testing and ¢ ¢ l ¢ ¢ ¢ ¢ ¢
certification,
entrepreneur network
facilitation)
Customer/market
¢
demand stimulation
Other market
l
development


Hughes et al., 2011

NYSERDA TMD1
CT Green Bank
Wessner, 2013
DOE APRA-E
USDOT, 2011
WHCC, 2013
DOE STTR1
DOE SBIR1
IEA, 2010
NMSBA
CEMI
mechanisms
Program- or Project-Level Research/Evaluation
Research questions l l l l
Methodology l l ¢ l
Conclusions l l l
Recommendations/Less
l l ¢ l
ons Learned
Outcomes (quantitative
l l ¢
and qualitative metrics)
1 Documents for these programs include at least one program evaluation report which covers topics of interest for this
review.

Investment Planning
• All of the peer programs noted close alignment between program investments and
relevant energy policy environment and frameworks. Typically, the federal, state, or
local policy environment has a direct bearing on how RD&D programs operate,
including how these program prioritize financial and other resource investments. All of
the programs included in this review appear to make a strong, connection with
overarching policy directives, most of which are energy-focused. For example, the
goals and objectives of the DOE-funded SBIR program and its sister program, STTR,
clearly reference the Department’s mission. As the 2008 evaluation of the SBIR
program states, “The SBIR program is structured to support all four mission objectives
of DoE [sic] in a manner that parallels the overall allocation of R&D funding to non-
weapons programs.” (Wessner, 2008). The Connecticut Green Bank similarly
recognizes that its goals are tightly aligned with the state’s energy policy. The 2015 and
2016 Comprehensive Plan outlines the Bank’s mission and goals and states further,
“These goals support the implementation of Connecticut’s clean energy policies be they
statutory (i.e., Public Act 11-80, Public Act 13-298), planning (i.e., Comprehensive
Energy Strategy, Integrated Resources Plan), or regulatory in nature.”
• The best practices literature indicates that programs should additionally allow
flexibility for individual projects to build on and enhance public—private
partnerships. These partnerships may include funding, but other resources such as
technical expertise and equipment are also highly valued. All of the RD&D programs
in this review rely to some extent on public sector funding, but documents reviewed
for these programs did not always indicate the extent to which public-private
partnerships exist.
• The program reports and best practices literature did not cover in great detail the
processes used to conduct market assessments, identify gaps and opportunities, or
develop investment plans. This may be an area to explore further through the in-depth
interviews.
Project Selection
• Rigorous selection criteria, strong alignment with overall goals and objectives, and a
peer review process appear to be fairly common among RD&D programs. The
documents for the eight programs featured in this review varied in the amount of
information regarding project identification and selection processes. The documents
from the DOE-funded programs – SBIR, STTR, and ARPA-E – were most likely to
provide detail regarding selection processes. These programs all appear to use rigorous
selection criteria that are strongly aligned with their overall goals and objectives. In

addition, the use of external peer reviewers for these programs ensures that individuals
with relevant expertise help inform awards, which may help bolster the programs’
credibility.
• Of the programs reviewed, the SBIR and STTR programs seem to have the most
transparent process for grantees; information regarding the type of grants, eligibility,
deadlines, etc. were featured more prominently on the programs’ joint website. The
stated aim of these programs to engage traditionally underrepresented segments of the
small business sector (e.g., women- and minority-owned businesses) may factor into
the more pronounced display of this information.
Project Assessment
• The best practices literature recognized the importance of identifying metrics and
systems for tracking progress. Examples of key project metrics include patents,
copyrights, trademarks, and scientific publications. Across a program or portfolio,
broader metrics related to economic impact (e.g., return on investment, increase in
revenue, or amount of leveraged funding or financing) or qualitative metrics associated
with technical assistance (improve operations, marketplace competitiveness, or
improved expertise) are tracked or assessed.
• The best practices literature on RD&D programs does not offer a lot of detail or
guidance related to how systems are implemented, how frequently metrics are
monitored or reported, and how internal systems help inform planning, design, and
implementation. These issues will be investigated further through in-depth interviews
with program stakeholders.
Program Management and Administration

• The peer program reports as well as the best practices literature reveal that effective
RD&D programs have a core internal staff that oversees design and implementation;
the program administration team facilitates relationships with investors, government
agencies, small and large companies, and other organizations.
• Staff are also responsible for ensuring that the program’s overall goals and objectives
are reflected in the portfolio of projects.
• According to the literature (program documents and best practices), program staff and
other institutional supports play an important role in facilitating technology transfer
and market development mechanisms.
o Successful technology transfer mechanisms include protocols and processes
that directly address barriers to commercialization. Examples of useful resources
include templates for licensing and other agreements such as software usage

agreements, material transfer agreements, standardized CRADA (cooperative
research and development agreement) templates or master CRADAs that allow
for single negotiation for several projects are (Hughes et al., 2011; US DOT,
2011).
o In order to implement and share these tools, programs and related institutions
provide support or foster information and knowledge sharing. For example,
universities or other research institutions may have offices to help researchers
navigate the technology development and commercialization processes.
Another mechanism is through federal agencies which may fund or partner
with private firms to facilitate information sharing via research consortia,
innovation hubs, or advisory committees. These opportunities help bring
together researcher and funders to unite the relevant technical expertise and
financial resources.
o Effective market development mechanisms address barriers that occur as a
product or pilot program is scaled up. Larger scale funding from federal entities
or private investors helps address financial barriers that researchers and
entrepreneurs face as they make this transition. Additional policy solutions such
as incentives, subsidies, and tax provisions help ease financial impediments to
market-readiness (Jenkins & Mansur, 2011; Wessner 2013). Other supports and
resources such as entrepreneurial business development programs and online
directories for investors seeking clean energy investment opportunities exist to
facilitate networks among private funders, clean energy companies, and
research institutions (WHCC, 2013; Murphy & Edwards, 2003).
Contextualized Considerations
Aside from specific issues discussed above, there were other contextual considerations
referenced in the literature.
• The best practices literature noted that the technology innovation process is not linear;
it requires program stakeholders, including administrators, implementers, and other
partners to be adept at identifying and capitalizing on opportunities. A few sources
noted that opportunities for feedback loops occur at key junctures, such as the
transitions from basic research to applied research, applied research to development, or
development to commercialization. For example, Chiavari & Tam (2011) note that
feedback from the market and technology users during commercialization and
diffusion phases can influence additional RD&D and promote continuous innovation.
The literature also indicated that monitoring of early-phase products or pilots can help
inform program design and implementation (Lal et al., 2012; Chiavari & Tam, 2011).

• Research and evaluation of RD&D programs is an important element for program
success, but only a few programs have publically-available evaluation reports, which
limits information and knowledge sharing in this field. Program-level research and
evaluation helps identify targeted areas of improvement and also establish credibility
and build the case for program investments. In addition, while the findings may be
specific to a particular program, they serve as a valuable contribution to the field as a
whole.
14.1.3 Conclusions, Lessons Learned, and Recommendations

The primary findings from this review of programs and best practices point to several key
takeaways regarding the design and implementation of RD&D programs:
• Program investment areas and priorities should be closely aligned with energy
policy environment.
• Project selection criteria must be rigorous and reflect overall program goals and
objectives. Transparent application and selection processes for grantees is a positive
asset for RD&D programs.
• Public-private partnerships are vital to addressing financial and other barriers to
commercialization.
o While financial investments from both the public and private sectors are
important, RD&D programs also benefit from sharing of other resources
such as technical expertise and equipment.
o Successful RD&D programs provide researchers and funders with the tools
and support to navigate the commercialization process. Formal mechanisms
might include systems for sharing license agreements, business coaching for
researchers, and networking opportunities for researchers and funders.
• Technology innovation typically starts with basic research and ends with a
commercialized product, but the process is not linear, which requires program
stakeholders such as administrators, implementers, and partners to be adept at
identifying and responding to opportunities for continuous improvement.
• Program- and project-level evaluation is important for identifying areas of
improvement and for establishing credibility and justifying program investments.
Such evaluation can be carried out by program staff to support internal monitoring
and reporting or by a third-party external evaluator.

14.1.4 Appendix A: Programs and Documents Included in the Review
This appendix provides brief descriptions of the programs, reports, and resources
included in the review of best practices.
Programs
We identified and reviewed websites and publicly-available reports for the following
programs.
DOE Small Business Innovation Research Program (SBIR)

Website: http://science.energy.gov/sbir/
Description: The SBIR Program awards Federal Research/Research and Development

(R/R&D) grants to small businesses through various federal agencies, including the
Department of Energy (DOE). According to the original charter, SBIR and STTR (described
below) have four program goals: 1) stimulate technological innovation; 2) use small
business to meet Federal R/R&D needs; 3) foster and encourage participation in
innovation and entrepreneurship by socially and economically disadvantaged persons;
and 4) increase private sector commercialization of innovations derived from Federal
R/R&D.105 With a strong emphasis on commercialization, small businesses that receive
awards through the SBIR and STTR programs maintain the rights to any technology
developed and are encouraged to commercialize the technology. The SBIR and STTR
programs offers participation in three phases. Phase I awards grants to fund up to $225,000
over 9 months for feasibility studies. Phase II grants, which are up to $1,500,000 over two
years, support more extensive R&D to develop the scientific and technical merit. Only
Phase I awardees may compete for Phase II funding. Phase III is the period during which
Phase II innovation moves from the laboratory into the marketplace with non-SBIR/STTR
funding. Small businesses are expected to acquire additional funds from private investors,
the capital markets, or from the agency that made the initial award.
Related Evaluation Reports and Documents:
Wessner, Charles W., Editor, Committee for Capitalizing on Science, Technology, and
Innovation, National Research Council. 2008. “An Assessment of the SBIR Program.”
http://www.nap.edu/catalog/11989.html.
Wessner, Charles W., Editor, Committee for Capitalizing on Science, Technology, and
Innovation, National Research Council. 2008. “An Assessment of the SBIR Program at the
Department of Energy.” http://www.nap.edu/catalog/12052.html.
105 From http://science.energy.gov/sbir/about/.

SBA Office of Investment & Innovation. October 2015. “SBA Office of Investment &
Innovation SBIR-STTR Presentation.”
https://www.sbir.gov/sites/default/files/SBA_SBIR-
STTR_Overview_October_2015.pptx.
Oliver, Manny. August 2016. “DOE’s Small Business Innovation Research (SBIR) and
Small Business Technology Transfer (STTR) Programs. Webinar slides.
http://science.energy.gov/~/media/sbir/pdf/docs/2017/FY17_Phase_I_Release_1_FOA
_Webinar.pdf.
DOE Small Business Technology Transfer Program (STTR)

Website: http://science.energy.gov/sbir/
Description: Like the SBIR program, STTR is administered by the DOE’s Office of
Investments and Innovation. As a sister-program of SBIR, the STTR program adheres to
the same goals and grant structure. While SBIR is focused on having small businesses
engage in federal R/R&D, STTR facilitates R&D cooperation between small businesses
and research institutions. As a result, there are two key differences between the two
programs: First, with an SBIR award, the principal investigator (PI) must be primarily
employed by the small business concern (SBC) while the PI of an STTR project can be
employed by either the SBC or research institution. Second, with an STTR, using a research
partner with a Phase I and Phase II award is required, and the minimum level-of-effort
expended by the SBC must not be less than 40% in both Phase I and Phase II. With an
SBIR, the SBC is not required to partner with a research institution; however, if using a
research partner, the minimum level-of-effort expended by the SBC must not be less than
60%.
National Academies of Sciences, Engineering, and Medicine. 2016. “STTR: An Assessment

of the Small Business Technology Transfer Program.”
https://www.nap.edu/catalog/21826/sttr-an-assessment-of-the-small-business-
technology-transfer-program.
See related SBIR/STTR references above.
DOE Advanced Research Projects Agency-Energy Program

(ARPA-E)
Website: https://arpa-e.energy.gov/
Description: The ARPA-E focuses on advancing energy technologies designed to reduce

the dependence on energy imports; reduce energy related emissions; improve energy
efficiency across all sectors of the economy; and ensure that the US remains competitive in

developing and deploying transformational technologies. The program provides funding
for technologies that are too early for private sector investment but have the potential to
lead to new ways to generate, store, and use energy. Project staff also receive direct
training and critical business information as part of the Agency’s Technology-to-Market
program. This support equips projects with a clear understanding of market needs to
guide technical development and help projects succeed in the marketplace.
Advanced Research Projects Agency-Energy Program (ARPA-E). “October 2013. ARPA-E

Strategic Vision 2013.” https://arpa-e.energy.gov/sites/default/files/ARPA-
E_Strategic_Vision_Report_101713.pdf.
Advanced Research Projects Agency-Energy Program (ARPA-E). 2016. “ARPA-E: The First
Seven Years, A Sampling of Project Outcomes.” https://arpa-
e.energy.gov/sites/default/files/documents/files/Volume%201_ARPA-
E_ImpactSheetCompilation_FINAL.pdf.
Advanced Research Projects Agency – Energy. July 2015. “Advanced Research Projects
Agency – Energy Annual Report for FY2014.” https://arpa-
e.energy.gov/sites/default/files/FY14%20Annual%20Report%207_27_0.pdf.
Clean Energy Manufacturing Initiative (CEMI)

Website: http://energy.gov/eere/cemi/clean-energy-manufacturing-initiative
Description: The Clean Energy Manufacturing Initiative (CEMI) is a DOE-wide endeavor

designed to enhance U.S. manufacturing competitiveness while advancing the nation's
energy goals, boosting the economy, and contributing to energy security. CEMI engages in
activities that focus on innovation and breaking down market barriers, including
technology research and development; new innovation models; competitiveness analysis;
stakeholder engagement; and energy productivity technical assistance.
U.S. Department of Energy. 2016. “The Clean Energy Manufacturing Initiative:

Strengthening American Manufacturing and Clean Energy Innovation.”
http://energy.gov/sites/prod/files/2016/10/f33/CEMI%20Publication-WebR.pdf.
Executive Office of the President National Science and Technology. February 2012. “A
National Plan for Advanced Manufacturing.”
https://www.whitehouse.gov/sites/default/files/microsites/ostp/iam_advancedmanuf
acturing_strategicplan_2012.pdf.

Executive Office of the President National Science and Technology Council Advanced
Manufacturing National Program Office. January 2013. “National Network for
Manufacturing Innovation: A Preliminary Design.”
http://energy.gov/sites/prod/files/2013/11/f4/nstc_jan2013.pdf.
New Mexico Small Business Assistance (NMSBA) Program

Website: http://www.nmsbaprogram.org/
Description: The NMSBA Program fosters collaboration between New Mexico small
businesses and the Los Alamos and Sandia national laboratories. Small businesses can
receive assistance from lab scientists or engineers for projects that require testing, design
consultation and access to equipment or facilities that are not available in the private
sector. The NMSBA Program offers three types of projects: individual, leveraged, and
contract. Individual projects involve a single small business tackling a problem with
national laboratory expertise. Requests for individual projects are accepted year-round
until funding is exhausted. Leveraged projects include multiple small businesses with
shared technical challenges. Proposals for leveraged projects are reviewed twice and
awards range from $20,000 to $100,000 per laboratory. Contract projects allow small
businesses to contract for services typically not available in the private sector at a
considerably reduced cost (such as courses on renewable energy development).
New Mexico Small Business Assistance (NMSBA). 2015. “2015 Annual Report for the New
Mexico Small Business Assistance (NMSBA) Program.”
http://www.nmsbaprogram.org/userfiles/2016NMSBAperspectivesFY15_Final_lowres.p
df.
NYSERDA Technology and Market Development Program

Website: http://www.nyserda.ny.gov/All-Programs
Description: The mission of the T&MD Program is to “test, develop, and introduce new
technologies, strategies, and practices that build the statewide market infrastructure to
reliably deliver clean energy to New Yorkers.” The specific objectives include: moving
new and under-used technologies and services into the marketplace to help achieve the
goals for the Energy Efficiency Portfolio Standard (EEPS) and Renewable Portfolio
Standard (RPS) Programs goals; validating emerging energy efficiency, renewable, and
smart grid technologies/strategies and accelerate market readiness; stimulating
technology and business innovation to provide more clean energy options and lower cost
solutions, while growing the state’s clean energy economy; and spurring actions and
investments to achieve results distinct from incentive-based programs. T&MD portfolio is
designed to support these objectives by funding nine initiatives in a range of areas,
including power supply and delivery, building systems, and clean energy infrastructure.

New York State Energy Research and Development Authority. June 2016. “NYSERDA
Technology and Market Development Program Semiannual Report through June 30, 2016,
Final Report.” https://www.nyserda.ny.gov/-
/media/Files/Publications/PPSER/NYSERDA/tmd-report-2016jun.pdf
Ridge, Richard and Helen Kim. “Value-Cost Assessment of New York Energy $martsm
Research and Development Program.” Prepared for the New York State Energy Research
and Development Authority, August 2005.
Washington State Clean Energy Fund

Website: http://www.commerce.wa.gov/growing-the-economy/energy/clean-energy-
fund/
Description: The Washington State Clean Energy Fund invests in clean energy
development by supporting the “. . . development, demonstration, and deployment of
clean energy technologies that save energy and reduce energy costs, reduce harmful air
emission, or otherwise increase energy independence for the state.” The fund provides
funding for a range of projects, including grants to electric utilities for smart grid projects,
grants to leverage support for research and development, financing opportunities for
renewable energy manufacturing, and grants to nonprofit lenders that provide capital to
residential and commercial consumers who install renewable energy systems and make
other energy-efficient upgrades.
Weed, Rogers, Washington State Department of Commerce. December 2011. “2012

Washington State Energy Strategy.” http://www.commerce.wa.gov/wp-
content/uploads/2016/06/energy-state-strategy-2012.pdf.
Washington State Department of Commerce. August 2016. “Clean Energy Fund Update.”
http://www.commerce.wa.gov/wp-content/uploads/2016/08/Energy-Fund-Update-
Aug-2016.docx.
State of Washington, Office of the Governor. 2014. “Executive Order 14-04 WA: Carbon
Pollution Reduction and Clean Energy Action.”
http://www.governor.wa.gov/sites/default/files/exe_order/eo_14-04.pdf.
Connecticut Green Bank

Website: http://www.ctgreenbank.com/
Description: The Connecticut Green Bank furthers the adoption of clean and renewable
energy solutions by making financing available to homeowners, businesses,
municipalities, and capital providers. The focus of the Green Bank is to attract and deploy

capital to fill the investment gap needed to support the successful implementation of the
state’s clean energy policy goals. To make clean energy more affordable and accessible to
consumers, the Green Bank is structured to address four consumer sectors: residential
(single and multifamily properties), commercial and industrial, institutional (state,
municipal, universities, schools, and hospitals) and infrastructure (grid-tied projects as
well as statutorily required programs such as the Residential Solar Investment Program
and the Anaerobic Digester Pilot Program).
Connecticut Green Bank. October 2015. “Connecticut Green Bank Comprehensive Plan:
Fiscal Years 2015 and 2016.” http://www.ctgreenbank.com/wp-
content/uploads/2015/11/CGB_FY15_and_FY16_Comprehensive_Plan.pdf.
Cadmus Group. September 2016. “Moving Forward with Green Energy: Market Potential
Assessment for Alternative Fuel Vehicles in Connecticut.”
http://www.ctgreenbank.com/wp-content/uploads/2016/09/CTGreenBank-Market-
Potential-Assessment-Alternative-Fuel-Vehicles-090816-FF.pdf.
Cadmus Group. March 2016. “Cost-Effectiveness Assessment of the Residential Solar

Investment Program.” http://www.ctgreenbank.com/wp-
content/uploads/2016/03/RSIP_Evaluation_II_Final_Report_and_cvr_ltr1.pdf.
Cadmus Group. January 2015. “Residential Solar Investment Program Evaluation.”

http://www.ctgreenbank.com/wp-
content/uploads/2016/03/RSIP_Evaluation_I_Final_Report_and_cvr_ltr.pdf.
Best Practices Literature

We reviewed the following sources to identify effective RD&D practices previously
identified in the literature.
Chiavari, Joana and Cecilia Tam. 2011. “Good Practices Policy Framework for Energy
Technology Research. Development and Demonstration (RD&D).” Prepared for the
International Energy Agency.
https://www.iea.org/publications/freepublications/publication/good_practice_policy.p
df.
Hughes, Mary Elizabeth, et al. June 2011. “Technology Transfer and Commercialization
Landscape of the Federal Laboratories.”
https://www.ida.org/idamedia/Corporate/Files/Publications/STPIPubs/ida-nsp-
4728.ashx.

International Energy Agency. 2010. “Global Gaps in Clean Energy RD&D: Update and
Recommendations for International Collaboration.”
https://www.iea.org/publications/freepublications/publication/global_gaps.pdf.
Jenkins, Jesse and Sara Mansur. November 2011. “Bridging the Clean Energy Valleys of
Death: Helping American Entrepreneurs Meet the Nation’s Energy Innovation
Imperative.” http://thebreakthrough.org/blog/Valleys_of_Death.pdf.
Lal, Bhavya, Gupta, Nayanee and Christopher L. Weber. January 2012. “Innovation
Pipeline Management: Lessons Learned from the Federal Government and the Private
Sector.”
https://www.ida.org/idamedia/Corporate/Files/Publications/STPIPubs/2014/D-
5367.ashx.
Murphy, L. M. and P. L. Edwards. 2003. “Bridging the Valley of Death: Transitioning from
Public to Private Sector Financing.” http://www.nrel.gov/docs/gen/fy03/34036.pdf
Ruegg, Rosalie and Irwin Feller, 2003, A Toolkit for Evaluating Public R&D Investment:
Models, Methods, and Findings from ATP’s First Decade. Prepared for the Economic
Assessment Office, Advanced Technology Program, National Institute of Standards and
Technology, Gaithersburg, MD.
Townsend, Brad and Erin Smith. March 2016. “U.S. Energy R&D Architecture: Discreet
Roles of Major Innovation Institutions.” http://cdn.bipartisanpolicy.org/wp-
content/uploads/2016/03/BPC-AEIC-Energy-RD-Architecture.pdf.
U.S. Department of Transportation, Research and Innovative Technology Administration.

March 2011. “Key Findings and Recommendations for Technology Transfer at the ITS
JPO.” http://ntl.bts.gov/lib/42000/42100/42107/FHWA-JPO-11-
085__Key_Findings___Recommendations_for_Tech_Transfer_at_ITS_JPO__PDF_508.pdf
Wessner, Charles. 2008. “Best Practices in State and Regional Innovation Initiatives:
Competing in the 21st Century.” https://www.nap.edu/download/18364.
Wessner, Charles. 2003. “Government-Industry Partnerships for the Development of New

Technologies.” Washington, D.C.: The National Academic Press.”
https://pdfs.semanticscholar.org/4546/9283a7c5b782a9d736d242dd5c5338050bfa.pdf
White House Conference Center. 2013: “Lab-to-Market Inter-Agency Summit:

Recommendations from the National Expert Panel.”
https://www.aau.edu/WorkArea/DownloadAsset.aspx?id=14535.

14.2 In-Depth Interviews
This section summarizes findings from interviews conducted with seven administrators
involved in the design and implementation of energy research, development, and
demonstration (RD&D) programs. The team conducted these interviews as part of the best
practices review, which seeks to identify key learnings from programs designed to
increase coordinated investment in new and emerging energy solutions.
14.2.1 Introduction
Objectives
The objective of the best practices review is to identify effective practices that have been
used to support RD&D programs. The team initially conducted a literature review to
summarize the best practices and lessons learned regarding the planning, design, and
implementation, of RD&D programs. This literature review identified a selection of
programs similar to the EPIC program, and compiled findings from publicly-available
evaluation reports and other documents about these peer programs. While the literature
provided valuable insights, the team identified specific areas to investigate further
through in-depth interviews with individuals involved in the administration of these
programs. This memo summarizes the findings from those interviews.
Methodology
The initial literature review identified several programs similar to the EPIC program.
Through the in-depth interviews and further review of program documents, the team
determined that four of the seven programs initially identified were very similar to the
EPIC program in terms of their objectives and mission. These primary peer programs
include:
• Department of Energy’s Small Business Innovation Research Program (SBIR)

• Department of Energy’s Small Business Technology Transfer Program (STTR)
• Department of Energy’s Advanced Research Projects Agency-Energy Program
(ARPA-E)
• NYSERDA’s Technology and Market Development (T&MD) Program
The literature review also identified three other peer programs that, upon closer
investigation, were revealed to be not as similar. Although these three programs support
innovative R&D efforts, they do not have an explicit goal related to commercialization,
which is a key feature of the EPIC program. These secondary peer programs include:
• New Mexico Small Business Assistance (NMSBA) Program

• Washington State Clean Energy Fund
• Connecticut Green Bank

The interviews covered a range of topics, including program background, project selection
processes, program administration and management, program tracking and reporting,
support for commercialization, knowledge dissemination and technology transfer, and
general program challenges and successes.
14.2.2 Summary of Findings

This memo builds on the initial literature review of best practices and focuses on the
following areas that were identified for further investigation:
• Processes used to conduct market assessments, identify gaps and opportunities,

and determine program or portfolio-level investment priorities.
• More detail on how program administrators coordinate with implementation
agencies and other partnering organizations.
• Information on how program and project-level systems are used to monitor and
evaluate program performance, including insights on the frequency of data
collection and reporting as well as how related findings inform planning, design,
and implementation.
• How RD&D programs adapt and respond to rapidly changing technologies and
markets.
We have grouped the key findings from these interviews in the five research categories
that the overall EPIC evaluation will cover: 1) Program Management and Administration,
2) Policy Alignment, 3) Investment Planning, 4) Project Selection, and Project Assessment.
Table 65 below summarizes the broad categories of effective practices that were referenced
during interviews with the peer programs.

Table 65: Program Comparison of Effective Practices
Primary Secondary
Peer Programs Peer Programs
NYSERDA T&MD
CT Green Bank
WA CEF
NMSBA
ARPA-E
STTR
SBIR
Effective Program Practices
Core internal staff that coordinates with internal ü ü ü ü ü ü ü
and external stakeholders
Require match funding ü* ü* ü ü ü NA
Offer support and resources to promote
commercialization activities ü** ü** ü ü NA NA NA
Explicit requirement for project-based knowledge
or technology plan ü ü ü NA NA NA
Policy Alignment
Explicit alignment with federal or state policy
goals ü ü ü ü ü ü ü
Investment Planning
Directly engage external stakeholders in
determining investment priorities ü ü ü ü
Offer focused funding opportunity ü ü ü ü NA
Offer focused and open funding opportunity ü ü NA
Project Selection
Transparent review criteria and selection process ü ü ü ü ü ü NA
Utilize external reviewers ü ü ü ü ü ü NA
Requirement for applicants to indicate initial
interest prior to submitting a full proposal (e.g.,
concept paper, letter of intent) ü ü ü ü NA
Project Assessment
Formally track metrics program-wide ü ü ü ü ü ü
Require regular progress reports ü ü ü ü ü ü ü
Use third-party evaluation to assess program
impact ü ü ü ü ü
* SBIR and STTR do not mandate match funding, the programs do encourage it for projects that fall
under their Phase II funding, which has an increased emphasis on commercialization.
** SBIR and STTR do not directly offer commercialization supports, but grantees may access the
Commercialization Accelerator Program, which provides assistance to all federal programs.

• The interviewees from all of the peer programs all spoke about the depth of
knowledge and expertise of the program administrators, consultants, and other
stakeholders that contribute to the program’s implementation. Each of the RD&D
programs employs staff with a strong technical expertise. Because these programs
support new and emerging technologies, the programs also rely on other experts
both within and outside of their respective agencies who may have specialized
knowledge of certain technologies or concepts. For example, when developing a
funding opportunity, the program administrators will convene industry experts to
help inform the initiative (see Investment Planning below for more detail).
Interviewees also spoke about coordinating with other state or federal agencies and
industry groups to stay current on topics affecting their work.
• Three of the seven peer programs, including two primary peer programs, require
projects to obtain match funding; while they do not directly assist with finding the
match funding, to the extent that they can, they try to help address any barriers the
project team may face. Washington State’s Clean Energy Fund requires a 50% cost-
share, NYSERDA’s T&MD Program requires 25% or 50% cost-share depending on
the size of the project, and ARPA-E’s cost-share varies by specific funding
opportunity. While SBIR and STTR do not mandate match funding, it is encouraged
for projects that fall under their Phase II funding, where the programs place
increased emphasis on commercialization.
• Contracting times vary substantially across the different peer programs. Among all
of the peer programs, one of the secondary peer programs has the shortest
turnaround of one week. On the other end of the spectrum, a primary peer program
takes roughly three to six months on average. Another primary peer program
reportedly takes six weeks but allows the project team to begin working on the
project at-risk up to 90 days before the contract is finalized.
• Two of the four primary peer programs offer support and resources to promote
commercialization activities. ARPA-E’s Tech-to-Market Team provides direct
business-related technical support and assistance to grantees. In addition to
providing guidance on the project team’s Technology-to-Market Plan, the Tech-to-
Market advisors assist projects in a variety of ways, including helping them to
conduct market assessments, identify potential investors, and develop and deliver a
business pitch. The ARPA-E program also holds an Annual Energy Summit that
convenes individuals and organizations from industry, academia, and government
and provides an opportunity for project managers to meet with potential partners,
collaborators, or investors.
Projects funded through NYSERDA’s T&MD can also access commercialization-
related resources available through the Entrepreneurs-in-Residence (EIR) Program,

which is very much like ARPA-E’s Tech-to-Market Team. In addition, NYSERDA
funds six business incubators across the state and supports a Proof-of-Concept
Center where businesses can access existing intellectual property (IP) and work on
commercializing it.
• All four of the primary peer programs have structures in place to help projects
identifying and capitalizing on opportunities to change course, when needed, to
maximize the projects’ success. Because the technology innovation process is not
linear, it often requires program administrators, grantees, and others involved in
the RD&D process to be adept at recognizing critical opportunities to maximize.
The SBIR program allows projects to be able to reallocate up to 10% of their budget
without prior approval. Because ARPA-E and NYSERDA’s T&MD Program both
have an explicit focus on commercialization, advisors for their respective programs
actively work with project teams to identify appropriate pivots, and may help
facilitate these course corrections.
Policy Alignment
• Interviewees from all seven peer programs indicated that their initiatives are
explicitly aligned with federal or state local policy goals. For example, as the
primary author for the New York State Energy Plan, NYSERDA played a key role in
defining energy policy goals and objectives. As a result, the T&MD Program is
closely coordinated with the Energy Plan recommendations and helps contribute to
the state’s energy goals. While the SBIR and STTR programs do not establish the
technology areas that they fund, the program administrators work closely with the
relevant R&D program offices at DOE. These program offices typically develop
their research priority areas, often driven by roadmaps or other investment
planning mechanisms, that are aligned with their mission. SBIR and STTR
programs help facilitate RD&D in these priority areas by stimulating technology
innovation among small businesses.
While the ARPA-E program is mission driven, it is the only peer program that
intentionally is not bound by policy roadmaps and other similar directives. As a
result, the program takes into consideration such documents, but specifically seeks
to fund the “disruptive technologies and solutions” that may not necessarily be
identified in these investment planning mechanisms. As the Strategic Vision for
ARPA-E states, the program is designed to advance “high-potential, high-impact
energy technologies that are too early for private sector or other DOE applied
research and development investment.”106
• When asked about potential reductions in federal funding and support for energy-
related RD&D, and what EPIC could do to address those gaps, interviewees offered
Advanced Research Projects Agency-Energy Program (ARPA-E). “October 2013. ARPA-E Strategic Vision
106
2013.” https://arpa-e.energy.gov/sites/default/files/ARPA-E_Strategic_Vision_Report_101713.pdf.

a range of responses but few concrete strategies. One interviewee stated that the
EPIC program should closely monitor federal funding for RD&D programs and
allocate funding in areas that are negatively affected by budget or policy changes,
but did not specify technology areas to target. Two interviewees suggested that
private investment might help fill the financial gap should the federal government
reduce RD&D funding, but did not recommend specific ways to do so. Two
interviewees also indicated that with the recent change in the administration, it was
too soon to predict what implications that would have on Congress’ funding
decisions.
Investment Planning
• All seven of the peer programs are informed in some way by external stakeholders;
interviewees from the four primary peer programs noted that they have formal
mechanisms to solicit input to identify and develop funding opportunities. As
noted above, the peer programs are directly influenced by policy directives, which
involves engagement with a variety of external stakeholders including federal and
state government agencies, legislators and legislative committees, businesses, and
other interest groups. The primary peer programs also formally engage external
stakeholders in identifying investment areas. The DOE convenes planning
workshops and other meetings that help inform funding priorities for individual
research offices. These investment areas support research funding, and a portion is
allocated to small business initiatives like SBIR and STTR. The ARPA-E program
uses a different approach. ARPA-E rotating Program Directors have a considerable
degree of autonomy in defining funding opportunities. ARPA-E’s use of rotating
Program Directors helps the program bring in fresh ideas based on their prior
experience and expertise. Also, because they have a three-year tenure with ARPA-
E, they typically are working within a relatively short timeframe and are attempting
to make a significant impact during their time with the program. These staff
conceptualize focused technical funding opportunities, which are informed by
meetings with external stakeholders. The Program Director takes the lead on
soliciting input from external stakeholders as well as other Program Directors and
leadership staff within the program, and ultimately has a great degree of ownership
in developing funding opportunities and selecting projects. Staff from NYSERDA’s
T&MD Program worked with an external consulting firm to determine the
program’s investment priority areas.
• Six of the seven peer programs offer focused and clearly-defined funding
opportunities; two of the four primary peer programs – ARPA-E and NYSERDA’s
T&MD Program additionally offer an open funding opportunity to encourage ideas
that not covered in focused program areas. In general, the focused funding
opportunities are developed by program staff and are designed to address the
mission and goals of the program. These opportunities offer funding to specific

technology areas such as battery storage, building efficiency, grid modernization,
etc. To foster innovative approaches and solutions, both ARPA-E and NYSERDA’s
T&MD Program also offer funding beyond the traditional focused opportunity.
ARPA-E’s Innovative Development in Energy-Related Applied Science (IDEAS),
accepts, on a rolling basis, proposals for single-phase projects of up to 12 months
and $500,000 of funding. NYSERDA similarly offers funding through The
Advanced Clean Energy Exploratory Research Program (ACE Program); proposals
are accepted on a rolling basis, and project funding and timelines vary.
Project Selection
• All six of the peer programs that offer grants, or, in the case of SBIR and STTR,
cooperative agreements, provide transparent information about RD&D funding
opportunities, proposal review criteria, and the selection process.107 Each of these
programs include information about funding opportunities on their website. In
general, the funding announcements describe technology-related parameters,
funding amount and duration, due date, and award date. SBIR, STTR, and ARPA-E
also have webinar archives that provide background on the program and eligibility
as well as information on how applications are reviewed and selected for funding.
• All four of the primary peer programs require applicants to provide a preliminary
research description prior to developing and submitting a full proposal. SBIR and
STTR require a letter of intent while ARPA-E and NYSERDA’S T&MD Program
expect a concept paper. While the specific requirements are different for each
program, this preliminary description serves a few purposes. It allows program
administrators to get a sense of the technology areas that are likely to be addressed
in response to a particular solicitation, which helps them determine the types of
external reviewers that will be needed for proposal review. It also allows the
applicant to describe their proposed research with relatively modest commitment in
terms of time and resources. Based on a review of the preliminary description, the
programs provide applicants with a notice of encouragement or discouragement to
submit a full proposal.
Project Assessment
• Six of the seven peer programs use formal metrics to track project performance.
Although the metrics of the peer programs vary based on their stated focus and
technology, the interviewees generally reported that they do track such indicators
and do so on a regular basis. Among the primary peer programs, common project-
based metrics include patents, copyrights, and publications. Metrics tracked across
program portfolios typically include indicators such as progress toward
commercialization, sales revenues, and leveraged funding.
107 The Connecticut Green Bank provides financing for commercially-available clean energy technologies.

• All seven of the peer programs stated that projects are required to report regularly
on their progress. For example, projects funded through NYSERDA’s T&MD
Program provided monthly or quarterly performance reports, which the assigned
Project Manager uses to track progress on milestones identified in the Statement of
Work. Many T&MD projects also receive support through the Entrepreneurs-in-
Residence (EIR) program, a resource available to companies engaged in
commercialization activities (described above under Program Management and
Administration). Through these interactions, the main points of contact between the
EIR program and the project often develop close relationships, and the EIR advisor
is often privy to project status updates on an ongoing basis. Interviewees
representing other peer programs similarly reported regular interaction between
program administrators and grantees through written reports (monthly, quarterly,
or annual), on-site visits, or phone calls.
• Five of the seven peer programs, including all four of the primary peer programs,
use an external evaluator to assess the program’s performance. In addition to
internal tracking reporting mechanisms, interviewees from the primary peer
programs stated that they have used third-party evaluations aggregate, assess, and
report on program-wide outcomes. The National Academies of Sciences recently
published assessments of the SBIR and STTR programs.108,109 NYSERDA also has a
separate evaluation group, which works with a third-party to evaluate the T&MD
Program’s effectiveness.110 ARPA-E recently completed an independent evaluation,
which is forthcoming. Interviewees generally indicated that these external efforts
help provide objective insights regarding the program’s performance are useful for
reporting to external program stakeholders.
14.2.3 Conclusions
The findings from these interviews fill important gaps and supplement information
previously identified in the team’s literature review.
• The interviewees confirmed findings from the literature regarding the importance
of a core internal program staff that the coordinates with both internal and external
stakeholders in the design and implementation of the RD&D program.
108 National Academies of Sciences, Engineering, and Medicine. 2016. STTR: An Assessment of the Small
Business Technology Transfer Program. Washington, DC: The National Academies Press.
https://www.nap.edu/catalog/21826/sttr-an-assessment-of-the-small-business-technology-transfer-
program.
109 National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy.
Washington, DC: The National Academies Press. https://www.nap.edu/catalog/23406/sbirsttr-at-the-

department-of-energy.
110 New York State Energy Research and Development Authority. June 2016. “NYSERDA Technology and
Market Development Program Semiannual Report through June 30, 2016, Final Report.”
https://www.nyserda.ny.gov/-/media/Files/Publications/PPSER/NYSERDA/tmd-report-2016jun.pdf.

• Interviewees provided greater detail on formal mechanisms used to identify
investment priorities. The various approaches indicate that EPIC’s current methods
are consistent with its peer programs.
• EPIC generally is consistent with the peer programs with regard to transparency
about funding opportunities, proposal review criteria, and the overall selection
process.
• Peer programs use formal metrics to regularly track projects’ performance. These
metrics are specific to the program’s stated focus, but common metrics include
patents, copyrights, publications, progress towards commercialization, and
leveraged funding.

15 Appendix D: Example of Project Tracking
Recommendation
To address unmet needs and missed opportunities in sharing relevant project-specific
updates and information in a timely and effective manner, we suggest that the
administrators collaborate to develop an electronic, searchable database that provides
relevant project updates on a quarterly basis. Initially, this database could be a simple
workbook sent to the CPUC, but there are opportunities to make it more widely available
by publishing it via the Internet.
Key attributes of this database should be:
• Search or sort capability by policy or technology area so users tracking a particular

topic can get an overview of EPIC activity and future outcomes in that topic area;
• Ease of administering so the assembly and updating of the database relies on
readily available existing information that can be easily compiled;
• Descriptive value so a reader can understand the general scope and scale of each
project (as defined by a brief project description, list of organizations involved, and
budget information); and
• Tracking value so a reader can follow the status of the project and any interim
results that can be shared or outputs that are being made public.
We have developed an illustrative example of how such a database could be structured

and the information it could contain. Figure 30 shows how projects related to energy
storage could be grouped and the kind of content it could provide. Some of the
example data is based on real information taken from existing Annual Reports or
project data, while other information is made up for illustrative purposes only. We
show the latter in blue text.
We note that our illustrative example is based on existing information about projects or
information that could easily be obtained consistently in project updates that
administrators already track (e.g., either internally by the IOUs, or by grantees to the
CEC). Specifically, we suggest that:
• Searchable fields include the EPIC Triennial Investment Plan, the administrator,
and one or two pre-existing tracking variables that identify the technology or policy
area that the project is intended to help inform or advance (columns B and C in our
illustrative example);
• Static descriptive information about the project, including its name, date of award,
date of expected completion, the value chain to which it is assigned (generation,

transmission, etc.), EPIC and matching funds budgeted; and grantees, vendors, and
partners involved in the project;
• Current project information, such as funds expended, activities to date, and any
updates to the project description; and
• References to outputs and outcomes that have already been delivered and are ready
to be made public.
This last point requires a bit more explanation. The intent of including outputs and
outcomes that can be made public is to facilitate information sharing with stakeholders
who are tracking a particular topic area in near-real time. We anticipate the inclusion of
references to papers, publications and presentations that have been given or are
anticipated in which project information and results are being made public anyway. At the
study team’s discretion, hypotheses and interim results could be included too, but those
would need to be clearly identified as such. We do not anticipate any proprietary
information that is not intended to be made public to be included in these reports.
The vast majority of the information in this illustrative list of project updates is already
available to administrators, and much of it is reported in current Annual Reports. Project
outputs and outcomes would be new additions, but we believe that information such as
presentations given and publications created can be easily gathered as part of the project
updates required of grantees.

Figure 30: Quarterly Report by Project Area
A. B. C. Sub-Technology / D. Program E. Project Name** F. Project G. A Brief Description of the Project H. Date I. Expected J. Assignment K. EPIC L. Matching M. Funds N.Partners (Primary O. Update
Investment Technology / Policy Area Administrator Type of the Date of Project to Value Chain Funds Funds Expended to Grantee or Vendor:
Program Policy Area Award Completion Committed Committed Date: Subcontractors)
Period ($) ($) Contract/Grant
Amount ($)
2012-2014 CSP Grid Scale Storage CEC EPC-14-003 Low- Applied The purpose of this project is the 1/15/15 3/15/18 Generation $1,497,024 $300,000 $188,054 Listed as Southern A project kickoff meeting was held in
Triennial Cost Thermal Energy Research and development and demonstration of a cost- California Gas Company April 2015 and technical progress is
Investment Storage for Development optimal, robust, and low-cost thermal in annual report, but underway. The project team has
conducted heat transfer modeling and
Plan Dispatchable CSP energy storage (TES) fluid, elemental should be UCLA
simulation as well as laboratory-scale
sulfur. Use of sulfur as a TES fluid will material compatibility experiments that
enable overall low system costs, long will feed into on-sun testing in 2017.
lifetime, and scalability for a wide range
of concentrating solar power (CSP)
applications and temperatures.
2012-2014 Grid-Scale Market Identification & PG&E PG&E #1.01: Energy Technology Develop and deploy technology to enable 9/19/13 9/19/16 Distribution; $2,030,000 n/a $1,833,110 DC Systems, Power After completing the process flow and
Triennial storage Market Pairing Storage for Market Demonstration fully automated resource response to Grid Settlements, Trimark software components of the project,
Investment Operations and CAISO market awards. Report financial Operation/Mark the project partners attempted to
market the products commercially in
Plan Deployment/ performance from participation in CAISO et Design
the CAISO markets. Currently, market
Renewables/D markets Report comparison of actual demand appears to be low although
ER Resource performance vs. hypothetical the administrators and other project
Integration performance quoted in industry reports. partners will continue to gauge market
Comply with regulatory requirements and response going forward if plans for
establish a framework/recommendations commercialization appear viable.
for accounting standards applicable to
energy storage.
Illustration for Quarterly Reports by Project Area (Columns A – O)
P. Activities to date Q. Outputs to date R. Outputs to watch for S. Short-Term Outcomes: Working hypotheses, T. Mid-Term & Long-Term Outcomes:
learnings, and results Anticipated project impact on energy
technology and commercialization
Laboratory testing of elemental sulfur's relevant Project presentation at Laboratory results will be documented in a laboratory Analysis of elemental properties of sulfur in storage setting Grantee is working with a newly formed
properties and storage container have been December 2015 EPIC demonstration report and available from Kevin Uy at CEC and storage container in laboratory have shown feasibility firm Element Sixteen Technologies,
completed. Symposium and four in March 2017. Grantee is submitting a manuscript to the of safe sulfur-based thermal energy storage. Results so far Incoporated, to further development and
presentations at ASME Power referreed journal Photovoltaics of the Future. If acccepted, suggest that performance metric of storage costs of below commercialization of sulfur-based
and Energy Conference in anticipated publication in Summer 2018 edition. Project $15/kWh and levelized cost of solar power below thermal energy storage. CEC is
June 2016. Available from report with detailed lab and field results planned for March $0.06/kwh will be achievable. Actual field results will be coordinating with CPUC, IOUs, and US
Kevin Uy at CEC 2018 and will be available at epic.cec.ca.gov. available by March 2018. DOE on an energy storage roadmap,
(kevin.uy@energy.ca.gov). which includes thermal energy storage.
Demonstrated the use of PG&E's Vaca-Dixon and Completed a final report in Project completed, no future outputs anticipated. However, Based on market participation analysis, current market Due to the achievement of the project,
Yerba Buena Sodium Sulfur (NAS) Battery September 2016, project results have led to the development of new flexible dynamics do not favor long-duration batteries given that administrator will continue to maintain the
Energy Storare Systems (BESSs) to provide summarizing project ramping product in CAISO market most significant revenues in the market are from frequeny Automated Disipatch System as a
energy and ancillary services in CAISO markets objectives, technical results regulation - a power rather than an energy product - project platform to automate the response of
as the first battery storage resources in California and lessons learned through administrators concluded a 30-minute BESS might be ablet current and future batter storage
to participate in the market. Developed and EPIC project. Presented at o provide the same FR capabilities as a 7-hour system with resources to CAISO market awards.
deployed a scalable technology platform to the 2015 EPIC Symposium as less capital investment. Administrator also plants to provide a
automate the response of current and furture well as several other industry new flexible ramping product that CAISO
PG&E battery storage resources to CAISO conferences during the plans to introduce in late 2016.
market awards vis Automated Dispatch System course of the project.
(ADS). Developed optimization models and
workflow processes for efficient bidding of battery
resources into the CAISO market,
Illustration for Quarterly Reports by Project Area (Columns P - T)

16 Appendix E: Supplemental Information on EPIC
Project Performance
This Appendix provides more details regarding the progress that each of the three
program areas has made with respect to its expected outputs and outcomes, which was
summarized in Section 10.5. These results are based primarily on 90 in-depth interviews
with members of EPIC project teams (CEC administrators, CEC Commission Agreement
Managers (CAMs), CEC grantees, IOU project managers, and, in some cases, IOU vendors)
associated with 54 sampled projects.
16.1 Applied Research and Development Projects

In this section, we present results to date with respect to the outputs and outcomes
depicted in the Applied R&D logic model in Appendix B. These results are based on the
analysis of a random sample of nine Applied R&D projects from a population of 121. We
reviewed project details and conducted in-depth interviews with CAMs and grantees to
develop a deep understanding of these nine sampled projects. Note that all of these
projects are administered by the CEC, since the IOUs are authorized to work only on
TD&D projects.
Because none of the sampled projects were (at the time of the interviews) complete, we
expected to observe some outputs but no measureable impacts with respect to any short-
term, mid-term or long-term outcomes. However, we were able to examine indicators as to
whether the projects are on a path to achieving these desired outcomes if the Applied R&D
proves successful. Such leading indicators include:
• Knowledge creation and knowledge utilization;

• Private and public investment (especially for technologies intended to be
commercialized);
• Follow-on research and development (if needed);
• Technology demonstrations (for technology-based projects); and
• Technology or tool adoption.
We discuss the degree to which existing Applied R&D projects show progress by these
indicators below. In order to connect the reader to the overarching framework provided
by the Applied R&D logic model, we have included, in parentheses next to each heading,
the letters associated with the output and outcome boxes in the Applied R&D logic model
in Appendix B.

16.1.1 Knowledge Creation and Knowledge Utilization (Logic Model
Boxes H, I, N, T)
Knowledge Creation
Project teams were asked to assess how successful they have been thus far in achieving
their technical performance targets. For these eight active projects, three of the eight
respondents indicated that they have been somewhat to extremely successful in achieving
their technical performance targets. Others said that it was too early to judge.
Another question is whether these active projects are likely to verify the targeted benefits.
Applied R&D projects targeted a range of benefits. We reviewed benefit questionnaires
from seven of nine Applied R&D projects. Table 66 below summarizes the project benefits
targeted by the following general benefit areas:
• Potential Energy and Cost Savings – for example, CEC project 14-030 developed,
demonstrated and evaluated scenarios that identify the most promising
opportunities for waste biomass distributed generation. Expected energy and cost
savings benefits of this project include an increase in distributed generation which
can be used to meet local system needs and lower costs by reducing the need for
transmission infrastructure investment and protecting ratepayers from changes in
fossil fuel prices. CEC project 15-021 will leverage mobile design practices,
hardware components and energy management software to evaluate the energy
consumption of residential and commercial plug-load devices. Expected energy and
cost savings benefits of this project are lower customer electric bills, and individual
ratepayer savings will make a significant contribution to California’s statutory
energy goals.
• Job Creation – CEC project 14-025 developed, demonstrated and evaluated a
photovoltaic solar system with air driven trackers for research purposes. It is
predicted that this project will add approximately 860 jobs per gigawatt.
• Economic Benefits – CEC project 14-030 developed, demonstrated and evaluated
scenarios that identify the most promising opportunities for waste biomass
distributed generation. An expected benefit of this project is a reduction in electrical
losses in the transmission and distribution system, and with integrated use of waste
biomass for distributed generation applications, providing the potential to produce
approximately 4.2 terawatt-hours of renewable electricity per year. CEC project 15-
021 will leverage mobile design practices, hardware components and energy
management software to evaluate the energy consumption of residential and
commercial plug-load devices. Potential economic benefits of this project are, at the
macro level, long-term, deep savings across a broad array of plug load devices.
• Environmental Benefits - for example, CEC project 14-040 will develop, test and
demonstrate Self-Tracking Concentrator Photovoltaic systems. Expected
environmental benefits of this project are GHG emission reductions from increased

solar power generation resulting from new low-cost PV panel technology. CEC
project 14-030, which developed, demonstrated and evaluated scenarios that
identify opportunities for waste biomass distributed generation, will provide
potentially significant environmental benefits. Expected environmental benefits for
this project are substitute energy sources worth significant savings in avoided grid
electricity and natural gas heating. This will result in avoided fossil fuel GHG
emissions of approximately 2.5 million metric tons of carbon dioxide per year.
• Safety, Power, Quality, and Reliability (Equipment, Electric System) – for
example, CEC project 14-040 will develop, test and demonstrate Self-Tracking
Concentrator Photovoltaic Systems. This is expected to provide greater reliability of
the equipment and electric system through increasing the use of distributed solar
generation, which will help improve reliability by reducing the sensitivity to
outages. CEC project 14-025, which developed, demonstrated and evaluated a
photovoltaic solar system with air driven trackers for research purposes, will also
have an impact on the reliability of the electric system and equipment. The expected
benefit of this project is to increase the flexibility of the system by capturing the sun
in the late afternoon to soften the net peak load, which now occurs in the evening.
Table 66: Summary of Sampled Project Projected Benefits

# with # with Total
Quantified Qualitative with Sample
Benefit Benefits Benefits Benefits Size
Potential Energy & Cost Savings 2 3 5 7
Job Creation 1 0 1 7
Economic Benefits 2 3 5 7
Environmental Benefits 4 1 5 7
Safety, Power, Quality and Reliability
1 1 2 7
We also asked grantees how successful they have been, thus far, in validating the expected
benefits from these Applied R&D projects that might eventually lead to longer-term
benefits once the technology/tool is more broadly deployed. For these eight active
projects, three of the eight respondents indicated that they have thus far been somewhat to
extremely successful in validating their targeted benefits. Other said that it was too early
to judge.
Other more concrete indicators of knowledge creation are the creation of progress reports,
final reports, databases, fact sheets, refereed journal articles and trade journal articles. The
CEC closely monitors their projects on a monthly or quarterly basis, with a few even
getting weekly progress reports (discussed in more detail in Section 9). There is also a

general consensus among the CAMs that these reports are very or extremely useful. The
use of these on-going reviews that allow for mid-course corrections are essential for EPIC’s
success and are considered best practice. In addition, two fact sheets have been produced,
but no articles in refereed journals or trade journals have been published, and no software
tools or databases have been created. However, given that all projects are still active, it is
premature to expect any more such outputs.
Knowledge Utilization
While it is important to create knowledge both during and at the conclusion of a project, it
is equally important for these results to be used. Knowledge gained during a project is
used to assess whether the project is on schedule and on budget and whether the
technology is working as expected. Any adjustments are triggered by consultation from
technical advisory committee (TAC) and policy advisory committee (PAC) members,
critical project reviews and discussions between the administrators and the CEC grantees
and IOU project managers. One question focused on the extent to which project teams, as a
result of knowledge gained thus far from their projects, have deviated from their original
CEC-approved research plan. Only two of the project teams reported that they had
experienced any deviations and they were all relatively minor. As a result, none of the
projects have been re-scoped thus far.
Market Actor Awareness

One of the earliest indicators of progress toward future market adoption is a perceived
need, awareness and appreciation of the value of new technology or tools by those market
actors who would use them. We asked grantees how much they thought that potential
buyers or users in the market understand the added value of the technology or tool they
are developing.
Three of the four projects teams111 developing technologies they intend to commercialize
indicated that their target audience has some level of understanding of their technology or
tool so far and the value that it may provide to them,112 while one reported that it was too
early to make that assessment. We did not ask this question of project teams working on
technologies or tools that were not anticipated to be commercialized.
This range of market understanding reflects the various stages of development of the
technologies and the degree to which they vary from existing approaches to achieve the
desired outcome. In some cases, the value of the technology is the key output from the
EPIC project. In these studies, the Applied R&D project is seeking to quantify the cost-
111 References to project teams refer to CEC staff overseeing projects and grantees and principal investigators
leading them. We had conducted separate interviews with both and were informed by their collective
responses.
112 Two of these projects teams characterized their target audience as having a significant level of
understanding, while one said there was a low level of understanding of the technology and its value.

effectiveness of a particular technology or solution to determine whether (or prove that) it
is an improvement over current alternatives.
Of the project teams that do plan to commercialize their technology/tool, three out of the
four believe that buyers and users in the broader market will have a high level of interest
in purchasing the end product regardless of their current understanding of the technology
or tool.
Patents/Copyrights
Two of the eights projects have filed a combined total of three patent applications since the
launch of these EPIC projects, although no patents have yet been granted. The broad
objective of obtaining these patents is to further the development of utility scale renewable
energy generation technologies as well as developing new technologies, tools and
strategies to lower the cost of distributed generation.
No project results have been copyrighted or licensed across all eight Applied R&D
projects; however, one project team stated that it is too early in the project to have
produced such results.
16.1.2 Private and Public Investment (Logic Model Box V)

The willingness of investors to provide additional funding for the development of a new
technology or tool is another indicator of progress toward future market adoption for
technologies or tools that will be commercialized for profit. For technologies or tools that
will be in the public domain or that have only indirect impacts on technology adoption,
such funding may come from public sources instead. To explore the degree to which
additional investment was being pursued and provided, we asked grantees whether they
had sought or secured additional funding from public or private investors since the start
of the EPIC project. We also examined matching funds as evidence of external support and
interest.
Table 67 below shows that the total budget of the eight sampled Applied R&D projects is
$8.9 million with $2.4 million of that budget being spent through 2016. Six of the eight
projects also include match funding totaling $8.9 million with those funds coming from
various private or public sources. Of the six projects with match funding (another
indicator of external support for Applied R&D projects), one project accounts for 68
percent of the total match funding amount.

Table 67: Private and Additional Public Investment for Sampled Applied R&D Projects
Total Project # Projects Seeking Seeking
Total Project Total $ Spent with Federal Private
Budget w/out Match (through Match and/or State Investor
Match Funds Funds 2016) Funds Funding Funding
$8.9 million $8.9 million $2.4 million 6 4 2
*Project total does not include the ninth sampled project that we excluded from this section because it had not yet
started by the time of our project interviews.
Extent of Future Private or Public Investment – Actual and Anticipated

Out of the eight sampled projects, four are seeking additional funding from various
federal and state sources such as the National Science Foundation, Department of Energy
(DOE) and Joint Bioenergy Institute. Two out of the four that are seeking additional
investments have also looked for private partners to invest from either U.S. venture capital
or other domestic private companies. One of the two project teams seeking private
investments has been granted funds, whereas the other is still pursuing their options.
Because the sampled projects are in the early stages, no project teams have had any
interaction with the CEC Market Facilitation program area thus far. However, it was noted
by one project team that although no interaction has occurred, they believe the Market
Facilitation program area will be helpful in overcoming non-technical barriers and
accelerating the commercial viability of their technology when they enter that stage in
their project.
Partnerships Formed
The addition of project partners is also indicative of external interest and can be a source of
private investment of funds (or intellectual capital). Thus far, two project teams have
added a combined eight project partners with each having either a formal written or
informal agreement. None of the project partners provide additional funding; however,
seven are essential in the role of providing their services and expertise in leading
discussions regarding the development of the technology or a relevant new Institute of
Electrical and Electronics Engineers (IEEE) standard. One partner had been working with
the team informally before the start of this EPIC project, and the partnership became
formal during the project. Typically, the project partners meet on an ad hoc basis,
depending on the need at that time of the project.
Finally, we note that all four project teams that have a technology or tool that they plan to
commercialize currently believe that it will be commercially viable, in that it will compete
effectively in the market and make a profit. However, due to the early stages of these
projects, it is not expected for any of these technologies to become commercially viable for
another one to three years. In most cases, the EPIC research will indicate whether their
belief that it is viable or cost-effective seems justified.

16.1.3 Follow-on Applied R&D (Logic Model Box P)
For some technologies or tools, another indicator of progress toward future market
adoption is their continued development with additional research and development. We
asked both CEC administrators and grantees if they planned to conduct additional applied
R&D for their technology or tool based on the results of the project.
Four of the eight project teams indicated that they do plan to conduct additional research
for their technology or tool depending on final results. Based on the current stages of the
existing research and interim results of our sampled projects, the technologies and tools
that are being developed seem to be on track with the original research objectives,
suggesting that additional research and development may be worthwhile.
Anticipated Future Research and Development via EPIC

Of the teams that anticipate pursuing follow-on applied R&D, two of the four project
teams plan to seek additional funding from EPIC.
Anticipated Future Research via Other Funding

In addition (or in place of additional EPIC funding), some project teams anticipate future
research funded from other sources. One team hoping for future EPIC funding plans to
seek funding from the DOE as well. Another team indicated that they might take some of
their findings and bring them to their start-up company to continue their research, while
seeking additional funding from non-dilutive investors (i.e., financing that does not
require the sale of company shares, and hence does not cause dilution of the existing
shareholders).
Finally, one of the four project teams planning to conduct follow-on research stated that it
was too early in the project to provide any details on potential next steps.
16.1.4 Technology Demonstrations (Logic Model Box U)

According to the program logic model, technology demonstrations would follow the
completion of Applied R&D efforts. Given that all sampled Applied R&D projects were
still in progress, it is still too early to know the extent to which the current set of projects
will lead to technology demonstrations.
While our review of the on-going projects and project team assessment of their progress
suggests that the technologies and tools being developed seem to be on track with the
original research objectives, the CAMs have only general ideas of what they want to see
out of any potential technology demonstrations. Clear next steps are not apparent at this
time.

16.1.5 Technology Adoption (Logic Model Boxes Q, R, S, and T)
While the Applied R&D logic model indicates that the adoption of technologies and tools
is expected to follow project completion, it is generally too early to assess the likelihood of
eventual adoption. However, we note that one grantee is reportedly already offering the
technology being studied in its Applied R&D project in the marketplace and is in the
process of its first open market installation in California. That technology is a PV-oriented
innovation that has received both EPIC and DOE funding and is transitioning to
commercial availability even as the EPIC project is still being completed. The other seven
project teams all appeared to anticipate the eventual adoption of their respective
technologies.
16.2 Technology Demonstration and Deployment Projects

In this section, for the Technology Demonstration and Deployment (TD&D) projects
designed and implemented by the CEC and the three IOUs, we present results to date with
respect to the outputs and outcomes depicted in the TD&D logic model in Appendix B.
These results are based on the analysis of a random sample of 38 TD&D projects, stratified
by administrator (17 active CEC projects, 12 active IOU projects and 9 completed IOU
projects). For the CEC projects, we conducted in-depth interviews with the CAMs and
grantees. For the IOU projects, we conducted in-depth interviews with the project
managers and, when necessary, the vendors. These in-depth interviews allowed us to
address a broader range of issues in greater detail.
Below, we present the results for the various outputs and outcomes, as depicted in the
TD&D logic model, for the 38 sampled TD&D projects.113 In presenting these results, there
are two things to keep in mind. First, since only nine projects have been completed at the
time of data collection, many of the observations are based on the informed judgment of
project teams regarding the likely eventual success of their projects. Answers to questions
for active projects should therefore be considered early indicators of project performance.
Second, a sample size of 38 is a relatively large sample fraction, but when some questions
are contingent on earlier responses and then broken out by different variables such as by
administrator and/or by active versus complete, the sample sizes within each cell become
quite small, making any generalizations to the larger populations less tenable. In order to
connect the reader to the overarching framework provided by the TD&D logic model, we
have included, in parentheses next to each heading, the letters associated with the output
and outcome boxes in the TD&D logic model in Appendix B.
At the time the TD&D sample was drawn in October of 2016, there were only nine completed projects.
113
The number of completed projects has grown to 18 by the time this evaluation report was prepared in July of
2017.

Boxes F, H, N and S)
From the TD&D logic model, it is clear that the key output of any project is knowledge and
that this knowledge should be used in some way to advance innovative technologies
leading eventually to commercialization of new products or wider adoption of new
products in the public domain. Over time, it is the combination of all of these projects that
is expected to increase the probability of achieving the mid- and long-term EPIC outcomes.
Respondents associated with completed projects were of course more confident about any
knowledge they created and its usefulness, while respondents associated with active
projects could only assess their progress to date, which at least provides us with a leading
indicator of success.
Knowledge Creation
The most basic question is whether, for completed projects, any new knowledge had been
created. For both completed and active projects, there was a consensus that knowledge has
been created and that this knowledge has given them greater confidence in the
performance of their technologies and tools. One way TD&D projects can increase
adoptions is to install the technology at a site where potential adopters can visit the site,
see it in operation and assess its applicability to their own situation. In general,
respondents felt that visitors to TD&D sites were, as a result, more confident in the
performance of the technology and more likely to adopt it.
Project teams were also asked to assess how successful they have been in achieving their
technical performance targets. For the completed IOU projects, nearly all indicated that
they had been very successful. For the active CEC and IOU projects, nearly all indicated
that, thus far, they have been very or extremely successful in meeting their technical
performance targets.
Another question is whether, if completed, they have validated or, if still active, are likely
to validate their targeted benefits. To address this question, we reviewed benefits
questionnaires from 14 of 38 TD&D projects. Table 68 below summarizes the project
benefits targeted by the following general benefit areas:
• Potential Energy and Cost Savings – for example, CEC project 15-053 developed,
demonstrated and evaluated energy retrofit packages to help the CPUC reach
residential Zero Net Energy (ZNE) goals. Expected benefits of this project are
significant savings in energy, money, resources, and operation and maintenance
costs that will accrue to ratepayers. CEC project 14-031, which is utilizing gas
turbines to convert dirty, low heating value fuels into heat, also anticipates potential
energy and cost savings. Expected benefits of this project are significant savings in
yearly electricity costs, reuse of resources, and reductions of nitrogen oxide
emissions.

• Job Creation – CEC project 14-081 is demonstrating a software tool that links
ground water extraction with smart meter data to provide growers with automated
information on energy and water consumption. It is predicted that this project will
add approximately 28 jobs in the state of California, with those jobs lasting
anywhere between 15 and 50 years.
• Economic Benefits – for example, CEC project 14-085 demonstrates how a
combination of PV generation and an energy management system could reduce the
community’s average daily power and peak energy demand. Expected benefits of
this project are reductions in electricity charges while reducing carbon dioxide
emissions and other GHG emissions associated with power products. CEC project
15-042, which is developing and evaluating a cost-competitive ZNE community,
also anticipates economic benefits. Expected economic benefits coming from this
project involve the construction of 50 ZNE new homes in a disadvantaged
community, favorable media attention leading to an increase in awareness and
knowledge of ZNE homes and an increase in property values in the community.
• Environmental Benefits – CEC project 15-079 is demonstrating a pre-commercial
flow battery storage control system at a Regional Wastewater Treatment Plant
located in a disadvantaged community. Expected benefits of this project are a
significant decrease in grid power consumption and GHG emissions. CEC project
14-050 is developing, demonstrating and evaluating microgrids to assist emergency
response facilities by providing three hours a day of power for critical loads during
a utility power outage. This project is expected to produce environmental benefits
by decreasing air emissions relative to the currently used diesel back-up power.
• Safety, Power, Quality, and Reliability (Equipment, Electric System) – CEC
project 15-042 is developing and evaluating a cost-competitive ZNE community.
Expected benefits of the project will directly improve grid reliability by reducing
peak demand through efficiency and renewable generation in the 50 homes
constructed in the community. CEC project 14-060, which is demonstrating a utility-
owned microgrid in a community commonly impacted by severe thunderstorms
and rugged terrain, also expects electric system reliability benefits. It is expected
that the microgrid will allow quicker restoration of services to customers in order to
sustain critical infrastructure during grid outages.

Table 68: TD&D Project Benefits Tracked
# with # with Total
3 3 6 14
For the IOU completed projects, nearly all project teams indicated that they were
somewhat to very successful in validating these expected benefits, with only one project
team indicating that they have not been at all successful. For active IOU projects, project
teams indicated that they have, thus far, been very successful in validating their expected
benefits. For the CEC active projects, nearly half of the project teams said that they have
thus far been very or extremely successful in validating their benefits. For the remaining
half, project teams indicated that it was too early to judge. Only two CEC project teams felt
that they have been not at all or a little successful thus far. In general, completed projects
and active projects thus far have been successful with some CEC project teams indicating
that it is too early to tell. In general, projects have either validated their expected benefits
or are on track to verifying them.
Other more concrete indicators of knowledge creation are the production of progress
reports, final reports, databases, fact sheets, refereed journal articles, and trade journal
articles. Both the CEC and IOU closely monitor their projects on a monthly or quarterly
basis, with a few even getting weekly progress reports (discussed in more detail in Section
9). There is also a general consensus among the CAMs and IOU project managers that
these reports are very or extremely useful. Furthermore, none of the projects have been re-
scoped even with such careful oversight, which is one indication that the research plans
finalized at the beginning of each project were very well designed and are being faithfully
implemented. This use of these on-going reviews allows for mid-course corrections that
are essential for the success of EPIC. The use of such feedback is considered best practice.
In addition, five of the 15 CEC grantees indicated that thus far they had published articles
in refereed journals (four more are in progress), three had created fact sheets, and two had
published articles in trade journals, but no software tools or databases have been created.
Thus far, the IOUs have published two articles in refereed journals and three articles in
trade journals, and have created four fact sheets, three databases and three software tools.

We emphasize that it is premature to expect a large number of such outputs, and more are
expected as these TD&D projects evolve.
Results in the form of final reports, articles, fact sheets and databases are posted on the
CEC, CPUC and IOU websites, presented at workshops for interested parties, presented at
professional conferences, and delivered to a variety of organizations and individuals that
comprise their professional network. These dissemination-of-knowledge activities are
described in more detail in the network analysis in Section 9.
While it is important to create knowledge both during and at the conclusion of a project, it
is equally important for these results to be used. Knowledge gained during a project is
used to assess whether the project is on schedule and on budget and whether the
technology is working as expected. The adjustments are triggered by consultation from
TAC and PAC members, critical project reviews and discussions between the
administrators and the CEC grantees and IOU project managers. For IOU projects, this
assessment is conducted by the project managers. Two questions focused on the extent to
which project teams, as a result of knowledge gained thus far from their projects, have
deviated from their original research plans and, as a result, re-scoped their projects. For
the CEC, there were only a few minor deviations and thus no reason to re-scope any of
their projects. For the IOUs, only one experienced a serious deviation based on interim
feedback resulting in its cancellation. These results suggest that knowledge gained during
the implementation of projects is being considered by project teams, resulting in some
minor adjustments and only one case of project cancellation. Such on-going feedback is
essential for EPIC’s success..
Finally, for six of the nine completed IOU projects, respondents indicated that they are
aware of other organizations or individuals, apart from their EPIC project team, who are
planning to use the findings or outputs from this project. Such use by others is one
indicator of the value of the IOU research.
Development of Patents and Copyright Material

In the application of TD&D activities, project scoping is normally directed toward
products and materials that have already been developed and are ready for integration
within a viable demonstration platform. Therefore, it is generally accepted that any patent
or copyright activities in product development would have taken place previous to project
scoping. For thoroughness of analysis, the sampled project interviews with CEC grantees
included questions to determine if TD&D project activities had led to any further
developments of patents, licenses and copyrights. While no such activity was identified, it
should be noted that many of these projects were in mid-cycle development, and it was
determined that any such activity, if they were to occur, would likely take place near the
end of project completion and prior to technology dissemination activities.

16.2.2 Private Investment (Logic Model Boxes M and O)
Commercial Viability
As mentioned in Section 10, many technologies, absent public funding, fall into the CVD
since the private sector does not typically conduct applied research and is risk-averse
regarding new, unproven technologies, often lacking the resources to analyze and evaluate
various technologies. We asked respondents who were planning to commercialize their
technology whether, absent EPIC funding, they thought that this technology would have
fallen into the CVD. Of course, accelerating a promising technology to market is still an
important contribution. Of the relatively few IOU projects for which the goal was
commercialization, many indicated that it would have fallen into the CVD, but a few also
noted that, while their technology would not have fallen into the CVD, EPIC funding had
basically helped them to accelerate its development in a more comprehensive manner and
also allowed them to take on more of the risk in some of these projects than they typically
would. Again, accelerating a promising technology to market and sharing the risk are still
important contributions. Only two indicated that the technology would not have fallen
into the CVD. Finally, many volunteered that private funding was not available.
To successfully commercialize a product or, if not to commercialize it, ensure the broader
adoption of a product in the public domain, one must have an understanding of the extent
to which the potential buyers or users in the broader market understand the added value
of the product and, once the product is commercially available, how willing they might be
to purchase or use the product. While markets are dynamic and willingness to pay is
difficult to predict, having some understanding of these two issues is an indication that
one is at least tracking these two issues and, as a result, has a greater chance of succeeding
in the market. The vast majority of the IOU project teams showed some level of
understanding of their respective markets, comprised mostly of other utilities. Because all
of the CEC projects are still active, fewer of the CEC project teams were able to answer this
question, and more felt that it is too soon to tell. However, those who were able to answer
also displayed some level of understanding of their respective markets.
As noted earlier, as one of the conditions for funding, grantees had to justify their projects
based on market forecasts/penetration estimates, and justify why EPIC funds are needed,
i.e., what would happen to the project if EPIC funds were not provided. They also were
asked to identify the market barriers to adoption. As EPIC projects evolve, additional
market intelligence and support can be provided by the Market Facilitation program area.
However, there is a disconnect between the Market Facilitation projects and TD&D
projects (as well as the Applied R&D projects) that are upstream from them. The CAMs for
Market Facilitation projects rarely mentioned these upstream CAMs and the IOU project
managers as potential users of their research and, not surprisingly, the CAMs and project
managers of these upstream projects are mostly unaware of the commercialization
resources available through the Market Facilitation program area. This disconnect is
primarily due to the fact that all three program areas were launched more or less at the

same time, making it extremely difficult to develop projects that are aimed to meet the
needs of TD&D (and Applied R&D) projects. The CEC recognizes this gap and reports that
it is committed to working more closely with those responsible for the upstream projects
to better meet their needs. This disconnect and a recommendation to address it are
discussed in Section 11.
Another question focused on the extent to which project teams, as a result of knowledge
gained thus far from their projects, have deviated from their original strategy to
commercialize this technology/tool and from their original assessment of market interests.
Nearly all of the IOU project teams and more than half of the CEC project teams indicated
that, based on what they have learned thus far, they had not changed their original
strategy, with the remaining CEC project teams indicating that it was too early to tell.
Nearly all of the CEC and IOU projects indicated that they have also not deviated from
their original assessment of market interests. This might mean that their original
understanding of the path to commercialization and the needs of the various users in this
market was reasonably sound. Of course, it could also mean that they have not yet learned
enough to challenge their original thinking, something that is certainly possible given the
disconnect between the Market Facilitation projects and the Applied R&D and TD&D
projects that are upstream from them. Again, this disconnect and a recommendation to
address it are presented in Section 11.
Assistance in Commercialization
There are a number of resources available to assist in the commercialization of a
technology including firms that specialize in providing such support. We asked CEC
grantees whether they have hired any outside firms to assist in the commercialization of
their technology. Thus far, none of the respondents said that they have hired any firms.
Another question is whether the CAMs have assisted the grantees in finding such firms.
Of the nine respondents, only two said that they have provided any assistance while three
suggested that it was soon. Such low levels of use of outside firms could indicate that they
do not see the need for such firms or they do not understand fully the services offered by
such firms.
Another potential source of assistance is the EPIC Market Facilitation program area in
which the CEC proposes funding initiatives to help overcome non-technical barriers to
accelerate the commercial viability of high-priority technologies and strategies in IOU
service territories. Project teams were asked how helpful the Market Facilitation program
area was in accelerating the commercial viability of their technology/tool. While this is
less of an issue for the IOUs due to their types of projects, those relatively few projects for
which Market Facilitation could be potentially helpful have had no contact with the
Market Facilitation program area. For the CEC grantees, none have had any contact with
this part of the EPIC portfolio. This seems in large part due to the fact that some projects
have not progressed to the point where such services could be effective and a general lack
of awareness of the Market Facilitation services.

16.2.3 Development of Partnerships and Investors (Logic Model Box O)
Business Development as a Result of Project Activity

As project grantees are selected and project implementation activities progress, it is
understood that a key benefit of these actions will be to increase the visibility of these
technologies and their parent organizations to potential partners and investors. The
underlying theory is that this increased visibility will increase the likelihood of the parent
organization receiving additional resources in the form of partnership developments,
third-party investors or even company acquisitions. These additional resources will
increase the chances that the technology will not fall into the CVD.
Of the active CEC TD&D projects with the intent to commercialize, very few have thus far
met with interested parties to form partnerships, and only one has added a partner. For
those that have not as yet met with any potential new partners, they often mentioned that
before engaging with interested partners or investors, they needed to first identify fuller
product configurations for the project application. In addition, some grantees were public
institutions or non-profit agencies that are not likely open to such external resource
activity. Furthermore, none of the grantees reported that they had discussed possible
acquisition of their companies with any manufacturers or venture capital firms. This is
likely due, at least in part, to the fact that all of the CEC projects are still active with the
final outcome of the projects yet to be determined. While not yet engaged with the Energy
Innovation Ecosystem that includes the four Innovation Clusters and the California
Sustainable Energy Entrepreneur Development Initiative (CalSEED), TD&D projects (and
Applied R&D projects) can clearly benefit from these resources. A recommendation is
made in Section 11 to increase the use of these resources.
Increased Technology Production (Logic Model Boxes O & T)

As illustrated in the TD&D logic model, the final project output leading toward short-term
outcomes includes increasing awareness and knowledge among the targeted populations
regarding the technologies in real-life situations. Subsequently, the intention of this
increased awareness is to help increase the number of producers and manufacturers in the
market for the demonstrated technologies as a short-term outcome of EPIC. To evaluate
the overall increase in technology production, we asked TD&D project teams about the
scalability of their EPIC technologies, the cost-effectiveness of their EPIC technologies at
commercial scale, and their knowledge of new producers or manufacturers of their
technologies that had entered the market since the launch of their EPIC project.
Technology/Output Scalability in Commercial Applications

The scalability of the EPIC-produced technologies and tools is one metric used to help
evaluate the effectiveness of increasing awareness and knowledge leading to increases in
private investments and leading eventually to increased levels of production. For the
sampled 22 TD&D projects that planned to commercialize their technology and were far
enough along in the program process to evaluate the potential scalability of their

technology outputs, all but one project team said their technology was either “very” or
“completely” scalable.
One of the primary reasons project teams said their EPIC technologies were scalable was
because they were applicable to a variety of commercial applications, were beneficial to
different types of communities (including disadvantaged communities), were software
applications that could be installed universally, and were modular to the extent that they
could be scaled depending on the size of the commercial application. For example, one
CEC-administered project focused on demonstrating biogas technologies that can be
installed individually for smaller applications or in parallel with multiple components for
larger landfills. Similar rationales were provided for TD&D projects focused
on compressed air monitoring systems and wastewater treatment facilities.
We also asked CEC grantees whether, since the launch of their projects, they had attracted
public and/or private investors. Thus far, very few grantees had attracted any public
and/or private investment beyond EPIC funding. Again, this is, at least in part, due to the
fact that all of the CEC projects are still active. Another possible explanation is that before
they received EPIC funding, they might already have received funding at the earlier proof-
of-concept stage of their technology.
Cost-effectiveness at Commercial Scale

Additionally, cost-effectiveness of the EPIC-supported technologies and tools when
manufactured at commercial scale is another metric used to evaluate a project’s push
toward increased levels of production and increased number of producers. Overall,
slightly less than half of the sampled TD&D projects (n=17) were either not far enough
along in their project or did not have plans to commercialize, and subsequently did not
attempt to evaluate the potential cost-effectiveness of their technologies and tools.
However, among the TD&D projects that were far enough along in their project process
and planned to commercialize, 81 percent across all administrators said their technologies
were either “likely” or “extremely likely” to be cost-effective when manufactured at or
near commercial scale, while only one project team acknowledged it was “unlikely” their
technology would be cost-effective.
One of the primary reasons that multiple project teams said they were optimistic their
technologies and tools would be cost-effective is because they anticipate production costs
for their technology will continue to decrease as production increases and the technologies
become more readily available. For example, one TD&D project team anticipates the
return on investment for their technology to drop as the price for their technology
continues to decrease. Additionally, project teams noted that being able to demonstrate
their technologies in demonstration or test sites further advances the cost-effectiveness of
their technologies because it indicates to other potential applications and manufacturers
that the technology can be effectively administered and produces the desired benefits that
warrant the required project costs.

New Producers of Demonstrated EPIC Technologies
In the short-term outcomes section of the TD&D logic model, increased awareness
regarding the real-life application of EPIC technologies is expected to result in an increase
in the number of producers of the demonstrated technologies. The increase in producers
may help increase the availability and cost-effectiveness of the EPIC technologies and
further promote renewable technologies within the commercial market.
As outlined above, given that several sampled TD&D projects—especially IOU-

administered projects—either did not have current plans to commercialize or were still in
the early stages of their project, only 58 percent (n=22) of project teams were asked about
the number of new producers for their technologies that had entered the market since the
launch of their project. As shown in Table 69, among the project teams that did respond, 11
indicated there were no new producers of their technologies or tools, while six said there
were new producers and five said they did not know.
Table 69: New Manufacturers Entering Market

Since the launch of this project, have
new manufacturers of this
technology/tool entered the market? CEC PG&E SCE SDG&E Total
Yes 4 2 0 0 6
No 6 2 1 2 11
Don’t know 4 0 1 0 5
Total 14 4 2 2 22
Follow-on TD&D (Logic Model Box V)

One possible outcome depicted in the TD&D logic model is follow-on TD&D research. The
CEC and IOU project teams were asked whether they plan to conduct additional TD&D
research for this technology/tool based on the results of their projects. Well over half of
the CEC project teams indicated that they plan to conduct additional TD&D research
based on the results of their projects thus far. For the IOU completed projects, nearly all
respondents indicated that they do not intend to conduct additional TD&D research. Since
the project teams for IOU active projects were not asked this question, it is difficult to
reliably compare the results. Nevertheless, at least some of these differences might be due
to the fact that the IOU projects are more narrowly defined for a very specific application
and therefore require less additional research.
Technology Adoption (Logic Model Boxes P, Q and R)

The adoption of a technology by industry stakeholders is a primary indicator of program
efficacy. This restricted our analysis of adoptions to the only completed TD&D projects in
our sample, the IOU projects. However, this does produce some difficulty in the overall

portfolio segmentation analysis, since the IOU TD&D projects tend to be targeted for
adoption within the specific activity of electric grid management and operations. In
addition, since the identification of future changes to grid-system operations are cycled
annually and through regulatory review, any future adoptions of these technologies or
processes would not be expected to occur until sometime in the next planning cycle.
General results from interviewed administrators indicate that these completed projects
had produced favorable results and that the further integration of these demonstrated
technologies and processes would occur—primarily by the administrator of the project.
Even though the adoptions were primarily by the administrator of the project, the IOU
project managers also noted that utilities inside and outside California could also benefit
from adopting these technologies.
16.3 Market Facilitation Projects

16.3.1 Background and Overview
In this section, for the Market Facilitation projects designed and implemented by the CEC,
we present results to date with respect to key outputs and outcomes depicted in the
Market Facilitation logic model in Appendix B. These results are based on a random
sample of eight active CEC Market Facilitation projects. For each sampled project, we
conducted in-depth interviews with the CAM and grantee. These in-depth interviews
allowed us to address a broader range of issues in greater detail. The evaluation team
asked interviewees about issues related to funding, knowledge creation, knowledge
utilization, project scalability, and follow-on research. This section summarizes findings
related to these topics.
In order to maintain respondents’ confidentiality, due to the small number of projects in

the sample and even smaller numbers by the logic model activity areas, this analysis
aggregates findings from all projects in the sample and does not discuss findings within
the main activity areas identified in the logic model. Additionally, although the sample
included eight projects, not all questions were asked of every CAM or grantee. In most
cases, those questions were not relevant to the specific project or the project had not been
active long enough to allow the respondent to provide a response. In some instances, the
question may not have been asked due to time constraints. As a result, some questions
have results from fewer than the eight respondents in the sample.
In order to connect the reader to the overarching framework provided by the Market
Facilitation logic model, we have included, in parentheses next to each heading, the letters
associated with the output and outcome boxes in the Market Facilitation logic model in
Appendix B.

16.3.2 Funding (Logic Model Box C)
In general, grantees reported that they were satisfied with the level of funding for their
project. When asked about the likelihood of whether their project would have been
conducted absent EPIC funding, four grantees said that it was “unlikely” while three said
that it was “extremely likely.” Of course, continued funding is always an issue. Since the
launch of their project, four grantees reported that they have sought additional funding.
Three said that they have pursued funding from federal agencies, one from private
investors, and one from a collaboration between the CEC and IOUs.

Boxes E, H, K, N, F, I, L and O)
From the Market Facilitation logic model, it is clear that the key output of any Market
Facilitation project is knowledge and that this knowledge should be used in some way to
advance innovative technologies, leading eventually to the achievement of mid- and long-
term outcomes. Of course, since all Market Facilitation projects are still active, respondents
could only assess their progress to date, which at least provides us with leading indicators
of knowledge creation and utilization.
Knowledge Creation
The most basic question is whether, for completed projects, any new knowledge had been
created. Since all of these Market Facilitation projects are still active, we asked the CAMs
how successful they have been, thus far, in achieving their project/research objectives.
Responses were evenly distributed from somewhat successful to extremely successful.
When asked to explain their answers, most indicated that they had achieved all of the
intermediate objectives thus far. A related question is how successful they have been, thus
far, in validating these benefits of their project that might eventually lead to longer-term
benefits once the findings/information/products are more broadly deployed. Four
responded that thus far, they have been somewhat successful, while one indicated that
they have been extremely successful. One indicated that it was simply too early to tell. In
general, projects are on track to achieve their research objectives and their expected
benefits.114
We also reviewed benefit questionnaires from four of seven Market Facilitation projects.
Table 70 below summarizes the project benefits targeted by the following general benefit
areas:
• Potential Energy and Cost Savings – for example, CEC project 14-037 is an applied,
data-driven study to understand the role and interactions of various factors such as
income, ethnicity, language and political orientation to understand the adoption
114 See Section 9 for a more detailed discussion of the results of our analysis of the benefits questionnaires.

and utilization of residential energy efficiency measures. Expected energy and cost
savings benefits for this project include a reduction in electricity use when results
are used to design energy efficiency programs, which, based on research findings,
will lead to a more effective and efficient program delivery. CEC project 15-010 will
seek to provide workers from disadvantaged communities with comprehensive
classroom and on-the-job training in the installation and maintenance of AutoDR
communication equipment. Expected energy and cost savings benefits for this
project are reduced electricity costs for building owners, based on the size of
buildings and energy consumption.
• Job Creation – for example, CEC project 15-010 will seek to provide workers from
disadvantaged communities with comprehensive classroom and on-the-job training
on the installation and maintenance of AutoDR communications equipment.
Expected benefits for this project are to increase the number of trained workers that
will fill jobs installing and maintaining AutoDR, specifically in disadvantaged
communities.
• Economic Benefits – CEC project 14-037 is an applied, data-driven study that will
serve to understand the role of various economic and societal factors in order to
provide insight into the trends of adoption and utilization of residential energy
efficiency measures. Expected benefits of this project will help energy efficiency
stakeholders to better understand trends of sociocultural groups, which will help
increase participation in energy efficiency measures, providing energy cost savings
and potential job creation. CEC project 15-010 will develop and train workers from
disadvantaged communities and provide them with the necessary skills to install
and maintain AutoDR. There are many expected economic benefits that will come
from this project, including more money coming in to disadvantaged communities
through skilled labor that will go back into businesses in those communities. Also,
improvement in system operation efficiencies will come from the increased utility
workforce, providing quicker dispatch of customer demand side management.
• Environmental Benefits - CEC project 15-010 will develop the skills of workers
from disadvantaged communities by providing comprehensive training on the
installation and maintenance of AutoDR communication equipment. Among many
other benefits, expected environmental benefits from this project are a decrease in
emissions; AutoDR will shift energy use from Peak periods to Off Peak periods and
reduce overall building energy use. CEC project 14-037 also predicts environmental
benefits including reductions in GHG emissions due to a decrease in residential
energy use.
• Safety, Power, Quality, and Reliability (Equipment, Electric System) – for
example, CEC project 14-037 is expected to lead to greater reliability in the electric
system and equipment. Expected benefits of this project are an improvement of the
accuracy of market adoption forecasts of the replacement of “generation-based”
resources with efficiency resources. CEC project 15-010, which is training workers

in disadvantaged communities to maintain and install AutoDR, also predicts
improved grid reliability benefits due to an increase in demand response resources
Table 70: Market Facilitation Project Benefits Tracked (n=4 number of Market
Facilitation projects with benefits questionnaire)
# with # with Total
0 2 2 4
Other more concrete indicators of knowledge creation are the publication of progress
reports, final reports, databases, fact sheets, refereed journal articles, and trade journal
articles. The CAMS closely monitor their projects on a monthly basis, and there is a general
consensus among the CAMs that these reports are extremely useful. Such on-going
reviews allowing for mid-course corrections are essential for EPIC’s success and are
considered best practice.
In addition, none of the projects have been re-scoped even with such careful oversight,
which is one indication that the research plans finalized at the beginning of each project
were very well designed. While all of these projects are still active, one article has been
submitted to a refereed journal. In addition, four of the seven respondents indicated that
they have published articles in trade journals and support a blog about their projects. As
projects evolve, we expect their numbers to increase.
In addition, the CAMs indicated that interim and final results in the form of final reports,
articles and fact sheets will also be shared through a variety of channels including the
CEC's various websites, workshops and presentations given at professional conferences.
These results will also be delivered to a variety of organizations and individuals that
comprise their professional network. These dissemination-of-knowledge activities are
described in more detail in the network analysis in Section 9.
When asked about the audiences they targeted for their findings, the CAMs most
commonly mentioned the projects’ TAC, members of other relevant project TACs, and the
Emerging Technology Coordinating Council. Interestingly, interviewees were not as likely
to cite the CEC Applied R&D grantees or TD&D grantees as primary audiences. Grantees

were even less likely than CAMs to do so. This result might be due to the fact that all three
program areas were launched more or less in parallel with each other. As a result, the
needs of those conducting Applied R&D and TD&D projects could not be carefully
reviewed prior to launching Market Facilitation projects. In the future, a more systematic
assessment of the needs of those conducting projects that are upstream from the Market
Facilitation projects should be considered.
While it is important to create knowledge both during and after the completion of a project,
it is equally important for these results to be used. For projects that are still active, CAMs
use the knowledge gained thus far to make any necessary mid-course corrections. One
question focused on the extent to which project teams, as a result of knowledge gained
thus far from their project, have deviated from their original CEC-approved research plan.
All of the project teams reported that they had not deviated, although two indicated that
their schedules had been somewhat modified due to unforeseen events. In addition, none
of the grantees indicated that they had changed the scope of work for their projects based
on feedback from the CAMs or the TACs. This suggests that their research plans were well
designed and are being faithfully implemented.
While only one regional Innovation Cluster was in our sample (EPC 15-038), given their
size and potential for knowledge creation and utilization, all four commercialization
projects are worth mentioning. These four regional Innovation Clusters are designed to
achieve a key strategic objective, S18, which is to foster the development of the most
promising energy technologies into successful businesses. Innovation Clusters are defined
as geographic concentrations of interconnected companies, specialized suppliers, service
providers, firms in related industries, and associated institutions in particular fields
(Tinguely 2013). Clusters emerge when a network of companies coexists within a
geographic location, allowing each of them to collaborate—and compete—in a way
that delivers greater productivity gains than they would achieve in isolation. The four
Clusters that have been funded (EPC-16-015, EPC-15-030, EPC-15-032 and EPC-15-038)
and their budgets are listed in Table 71.
Table 71: Innovation Clusters and Funding

Innovation Clusters EPIC Funding Match Funding Total
Bay Area Regional Energy Innovation Cluster $ 4,980,000 $ 9,000,000 $13,980,000
Central Valley Energy Innovation Cluster: BlueTechValley Energy Cluster $ 5,000,000 $ 2,655,684 $ 7,655,684
San Diego Regional Energy Innovation Cluster $ 5,000,000 $ 3,097,934 $ 8,097,934
Los Angeles Regional Energy Innovation Cluster $ 4,999,247 $ 3,658,099 $ 8,657,346
Total $ 19,979,247 $ 18,411,717 $38,390,964

The EPIC funding combined with the match funding make these regional Innovation
Clusters the largest EPIC Market Facilitation project by far. In general, these four Clusters
are designed to help entrepreneurs avoid the TVD and the CVD by facilitating a network
of stakeholders to provide commercialization assistance and services to clean energy
entrepreneurs and start-up companies. These networks will exchange ideas and best
practices, information on promising technologies, and insights into specific market
opportunities and customer needs. Also, this network will provide opportunities for
entrepreneurs to engage with industry and investor stakeholders and receive market
feedback and validation, as well as services to match entrepreneurs with customers in IOU
service territories and guidance on incubators for product testing.
Each cluster has three basic activities: 1) engage regional hubs, 2) provide business and
technology serves, and 3) assess energy research. In Table 72, for each Innovation Cluster,
we present the number of regional hubs that are being engaged, the number of firms
providing business and technology services, and the number of energy research
organizations being engaged.
Table 72: Innovation Clusters, by Key Activities

Energy
Hubs Business & Research
Innovation Clusters Engaged Tech Firms Organizations
Bay Area Regional Energy Innovation Cluster 2 2 4
Central Valley Energy Innovation Cluster: BlueTechValley Energy Cluster 2 2 8
Los Angeles Regional Energy Innovation Cluster 4 5 6
San Diego Regional Energy Innovation Cluster 3 3 3
Total 11 12 21
Not only are small entrepreneurs across California eligible to participate, but both active
and completed EPIC projects are also eligible.
These Innovation Clusters will be evaluated over time on a number of performance

metrics including job creation and economic development in California, future energy and
environmental savings, dollar value of Ecosystem115 resources, increased market size and
potential as a result of using Ecosystem resources, improved resource effectiveness
through use of market-specific and region-specific market research, number of referrals
made from one Cluster to another to reward collaboration, level of customer awareness of
emerging technology options in Cluster regions, and improved diversity in
115The Energy Innovation Ecosystem includes the four Innovation Clusters and the California Sustainable
Energy Entrepreneur Development Initiative (CalSEED), which includes 1) at least $24 million in small
grants for entrepreneurs, 2) at least $44 million set aside for underrepresented groups, and 3) support for
early development of promising new energy concepts.

entrepreneurship.116 In addition to the CEC annual reports, the CPUC should also receive
any interim progress reports.
16.3.4 Project Scalability (Replicability) (Logic Model Boxes F, I, L and O)

The team asked each project grantee how scalable117 they thought their project was, using
a scale of “not at all” to “completely.” Of the six grantees for whom this question was
relevant, four said that their project’s approach was “completely” scalable. Of the
remaining grantees, one said that the project was “very” and other said that it was “a
little” scalable. One grantee’s interpretation of “scalable” was that it could be replicated
anywhere in the world. Another said that it was scalable as long as there was funding.
16.3.5 Follow-On Research (Logic Model Boxes F, I, L and O)

One possible outcome depicted in the Market Facilitation logic model is follow-on
research. The CEC project teams were asked whether they plan to conduct additional
studies or research based on the results of their projects. Although all eight of the Market
Facilitation projects are still active, two grantees responded that they would conduct
additional studies. One grantee pointed out that since their project is very broad, it could
apply to any of a number of areas such as community design, alternative vehicle
technology, fuel cell technology, battery energy storage or PV system design. This grantee
went on to note that they could conduct additional research in any one of these areas in
the future.
116 Navigant Consulting is currently under contract to the CEC to develop and finalize a written report
detailing a common methodology that can be used to evaluate the benefits accomplished by the four
Regional Energy Innovation Clusters.
117 Scalability, in the traditional sense, is the capability of a system, network, or process to handle a growing
amount of work, or its potential to be enlarged in order to accommodate that growth. In an economic
context, the scalability of a company implies that the underlying business model offers the potential for
economic growth within the company. Scalability for Market Facilitation projects can also include the ability
of a research project to be replicated in other states and jurisdictions.

17 Appendix F: Selected References Regarding Theory-
Driven Evaluation
For detailed descriptions of theory-driven evaluation, see the following references:
Chen, H.T. Theory-Driven Evaluations. Thousand Oaks, CA: Sage, 1990.

Coryn, C.L., L.A. Noakes, C.D. Westine, and D.C. Schröter. “A Systematic Review of
Theory-Driven Evaluation Practice from 1990 to 2009," American Journal of Evaluation,
32(2) (2011).
Rogers, P.J. “Program Theory Evaluation: Not whether programs work but how they
work.” In: D.L. Stufflebeam, G.F. Madaus, and T. Kelleghan (Eds.), Evaluation Models:
Viewpoints on Educational and Human Services Evaluation (pp. 209-232). Boston, MA:
Kluwer, 2000.
Rogers, P.J.“Using Program Theory to Evaluate Complicated and Complex Aspects
of Interventions.” Evaluation 14 (2008): 29-48.
Rogers, P.J., A. Petrosino, T.A. Huebner, and T.A. Hacsi. “Program Theory Evaluation:
Practice, Promise, and Problems.” In: P.J. Rogers, T.A. Hacsi, A. Petrosino, & T.A.
Huebner (Eds.), Program Theory in Evaluation: Challenges and Opportunities (pp. 5–14).
New Directions for Evaluation, No. 87. San Francisco, CA: Jossey-Bass, 2000.
Ruegg, Rosalie and Irwin Feller, A Toolkit for Evaluating Public R&D Investment: Models,
Methods, and Findings from ATP’s First Decade. Prepared for the Economic Assessment
Office, Advanced Technology Program, National Institute of Standards and
Technology, Gaithersburg, MD, 2003.
Weiss, C.H. “Nothing as Practical as Good Theory: Exploring Theory-based Evaluation for
Comprehensive Community Initiatives for Children and Families.” In: J. Connell, A.
Kubisch, L.B. Schorr, & C.H. Weiss (Eds.), New Approaches to Evaluating Community
Initiatives: Volume 1, Concepts, Methods, and Contexts (pp. 65-92). New York, NY: Aspen
Institute, 1995.
Weiss, C.H. 1997a. “How Can Theory-based Evaluations Make Greater Headway?”
Evaluation Review, 21: 501-524.
Weiss, C. H. 1997b. “Theory-based Evaluation: Past, Present and Future.” In: D.J. Rog & D.
Fournier (Eds.), Progress and Future Directions in Evaluation: Perspectives on Theory,
Practice and Methods (pp. 55-41). New Directions for Evaluation, No. 76. San Francisco,
CA: Jossey-Bass.
Weiss, C.H. On Theory-based Evaluation: Winning Friends and Influencing People. The
Evaluation Exchange, IX, 1-5, 2004.

EPIC Final Report 090817 Compressed

Uploaded by

Copyright:

Available Formats

EPIC Final Report 090817 Compressed

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

EPIC Final Report 090817 Compressed

Uploaded by

Copyright:

Available Formats

Electric Program Investment

Charge (EPIC) Evaluation

Evergreen Economics Page i

Evergreen Economics Page ii

Evergreen Economics Page iii

Evergreen Economics Page iv

Evergreen Economics Page v

Evergreen Economics Page vi

Evergreen Economics Page vii

Evergreen Economics Page viii

Evergreen Economics Page ix

Evergreen Economics Page x

Table 1: EPIC Program Budget (2012–2017)

1.1 Evaluation Objectives and Approach

Evergreen Economics Page 1-1

1. Electricity IOU Ratepayer Benefits: Providing electric The evaluation

2 D.11-12-035, D.12-05-037, D.13-11-025, D.15-04-025, and D.15-09-005.

Evergreen Economics Page 1-2

1.2 Key Findings and Recommendations

1.2.1 Program Requirements

Based on our review of program administrative procedures

1.2.2 EPIC Project Impacts

Evergreen Economics Page 1-3

Overall, the EPIC portfolio appears to be on track, thus far, in

Evergreen Economics Page 1-4

1.2.4 Effectively Engaging Stakeholders

Evergreen Economics Page 1-5

• 4b) The administrators collaborate and jointly convene a quarterly workshop to

Evergreen Economics Page 1-6

1.2.5 Independent Coordination Body

• 6a) The CPUC and/or the administrators fund and

Evergreen Economics Page 1-7

1.2.6 IOUs in Compliance, but Not Meeting Intent of All Requirements

To verify compliance with these requirements, we relied on a combination of sources,

Evergreen Economics Page 1-8

Evergreen Economics Page 1-9

Evergreen Economics Page 1-10

In order to improve the transparency and comprehensiveness of IOU investment planning

Evergreen Economics Page 1-11

We offer additional recommendations in Section 11 that focus on program administration

Evergreen Economics Page 1-12

• Providing societal benefits;

Evergreen Economics Page 2-1

Program funding is approved in the following defined investment areas: Applied

• Applied Research and Development (Applied R&D) – Applied R&D activities

• Grid operations/market design

Evergreen Economics Page 2-2

2.2.1 Greenhouse Gas Reduction Legislation and the Renewable Portfolio

SB 350 increases California’s renewable electricity procurement goal from 33 percent by

4 De León, Chapter 547, Statutes of 2015.

Evergreen Economics Page 2-3

7 Padilla. SB-17 Electricity: smart grid systems, Chapter 327. 2009.

California Energy Commission. 2015. http://www.energy.ca.gov/2015publications/CEC-500-2015-

Evergreen Economics Page 2-4

2.2.3 Energy Efficiency and the Loading Order

2.2.4 Distributed Energy Resources

13 California Code, Public Utilities Code 8360.

Action Plan. 2003. http://www.energy.ca.gov/energy_action_plan/index.html

Evergreen Economics Page 2-5