Energy Comsumtion Ashare

This file is licensed to sergio torres morales (retos@correoinfinitum.com).
License Date: 9-26-2012

©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. (www.ashrae.org).
For personal use only. Additional reproduction, distribution, or transmission in either print or
digital form is not permitted without ASHRAE's prior written permission.
This file is licensed to sergio torres morales (retos@correoinfinitum.com). License Date: 9-26-2012
Any updates/errata to this publication will be posted on the

ASHRAE Web site at www.ashrae.org/publicationupdates.
ISBN 978-1-933742-73-1
©2009 American Society of Heating, Refrigerating

and Air-Conditioning Engineers, Inc.
1791 Tullie Circle, NE
Atlanta, GA 30329
www.ashrae.org
All rights reserved.

Printed in the United States of America
Cover design by Joe Lombardo, DLB Associates.
ASHRAE has compiled this publication with care, but ASHRAE has not investigated,
and ASHRAE expressly disclaims any duty to investigate, any product, service, process,
procedure, design, or the like that may be described herein. The appearance of any
technical data or editorial material in this publication does not constitute endorsement,
warranty, or guaranty by ASHRAE of any product, service, process, procedure, design, or
the like. ASHRAE does not warrant that the information in the publication is free of
errors, and ASHRAE does not necessarily agree with any statement or opinion in this
publication. The entire risk of the use of any information in this publication is assumed
by the user.
No part of this book may be reproduced without permission in writing from ASHRAE,
except by a reviewer who may quote brief passages or reproduce illustrations in a review
with appropriate credit; nor may any part of this book be reproduced, stored in a retrieval
system, or transmitted in any way or by any means—electronic, photocopying, recording,
or other—without permission in writing from ASHRAE. Requests for permission should
be submitted at www.ashrae.org/permissions.
____________________________________________
Acknowledgments.......................................................................... ix
PART 1 BASICS
CHAPTER 1 INTRODUCTION................................................ 3
1.1 Objectives for this Book ............................................................. 11
1.2 How to Use this Book ................................................................ 12
CHAPTER 2 HOW, WHAT, & WHERE TO MEASURE ... 15

2.1 Overview ..................................................................................... 15
2.2 Quantifying Energy Efficiency Metrics .................................. 17
CHAPTER 3 MEASUREMENT DEVICES ........................... 21

3.1 Overview ..................................................................................... 21
3.2 Sensor Accuracy ......................................................................... 23
3.3 Temperature ................................................................................ 24
3.4 Pressure ........................................................................................ 29
3.5 Flow—Liquid .............................................................................. 31
3.6 Flow—Gas ................................................................................... 40
3.7 Current ......................................................................................... 44
3.8 Voltage ......................................................................................... 49
3.9 Power ........................................................................................... 53
CHAPTER 4 MEASUREMENT COLLECTION

SYSTEMS—ARCHITECTURE & SOFTWARE ........... 59
4.1 Overview ..................................................................................... 59
4.2 Business Questions..................................................................... 60
4.3 Scalable Hardware/Software Architecture .............................. 63
4.4 Measurement Levels .................................................................. 64
Real-Time Energy Consumption Measurements in Data Centers
PART 2 COOLING SYSTEMS—AIR MEASUREMENTS
CHAPTER 5 AIR HANDLERS ............................................... 79

5.1 Overview ..................................................................................... 79
CHAPTER 6 COMPUTER ROOM UNITS ........................... 85

6.1 Overview ..................................................................................... 85
PART 3 COOLING SYSTEMS

—HYDRONIC MEASUREMENTS
CHAPTER 7 PUMPS .............................................................. 101

7.1 Overview ................................................................................... 101
7.2 Measurement Levels—Electrical............................................ 101
7.3 Measurement Levels—Fluid ................................................... 105
CHAPTER 8 COOLING TOWERS...................................... 113

8.1 Overview ................................................................................... 113
8.2 Measurement Levels ................................................................ 116
CHAPTER 9 CHILLERS ....................................................... 125

9.1 Overview ................................................................................... 125
CHAPTER 10 HEAT EXCHANGERS ................................... 141

10.1 Overview ................................................................................... 141
PART 4 POWER SYSTEMS MEASUREMENTS
CHAPTER 11 INTRODUCTION TO
CRITICAL POWER DISTRIBUTION ......................... 149
11.1 Overview ................................................................................... 149
11.2 Critical Power versus Essential Power .................................. 150
vi
Contents
CHAPTER 12 UPSTREAM CRITICAL

POWER DISTRIBUTION .............................................. 155
12.1 Overview ................................................................................... 155
12.2 Service Entrance Equipment ................................................... 155
12.3 Automatic Transfer Switch (ATS) ......................................... 159
12.4 Primary Electrical Distribution Switchgear .......................... 161
CHAPTER 13 UNINTERRUPTIBLE
POWER SUPPLY (UPS) ................................................. 165
13.1 Overview ................................................................................... 165
13.2 UPS Metering, Power Module Level ..................................... 168
13.3 UPS Metering, System Level.................................................. 169
CHAPTER 14 COMPUTER ROOM TRANSFORMER

& POWER DISTRIBUTION UNIT (PDU) ................... 173
14.1 Overview ................................................................................... 173
14.2 Stand-Alone Transformers ...................................................... 173
14.3 Computer Room Power Distribution Units (PDU) .............. 175
14.4 Rack-Mounted Power Distribution Unit (RPDU) ................ 179
PART 5 IT SYSTEMS MEASUREMENTS
CHAPTER 15 COMPUTE & STORAGE SYSTEMS........... 185

15.1 Overview ................................................................................... 185
CHAPTER 16 NETWORKING SYSTEMS ........................... 201

16.1 Overview ................................................................................... 201
APPENDIX A PUMPS .................................................................... 213

A.1 Power and Efficiency ......................................................... 213
A.2 Real-Time Power Measurements ....................................... 216
vii
APPENDIX B CHILLERS ............................................................ 223

B.1 Variables Affecting RLA and Power Rating ......................... 223
B.2 Integrated or Non-Standard Part Load Value ..................... 224
APPENDIX C MIXED-USE FACILITIES................................ 227

C.1 Real-Time Cooling Tower Power Consumption ................ 227
C.2 Real-Time Chiller Power Consumption ............................. 229
APPENDIX D UNINTERRUPTIBLE
POWER SUPPLY (UPS) ...................................................... 233
D.1 Technology ......................................................................... 234
D.2 Redundancy and Availability ............................................. 236
D.3 Rules of Thumb for Minimum Practical Level
of UPS Instrumentation ...................................................... 241
D.4 Sample Case Study: A Partial PUE and DCiE Determination
for the Critical Power Path within the Data Center ............ 243
APPENDIX E ONSITE POWER GENERATION

AND CCHP IN DATA CENTER APPLICATIONS ..... 249
E.1 Overview ............................................................................ 249
E.2 CCHP.................................................................................. 251
E.3 Measurement Levels........................................................... 258
E.4 Example Calculations for a CCHP Installation ..................... 263
APPENDIX F ABBREVIATIONS AND GLOSSARY ........... 267
REFERENCES ................................................................................... 283
viii
ASHRAE TC 9.9 and The Green Grid would like to thank the
following individuals for their substantial contributions to the
book:
Lead editor / author – Tahir Cader, HP (formerly SprayCool)

Co-editor after First Draft – Don Beaty, DLB Associates
Chapter 1 – Tahir Cadera,b, HP (lead)

Chapter 2 – Tahir Cadera,b, HP (lead); Mike Mangan, DLB
Associates; Jeff Jaworksi, DLB Associates
Chapter 3 – John Beana,b, APC/Schneider; Randall Woffordb, Dell;
Ross Ignalla, Dranetz-BMI; Michael Kennedy, DLB Associates
Chapter 4 – Ken Uhlmanb, Eaton (lead); Harry Rogersb, Microsoft
Chapter 5 – Robert Wasilewski, DLB Associates (lead)
Chapter 6 – Jeff Trowera, DataAire (lead); Cliff Federspiel,
Federspiel Controls
Chapter 7 – John Beana,b, APC/Schneider (lead)
Chapter 8 – Daryn Clinea, Evapco (lead)
Chapter 9 – Jonathan Spreemana, Trane (lead); Tahir Cadera,b, HP
Chapter 10 – Robert Wasilewski, DLB Associates
Chapters 11, 12, 13, 14 – Steve McCluera,b, APC/Schneider
Electric (lead); Bill Campbellb, Emerson Network Power; John
Messerb, Emerson Network Power
Chapter 15 – Mike Pattersona,b, Intel (lead); Bob MacArthurb,
EMC
Chapter 16 – Kevin Engelberta,b, Cisco (lead)
Chapter 17 – Kevin Wymana, Carrier Corporation (lead); Greg
Palmerb, HP (formerly UTC Power) (lead);
Appendix A – John Beana,b, APC/Schneider (lead)

Appendix B – Jonathan Spreemana, Trane (lead)
Appendix C – Tahir Cadera,b, HP (lead); Jonathan Spreemana,
Trane
ix
Appendix D – Steve McCluera,b, APC/Schneider (lead); Bill

Campbella, Emerson Network Power; John Messerb, Emerson
Network Power
Appendix E – Kevin Wymana, Carrier Corporation; Greg Palmera,b,
HP (formerly UTC Power) (lead);
The following individuals also provided significant feedback and

guidance in the writing of this book: Roger Schmidt, IBM; Don
Beaty, DLB Associates (major commenter on 1st edition).
Production of final book including creation of most graphics – Jeff

Jaworski, DLB Associates; Mike Mangan, DLB Associates.
Book cover design – Joe Lombardo, DLB Associates.
a
Member ASHRAE TC 9.9
b
Member The Green Grid
Over the last several years, energy consumption by data centers in

the US as well as worldwide has become a topic of intense discussion
within the Information and Communication Technologies (ICT) world.
There are numerous publications presenting statistics on the impact of
data center power consumption on the supply of electricity. One of the
more comprehensive studies was that requested by the US Congress in
Public Law 109-431, in which the EPA was mandated to quantify the
electricity usage by US data centers, resulting in Report to Congress on
Server and Data Center Energy Efficiency Public Law 109-431, 2007.
The key finding of this study is that in 2006, US data centers consumed
1.5% of all electricity used in the US and that according to historical
trends, this consumption would rise to 2.9% by 2011. The 1.5%
electricity usage included servers and the infrastructure to support
servers, but did not include network or storage equipment. A graph of
the findings is shown in Figure 1.1.
Figure 1.1 - Projected data center

energy use scenarios (EPA, 2007)
The alarming trend of escalating electricity consumption in US data

centers has spurred the ICT industry to aggressively increase energy
efficiency in order to dramatically reduce power consumption in data
centers. Together, the DOE and The Green Grid have stated that a goal
for 2011 is to achieve a reduction of energy to 100B kWh / year instead
of the current projection of 120B kWh / year for 2011. One of the key
ways in which the industry can achieve the state-of-the-art curve is via
real-time energy efficiency, which is achievable only through the use of
Introduction
real-time energy consumption data using energy efficiency and

productivity metrics. An example of a real-time energy efficiency metric
is the real-time version of the Power Utilization Effectiveness (PUE)
metric as proposed by The Green Grid (Green Grid Data Center Power
Efficiency Metrics: PUE And DCiE, 2008). This metric is defined and
discussed further in Chapter 2. The focus of this book is real-time
energy consumption measurements, with the resulting data to be used in
all the relevant energy efficiency and productivity metrics.
Real-time energy consumption measurements are only possible if all

key subsystems are appropriately instrumented and properly
communicating through use of data center level software. Existing data
centers have varying levels of instrumentation, ranging from very poor to
excellent. For this book, three approaches to instrumentation and
measurement for any given subsystem will be followed. The following
loose guidelines are provided:
 Minimum Practical Measurement

 Best Practical Measurement
 State-of-the-Art Measurement
When deciding what level of measurement to target, a data center

owner / operator needs to keep in mind key items such as capital cost,
data accuracy and resolution, and end-use of the data. These factors will
be dealt with in further detail in subsequent chapters. The following
guidelines, summarized in Table 1.1, are suggested:
 Minimum Practical Measurement – This will require some level of

human activity to perform periodic measurements. This approach
will require zero to limited infrastructure upgrades, and zero to
limited investment in instrumentation. This approach may rely more
heavily on staff (most likely existing) to manually record data, and
will also rely on manufacturers’ equipment data.
 Best Practical Measurement – This will require a lower level of

human activity than the minimum case in order to manually record
data. For this case, it is anticipated that data will be logged in real-
time with extensive trending possible. The instrumentation used
may not necessarily be of the highest accuracy nor will it likely be
the most extensive, with the more difficult to instrument parts of the
facility remaining uninstrumented. Limited modification to
infrastructure should be expected, and some tasks may be beyond the
competency of the existing staff. Less reliance on manufacturers’
data is expected.
 State-of-the-Art Measurement – This will not require human activity

to gather and record data. Data will be collected by automated
systems in real-time and will support extensive trending and analysis.
The instrumentation will be of accuracy suitable for revenue grade.
There will likely be a requirement to upgrade the existing
infrastructure, and it is very likely some level of contractor or
consultant support will be needed for the implementation.
Mixed-use facilities offer the greatest challenge in which to quantify

real time energy consumption. Figures 1.2 through 1.4 show a generic
layout in a mixed-use facility. These figures are schematic in nature and
are not intended to be fully representative of all possible configurations.
Figure 1.2 shows a schematic representation of the electrical distribution
system in a mixed-use data center, while Figure 1.3 shows the
mechanical layout of the same data center type.
Introduction
Figure 1.2 - Schematic representation

of the electrical system in a mixed-
use facility

of the mechanical layout of a data
center housed in a mixed-use facility
Introduction
Figure 1.4 is a simple graphic representing the key metering

locations in a typical data center. The meters acquire power
consumption data from all the electrical and mechanical subsystems
shown in Figures 1.2 and 1.3, respectively. Each metering location is
associated with its own hardware and software protocols, and in many
cases hardware and software is provided by multiple vendors. These
systems generally do not communicate with each other, which creates a
significant issue in progressing toward the display of real-time energy
and productivity metrics. The subsequent chapters will discuss these
issues in further detail.
Figure 1.4 - Key metering locations in

a data center
10
Introduction
1.1 OBJECTIVES FOR THIS BOOK
The following are key objectives for the book:
 Provide an overview of the state of real-time energy consumption

measurements in the data center. The book will cover both legacy as
well as state-of-the-art data centers.
 Discuss the minimum and best practical levels of measurement, as
well as state-of-the-art measurement for real-time energy
consumption measurements (see Chapter 1 – Introduction).
 Provide a detailed discussion of how the measured real-time data
will be used, and in particular how this information will be turned
into knowledge that can lead to actionable items. This will cover the
latest industry data center productivity and energy efficiency metrics
from organizations such as The Green Grid and ASHRAE TC9.9.
Emphasis will also be placed on quantifying the data center’s power
consumption for a data center housed in a mixed-use facility.
The idea behind the state-of-the-art measurement is that the industry

will eventually arrive at the ―pl ug and play‖ data center. Such a data
center will rely on the widespread availability of network-enabled
equipment. For example, at some point in the future, a data center owner
/ operator can expect to ― plug in‖ a key subsystem such as a pump and
have the data center’s operating system recognize the pump in real-time.
This will be followed shortly thereafter by real-time reporting of energy
consumption measurements, and in turn real-time data center
productivity and energy efficiency metrics.
This book will focus on monitoring and control for optimization of

data center energy efficiency. There are, in fact, other benefits that may
arise from real-time monitoring and control. One key benefit includes
predicting the health of the infrastructure by tracking performance trends.
Additionally, while the book presents several examples using the

Power Usage Effectiveness (PUE) metric from The Green Grid, use of
any specific metric such as the DOE’s Energy Usage Effectiveness
(EUE), or The Green Grid’s Data Center Energy Productivity (DCeP) is
left entirely up to the data center owner / operator.
11
1.2 HOW TO USE THIS BOOK
While it is recommended that this book be read in its entirety, it is

possible to benefit from reading only parts of the book. For the benefit
of the reader, the book has been divided into five parts, each containing
chapters dedicated to key components or subsystems. The five parts
included are:
Part 1 Basics
Part 2 Cooling Systems – Air Measurements
Part 3 Cooling Systems – Hydronic Measurements
Part 4 Power Systems Measurements
Part 5 IT Systems Measurements
Part 1 will provide an overview of the book including measurement

devices and software. Chapter 1 sets the stage for the book. Chapter 2
will focus on How, What & Where To Measure. Chapter 3, the
Measurement Devices chapter, will provide an overview of the various
sensor types available. The chapter will provide an overview of sensors
for all electrical (e.g., voltage, current, etc.) and mechanical (e.g.,
pressure, temperature, flow, etc.) systems in the data center. Chapter 4,
the Measurements Collection Systems chapter, will cover the business
objectives that will guide a data center owner / operator to a given level
of instrumentation (i.e., minimum practical, best practical, or state-of-
the-art level of measurement). This chapter will also provide an
overview of the various standards and protocols to facilitate
communication with IT equipment and facilities equipment. The
objective of such protocols is to acquire the real-time power consumption
data and make it readily available to the data center owner / operator.
Finally, this chapter will provide some discussion of how the acquired
and reduced data can be turned into knowledge and subsequent
actionable items that affect the business.
Parts 2 and 3 will provide an overview of the various cooling

systems and subsystem types (e.g., chillers) that are deployed today.
Each chapter will focus on the single most widely deployed subsystem
type and provide a more detailed discussion of the three levels of
instrumentation. The reader will be shown a high level discussion (not a
detailed description) of how to, at each level of instrumentation, use
12
Introduction
measured data and manufacturer’s data to quantify the power

consumption of the specific subsystem type. Special attention is given,
where appropriate, to show the reader how to quantify that part of the
subsystem’s power consumption that is attributable to a data center
housed in a mixed-use facility. For example, in mixed-use facilities, the
cooling towers, chillers, and pumps typically support all parts of the
facility, including the data center.
Part 4 will focus on the power delivery path from the point of entry
into the facility, to the point of delivery to the IT equipment. Specific
attention is paid to Uninterruptible Power Supplies (UPS) and
transformers. As with the other chapters, an overview is provided with
emphasis on the most widely deployed UPSs and transformers.
Part 5 will provide a description of the servers, storage, and

networking equipment deployed in data centers. The three levels of
instrumentation will be discussed, and the reader will be shown how to
roll the total IT equipment power consumption into a single power
consumption number for later use by the data center owner / operator.
There are also Appendices meant to provide additional information

or detail for different subsystems or components. Appendix A provides
additional information for calculating real-time pump efficiency.
Appendix B describes additional methods for quantifying chiller
efficiency. Appendix C focuses on a specific example for calculations
within a mixed-use facility. Appendix D provides additional information
on Uninterruptible Power Supply efficiency measurements. Some rule-
of-thumb calculations are also provided to enable the reader to perform
calculations on power conversion losses. Appendix E, Onsite Combined
Cooling, Heat, and Power (CCHP), describes in some detail the specifics
of CCHP and waste heat recovery. An important feature of the chapter is
the description of how to accommodate CCHP within the calculation of
energy efficiency metrics such as PUE for data centers. Appendix F lists
the nomenclature in the book.
A references section is located at the end of the book.
13
2.1 OVERVIEW
Understanding the overall goals for measurements is as important as

implementing the measurement system and obtaining measurements
within the data center. While accuracy of the measurement devices can
be critical, benefits can be realized through simply obtaining useful data.
Depending on what, where, and how measurements are taken, varying
levels of accuracy of the devices may be implemented.
Potential uses for measured data can include understanding energy

usage as a whole, trending over time, understanding the instantaneous
power consumption of key pieces of equipment, billing, or calculating
energy efficiency using one of the metrics described in this book.
Depending on the purpose, different factors may be paramount for the
collected data. Assuming that a fixed budget exists for obtaining a set of
measurements or calculating a metric, the owner / operator may need to
balance measurement accuracy with frequency, sensor quantity and
location. These decisions ultimately need to be made in accordance with
understanding how, what, and where to measure.
Common to all systems is the opportunity to measure the real-time

power consumption for each subsystem. The real-time energy
consumption can be measured directly via current and voltage
measurements, or accurately via the measurement of power. For purely
electrical equipment such as UPSs and transformers, the only choice of
accounting for power consumption (losses in the case of this type of
equipment) is via direct measurements of current and voltage or power.
For mechanical subsystems such as pumps, compressors, and

blowers, estimated power consumption can be indirectly calculated via
the measurement of temperature, flow rate, and pressure drop. Power

consumption can also be obtained through direct measurements on each
subsystem. The measured data can then be used in conjunction with
manufacturers’ performance data in order to determine efficiency. It is
instructive for the user to consult ASHRAE Guideline 22-2008 for some
guidance with respect to instrumentation of the chilled-water plant
(includes cooling towers, condenser water pumps, chillers, and chilled
water pumps). It is important to note, however, that the level of
instrumentation required for isolating the real-time power consumption
of a data center housed in a mixed-use facility is higher than that shown
in ASHRAE Guideline 22-2008.
For facilities using air side economizers it is important to understand

the condition of the air external to the data center to properly react to
changing external conditions. Particulate and gaseous contamination
will not be discussed in this book, however, more information can be
found in ASHRAE’s ―P articulate and Gaseous Contamination in
Datacom Environments‖.
Measurements can be taken either manually or automatically. The

specific device installed will dictate which option can be used. Generally,
manual readings will be the minimum practical measurement, while
automatic readings will occur in the best practical, and state-of-the art
data center. Automated readings can be stored electronically and trended
over time. Trending can also be achieved with manual readings, but will
take more time and effort to produce.
The decision whether to use minimum practical, best practical, or

state-of-the-art measurements is ultimately a function of the facility and
its stakeholders. This book will introduce a multitude of measurement
devices, locations, and techniques to understand the energy consumption
of common components of a data center. Obtaining the most useful data
should always dictate the techniques employed.
16
How, What, & Where to Measure
2.2 QUANTIFYING ENERGY EFFICIENCY METRICS
The Green Grid recently proposed Power Utilization Effectiveness

(PUE) as an energy efficiency metric (Data Center infrastructure
Efficiency (DCiE) is the reciprocal of PUE). This metric highlights the
amount of power that is consumed in total by the data center, including
IT loads, and the amount for IT and physical infrastructure to support the
IT. Figure 2.1 shows a simple schematic detailing the key data center
subsystems that are accounted for in PUE or DCiE.
Figure 2.1 - Data center energy

efficiency metrics
In keeping with the discussion of each of the data center subsystems
covered in the remainder of the book, Power Utilization Effectiveness
(PUE) is defined as:
Pfac
PUE (2.1)
PIT
17
where Pfac is total power consumed by the facility and PIT is total
power consumed by the IT equipment.
The IT equipment consists of the servers, network gear, storage

equipment etc. At the facility level, the key power consuming
subsystems include the chillers, IT equipment, Computer Room Air
Conditioners / Handlers (CRACs / CRAHs), cooling towers, pumps,
UPSs, etc. Taking these into account, Equation 2-1 can be re-written as:
Pserv Pnet Pstor Pc Pcrac Pct Pp Pn

PUE (2.2)
Pserv Pnet Pstor Pn
where Pserv is total power consumed by the servers, Pnet is total power
consumed by the network equipment, Pstor is total power consumed by
the storage equipment, Pc is total power consumed by the chiller(s), Pcrac
is total power consumed by the Computer Room Air Conditioner(s)
(CRACs), Pct is total power consumed by the cooling tower(s), Pp is total
power consumed by the pump(s), and Pn is total power consumed by the
nth subsystem.
As previously mentioned, the individual chapters on subsystems will

describe how to measure the real-time power consumption for each given
subsystem. For example, ―Chapt er 9 Chillers‖ describes the real-time
measurement of Pc for the three levels of instrumentation described in
―Chapt er 1 Introduction‖. In addition, examples are given in the
Appendices for each subsystem describing how to quantify what
percentage of the full facility power, Pfac, the data center is responsible
for in a mixed-use facility.
A simple example showing the calculation of DCiE and PUE is

illustrative of this point. Assume that a data center’s total facility power
consumption is 2.2 MW, while the IT equipment power consumption is 1
MW. Under this scenario, the DCiE is calculated as:
PUE = (Pfac/PIT)
= (2200 kW/1000 kW) (2.3)
= 2.2
18
How, What, & Where to Measure
DCiE = (PIT/Pfac) · 100%

= (1000 kW/2200 kW) · 100% (2.4)
= 45%
The Green Grid is currently gathering data center performance

information and intends to publish a white paper, in the near future, that
will put the above-calculated values of PUE and DCiE in perspective. In
order to improve the efficiency of the data center, strategies should be
developed that address the efficiency and utilization of both the IT and
facilities systems. Depending on which strategies are implemented, the
numerator or denominator could change, thereby impacting these metrics.
The data center operator should be aware that technologies like
virtualization can lower total IT power, effectively increasing PUE,
while efficiency has gone up. It is recommended that the user follow
The Green Grid’s ―Usag e and Public Reporting Guidelines for The
Green Grid’s Infrastructure Metrics PUE/DCiE‖ for PUE of DCiE.
While the usage of PUE and DCiE has been highlighted here, the
objective is not to focus on any metric in particular. To re-iterate, a key
objective for the book is to educate the data center owner / operator on
how to acquire the real-time power consumption measurement data.
This data can subsequently be used in the determination of any energy
efficiency, data center productivity, or other metric of choice.
19
3.1 OVERVIEW
In the data center, sensors are used to measure key variables such as
temperature, flow rate, current, voltage, pressure, humidity, etc. Meters
generally form part of a monitoring system that has its information
gathered by the appropriate software to display an aggregate view of the
information from a device (e.g., a data center subsystem), a facility, or an
enterprise. Clearly, the various types of software (see Chapter 4) need
the capability to communicate with the various sensor / meter
combinations.
Sensors and meters come in a wide variety of configurations,

accuracies, and connectivity styles. This chapter is intended to provide
the reader with an overview of the various sensor and meter styles, their
basic modes of operations, their general accuracy levels, and their
general applicability to the various subsystems in the data center.
Table 3.1 lists the key data center subsystems covered in this book,
each subsystem’s major components, and the key variables to be
measured for each subsystem. The table shows that there are some
commonalities between subsystems and components. For example,
pumps are deployed in cooling towers for spraying water, for moving
condenser and chilled water, and for moving condensate in CRACs. The
condenser and chilled water pumps will be large pumps (>20 hp [15
kW]), the cooling tower pumps will be mid-size (>5 hp [4 kW]), and the
condensate pumps will be the smallest pumps (<1 hp [0.75 kW]). The
real-time power consumption for each of these pump types can be
directly measured via current and voltage, or with a power meter. The
power consumption for the pumps can be indirectly measured using flow
rate and pressure drop, but this approach is not likely to be effective for
the small volumes of water moved by the condensate pumps. Chapter 7 -
Pumps provides a detailed discussion with respect to the measurement of
the real-time power consumption of pumps.
With respect to the instrumentation, the key specifications such as

span (instrument range), resolution, and accuracy will change as a
function of the equipment that is being measured. For example, it is not
recommended to use a power meter capable of measuring over a range of
0 to 1,000 kW on a pump that consumes a maximum of 20 kW.
Additionally, an important part of sensor accuracy is system calibration
and understanding the potential drift inherent in the type of sensor
selected. Calibration should always be performed with the system
installed compared with from the factory. System calibration can be
impacted by software filtering, slope of the sensor reading and software
scaling.
This chapter will discuss the various sensor types to be used to

measure the key variables, and will provide some high level guidance
with respect to specifying the most accurate instrumentation within the
bounds of the minimum practical, best practical, and state-of-the-art
measurements. This guidance will be provided for all the data center
subsystems and associated components listed in Table 3.1.
22
Measurement Devices
This chapter will provide details on sensors and in some cases,

sensor / meter combinations for the measurement of the following key
variables:
 Temperature
 Pressure (and pressure drop)
 Flow
 Current
 Voltage
 Power (separate from combination of current and voltage)
Sensor selection is dependent upon the quality (accuracy, precision,

drift, rate of response), quantity, installation restrictions, method of
measurement required, signal output requirements (or signal
conditioning), measurement range, turndown ratio, the capabilities of the
intended data recording devices, and the resources available to purchase
and / or support them.
3.2 SENSOR ACCURACY
The impact of inaccurate data can have a dramatic effect on energy

consumption. For example, a 1 °F (0.56 °C) decrease in chilled water
temperature caused by an inaccurate high reading can create a 2 - 4
percent increase in energy usage to maintain that unnecessary low
temperature. Not knowing the real temperatures can cost a fortune in
wasted energy, not to mention wear and tear on the chiller components
by running outside of intended parameters. The four main contributors
to bad data are inaccurate temperature sensors, pressure sensors,
turbulent flow, and human error.
The following example of how inaccurate data can affect energy

efficiency:
Example
Assume a chiller has a design specification of 600 tons (2.11 MW),
drawing 500 full load amps at 460 V with a power factor of 0.9,
producing a 10 °F (5.6 °C) ∆T at a flow rate of 1,440 gpm (327 m3/hr)
and a design efficiency of 0.598 kW/ton, where ― kW‖ refers to the
23
electrical input to the chiller and ―

ton‖ refers to the cooling output of the
chiller (1 ton = 12,000 Btu/h [3.52 kW]). If this chiller’s evaporator
water temperature sensor is reading low by 1 °F (0.56 °C) and the
evaporator water out temperature sensor is reading high by 1 °F
(0.56 °C), a combined error of 2 °F (1.1 °C) or an 8 °F (4.4 °C) ∆T at full
load exists. When the efficiency is calculated, it equals 0.747 kW/ton.
Dividing 0.598 by 0.747 gives 80 percent actual efficiency. A 2 °F
(1.1 °C) error can make it appear that the chiller is 20 percent inefficient.
The inaccuracies shown in the above example can alter scheduling of

maintenance, produce inaccurate cost analysis, and skew the plant load
profile by 20 percent, making decisions concerning chiller sizing very
difficult. Operating at 80 percent of full load rated efficiency, a 600 ton
(2.11 MW) chiller running at 50 percent load, 24 hours / 365 days, at
$0.06 / kWh would indicate a $24,912 loss. This emphasizes the reason
that sensors and gauges should be accurately calibrated to their specified
accuracies.
3.3 TEMPERATURE
There are many different types of temperature sensors, each based on

different technologies. This section will not discuss every type, but
instead the ones most common to data center systems. The main sensor
types focused on are thermocouples, thermistors, and RTDs (resistance
temperature detectors). The reader should be aware that there are lead
length limitations for each of these sensor types.
Table 3.2 gives a brief overview of the sensors to be discussed in the

subsequent subsections, based on ASHRAE Handbook – Fundamentals,
2009.
24
Measurement Devices
Source: ASHRAE Handbook – Fundamentals
3.3.1 Thermocouples
Figure 3.1 shows a typical thermocouple probe. A thermocouple

consists of two electrical conductors that are made of dissimilar metals
and have at least one electrical connection (see for example Figliola and
Beasley (1991), p. 282). The two conductors are formed into a junction
via soldering or twisting to make good electrical contact. The junction of
the two dissimilar metals, typically called the hot junction, produces a
small voltage signal in proportion to the temperature of the junction. One
junction (the hot junction) is typically encased in a sensor probe at the
point of measurement, while the other junction (the cold junction) is
connected to the measuring instrument. Thermocouples are among the
easiest temperature sensors to use and have the advantage of being self-
powered, relatively low cost, stable and durable.
Figure 3.1 - Photograph of a

thermocouple probe
25
Thermocouples are also rugged and very reliable. An example of a

typical use of thermocouples is shown in Figure 3.2. For the chiller plant
(shown with condenser and chilled water pumps), thermocouples are
installed in wells in the water piping. These should be installed close to
the upstream (inlet) of the temperature-changing device, such as chillers
and cooling towers, but as far downstream (outlet) as practical to ensure
that there is no temperature stratification in the outlet flow where the
measurement is taken.

of the use of thermocouples in a
chilled water plant
26
Measurement Devices
3.3.2 Thermistors
Thermistors are special solid-state transducers that behave like

temperature-sensitive electrical resistors that exhibit a change in
electrical resistance with a change in temperature (see for example
Figliola and Beasley (1991), p. 275). Thermistors are special solid-state
transducers that behave like temperature-sensitive electrical resistors that
exhibit a change in electrical resistance with a change in temperature.
Figure 3.3 shows three thermistor probes. A small and measured direct
current is forced through the thermistor, the voltage drop is measured,
and the resistance determined. A functional relationship between
resistance and temperature is used to determine the temperature.
Compared to other temperature sensors, thermistors are very accurate
and precise over relatively small temperature range. An advantage of a
thermistor is the ability to retain specified characteristics after being
subjected to designated environmental or electrical test conditions.
Thermistors can provide high sensitivity and accuracy, but can be more
costly.
Figure 3.3 - Photograph of thermistor

probes
3.3.3 Resistance Temperature Detectors

(RTDs)
Resistance Temperature Detectors (RTDs) are mature and well

understood temperature sensors that have been in use for many years.
Full technical descriptions for RTDs are readily available in any
27
engineering text (see for example Figliola and Beasley (1991), p. 267).
The following text has been excerpted from the website
www.temperatures.com/rtds.html:
―R esistance Temperature Detectors (RTDs) are wire wound and thin

film devices that measure temperature by making use of the physical
principle of the positive temperature coefficient of electrical resistance of
metals (see Figure 3.4 for selected RTDs). The hotter they become, the
larger or higher the value of their electrical resistance. They are among
the most precise temperature sensors available with resolution and
measurement uncertainties of ±0.2 °F (±0.1 °C) or better possible in
special designs. The most commonly used type of RTD element is
platinum; these models are often referred to as Platinum Resistance
Thermometer (PRTs). Additionally, platinum elements are popular
because they can be used over a wide range of temperatures and feature a
quick response time. Platinum's coefficient of resistance is nearly linear.
By using a platinum element, resolutions of ±0.2 °F (±0.1 °C) or better
are possible.‖
Figure 3.4 - Photographs of RTDs
28
Measurement Devices
―The advantages of RTD’s include: stable output over a long period

of time, ease of recalibration, and highly accurate readings over
relatively narrow temperature spans. When compared to thermocouples
their disadvantages are smaller overall temperature range and a greater
initial cost. ―
3.4 PRESSURE
There are numerous types of pressure sensing methods utilizing

different electrical signals and various means of local indication. Local
indicating sensors will have some form of attached display that allows
observation of the reading, typically in engineering units. Local
indicating sensors may also include a provision for an electrical signal
output for remote monitoring. For simplicity only the more common
subset of what is available is discussed. The scope of information
provided is not intended to eliminate the use of other styles of devices
but rather to maintain focus on the more likely scenarios. In all cases, it
is necessary to select wetted sensor materials that are compatible with
fluids to be measured, that the pressure range is appropriate for the
particular application, and that the electrical output signal is compatible
with the analog inputs of the particular BMS (Building Management
System) or Supervisory Control and Data Acquisition (SCADA) system
used.
It is important to understand that not all pressure sensors are suitable

for use with liquids and that they may only be intended for applications
using air or compressed gasses. Another important consideration when
selecting a pressure instrument is to understand whether it measures
absolute or gauge pressure. Gauge pressure is referenced to atmospheric
pressure and is appropriate for most non-laboratory measurements of
fluids within a piping system. Gauge pressure can either indicate
positive pressure above atmospheric conditions or vacuum level below
atmospheric conditions. Compound gauges are capable of measuring
both positive pressures and vacuum.

2009.
29
3.4.1 Bourdon Tubes
Bourdon tubes (Figure 3.5) are offered in a variety of configurations;

from a simple indicating mechanical gauge or indicating gauge combined
with electronic transmitter circuits to generate an electrical signal
proportional to the pressure. Regardless of indication method, all of
these are comprised of a tube (C, Spiral or Helical) that provides a
mechanical motion response that is proportional to the pressure within
the tube. Typically, bourdon tubes would be found on local mechanical
indicating gauges. This type of instrument is suitable in most cases for
minimum practical measurement.

of a Bourdon tube gauge
30
Measurement Devices
3.4.2 Strain Gauge
The strain gauge (Figure 3.6) is typically combined with an

electronic transmitter circuit to provide the desired electrical signal
output. These devices have an element bonded to a flexible substrate.
Pressure applied to the substrate flexes the element, thereby creating a
strain (elongation or compression) of the material. The sensing element
mounted to this substrate undergoes a change in resistance proportional
to the strain applied. The electronic circuit detects the change of
resistance and modifies the signal to one of many possible standard
output signals (4-20 ma, 0-10 Vdc, serial data, etc.). These devices may
also include a local electronic indicating display. This type of device
would be suitable in most cases for best practical or state-of-the-art
measurement.

of a strain gauge
3.5 FLOW—LIQUID
There are a number of technologies for measuring the flow of liquids

in piping systems. They offer varying ranges of accuracy, pressure loss,
31
and cost as well as installation complexity. A brief description of the

predominate technologies is offered. As with pressure sensors, it is
important to select a flow meter that has wetted surface material
compatibility with the fluid to be measured. It is also important to
evaluate system effects due to pressure losses caused by the flow meter,
and to assure that the selected flow meter has an appropriate range for
the particular fluid monitoring application.
In many cases, the flow meters may be referred to as a flow

transmitter and may further be classified as either an indicating or non-
indicating transmitter. In the case of a flow transmitter, the native signal
(pulse, frequency shift, etc.) is converted into an analog signal (4-20 mA,
0-10 Vdc, etc.) that is then scaled to engineering units. It is important to
select a transmitter with electrical characteristics that are compatible with
the SCADA system analog channels. It is also possible to have a
transmitter that provides serial communications interface; Lon, ProfiBus,
BACnet, MODBUS, GPIB, etc. Regardless of the interface means
(analog or serial communication) the distinction of indicating versus
non-indicating is in reference to a local display. An indicating
transmitter will typically have an LCD or similar display of the flow rate
along with other key values. Indicating instruments are often beneficial
during installation and troubleshooting as they offer a convenient
reference for checking against values registered on the SCADA System.
Table 3.4 provides a summary of the flow sensors discussed in this

subsection, based on ASHRAE Handbook – Fundamentals, 2009.
Information such as application, flow range, and precision should be
used by the data center owner / operator to select the correct level of
instrumentation to best map to the required level of instrumentation. The
table and the information provided in this subsection also provides useful
guidance with respect to the types of flow sensors that are best used in
various locations in the data center.
ρ v dh
Reynolds Number Re (3.1)
μ
where ρ is density, v is velocity, dh is hydraulic diameter, and μ is

viscosity.
32
Measurement Devices
33
3.5.1 Paddle Wheel
An assembly including a paddle (Figure 3.7) is placed into the fluid

flow to be measured. As the fluid moves past the paddle, the fluid causes
the paddle to spin. The paddle typically has permanent magnets attached
to it that rotate past a stationary sensing element. The sensor then
produces a pulse output proportional to the fluid speed. These devices
have sensitive installation requirements, needing a certain amount of
straight pipe both upstream and downstream from the paddle’s point of
installation as well as that the fluid is clean and without deposits. Device
accuracy is also influenced by density and viscosity of the fluid being
monitored. Typically these devices will work with bi-direction flow,
although this is not a consideration in most piping applications.

of a paddle wheel
34
Measurement Devices
3.5.2 Turbine Wheel
A turbine wheel (Figure 3.8) is placed into the flow to be measured.

As the fluid moves past the turbine blades, the fluid causes the turbine
wheel to spin. The turbine wheel typically will have permanent magnets
attached to it that rotate past a stationary sensing element. The sensor
gives a pulse output proportional to the fluid speed. These devices are
sensitive to installation needing a certain amount of straight pipe both
upstream and downstream from the instruments point of installation.
Device accuracy is also influenced by density and viscosity of the fluid
being monitored. Typically these devices will be sensitive to the
direction of flow and may not work if installed backwards.

of a turbine wheel
35
3.5.3 Shedding Vortices
The shedding vortices flow meter (Figure 3.9) has no moving parts.
A blunt object is placed into the fluid flow stream and as the fluid moves
past the object, vortices are created, shedding alternately from side to
side of the blunt object. These vortices produce minute local pressure
disturbances that may be monitored. The pressure disturbance frequency
created by the vortices is a function of the fluid velocity, and may be
processed to create a signal that represents the fluid velocity in
engineering units. These meters also need straight pipes both upstream
and downstream from the point of installation. The straight lengths
required are frequently less stringent than for the paddle or turbine wheel.

of a shedding vortices flow meter
36
Measurement Devices
3.5.4 Venturi Nozzle
A Venturi device (Figure 3.10) is placed into the fluid flow that
tapers in the convergent entrance from the nominal flow area to a point
of constriction known as the throat and then gradually increases back to
the nominal flow area. Given the correct geometry, the majority of
pressure losses experienced at the point of maximum constriction are
recovered over the length of increasing cross sectional flow area, or
divergent outlet. Pressure taps are installed within the section
immediately before the convergent entrance and at the point of maximum
constriction. The relationship to pressure change is governed by
Bernoulli’s Equation regarding the conservation of mass and energy.
This establishes a mathematical relationship between the difference in
pressure between the two tap locations and the fluid mass flow rate.
These devices are also sensitive to installation design, and density of the
fluid being measured. The Venturi nozzle has no moving parts and, if
correctly selected for the flow, has minimal pressure losses.
Figure 3.10 - Schematic

representation of a Venturi nozzle
37
3.5.5 Ultrasonic
Ultrasonic technology (Figure 3.11) is noninvasive, and does not

require penetration of the piping. In most cases the instrument is
strapped onto the outer surface of the pipe containing the fluid flow to be
measured. An ultrasonic sound wave is emitted into the fluid and a
separate receiver detects the resulting reflected sound wave frequency.
The Doppler Effect from sound waves reflecting off of moving particles
creates a frequency shift, which is a function of the fluid velocity. These
devices need some quantity of solids or bubbles suspended in the fluid to
get an accurate reading. It is advisable to consult the manufacturer of
such devices to understand minimum particle size (microns) and particle
concentration (ppm).

representation of Doppler ultrasonic
technology
38
Measurement Devices
3.5.6 Variable Area
The variable area flow meter (Figure 3.12) typically requires visual
observation against a graduated scale, although some implementations
may include an electronic sensor. A float is placed in a vertical metering
tube that has an increasing cross sectional area along its height. The tube
is installed such that the flow is upwards through the tube acting against
gravitational forces on the float. The higher the fluid flow rate the higher
the float will be lifted in the tube. These devices are very sensitive to
fluid specific gravity, due to buoyancy forces acting on the float within
the particular fluid. They are less common on larger pipes, although they
may be installed in parallel with an orifice plate on larger pipes.

representation of a variable area flow
meter
39
3.6 FLOW—GAS
The airflow rate in a conduit can be measured using some very basic
instrumentation. A pitot tube coupled with a liquid column manometer is
the most basic instrument used to measure the velocity pressure of the air
in the conduit. Electronic instruments, such micro-manometers, can be
used in lieu of the liquid column manometer and, sometimes, are easier
to use and could be more accurate.
More times than not and irrespective of the type of instrument used
to gather the data, the velocity profile of the air in the conduit is not
uniform. In other words, the air in the conduit has different velocities
across the cross-sectional area of the conduit at the point where it is
being measured. As a result, multiple readings must be taken; each
reading should sample a different location across the cross section of the
conduit. It is important that the samples are taken in a fixed pattern
across the area so that a representative velocity profile is obtained.
These readings are then averaged. The number of readings that are taken
at this point will determine the accuracy of the calculated airflow value.
See ASHARE Handbook – Fundamentals, 2009 for details on duct
measurement traverse patterns and dimensions.
Another value that needs to be determined is the cross-sectional area

of the conduit at the location of the pitot tube readings. If the location
where the readings of the pitot tube is carefully selected, the velocity
profile of the air in the conduit is fairly uniform, and the area of the
conduit can be easily measured and calculated.
Once the cross-sectional area of the conduit and the average velocity
pressure are known, then the following formula is used to calculate the
airflow in the conduit at that particular location:
Q 4005 A q (3.2)
where Q is the volume of air flowing in the conduit (in cfm [m3/hr]),
4005 (use 1.2888 when using metric units) is a constant, A is the cross-
sectional area of the conduit (in ft2 [m2]), q is the average velocity
pressure of the air in the conduit at the prescribed location (in ft2/min2
40
Measurement Devices
[m2/hr2]). Equation 3.1 is simplified and assumes standard pressure and

temperature of the airflow being measured.
More sophisticated, and potentially more accurate, electronic devices,

such as hot-wire anemometers, can take the place of both the pitot tube
and the manometer. Mechanical devices, such as rotating vane
anemometers, can also be used if the conduit has a large enough cross-
sectional area that the size of the instrument does not interfere with the
flow of the air in the conduit. These types of instruments are used to
directly measure the velocity of the air in the conduit. Again, multiple
readings should be taken at the cross-section in question so that an
average velocity can be calculated for that point.
Once the average velocity and the cross-sectional area of the test
location are known, then the volume of air flowing in the conduit can be
calculated by the following formula:
Q V A (3.3)
where Q is volume of air flowing in the conduit, V is the average

velocity of the air in that conduit, and. A is the cross-sectional area of the
conduit.
The next level of ease of measurement and accuracy involves

installing an airflow monitoring station in the conduit. An airflow
monitoring station is a device that is installed permanently in the conduit
and uses pick up points that are mounted in a grid to directly measure the
average velocity of the air that flows past it. Because the cross-sectional
area of the station is known, the airflow calculation is relatively easy.

2009.
41
3.6.1 Pitot Tube and Manometer
The pitot tube (Figure 3.13) is a standard air velocity meter. This
device utilizes two concentric tubes bent at a right angle. One tube
measures the total or impact pressure existing in the air stream while the
other measures only the static pressure. The difference in total pressure
and static pressure can be used to determine the velocity of the air flow.
A manometer is generally connected to this device to measure the
difference in the two pressures.
Typically when using a pitot tube to measure air flow in ventilation

duct, two or more traverses across the diameter of the duct are made at a
location down stream of any major air disturbances in order to take
multiple readings at the center of annular rings of equal area. Multiple
readings are taken to compensate for any potential non-uniform airflow.
Once the velocity pressures are converted to velocities, they can then be
averaged to determine the average velocity across a particular duct. The
number of readings needed will be proportional to the diameter of the
duct with fewer readings being needed for smaller diameters.
The pitot tube is a relatively cheap and inexpensive way to record

airflow readings in the field. Pitot tubes can be constructed to operate
under various turbulence, airflow and vibration characteristics. These
qualities favor pitot tube as a measurement in the field where conditions
are not necessarily predetermined. However, the accuracy of this device
decreases as the velocity of air declines. Pitot tubes are generally not
used when attempting to measure airflows below 600 fpm (3 m/s). The
device is also prone to clogging of the various ports used to take readings.
42
Measurement Devices

representation of a pitot tube
3.6.2 Hot Wire Anemometer
A hot-wire anemometer (Figure 3.14) consists of a very fine wire

heated up to a temperature above that of the ambient air. The velocity of
air will dictate how much heat can be removed from the wire into the air
stream. Therefore, the temperature of the wire and its resistance will
vary based on the air flowing across it.
The relationship of airflow versus resistance can be measured

through several forms of hot wire devices such as Constant-Current
Anemometer (CCA), Constant-Voltage Anemometer (CVA) and
Constant-Temperature Anemometer (CTA). The voltage output from
these anemometers is determined through a given circuit within the
device trying to sustain the specific variable (current, voltage or
temperature) constant.
Hot-wire anemometers are typically preferred in field environments

where rapid velocity fluctuations and turbulent flows are present due to
43
their extremely high frequency-response and fine spatial resolution

compared to other measurement methods. However, the hot-wire
anemometer has proven to be very fragile while also needing to be
recalibrated often due to the accumulation of dust and debris.

representation of a hot-wire
anemometer
3.7 CURRENT
Current in an electric circuit represents electricity consumed by the

load. Current is measured in amperes and is proportional to power
consumed. A current sensor detects the current flow in an electric circuit
and provides a proportional output that is matched to the meter or system
reading the current. Not all applications require current sensors. Some
have a current draw that can be safely and efficiently measured directly
by the meter or the system reading the current. However, the current
draw of most industrial loads is either too high or unsafe to be measured
44
Measurement Devices
directly by such devices so current sensors act as the interface to the

measurement system.
The job of the current sensor is to transform the current flowing

through the circuit into a signal compatible with the measurement system
while providing the appropriate accuracy, resolution, etc. to meet the
overall system requirements. A current sensor is often called a current
transformer (CT). The current sensor must be both physically and
electrically compatible with the type of load being measured while also
meeting overall installation requirements (e.g., invasive versus
noninvasive installations).
Note that the information below is for reference only and does not
include the various safety and training aspects in the application and use
of current sensors. All electrical work should only be performed by
qualified personnel.

2009.
3.7.1 Current Transformers (CT)
Current transformers come in many shapes and sizes and must be

matched to both the load and requirements of the installer. The most
commonly used types are solid core and split core. The size and shape
45
are chosen based upon the wire or bus size carrying the current load.
Figure 3.15 depicts four examples of commercially available current
sensors.
Solid Core - Solid core CTs look like doughnuts. It consists of a

ferrite ring as the transformer core, and a primary winding of the
transformer from which the current will be measured. A multiple turn
secondary winding is integral to the transducer and provides the output
signal. The current flowing through the primary winding produces a
very small current in the secondary winding. When the secondary is
terminated into a resistor, a small voltage proportional to the sensed
current is developed across the resistor. This voltage can be easily
measured.
Solid core CTs are typically used in permanent installations, are very
cost effective and small in size but require that the circuit is broken
during installation. This is acceptable for new construction but may not
be acceptable for retrofit installations since an active load may have to be
shut down while the circuit is de-energized for CT installation.
Split Core - Split core CTs are also used in permanent applications
and work in a similar fashion but have a two-piece core that can be
opened to place the conductor through the CT without breaking the
circuit. Such CTs are larger and more expensive but may be the only
choice for retrofit or other installations where the circuit cannot be
broken to install the CT. The CTs is usually opened for installation by
removing a nut or other fastener.
Clamp-on and Flex CT - Variations of the split core CT are the

clamp-on and flex CTs. Both can be opened for installation without
breaking the circuit. Clamp on CTs have rigid jaws that open by
squeezing the handle. Flex CTs, also called rope CTs, are pliable and
can be bent and wrapped around odd shapes formed by busses and wire
bundles.
46
Measurement Devices
Solid Core CT Split Core CT
Clamp-on CT Flex CT
Figure 3.15 - Examples of

commercially available current
transformers
3.7.2 Selection Considerations
The accuracy of a current sensor is a function of its type, design and

application. Within a given type of current sensor, better accuracy
usually comes at a higher price. The accuracy is commonly specified as
a percent of the full-scale measurement. Accuracy at a specific
frequency (e.g., 60 Hz) or voltage level may also be used.
47
The application of the current sensor needs to be considered when

determining the accuracy requirements of the current sensor. Current
sensors used for billing or revenue purposes often have specific and strict
requirements for accuracy, current and voltage ranges and other
requirements to be matched properly to the overall application. The
American National Standards Institute (ANSI) for North America and
the International Electrotechnical Commission (IEC) for Europe and
other regions set standards for current sensors used in metering and
safety relay applications. It is important to note that most data centers
and industrial and commercial applications are for monitoring only so
adherence to such standards is usually not required.
Current sensors can be used in a variety of applications including

current indicators, current meters, power and energy meters. When
choosing a current sensor you must consider the overall acceptable error
in the final measurement when selecting the appropriate current sensor.
Current can be a standalone measurement or be combined with voltage to
compute energy parameters such as watts (W), volt-amperes (VA), volt-
amperes reactive (VAR), and power factor (PF). In such cases,
measurement errors have an accumulated effect so errors in current
measurement must be combined with the errors in the voltage
measurements to determine overall measurement system accuracy.
ASHRAE Guideline 22-2008 recommends power measurement devices
and electric energy measurement devices have accuracies of 1.5% and
3.0% of reading, respectively.
The bandwidth of a current sensor is the range of frequencies that

can be measured. The current sensor must be capable of measuring the
frequency range required by the application. Although other applications
such as direct current (DC) devices and variable frequency drive (VFD)
exist, most alternating current (AC) power systems in the US operate at
60 Hz, which is referred to as the fundamental frequency. Ideally, this is
the only frequency component that exists. In reality, however, many
loads are non-linear and cause distortion of the current and ultimately the
voltage. Such distortion is usually referred to as harmonics. Harmonics
are multiples of the fundamental frequency and must be measured by
many systems. It is not uncommon to have harmonics beyond the 11th
which is 660 Hz (60 Hz * 11). The current sensor must have an adequate
bandwidth to meet the harmonic measurement requirements of the
48
Measurement Devices
application; otherwise, the current sensor may act as a filter that can
result in inaccurate measurements.
Bandwidth is typically specified as a frequency or frequency range.

When specified as a single frequency number such as 60 Hz the current
sensor is designed for the fundamental frequency only. When specified
as a range such as 60 Hz – 400 Hz the current sensor will pass
frequencies up to 400 Hz before the accuracy is greatly affected. Most
common current sensors do not go beyond 400 Hz, so more expensive
and specialized sensors may be required for higher frequency
measurements.
3.8 VOLTAGE
Voltage is like the pressure of a fluid in a pipe and is a measurement

of potential in an electrical circuit. Voltage is measured between two
points in a circuit, usually between a phase conductor (wiring) and a
reference such as neutral, ground or another phase. As a result, any
voltage measurement requires two connections, plus and minus
(reference).
A voltage sensor provides a proportional output that is matched to

the meter or system measuring the voltage. Not all applications require
voltage sensors. Some accept a voltage that can be safely and accurately
measured directly by a meter or instrument.
The job of the voltage sensor is to transform the voltage into a signal
compatible with the measurement system while providing the appropriate
accuracy, resolution, etc. to meet the overall system requirements. The
voltage sensor must be both physically and electrically compatible with
the circuit being measured while also meeting overall installation
requirements.
Please note that the information below on potential transformers,

voltage dividers, and voltage transducers is for reference only and does
not include the various safety and training aspects in the application and
use of voltage sensors. All electrical work should only be performed by
qualified personnel.
49

2009.
3.8.1 POTENTIAL TRANSFORMER
Potential transformers (PT) are typically used in medium voltage,

high voltage and other AC applications where the meter cannot connect
directly to the circuit to be measured. PTs are also used in some low
voltage systems where the voltage is too high for the meter or
measurement system. PTs are used to accurately and safely meter
potentially dangerous voltage levels. A PT is a transformer so it also
provides electrical isolation from the circuit to be measured. Some
applications use PTs solely for their isolation characteristics and not to
step down the voltage.
Windings, material and design of a PT determine its voltage output,

accuracy and other characteristics. Important characteristics of PTs are
primary voltage, secondary voltage and ratio. The primary voltage is the
voltage of the circuit to be measured. It is typically referenced to the
nominal circuit voltage (2.4 kV, 13.8 kV, etc). Secondary voltage is the
output of the PT and is what will be measured by the meter or system. A
typical PT secondary is 120 V. The ratio is the ratio of the primary to
secondary voltage of the PT and is used by your meter or system to
properly scale the measurement. As an example, a PT with a 20:1 ratio
connected to a 2.4 kV circuit will have secondary voltage of 120 V. This
may also be referred to as a 2400:120 PT.
50
Measurement Devices
3.8.2 Voltage Divider
A voltage divider is a simple resistive circuit that steps a voltage

down to a lower level that is compatible with the meter or measurement
system. With some exceptions, voltage dividers are typically used in low
voltage AC or DC applications. Important characteristics of voltage
dividers are input voltage and output voltage. The input voltage is the
voltage of the circuit to be measured and the output voltage is the
stepped down output of the divider. Voltage dividers are directly
connected to the circuit to be measured and provide no electrical
isolation. As a result, when applying a voltage divider it is important to
consider the load the meter presents to the circuit as it can affect the
voltage output.
3.8.3 Voltage Transducer
Unlike other voltage sensor types, voltage transducers (Figure 3.16),

as defined here, do not produce a direct voltage output that is
proportional to the primary voltage. Instead they produce a proportional
output that is scaled to a standard transducer interface signal such as 4-20
mA or other standard signals such as 0-5 Vdc, 0-10 Vdc, etc. Such
devices are often used by building monitoring or other data acquisition
systems that can compute the voltage from the measured signal. Some
voltage transducers do not produce proportional outputs but produce a
Root Mean Square (RMS) equivalent value of the voltage. In other
words, they are producing the computed value of RMS voltage instead of
the meter or data acquisition system.
51
Potential Transformer Voltage Transducer
Figure 3.16 - Examples of voltage

sensors
3.8.4 Direct Connection (no voltage sensor)
Many data centers do not require the use of a voltage sensor. In such
low voltage applications the meter or measurement system can directly
accept common voltages such as 120 V, 220 V, 277 V and 480 V without
the use of a voltage sensor. Consult with the meter or measurement
system manufacturer for proper application.
3.8.5 Selection Considerations
The accuracy of a voltage sensor is a function of its type, design and

application. Usually, within a given type of voltage sensor, better
accuracy comes at a higher price. The accuracy is usually specified as a
percent of the measurement and usually represents the accuracy over the
specified measurement range of the sensor. You may also see references
such as accuracy at a specific frequency (e.g., 60 Hz) or voltage.
52
Measurement Devices
The application of the voltage sensor needs to be considered when

determining the accuracy requirements. Voltage sensors used for billing
or revenue purposes often have specific and strict requirements for
accuracy, voltage ranges and other requirements to be matched properly
to the overall application. The American National Standards Institute
(ANSI) for North America, the International Electrotechnical
Commission (IEC) for Europe and other regions and other standards
bodies set standards for voltage sensors used in metering and safety relay
applications. It is important to note that most data centers and industrial
and commercial applications are for monitoring only so adherence to
such standards is usually not required.
Voltage sensors can be used in a variety of applications including

voltage indicators, voltage meters, power and energy meters. When
choosing a voltage sensor you must consider the overall acceptable error
in the final measurement when selecting the appropriate voltage sensor.
Voltage can be a standalone measurement or be combined with current to
compute energy parameters such as watts (W), volt-amperes (VA), volt-
ampere reactive (VAR), and power factor (PF). In such cases
measurement errors have an accumulated effect so errors in voltage
measurement must be combined with errors in the current measurements
to determine overall measurement system accuracy. ASHRAE Guideline
22-2008 recommends power measurement devices and electric energy
measurement devices have accuracies of 1.5% and 3.0% of reading,
respectively.
3.9 POWER
Power is the rate at which energy is transferred and is measured in

watts (W). In simple terms, power is the product of voltage and current.
As a result, a power meter must measure both voltage and current in
order to compute watts and other related parameters such as volt-amperes
(VA), volt-ampere reactive (VAR), power factor (PF), demand, energy
and others.

2009.
53
3.9.1 Examples of Power Meters
Portable Meter (Figure 3.17) - Portable power meters come in many

varieties ranging from hand-held single-phase multimeters to
sophisticated three-phase power analyzers with recording and triggering
capabilities. Most have a built in LED or LCD display for manually
recording measured data. Capabilities can vary greatly so it is important
to make sure the meter is appropriate for the application. Portable power
meters are typically used for spot measurements, power surveys and
other short term short measurements.
Panel Meter (Figure 3.17) - Panel meters are usually permanently

installed in switchgear, UPS systems, generators or other electrical
enclosures or devices. Older-style analog panel meters are indicators
that only display one parameter such as watts. More modern digital
panel meters typically have LED or LCD readouts that can also display
many other parameters such as V, A, VA, VAR, and PF. More advanced
digital panel meters can also compute and display some power quality
parameters such as harmonics.
Revenue Meters - As the name implies, revenue meters are used for
billing, revenue or other applications where the meter must accurately
compute power usage. Revenue meters are mostly used by electric
utilities, landlords and others who bill their customers for power usage.
This type of meter is rarely used in data centers unless co-generation,
renewable energy or other such systems exist. It is important to note that
54
Measurement Devices
some utilities can provide a transducer output from their revenue meters
that enables the customer to record the power usage computed by the
utility meter. Such outputs are often in the form of pulse outputs where
the meter generates pulses that represent a certain amount of energy
consumed. The customer’s monitoring system counts these pulses to
record their own energy usage.
Power Transducer (Figure 3.17) - Power transducers produce an

output that is proportional to the computed power. The output is often
scaled to a standard transducer interface signal such as 4-20 ma or other
standard signals such as 0-5 Vdc, 0-10 Vdc, etc. Such devices are often
used by building monitoring or other data acquisition systems that record
power from various points using these transducers.
55
Portable Single Phase

Power Meter Portable 3 Phase Power Meter
Panel Mount Power Meter Power Transducer
Figure 3.17 - Examples of power

meters
56
Measurement Devices
3.9.2 Communications
Power meters that are used with a building monitoring or other data
acquisition system often require remote communication capabilities. In
simplistic terms, there are two main considerations when communicating
with a power meter:
Hardware Communication Interface - Hardware interfaces are often

built into the meter in the form of industry standard serial (RS232 /
RS485), Ethernet (RJ45) or other interfaces. More than one hardware
interface may be provided with the meter. The hardware interface is the
physical connection to the monitoring systems communications network
and is typically part of a network of meters and other devices.
Software Communication Interface - The software interface is the

communication protocol that rides on the hardware interface and often
dictates that certain compatible hardware interfaces be used. Software
interfaces can be proprietary but often comply with industry standards
such as BACnet (ASHRAE / ANSI Standard 135), LonWorks, Modbus
and other interfaces.
It is important to make sure that the meter’s communication interface

is appropriate for the application and is compatible with the monitoring
system in use.
3.9.3 Accuracy
The accuracy of a power meter is a function of its type, design and

application. Usually, within a given type of meter, better accuracy
comes at a higher price. The accuracy is usually specified as a percent of
the measurement and usually represents the accuracy over the specified
measurement range of the meter. Power meters are usually only
specified to work at one frequency such as 60 Hz.
The application of the power meter needs to be considered when

determining the accuracy requirements. Power meters used for billing or
revenue purposes often have specific and strict requirements for accuracy,
voltage ranges and other requirements to be matched properly to the
overall application. The American National Standards Institute (ANSI)
57
for North America, the International Electrotechnical Commission (IEC)

for Europe and other regions and other standards bodies set standards for
power meters for such applications. It is important to note that most data
center, industrial, and commercial applications are for reference only so
adherence to such standards is usually not required.
Power meters can be used in a variety of applications including

power indicators, local display of power readings, and in remote data
acquisition systems for use with a building monitoring or other data
acquisition system. When choosing a power meter you must consider the
overall acceptable error and accuracy of the final measurement. Overall
errors include those of the sensors and the meter. ASHRAE Guideline
22-2008 recommends power measurement devices and electric energy
measurement devices have accuracies of 1.5% and 3.0% of reading,
respectively.
58
4.1 OVERVIEW
At any level of instrumentation, software is needed to aggregate the

power being consumed by the various subsystems in the data center. As
is typical in large and mixed-use facilities, several monitoring and
controlling systems may coexist. Some software packages will be able to
communicate with each other, and others will not. The challenge for the
industry is to standardize on a common protocol that will allow these
various software packages to be able to communicate with each other. It
is particularly critical if a comprehensive roll-up of the power
consumption of all the data center subsystems is to be achieved.
In the data center, sensors acquire data (energy consumption,

temperature, pressure, flow, etc.) which needs to be aggregated into
information either manually or automatically with monitoring systems to
reveal trends and potential critical issues that the data center manager
needs to manage. This information can be analyzed to determine the
actionable items that can be implemented to improve the data center’s
performance. Ensuring that an accurate history of the key data is
available will allow the organization to determine if the expected results
are obtained. The software strategy adopted by the organization should
be in line with the organization’s sustainability objectives, however
simple or complex they are.
Taking single readings of key parameters used to analyze the

operation or efficiency of a data center can lead to misleading results.
Data center loads, both IT loads and non-IT loads, vary over time
depending on many variables including:
 Variation in IT load due to normal fluctuation of business demands

on the computing environment
 Variation in heat loads due to thermal heat transfer, solar heat gains,
air leakage and daylighting.
To develop metrics that can be used to describe energy efficiency or

for planning future needs as IT equipment is added and removed, it is
best to take many power, temperature or other readings of interest over
long periods of time. By using many readings over time, it is possible to
get a more accurate picture of the true or average infrastructure loads.
One possible approach to using many readings over time is the risk
adjusted value derived from readings taken over several days. The mean
and variance of these readings are then used to derive a single number
based on a desired confidence level. This derived number is then used as
representative of the actual load. The issue of data acquisition frequency
will be addressed at a relatively high level in this chapter.
In accordance with the minimum practical, best practical, and state-

of-the-art levels of measurement adopted in the book (Chapter 1) this
chapter will discuss three corresponding levels of software to take
advantage of the three levels of instrumentation. Finally, the chapter will
discuss the metrics and methodologies organizations may implement to
determine their data center’s efficiency.
4.2 BUSINESS QUESTIONS
The choice of software, for any of the previously discussed levels,

should not be made without considering the key business objectives. For
example:
 What will the organization do with the information?

 What is the current energy consumed by the data centers?
 Will the energy consumption be normalized to account for an
increase or decrease in business activity, weather, or other metrics?
 Who currently pays for the energy bills (IT or facilities)?
60
Measurement Collection Systems—Architecture & Software
 Who is driving the software installation initiative (IT, facilities,

Corporate Sustainability, etc.)?
 Who benefits (and who may be penalized) with information gathered
by the software?
 Is there executive sponsorship including from both IT and facilities?
 Will this be an ― event‖ or ―i nstitutionalized‖ as part of the
organization’s near and long-term strategy to optimize energy?
 What efficiency metrics will the organization use (DCiE, PUE, etc.)?
 Will the organization’s supply chain policies be updated to ensure
that Total Cost of Ownership (TCO) is used to procure IT and
facilities equipment in the future versus lowest initial cost?
 What is the retention policy for the data and are adequate backup
processes in place if the primary data is no longer available?
In many organizations, IT and facilities report into different parts of

the organization. Facilities may report into corporate real-estate and a
Chief Operating Officer (COO). The data center facilities personnel may
be part of a general facilities staff which will require additional
coordination. IT personnel typically report to a Chief Information
Officer (CIO) who may or may not be directly responsible for facilities
and its associated operating expenses. In these types of organizations, it
is important that the Chief Financial Officer (CFO) also be engaged to
ensure that the proper financial methodologies and incentives are in place
to ensure long-term success of an energy conversation program.
Coordination between IT and facilities to understand how the energy
profile, technologies, power densities, and energy consumption are
forecasted to ensure that the data center is optimized to reduce energy
consumption and costs.
Technology may not be the limiting factor in implementing an

energy efficiency policy in an organization, but rather the organizational
behaviors of each group need to be aligned for the program to be
institutionalized over time. Executive sponsorship of the energy
efficiency program should be obtained with the objectives, costs, key
metrics, and potential interruptions to the business for the data center to
be instrumented.
The supply chain procedures should ensure the organization is

obtaining solutions that satisfy the lowest total cost of ownership (TCO)
61
versus the lowest initial cost for each system. Oftentimes, IT procures
the IT kit while facilities is responsible for the energy bills. Charge back
policies may be in place in the organization, but may be allocated by
square footage, especially if no instrumentation or allocation process
exists to gather data center energy consumption.
New technologies such as blade servers and virtualization software

enable data centers to operate much more dynamically for certain
applications such as internet applications. Blade servers, versus legacy
volume servers, enable the number of servers, racks, and the total power
to be significantly reduced. However, power densities often increase
from 5kW/rack to 10, 15, 25 kW/rack or more, driving the need for real-
time power measurements to ensure systems have adequate power and
cooling while conserving energy.
An additional benefit that a data center energy efficiency initiative

may provide to an organization is to proactively link IT, facilities, and
electric utilities with the line of business managers to proactively manage
the ecosystem to meet the organization’s business needs. With a
minimal practical measurement, basic energy and IT loads are revealed
and efficiency is available either manually or semi-automatically. By
incorporating a best practical measurement system, real-time information
is available for both IT and facilities to utilize. With state-of-the-art
measurement, some software processes can be automated including when
and where to run IT applications, chargebacks to end-users for energy,
power and cooling capacities, and system health reporting.
Gathering the electric utility bills for the data center or facility will
help in understanding the amount of power (kW) consumed during each
demand period of the utility meter (typically 15 minutes), the amount of
energy (kWh) consumed over the period (typically one month), and all of
the other charges that should be managed to optimize the data center’s
energy. The energy bill(s) will include generation, transmission, and
distribution charges from one or multiple companies. Understanding
these bills may be perplexing at times, but public utility rate information
is usually available from the local utility’s website.
62
4.3 SCALABLE HARDWARE / SOFTWARE

ARCHITECTURE
A typical data center monitoring infrastructure consists of a mixture

of sensors and communications protocols. Infrastructure components
such as Power Distribution Units, Branch Circuit Breaker panels, and
power strips can provide telemetry over Simple Network Management
Protocol (SNMP), BACnet, or Modbus networks. Building management
systems may be available with open presence networks (OPN) interfaces,
and individual devices such as servers may provide an Intelligent
Platform Management Interface (IPMI).
Both the best practical and the state-of-the-art measurement

situations require some automated data collection and recording.
Maximum benefit can be gained through a scalable architecture that can
carry an enterprise from the best practical through to the state-of-the-art,
as well as support intermediate states as the enterprise matures.
A software / hardware architecture to gather the available telemetry,

store it in a central database, and provide useful analysis and display
needs to have the following attributes / capabilities:
 Provide a uniform view of the telemetry data by allowing for the

normalization of data from many different devices, interfaces, and
protocols.
 Reassurance that telemetry can be reliably obtained at the required
rates. It also needs to provide the capability to poll the telemetry in a
way that accounts for differing response times of the various
interfaces and protocols.
 Ability to perform analysis and provide metrics that are reliable to a
known level. The system must be able to poll telemetry data
frequently over long periods of time and statistically account for the
natural variations that the IT load presents.
 Allow for the storage of the telemetry data in a central location for
presentation and analysis.
 Ability to support an enterprise model while providing the ability to
bring telemetry data together from multiple sites.
 Allow for simple scaling of the architecture in tandem with the
expansion and maturing of the enterprise.
63
An essential component of any scalable software / hardware

architecture is the data collection component. The data collection
component should perform, at a minimum, the following functions:
 Auto-discovery and identification of devices capable of providing

telemetry.
 Polling of telemetry from the mixture of sources and protocols found
in the environment.
 Access many different devices based on a ―Dev ice Description‖ file
that describes the specifics relating to each device to be polled.
 Normalization of the telemetry data to standard units for storage,
display, and analysis.
 Delivery of normalized data to a central database for storage, display,
and analysis.
 Provide scalability.
 As the number of telemetry points or sites increases additional data
acquisition devices can be added to accommodate the increased data
load.
 As new devices become available on the market, the software can be
easily extended to incorporate the new devices.
4.4 MEASUREMENT LEVELS
4.4.1 Minimum Practical Measurement
This scenario requires zero to limited infrastructure upgrades and

investment in instrumentation. Chapter 1 provides additional discussion
as to how to differentiate between the various levels of instrumentation
in the data center.
An important data center energy efficiency metric that can be

quantified using any level of software is the PUE, or its inverse the DCiE.
By understanding this metric and knowing how it is trending,
organizations can begin to have a common metric with which to drive
efficiency improvements. ―Hom e grown‖ spreadsheet software for this
level of instrumentation may be utilized for analysis. Backup and access
processes should be incorporated to ensure retention of the data.
64
Even with minimal instrumentation and data logging, the data center
owner / operator can start to perform analysis on the data to observe
trends and anomalies. For example, consider an illustration that uses the
data shown in Tables 4.1 and 4.2. Table 4.1 shows the monthly energy
bills, along with the cost of energy for 2007 and 2008. The table also
shows the average monthly PUE, along with annual averages for all the
data. Table 4.2 summarizes the annual average PUE, energy usage, and
cost of energy. While the energy efficiency for the data center improved,
the total energy bill increased by 8% from 2007 to 2008. The
explanation is provided when considering that the cost of energy rose
from $0.09 to $0.14 / kWh over the same period of time.
If this data center operated at the same efficiency and load in 2008 as
in 2007, the resulting energy bill would have been $1.08M - this
corresponds to a $426,000 or 52% increase instead of the 8% with rising
energy costs. This example illustrates the importance of understanding
data center efficiencies and operating expenses in aggregate to develop a
comprehensive data center energy efficiency strategy.
65
In a minimal practical measurement scenario, the organization may

or may not have an online data acquisition system. If no online system is
in place, a documented process should be established to ensure the
metered data from key subsystems is read on a consistent and timely
manner, and that the data is reviewed for consistency and is accurately
entered into the spreadsheet software.
Consistency is critical, especially in mixed-use facilities where

changes in the power or cooling system may impact the subsystem
metering in place without the user’s knowledge.
Commentary should also be included to account for any anomalies

such as data center outages, new acquisitions, new equipment installed or
new applications installed in the data center to help normalize the data.
4.4.2 Best Practical Measurement
This scenario assumes a combination of manual and automated

measurements. For the automated measurements, a data acquisition
system is in place to gather and store the subsystem metering data. Some
human intervention may be required on a periodic basis.
Without a data center level (or enterprise level) energy efficiency

strategy, changes in one part of the system (IT, power, or cooling) may
have profound changes on other parts of the system. These impacts may
not be readily apparent, especially when time delays are introduced into
the scenario. For example, higher than expected PUE values may be
generated if the IT applications are delayed in a new data center.
66
The level of software and data acquisition should be aligned with the
business objectives, and will also be influenced by the available budget.
A best practical measurement level of software should provide an
organization with the information needed to operate its facilities systems
in real-time to support IT and the organization.
Facility software has historically focused on power and cooling

systems, and IT software has focused on storage, network and computing
systems. In this scenario, organizations are able to display facilities
information to IT Network Operation Centers and vice versa thereby
linking the organizations. While the IT and Facility software systems
may not be fully integrated, having the key metrics available to each
team in real-time helps optimize the system. The investment in these
systems to improve efficiency also benefits the organization in
emergencies when critical information such as run-time on diesel
generators or network traffic is available to both teams to drive business
system performance.
Figure 4.1 provides an example of an energy dashboard that can be

used with a best practical level of measurement (note that this can also be
used with the other levels). To summarize, the dashboard can present
information relating to:
 PUE
 Power quality (utility and data center)
 Power and cooling capacities
 IT network traffic
 Environmental information including outside temperature, humidity,
degree-days
 Utility real-time pricing, demand reduction or interrupt requests
 Carbon footprint
 System health (power, cooling and IT)
67
Figure 4.1 - Sample Enterprise

Energy Dashboard
Development of the energy dashboard can be expensive. To help

justify these software systems, the addition of system health to the
efficiency methodology may be a benefit that supports multiple groups
within the organization and could help gain support enterprise-wide.
To implement these software solutions, a cross-divisional team of IT

and facilities should be established to help each member understand the
goals and critical issues each group may be facing. For example, if
sensors are required to be installed, an orderly shut-down may be
scheduled at the next preventive maintenance interval. In high
availability applications, this may require approval from other members
of the organization.
Linking the IT and Facility software applications will also involve

reviews with both organizations to evaluate network strategies, security,
database strategies and backup procedures. IT may not want the
68
facilities monitoring system running on the business IT network and may

require them to operate an independent network.
Network security should be reviewed with the proper personnel in

the organization to ensure adequate measures are taken to prevent misuse
of the data. Customized screens may be developed for different levels in
the organization from the business managers, financial managers to IT
and facilities operators.
4.4.3 State-of-the-Art Measurement
In this scenario, instrumentation is required at key measurement

points throughout the data center and facilities including automatic data
collection and logging software. With this level of instrumentation,
proactive control of the IT and Facility loads linked with electric utilities
are more practical.
Organizations typically focus on the power and cooling systems for

the data centers and may have sensors installed at various locations with
monitoring systems for facilities personnel. In most organizations, IT
monitors its equipment on a network that may or may not be separate
from the facilities network. Historically, they have not been linked either
physically or organizationally. This chapter tackles this scenario directly.
Many servers are instrumented with systems that can report system
metrics whether or not its host operating system is running. Such servers
have Baseboard Management Controllers (BMC) that are powered by a
small standby power supply. The standby power supply is active
whenever primary power is connected to the unit. BMCs typically share
a network interface card (NIC) with the host microprocessor and
operating system. A portion of the shared NIC is also powered with
standby power. The NIC is connected to the BMS using data center
networking infrastructure. In some instances it may be desirable, for
redundancy or security reasons, to use a separate NIC and networking
infrastructure.
Baseboard management controllers typically execute management

firmware that is proprietary to the IT equipment OEM. BMCs respond
69
to management consoles by way of network protocols, such as TCP/IP or

Remote Management Control Protocol (RMCP+), and session protocols,
such as the Intelligent Platform Management Interface (IPMI) or Web
Services for Management (WS-MAN).
Servers instrumented as above can report physical sensor data, such

as ambient and component temperatures, and when equipped with
instrumented Power Supply Units (PSU), power metrics such as primary
(typically AC) current / voltage and power factor. Many instrumented
power supplies can also report secondary (DC) voltages and currents.
Most servers also maintain a log of critical system or hardware events in
non-volatile memory. Such logs can be helpful for diagnosing physical
events or conditions that cause equipment failure. Data from sensors and
logs are reported to the BMC via an I2C Bus variant such as PM-BUS.
The management controller, in turn, can be queried by a data center
management controller to retrieve and report the data.
Using IT-equipment-based sensors, data center power and thermal

metrics can be gathered and used for real-time control of data center
thermal profiles and real-time power provisioning. Since the sensors are
integrated into equipment, configuration errors and correlation of IT
equipment to PSUs are reduced. Additionally, fine-grain, real-time
thermal profiles and power provisioning are achievable without
significant capital expense. Finally, time-stamped logging of these
metrics can be used to correlate to applications running on the IT
equipment to measure and optimize application efficiency.
Facilities and IT equipment sensing processes have different

constraints imposed on their implementations but are nevertheless
implemented in similar, hierarchical architectures. Facilities
management is a real-time process and therefore requires localized
monitoring and control in order to affect a workable control loop. IT
equipment is deployed at a very large scale and therefore requires
localized monitoring to cope with the volume of data produced by a large
number of computing and networking devices. Both facilities and IT
equipment rely on local monitoring for consolidating local data to be
logged to a central management database.
70
Figure 4.2 illustrates the sensing paths used by IT equipment in data

centers, and how they may be aligned with the facilities monitoring done
in the building management system. Figure 4.3 provides an additional
illustration of the sensing paths along a broadening to multiple sites.
Figure 4.2 - Data center sensing

hierarchy
71
Figure 4.3 - Collecting data center

telemetry in an enterprise
In a typical IT server sensor implementation, the BMC provides a

programmable interface between the physical sensors in the system and
the operating system running on a server. A hardware driver is used to
abstract the BMC’s interface to the management provider. The
management provider then populates sensor metrics into a server
management data structure that is accessible by a management agent.
The management agent can then monitor the data structure and report
metrics to a local management server on a query basis. The agent can
72
also monitor the data structure for critical sensor values and alert a local
management server that an anomaly has occurred. The local
management server collects sensor values and alerts from the managed
server, logs them in a local database and can perform automated tasks to
act on alerts. The local management server can typically manage
hundreds of servers.
Figure 4.4 shows a schematic implementation of a server being

monitored by a Manager Server. In this case, the BMC is accessed by an
Management Interface Driver, which in turn is accessed by a
Management Interface Provider. The Management Interface Provider
populates a management data structure and values that are monitored by
the Management Agent. The Management Agent answers queries from
and send alerts to the Management Client on the Manager Server via
some protocol. The communications between the managed server and
the management server may be carried by a local area network using
TCP/IP transport. There are many examples of similar architectures
using agents and management servers by various companies.
73
Figure 4.4 - Managed server

monitored by a manager server
In a large-scale IT (>1000 managed servers), a central management

server can collect consolidated data from local management servers, as
shown in Figure 4.4. In this case, care must be taken to compress or
otherwise summarize data from the local management servers, as
management datasets from thousands of servers can become extremely
large.
Figure 4.5 shows a data center level data collection architecture,

which in turn points to the various software levels needed to acquire,
reduce, and display the data. The software architecture of the data
collector allows different interfaces and protocols to be accessed for
74
telemetry. To provide the ability to access the different devices that may
exist in the environment, the collector should be driven from device
descriptor files that can be added and extended to support any device that
may be present.
Figure 4.5 - Data collector software

architecture
The Data Collector Manager component is the heart of the system

interpreting the device descriptions, polling for telemetry on the required
schedules, and queuing the data to be pushed out to the controlling
Manager.
75
The Data Collector Interface is the level at which telemetry data is

presented; ―Ri sk Adjusted‖ values are calculated, and derived efficiency
or operational metrics are computed and presented to users. In an
enterprise deployment, this component should be a Web-based Graphical
User Interface (GUI). Web-based GUIs make the system simple to
deploy and maintain thus reducing the cost of ownership for the system.
The basic requirements for this layer of the monitoring system should
include:
 Web browser based GUI for access to all functions of the system.
 Secure access to prevent unauthorized access to the information.
 Ability to aggregate and summarize data for presentation and
reporting.
 Presentation of various dashboards that clearly show the current
efficiency and operational metrics for each facility.
76
5.1 OVERVIEW
Air handler or air-handling unit (AHU), is the name of the general

category of HVAC equipment that is used to convey air in a system. In
its most basic form, an air-handling unit consists of filtration, a heat
transfer coil (or coils) and an Air Moving Device (AMD). Figure 5.1
depicts a schematic of a typical AHU.
Figure 5.1 - Schematic view of Air-

handling Unit (AHU)
The air that is moved by an air handler has mass and this mass is
conveyed between two points or areas in a system because of a
difference in pressure between these two points or areas. This difference

in pressure causes the air to flow from the high-pressure zone to the low-
pressure zone.
Because this air has volume and mass, the flow rate between the
high- and low-pressure zones in the system can be measured. Units of
airflow are cubic feet per minute (cfm) or cubic meters per hour (m3/hr).
Generally, when air is measured in cfm, the flow rate is based on the
volume of the air. It is assumed that the density, and therefore the mass
of the air, does not change as it progresses in the system. This approach
takes certain liberties with the accuracy of the calculation but for the
condition of the air normally seen in a data center, the error is very small.
The pressure difference between the high- and low-pressure zones in

the system can also be measured. The customary units used to measure
this pressure are inches of water column (in WC) or millibars (mbars).
Moving air has two pressure components; a static pressure (SP) and a
velocity pressure (VP). When a mass of air is moving at some velocity,
it has momentum. This momentum is commonly known as velocity
pressure. Velocity pressure is always measured parallel to and against
the direction of the airflow. In order for air to have a velocity pressure
component, it must be moving. Static pressure is the pressure that the air
exerts on the walls of the ―conduit‖, whether it is in motion or not. This
pressure is measured perpendicular to the walls of the conduit.
When air is moving in a conduit (ductwork, plenum, air handler

casing, room, etc.), it has a velocity. The customary units for velocity
are feet per minute (fpm) or meters per second (m/s).
The air moving device (AMD) is the component of the air-handling

system that imparts the energy on the air to create the above-mentioned
differential pressure in the conduit. The AMD motor is the system
component that consumes the power. AMDs come in many different
sizes and configurations and some are better suited for use in air-
handling systems than others. Because of depth and the breadth of this
topic, AMDs will not be discussed in detail in this chapter.
80
Air Handlers
While it is intended that this chapter be as standalone as possible, it

is also recommended that the reader review Chapters 1 – 4 for supporting
material.
Table 5.1 gives an overview of the minimum practical, best practical,

and state-of-the-art measurement levels. Further detail will be presented
in the subsequent subsections.
Various manual velocity measurement procedures can be used to

determine the volume of air flowing in a conduit in an existing system.
With the possible exception of having an airflow monitoring station
installed in a section of conduit, all of these procedures would rely
heavily on staff to gather the required data. Any readings obtained using
this methodology would be instantaneous readings and, as a result, would
only be valid for that particular point in time. However, if enough of
these readings were taken across a number of months or years, this
information could be used to develop high level trends.
81
The power used by the AMDs in an air-handling system can be

measured using a clamp-on type ammeter. Again, this type of reading
would rely heavily on staff and would provide a ― snapshot‖ that was
accurate for that particular point in time. The trending that could be
developed from this approach tends to be more accurate for constant
volume fan systems compared to variable volume systems.
Since the airflow across a section of conduit is generally not uniform,

it is difficult to rely on the information gathered from a single pitot tube
and / or anemometer. Multiple instruments can be installed at the
sampling location and the data could be trended, but the installation of an
airflow monitoring station is a better approach.
An airflow monitoring station can be utilized to gather information

(average air velocity, air flow, pressure and temperature) and then
transmit this information to the front end of a BMS. These stations can
be installed in strategic locations so that the airflow at those points can
be monitored on a continual basis. This approach would provide a
continuous flow of information so that trending can be performed in an
office-type environment.
The power used by the AMD(s) can be gathered using stand-alone,

field-mounted, data gathering devices. A number of different
manufacturers can provide small systems that can be installed and then
left to gather a data point at a predetermined interval. The device can be
programmed to record a sample anywhere from several seconds to hours.
Operators can then download the data stored on the device into a PC and
use the software that is provided with the device to develop trends.
For the state-of-the-art installation, multiple air monitoring stations

could be employed in the main ducts of the facility or inside each of the
air-handling units. The quantity of outdoor air introduced into the data
82
Air Handlers
center can also be monitored. Real-time information regarding the flows

to the data center could be monitored and trended by the BMS.
Some data centers are adding rack temperature sensors to measure

the temperature entering the server as well as to detect any hot air
encroachment over the top of the racks. With advanced control systems,
this data, or the data directly from the server microprocessor can be used
to control the air flow being delivered from the CRAC/H units.
The power input to the AMDs could be monitored using a wattmeter

and this information could also be trended at the BMS. If the volume of
air supplied to the center is controlled, then the speed of the AMDs can
be effectively adjusted using variable frequency drives (VFD). Even
though most VFDs have the capability to transmit power consumption as
well as key electrical data such as voltage and frequency to a BMS, they
may not take into account the inefficiencies (losses) of the drive itself. If
the data center operators feel that it is important to track this loss, then
the information available from the drive could be supplemented with a
wattmeter installed on the incoming service to the drive. (Note: VFD’s
develop harmonics that will confuse greatly a standard wattmeter. Take
this into account when selecting your measuring device.)
83
6.1 OVERVIEW
The category of Air Conditioning (AC) equipment that is most often

utilized in datacom applications is commonly known as Computer Room
Air Conditioners (CRAC) (Figures 6.1 and 6.2). Other terms that are
sometimes applied include precision cooling, precision air conditioning
and close control air conditioners. There is also a distinction made
between direct expansion (DX – compressorized with integrated
refrigerant circuits) as opposed to chilled water air handlers. It is
somewhat common to refer to chilled water Computer Room Air
Handlers as CRAH (Figure 6.3).
Figure 6.1 - Schematic View of a

CRAC
86
Computer Room Units
Figure 6.2 - Typical DX CRAC unit
87
Figure 6.3 - Schematic View of a

CRAH
Precision cooling equipment is required to maintain environmental

conditions, particularly temperature and humidity, as defined in the
ASHRAE Datacom series Thermal Guidelines for Data Processing
Environments. Precision cooling equipment also typically has a high
sensible heat ratio to address the sensible heat given off by the electronic
equipment without spending unnecessary energy on latent cooling.
ASHRAE / ANSI Standard 127 - 2007 Method of Testing for Rating

Computer and Data Processing Room Unitary Air Conditioners refers to
the various types of unitary AC systems employed in data centers. The
main types include:
88
Computer Room Units
 Single-Package Unit
 Unit with Remote Condenser
 Unit with Remote Condensing Unit
 Unit with Remote Air-cooled fluid cooler or cooling tower
 Chilled Water
 Dual Source Chilled Water Primary, DX Backup
 Economizers
Figure 6.4 - Typical Downflow CRAC

unit
The configuration of dual source chilled water primary and DX

backup provides inherent redundancy and has become popular in large
enterprise data centers. The current focus on energy efficiency has led to
increased interest in the use of economizers. Waterside economizers
have been available and utilized for years and there is much discussion
on the issues of humidity control, particulate and gaseous contamination
89
when applying airside economizers. This book will not address the real-
time power consumption measurement or energy savings of such cooling
methods, however there should be a high degree of similarity when
measuring the power consumption of these various configurations.
The equipment that is most often identified as a CRAC unit is a

machine that sits on the raised floor and discharges air into an underfloor
plenum space (Figure 6.4). However, CRAC units can just as easily be
upflow / top discharge or ceiling units.

material.
6.1.1 KEY COMPONENTS
For the purposes of measuring the real-time power consumption of

CRAC units and CRAHs, it is important to have a high-level
understanding of the power consuming components of these types of
units. Table 6.1 provides an overview of the various components that
can be found in air-cooled CRACs, water-cooled CRACs and chilled-
water CRAHs.
Humidification, dehumidification, reheat and outside / makeup air

are important considerations when discussing power consumption
measurements of CRACs and CRAHs, since they are often only parts of
the full system. The power used by humidification, dehumidification and
reheat can be measured or accessed directly from the CRAC unit while
outside / makeup air is typically provided by a separate system.
90
Computer Room Units
When discussing power consumption of a CRAC unit it is important

to recognize that there may be other components that are operating in
conjunction with the device that is actually on the data center floor. Such
a component could be an integral part of a CRAC unit system such as a
remote condenser or condensing unit, or it could be part of a larger
system including chillers and the attendant components of a chilled water
system. In this book, related subsystems such as chillers and cooling
towers are treated in separate chapters.
All of the preceeding information also applies to the CRAH category

except that there are no on-board, close-coupled compressors. However,
one distinguishing characteristic of CRAHs is that they lend themselves
to apply VFDs or, increasingly, electronically commutated motor
controls for varying fan speed based on need (temperature, static
pressure, etc.).
Variable frequency drive (VFD) and Electronically Communicated

Motor (ECM) technology lends itself well to reporting data directly to
BMS and those characteristics will generally be on-board from day one.
Advisements concerning harmonics caused by VFDs, and their
subsequent impact on either communication of data or on any other
components (including IT equipment), may also apply to ECM fans and
should be taken into consideration.
6.1.2 MODES OF OPERATION
A typical CRAC unit has four basic modes of operation; cooling,

reheat, humidification and dehumidification. The mode of operation that
the CRAC unit is operating will dictate which main components are
energized, which in turn drives its overall power consumption. Table 6.2
provides a list of the main components that are energized in each of the
modes of operation.
91
There are several types of reheat that may be utilized in a CRAC

unit; electric, hot water, steam and hot gas. Each type of reheat might
incorporate different functional components that might be internal or
external to the CRAC unit. See Table 6.3 for the list of components that
are energized for each type of reheat.
Compressors with a variety of methods for varying capacity need to

be taken into consideration. These methods include dual compressors,
cylinder unloading, inverter control, tandem circuits, digital control and
other methods of varying capacity to meet the cooling load. The various
modes of operation will result in a variation in energy consumption
including the combination effect of unloading across two circuits in all
possible combinations.
92
Computer Room Units
Fan motors and pumps hooked up to VFDs or utilizing ECM

technology will have the ability to ramp up and down and therefore
energy consumption will vary.
Humidifiers, electric reheat and condensate pump energy usage as

well as compressors, condenser and fluid cooler fan motors and
associated pumps all could have an intermittent usage profile dependent
on a call for any particular mode of operation.
CRAC units are usually part of the initial infrastructure build out and
therefore, similar to the chillers and other large infrastructure equipment
can be considered legacy for the purpose of an older data center that is
being fitted for real-time power consumption measurement. Older
equipment may have limitations in terms of any inherent, onboard
instrumentation or the ability to retrofit with measurement
instrumentation.
In order to measure the real-time power consumption of an

individual CRAC unit the power draw from the various power
consuming components, which may be operating simultaneously, must
be determined. The overall power consumption of a CRAC unit can be
represented as as:
Pcrac Pfan Pcomp Pheat Pcp Phum (6.1)
where Pcrac is the overall CRAC power consumption, Pfan is the fan
power consumption, Pcomp is the compressor power consumption, Pheat is
the power consumption for heaters, and Pcp is the condensate pump
power consumption. The power consumption for the other CRAC modes
of operation can be determined similarly.
Once the overall power consumption of the CRAC has been

determined, this data can then be used for any number of purposes. For
example, the Pcrac can be made available to the facility software for use
in the calculation of the PUE. While PUE is used as a particular example
in this book, the point to be made is that the data can be used in any data
center energy efficiency, productivity, or other metric.
93

A base unit configuration will be used to illustrate each level of

instrumentation and measurement capabilities. This configuration will
be a water / glycol cooled CRAC unit with compressor(s), evaporator fan
motor(s), humidifier, reheat and condensate pump. Some source of
condenser water needs to be supplied to this unit whether it comes from a
fluid / dry cooler, cooling tower or other source. Cooling towers and
other sources of condenser water and the methods for measuring power
associated with these devices are covered in Chapter 8 of this book and
the pump(s) that would be used to move that condenser water are
covered in Chapter 7. This book will examine a dual circuit machine
having two compressors / circuits. This will allow a discussion of part
load conditions in terms of the refrigeration circuit.
The least sophisticated method to obtain real-time measurement of

energy consumption would be to utilize the data that is typically reported
by most modern CRAC unit microprocessors. Generally, this data
consists of run times for high voltage components that have high
amperage draw such as evaporator fan motors, compressors, reheat and
humidifiers. If the power consumption of these components has been
94
Computer Room Units
measured and recorded, then energy usage can be calculated by

determining how many hours a particular component has been operating.
This energy consumption can then be tracked, reported, correlated,
charted and analyzed.
Data can be collected by the building BMS via standard protocols

such as Modbus, BACnet, LonWorks or SNMP. If there is no BMS
available then it could be collected by an operator as a manual task. All
of the data collected should result in the input values for the equations of
DCiE and PUE.
Moving towards higher levels of sophistication in terms of

measurement and instrumentation provides opportunities for higher
degrees of accuracy, greater usability and reporting of data. Figure 6.5 is
an electrical schematic representation of a typical CRAC unit. Also
shown in this figure are recommended instrumentation points for
measuring the CRAC unit’s overall current draw, along with the
measuring points for the power consumption of the key CRAC unit’s
components.
Simple devices such as current transformers (CT) can allow real-time

measurement of power consumption for the power drawing components.
All such devices can be reported in real time through data loggers or
building management systems and can be used to measure consumption
of each of the power drawing components.
Alternatively, a CRAC unit’s overall power consumption and energy

usage can be measured at a single point, prior to the main disconnect.
With voltage taps and CTs, real and reactive power can be measured.
Some devices can also provide power quality analysis. Power and power
quality measurement devices may be hand-held devices designed for spot
measurements, or embedded sub-meters designed for long-term
monitoring. If the CRAH / CRAC unit has a variable frequency drive
(VFD), then the user should select a sub-meter that is designed to handle
the harmonics generated by the VFD. Most VFDs can report drive
current and power consumption. Most VFDs can also map either
current or power to an analog output or to a serial communication port
(e.g., a Modbus register or a BACnet object) for continuous monitoring.
95
The power measurements built into modern VFDs compare favorably to

measurements taken with expensive power quality measurement devices.
Figure 6.5 - Current draw

measurement points for a typical
CRAC
This section discusses the best practical measurement of real-time

power consumption. This would be the case when a data center is either
already instrumented or it is practical to instrument via retrofit with
acceptable, minimal or scheduled downtime. In the case of new
construction there is the opportunity to specify, provide and install all of
the necessary components to accomplish a predetermined energy
management goal. The measurements that should be taken are similar to
those stated previously and are dependent on the cooling methodology
(DX, chilled water or dual source). These values will also increasingly
be measured and even analyzed and trended automatically with little and
ultimately no human involvement.
96
Computer Room Units
In the previous section the least sophisticated method of data

acquisition correlated component run-times with known power
consumption. For the best practical measurement the technician
physically and permanently installs the instrumentation for real-time
monitoring and measurement. An example of location would be current
sensors in place on all connected loads (compressors, motors,
humidifiers, electric reheat, condensate pumps, control transformers).
These sensors should report to a BMS directly or indirectly through data
loggers.
The BMS is then able to report in real-time the energy consumption

and is able to analyze and trend the data in pre-determined reporting
formats that are useful for and useable by the operators.
State-of-the-art measurement of CRAC unit power usage refers to

newer units incorporating technologies that either have inherent
measurement capabilities or that lend themselves to easily capture usable
and relevant measurements from components or to incorporate
measurement instrumentation with a minimum cost or effort. A BMS
should be used to gather, trend and report such data.
97
7.1 OVERVIEW
Any one of a number of methods can be used to determine the

efficiency (or energy consumption) of pumps. These same methods can
also be used to determine the amount of power used by the data center
when it is a part of a multiple use facility. This chapter outlines possible
approaches for minimum practical, best practical and state-of-the-art
measurement. These scenarios are presented to give the reader a number
of options for consideration and are not intended to cover all possible
solutions. While the sensors and software necessary to acquire and
record the real-time power consumption measurements may be discussed
in this chapter, the details were discussed in Chapters 3 and 4.
The subsequent sections discuss the means used to measure the

electrical energy consumed by the data center pumps, as well as the
resulting mechanical energy produced by them.
A subset of possible configurations for centrifugal pumps is

discussed in Appendix A, including potential accounting methods for
applications in a mixed-use facility.

material.
7.2 MEASUREMENT LEVELS—ELECTRICAL

and state-of-the-art electrical measurement levels. Further detail will be
presented in the subsequent subsections.
The sophistication of electrical monitoring may be broken into the

same three levels of effort mentioned above and Figure 7.1 offers
examples of each of the three scenarios. These figures are for illustrative
purposes only and are not intended to be all inclusive.
.
102
Pumps
Figure 7.1 - Electrical Monitoring of

Pumps
Figure 7.1, Column A illustrates a potential set-up for the minimum

practical methods that could be used to measure the amount of energy
used by a pump. Each level generally includes some method to measure
the electrical current consumed by the motor. For this level, a split core
or clamp-on type current transformer can be used to measure the power
flowing in each of the three phase legs of the pump circuit. Pump power
may then be calculated as follows:
103
Power 3 I RMS VRMS,(L-L) PF (7.1)
where IRMS is root-mean-square current, VRMS,(L-L) is root-mean-square

leg-to-leg voltage and PF is power factor of pump’s motor. (A nominal
power factor is generally available from motor manufacturer.)
Alternatively, if using leg-to-neutral voltage, the following

relationship can be used:
Power 3 I RMS VRMS,(L-N) PF (7.2)
where IRMS is root-mean-square current, VRMS,(L-N) is root-mean-

square leg-to-neutral voltage and PF is power factor of pump’s motor.
(A nominal power factor is generally available from motor
manufacturer.)
A manual reading using the clamp-on meter should be recorded at

regular intervals to establish a ―load profile‖. However, a better
approach would be to have this value continuously monitored, recorded
and trended by the Building Management System (BMS) or a stand-
alone data gathering system. (See Chapter 3 – Measurement Devices for
detailed recommendations on instrumentation.)
Figure 7.1, Column B illustrates a possible configuration for best

practical measurement. This level assumes that the pumps are powered
by VFDs. Frequently the VFDs will offer the ability to monitor the
power either by manual means or through the BMS. It is important to
understand that the VFDs are likely to report only the output power and
will not account for the losses of the VFD itself. It is reasonable to apply
a drive loss value to determine total power required by the pump circuit.
This drive loss value is the electrical inefficiency of the drive,
determined by the ratio of power delivered to input power. VFD
manufacturers should have published data regarding drive efficiency but
104
Pumps
the user should be aware that the efficiency of the VFD changes with the
loading and speed of the motor so this might be an arduous task.
In many cases the pump may not be supported by a VFD or it may

not be possible to extract the total amount of power consumed by the
pump from the VFD. In this case, one leg of the pump power circuit
should pass through a current transformer that is monitored by a BMS
system. In this case, an average voltage and power factor may be
assumed and the current drawn by the leg with the current transformer
could also be considered the average value of the three legs. The power
may easily be calculated using Equation 7.1. Alternatively, if using leg-
to-neutral voltage, Equation 7.2 can be used.
Figure 7.1, Column C illustrates a potential configuration for a

situation that requires state-of-the-art measurement. For this level of
effort, the power circuit feeding the VFD should be monitored
continuously with these values trended and logged to the BMS at regular
intervals. Additionally, it may be desirable to have the VFD networked
into the BMS system with the appropriate level of integration to allow
for real time data retrieval from the VFD.
A power monitor is a good choice for this application, and should

measure the real-time voltage of all 3 phases as well as the current of
each phase leg. Ideally, the power monitor and the data that it gathers
should be ―r evenue grade‖. The parameters measured and / or displayed
by a power monitor could include power factor, voltage of each leg,
current of each leg, average voltage, average current, real time power and
ability to totalize energy consumption over time. Power monitors are
able to support a variety of communications protocols including ModBus,
BACnet, LonWorks, etc. Flexibility of the communication interface
protocols should reduce the burden of integration into an existing BMS
deployment.
7.3 MEASUREMENT LEVELS—FLUID
While it is important to understand that a portion of the pumps’

electrical power is consumed by a data center for a given facility, it is
105
also beneficial to understand how much of this power is converted to

mechanical energy in the pumped loop. By understanding this secondary
metric it then becomes very easy to assess the efficiency of the pumping
system. However, even the most efficient pumps (ratio of fluid power to
electrical power) may not necessarily be accurate indicators of an
efficient fluid loop. It is possible that the piping system could have high
pressure losses as a result of excessive pipe length, excessive fittings,
fouled strainers, or high velocities. All of these factors have the ability
to reduce the system’s pumping efficiency.
To understand the pumping efficiency of a system, several

parameters must be measured or inferred. These include the pressure rise
across the pump (the difference in pressure from inlet to outlet) along
with the flow produced by the pump. It will also be necessary to know
the specific gravity of the cooling fluid. This value will be relatively
constant based upon the glycol concentration (if glycol is used) for a
given facility.
The next several sections describe practices that may be categorized

as minimum practical, best practical and state-of-the-art measurements.
Again, these are for illustrative purposes and are not intended to cover all
possibilities.
Table 7.2 gives an overview of the minimum practical, best practical

and state-of-the-art mechanical measurement levels. Further detail will
be presented in the subsequent subsections.
106
Pumps
For constant speed pumps, the working head can be determined by

taking the difference between the discharge and suction pressures.
Frequently, pumping systems have mechanical dial pressure gauges
already installed at pump suction and discharge connections. If no
pressure gauges exist, it is very likely that pressure / temperature (P/T)
taps have been provided. A probe connected to either a digital or a
mechanical meter may then be used to measure the pump pressure at
both the suction and discharge.
This pressure data may then be compared to the performance curve

of that pump at the specific speed and impeller trim (the impeller
diameter) to infer the fluid flow rate. Figure 7.2 depicts generic pump
curves for three different impeller sizes, and three efficiency curves
denoting the percentage of power delivered to the transport of the fluid
versus the power delivered to the pump impeller shaft. It is important
that the actual curve for the pump under consideration be used for this
step in the process. The correct pump curve can (and should) be
obtained from the pump manufacturer.
To use the curve, subtract the suction pressure (pressure at the pump
inlet) from the discharge pressure (pressure at the pump outlet). The
resulting value will be the total differential head developed by the pump.
Be sure that the units of the values obtained from the field agree with
those shown on the pump curve. If they do not, convert the field value to
the appropriate units. Locate this value on the y-axis of Figure 7.2 and
project a horizontal line across to the correct impeller trim. Then draw a
vertical line from this point of intersection on the impellor curve down to
the x-axis. The value on the x-axis will correspond to the flow of the
pump in gallons per minute (gpm).
107
Figure 7.2 - Typical pump operating

curve
Example
STEP 1 - Assume that the pump has a suction pressure of 5 psi and
develops a discharge pressure of 20.6 psi. Subtracting these 2 values
yields a differential pressure of 15.6 psi. Because the y-axis of the pump
curve is shown in ― feet of head‖, we must convert the pressure to the
correct units before we can plot the operating point. One psi is equal to
2.31 feet of head. We can obtain the appropriate value by multiplying
the differential pressure measured in the field of 15.6 psi by 2.31. This
operation yields a value of 36 feet of head.
STEP 2 - Referring to Figure 7.2, a horizontal line has been drawn

from 36 feet of head on the y-axis across to the 10‖ pump impeller. A
vertical line has then been drawn from this point of intersection down to
the x-axis. The corresponding flow is 360 gpm.
108
Pumps
Unfortunately the BMS will not be able to read the pump curve.
However a 2nd or 3rd order polynomial equation may be produced that
will reasonably approximate the pump curve. This curve is developed
using the pressure and flow values from a number of operating points
along the actual pump curve.
The curve for the 10-inch impeller shown in Figure 7.2 approximates
the 3rd order polynomial equation shown below. This equation gives an
R2 value, or the proportion of variability in the data set that is accounted
for by the model, of 99.88%.
Flowrate( gpm) 1920 123.9 h 3.6 h 2 0.03778 h 3 (7.3)
where h is head in feet of water column.
This equation could be programmed into the BMS to provide for an

automated conversion of the differential head measured across the pump
into a flow rate in gallons per minute. The equation is representative for
this particular pump and impellor diameter only and must be calculated
for each pump and impeller trim. Ultimately, the accuracy of the flow
rate will be only as good as the pump curve and curve fit equation. Keep
in mind that any errors introduced by actual pump installation and
operating conditions will also be reflected in this equation. Factors that
could adversely affect pump performance include, but are not necessarily
limited to: inadequate net positive suction head, fluid specific gravity,
fluid viscosity and unusual (non-conforming) inlet or outlet piping
connections to the pump. Ultimately, the flow values derived from such
a means should be viewed as a relative indication of performance or an
approximation and should not taken as an absolute value.
Pump data should be taken on a regularly scheduled basis and

recorded. Again, as with the electrical data, a far better practice would
be to have the BMS continuously monitor, record, and trend the suction
and discharge pressures of the pump at some predetermined interval.
109
The best practical level of measurement could include permanently

mounted pressure transducers installed across the pump and a pump
curve that has been confirmed with a temporarily installed ultrasonic
flow meter. A small portion of insulation will need to be removed from
chilled water piping to ensure an accurate reading. This data may then
be used to calibrate the pump equation to reflect performance of the
pump as it is actually installed.
Alternately if pump curves are not available, a table of data may be

created by manually imposing different pressure drops across the pump.
This may be done by manually opening or closing a flow regulation
valve on the pump discharge (a circuit setter, a triple duty valve, a ball
valve or a butterfly valve). The table should have adequate resolution of
the pressure differentials imposed on the pump and the recorded values
should also extend slightly above and below the anticipated operating
range of the system. Caution should be used when performing such an
activity as temporary flow rate disturbances may cause problems for
operating equipment.
A curve fit polynomial equation may then be developed using the

data recorded in the table. This equation could then be programmed into
the BMS to automatically convert pressure rise across the pump to its
flow rate. When dealing with variable speed pumps, the approach of
inferring pump flow rate by means of curve fit equation using pumping
head as the input variable becomes more problematic as there will be a
unique flow rate for each pump head and speed combination. This
problem could be addressed by a series of equations for various
operating speeds and interpolations between these pump speeds as
needed. However, due to the complexity and variability involved, it
would be best to utilize a flowmeter to determine the pump flow. This
approach is discussed in the next subsection.
The state-of-the-art measurement level involves permanently

mounting pressure transducers across the pump suction and discharge,
connected to and monitored by the BMS. To obtain this level of
110
Pumps
accuracy, the flow must be measured and not calculated as was done for
the previous level of measurement. This approach will require the
installation and monitoring of a flow meter. There are many choices for
the types of flow meters that can be used for an HVAC application.
These include: ultrasonic, in-line turbine, in-line paddle wheel, in-line
shedding vortices, venturi, etc. All but the ultrasonic flow and in-line
paddle wheel meters will likely require at least some system down time
to allow for the installation of the flow meter(s).
It may be possible to install an in-line flow meter in a section of

piping that has isolation and bypass valves installed. An important
consideration for installation of flow meters is to strictly adhere to
manufacturer’s guidelines for minimum lengths of straight piping
upstream and downstream of the flow meter. Failure to follow these
guidelines will surely compromise the accuracy of any readings obtained
with this meter.
Additionally, it is always good practice to install bypass and isolation

valves around any in-line instrumentation that is inserted in, or is a part
of, the piping system. Figure 7.3 schematically depicts such an
installation.
111
Figure 7.3 - Typical Flow Meter

Installation
112
8.1 OVERVIEW
Cooling towers are typically the lowest energy consuming subsystem

in a chilled water loop. Consequently, cooling towers have been the least
likely to be sub-metered for energy usage. The power measurement
methods presented in this chapter provide the means to meter power
consumed by a cooling tower system at various levels of sophistication
and accuracy. These methods will provide industry acceptable means for
minimum practical, best practical and state-of-the-art measurements for
cooling towers. Note that the sensors and software necessary to acquire
real-time power consumption measurements for cooling towers will not
be discussed in this chapter.
The term cooling tower is often used to refer to ― open‖ (Figures 8.1
and 8.2) or ―c losed‖ (Figures 8.3 and 8.4) cooling tower equipment
designs, each having their own inherent operational advantages and
levels of energy usage. The most common and energy efficient cooling
tower designs used in data center applications are axial fan open circuit
cooling towers and axial fan closed circuit cooling towers. Open and
closed cooling towers are available in a variety of mechanical designs for
heat rejection, including:
 Counter Flow – The flow of air and water are in opposite directions
 Cross Flow – The flow of air and water are in perpendicular
directions
 Combined Flow – A combination of cross flow and parallel flow
with air and water flows moving in perpendicular directions and in
the same direction.
Figure 8.1 - Axial fan counter flow

open circuit cooling tower
Figure 8.2 - Axial fan cross flow open

circuit cooling tower
Figure 8.3 - Axial fan counter flow

closed circuit cooling tower
114
Cooling Towers
Figure 8.4 - Axial fan combined flow

closed circuit cooling tower
Please note that axial fan open circuit cooling towers typically
consume less energy than axial fan closed circuit cooling towers;
however, closed circuit cooling towers have inherent design advantages
that reduce equipment downtime including:
 The ability to operate with reduced or zero water use in the event of
water loss, if the closed circuit cooling tower is designed with
extended surface tubes for dry operation. (Please see ― Evaporative
Cooling Choices to Maximize Waterside Economizer Use in
Datacom Installations‖ by Lane and Cline for further information.)
 Reduced fouling of heat exchanger surfaces by isolating the closed
loop from the evaporative cooling loop.
 Simplified water treatment due to smaller volume of water to treat in
the evaporative cooling loop. This is also advantageous for the use
of non-chemical water treatment methods.
For an in-depth description of cooling towers, see the ASHRAE

Handbook HVAC Systems and Equipment Chapter 39 on Cooling
Towers.
The cooling tower energy consuming components that must be

metered include fan motors for open towers; fan and pump motors for
closed circuit cooling towers. The method of energy measurement of
both will be addressed in this chapter.
115

material.

The minimal practical measurement level required to capture cooling

tower fan and pump motor power consumption is the use of a true RMS
power digital sampling meter or handheld wattmeter that records ―direct-
read‖ power measurements. The power measurement device shall meet
the requirements as listed under ASHRAE Guideline 22-2008
Informative Appendix E – Example Specification Language Section IV
Instrumentation and Data Requirements.
For a drive system using a variable frequency drive (VFD) to

optimize system conditions and to reduce energy consumption in a fan
drive system, it is important to meter power before the variable
116
Cooling Towers
frequency drive for an accurate energy consumption measurement.

Efficiency losses occur from the line power to the fan motor because the
variable frequency drive converts the three-phase voltage source to a
chopped DC voltage. Efficiency ratings typically range from 95-98%
with efficiency decreasing with decreased motor load. The efficiency of
the variable frequency drive can be obtained from the supplier.
The handheld device must directly calculate power across all three
legs as shown in Figures 8.5 and 8.6. Data is logged manually for re-
entry into a spreadsheet. Expected accuracy is typically in the range of
±1.5%.
Figure 8.5 - Junction Box for a Fan

Motor
117
Figure 8.6 - Variable Frequency Drive
T
he handheld device will typically calculate power using the Equation 8.1.
If this type of device is not available, determine voltage and current
readings on all three legs and the power factor from the motor nameplate
to calculate power as follows:
Pi I RMS( L L) VRMS 3 PF (8.1)
where Pi is power in watts, VRMS is the RMS mean line-to-line

voltage of three phases, IRMSis the RMS mean current of three phases,
and PF is the power factor.
118
Cooling Towers
The best practical measurement for determining energy consumption

is to utilize current transducers that are connected to power meters. This
will eliminate the human element in the measurement and can gather
continuous data for logging and trending. The current transducers are
either solid core for new installations (Figure 8.7) or split core for easy
retrofit onto existing conductors for legacy installations (Figure 8.8).
Further information on these devices can be found in Chapter 3 –
Measurement Devices.
Figure 8.7 - Solid core transducer
Figure 8.8 - Split Core Transducer
The use of current transducers will transform the fan motor current to
a low voltage that can be read by a data acquisition system. In addition,
the power measurement accuracy of the transducer improves to ±0.2%.
119
Figures 8.9 and 8.10 illustrate the best practical level of

instrumentation for a pump motor using current transducers in a junction
box and a variable frequency drive.
Figure 8.9 - Junction Box for a Pump

Motor
120
Cooling Towers
Figure 8.10 - Variable frequency drive

application
Figure 8.11 is an example of real-time power consumption of a

single cooling tower utilizing current transducers. This is the best
practical measurement for a new installation.
121
Figure 8.11 - Current transducers
Energy management using sub-metering is critical to energy cost

savings. Utility providers are beginning to implement time-of-use rates
once again so that customers can take advantage of shifting peak demand
to off-peak hours. Also, many customers are buying power under real
time pricing with energy costs changing hourly and seasonally. In
addition, advanced metering can pick up electrical supply issues such as
transients, power factor and harmonic distortion, all critical to data center
operation.
122
Cooling Towers
Advanced or smart meters are required in order to measure and

record interval data, make real time energy purchases and to monitor
power quality. Typical metering specifications for revenue grade
applications will include energy pulse output and meet ANSI
specifications for metering accuracy at ±0.1%. Figures 8.12 and 8.13
illustrate a state-of-the-art measurement application of a revenue grade
meter system for a standard junction box and with a variable frequency
drive. Further information on these devices can be found in Chapter 3 –
Measurement Devices.
Figure 8.12 - Revenue grade power

metering with a junction box
123
Figure 8.13 - Revenue grade power

metering for a variable frequency
drive
124
9.1 OVERVIEW
This chapter covers real-time measurement of chillers using three

different levels of instrumentation. The minimum practical measurement
covers legacy chillers, those chillers without any instrumentation on the
chiller and minimal instrumentation on the chilled water loop. The best
practical measurement assumes that the chiller has some instrumentation
and that the chilled water loop is instrumented for water temperature and
flows. Finally, the state-of-the-art measurement has all the
instrumentation necessary to determine the instantaneous energy
consumption and possibly additional instrumentation to determine if the
chiller is running most efficiently. It should be noted that installation of
instrumentation when practical may allow for higher measurement levels
to be obtained on older equipment. In each section of this chapter, it will
be shown how to get the energy consumption (kW) of the chiller to
determine the PUE and the efficiency (kW/ton) of the chiller to analyze
how the chiller is operating at a specific point in time.
Chillers represent the largest single power consumer in the

infrastructure that supports the IT equipment and IT area, and are
consequently receiving significant attention with respect to energy
efficiency. Chillers typically have a long lifespan that ranges from 20 to
30 years. As a result, many data centers may still be using very old
legacy chillers that provide little or no data, and have little or no built-in
instrumentation. This chapter will provide an overview of the various
types of chillers used by data centers. In addition, with the ultimate goal
of getting to real-time power consumption data, the chapter will describe
how to get to a level of minimum practical, best practical, and state-of-
the-art measurement.
The two most common types of chillers used in data centers are air-
cooled and water-cooled chillers. Both chiller types include an
evaporator, compressor, condenser, and expansion device. The main

difference between the two chiller types is the form of final heat rejection.
Air-cooled chillers reject heat through an air-cooled condenser with the
use of fans to move air across the fins of the condenser coil. Unlike
water-cooled chillers, air-cooled chillers do not need a cooling tower for
the final heat rejection. The effectiveness of any air-cooled condenser is
a function of the ambient dry bulb temperature. The lower the dry bulb
temperature, the more efficient the condenser is at converting the
chiller’s gas refrigerant into liquid refrigerant.
Air-cooled chillers (Figure 9.1) range in size from 10 to 500 tons (35
to 1,750 kW) of cooling capacity, and they most commonly use scroll or
helical rotary compressors. Helical rotary compressors are typically used
with the larger capacity chillers. More information on chillers can be
found in the ASHRAE Handbook – HVAC Systems and Equipment.
Figure 9.1 - Helical rotary air-cooled

chiller
Water-cooled chillers reject the heat from the condenser through a
cooling tower. The cooling tower rejects the waste heat and sends the
condenser water back to the chiller via a pump. Water-cooled chillers
can have capacities from 70 to 4,000 tons (250 to 14,000 kW) of cooling
capacity. Most commonly, water-cooled chillers (Figure 9.2) in the
capacity range of 70 to 500 tons (250 to 1,750 kW) use helical rotary
compressors, while those in the capacity range of 300 to 4,000+ tons
(1,050 to 14,000+ kW) use centrifugal compressors.
126
Chillers
Table 9.1 shows some of the relative advantages and disadvantages

of air-cooled and water-cooled chillers, although some of these may vary
depending on required capacity.
Either air- or water-cooled chillers may be used, but the size of the
data center and facility will impact the selection process. Typically, data
centers use water-cooled chillers for several reasons. First, there is value
in the ability to increase redundancy through combinations of chillers,
pumps, and cooling towers. Secondly, the size of the data center may
dictate a water-cooled chiller. Finally, water-cooled chiller systems can
be operated much more efficiently than air-cooled chilled water systems.
127
Figure 9.2 - Helical rotary water-

cooled chiller & Centrifugal rotary
water-cooled chiller
While it is intended that this chapter be as standalone as possible, it is

also recommended that the reader review Chapters 1 – 4 for supporting
material. ASHRAE Handbook – HVAC Systems and Equipment, 2008
and ASHRAE Handbook – Refrigeration, 2006 are two other sources of
information.
128
Chillers

Legacy data centers are assumed to be 15 years old or older. It is

also assumed that such data centers will be using their originally installed
chillers. In general, these chillers will have very limited instrumentation
associated with them, and are assumed to not report key data such as
percent load and efficiency in real-time.
In order to determine the real-time power consumption of a chiller,

key data such as entering and leaving evaporator water temperature,
entering and leaving condenser water temperature, chiller full load and
instantaneous rated load amps (RLA), and condenser or evaporator water
flow rates can be used with varying levels of accuracy. For legacy
chillers, much of this data is not typically available. In order to
determine the real-time power consumption of any legacy chiller, the
minimum practical amount of data needed includes the instantaneous
129
RLA, the entering condenser water temperature (ECDW), and the

leaving evaporator (chilled) water temperature (LCHW). If this data is
not available from the chiller controller, the data center operator will
need to instrument accordingly. In addition to this measured data, it is
assumed that the chiller manufacturer can provide unloading curves for
the chiller at different conditions, and the RLA curves of the chiller at
varying capacity and conditions. An example is provided in Appendix B
to illustrate how to determine real-time power consumption with limited
metering.
Assume that a chiller plant has a legacy chiller running, and that the
following information is known:
Figure 9.3 - Example Chiller

Conditions
Example
The following information was provided by the manufacturer (be

sure that the units of the values obtained from the field agree with those
provided by the manufacturer. If they do not, convert the field value to
the appropriate units):
Chiller capacity at full load = 1,000 tons

Chiller RLA at full load = 800 amps
130
Chillers
The following was measured by the data center owner:

Instantaneous RLA = 545 amps
Entering condenser water temperature = 75 °F
Leaving chilled water temperature = 44 °F
Using the RLA curve provided by the manufacturer (see Figure 9.4),
at 545 amps, the percent load is shown to be 80% (follow the arrows to
the horizontal axis). At this point, the unloading curves (efficiency
curves denoting RLA at various entering condenser water temperatures
and percent loads) of Figure 9.5 can be used. At 80% load, the
unloading curves show that the chiller efficiency is approximately 0.44
kW input energy per ton of cooling (kW/ton).
131
Figure 9.4 - % Load versus RLA at

design conditions
132
Chillers
Figure 9.5 - % Load versus efficiency

(kW/ton) for varying values of ECWT
133
The chiller efficiency has been determined to be 0.44 kW/ton. At

80% load, the instantaneous chiller capacity is calculated as:
Chiller capacity at 80% load = 0.80 · 1,000 tons = 800 tons
In turn, the real-time chiller power consumption (demand) is

calculated using the following equation:
Chiller Power Consumption = Chiller Efficiency (kW/ton) · Load (tons) (9.1)
Chiller power consumption = 0.44 kW/ton · 800 tons = 352 kW
An alternative approach may be to calculate the % load as follows:

545 amps / 800 amps = 0.68. This value of 68% load is then used in
conjunction with Figure 9.5 to estimate the chiller efficiency, after which
the real-time power consumption can be calculated. At 68%, the load is
12% lower than the 80% value provided by Figure 9.4 at an RLA of 545
amps. However, this approach is less accurate because more than one set
of operating conditions can lead to a chiller amp draw of 545 amps, and
therefore should not be used. For example, an RLA of 545 amps may be
reached when the chiller is operating near full load with cooler entering
condenser water temperature, or at a low load condition with warm
entering condenser water. It is very difficult to determine the actual
operating capacity of the chiller using this method because of the
variability in the chilled water system.
Another approach using integrated part load values or non-standard

part load values can be found in Appendix B.
This section discusses the best practical measurement for real-time

power consumption. This scenario implies that the data center is either
already instrumented in an appropriate manner, or that it is practical to
134
Chillers
instrument in this manner. Given the stated scenario, the following data
is needed for the measurements:
 Water flow rate through the evaporator

 Leaving chilled water temperature (LCHW)
 Entering chilled water temperature (ECHW)
 Entering condenser water temperature (ECDW)
 Chiller performance curve (i.e., efficiency versus % load)
In every case, the key questions of what kind of sensor, what level of
accuracy is needed, and where to install the sensors have to be answered.
The flow rates through the evaporator and condenser are generally
measured using a flow meter or differential pressure transducer. Both of
these flow measurement devices have requirements for placement which
may include straight pipe distance before and after the measurement
device. Further information can be found in Chapter 3 – Measurement
Devices.
For the entering and leaving evaporator and condenser water

temperatures, thermocouples placed in thermocouple wells are generally
used. These thermocouples need to be placed in a location that ensures
an accurate and repeatable temperature reading. This can be
accomplished in many ways, but several issues need to be considered.
First the thermocouple needs be in a location that will allow for a mixed
water temperature reading ensuring that the temperature is not being
recorded from an area where temperature stratification is prevalent.
Secondly, a specific straight pipe distance before or after the sensor may
be required.
The placement of measurement instrumentation is critical to the

precision and accuracy of the measurements. For a chiller the basic
types of meters needed are flow meters, temperature sensors, and
pressure sensors. Each of these meters and sensors has manufacturer
recommendations for installation and use in a chilled water system.
The measured data is used to calculate the chiller capacity (tons) as

follows:
135
Q Evaporator · (ECHW – LCHW)

Chiller Capacity (tons) = (9.2)
24
where Q Evaporator is water flow rate through the evaporator (gpm),

ECHW is entering chilled water temperature (°F), and LCHW is leaving
chilled water temperature (°F).
Example
This example is provided to illustrate the calculation of chiller
capacity. Figure 9.6 shows the values to be used in the example
calculation (note that it is important to preserve the units shown). For the
example, the LCHW = 44 °F, the ECHW = 52 °F, and the evaporator
flow rate = 2,400 gpm, with the chiller capacity as according to Equation
9.2:
2400 · (52 – 44)

Chiller Capacity (tons) = = 800 tons
24
Figure 9.6 - Schematic View Of A

Chiller
The chiller has a capacity of 1,000 tons at 100% load. Assume that
the entering condenser water temperature is measured at 75 °F. The data
center operator can now use the chiller’s performance curve to quantify
the chiller’s power consumption. At 800 tons, the chiller is operating at
136
Chillers
800 tons / 1,000 tons = 80% load. Starting at 80% load on the horizontal
axis of the unloading curves shown in Figure 9.5, a vertical line is drawn
until it intersects the 75 °F ECWT curve, at which point a horizontal line
is drawn over to the vertical axis to an efficiency of 0.44 kW/ton. The
power consumption for the chiller is then determined using Equation 9.1.
The chiller power consumption is:
Chiller power consumption = 0.44 kW/ton · 800 tons = 352 kW
Consider the case of a data center housed in a mixed-use facility. In

this mixed-use facility, load may originate from the data center (400
tons), from the office space (150 tons), and from additional lab space
(250 tons). In order to quantify the percentage of the chiller’s power
consumption, at 80% load, associated with the data center, the following
calculation is conducted again using Equation 9.1:
Data Center Chiller Power Consumption = 0.44 kW/ton · 400 tons = 176 kW
This proportioning of cooling in a mixed-use facility is relatively

accurate regardless of where in the chiller performance operating space
the facility is operating.
Chiller controls have been the subject of significant advancement

and now have capabilities to not only measure and display temperatures,
amperage (per phase), voltage (per phase), power factor, and
instantaneous power and energy (kW and kWh), but can help to diagnose
issues that are causing the chiller to be less efficient. A state-of-the-art
chiller controller will also have the ability to send the data collected at
the chiller directly to the BMS. Most chillers come standard today with
instrumentation to not only operate, but to protect itself from damage
caused by mechanical or electrical malfunction. Most of these
137
measurements are displayed at the chiller through a user interface, or this

data can be reported to a BMS.
State-of-the-art chiller plants may have the following measurements

available:
 Leaving chilled water temperature

 Entering chilled water temperature
 Entering condenser water temperature
 Leaving condenser water temperature
 Amperage per phase
 Percent amperage
 Voltage per phase
 Power demand (kW)
 Power consumption (kWh)
 Instantaneous chiller capacity
 Evaporator refrigerant temperature
 Condenser refrigerant temperature
Having these parameters will allow for accurate instantaneous

measurements because this data will account for the chiller power
consumption even with fouled tubes, changing condenser water
temperatures, or if a cooling tower fan is out of service.
A state-of-the-art chiller plant will have the ability to diagnose

potential efficiency losses in the chiller plant. Additional measurements
will be useful to ensure that the chiller is operating most efficiently. For
example, some chillers report evaporator and condenser approach
temperatures. These approach temperatures can be monitored over time
and if the temperature begins to increase then the owner knows that the
tubes need to be cleaned or the refrigerant charge needs to be checked. If
the chiller plant is utilizing an air-cooled variable frequency drive on the
chiller, an increase in the variable frequency drive operating temperature
over time could be an indicator to check if the air filter is clean. Other
measurements can provide information to identify why a chiller is not
working or if it needs maintenance. Using additional measurements on
the entire chilled water system will help make certain the chiller plant is
operating at the optimal efficiency.
138
Chillers
The state-of-the-art chiller plant will monitor the energy

consumption of all equipment and determine the optimal conditions to
run the chilled water system. When a chilled water system is optimized
the owner will have a chilled water system that will be reliable and will
exhibit reduced energy costs. Using the ASHRAE Green Guide: The
Design and Operation of Sustainable Buildings can introduce methods to
help optimize chilled water systems.
139
10.1 OVERVIEW
Even though a heat exchanger (Figure 10.1) is a static piece of

equipment with no moving parts, the parameters used to determine its
configuration during the selection process and the way it is maintained
has a direct bearing on the plant’s energy consumption.
A heat exchanger is a piece of equipment that allows heat to be

efficiently transferred across a physical boundary from one fluid to
another while not allowing the two fluids to mix. For data centers, the
fluids are most likely liquids (water or a glycol solution), as opposed to
vapors or gases.
Figure 10.1 - Schematic View Of A

Heat Exchanger
In a data center, heat exchangers are typically used to isolate clean

water, typically found in a closed cooling system, from dirty water
(passed through an open cooling tower) or a glycol (drycooler) system.
Heat exchangers can also be used to hydraulically isolate open water
systems from closed water systems.
Since a heat exchanger is used to transfer heat from one water stream
to another, there is hot side flow (heat load) and a cold side flow (heat
rejections). For a data center application, heat would be typically
transferred from the warmer load (cooling system) flow to the cooler
source (cooling tower or drycooler) flow.
In order to efficiently transfer heat across the physical boundary

inside the heat exchanger, the cold-side fluid flow rate must be
marginally greater than, or at least equal to, the hot-side fluid flow rate.
A flow ratio of approximately 1.2 between the cold-side flow and hot-
side flow is considered ideal by some heat exchanger manufacturers, and
can provide a small approach temperature across the heat exchanger.
The approach temperature is a temperature difference in a heat

exchange system between the ideally obtainable temperature and the
temperature that is actually obtained. In a heat exchanger it is the
positive difference between the temperatures of the cold-side fluid
entering the heat exchanger and the hot-side fluid leaving the heat
exchanger. A small approach temperature is important in a data center
application because it increases the number of hours per year when a
water-side economizer can be utilized.
Why is all of this background important? If the parameters of a heat

exchanger were not optimized during the selection process or if the
maintenance protocol is lacking, then heat cannot be efficiently
transferred across the heat exchanger and the remainder of the plant must
work harder to provide the required cooling capacity.
The only way to determine if the heat exchanger is operating at or

near its peak efficiency is to measure the flow rates of the fluid streams
on both sides of the exchanger and compare the entering and leaving
temperatures to a characterized performance curve. The flow rates can
be determined by either directly measuring the flows with a flowmeter or
by calculating the flows based on the pressure drop of the fluid streams
through the exchanger. Measuring the flow through the exchanger is a
more accurate approach to establishing the flows since fouling of the
heat exchange surfaces inside the exchanger will affect the pressure drop
of the fluid through the exchanger for a given flow. Changing fluid
viscosities, especially when a glycol solution is used as one of the fluids,
also will affect the flow versus pressure drop relationship.
142
Heat Exchangers
Even though a heat exchanger has no moving parts, it is not a simple

device.

material.
Table 10.1 gives an overview of the minimum practical, best

practical, and state-of-the-art measurement levels. Further detail will be
Establishing the performance of a heat exchanger requires measuring,

at a minimum, flow and change in temperature (ΔT). These values must
be measured across both (the cold and hot) sides of the heat exchanger.
Starting at the most basic (and least accurate) methodology of

measuring these parameters, pressure / temperature (P/T) taps may be
permanently installed in the cold-side and hot-side piping. Probes are
inserted into these taps and the appropriate values are indicated on a
temporary or handheld meter. As the name implies, both pressure and
143
temperature can be measured at each point. When using a temporary or

handheld meter, the indicated values are generally recorded by hand.
In lieu of the P/T taps, a permanent pressure gauge bridge can be

installed around both sides of the exchanger. Readings obtained from a
pressure bridge are also recorded by hand. Using a pressure gauge
bridge eliminates some of the error that might be seen if four, individual
pressure gauges were used. Again, the values of interest are the change
in pressure or ΔP across both sides of the heat exchanger and not the
actual entering and leaving pressure readings themselves. The
differential pressures are then compared against the data obtained from
the exchanger manufacturer to approximate the fluid flow through the
exchanger.
Glass or bi-metal thermometers can be installed in the entering and

leaving connections of the cold-side and hot-side piping to read the fluid
temperatures on both sides of the exchanger. Again, these readings must
be read and recorded manually. The accuracy of the temperature data
may be skewed because of the different errors inherent to the four
individual devices, as well as the human error when these devices are
read.
It must be cautioned that the values obtained using the minimum

practical approaches must be considered rudimentary at best, and may
not provide the accuracy that is required to perform a comprehensive
analysis of the heat exchanger performance. However, the values
obtained using this level of instrumentation will provide a good
understanding of the overall function of the heat exchanger.
This level of measurement focuses on automating the data gathering

process and increasing the accuracy of the fluid flow readings through
the exchanger.
The reader should be cautioned that the accuracy of the data obtained
from this quality level of electronic instrumentation should always be
considered suspect, especially if the data is extracted beyond the decimal
point. Additionally, the measurement intent should always be considered
144
Heat Exchangers
when selecting instruments. When trending data over time, it may be

more important to use instruments that have high repeatability
(precision) as opposed to outstanding accuracy. In other situations,
highly accurate measurements may be required.
A temporarily installed ultrasonic flowmeter can be used to measure

flow through each side of the heat exchanger so that the flow versus
pressure drop curve of the exchanger can be verified. Even though the
flow versus pressure drop curves provided by the heat exchanger
manufacturer are generally close enough, the values used to establish
these curves are the result of calculations used in the heat exchanger
selection program and can vary from the actual values. In an ideal world,
the flowmeter would be a permanent installation so that the flow could
be read directly at all times.
If physical conditions or budget do not allow for the installation of

flowmeters, bourdon tube-type or poppet valve-type differential pressure
transducers can be installed across both sides of the heat exchanger to
record the differential pressure across exchanger. This information, in
conjunction with the manufacturer’s data, can then be used to
approximate the fluid flow through both sides of the exchanger.
Thermocouple or thermistor-type temperature probes installed in the

cold side and hot side piping can be used to provide the accuracy
required by this level of measurement.
The above-mentioned approaches may not provide more accurate

data than what was outlined in the minimum practical level but the big
advantage with stepping up to this level is that data can be collected from
these devices automatically by the BMS at predetermined intervals over
long periods of time with little staff involvement. This data can then be
used to develop comprehensive trending.
Obtaining accurate flow versus pressure drop data has advantages

beyond just approximating the rate of the fluid flow through the
exchanger. If the flow versus pressure drop characteristics of the
exchanger are monitored constantly, then the effects of plate fouling can
145
be documented. Most heat exchanger manufacturers agree that the

pressure drop of the fluid through an exchanger increases before the
effects are seen on the temperature side.
State-of-the-art flow measurement would require that flowmeters be

installed in the two water streams feeding the heat exchanger. The type
of flowmeters that are employed should be compatible with the
characteristics of the fluid and should also take into account any
installation restrictions.
Strain gauge-type differential pressure transmitters could then be

installed across both sides of the heat exchanger so that the information
gathered with these devices could be used in conjunction with the flow
data and other historical information for a complete analysis of the flow
versus pressure drop performance of the exchanger.
RTD temperature transmitters installed across both sides of the heat

exchanger would round out the installation for state-of-the-art
measurement so that the flow versus temperature performance of the
exchanger could be evaluated.
Any and all of the data can be collected from these devices
automatically by the BMS at predetermined intervals over long periods
of time with little staff involvement. This data can then be used to
develop comprehensive trending and would also be valuable to anticipate
plant maintenance requirements.
146
11.1 OVERVIEW
This chapter focuses on the elements of a critical power path for a

data center. The critical power path is in fact not a single path but can be
several meandering paths that cross and split at various places to allow
redundancy. Because there are so many variations and combinations it
would be impossible to identify them all. This chapter attempts to give a
high level view of the places where power could be delivered to and
within a computer room. It also considers how power could be measured
and it identifies the types of data that could be gathered.
Highly precise meters are typically required for power quality and
forensic analysis, but such precision can be overkill for routine efficiency
studies.
One application of the information given here is to identify the

aggregate power consumed by just the computer room within a data
center in order to calculate the data center efficiency within that space.
The dynamics of energy use in a data center can be quite fluid. In a
high-availability, highly redundant data center, capturing data on the
total consumption can be very challenging. In a multi-tenant commercial
space, such as a collocation facility, it could be all the more challenging.
There can be literally thousands of data points gathered at many points
along the critical power path. For power consumption measurement, the
primary objective is typically not to measure the efficiency of every
component in a string, but to determine overall performance and to
identify places where improvements can be made, either through better
operating practices, better design, or upgraded equipment. Secondary
objectives can be to measure power quality or energy consumption by a
particular device.
While high levels of instrumentation might be available throughout

the critical power path, the user must do a cost-benefit analysis to
determine if any additional information that might be gained can justify
the additional cost.
For purposes of this section, we assume that the data center is rated
somewhere between 200 kW and 5000 kW. Smaller data centers may
not have all of the features described herein, and larger data centers are
highly engineered facilities that are not typical. For convenience, we
also limit the discussion to features such as voltages and frequencies
typically found in North America. Although voltage, frequency, and
distribution methods may differ in other parts of the world, the
fundamental principles are the same.

material.
11.2 CRITICAL POWER VERSUS ESSENTIAL

POWER
11.2.1 Critical Power Path
The term ―c ritical power path‖ applies to the entire path through
which electrical power will flow from its entry into a facility to its point
of final consumption by IT equipment. The total critical power path can
consist of many elements, including:
 Utility service entrance equipment such as utility substation
transformers, along with associated transfer switch equipment
 Distribution equipment such as switchboards, switchgear,
panelboards, and power distribution units (PDUs), all of which
contain overload circuit protection devices such as circuit breakers
and / or fuses. Power is distributed over cables or busbars.
 Transformers for voltage conversion and / or power conditioning and
/ or creation of separately derived neutrals
150
Introduction to Critical Power Distribution
 Uninterruptible Power Supply (UPS) systems with batteries or other

energy storage subsystems
For our purposes we shall limit discussion of the critical power path
from the point where the alternating current (AC) power enters the
building to the point where the power is consumed by the information
technology equipment. Power characteristics further upstream through
the distribution system and toward the source of power (for example,
electric utility power plant or on-site generation) may add some
interesting variables, but such discussion would be mostly a distraction.
Likewise, this discussion will not attempt to delve into the complexities
introduced by such innovative techniques as co-generation and waste
heat utilization. Although they are valid topics, they are too specialized
and atypical for a generalized discussion.
Figure 11.1 illustrates the elements typically present in a critical

power path. All of the elements shown can have losses that result in heat
generation to some degree, and all can be points of measurement and
instrumentation. However, losses in distribution equipment are quite
small, typically accounting for less than 1% of total losses. The
dominant sources of power consumption (i.e., efficiency losses) in the
critical power path tend to be the points where power is converted,
primarily in the UPS and transformers.
The focus of Chapters 12 – 16 is on the elements in the Critical

Power Path
151
Figure 11.1 - Electrical Flow Diagram

Denoting Critical And Essential
Power Paths
152
Introduction to Critical Power Distribution
11.2.2 Essential Power Path
Elements in the essential power path are ancillary devices, such as

cooling equipment and motor controls. Such devices are necessary for
the operation of the equipment in the critical path, but are not directly in
the critical path. Elements of essential power are addressed in other
chapters of this book.
The key distinction is the continuity of power. Essential equipment

may be able to tolerate a few seconds or even minutes of power
interruption, whereas critical path equipment cannot tolerate any
unscheduled disruption. Figure 11.1 is a simplistic illustration showing
the difference between the critical power path, the essential power path,
and non-critical loads.
11.2.3 Application to Energy Efficiency Studies
For efficiency studies, the user typically wishes to establish a base

line, make changes, and then measure the effect of the changes against
the base line. This should be done using the same instruments for both
studies. Where critical data is missing, adding low-cost power / energy
meters should be sufficient. The precision of such instruments is usually
good enough to measure improvements. The best place to do this is at
the utility input and the output of UPS systems or the output of the power
distribution cabinets. The best way to do this is to have the ability to
collect data systematically and to aggregate it over a period of weeks to
allow for variations in power consumption.
The Appendices include examples of how the measured data can be

used to calculate efficiency metrics for the entire data center. The first
example, in D.3, is a rule of thumb for those who are unable or unwilling
to collect detailed data, but who need a quick estimate. It provides a
gross estimate at a single moment in time. The other, in D.4, is for a
fairly accurate calculation using power measured at RPDUs in every
equipment rack or cabinet.
153
12.1 OVERVIEW
The electrical service entrance in most data centers includes one or

more medium voltage transformer(s) with protective fuses or switchgear
on the primary side and distribution protective switchgear on the
secondary side. These are generally owned by the electrical utility but
could be installed as part of the data center infrastructure. The
transformer, if included within the infrastructure, should be considered in
the overall critical power path power consumption calculations. The
upstream power equipment also includes associated switchgear and
automatic transfer switches.

material.
12.2 SERVICE ENTRANCE EQUIPMENT
The first point of power measurement is where the AC power enters

the facility. The first step is to determine if elements of the service
entrance equipment are within the control of the facility (and therefore
included in critical power path calculations), or if they are within control
of the utility. A quick way to determine this is to check where the
revenue meters are connected – before or after the transformer. When
included after the transformer, the transformer losses are typically not
included in the calculation, but can be estimated if it is desired to include
them. Figure 12.1 shows a typical service entrance meter.
Figure 12.1 - Typical Service

Entrance Meter
Service entrance transformers typically receive medium voltage and

step it down to a lower voltage (e.g., 13.8 kV to 480 V). The utility or
transformer manufacturer can provide an efficiency curve, but keep in
mind that the efficiency varies with the amount of load (see the
transformer discussion in Chapter 14.2) and should not be considered to
be constant. For our purposes, we assume that utility-side power
measurement is beyond our scope. Ideally, the service entrance
switchgear will have circuit protecting devices and metering for each
branch of the critical power path feeding the data center. In practice,
such separation of the data center load from the rest of the building may
occur further downstream. A typical metering location for the service
entrance is shown in Figure 12.2. For the service entrance, the following
three levels of instrumentation are considered:
156
Upstream Critical Power Distribution
Figure 12.2 - Service Entrance

Metering Location
157

practical, and state-of-the-art measurement levels at the service entrance.
The minimum practical measurement utilizes voltage and current

meters, usually installed on the first fuse or circuit breaker panel
downstream of the service entrance transformers. Readings are taken by
visual inspection and recorded manually, having little or no levels of
automation. The estimated accuracy of meters in this application is
approximately 1% or better, but data collection is prone to human error.
The best practical measurement utilizes power meters mounted on

the first distribution panel downstream of the service entrance
transformers. They are capable of true RMS measurements of line-to-
line (and possibly line-to-neutral) voltage and current per phase.
158
Secondary power quality metering may also be available, such as

calculating and displaying frequency, real load (kW), apparent load
(kVA), power factor, K-factor, flicker, harmonics, inter-harmonics,
mains signaling, voltage deviations, transformer over-temperature, and
other power quality indices. They should also be able to communicate
measurements and alarm conditions through a network with facility-level
controls. The typical accuracy of this method is 1% or better. Figure
12.1 shows a typical socket-mounted service entrance meter.
The state-of-the-art measurement involves all of the features of the

best practical measurement, as well as instrumentation including
automatic controls integrated with facility and remote management
systems via LAN, WAN or Internet. In some applications, such as
mobile applications or restricted access locations, wireless sensors with
battery backup may be useful. Revenue-grade meters are readily
available in a variety of form factors, and may be able to integrate
readings from electric, gas or water meters or to pair with the facility’s
existing energy management system through industry-standard
communication protocols. Such devices are suitable for tenant metering
and sub-billing solutions. The accuracy of these devices is better than
1% and is verified to recognized standards using hundreds of ANSI-
compliant measurements and all standard demand calculation methods,
even including features such as time-of-use metering and line loss
compensation values.
12.3 AUTOMATIC TRANSFER SWITCH (ATS)
Most data centers will have access to more than one source of power.
Typically, the primary source is from the electric utility and the
secondary source is from an on-site back-up source (such as diesel or
turbine generator system, microturbines, or fuel cell system). Automatic
transfer switches, such as the one shown in Figure 12.3 recognize when
the existing power source becomes unsuitable and switch to the alternate
source. An ATS is self-acting for transferring the connected load from
one power source to another, but it can be manually initiated as well.
Larger facilities will typically have several transfer switches to allow
multiple power paths at any point within the overall critical power path.
159
Figure 12.3 – Automatic Transfer

Switch (ATS)

practical, and state-of-the-art measurement levels of the ATS.
160
The minimum practical measurement typically involves status lights

to indicate active and available power sources. Displays may also
provide voltage and frequency measurements to be recorded manually.
The units must communicate their status either visually or via contact
closures. These voltage and frequency measurements, if available,
typically have accuracies of 1% to 2%.
The best practical measurement involves measuring voltage,

frequency, and sometimes current as well as calculated values such as
apparent load (kVA), real load (kW), energy consumption, power factor,
total harmonic distortion (THD), and K-Factor. Status indicators are
present to determine which power source is most within acceptable
parameters. Programmable voltage and frequency synchronization is
also included. Power meters and status indicators mounted on a control
panel communicate with a network to decrease the required level of
human interaction. Estimated accuracies are 1% for voltage
measurements and 0.2% for frequency measurements.

best practical measurement, as well as the ability to integrate with facility
management systems. Remote control capabilities allow remote override
or bypass of a transfer switch. Estimated accuracies are 0.5% for voltage
measurements and 0.2% for frequency measurements.
12.4 PRIMARY ELECTRICAL DISTRIBUTION

SWITCHGEAR
On the secondary side of the service entrance transformer are

protective switchgear and / or transfer switchgear as well as AC bus
wiring that distributes the power to specific loads or downstream
switchboards and panelboards. The switchgear is typically low-voltage
(600, 480 or 208 in North America). Circuit breakers or fuses interface
via bus wiring and cabling to the UPS input. Both the output switchgear
161
and the bus wiring exhibit losses that are equal to the square of the
current through them multiplied by the resistance of the circuit, generally
2
referred to as ―I R losses.‖ Typically, the switchgear / distribution
wiring combination amounts to less than 0.5% loss in the critical path.
Figure 12.4 shows a typical meter that could be installed with each of the
various circuits from the primary electrical distribution switchgear,
suitable for tenant metering.
Figure 12.4 - Utility Grade Metering At

Main Switchboard Critical Buss

practical, and state-of-the-art measurement levels of the primary
electrical distribution switchgear.
162
The minimum practical measurement typically utilizes voltage and

current meters for the entire switchboard; however, readings are taken
visually and recorded manually. The estimated accuracy of this method
is 1% to 5%, and is prone to human error.
The best practical measurement utilizes power meters mounted on

the switchgear for each outgoing circuit. Individual circuit breakers
typically include monitoring and status points. The meters are capable of
true RMS measurements of line-to-line (and possibly line-to-neutral)
voltage, and current per phase. They may also be able to determine
frequency, apparent load (kVA), real load (kW), power factor, harmonic
content and K-factor. System-level alarms may also be included. All
monitoring devices can communicate measurements and alarm
conditions through a network. The estimated accuracy of this method is
between 1% and 3%.
163

management system with microprocessor-based controls to automatically
connect or disconnect specific loads under predefined conditions.
Depending upon the application, metering might be bidirectional in both
generation and standby modes, allowing tracking of on-site power
generation onto the grid. The estimated accuracy of this method is
revenue grade with better than 0.5% for voltage and accurate up to 0.001
A for current.
164
13.1 OVERVIEW
Large data centers typically have one or more centralized

Uninterruptible Power Supply (UPS) systems. The largest data centers
will have multiple UPS systems physically and electrically separated
from each other, with each fed from separate upstream electrical
switchgear, but connectable via ATS. A data center with a centralized
UPS is illustrated in Figure 13.1. The UPS output voltage is typically the
same as the input voltage (e.g., 480 volts input and output). The UPS
system typically consists of multiple power modules whose outputs are
connected in parallel. Individual modules will have some metering on
the input and output, with further metering at the overall system level.
Small facilities or low criticality loads may have only a single module
UPS, in which case module and system level features are the same.
There will also be an automatic bypass and usually there will be a
maintenance bypass, both of which are typically fed from sources
different from the power module source. Power modules have some
metering at their inputs, but the optimum location for capturing detailed
power consumption is upstream at the primary electrical switchgear
described in Chapter 12. UPS metering and instrumentation at the power
module level is described in detail in Section 13.2. System-level
metering is described in Section 13.3
Figure 13.1 - Centralized UPS
Small to medium data centers may have one or more distributed UPS
systems located in the computer room as close to the IT equipment as
possible. Figure 13.2 shows a UPS complete with batteries and power
distribution unit installed with the IT equipment racks on the raised floor
of a computer room. The UPS input voltage is typically stepped down
internally to IT equipment utilization voltage (e.g., 480 V input to
208/120 V output), but it may operate at the IT equipment utilization
voltage (e.g., 208/120 V input and output). Depending on the
construction, there may be multiple power modules, each with a
dedicated input power circuit, or a single input for all power modules.
There is usually an automatic bypass and a maintenance bypass, both of
which can be fed from sources different from the power module source.
166
Uninterruptible Power Supply (UPS)
Figure 13.2 - Distributed UPS in IT

Equipment Room

practical, and state-of-the-art measurements for both power module and
system levels. Further detail will be presented in the subsequent
subsections.
167

material.
13.2 UPS METERING, POWER MODULE LEVEL
The types, features, and accuracy of metering and instrumentation at

the power module will vary from one manufacturer to another and / or
depend upon the rating of the equipment. Module-level metering may be
minimal on distributed UPS systems, with more metering available on
larger, centralized systems. Because most UPS systems will have
multiple power modules, inputs to each module must be aggregated to
calculate total system input power consumption. Sometimes this is most
easily accomplished at upstream electrical input distribution switchgear.
The minimum practical measurement typically provides

measurement of voltage phase-to-phase (and phase-to neutral where
applicable) for input and output. Output current for each phase is
standard, but input current might not be available, in which case current
168
must be measured and power calculated manually or derived from

upstream switchboards. The units must communicate their status either
visually or via contact closures. The estimated measurement accuracy of
this method is 3% to 5%.
The best practical measurement includes a local display of voltage,

frequency, and current measurement for both input and output.
Calculated output values such as apparent load (kVA), real load (kW),
crest factor, and percent load might be available at the module level, but
such values are more common at the system level. Input power factor
and harmonic current may be available. Communication is usually direct
to the system controls, via open source or proprietary systems to a
system management network for manual control. Data can be accessed
via LAN or WAN. The estimated measurement accuracy of this method
is 2% to 4%.

management systems for automatic control of the UPS operation. A
graphical user interface (GUI) display of power flow and operating status
may be included on each module. Remote monitoring capabilities allow
remote manual module operation via system-level controls. Power
modules can be automatically added-to or deleted-from the parallel bus
to achieve optimum performance. The estimated measurement accuracy
of this method is 1% to 3%.
13.3 UPS METERING, SYSTEM LEVEL
13.3.1 Overview
The UPS ―s ystem‖ includes all of the distribution switchgear,

paralleling switchgear, static bypass, maintenance bypass, and possibly
output distribution (See Figure 13.3). The primary consideration is for
the output side of the UPS system (i.e., what is being delivered to the IT
169
loads). For multi-module UPS systems, data points are usually measured
at the paralleling switchgear or output distribution switchboard.
Calculated values may be accomplished at the power equipment itself or
calculations might take place at a remote intelligent monitor.
Figure 13.3 – UPS ―System‖ Within

Power Distribution Scheme
170
The minimum practical measurement typically provides output

voltage phase-to-phase (and phase-to neutral where applicable), current
per phase, and status indicators. Control panels display power flow on a
mimic buss. Under this scheme, power may have to be calculated
manually. The units must communicate their status either visually or via
contact closures. The estimated measurement accuracy of this method is
3% to 5%.
The best practical measurement includes local and remote display of

voltage phase-to-phase (and phase-to-neutral where applicable),
frequency, current each phase (and neutral where applicable). Calculated
values include output apparent load (kVA), real load (kW), power factor,
crest factor, and percent load. A GUI mimic display shows the operating
condition of all system elements including power modules. Alarms and
thresholds are programmable, both locally and remotely.
Communication may be via open source or proprietary systems to a
system management network, and can be in user-selected language. Data
can be accessed via LAN or WAN. Protocols include HTTP, HTTPS,
SMS, SMTP, SSL, and TCP/IP. Typical interfaces are RJ-45 10 / 100 /
1000 Base-T. The estimated measurement accuracy of this method is 2%
to 4%.

management systems. Control capabilities allow remote automatic or
manual operation of power modules, system controls, and circuit
breakers or switches. Power modules can be automatically added or shut
down to achieve optimum performance. The estimated measurement
accuracy of this method is 1% to 3%.
171
14.1 OVERVIEW
Transformers can be used throughout the critical power path. For

purposes of this section, transformers integrated into other equipment are
not discussed. For instance, UPS systems usually contain one or more
transformers, but they are not metered separately; instead, they are
factored into the overall performance of the UPS. This chapter looks
specifically at three-phase transformers introduced into the system as
stand-alone devices, or as Power Distribution Units (PDU) that include
distribution with the transformers. From an efficiency and first-cost
perspective, transformers should be eliminated whenever possible.
However, there may be reasons of cable installation costs or safety for
keeping transformers in the system. One way to eliminate transformers
may be to match the IT input voltage to the UPS output voltage (for
example, 240 volt UPS output and 240 volt server input).

material.
14.2 STAND-ALONE TRANSFORMERS
Transformers are almost always used to convert from one voltage to

another, although this is not a 100% rule. Auto-transformers, used
exclusively for voltage conversion, are small and highly efficient, but
they lack any power conditioning. Transformers can also be used for
power conditioning. The most common is a Delta-Wye isolation
transformer (three-phase input and three-phase plus neutral output).
Utility and off-the-shelf supplied distribution transformers will often
exhibit maximum efficiency at loads between 40% and 60% of their

kVA rating; their efficiency will drop gradually when loaded above 60%
of their rated load, and will drop significantly when loaded below 40% of
their rated load.

practical, and state-of-the-art measurement levels.
The minimum practical measurement for stand-alone transformers,

which covers the great majority of transformers, typically involves no
metering at all. Most stand-alone transformers rely on handheld
instruments, or upstream and downstream devices external to the
transformer for metering.
174
Computer Room Transformer & Power Distribution Unit (PDU)
Figure 14.1 - Typical Transformer

With No Metering (Side Panel
Removed)
The best practical measurement involves measuring voltage and

current, typically by a customized meter for the particular transformer.
Estimated accuracy for these measurements is between 2% and 4%.
The state-of-the-art measurement involves measuring input voltage

from phase to phase, output voltage from phase to phase and phase to
neutral, and current for each phase. Calculated values can include K-
factor, total apparent load (kVA), total real load (kW), power factor, and
transformer temperature. Two-stage over-temperature alarm and
contacts may be utilized to communicate the unit’s status via network
communications. Estimated accuracy for these measurements is between
1% and 2%.
14.3 COMPUTER ROOM POWER DISTRIBUTION

UNITS (PDU)
A PDU is a cabinet used to distribute power within a computer room.

Connection to the PDU usually marks the demarcation between facility
wiring and the IT equipment. The PDU contains one or more power
distribution panels, in which circuit breakers connect via interconnect
175
cables to equipment racks, cabinets, or freestanding IT devices. PDUs

can be stand-alone devices or they can be integrated into a distributed
UPS system. PDUs come in two versions: with or without a transformer.
Figure 14.2 shows a floor-mounted PDU with a transformer and
distribution panelboards.
PDU WITH TRANSFORMER - The most common use of a power

distribution unit is to step down voltage from a centralized UPS system
to IT equipment utilization voltage (e.g., 480 volt input to 208 / 120 volt
output). The transformer is usually an isolation transformer, but it can
sometimes be designed for additional power quality features such as
creation of a common ground, creation of a separately derived neutral,
and harmonic cancellation. Unlike some power supplies, today’s
transformers are typically most efficient at around 50% load.
PDU WITHOUT TRANSFORMER - In distributed UPS

applications it is common for the PDU to have no transformer, in which
case it might depend upon upstream fuses or circuit breakers for overload
protection. Metering may be reduced or totally absent, relying instead
upon upstream monitoring (e.g., at the UPS).
As described above, the main components of a PDU are a

transformer and one or more distribution panelboards. They can range in
rating from 15 kVA to over 300 kVA and are most commonly used to
step down voltage and distribute power in the computer room when fed
from centralized UPS systems located in separate equipment rooms. A
typical application is for a branch circuit to connect to a rack-mounted
PDU (RPDU) device serving an equipment rack. PDU controls can
operate one or more panelboards or, in some cases, individual branch
circuit breakers. More than one PDU can be supported by a single UPS
system. PDU control panels are the prime source of information about
power in the computer room. Figure 14.3 shows the power flow through
a basic PDU with transformer.
176
Figure 14.2 - Floor-Mounted Power

Distribution Unit (PDU)
Figure 14.3 - Power Flow Through A

Power Distribution Unit
177
A variation of a PDU is a remote power panel (RPP). This is usually

nothing more than one or more distribution panel boards in an enclosure
that is sub-fed from an upstream PDU in order to add more branch
circuits closer to the load(s). RPPs usually have no additional
monitoring, but could be a point of measurement with a clamp-on meter.

practical, and state-of-the-art measurement levels of a PDU.
The minimum practical measurement will include analog input

voltage from phase to phase, output voltage from phase to phase and
from phase to neutral, output current for each phase, and total apparent
178
load (kVA). Two-stage over-temperature alarm and contacts may be

utilized to communicate the unit’s status via contact closures and RS232
ports. Estimated accuracy for these measurements is between 2% and
5%.
The best practical measurement involves measuring true RMS input

voltage from phase to phase, input phase rotation, output voltage from
phase to phase and phase to neutral, frequency, output voltage total
harmonic distortion (THD), output current for each phase, K-factor, total
apparent load (kVA), total real load (kW), energy consumption (kWh),
power factor, percent load, and transformer temperature. Out of
threshold alarms may be communicated via network communications.
Estimated accuracy for these measurements is between 1% and 2%.

best practical measurement, as well as the current measurement and on /
off control per output branch circuit. Network communications will
enable the use of a facility management interface. Data can be accessed
via LAN or WAN. Protocols include HTTP, HTTPS, SMS, SMTP, SSL,
and TCP / IP. Typical interfaces are RJ-45 10 / 100 / 1000 Base-T. The
estimated accuracy of these measurements is 1%.
14.4 RACK-MOUNTED POWER DISTRIBUTION UNIT

(RPDU)
An RPDU is a device typically mounted in an equipment rack or

cabinet into which IT equipment connects (usually by an equipment
power plug). There can be several RPDU devices mounted in a cabinet
and each is usually powered from a different upstream UPS or alternate
power source. Instrumentation in an RPDU can range from no metering
at all to limited metering to metering with limited controls. Current per
phase, when available, enables a technician to load balance within a rack.
An RPDU can interface with environmental monitoring devices to give a
total picture of the power, temperature and humidity in a single
equipment rack or cabinet. Most RPDU current sensors are low cost
179
components designed to protect the circuit against overload, not for

precise measurement. As RPDUs are used more as an energy efficiency
tool, their accuracy is predicted to improve.
An RPDU can be mounted horizontally or vertically. Its purpose is

to allow fast connection and disconnection of power to IT equipment
mounted in the rack or cabinet. An RPDU can have a wide range of
monitoring and control features. If mounted horizontally the unit takes
up ―U‖ space in the rack and can be read from the front of the rack. If
mounted vertically, the unit takes up zero ―U‖ space but must be read
from the rear of the rack or cabinet.
Some modern servers are able to measure and report power

consumption. Proposed ENERGY STAR server specifications
(ENERGY STAR Program Requirements for Computers, Version 5.0)
require 5% accuracy. Users may wish to gather power consumption data
directly from servers in the long term rather than from RPDU at the rack
level.

practical, and state-of-the-art measurement levels of an RPDU.
180
The minimum practical measurement provides analog input current

for each phase, monitored via the web, Telnet, SNMP, SSH or a
proprietary network. The estimated accuracy for this method is typically
5% of the full scale.
The best practical measurement involves measuring true RMS

voltage from phase to phase and from phase to neutral, and current for
each phase. Alarm thresholds, remote individual outlet control, time
delays, overload and warning alarms can all be communicated and
managed web, Telnet, SNMP, SSH or a proprietary network. The
estimated accuracy for this method is typically 5% of the full scale.
Figure 14.4 - Typical Rack-Mounted

Power Distribution Unit (RPDU) With
Metering
181

best practical measurement, as well as the output current per connected
load, real load (kW), and power factor. It can also utilize a facility
management system interface. The estimated accuracy for this method is
typically between 2% and 3%.
182
15.1 OVERVIEW
The IT equipment in the data center is generally comprised of a

combination of multiple types of datacom equipment including servers,
storage, and networking gear. Each data center is made up of different
percentages of these types of equipment. The first step in the process of
determining how to measure power consumption data should be a
detailed inventory of all equipment in the data center. This inventory
should include type, model number, age, configuration, and rack, row
and room location of the equipment. Also useful in the inventory is a
listing of the typical workloads run on each machine, and whether the
workload changes, is fixed, or if the machine is virtualized. Figures 15.1
and 15.2 show the typical power architecture from the UPS to the rack,
servers and within the servers, for both single and double power supply
servers. This architecture will be referred to within the chapter in terms
of where power can and should be measured to gauge IT power
consumption. This architecture is not the only available architecture but
is shown here because it is the most pervasive. Different combinations
of power distribution components can provide alternate architectures,
possibly optimized for specific types of workloads, alternate voltages, or
reliability requirements. The Green Grid has published a White Paper on
alternate architectures that should be reviewed for further information.
(Please see ― Qualitative Analysis of Power Distribution Configurations
for Data Centers‖ for further information, Oct 23, 2007,
www.thegreengrid.org)
Figure 15.1 - Typical Power

Architecture With Single Power
Supply Servers
186
Compute & Storage Systems
Figure 15.2 - Typical Power

Architecture With Double Power
Supply Servers
Storage equipment within the data center can take many different
forms, ranging from a large monolithic system to smaller storage
processors which mount in a standard 19-inch (483 mm) rack (Figure
15.3). In addition, these systems can be active very close to 100% of the
time, or they can be used for backup purposes and may only run as a
batch job as infrequently as once per day or less. For these reasons, it is
important to monitor the equipment for as long a period of time as
practical to be able to capture these variations in energy consumption.
187
Unlike servers, much of the storage equipment designed over the last few
years does not provide power consumption data as a standard output, and
measurements need to be taken in order to determine actual power
consumption.
Figure 15.3 - Examples of storage

systems
The importance of determining the age or model number of a server,

storage device or piece of network gear is based upon determining a
device’s capability for power reporting and power management. Modern
devices can generally report energy (or alternatively current) being
consumed. Most older devices do not have this capability. Some
research may be required to understand the capability of each type of IT
equipment deployed. Knowing the capabilities of the IT base equipment
can assist in determining the appropriate methodology for IT equipment
power measurement.
Depending on the IT layout of a data center, the IT equipment could

be intermingled or often the storage equipment can be located in a
completely different space. The storage could be spinning media (hard
drives) or tape back-up. In capturing the total efficiency of a data center,
remotely located storage must be accounted for since it is important to
capture all IT loads. The equipment may be in a different space for
reliability (the entire space may not have UPS) or density reasons (tape
back-up systems do not have cooling requirements as high as servers),
188
but nonetheless it needs to be accounted for if it is part of the overall IT

system under review.
One of the most important aspects to measuring the power

consumption in the data center is understanding the workload and use
profile. Unlike typical HVAC equipment, which has predictable diurnal
and seasonal power use variations, IT equipment will likely have a less
intuitive pattern of power consumption. For example, a server
supporting typical office use (e-mail, web access, document storage)
typically has a pattern of peak usage and power consumption in the first
several hours of the workday, another peak right after the lunch hour, and
a smaller one near the end of the workday. This pattern is generally true
for the five working days, and the rest of the week has very low
utilization. To best understand the energy use of such a server, a
measurement window of 24 hours is recommended at a minimum, but a
full week provides an even better representation. Only measuring first
thing in the morning, or the middle of the afternoon, or over a weekend
(so as to not potentially disrupt server operation) would all yield different
results with none being suitable for understanding the full energy picture.
Another possible workload would be that of an engineering or High

Performance Computing (HPC) nature. For these types of workloads,
utilization and power draw can be quite high during the actual
computation and relatively low once the job has completed. It is
important to note that the job run-length is typically dependent on the
size or complexity of the job rather than the time of day. The jobs may
run for just minutes or a few hours, and some can even run for days or
weeks. The energy consumption must be characterized over a long
enough period to capture both the run time utilization as well as the
typical duration of idle time between jobs. Detailed coordination with
the workload scheduler will improve the understanding of these patterns
as well as the validity of the results.
Another over-riding feature of the IT equipment that needs to be

understood in any measurement protocol is that of redundant power feeds.
Often the equipment will have two power supplies with each server fed
from a separate and independent power system. The power supplies are
generally tied together inside the IT equipment, on the motherboard
power distribution, such that if either source of power fails the other will
189
ramp up to carry the full load. Caution must be exercised by

understanding where redundant supplies are being used, and accounting
for these will be critical to accurately measuring the power consumption.
There is a variety of methods that can be used to determine IT power

consumption. As this value is fundamental to understanding the data
center’s energy efficiency, these methods must be explored and
appropriate choices made based on need, cost, and capability. As
discussed in Chapter 2, a commonly used metric to determine a facility’s
energy efficiency is PUE or DCiE (the inverse of PUE). Accurate
quantification of this metric is dependent upon accurate power
consumption measurements. The IT equipment’s power consumption
should be aggregated into a total represented by PIT, which can be is used
in the calculation of the PUE. (See Chapter 2 for PUE discussion)

material.

190
Design Values
The simplest way to determine IT equipment power consumption is

to use design values for the IT equipment to estimate the power usage.
This could include vendor-supplied data for projected power
consumption for a given workload (which may be found in the product
literature), or even a ―
rule of thumb‖ percentage of the nameplate data.
Figure 15.4 shows a typical nameplate sticker for a server. The
nameplate power rating listed is generally much higher than actual
consumption, so derating by 50% is often suggested. The Green Grid
White Paper Proper Sizing of IT Power and Cooling Loads
(www.thegreengrid.org, July 27, 2009) discusses the difficulties involved
with this approach as well as more appropriate methodologies to gain
this information and should be read in conjunction with this book to
obtain the most comprehensive understanding of the challenges. Use of
nameplate data is the simplest but least accurate methodology and is not
recommended. The only benefit to this method is that it is better than
doing nothing at all.
Figure 15.4 - Example of a server’s

nameplate sticker
In response to customer requests, IT equipment manufacturers are

now providing much more information about server power consumption
than in the recent past. The ASHRAE Thermal Guidelines book
(Thermal Guidelines for Data Processing Environments, ASHRAE,
191
2008) suggested this level of visibility from IT manufacturers. In

addition, there are a number of power consumption calculators available
on vendor’s websites. Figure 15.5 shows an example of a power
consumption calculator. These tools can be reasonably accurate and
allow different configurations and workloads to calculate a predicted
power usage. It should be kept in mind that these tools are not as good
as actual measured data, but are far better than using nameplate data.
Figure 15.5 - Example of an IT

Equipment power consumption
calculator
192
In the case of equipment used for backup storage purposes it may be

necessary to take a percentage of the nameplate rating in order to account
for the fact that the equipment may be idle for a very large percentage of
time. As in the case for servers, this is the least accurate methodology
and is not recommended, but is certainly better than doing nothing at all.
An alternative method for storage power consumption measurements,

which does not require the use of nameplate ratings or actual
measurements, is to use a vendor supplied power calculator that provides
a very accurate method of determining power consumption. These
calculators do not normally take into account how the equipment is used
within the data center, but can provide very accurate power consumption
calculations based on the components that make up the system. These
include the hard drive size and rotational speed, the number of storage
processors, and the amount of cache memory. In many cases, the power
consumed by the configuration can then be derived from a vendor
provided chart that plots Input / Output Operations per Second (IOPS)
and watts for the entire array, by using a 24-hour averaged IOPS value.
When it is not possible to determine an averaged value IOP, applying a
fixed duty cycle (e.g., 60%) to the value provided from the calculator
provides a reasonably accurate estimate of the power consumption.
PDU Output
Data center floor-mounted PDUs often have instrumentation to show

instantaneous power draw. If the goal is only to determine data center
efficiency as a whole, collection of this information may be ideal.
Generally, all of the IT equipment is fed from a number of PDUs in the
space, so if the information is available at that level then the IT power
consumption calculation portion of the PUE is reasonably
straightforward.
The PDU may also aggregate power consumption over time. This
can be an asset in terms of removing variables such as workload and
utilization of individual rack-mounted devices from calculations of total
energy use. The average provided by the PDU can be examined for
time-based variability and may offer an advantage if the equipment
served by the PDU has multiple workload characteristics.
193
The total power output from the PDUs is a fairly accurate

representation of all the IT equipment use. The assumption made is that
the line losses (power wasted thru resistance losses in the power cabling)
during power delivery from the PDUs to the IT equipment are minimal
so that PDU output ≈ IT consumption. Experience indicates that these
line losses are generally less than 1 or 2% and as such a useful
measurement. The specifics of PDU power measurement were discussed
in Chapter 14. Unfortunately, the PDU values offer no visibility into
individual or specific server, storage, network gear, or component level
power consumption and will only provide a space-aggregate value.
Portable Plug-in Meters
Another option for power consumption measurement is to measure

and characterize a single server, storage array or network device and
multiply that value by the number of the same or similar devices in the
data center. The individual server can be measured using a clamp-on
ammeter or a plug-in type power meter to get an instantaneous power
reading, or that value could be recorded over time to calculate a
workload-based average energy use. A similar approach can be taken for
the storage and network equipment.
One issue with the clamp-on ammeter is the requirement to measure

only a single conductor at a time. A specially configured measurable
extension cord, with separate conductors, would need to be employed. A
plug-in meter is simpler but would also require unplugging the server to
be able to get a power consumption reading. If the server has a
redundant power supply, two meters are required. Each power supply
may be unplugged one at a time so that the power meter can be installed.
Coordination with the server owners is suggested as there is some
additional risk of an upset and installation of the meters or special power
cords during an idle period is advisable.
The extension of the measured value over a time domain or averaged

over the data center can be very inaccurate, especially if there are a
number of different server types or a number of different workloads
being run. The logical refinement of this process to measure one of each
type of server and workload for a sufficient time and weight these values,
scaled-up based upon the numbers of servers and workloads typical of
194
the data center. Being able to measure the server long enough to get a
fair time-based type average is critical (avoiding over-sampling peak
utilization or idle time), so workloads must be understood.
A simple example is the peak use in e-mail systems first thing in the
morning and right after the lunch hour. Sampling only during these
times will yield higher energy consumption than is actually used over
time. Similarly, not including these times will yield inaccurately lower
energy consumption. However, this extrapolation to get total room
energy use will not be as good as the summation the PDU provides.
If the goal is to characterize a single server, the methods above are

suitable. However, a more detailed analysis is available. The Standard
Performance Evaluation Corporation (SPEC) has published a
methodology to capture both server power consumption and performance
in a combined format. Details on SPEC power can be found at
http://www.spec.org/power_ssj2008/. Additionally SPEC provides
guidance on power meters at
http://www.spec.org/power_ssj2008/docs/device-list.html. SPEC has
collaboration agreements with The Green Grid and other industry
organizations and is a good source of additional power measurement
information.
Similar benchmarking standards are being developed for network

gear by ATIS (www.atis.org) Network Interface, Power and Protection
Telecommunications Energy Efficiency (NIPP TEE) committee. These
standards provide reasonable utilization rates for various classes of
network equipment and associated power usage at these conditions.
These values can be used to make a reasonable power use estimate for
various network devices under various load conditions.
Many different power and energy consumption meters are available

that can be used to measure the storage equipment’s power consumption
within the data center. Most storage systems utilize redundant AC input
power feeds that are fed directly from the rack PDUs, or from dedicated
AC input power connectors. An example is the power connectors for a
monolithic storage system where no other IT equipment is being fed
from these AC power feeds. In order to account for the losses which
occur when both of the redundant power supplies are operating at half
195
load and to also account for any current share differences that may exist
between the redundant power supplies, the preferred method for
measuring power is to connect a power meter in series with each of the
two AC inputs and collect an aggregate measurement over a specific
period of time (24 hours minimum).
Storage equipment is used for a variety of purposes and it is

important to understand if the measurements are being collected at a
peak workload, when the system is idle, or something in between. It
should also be noted that in order to collect power data, the meters
should be inserted in series with the incoming AC power feeds. This
process requires the data center operator to disconnect one of the two
redundant AC power sources, thereby forcing the system to run on only a
single power feed until the connections are completed and the power
supply or power system is restarted. There is always risk associated with
forcing the system to run on a single AC feed so the ideal time to insert
these meters is when the system is off-line for maintenance, or when the
system is idle and there is less chance of a disruption creating any
problems.
The next most precise methodology for obtaining the power

consumption for IT equipment is with direct measurements. These may
be taken using instrumented power strips along with the output of the
PDUs.
Instrumented Power Strips
Power strips that measure power and energy consumption and report
this data over the network provide an elegant method for obtaining IT
equipment power consumption. An example of such an instrumented
power strip is shown in Figure 15.6. The power strip can report via the
network to the data center management console, or to a number of
commercial data center management software packages. Data center
software is discussed in-depth in Chapter 4. To ensure that all IT devices
196
are accurately accounted for, it is important to have a detailed mapping

of the devices across the power strips.
Figure 15.6 - Instrumented Power

Strip
When dealing specifically with storage systems, rack mountable

instrumented power strips provide an excellent method for measuring the
power consumption. These power strips report their energy consumption
over a network connection and can be permanently mounted in the rack
for each of the storage arrays in the data center.
It should be noted that for some monolithic-type storage systems or

network systems, these power strips will not work because the array does
not utilize a standard rack design with individual power cords for each of
the rack-mountable modules.
PDU Output
As reported in Chapter 14, PDUs that integrate power over time and
report this to a Building Management System (BMS) provide an
excellent summation of the IT loads fed by the PDU. In turn, the data
acquired by the BMS can be reported up to a data center level
management system that is tracking the power consumption of all the
key subsystems in the data center.
The ideal case for measuring IT equipment power consumption is for

the equipment to self-report this information over a network. This
197
capability currently exists in many, if not most, new servers today.

Unfortunately, data centers typically have a range of equipment of
different ages. The older equipment may not have these features and as
such, to get a full set of data, the methods outlined in the minimum and
best practical measurements sections must also be implemented.
To get data from the IT equipment requires coordination with the IT

staff and data collection across the IT network. There are a number of
protocols that govern communication with different IT equipment,
including, but not limited to:
Lon,
ProfiBus
BACnet
MODBUS
GPIB
The selection of the best protocol should be made with cooperation

from the IT equipment owners. There is a wide range of management
and measurement protocols, and each of the above will support power
measurement as well as a range of other platform data, ranging from
server inlet temperatures to CPU utilization and fan speeds. There is a
wealth of data collected in each server, and integrating it with the
building management system (BMS) will allow for advances in overall
data center efficiency.
Once again, the state-of-the-art for obtaining power and energy

measurements on a storage array is the same method described for a
server and that is for the system to self-report the data over the network.
The difference being that although this is practically a standard feature
on server designs today, this is not the case for storage equipment. Most
storage suppliers today are currently developing this feature, but this is
lagging behind servers and may be difficult to find in the market.
There are also input power monitoring products available today that
can be hard wired into the AC input of the rack or array and will send
energy consumption data back to a central location using wireless
technology. These add-on modules can measure power consumption,
temperature, and humidity in the environment where the rack or storage
array is located. With enough of these wireless transmitters placed
198
within the data center, the hot spots can be mapped. It is then possible to
move the IT load around in order to balance temperatures and to reduce
the overall data center cooling requirement.
Network gear suppliers have provided self-reporting capabilities for

some time. However, more recently, software suites have been
developed that collect, analyze and report power data in a way that
makes the data more accessible to everyone. Graphical User Interfaces
and HTML formatting allow quick and easy reporting and scorecard
creation. Overall, this allows power usage data from many disparate
devices to be centrally monitored. In addition, newer devices can be
controlled to reduce energy consumption when in idle states by
implementing appropriate energy management policies, such as
powering down a device during at night or over the weekend. The ability
to easily aggregate power data from many different network devices
from a central location and in real time represents the state of the art
power measurement practice.
199
16.1 OVERVIEW
The primary function of network communications equipment

(routers, switches, etc.) in data centers is to pass digital data between
other devices (or nodes) on the network. These other devices may
include servers, storage equipment, and other communication devices,
such as wireless ports, switches and routers. Network equipment resides
alongside server and storage equipment in data center racks, sharing
power supplied by rack level PDUs, and generally operates at the same
voltage and current levels. Network devices generally contain the same
single and dual redundant power supply architectures that are commonly
found in servers, and therefore employ the same switched power supplies
found in servers. As such, network equipment is saddled with the same
power conversion losses encountered in servers.
In larger, higher data throughput network systems, such as multi-

blade fiber optic routers, power may be supplied at other voltages in AC
and DC, such as 48 Vdc.
Some network devices are now equipped with the ability to provide
power to remote devices (such as IP phones, wireless ports, etc.) directly
through the Ethernet cable, thus negating the need for 120VAC power
drops at the remote device locations. Power over Ethernet (PoE), as it is
called, is governed by the IEEE 802.3-2005 standard, which continues to
evolve to include expanded capabilities and increased total power levels.
A device that supplies power is called Power Sourcing Equipment (PSE)
and the device that is powered is called the Powered Device (PD). IEEE
802.3 recommends maximum power to be transmitted over Ethernet
conductors (currently at 15.40 W at the PSE and, due to line losses,
12.95W at the PD); however, some vendors may not comply with these
recommended standards. Consult with equipment vendors to obtain
specific PoE power values for each device.
In estimating the utilization conditions for a network device (such as

a router or switch), it is important to consider the additional power draw
imparted by remote Powered Devices (PDs) when Power over Ethernet
(PoE) is present. The total amount of additional power that a PSE draws
due to integral PoE functionality (such as a PoE enabled 48 port PoE
switch) depends how many remote PDs are connected, how much power
each PD requires and the length of Ethernet cable between the PSE and
the PD. Because cable lengths between a PSE and PD vary, it is
recommended that power measurements be taken at the PSE, not at the
PD.
PoE functionality can be retrofitted to legacy data centers by

installing mid-span PoE power injectors onto existing Ethernet cable
runs. Power injectors are small devices that introduce power into an
Ethernet cable at some midpoint between the switch and the remote PD.
These injectors are typically located near the remote PD, but, in some
cases, they are located within the data center. Mid-span power injectors
may be equipped with self-reporting capabilities including power usage
reporting, that are accessible via Ethernet cable and query commands,
such as SNMP. When power injectors do not include the self-reporting
capabilities, power must be obtained using one of the AC line power
measurement techniques described below, such as AC power clamp
meter or similar.
A typical PoE enabled PSE network device (say a 48 port switch)

may be capable of providing a maximum power of 15.4W per Ethernet
port; however, PDs may require less than the maximum power available.
When setting up PoE networks various operational parameters are
programmed into the switch, including the maximum power to be
supplied to each Ethernet port. Target power level settings for each PD
can typically be found in the PD product literature. During actual PoE
network operation, the actual power usage on each Ethernet port can be
monitored and recorded at the PoE switch. To obtain actual momentary
power and power usage over time of each remote PD, a script can be
written to log the actual power usage over time on the PoE switch. As a
reasonable estimate it is appropriate to de-rate the nameplate PoE values
by some percentage. With the power usage over time data it is possible
to accurately estimate the actual percentage de-rating of nameplate or
literature power values.
202
Networking Systems
For legacy networks (that do not include PoE capabilities) PoE can
be retrofitted to a network by installing midspan power injectors between
regular Ethernet switches and powered Ethernet devices. These small,
midspan power units inject power into the Ethernet cable conductors
without affecting the data traffic, and are powered by common AC power
adapters plugged into rack level PDUs.
This chapter will consider only the network equipment power

consumption downstream of rack level PDUs. Other areas of power
consumption in the data center are addressed in other chapters.
In network communications equipment power is consumed in the

following areas (refer to Figures 16.1 and 16.2):
 Internal power supply conversion losses

 Internal cooling fans
 Circuit board level components (varies with data traffic)
 PoE AC power adapter conversion losses
 PoE transmission line losses
203
Figure 16.1 – Example Power Usage

In Network Equipment With PoE
Switch
204
Networking Systems
Figure 16.2 – Example Power Usage

In Network Equipment with PoE
Injector
205

material.

Design Values
Network vendors provide power consumption information on

nameplates and in product literature. In most cases, especially with
newer products, power information includes power consumption at
various system utilizations levels.
The simplest way to estimate power consumption of network

equipment is to use design values provided by the equipment vendor. If
power data is provided as a maximum value, appropriate de-rating is
necessary to more accurately estimate actual power usage under various
206
Networking Systems
conditions. In estimating the utilization conditions for a device, it is

important to consider the additional power draw imparted by remote
Powered Devices when PoE is present.
The total amount of power that a PSE draws (such as a 48 port PoE
switch) depends how many remote Powered Devices are connected to the
PSE, how much power each PD requires and the length of Ethernet cable
between the PSE and the PD. Network equipment that does not have
PoE capabilities (such as a basic 48 port switch), yet has PoE midspan
power injectors installed to supply power to remote PoE devices, will not
experience any additional power draw from those PoE devices. Instead
the power for the PDs will come from the midspan power injector.
These power injectors must be included in the total network power
calculation.
Figure 16.3 shows a typical nameplate sticker for a network device,

in this case a router. Figure 16.4 and Table 16.2 show typical
information found in a router operator’s manual.
Figure 16.3 - Router Name Plate
207
Figure 16.4 - Power Information From

Product User Guide
Although they lag somewhat behind server manufacturers in

providing power consumption information, network vendors are
beginning to provide online power calculators for some equipment.
Figures 16.5 and 16.6 show examples of online power calculators for a
switch. These tools can provide a better estimate of power usage under
varying conditions and configurations (including the addition of PoE
devices) than estimating power from nameplate data.
208
Networking Systems
Figure 16.5 - Online Power Calculator
Figure 16.6 - Online Power Calculator
209
PDU Output
Network equipment is fed power from rack mounted PDUs (along

with servers and storage equipment). As mentioned in Chapter 14, these
PDUs often have instrumentation showing instantaneous and power
consumption and the means to collect power consumption over time.
While this provides an accurate measure of power consumption in each
rack, it does not provide the power consumption of just the network
equipment. Still, by combining this measured data with other estimated
data it is possible to derive power consumption of just the network data.
Portable Plug-in Meters
Similar to the direct power measurement process described in

Chapter 15, individual network equipment power can be measure using a
clamp-on ammeter or a plug-in type power meter to get instantaneous
power reading, or the value could be recorded over time to calculate a
workload-based average energy use. Similar up-time related challenges
exist for measuring power in this way on network equipment, and proper
care should be exercised in coordinating with data center management
prior to making these types of measurements. The reader is also
reminded to consider variations in loading over time when deciding how
long and over what periods to collect data. Ensure data is collected over
representative network activity and for appropriate durations. Network
traffic often varies in predictable ways, often synchronized with regular
related daily, weekly or monthly events.
The next most precise methodology for obtaining the power

consumption for IT equipment is with direct measurements. These may
be taken using instrumented power strips along with the output of the
PDUs.
Instrumented Power Strips
Power strips that measure power and energy consumption and report
this data over the network provide an elegant method for obtaining IT
210
Networking Systems
equipment power consumption. An example of such an instrumented

power strip is shown in Figure 15.6.
The power strip can report via the network to the data center
management console, or to a number of commercial data center
management software packages. Data center software is discussed in-
depth in Chapter 4. To ensure that all IT devices are accurately
accounted for, it is important to have a detailed mapping of the devices
across the power strips.
When dealing specifically with storage systems, rack-mountable

instrumented power strips provide an excellent method for measuring the
power consumption. These power strips report their energy consumption
over a network connection and can be permanently mounted in the rack
for each of the storage arrays in the data center.
It should be noted that for some monolithic-type storage systems or

network systems, these power strips will not work because the array does
not utilize a standard rack design with individual power cords for each of
the rack-mountable modules.
PDU Output
As reported in Chapter 14, PDUs that integrate power over time and
report this to a Building Management System (BMS) provide an
excellent summation of the IT loads fed by the PDU. In turn, the data
acquired by the BMS can be reported up to a data center level
management system that is tracking the power consumption of all the
key subsystems in the data center.
The ideal method for determining power usage in network equipment

is for the equipment itself to report power information over the IP
network. While many disparate building management and monitoring
protocols exist (see list in Chapter 15), Internet Protocol has emerged as
the protocol of choice going forward. As such, adapters are
commercially available to convert all of the various protocol formats to
211
IP format so that even older legacy equipment report to and can be

controlled by the management software. The management software has
an easy to use graphical user interface that allows users to customize
equipment responses to varying load conditions from central or
distributed locations. Implementing an energy management system of
this level of sophistication will most likely require the support from
system integrators and / or consultants with deep experience in the
proper deployment of such systems.
212
A.1 POWER AND EFFICIENCY
There are numerous types of pumps available. Figure A.1 below

depicts three possible pump types / configurations. A primary discussion
topic will deal with the physical means by which motor shaft power is
converted into water power. These pump types include but are not
limited to: centrifugal, rotary vane, turbine, reciprocating, gear, gerotor,
diaphragm, etc. The vast majority of pumps used for the purposes of
moving liquid based coolants throughout a mechanical chilled water
plant for a data center will be centrifugal. Additionally, this category is
further divided into a multitude of subcategories most of which, if not all,
could be found in the typical range of pumping solutions for data centers.
Figure A.1 - Three Pump Types /

Configurations
214
Appendix A—Pumps
Ultimately, all of these configurations transform electrical power into

mechanical shaft power and finally the mechanical shaft power into
water-horsepower.
Water-horsepower (WHP) is defined as:
V H
WHP (A.1)
33,000
where V is volumetric flow rate (ft3/min), is density (lb/ft3),

H is differential pressure (ft of liquid), and 33,000 is a conversion from
ft∙lb/min to water-horsepower (1 WHP = 0.7547 kW) or:
Q SG H
WHP (A.2)
3,960
where Q is the volumetric flow rate (gpm), SG is the fluid’s

specific gravity, H is differential pressure (feet of liquid), and 3,960 is
a conversion factor from gpm∙ft to water-horsepower (1 WHP = 0.7547
kW):
Not all of the impeller shaft power produces water-horsepower.

Some fraction of this horsepower results in heating of the pumped fluid
rather than its locomotion. The mechanical losses of the pump are
referred to as static efficiency pump.
This yields the equation for shaft-horsepower or brake horsepower:
Q SG H
BHP (A.3)
3,960 pump
The static efficiency, pump, of a pump is dependent upon several

parameters: pump speed, impeller trim and operating point on the pump
curve. The typical range for pump may vary from as low as 0.4 to as
high as 0.8, although wider ranges are also possible.
215
Overall system losses are further compounded by the pump motor

efficiency losses, and when employed variable frequency drive
efficiency losses. Typically, the efficiency for the motors and variable
frequency drive will be higher than 0.8 and range as high as 0.95 or
better. However, the efficiency losses are compounded, ultimately
yielding total pump power requirement for fluid transport of:
Q SG H
Power (kW ) 0.7547 (A.4)
3,960 pump motor VFD
It is easily seen that the compounding effects of inefficiencies can

quickly diminish the useful work delivered by the pump versus input
power. For example in using typical efficiencies:
pump (0.67) motor (0.85) VFD (0.95) total (0.54) (A.5)
thereby allowing as little as half of the electrical power delivered to

the pump to become working fluid power.
A.2 REAL-TIME POWER MEASUREMENTS
Figure A.2 depicts typical system and power curves for a variable
speed pump application. This curve shows a two to one reduction in
flow (50%) provides an eight to one reduction in power (12.5%). It is
worth mentioning that many systems with variable speed pumps and 2-
way flow control valves do not follow a single system curve or power
curve.
216
Appendix A—Pumps
Figure A.2 - Variable Speed Pump

Curve
217
It is desirable to understand and monitor both the electrical power

delivered to the pumping system, as well as the mechanical fluid power
derived from the input power.
Multiple use facilities serviced by a common mechanical plant add

the challenge of proportioning the power demand for common pump
loops. Consider the chilled water pump that supports fluid flow for both
general facility needs (office comfort cooling) and data center (mission
critical cooling). Figure A.3 depicts such a scenario with a common
primary loop and dedicated secondary loops. For clarity redundant items
and piping accessories have been omitted from the figure.
The primary loop has a flow of V and power consumption of PP1 that
is not fully allocated to the data center; therefore only the portion of the
flow used for the data center should be applied to PP1. The total pump
power for the data center may be obtained by the following equation.
VP 2
PP,dc ( PP1 ) PP 2 (A.6)
VP1
where VP1 is the volumetric flow rate for common loop pump P1,
VP 2 is the volumetric flow rate for data center pump P2, PP1 is the power
consumption of common loop pump P1, PP2 is the power consumption of
data center pump P2.
218
Appendix A—Pumps
Figure A.3 - Chilled Water Pump In

Multi Use Facility
The flow in the primary loop ( VP1 ) will not necessarily equal the
sum of the data center secondary loop ( VP 2 ) and the general HVAC
secondary loop ( VP 3 ). A more accurate assessment makes it also
necessary to quantify VP 3 . The following equation is more accurate as it
will also allocate pumping power consumed for the excess flow in the
primary loop apportioned to the data center:
219
VP 2
PP,dc ( PP1 ) PP 2 (A.7)
VP 2 VP3
where VP1 is the volumetric flow rate for common loop pump P1,
VP 2 is the volumetric flow rate for data center pump P2, VP 3 is the
volumetric flow rate for data center pump P3, PP1 is the power
consumption of common loop pump P1, PP2 is the power consumption of
data center pump P2.
It is not necessary to measure the power of pump P3, it is only

necessary to know this loop’s flow to allow for accurate proportioning of
common loop pump power by P1 to the various parts of the mixed-use
facility.
Figure A.4 depicts an even more complicated mixed-use pumping

environment. While again it is the chilled water circuit shown,
condenser water circuits may share similar complexity and solutions.
The primary chilled water loop is supported by three parallel pumps P1 –
P3. It is possible the pumps could be setup for N, N+1 or 2N redundancy.
To this point any combination of pumps could be running at any given
time depending upon the redundancy schema deployed. Additionally,
the secondary chilled water loop as shown is also supported by three
parallel pumps P4 – P6. The secondary loop could also be setup for N,
N+1 or 2N redundancy for the pumps. The secondary loop is split into
two circuits, one for the building HVAC needs and the other for the data
center needs. Ultimately, it is desirable to determine the portion of total
pumping power that is required by the data center.
220
Appendix A—Pumps
Figure A.4 - Parallel pumps in a

mixed-use facility
Once again, the data center’s flow requirements versus total chiller
plant must be proportioned in order to assign the appropriate pump
power requirements to the data center. This is simply the ratio of data
center flow over total flow:
Vdc 6
PP ,dc PP, Pn (A.8)
Vdc Vhvac n 1
221
where PP ,dc is pumping power attributed to the data center, Vdc is

pump flow to the data center, Vhvac is pump flow for the general HVAC,
and PP , Pn is pump power for pumps P1 through Pn.
222
B.1 VARIABLES AFFECTING RLA AND POWER

RATING
Each chiller will have a certain rated load amps (RLA) and power
rating at design conditions, for which RLA curves are typically provided.
Design conditions are the most severe set of conditions for which the
chiller and HVAC systems are sized. However, the RLA vs. power
rating graph will vary if the chiller is not operating at one of the four Air-
Conditioning Heating and Refrigeration Institute (AHRI) conditions.
These conditions can be found in AHRI Standard 550/590 – 2003.
When centrifugal chillers are selected they are optimized for a

specific set of conditions, most commonly the design conditions, and the
design conditions set the maximum RLA and power rating for the chiller.
Chillers are only one component of the chilled water system and as
conditions change in this dynamic chilled water system all components
are affected. The chiller RLA and power rating can be affected by
several factors including entering condenser water temperature (ECDW),
leaving evaporator water temperature (LCHW), condenser water flow,
evaporator water flow, fouled tubes in the evaporator or condenser,
improper refrigerant charge or a cooling tower with a fan that is out of
service.
Increasing the entering condenser water temperature will force a

chiller to work harder by creating more lift (the difference between
condenser refrigerant pressure and evaporator refrigerant pressure) on the
chiller, consuming more energy. The same is true by decreasing the
leaving chilled water temperature and the chiller RLA; power rating will
be higher given all the other set points remain the same. Water flows
through the condenser and evaporator have an affect on the chiller as
well. More flow through the condenser can help to reduce the amount of
work the chiller has to do thereby decreasing the RLA and power rating.
The entire chilled water system needs to be considered when determining
chiller efficiency, or at what conditions to operate the chiller. The chiller

will be more efficient when the ECDW temperature is lower, but this
will cause the cooling tower fans to consume even more energy and this
could make the entire chilled water system less efficient.
At design, the chiller is selected to allow for some fouling of the

evaporator and condenser tubes. Excessively fouled tubes will lower the
heat transfer of the tubes, and the RLA for the chiller may not be
representative of the RLA graph provided by the manufacturer. Each of
these issues can falsely increase or decrease the chiller RLA thereby
providing the wrong percent load efficiency and the wrong real-time
energy consumption.
B.2 INTEGRATED OR NON-STANDARD PART LOAD

VALUE
Another approach to understanding chiller efficiency is to utilize

either Integrated Part Load Value (IPLV) or Non-Standard Part Load
Value (NPLV) curves, if they can be obtained, to determine the real-time
power consumption of legacy chillers. The general belief is that these
curves apply to all load conditions. Consider the use of NPLV curves for
the determination of the real-time power consumption, with an example
curve provided in Figure B.1. At 80% load in the given example, Figure
B.1 shows that the chiller has a part load efficiency of 0.49 kW/ton, as
opposed to 0.44 kW/ton originally determined in the example given in
Chapter 9 (See section 9.2.1). In turn, the real-time power consumption
of the chiller is calculated as 0.49 kW/ton * 800 tons = 392 kW, which is
11.4% higher than the value originally calculated in Chapter 9 of 352 kW.
224
Appendix B—Chillers
Figure B.1 - Load Efficiency (kW/ton)

at NPLV (Non-Standard Part Load
Values)
A source of error related to the NPLV curves is that they do not

account for events that may increase or decrease the input power (kW) to
the chiller. For example, these graphs assume a certain fouling factor in
the tubes of the evaporator and condenser and AHRI relief. If the fouling
factor is greater than that assumed by the curves, then the chiller may be
consuming more power (kW) to get the same capacity as compared to
clean tubes. But more importantly the NPLV or IPLV curves are only a
snap shot of the chiller performance at four specific points. If the chilled
water system is not operating exactly at one of these four specific points,
then the power rating will only be an estimation of the actual power
consumption at best. Finally, AHRI conditions only account for loads
between 25% and 100% therefore if the chiller is operating at a load less
than 25%, then Figure B.1 is not useful.
A limited number of approaches to arriving at the real-time power

consumption of legacy chillers have been provided. The chiller expert is
well aware that there are in fact numerous approaches to solving this
problem. The bottom line is that the data center operator needs to
225
determine the legacy chiller’s real-time capacity and determine percent-

load, which can be used with the chiller’s performance curves (similar to
Figure 9.5) to determine the real-time efficiency and power consumption.
226
C.1 REAL-TIME COOLING TOWER POWER

CONSUMPTION
It is common for data centers to be incorporated into multi-use

facilities where the chiller plant is supplying chilled water to the data
center as well as to a retail / office space. Figure C.1 shows a typical
chiller plant with five chillers and four open cooling towers.
Figure C.1 - Schematic layout of a

chilled water plant for a mixed-use
facility
228
Appendix C—Mixed-Use Facilities
The open cooling tower power consumption (kW) for a multi-use

facility with multiple cooling towers can be calculated as follows:
(Pct1 + Pct2 + Pct3 + Pct4) · Qdc

Pct,dc = (C.1)
Qtotal
where Pct,dc is the total cooling tower plant power consumption

attributable to the data center, Pct1…Pct4 is the power consumption of the
individual cooling tower fan motors, Qdc is the total heat load from the
data center, and Qtotal is the total heat load from the mixed-use facility.
Since open cooling towers have been assumed, there are no integral
pumps to include in the total power consumption calculation. The power
consumption for the condenser water pumps is discussed in Chapter 7. If
closed circuit cooling towers had been assumed, the pump power
consumption per cooling tower would have to be included in the total
power consumption for the cooling tower plant. If there is interest in
knowing the power consumption of individual cooling towers
attributable to the data center, the data center owner / operator should
measure condenser water flow rate to each cooling tower. This specific
case is not covered in the book.
Once the overall power consumption of the cooling tower plant Pct,dc ,
attributable to the data center has been determined as illustrated by
Equation C.1, this data can then be used for any number of purposes.
For example, the Pct,dc can be made available to the facility software for
use in the calculation of the PUE. While PUE is used as a particular
example in this book, the point to be made is that the data can be used in
any industry data center energy efficiency, productivity, or other metric.
C.2 REAL-TIME CHILLER POWER CONSUMPTION
Mixed-use data centers present a challenge when it comes to

quantifying the total power consumption attributable to the data center
itself. This stems from the fact that when a given subsystem, such as a
chiller plant, receives heat loads from multiple sources, the total power
consumption for the chiller plant is attributable to these multiple sources.
This section describes how to separate the chiller plant’s power
229
consumption that is attributable to the data center. Keep in mind that the
heat load originating from the data center, as transported in the returning
water, is the sum of the heat loads from the IT equipment, heat losses
from the CRAHs, heat losses from the lighting, and several other key
sources within the data center. For this section, all these sources are
lumped into a single source from the data center (i.e., total heat load of
the entire data center).
Shown in Figure C.1 is a graphic representation of a data center that

uses five chillers in its chiller plant. For clarity, the five chillers are
shown as a single component labeled ― Five Chillers‖. The figure shows
the schematic plumbing layout, the pump locations, and the connection
to the cooling towers. The figure also shows the points at which the
water flow rate (supply only) and temperatures (supply and return) are
measured. The load on the chiller plant can be calculated from this
measured data as follows:
Qtotal = m
 total · Cp · (EEWT – LEWT) (C.2)
where Qtotal is the total heat load on chiller plant, m total is the
measured mass flow rate of chilled water calculated from volumetric
flow rate, Cp is the specific heat (constant pressure) of evaporator
(chilled) water, EEWT is the entering evaporator water temperature, and
LEWT is the leaving evaporator water temperature.
The total heat load, i.e., Qtotal, consists of the sum of the heat loads
from all sources within the mixed-use facility, including from the data
center itself. For example, loads could originate from the data center
(Qdc), an adjacent laboratory (Qlab), office space (Qoffice), etc. Once these
heat loads combine into a single mass flow rate of water ( m  total ), the
mixture will attain a temperature that can be named Ttotal. The heat load
from the data center that returns to the chiller plant will be split into five
separate water streams (to the five chillers), assumed to all be at the same
temperature Ttotal. The data center heat load is represented as:
Qdc Qdc,c1 Qdc,c 2 Qdc,c 3 Qdc,c 4 Qdc,c 5 (C.3)
230
Appendix C—Mixed-Use Facilities
where Qdc,c1…Qdc,c5 are the heat loads from the data center, split
between chillers 1 through 5 respectively.
It is assumed that the temperature of the water entering each chiller

is the same, i.e., Ttotal. Note that in this case, Ttotal = EEWT. The portion
of the total heat load from the data center, delivered to any chiller, can be
calculated. Consider the case of chiller 1:
Qdc,c1 m dc,c1 C p (EEWT – LEWT) (C.4)
where m
 dc,c1 is the proportion of total mass flow rate from the data
center passing through chiller 1.
The total water mass flow rate for the mixed-use facility, m  total ,
splits into five separate mass flow rate streams corresponding to the five
chillers (i.e., m
 dc,c1 through m dc,c 5 ) respectively. The mass flow rate to
each chiller is driven by a pump that serves each chiller, and it is
assumed that these flow rates are measured. Each chiller will also
receive a contribution from the total mass flow rate of water returning
from the data center (i.e., m
 dc ). It is assumed that this flow rate is
measured as well. The contribution for chiller 1, from m  dc , is calculated
as the ratio of the chillers mass flow rate times the mass flow rate of the
data center to the total mass flow rate of the facility, as follows:
m c1 m dc
m dc,c1 (C.5)
m total
The contributions to chillers 2 through 5 are calculated in the same

manner as shown in Equation C.5.
Once the fraction of the total heat load passing through a given
chiller, attributable to a source such as the data center, is determined, the
fraction of the chiller’s total power consumption attributable to the data
center can be calculated as:
231
Pdc,c1 Qdc,c1 c1 (C.6)
where Pdc,c1 is power consumption by chiller 1, attributable to the

heat load from the data center (tons) and c1 is the efficiency of chiller 1
(kW/ton).
The efficiency for the chiller is determined as described in Chapter 9.

Equation C.7 can be used to calculate the separate power consumptions,
attributable to the data center, for chillers 2 through 5. The total chiller
plant power consumption, attributable to the data center, is then found
from:
Pdc Pdc,c1 Pdc,c 2 Pdc,c 3 Pdc,c 4 Pdc,c 5 (C.7)
where Pdc is the total chiller power consumption attributable to the

data center and Pdc,c1…Pdc,c5 is the power consumption by chillers 1
through 5 (respectively) attributable to the data center.
The analysis above presents one approach to quantifying the portion

of the chiller plant’s power consumption associated with the data center.
There are, of course, additional ways to do this. The best approach for a
given data center will be driven by the available level of instrumentation
and the real-time data that is available.
232
A UPS is an electrical power system that receives power from the

electric utility or local on-site generators. It has two functions: (1) under
normal operating conditions, the UPS supplies precise power to critical
IT loads within the data center; and (2) upon loss or severe degradation
of utility power, it supplies uninterrupted power to the IT equipment until
its energy storage is depleted or until main power is restored from the
utility or generators. Incorporated within the UPS function is an energy
storage device such as a battery system. This energy storage device
allows the UPS to continuously provide power for some load-related
period of time, even with the total loss of electrical power to its input.
This energy storage can be used to provide enough time to start an
engine-generator or provide enough time for an orderly shutdown of the
critical operations. The ride-through time may range from 5 minutes to
hours, but typically is in the 10 to 30-minute time frame.
The UPS can be a significant contributor to the total critical power

path data center power consumption. Although state-of-the-art UPS
modules are efficient at full load (typically 93 – 97% between half load
and full load), legacy UPS system efficiency losses can be significant at
lower loads. Inefficiency can be compounded in some redundant
configurations in which UPS modules must operate at low or very low
percentage of capacity by nature of the system design. Inefficiency
becomes a significant factor of loss within the critical power path and
adds to the overall electrical load demand for the center. In addition,
because this loss appears as heat, it increases the air conditioning
requirements, thereby further driving the electrical needs. As noted, the
power ratings of the UPS and other critical power path infrastructure are
highly leveraged by the IT loads, which consume about 30-50% of the
total data center electrical usage. Figure D.1 shows that losses associated
with the critical power paths can represent a significant amount of the
total data center power consumption. IT equipment, PDUs, and UPS,
which are all in the critical power path, account for about half of total
power consumption. Although the values in Figure D.1 can vary greatly
from one data center to another, the relationships are valid. A difference
of 1 watt at the IT load can equate to more than 2.4 watts at the facility’s
electrical service entrance.
An automatic static bypass switch is included in the UPS

configuration to automatically and seamlessly transfer the critical load
input to an alternate source (utility or generator) if the UPS fails. The
bypass significantly improves the availability of power to the IT load’s
input by a factor of 6 to 10 and permits seamless transfer to a
maintenance bypass.
Figure D.1 - Schematic

representation of data center total
power consumption
D.1 TECHNOLOGY
Performance of the UPS will vary with the underlying technology.

Choice will depend upon the owner’s assessment of many factors
234
Appendix D—Uninterruptible Power Supply (UPS)
including degree of risk tolerance, criticality of the IT operation, space

available, floor loading, environmental conditions, installed cost,
operational cost, availability, and vendor relationships to name only a
few.
By far the most common type is the on-line static (solid state) UPS,
although it should be noted that other technologies are used. These
include standby UPS, line-interactive UPS, rotary UPS, hybrid rotary /
static UPS, and hybrid rotary / generator systems. Because of its
overwhelming dominance in the industry, this document will focus on
the static online UPS.
Figure D.2 shows a simple representation of an on-line solid state

UPS, commonly called a double conversion UPS.
Figure D.2 - Simplified single-line

diagram - solid state UPS system
235
Because power is typically distributed as alternating current (AC) at

50 Hz or 60 Hz, and battery systems create and require charging with
direct current (DC), on-line double conversion UPS convert utility power
to DC to charge the batteries and then convert it back into AC in order to
provide seamless transfers. Other types do not convert all power to DC.
D.2 REDUNDANCY AND AVAILABILITY
Realistically, most data centers in the range of our discussion would

not have a stand-alone, non-redundant UPS system as depicted in Figure
D.2. UPS modules would typically be in parallel for output capacity,
redundancy, or both.
D.2.1 PARALLEL FOR CAPACITY (NO

REDUNDANCY)
Two or more UPS power modules can be in parallel to achieve a

higher kVA system rating. For example, two 500 kVA modules could be
parallel to achieve a 1000 kVA system rating. This is usually done to
achieve a system rating higher than the manufacturer can provide with a
single module. Such configurations are uncommon because the increase
in modules without redundancy increases the probability that a single
module failure could cause the entire UPS system to fail.
D.2.2 PARALLEL FOR REDUNDANCY OR

BOTH CAPACITY AND REDUNDANCY
Figure 11.1 showed a simplistic system with a single critical power

path without redundancy. An alternate design could use parallel systems
fed by separate power modules. If in parallel for capacity, failure of
either power module could cause the system to become overloaded and
fail. If in parallel for redundancy, in this example each module carries
approximately 50% of the connected load, potentially resulting in a
lower efficiency per module. If either one of the power modules fails,
236
the remaining module picks up the faulted module’s share of the load in
one step (in this case, from 50% to 100% of the load).
Figure D.3 shows a more realistic (although simple) scenario for a

large data center in which multiple systems can exist and interconnect.
In this example, there are two N+1 systems paralleled for redundancy. In
normal conditions, each represents a critical power path. Cross ties at
various points in the system (not shown) allow elements of one system to
switch into the critical power path of the other system for maintenance or
repair. The PDUs distribute power to IT equipment racks. IT equipment
(such as a server) can either deploy a single cord (e.g., server rack(s) 1)
or dual cords (e.g., server rack(s) 2 & 3). For racks with single cord
devices, a rack automatic transfer switch (ATS) can be provided to allow
operation from either critical power path. A dual cord device is always
connected to both sources with each source providing approximately
50% of the power. This is a better design because each source supports
half the load and then each incurs only a 50% increase in connected load
in the event of a failure on one of the sources. As can be seen, the
introduction of redundancy throughout the system can make it very
challenging to determine the location of loads in a power path at any
given moment. Data center power consumption is most easily measured
at each input switchgear.
237
Figure D.3 - 2(N+1) UPS configuration
238
D.2.3 UPS Efficiency Considerations For

Parallel Systems
Because the point of optimum efficiency can vary between models of

UPS power modules, the impact of redundancy on efficiency must be
considered. Figure D.4 illustrates the wide range of performance for
different technologies. Newer designs are optimized at lower loads,
whereas many older systems were optimized near their full rated capacity.
State of the art may exhibit higher efficiency than shown on the graph.
Figure D.4 - Legacy UPS system

distribution efficiency versus load
239
Example
Most UPS systems run somewhere below 80% of their non-
redundant rating. Assume a system is rated for 100 kW [N] with one
additional module for redundancy [N+1] and it is operating at 80% of its
full rated capacity (i.e., 80 kW). Table D.1 shows the hypothetical effect
on system efficiency with variations in the quantity and rating of parallel
power modules.
Table D.1 demonstrates the importance of ― right sizing.‖ A system

should be designed so that power modules operate as closely as possible
to their optimum efficiency. Multiple small modules may operate more
efficiently than fewer large modules. Ideally modules can be easily
added or deleted as required to achieve maximum efficiency. Not shown
in the table is the size of the load step that would be added to each of the
remaining modules if one module is taken down. The higher the quantity
of modules, the smaller the load step and the smaller the possible
disturbance on voltage regulation. Other considerations, such as initial
cost, maintenance, and reliability must also be weighed against operating
cost, which are not part of this discussion.
240
D.3 RULES OF THUMB FOR MINIMUM PRACTICAL

LEVEL OF UPS INSTRUMENTATION
It can be assumed that with a best practical level of instrumentation

or state-of-the-art level of instrumentation, measurable and accurate data
will be available for determining the distribution of losses within the
critical power path. The same data would also show the operating
efficiencies for each of the series components. However, in this example
it is assumed that for the minimum practical level of instrumentation
only the UPS output power is measurable. Therefore, to estimate the
operating losses of the critical power path elements, representative
efficiencies must be used to obtain a broad look at where the losses occur.
When it is available, one should use manufacturer’s data such as
efficiency versus load curves. If such data is unavailable, then one must
resort to a rule of thumb alternative as described in this section. Rules of
thumb are clearly less accurate than using manufacturer supplied curves,
but they can quickly provide a first pass view at the losses within the
critical power path.
Keep in mind that each of the series elements within the critical
power path will have losses that do not vary in a linear fashion with load
level. Also keep in mind that IT loads are dynamic and change in value
with respect to time. Therefore, this technique only takes one snapshot
in time and only provides a relative performance profile for which
significant inaccuracies are possible.
Following are two examples that illustrate rule of thumb calculations

of the power distribution losses.
Example 1: Power distribution loss calculations
IT Load PUPS , n IT (D.1)
where PUPS,n is the output power of each UPS from 1 to n, and IT is

the efficiency of the power distribution system. According to Table D.2,
IT = 0.98 without a transformer or 0.965 with a transformer.
241
Key assumptions include that all IT loads are on UPS power, the
UPS output power is metered, the UPS batteries are fully charged, and
efficiency of the power distribution is an average value.
Example 2: Power distribution loss calculation
PUPS ,n
Service Entrance Input Power (for Critical Power Path) =
UPS IT
(D.2)
where UPS is the average efficiency for the UPS, IT is the average
efficiency of the total IT load and PUPS ,n is the total UPS power.
A key assumption is that all UPS are operating between 40% and
80% of capacity.
Using the efficiency values provided in the table and placing these in
Equation D.1, the following result is obtained:
1
IT Load PUPS,n 1.11 PUPS ,n (D.3)
0.92 0.98
The rule of thumb scenarios are summarized in Table D.2.
242
D.4 SAMPLE CASE STUDY: A PARTIAL PUE AND

DCIE DETERMINATION FOR THE CRITICAL
POWER PATH WITHIN THE DATA CENTER
The following is an example of how to calculate a partial Power

Usage Effectiveness (PUE) and a partial Data Center Infrastructure
Efficiency (DCiE) for the critical power path in a hypothetical data
center.
(Note: This example addresses only the critical power path; it does
not include other elements such as cooling equipment in the essential
power path that are addressed in other chapters. Whereas the PUE
considers all power-using components within a facility, a "partial" PUE
considers only the power-using components within a boundary. A
boundary could be a physical boundary such as a container, a computer
room, a modular pod, or a building. A boundary could also be a logical
boundary such as equipment owned by a department or a customer, or
owned versus leased equipment, or any other boundary that makes sense
for the management of the assets. In that sense, the power consumed by
the IT equipment compared to the power consumed by all of the
elements in a critical power path would constitute only a partial PUE.)
This example assumes that the data center has more metering than
would be expected with a minimum level of instrumentation. (Note:
Elements of the essential power path are described elsewhere in this
document and are not included in this calculation.)
243
Figure D.4 - Measuring points for

PUE or DCiE methods of efficiency
calculation
244
For this example, the critical AC power path starts at the electrical
distribution panels. Instruments at these panels measure and display
current, voltage and power in each feeder to the respective downstream
UPS systems. Each UPS further connects to a downstream PDU, from
which originate the branch circuits to the IT loads.
The IT load consists of ten computer racks. Two separate 208/120

volt three-phase AC power paths (A & B) are distributed to each rack via
separate branch circuits to rack-mounted power distribution units
(RPDU), so that each rack has two RPDUs. Dual-corded IT equipment
(e.g., servers) connect one cord to source ―A‖ and the other cord to
source ― B.‖
With this information we can calculate both Data Center

infrastructure Efficiency (DCiE) and Power Usage Effectiveness (PUE).
Note that measurements at all points in the critical power path must be
taken as nearly simultaneously as possible. This is possible with
automated, real-time measurement. Manual measurements may require a
team of people in order to minimize the time required to gather data.
STEP 1 - Measure input power to the data center
This can be obtained by careful measurement at the electrical

distribution panels, ensuring that the power supplied only to the critical
power path is included in the measurement. In this example, there are
four feeders to each UPS. Power measurements in watts are assumed to
be displayed on meters for each feeder, and are tabulated in Table D.3
below:
245
STEP 2 - Calculate the total IT load power.
This is most easily accomplished by taking power measurements

from the output of the PDUs and aggregating them. The PDU meter
accuracy is typically more precise than the RPDU meter accuracy. If, for
some reason, that is not possible, or if the user wishes to know the exact
power from each source at each rack, power consumption can be
measured at the RPDU. The data in Table D.4 represents measured
currents at the rack RPDUs and calculated power. Note that in this
example current is measured but power is calculated using available data.
In an actual data center, power (watts) might actually be displayed on a
meter, thereby negating the need to calculate the power from the current
measurements. Power must be calculated for each source (A + B) and
aggregated to get the true power consumption.
Power (W) = Σ(V · I · 3)

= (208 · IA · 3 ) + (208 · IB · 3) (D.4)
where V is the three-phase voltage and I is the current for each

source.
246
STEP 3 (Optional) - Calculate RPDU losses.
In most cases the efficiency of the RPDU itself is considered to be an

integral part of the IT load. If one wishes to count RPDU as part of the
infrastructure losses, it may be necessary to use calculated losses
provided by the RPDU manufacturer.
For this example, the IT load will be adjusted to obtain just the
aggregate computing losses, assuming an RPDU efficiency of 98.5%.
Therefore:
Net IT Load Server Rack PDU Load RPDU Efficiency (D.5)
247
Net IT Load 66,354W 0.985 65,359W
STEP 4 - Calculate Critical Power Path PUE
By taking the reciprocal of DCiE, the Power Usage Effectiveness

(PUE) can be calculated. In this example:
1
PUE(CPP ) (D.6)
DCiE (CPP )
where PUE(CPP) is the Power Usage Effectiveness for the Critical

Power Path.
1
PUE(CPP ) 1.152
0.8678
STEP 5 - Calculate DCiE
For this example, a reading was obtained of 76,470 watts of three-

phase power at the input to the data center and 66,354 watts were
consumed by the IT loads.
Total IT Load
DCiE (CPP ) 100% (D.7)
Critical Power Path Input Power
where DCiE(CPP) is the Data Center Infrastructure Efficiency for the

Critical Power Path.
66,354W
DCiE (CPP) 100% 86.78%
76,470W
Further analysis or breakdown of element efficiencies within the

critical power path including the UPS and power distribution elements
can be conducted. This example just provides an overall value of PUE
for the entire critical power path.
248
E.1 OVERVIEW
Onsite power generation has the potential to benefit a data center in

numerous ways. The most direct method is by reducing high energy
costs by eliminating or reducing the demand for power from the
electrical grid. Although there is a cost associated with any onsite
generation methods, overall energy cost savings can be achieved by
reducing the peak demand from the grid, which can often result in
reduced peak usage fees and ratchet charges. A ratchet charge is a utility
rate provision under which the demand charge for each month (or other
period) is based on the highest measured demand (or its percentage) over
the previous year (or other period). Depending on the electrical
generation method, peak shaving (reducing peak demand) or base
loading (constant electricity generation) schemes may be appropriate.
Onsite power generation can be categorized into two types; methods

that are stand-alone or methods that can be part of a Combined Cooling
Heat and Power (CCHP) system. The first type is normally associated
with Green technologies, or power generation methods that do not utilize
fossil fuels. Alternative energy sources include solar, wind,
hydroelectric, and geothermal. These environmentally friendly power
generation technologies have advanced in recent years to the stage where
implementing these systems can be cost effective, with paybacks in just
years instead of tens of years.
In addition to the local economic and data center infrastructure

benefits that can be obtained, the displacement of inefficiently generated
power off the utility grid provides real environmental gains in the area of
emissions and reduces the burden on natural reserves.
The second category of power generation technologies all generate

heat as a byproduct, and can therefore be used with a CCHP system.
CCHP refers to a system that couples power generation equipment with
equipment designed to do additional work using the leftover heat energy
from the electrical generation process. These systems are located at the
point of use and are a subset of the industry category of Distributed
Generation (DG) or Distributed Energy (DE). In data center applications,
CCHP systems not only generate electricity for use, but also use the
remaining heat for water-cooling of the data center, and may include
heating and steam generation components for use outside the data center
and for dehumidification.
CCHP systems can utilize prime movers such as gas turbines, gas
engines, diesel engines, and microturbines to power generators. These
technologies can be used with a number of different fuel sources,
providing the opportunity to optimize the availability, cost, as well as
performance of the power generation fuel to suit the needs of the data
center facility.
From an input power perspective, CCHP systems directly displace

power provided by the utility grid while simultaneously displacing the
electrical load associated with the chiller systems being displaced by the
cooling capacity of the CCHP system. In a perfectly sized system, the
cooling output of the CCHP system will be sized to eliminate 90% to
95% of the electric power required to run conventional cooling
equipment and will displace about 50% to 55% of the IT load, meaning it
will not be metered at the utility grid meter. Figure E.1 illustrates a
typical application of CCHP in the data center.
Onsite power generation alone does not affect the calculation of PUE
or DCiE. These metrics are based on electrical power consumption, and
are not biased by how the electricity is being supplied. Regardless of the
source, the power consumed by the IT equipment as well as the power
consumed by the data center facility (counting the power generation
outside of the facility) both remain the same.
However, when calculating PUE and DCiE for a CCHP system, a

non-electric energy source is used to operate an absorption chiller. The
reuse of what would otherwise be waste heat improves the PUE and
250
Appendix E—Onsite Power Generation and CCHP in Data Center Applications
DCiE ratios by reducing the total metered power into the data center by
the following equation:
Pdp = Pgen + Pcool (E.1)
where Pdp is the total power displaced by CCHP at the utility meter,
Pgen is CCHP electric output power as metered at the combined output
point (collector bus) of the CCHP system, and Pcool is displaced cooling
power (power displaced by the absorption chiller providing cooling to
the data center).
E.2 CCHP
Key to all applications for CCHP is using the exhaust heat for useful
purpose. A data center is an excellent application since there is a
relatively constant thermal load requirement.
Sample Data Center Application of CCHP
This section introduces a real-world application of CCHP to the data

center environment, introducing critical design guidelines for maximum
efficiency and availability as well as serving as the basis for the follow-
on instrumentation discussions.
A CCHP system used in the data center could be set up with grid
independent capability allowing it to not only provide economic benefit
(return on investment), but to also operate in the event of a grid outage,
just as an emergency generator set would. The CCHP system could also
be used as an extra layer of redundancy in addition to a conventional
diesel generator(s). When configured in this fashion, the system can
sustain the combined power and cooling requirements of IT equipment
indefinitely, provided there is no interruption in CCHP fuel supply.
Figure E.1 illustrates how a CCHP system would be used in this

configuration. This layout is microturbine based but can essentially use
any CCHP prime mover technology. The CCHP system serves a data
center with a UPS and electric driven cooling. The bold lines represent
251
paths of normal or possible power flow. This design requires inclusion

of a Dual Mode Controller (DMC).
Figure E.1 - CCHP schematic for data

center operation
The DMC is a device that seeks to remain closed at all times when
the grid is present. This allows the microturbines to flow surplus power
to the entire building while the CCHP system is running in parallel with
the grid. In this mode the system is running in a maximum, base-loaded
condition. If the grid goes down, the DMC will immediately open the
252
connection to the grid. This is a system safety interlock and protection

feature that prevents inadvertent power flow back to the grid. For a brief
period of time, the data center operates on UPS, as the CCHP system
resets itself to run independent of the grid. Once reset, the CCHP system
feeds power to the critical data center loads.
The automatic transfer switch (ATS) logic is set up so that the CCHP
system is the normal source. To prevent a false start of the diesel, a brief
time delay is introduced into the logic of the transfer switches. After the
delay, if the CCHP system is not online, the transfer switch will close its
starting contacts activating the diesel backup system.
A key element to a successful design of the cooling plant is the use

of a hybrid chiller plant. The hybrid chiller plant includes both electric
driven chillers and waste heat driven chillers in combination for
reliability and energy savings. The layout of the equipment will depend
on loading, rate structure and system requirements. Typical decoupled
systems can be utilized as well as side stream applications and series
flow applications.
The series flow arrangement can provide additional energy savings

over normal operations by reducing the inlet temperature of the chiller
water to the electric chiller. This arrangement can provide additional
savings if the electric chiller is sized to provide full load without an
absorption chiller but is operated most of the time in a part load
condition. This provides not only superior energy savings but also
reliability.
Absorption Chillers
Absorption chilling is a mature, stable technology that uses heat

instead of mechanical energy to provide cooling. The single-effect
absorption chiller system consists of an evaporator, an absorber, a
condenser, a generator, and a solution heat exchanger. Water is typically
used as the refrigerant in vessels maintained under low absolute pressure
(vacuum). In cooling mode, the chiller operates on the principle that
under vacuum, water boils at a low temperature. Under typical operating
conditions, this occurs at approximately 40 °F (4.4 °C), thereby cooling
the chilled water that circulates through the evaporator tubes. A
253
refrigerant pump is used to spray the refrigerant water over the

evaporator tubes to improve heat transfer. A single-effect absorption
chiller system diagram is shown in Figure E.2.
Figure E.2 - Schematic representation

of a single-effect absorption chiller
system
254
In recent years, double-effect absorption chillers have been widely

adopted. These use two generators (one low temperature, the other for
high temperature) and two solution heat exchangers for improved
efficiency. To make the cooling process continuous, the refrigerant
(water) vapor must be removed as it is produced. To accomplish this, a
lithium bromide solution, which has a high affinity for water, is used to
absorb the water vapor. As this process continues, the lithium bromide
becomes diluted, reducing its absorption capacity. A solution pump then
transfers this weak (diluted) solution to the generators where it is re-
concentrated to boil off the previously absorbed water. The solution
pump maintains optimum solution flow to the generators at all operating
conditions for maximum efficiency. The diluted solution is pumped to
the high-temperature generator where it is heated and re-concentrated to
a medium concentration solution. The medium concentration solution
from the high-temperature generator flows to the low-temperature
generator where it is heated and re-concentrated to a strong solution by
the high temperature water vapor released from the solution in the high-
temperature generator. The water vapor released in the shell side of the
low-stage generator, in addition to the now condensed water vapor from
the tube side of the low-stage generator, enters the condenser to be
cooled and returned to a liquid state. The refrigerant water then returns
to the evaporator to begin a new cycle.
To remove heat, relatively cool water from a cooling tower or other

source is first circulated through the tubes of the absorber to remove the
heat of vaporization. The water is then circulated through the tubes of
the condenser. The strong (re-concentrated) solution from the low-stage
generator flows back to the absorber to begin a new cycle.
Steam Generation
Many of the prime movers used in combined heat and power systems
can generate steam through the use of a heat recovery steam generator
(HRSG). Exhaust gas from microturbines, gas turbines, reciprocating
engines and molten carbonate fuel cells can be used to drive the HRSG
to generate high quality steam. This steam is used to provide
humidification in the data center or to provide steam heating within the
facility surrounding the data center. Where steam absorption cooling is
255
already in place, this steam can be used to displace the high cost of site-
made steam for cooling. Larger CCHP systems can provide additional
electrical power through a steam turbine supplementing the main prime
mover.
Desiccant Dehumidification
The waste heat from the electrical generation process is also used to
provide regeneration of desiccant wheels used to dehumidify air streams.
This is especially applicable in applications where the outside conditions
introduce moist air into the data center or supporting facility. The loads
from the IT equipment are sensible loads, but outside air can contain a
latent component to meet ventilation requirements. Desiccant
dehumidifiers can be used to dry this air before being introduced into the
data center environment. New desiccant designs include reactivation of
the wheel with medium temperature water (140 – 180 °F [60 – 80 °C]).
CCHP Operation Schemes
In normal power loading conditions, the CCHP system is loaded to

its maximum efficiency point in relation to both electricity and thermal
load. The facility utilizes electric grid power in excess of constant
CCHP base power to meet the customer power demand, referred to as
base loading (see Figure E.3). In this case, the electric and thermal
outputs are optimized to preserve overall system efficiency.
256
Figure E.3 - Example of base loading

with a CCHP system
The CCHP system may also be used for peak shaving during periods
of the day when electricity from the utility is at a premium. This mode,
known as Load Following, allows the user to selectively determine start /
stop commands and / or power output levels, depending on the
technology of the prime mover. When control is based on a utility input
power set point, the system regulates the utility power flow to an
adjustable maximum: the utility power set point. If the local demand
rises above this level by an adjustable amount for a set time, the CCHP
system is directed to supply the difference, up to its capacity. The goal
of the peak shaving mode is to minimize peak and standby electricity
charges (see Figure E.4).
257
Figure E.4 - Example of peak shaving

with a CCHP system
E.3 MEASUREMENT LEVELS
Table E.1 gives an overview of the minimum practical, best practical,

258
E.3.1 Minimum Practical Measurement
The minimum practical measurement requires measurement of the

power (kW) at the CCHP system output breaker to the data center (kWh
meter recommended), the input and output evaporator chilled water
temperatures (ECHW and LCHW), and the evaporator chilled water flow
rate. The minimum requirement will be to manually record these values
during steady-state conditions at fixed time intervals.
F
igure E.5 is essentially Figure E.1, but with the key power metering
points indicated. Similarly, Figure E.6 is a modified version of Figure
E.2, with the key metering points indicated. More specifically, Figure
E.5 indicates the power measurement points in the power distribution
schematic, while Figure E.6 shows the measuring points for determining
the power displaced by the CCHP system.
259
Figure E.5 - Power instrumentation

points
260
Figure E.6 - Cooling instrumentation

points
261
E.3.2 Best Practical Measurement
The best practical measurement requires installed sensors for

measurement of power at the CCHP system output / input breaker to the
data center, input and output absorber chilled water temperatures, and
absorber chilled water flow in order to calculate the revised displaced
power (Pdp). The best practical measurement will be to automatically
record these values during fixed time intervals.
The power displaced by the CCHP will be calculated by determining

the temperature compensated power usage per ton of chilled water
produced for the average installed electrical chiller, multiplied by the
actual tons of chilled water being produced by the CCHP system. This
allows the PUE to be calculated in real-time.
E.3.3 State-of-the-Art Measurement
The state-of-the-art measurement requires factory installed sensors

for measurement of power at the utility / data center input, power at the
CCHP system output / input breaker to the data center, input and output
absorber chilled water temperatures, absorber chilled water flow, and
ambient temperature in order to calculate the revised displaced power
(Pdp). The state-of-the-art measurement will be to automatically record
these values during fixed time intervals and automatically calculate Pdp
based on the data collected.
The software system will be programmed with OEM performance

curve data for the chillers that are being replaced by the CCHP system.
The power displaced by the CCHP will be calculated by determining the
temperature compensated power usage per ton of chilled water produced
for the replaced electrical chiller, multiplied by the actual tons of chilled
water being produced by the CCHP system. The net electric power of
the CCHP system, Pcool, may be subtracted from the total power
displaced by CCHP to derive the CCHP net improvement in kW. This
amount of kW may then be used in conjunction with algorithms as
described above to calculate the benefit of CCHP to PUE on both an
instantaneous basis and on a totalized basis.
262
E.4 EXAMPLE CALCULATIONS FOR A CCHP

INSTALLATION
While the sensors and software necessary to acquire and reduce the
real-time power consumption measurements for CCHP systems may be
discussed in this Appendix, the details with respect to the sensors and
software are left to Chapter 4 Measurement Collection Systems –
Architecture & Software, and Chapter 3 Measurement Devices.
The primary objective of this Appendix is to educate data center

owners / operators on real-time power consumption measurements for
CCHP systems. With this primary objective in mind, it is instructive to
provide an illustration of one of the uses of the real-time power
measurements. One such illustration is provided with the calculation of
an energy efficiency metric such as the PUE. The PUE has been defined
in Chapter 2 How What & Where To Measure, and more specifically in
Equation 2.1. Some owners / operators are also using the DCiE, which is
given as DCiE = 1/PUE. The power consumption for a facility that has a
CCHP system installed is given by:
Pfac,CCHP Pfac Pdp Pgen (E.2)
and
Pdp Pgen Pcool (E.3)
where Pfac,cchp is facility power consumption measured at the utility

meter after the installation of CCHP, Pfac is facility power consumption
before the installation of CCHP, Pdp is the total power displaced at the
utility meter by CCHP, Pgen is power generated by the CCHP system as
metered at the combined output point (collector bus) of the CCHP
system, and Pcool is power not consumed by the chillers due to the
presence of the absorption chillers that are part of the CCHP system.
The PUE of the data center will change after the installation of a
CCHP system, so this book will differentiate between the PUE for
263
facilities with and without CCHP systems. The PUE for a facility after
the installation of a CCHP system is given by:
Pfac Pdp Pgen Pfac,CCHP

PUECCHP (E.4)
PIT PIT
The improvement in PUE, after the installation of a CCHP system, is

given by:
PUE PUECCHP
Percent Change in PUE (E.5)
PUE
If Pdp Pgen Pcool is used in Equation E.3, the following is

obtained:
Pfac – (Pgen + Pcool) + Pgen Pfac – Pcool

PUECCHP = = (E.6)
PIT PIT
This shows that PUECCHP has improved by the value of the power of
the displaced cooling, Pcool. It is assumed that the power consumed by
the IT equipment, PIT, remains constant both before and after the
introduction of a CCHP system. Note that once CCHP is implemented,
the original Pfac can only be mathematically approximated. To illustrate
the changes in PUE with the introduction of CCHP, an example is
provided.
PUE example calculation:
Assume that the total facility power, Pfac, is measured at the utility
meter, prior to the installation of CCHP, to be 2,000 kW. The CCHP
system generates Pgen = 750 kW of power, and offsets cooling that
requires Pcool = 250 kW of power to generate. It is also assumed that the
IT load is 1,000 kW. The PUE, prior to the installation of CCHP, is
calculated as:
264
2,000kW
PUE 2.0
1,000kW
Upon introduction of the CCHP system, and using Equation E.4, the
PUECCHP is calculated as:
2,000 kW – (750 kW + 250 kW) + 750 kW

PUECCHP = = 1.75
1,000 kW
The improvement in PUE for the data center after the installation of
a CCHP system is 12.5%.
The Total Cost of Ownership (TCO), which will account for the cost
of the fuel used to run the generators, as well as maintenance cost are
beyond the scope of this discussion and are not covered in this book.
CCHP systems provide economic return where a differential between
natural gas and electrical grid electricity exist known as ―s
park spread‖.
These energy economics are an important factor in the decision process
in deploying CCHP systems as are both assured power and assured
cooling decisions.
265
ABBREVIATIONS
A cross-sectional area
AC alternating current; air-conditioning
AHU air-handling unit
AHRI Air-conditioning Heating and Refrigeration Institute
AMD air moving device
ANSI American National Standards Institute
ASHRAE American Society of Heating, Refrigeration, and Air-
Conditioning Engineers
ATIS Alliance for Telecommunications Industry Solutions
ATS automatic transfer switch
BHP brake horsepower
BMC baseboard management controllers
BMS building management system
CCA constant-current anemometer
CCHP combined cooling heat & power
cfm cubic feet per minute
CFO chief financial officer
CIO chief information officer
COO chief operating officer
CPU central processing unit
CRAC computer room air conditioner
CRAH computer room air handler
CT current transformer
CTA constant-temperature anemometer
CTTS closed transition transfer switch
CVA constant-voltage anemometer
d diameter
DC direct current
DCeP Data Center Energy Productivity
DCiE Data Center Infrastructure Efficiency
DE distributed energy
DG distributed generation
DMC dual mode controller
DP dew point
DX direct expansion
E energy
ECDW entering condenser water temperature
ECHW entering chilled water temperature
ECM electronically commutated motor
EEWT entering evaporator water temperature
EPA Environmental Protection Agency
EUE Energy Usage Effectiveness
fpm feet per minute
GPIB General Purpose Interface Bus
gpm gallons per minute
GUI graphical user interface
H head
h height; head of water
Hp horsepower
HPC high performance computing
HRSG heat recovery steam generator
HTTP Hypertext Transfer Protocol
HTTPS Hypertext Transfer Protocol Secure
HVAC Heating, Ventilating, and Air Conditioning
I current
I2C Inter-Integrated Circuit
ICT Information and Communications Technologies
IEC International Electrotechnical Commission
IEEE Institute of Electrical and Electronics Engineers
IOPS Input / Output Operations Per Second
IP internet protocol
IPLV integrated part load value
IPMI Intelligent Platform Management Interface
IT information technology
KVA kilovolt-amperes
kW kilowatt
kWh kilowatt-hour
LAN local area network
LCD liquid crystal display
LCDW leaving condenser water temperature
268
Appendix F—Abbreviations and Glossary
LCHW leaving chilled water temperature

LED light-emitting diode
LEWT leaving evaporator water temperature
m mass flow rate
ma milliamp
NFPA National Fire Protection Association
NIC network interface controller; network interface card
NIPP TEE Network Interface, Power and Protection
Telecommunications Energy Efficiency
NPLV nonstandard part load value
OEM original equipment manufacturer
OPN open presence networks
P power
p pressure
P/T pressure / temperature
PD powered device
PF power factor
PM-Bus Power Management Bus
PoE Power over Ethernet
ppm parts per million
PRT platinum resistance thermometer
PSE power supplying equipment
psi pounds per square inch
PSU power supply unit
PT potential transformers
PUE Power Usage Effectiveness
Q volumetric flow rate
Q volume of airflow
q average velocity pressure
Re Reynold’s Number
RH relative humidity
RLA rated load amperes
RMCP+ Remote Management Control Protocol
RMS root mean square
RPDU rack-mounted power distribution unit
RPP remote power panel
RTD resistance temperature detectors
SCADA supervisory control and data acquisition
SG specific gravity
269
SMS Short Message Service

SMTP Simple Mail Transfer Protocol
SNMP Simple Network Management Protocol
SP static pressure
SPEC Standard Performance Evaluation Corporation
SSH Secure Shell
SSL Secure Sockets Layer
T temperature
TCO total cost of ownership
TCP/IP Transmission Control Protocol /Internet Protocol
THD total harmonic distortion
UPS uninterruptible power supply
V volumetric flow rate
V volt; average velocity
v velocity
VA volt-ampere
VAR volt-amperes reactive
VFD variable frequency drive
VP velocity pressure
W watt
WAN Wide Area Network
WC water column
WHP water horsepower
WS-MAN Web Services for Management
ΔT delta (difference of) temperature
ΔP delta (difference of) pressure
change or difference
density
viscosity – dynamic
viscosity – kinematic
Cp specific heat (constant pressure)
k thermal conductivity
efficiency
270
ABBREVIATIONS (SUBSCRIPTS)
c chiller
c1, etc. chiller 1, etc.
CCHP Combined Cooling Heat and Power
comp compressor
cool cooling power saved by CCHP
cp condensate pump
CPP critical power path
crac CRAC
crah CRAH
ct cooling tower
ct1 cooling tower 1, etc.
dc data center
deh dehumidifier
dp displaced power
fac facility
fan fan
gen generator
heat heating
hum humidifier
i input
ipd input power distribution
IT IT equipment
L-L leg-to-leg
L-N leg-to-neutral
net network equipment
p pump
p1 pump 1, etc.
pd power distribution
RMS root mean square
serv server(s)
stor storage equipment
ups UPS
ups1 UPS 1, etc.
VFD variable frequency drive
271
GLOSSARY
ACPI:
cabinet:
crest factor: a waveform measurement calculated by peak amplitude

divided by the RMS.
272
circuit setter: a valve that is carefully calibrated to provide flow

balancing and flow metering.
cylindrical unloading:
data center:
the ratio of the total

amount of power used by a data center facility to the power delivered to
IT equipment, as a percent.
DCMI:
273
of total source energy

by total UPS energy; source energy differs from site energy in the fact
that it accounts for the types of fuels that are consumed to provide energy
to the facility.
flicker: a rapid fluctuation in voltage. Measurements are defined by

IEC 61000-4-15, Testing and Measurement Techniques – Flickermeter –
Functional and design specifications, Ediction 1.1, 2003-03.
harmonics: electrical voltages and currents resulting from certain

loads that are the source of many power quality issues.
274
a simple and inexpensive means of addressing lower than

designed or expected capacity by diverting hot discharge gas to falsely load an
otherwise lightly loaded compressor
275
IPMI:
PMBus:
Power Usage Effectiveness (PUE): the ratio of power delivered to

IT equipment to the total amount of power used by a data center facility.
precision: the quality of being reproducible
rack
276
rack-mounted equipment
ratchet charge: a utility rate provision under which the demand

charge for each month (or other period) is based on the highest measured
demand (or its percentage) over the previous year (or other period).
reactive power: portion of apparent power that does no work. It is

measured commercially by kilvars. Reactive power must be supplied to
most types of magnetic equipment, such as motors. It is supplied by
generators or by electrostatic equipment, such as capacitors.
277
278
SMASH:
SMBus:
SNMP:
spark spread: the difference between the market price of electricity

or natural gas and its production costs. To calculate the spread, the heat
rate of a generating unit or power system is multiplied by the cost of
energy measured in dollars per BTUs (British thermal units).
standard pressure: reference condition for the pressure (assumed sea

level) is defined by the ICAO (International Civil Aeronautics
Organization) as 101.325kPa, which is exactly 1013.24 millibars. In I-P
units the value is approximately 14.696 psi, or 29.921 inches of mercury
at 32 °F.
standard temperature: reference condition for the rating of

equipment, usually 68 °F (20 °C).
279
ventilation
280
281
AHRI. 2003. AHRI Standard 550/590-2003, Standard for Water Chilling

Packages Using the Vapor Compression Cycle.
http://www.ahrinet.org/ARI/util/showdoc.aspx?doc=1472
ANSI / ASHRAE. 2007. ANSI / ASHRAE 127-2007, Method of Testing
for Rating Computer and Data Processing Room Unitary Air-
Conditioners. Atlanta, GA: American Society of Heating,
Refrigerating, and Air-Conditioning Engineers, Inc.
ANSI / ASHRAE. 2008. ANSI / ASHRAE 135-2008, BACnet – A Data
Communication Protocol for Building Automation and Control
Networks. Atlanta, GA: American Society of Heating, Refrigerating,
ASHRAE. 2002. Guideline 14-2002 – Measurement of Energy and
Demand Savings. Atlanta, GA: American Society of Heating,
ASHRAE. 2006. Liquid Cooling Guidelines for Datacom Equipment
Centers. Atlanta, GA: American Society of Heating, Refrigerating,
ASHRAE. 2006. ASHRAE Green Guide: The Design Construction and
Operation of Sustainable Buildings, Second Edition. Atlanta, GA:
American Society of Heating, Refrigerating, and Air-Conditioning
Engineers, Inc.
ASHRAE. 2006. 2006 ASHRAE Handbook – Refrigeration. Atlanta, GA:
Engineers, Inc.
ASHRAE. 2008. Thermal Guidelines for Data Processing Environments
– Second Edition. Atlanta, GA: American Society of Heating,
ASHRAE. 2008. Evaporative Cooling Choices to Maximize Waterside
Economizer Use in Datacom Installations. ASHRAE Transactions
2008, Vol. 114, Part 1 Paper NY-08-006
ASHRAE. 2008. 2008 ASHRAE Handbook – HVAC Systems and
Equipment. Atlanta, GA: American Society of Heating,
ASHRAE. 2008. ASHRAE Guideline 22-2008, Instrumentation for

Monitoring Central Chilled-Water Plant Efficiency. Atlanta, GA:
Engineers, Inc.
ASHRAE. 2009. Best Practices for Datacom Facility Energy Efficiency.
Atlanta, GA: American Society of Heating, Refrigerating, and Air-
Conditioning Engineers, Inc.
ASHRAE 2009. Fundamentals Handbook Atlanta, GA: American
Society of Heating, Refrigerating, and Air-Conditioning Engineers,
Inc.
ASHRAE 1991. ASHRAE Terminology of Heating, Ventilation, Air
Conditioning, & Refrigeration - Second Edition. Atlanta, GA:
Engineers, Inc.
Cisco 2009 “Managing EnergyWise Using CiscoWorks LAN
Management Solution.” Cisco White Paper C11-558123-00 Sep.
2009
Clark, Don 2005. How Sensor and Gauge Accuracy Impact Chiller
Efficiency Do You Really Know How Your Chillers Are Running.
http://www.efftec.com/theimpactofsensorcalibration.html
DMTF. 2008. Web Services for Management (WS-Management)
Specification v1.0.0. Distributed Management Task Force,
Inc. http://www.dmtf.org/standards/published_documents/DSP0226
_1.0.0.pdf
EPA. 2007. Report to Congress on Server and Data Center Energy
Efficiency Public Law 109-431
EPA 2008. Energy Star Program Requirements for Computers, Version
5.0
www.energystar.gov/ia/partners/products_specs/program_reqs/Comp
uter_Spec_Final.pdf
Efficiency Valuation Organization. 2007. EVO 10000-1.2007:
International Performance Measurement and Verification Protocol,
Concepts and Options for Determining Energy and Water Savings
Volume 1. http://www.evo-world.org/
Figliola, R. S., and Beasley, D. E., 1991. Theory and Design for
Mechanical Measurements. John Wiley & Sons.
IEEE. 2003. 802.3af: Amendment: Data Terminal Equipment (DTE)
Power via Media Dependent Interface (MDI)
284
References
Intel. 2004. Intelligent Platform Management Interface IPMI v2.0

Specifications Document Revision 1.0.
http://download.intel.com/design/servers/ipmi/IMPIv2_0rev1_0.pdf
Lippis, Nicholas John III. 2009. “Controlling Corporate Energy
Consumption via the Enterprise Network: A New Approach to
Achieving Energy Efficiency by Leveraging Network Infrastructure.”
Lippis Report. Jan. 2009
Philips Semiconductors. 2000. I2C Bus Specification Version 2.1.
SPEC, 2008. SPECpower_ssj2008, SPEC – Power and Performance
TGG. 2007. Qualitative Analysis of Power Distribution Configurations
for Data Centers. The Green Grid
TGG. 2008. A Framework for Data Center Energy Productivity. The
Green Grid
TGG. 2008. Green Grid Data Center Power Efficiency Metrics: PUE
And DCiE. The Green Grid
TGG. 2009. Proper Sizing of IT Power and Cooling Loads. The Green
Grid
TGG. 2009. Usage and Public Reporting Guidelines for The Green
Grid’s Infrastructure Metrics PUE/DCiE. The Green Grid
United States Department of Energy (DOE) Federal Energy Management
Program (FEMP). 2008. M & V Guidelines: Measurement and
Verification for Federal Energy Project, Version 3.0.
http://www.eere.energy.gov/femp
285

Energy Comsumtion Ashare

Uploaded by

Copyright:

Available Formats

Energy Comsumtion Ashare

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Energy Comsumtion Ashare

Uploaded by

Copyright:

Available Formats

This file is licensed to sergio torres morales (retos@correoinfinitum.com).

License Date: 9-26-2012

Any updates/errata to this publication will be posted on the

©2009 American Society of Heating, Refrigerating

All rights reserved.

Cover design by Joe Lombardo, DLB Associates.

CHAPTER 2 HOW, WHAT, & WHERE TO MEASURE ... 15

CHAPTER 3 MEASUREMENT DEVICES ........................... 21

CHAPTER 4 MEASUREMENT COLLECTION

Real-Time Energy Consumption Measurements in Data Centers

PART 2 COOLING SYSTEMS—AIR MEASUREMENTS

CHAPTER 5 AIR HANDLERS ............................................... 79

CHAPTER 6 COMPUTER ROOM UNITS ........................... 85

PART 3 COOLING SYSTEMS

CHAPTER 7 PUMPS .............................................................. 101

CHAPTER 8 COOLING TOWERS...................................... 113

CHAPTER 9 CHILLERS ....................................................... 125

CHAPTER 10 HEAT EXCHANGERS ................................... 141

PART 4 POWER SYSTEMS MEASUREMENTS

CHAPTER 12 UPSTREAM CRITICAL

CHAPTER 14 COMPUTER ROOM TRANSFORMER

PART 5 IT SYSTEMS MEASUREMENTS

CHAPTER 15 COMPUTE & STORAGE SYSTEMS........... 185

CHAPTER 16 NETWORKING SYSTEMS ........................... 201

APPENDIX A PUMPS .................................................................... 213

Real-Time Energy Consumption Measurements in Data Centers

APPENDIX B CHILLERS ............................................................ 223

APPENDIX C MIXED-USE FACILITIES................................ 227

APPENDIX E ONSITE POWER GENERATION

APPENDIX F ABBREVIATIONS AND GLOSSARY ........... 267

REFERENCES ................................................................................... 283

Lead editor / author – Tahir Cader, HP (formerly SprayCool)

Chapter 1 – Tahir Cadera,b, HP (lead)

Appendix A – John Beana,b, APC/Schneider (lead)

Real-Time Energy Consumption Measurements in Data Centers

Appendix D – Steve McCluera,b, APC/Schneider (lead); Bill

The following individuals also provided significant feedback and

Production of final book including creation of most graphics – Jeff

Book cover design – Joe Lombardo, DLB Associates.

Over the last several years, energy consumption by data centers in

Real-Time Energy Consumption Measurements in Data Centers

Figure 1.1 - Projected data center

The alarming trend of escalating electricity consumption in US data

real-time energy consumption data using energy efficiency and

Real-time energy consumption measurements are only possible if all

 Minimum Practical Measurement

When deciding what level of measurement to target, a data center

 Minimum Practical Measurement – This will require some level of

 Best Practical Measurement – This will require a lower level of

Real-Time Energy Consumption Measurements in Data Centers

 State-of-the-Art Measurement – This will not require human activity

Mixed-use facilities offer the greatest challenge in which to quantify

Figure 1.2 - Schematic representation

Real-Time Energy Consumption Measurements in Data Centers

Figure 1.3 - Schematic representation

Figure 1.4 is a simple graphic representing the key metering

Real-Time Energy Consumption Measurements in Data Centers

Figure 1.4 - Key metering locations in

1.1 OBJECTIVES FOR THIS BOOK