Overview

 of  Largest  Data  Centers  

   Ali  Ghiasi                  Rich  Baca  

Ghiasi  Quantum  LLC    Commscope  
 
802.3bs  Task  Force  
May  2014  –  Interim  MeeFng  
 
 Overview*    
q  Previously  presented  overview  of  some  of  the  largest  US  data  
centers  
–  h4p://www.ieee802.org/3/100GNGOPTX/public/sept11/ghiasi_01_a_0911_NG100GOPTX.pdf  

q  Several  assumpFon  were  made  in  the  original  study  

–  Building  assumed  to  be  square    
–  Examining  satellite  photo  indicate  most  building  are  rectangular  likely  
due  to  HVAC  requirements  so  actual  length  can  be  li4le  longer  
–  3  Years  ago  Google  Mayes  county  with  ~1000,000  sq-­‐Y  was  the  largest  
known  data  center  but  now  there  are  several  comparable  size  data  
center    
–  SuperNAP  Las  Vegas  with  2,200,000  sq-­‐Y  is  the  largest  known  
opera\ng  data  center  and  more  than  twice  largest  Google  data  center  
q  Bigger  building  may  translate  to  longer  fiber  runs!  
  provided  here  is  as  is,  there  is  no  guarantee  on  their  accuracy  as  it  relies  on  3rd  party  ar\cles  and/or  es\mate.    
*  Data  
A.  Ghiasi   802.3  BS  Task  Force   2  
 Google  Mayes  County  vs  SuperNAP  Las  Vegas    
Google  Mayes  County  

A.  Ghiasi   802.3  BS  Task  Force   3  

 Some  Facts  About  Mega  Data  Center  
Trend  
q  Google  has  ~14  data  centers  around  the  world  consuming  260  
MW  or  0.01%  of  total  power  capacity  on  the  planet  
q  As  of  July  2013  both  Google  and  MicrosoV  each  have  in  the  
900k-­‐1000k  servers  
q  SuperNAP  Las  Vegas  based  on  Dell  Scorpion  pulling  in  35  kW/rack  
–  SuperNAP  is  located  in  Las  Vegas  for  having  least  amount  of  natural  
disasters    
–  Las  Vegas  also  is  a  hub  for  dark  dark  fiber  from  .COM  era  
–  At  power  density  of  ~1400  W/sq-­‐Y  the  challenge  is  cooling  Vegas  
q  Standard  Facebook  storage  rack  has  power  capacity  of  8  kW  
–  Facebook  uses  3  \ered  storage  with  total  storage  capacity  of  100  PB.      

A.  Ghiasi   802.3  BS  Task  Force   4  

 Original  Study  Presented  in  2011  .bm      
Data  Center  LocaFon     Status   Square  footage   Power  (MW)   Longest  Run(m)*  
Amazon          

q  *  Dana  Hull,  San  Jose  

Boardman,  OR   Not  operaFonal   100000   5   192.8  
Mcnary,  OR   Under  ConstrucFon   120000   6   211.2  
Manasas,  VA   OperaFonal   110000   5   202.2  

Mercury  News,  April  22   Ashburn,  VA  

Stering,  VA  
OperaFonal  
OperaFonal  
180000  
125000  
9  
6  
258.6  
215.5  

2011     Apple  
Maden,  NC  
Newark  
 
Nearly  Completed  
OperaFonal  
 
500000  
100000  
 
100  
15  
 
431.1  
192.8  

q  **  Status  as  of  April  2011     Facebook  

Princeville,  OR  
Forest  City,  OR  
 
Nearly  Completed  
Under  ConstrucFon  
 
307000  
300000  
 
40  
40  
 
337.8  
333.9  

q  ***  Longest  Run  EsFmated   San  Jose,  CA  

Santa  Clara,  CA  
Lease  
Lease  
25000  
86000  
5  
15  
96.4  
178.8  

to  be  =  2*√(area)=603  m     Santa  Clara,  CA  

Ashburn,  VA  
Ashburn,  VA  
Lease  
Lease  
Lease  
50000  
49000  
45000  
8  
8  
8  
136.3  
134.9  
129.3  

q  #  Tulsa  Times  May  3  2007   Google  

Berkely  County,  SC  
 
Likely  OperaFonal  
 
200000  
 
76  
 
272.6  

states  1.4M  Sq-­‐V  plant  

Council  Bluff,  IA   Likely  OperaFonal   200000   76   272.6  
Dalles,  OR   OperaFonal   200000   70   272.6  

assume  data  center  is  70%  

Lenoir,  NC   Phase  2   337000   76   353.9  
Mayers  County,  OK  #   Likely  OperaFonal   980000   100.8   603.5  
HP          

of  plant     Atlanta,  GA  

Atlanta,  GA  
OperaFonal  
OperaFonal  
200000  
200000  
20  
20  
272.6  
272.6  

q  ##  Various  media  reported  

AusFn,  TX   OperaFonal   100000   20   192.8  
Colorado  Spring,  CO   OperaFonal   250000   10   304.8  
Tusla,  OK   OperaFonal   250000   20   304.8  

likely  size  to  be  ~500K  Sq-­‐V     IBM  

Boulder,  CO  
RTP,  NC  
 
OperaFonal  
OperaFonal  
 
300000  
100000  
 
60  
30  
 
333.9  
192.8  

q  Median  size  of  data   MicrosoV  

Boydton,  VA  ##  
 
Likely  OperaFonal  
 
500000  
 
NA  
 
431.1  

centers  in  US  190000  sq   Chicago,  IL  

Quincy,  WA  
OperaFonal  
OperaFonal  
700000  
500000  
60  
27  
510.0  
431.1  

feet    
San  Antonio,  TX   OperaFonal   477000   27   421.0  
Des  Moines,  IA   Under  ConstrucFon   NA   NA    
Twiner          

q  Longest  run  assuming   Salt  Lake  City,  UT  

Sacramento,  CA  
Lease  
Lease  
15000  
NA  
2  
2  
74.7  
 

190000  sq  feet  area  

Yahoo          
Ashburn,  VA   OperaFonal   112000   10   204.0  
Lockport,  NY   OperaFonal   NA   18    

=√(area)=132.9  m     Omaha,  NB  

Quincy,  WA  
OperaFonal  
OperaFonal  
180000  
180000  
20  
26  
258.6  
258.6  

 
Quincy,  WA   Under  ConstrucFon   NA   72    
190000   132.9  
A.  Ghiasi   802.3  BS  Task  Force   5  
 Updated  Data  Center  Study  Data$Center$Location$ Status Square$footage Power$(MW) Longest$Run(m)
Amazon
Boardman,$OR Operational 100000 25 192.8
Mcnary,$OR Operational 120000 30 211.2
Manasas,$VA Operational 110000 27.5 202.2

q  Largest  operaFng  data  center  

Ashburn,$VA Operational 180000 45 258.6
Stering,$VA Operational 125000 31.25 215.5
Apple

grow  from  ~  1M  sq-­‐V  to  2.2M   Maden,$NC

Newark
Operational
Operational
500000
100000
100
15
431.1
192.8

sq-­‐V  
Facebook
Princeville,$OR Operational$ 307000 40 337.8
Forest$City,$OR Operational$ 300000 40 333.9
San$Jose,$CA Lease 25000 5 96.4

q  Longest  cable  length  grow   Santa$Clara,$CA

Santa$Clara,$CA
Lease
Lease
86000
50000
15
8
178.8
136.3

from  600  m  to  900  m  

Ashburn,$VA Lease 49000 8 134.9
Ashburn,$VA Lease 45000 8 129.3
Altoona,$IA Phase$I$Operational 1400000 117 721.3
Google

–  Longest  cable  when   Berkely$County,$SC

Council$Bluff,$IA
Operational$
Operational$
200000
200000
76
76
272.6
272.6

Langfeng  Data  Center   Dalles,$OR

Lenoir,$NC
Mayes$County,$OK$
Operational$
Operational$
Operational$
200000
337000
980000
70
76
100.8
272.6
353.9
603.5

become  opera\on  will  be   HP

Atlanta,$GA Operational 200000 20 272.6

~1.5  km  
Atlanta,$GA Operational 200000 20 272.6
Austin,$TX Operational 100000 20 192.8
Colorado$Spring,$CO Operational 250000 10 304.8

q  Longest  Run  EsFmated  to  be  =  

Tusla,$OK Operational 250000 20 304.8
IBM
Boulder,$CO Operational 300000 60 333.9

2*√(area)=603  m     RTP,$NC
Microsoft
Boydton,$VA
Operational

Operational$
100000

500000
30

21
192.8

431.1

–  A  typical  rectangular  building   Chicago,$IL

Quincy,$WA
Operational$
Operational$
700000
500000
60
27
510.0
431.1

will  have  slightly  longer  

San$Antonio,$TX Operational$ 477000 27 421.0
Dublin,$Irland Operational$ 330000 22 350.2
Des$Moines,$IA Under$Construction 1160000 NA 656.6

cables   Twitter
Salt$Lake$City,$UT
Sacramento,$CA
Lease
Lease
15000
NA
2
2
74.7
$

q  Median  size  of  data  centers   Yahoo

Ashburn,$VA
Lockport,$NY
Operational
Operational
112000
150000
10
18
204.0
236.1

grow  from  190000  sq-­‐feet  tp   Omaha,$NB

Quincy,$WA
Operational
Operational
180000
180000
20
26
258.6
258.6

225000  sq-­‐V  
Quincy,$WA Operational 151000 26 236.9
Other
Switch$SuperNAP`Las$Vegas,$NV Operational$ 2200000 200 904.2

q  Median  longest  run  assuming  

NSA`Blufdale$UT Operational 1000000 65 609.6
Lakeside$Technology`Chicago,$IL Operational$ 1100000 100 639.4
QTS$Metro$`$Atlanta,$GA Operational$ 970000 120 600.4

square  building  grow  from  

NAP$of$America`$Miami,$FL Operational$ 750000 527.9
IO$Phoenix,$Phoneix,$AZ Operational$ 538000 40 447.1
International$

~130  m  to  ~145  m.   China$Langfang

Tulip,$Bangalore$India$
NGD$Europe,$Wales$UK
Under$Construction
Operational$
Operational
6300000
900000
750000
40
90
1530.1
578.3
527.9
  A.  Ghiasi   802.3  BS  Task  Force   Sum
Median$sq`ft
25012000
225000
1915.55
Median$L$m 144.6
6  
 Equinix  Data  Centers   Equinix'Data'Centers'
Square'footage* Power'(MW) Longest'Run(m) Equinix'Data'Centers
Square'footage* Power'(MW) Longest'Run(m)
Atlanta' 58000 11.05 Amsterdam 137000 54.7
AT1 44615 8.5 128.8 AM1 30556 12.2 106.6
AT2 9448 1.8 59.3 AM2 30556 12.2 106.6
AT3 3937 0.75 38.2 AM3 75888 30.3 167.9

q  Equinix  has  83  data  centers  world   Boston'

Chicago
13000
190000
2.25
31.5
69.5 Dubai
Dusseldorf
19000
53000
5.6
7.8
84.0

wide  head  to  head  in  term  of   Ch1

CH2
54286
48254
9
8
142.0
133.9
DU1
DU2
28538
24462
4.2
3.6
103.0
95.3
area  with  MicrosoV     CH3
CH4
15079
72381
2.5
12
74.9
164.0
Frankfort'
FR1
344230
10124
47.6
1.4 61.3
–  Equinix  website  state  5.8  M  sq-­‐Y   Dallas 150000 21 FR2 195257 27 269.4

of  data  center  but  adding  all  the   DA1

DA2
32143
53571
4.5
7.5
109.3
141.1
FR3
FR4
34712
74487
4.8
10.3
113.6
166.4
data  centers  only  sums  to  3.3  M   DA3
DA4
32143
10714
4.5
1.5
109.3
63.1
FR5
Hong'Kong
29650
171000
4.1
85.7
105.0

sq-­‐Y   DA6 21429 3 89.2 HK1 76321 38.25 168.4

Washington'DC 478000 139.65 HK2 40405 20.25 122.5
q  Equinix  are  moderate  size     DC1
DC2
10782
41074
3.15
12
63.3
123.5
HK3
Jakarta'
54273
32500
27.2
2
142.0
109.9
–  Expect  cable  distribu\on  to  be  bi-­‐ DC3 47920 14 133.4 Geneve' 38000 7.5

modal     DC4
DC5
59900
59900
17.5
17.5
149.2
149.2
GV1
GV2
15200
22800
3
4.5
75.2
92.0

–  100  m  within  one  data  center  or   DC6

DC7
61611
26527
18
7.75
151.3
99.3
London
LD1
34000
1605
11.44
0.54 24.4
vault   DC8
DC10
2567
82148
0.75
24
30.9
174.7
LD2
LD3
19318
6538
6.5
2.2
84.7
49.3

–  1-­‐2  km  to  adjacent  building     DC11

Denver'
85571
12000
25
2.5
178.3
66.8
LD5
Mellborne'
6538
36000
2.2
12.5
49.3
115.7

q  Longest  Run  EsFmated  to  be  =   Miami

MI2
30000
2477
10.9
0.9 30.3
Munich
MU1
84000
53014
19.41
12.25 140.4

2*√(area)=300  m     MI3
Los'Angles'
27523
215000
10
44
101.1 MU2
MU3
692
30294
0.16
7
16.0
106.1

–  A  typical  rectangular  building  will   LA1

LA2
21989
12216
4.5
2.5
90.4
67.4
Paris'
PA1
26000
4480
29.6
5.1 40.8
have  slightly  longer  cables   LA3
LA4
107500
73295
22
15
199.9
165.0
PA2
PA3
11946
9574
13.6
10.9
66.6
59.6
  New'York 484000 90.5 Rio'de'Janeiro 29594 17.1

*  Total  area  for  each  city  was  only   NY1

NY2
16044
74873
3
14
77.2
166.8
RJ1
RJ2
3634
25960
2.1
15
36.8
98.2

provided  to  calculate  each  size  of   NY4

NY5
NY7
240663
53481
98939
45
10
18.5
299.1
141.0
191.7
Sao'Paulo
SP1
SP2
79500
27369
52131
8.54
2.94
5.6
100.8
139.2
each  Bldg  it  was  es\mated  based   Philadalphia'
Seatle
39990
53000
3
19
121.9 Shanghi
SH1
52000
6411
14.6
1.8 48.8

on  the  ra\o  of  Bldg  power  capacity   SE2

SE3
25105
27895
9
10
96.6
101.8
SH2
Sh3
22795
22795
6.4
6.4
92.0
92.0
Silicon'Valley' 448500 33.125 Singapore 117000 45.5
to  total  power  to  calculate  the   SV1
SV2
27079
27079
2
2
100.3
100.3
SG1
SG2
55286
61714
21.5
24
143.3
151.4

Bldg  size.   SV3

SV4
27079
27079
2
2
100.3
100.3
Sum
Median'Size
3313314
29094
742.865
Median'L 104
SV5 243713 18 300.9
A.  Ghiasi   SV6 802.3  BS  Task  Force  
42311 3.125 125.4 7  
SV8 54158 4 141.9
 How  Big  Are  They  

q  Total  floor  space  for  top   Area  sq-­‐V   MicrosoY  

operator  is  ~29  M  sq-­‐V     Equinix  

Facebook  
–  The  total  power   Switch  SuperNAP-­‐Las  Vegas,  
consumma\on   NV  
Google  

assuming  if  all  operated   HP  

at  1400  W/sq-­‐Y  would   Amazon  

be  ~41  GW!   Lakeside  Technology-­‐

Chicago,  IL  
NSA-­‐Blufdale  UT  
–  The  networking  related  
QTS  Metro  -­‐  Atlanta,  GA  
PD  could  be  ~10  GW   Tulip,  Bangalore  India    
and  ~  5  GW  could  be   IBM  
SerDes+Opics  PD  and   Yahoo  

directly  impacted  with   NAP  of  America-­‐  Miami,  FL  

what  we  do  in  .bs!   NGD  Europe,  Wales  UK  

Apple  

IO  Phoenix,  Phoneix,  AZ  

A.  Ghiasi   802.3  BS  Task  Force   8  

Without  Efficiency  Improvement  the  Total  (PON,  Router,  
Transport)  Projected  Power  Consump\on  Could  be  ~1/3  
World  Power  Genera\on  

*  

*  Energy  Consump\on,  Rodney  Tucker,  Photonic  IO  Conference,  Santa  Fe  2012    
A.  Ghiasi   802.3  BS  Task  Force   9  
 Summary    
q  During  802.3bm  study  phase  an  SMF  reach  of  500  m  was  more  
than  adequate  to  address  nearly  all  data  center  but  no  longer  is  
the  case  
–  Minimum  reach  needs  to  be  1  km  with  target  of  2  km  
–  A  2  km  reach  will  also  address  building  to  building  applica\on  
q  In  2011  Google  was  the  1st  to  build  a  data  center  with  area  of  ~  
1M  sq-­‐V  several  Fme  larger  than  other  operaFng  data  centers  
–  SuperNAP  in  Las  Vegas  with  area  of  2.2M  sq-­‐Y  takes  the  crown    
–  In  2014  there  are  numerous  data  center  approaching  or  even  larger  
than  1  M  sq-­‐Y  
•  Building  ~1  M  sq-­‐Y  data  center  is  now  the  norm    
•  Bigger  is  be4er  
q  With  data  centers  power  dissipaFon  growing  exponenFally  power  
dissipaFon  has  to  be  a  key  metric  in  .bs.  
A.  Ghiasi   802.3  BS  Task  Force   10  
 References    
–   Dana  Hull,  San  Jose  Mercury  News,  April  22  2011    
–  h4p://www.cnet.com/news/inside-­‐one-­‐of-­‐the-­‐worlds-­‐largest-­‐data-­‐centers/
h4p://www.forbes.com/pictures/qgl45ijg/range-­‐interna\onal-­‐informa\on-­‐hub/  
–  h4p://wikibon.org/blog/wp-­‐content/uploads/2011/10/5-­‐top-­‐data-­‐centers.html  
–  h4p://www.datacenterknowledge.com  
–  h4p://storageservers.wordpress.com/2013/07/17/facts-­‐and-­‐stats-­‐of-­‐worlds-­‐largest-­‐
data-­‐centers/  
–  h4p://www.desmoinesregister.com/story/money/business/2014/04/18/microsoY-­‐
alluvion-­‐data-­‐center-­‐west-­‐des-­‐moines/7859973/  
–  h4p://www.datacenterknowledge.com/special-­‐report-­‐the-­‐worlds-­‐largest-­‐data-­‐centers/
largest-­‐data-­‐centers-­‐ngd-­‐terremark-­‐qts/  
–  h4p://www.equinix.com  
 

A.  Ghiasi   802.3  BS  Task  Force   11