Awesome! The PUE tells only part of the story - the efficiency of the servers themselves is "hidden" under PUE - more MIPS for your kWh is in there somewhere too....

The servers show a lot of innovative approaches - and pose the question, where to now? I've often wondered when one of our leading server manufacturers will deliver a solution as "energy smart" as good old boradcast TV kit - which runs on direct outside air up to 50 or 55 C (that's 122 to 131 F).... the chips can handle it, they just need to be packaged to let the heat out. A different heat sink approach might offset some of the other savings, but overall it should stack up.... then there's the issue of how to get the air in and then out again, and still have a livable workspace to walk around.... but we actually know how to do that already, we've been looking at air distribution solutions as an industry for a while and that bit is quite mature.

So, hands up any vendors out there who are game to give it a crack (or perhaps no-one will speak up, because someone already is?)!

I would love to know what is the redundancy level of their data center. I suspect it is all based on N level design.

What does "Ethernet-powered LED lighting" mean?

Does that imply that they used POE (Power Over Ethernet) ports to power lighting ?

@Yigal,
That's a good point, but if they have the redundancy distributed amongst sites, instead of in a single site, does that make their resiliency any less? If Tier I is handling a % of load that can be handed to another site seamlessly (or near seamlessly) through virtualization of sites, then this wouldn't be a big deal. Facebook may be a great candidate for this as well, as at the end of the day, this isn't a financial system, and though it may sound glib, what is the impact of an outage to Facebook users (and it would be localized to those users) if for a few minutes they are not able to get to their accounts? They don't pay for them, you could argue some revenue is lost from revenue generating sources for Facebook, but this isn't like an outage to eBay, or iTunes.
I suspect they may be leading the charge of what I've been predicting for a while. That Data Centers are the next convenience store... there will be one on every street corner (literally). But they won't look like the legendary Tier IV sites. Instead, they'll be highly virtualized (between sites), high density, and un-staffed (though secure, like power substations). PUE is a lot easier to achieve at Tier I level than at any other Tier.

The Uptime Institute Blog has our technical review of the Prineville site.

http://blog.uptimeinstitute.com/2011/04/technical-response-to-facebook%E...

Overall, the design is very innovative and I laud Facebook for sharing with the community to drive improvement. I think we can all agree the design does not support the same availability objective that many owners require.

Additionally, Facebook's drive towards efficiency reduced aniticipated PUE but their cooling selection is water based in a high desert. My colleague Chris Brown summed it up best: “The term ‘green’ cannot just be about reducing electrical power consumption. It has to involve the natural resource limitations of the local area. Green must be centered on designing data centers that minimize the consumption of all natural resources not just one,” Would you call a data center green that relies on a small community's ground water supply for 100% evaporative cooling?

Yes Julius, that is correct. Facebook have used POE (Power Over Ethernet) ports to power lighting.

This is what is was able to find:
"Energy-efficient LED lighting is used throughout the data center interior. / Innovative power over Ethernet LED lighting system. / Each fixture has an occupancy sensor with manual override. / Programmable alerts via flashing LEDs."

LED lamps use DC power. The LED lamps that you can buy that screw into a standard Edison socket contain electronics to convert the AC line voltage into DC. This is inefficient -- generates heat and wastes power (though still not nearly as much as a traditional incandescent bulb).

By using power over Ethernet for their lighting, the datacenter can use cheaper, cooler, more efficient bulbs, and save a lot of money on the wiring too (remember copper is expensive)

One other very important aspect of the POE lighting approach is that each light is actually individually controlled, in essence a single circuit, so other applications such as motion activation, and level control, are inherently available options. While this has only marginal energy reduction by comparison to the overall in a Data Center it has enormous impact in more conventional lighting applications.

@Rowan

Such a good point about the servers.

We have been suppyling servers to the military for years that operate in high ambient environments with no active cooling. I always questioned why OEM's could do it for so called "hardend" servers but never for general use.

Availability is the big question but Scot hit the nail on the head about PUE and Tier 1 facilities.

Simon:
I dont think the OEMs would have any probelm with supplying Military grade servers but who is going to pay the price difference between hardened Military grade servers and conventional off-the-shelf servers us mortals use. From my experience most of the companies would prefer to stick to the normal servers

Keith - The UTI comments about water usage are, of course, correct but may not be fully fleshed out. The Facebook PUE is 1.07 and it uses a lot of water locally. The delta between that 1.07 PUE and a PUE of, say, 1.5, is also reflected in the water consumption at the power station(s) feeding the grid. In fact, having checked the typical water consumption at US stations, the Facebook facility uses marginally less water than a non-adiabatic PUE 1.5 facility would in the same area and a lot less than a more typical PUE 2.0 facility. Indeed that is also pertinent at partial loads. So the point comes down to LOCAL consumption vs OVERALL consumption. Facebook use less power and less water overall.

Let's explore some of the power lowering techniques. First of all there must be a hats off to a key player who you may not know. His name is Giovani Coglatore whose vision helped Rackable Systems (Now SGI) create the Ice Cube Air whose PUE rating is similar to Facebook. He moved to Facebook and I believe that his ideas may have been instrumental. Lets explore some of those ideas..

480 / 277 is used in all industrial lighting circuits. It uses about 1/3rd the copper of the 208 system. Second is phase perfect loading. Yes, this is another technique of reclaiming power that would be otherwise lost as heat in a building transformer due to phase imbalance sometimes called loping. How is this accomplished ? Today's power supplies have evolved. Here is how.

- Synchronous decommutation of the AC input
- Faster Switching Speed
- Planar Magnetic Transformers
- Lower ripple currents
- The ability to synchronize 3 power supplies PWM so no steering diodes are needed on the output.
- 1 unified DC Bus at the Rack Level

Synchronous decomutation is a technique which removes the bridge rectifier in the front end of the power supply. This circuit does two jobs. One performs the function of a full wave rectifier but without the losses and second it aligns the voltage and current. A bridge rectifier in the front end of the power supply can attribute up to 10% of the losses in the power supply. With this new circuit those losses are cut an order of magnitude.

Faster switching speeds are due to new science in the semiconductor industry which takes switching Mosfet's to a whole new level of performance by almost eliminating the problems of gate capacitance in the device. Faster switch speeds means higher power density / the power supply shrinks in size.

The side benefit to faster switching speed is lower ripple currents. Smaller less expensive capacitors with lower ESR ratings can be used with good sucess.

Planar magnetic transformers are not wound. Instead they are etch Printed Circuit Board types and the stamped flat winding type. These transformers with flat windings do not suffer from coupling losses their wound cousins suffer from. Also capacitance is also reduce which helps in the area of efficiency. A planar transformer switching above 500 Khz can be as much as 99% efficient.

Synchronizing the PWM's Pulse with modulators (The switching / timing part of the power supply) of multiple power supplies allows for a common output bus to be connected to the DC outputs of the power supplies without the need of steering diodes. Why was there a need for steering diodes in the past ? In the past power supplies ran asynchronously. What this means is that they all had their feedback and control circuits to manage the voltage and current on its outputs. Running in the manner would cause problems when two asynchronous power supply outputs were connected. I would cause a tug of war and one of the power supplies would typically burn up. The steering diodes just like the bridge rectifiers have a voltage drop across the device which cause losses.

So you see much has changed at the rack level when advanced power solutions are necessary for those who are efficiency minded. One such rack level product is the SGI Cloudrack which employs one or two clusters of 3 power supplies depending upon the configuration. The Cloudrack has one unified rack level 12 Vdc Bus to power all the systems. So, it presents perfect phase balance to the data center building transformer.
Rack Level Single Conversion Power Distribution will one day become the standard for cabinet power.

-

Individual Cooling at the rack level is responsible not radical ! I prefer water from a northern area where lakes and rivers can supply water and cooling for $0.00. Well the cost of materials and a settling pond for the EPA, yes of course but otherwise no power for cooling except for the localized air movement at the cabinet level.

-

More in another post.

I know not much has been said about hysteresis when it comes to boot times and provisioning. This really is a serious power wasting issue. Data center managers should demand " INSTANT ON " servers and equipment. With the Mac book Air we see that with the ample use of flash memory the laptop device is an "Instant on / Instant off" machine. These are sometimes called "State Machines" as they save the state of the computer in flash memory. So what transpires is a very fast data dump from flash to RAM to start and vise versa to stop. Very fast and saves power ! Provisioning becomes simple and ramp rates and histograms are not necessary as load / demand increases.

I have been involved in mechanical design for the past 20 years and 18 years before that I was in electronics design. For the past 11 years I have designed server system, cabinet systems and data center level systems. My current position has me designing a large optical router for one of the industries Tier One IT companies.

Ian, Thank you for your comments. I think what Facebook missed, even with an excellent program to become more efficient , is that data center sustainability is a much broader challenge requiring thoughtful deliberation of many factors. Locating a 30 MW Data Center (with another expansion to possibly double the size) in a rural area with limited water has consequences to others. The commonly reported example that Facebook may have missed some broader issues is evident in the Greenpeace program asking Facebook to “Un-friend Coal”. Facebook is just now installing 100 kW (a drop in the power consumption bucket) to improve perceived use of renewables.

Let’s take a look at your point which is a good one. The Prineville Data Center overall water use is projected better than traditional US Data Centers with Chilled Water Machines and Cooling Towers. Facebook has stated their Water Usage Effectiveness WUE of 0.3 L/kWh versus a national average of 1 L/kWh. But to your point this can be examined further with The Green Grid’s “WUE source” which accounts for the use of water in power generation and the data center. According to The Green Grid the water usage for power in Oregon is 3.1 L/kWh. The power industry in Oregon uses quite a bit of water and is not so impressive compared to other western states (California 0.19, Washington State 1.1 or even Colorado 1.93 L/kWh respectively). The total WUE Source for the Prineville Data Center is then approximately 3.4 L/kWh. This figure is quite high when compared to an average data center in CA that would be 1.2 l/kWh. WUE source makes the site a large water user even if the data center is 38% percent more efficient than Facebook's current COLO spaces.

I’d like to once again look at the local impact. Facebook entered a rural community with only 9 shallow aquifer wells as a water source. There is no surface water available. Facebook has sunk one or more wells into the same aquifer. From centraloregonian.com "The City Public Works Superintendent Jerry Brummer explained that on the valley floor (where Prineville is located) there is a finite amount of water available "“If we just continue to drill wells, we’re going to draw the same amount of water out of 50 wells as we are going to draw out of 10. We are just sticking more straws in the bowl.”" Water issues are very real issues in desert regions, I have been in communication with the Facebook team and requested their water consumption calculations and aquifer impact. We’ll see.

Opening yourself to scrutiny to the whole world via the web can be difficult as Facebook is probably learning, yet whoever said leading was easy.

Would be good to understand what redundancy is designed in, sorry to bang on about SLA's, but do they any customers requiring them, and if not, then how relevant is this design in a critical facility perspective??

To pull the discussion back to what is useful to folks looking for an efficiency roadmap to their own enterprise data centers from the FAcebook experience:
1. Regardless of any debate about social irresponsibility over the impact on the local water table, this cooling system will use less water on the site than a traditional data center using tower evaporation for energy removal,
2. The direct evaporative cooling system with no refrigerant back-up to a serious episodic pollution event probably means this is not a viable option for a company without a robust disaster-recovery data center someplace.
3. An indirect evaporative cooling solution with integral refrigerant back-up would give a very similar energy efficiency, with everything else being equal, though would obviously increase the initial capital investment.
4. Just as a side note - the POE LED's actually go into the diviser of the PUE equation so they would simultaneously increase the energy usage of the data center while reducing the PUE. But I suspect it is a relatively small fudge.

Very interesting , when we as a DC Power System manufacture in India , working on such a high efficient high wattage DC UPS for Data center , same is already done at some part of world . This the first leapfrog at Green Data center , surely a big thing still possible .

HVDC (e.g. >400VDC) does increase efficiency compared to a 460/208V North American three-wire system but offers no advantage to four-wire 400/230V EMEA solutions. In fact there is a downside with HVDC; every system has to be 2N (like a Tier IV topology) due to concurrent maintenance requirements. The change to 230V ph-N is in fact already happening in the US with reported sales of servers on the increase. The lack of codes and recepticles etc is also a hurdle - as is arc-flash, although that could be solved by clever plug/socket and switch designs.

The Facebook datacenter is much more energy-efficient than the "State of the art" industry average datacenter with PUE equal to 1.5. This is surely the average of the installed base, nog for nee datacenters. Or am I mistaken?

@Karel, no Facebook is not state of the art. Modular data centers are more energy efficient than the Facebook model. Here is a link to a modular data center which has a PUE of 1.06. http://www.sgi.com/products/data_center/ice_cube_air/

As to sustainability, modular data centers are better at sustainability as they are convertible replaceable units. Unlike fixed data centers, modular data centers are able to capitalize on square footage. Stacking them can allow a datacenter to expand without having to break new ground. Modularity like this saves huge $$ on infrastructure, logistics, and improves site longevity. ---> No <--- fixed datacenter can sport these qualities. Some say this is a radical departure from the "Norm". I say its a responsible departure from the old and embracing this kind of change embraces a newer better future for datacenters.

So you are selling containers then, Ron? Ones with a core backbone of power and cooling infrastructure or self contained?

I think Ron made a good point on several levels. Not the least of which are first cost build, portability, etc. And the etc part can involve quite a few considerations. I attended a conference 6 months ago which was heavily attended by DC managers and to a person they all had the same basic comments and reservations. The comments were that the modular or containerized data center was probably inevitable and the reservations were also similar in that this approach was better suited to areas of data processing where the application was indentical or very similar. I am not sure why that is the case but I am sure someone on this discussion can enlighten me.