In the heady world of air conditioning, we have our familiar Q = M x C x Delta"t" equation.
Q is the sensible cooling load from the servers in kW
M is the air flow rate in cubic metres per second
C can be considered a constant with a numerical value of around 1.2
Delta "t" is the difference between supply and return air temperature in Deg C (all right K for purists!)

So I think the increased set points simply allow more efficient heat rejection, rather than a reduction in air flow rate, or fan speed.

Mike I think it's an issue of server design. Certain components at certain locations in the server have a maximum temperature they can withstand reliably, or that the mfg is comfortable with. Most servers are series airflow in that the air flows over, say the storage devices first, then perhaps RAM, then the CPUS, and then perhaps the power supplies. Multiple temperature sensors monitoring the air temperature at each step. If the temperature experienced at any point exceeds the design value fan speed is increased to ensure the temperature is back under the required value. This increase airflow reduces the delta T across the piece of equipment, and consumes more energy at the IT load via the fans. This in turn decreases the efficiency of the overall cooling system due to the lower delta T.

Now the goal, at least I believe, in increasing inlet air temperatures is to widen the window for air and water side economization. For air side, delta T isn't an issue, as most system direct reject the heat back into the outside environment. Water side isn't quite as important, but the coils used for air/water rejection must be ok with it.

The killer though might be the fan side energy increase. David Moss over at Dell has put together a whitepaper that covers the increase in energy usage they found with a few server types I believe, and it wasn't insignificant.

All this however can go away if the server mfgs increase the allowable temperatures of their components by hardening them in some way to allow the higher inlet temperatures and higher internal temperatures. Do this, and the fan curves go back to before, but with higher trip points for fan speed increases, and the higher inlet temperatures will allow more efficient DC operation, with no penalty on the fan side of the servers. Or maybe get rid of one of the fans, either server side or mechanical side ;-)

Several considerations.
* New standard from ASHARAE encourages new servers to operate at higher inlet temperatures. Good only if you have all new servers - not the case in most instances.
* Raising inlet temperature is good to a point. I interpreted the Dell/APC report to indicate 77F as an ideal temp with minimum server kW from server fans.
* Bigger deal is raising underfloor temperature so if you contain the cold aisle and have sufficient underfloor air, the underfloor approaches 77F. If your cold aisle has excessive recirculation, you may have 77F inlet but underfloor of less than 60F - an indication you have a serious problem going undetected.
Key Success Factors:
- Maximum delta T across CRAHS - good. Get rid of wasted cold air.
- Server inlet T that keeps server fans at minimum (raising inlet Ts that causes server fans to ramp up, artificially improves PUE and kills your utility bill - bad.)
- Maximum underfloor T - as close to cold aislet temp as possible
- Trial higher and lower chilled water T. You may find higher T produces more chiller and pump load even though the actual chiller kW/ton goes down.
Measure, measure, measure. ....and measure everything. If you only measure fan loads, you may miss the pump load increases for example.
It soulds complicated but it is only complicated if you aren't thorough. Whatever you don't check will probably hurt you.

At server level, there is a huge increase in fan power with variable speed server fans when inlet air temperatures are increased. Howerver the server fan power increase pales when compared to the savings that higher server inlet air temperatures make possible on the cooling infrasturcture.

Cooling units fan power reduction can be a significant factor on power consumption in higher density applications where the cooling unit fans require UPS support. Lower fan power, smaller UPS and battery.

Higher server inlet air temperatures translate dircetly into higher hot aisle temenratures.

When using a rack or row level close coupled cooling (CCC) systems (Datacentience, Rittal, APC etc), warmer return air temperatures from the hot aisle enable use of much warmer chilled water temperatures. Warmer chilled water temperature allow dramatic reduction in chiller compressor energy use. Compressors are the real energy guzzlers in datacentres.

For every one degree above the rated 7C (45F) leaving water temperature from a chiller you can save 3.5% of the compressor power that is stated at standard 7C leaving water condition.

When using close coupled cooling (CCC) units it is easily possible to design at 16C chilled water temperature supply the CCC units. This is yields a massive 31% savings in compressor power.

For maximum savings in energy, you must reduce or eliminate chiller compressor operation or run hours. This is commonly called free cooling. Substantially more free cooling hours can be obtained when using warmer chilled water.

So, reduced server fan power combined with reduced cooling unit fan power is only part of the solution (our CCC units deliver 50kW with 930 Watts of fan power). Look to reduce the compressor run time to save energy. CCC and free cooling combined are the way to save energy.

I experimented with 78 degrees a couple years ago. Yes it dropped the load on the building fans, but all the multi speed fans in the servers kicked to high! and then within a week started failing. Turns out some of the fans are not rated to 100% duty cycle at their highest speed. It also raised the server power consumption and offset the savings in the building system. (bigg installation)

Overall MTBF still drops as you go past 75 degrees. Even past 70 I saw a drop in service life of hard drives.

I was lucky enough to have a significant sample number (> 5500 units) in one room so If we made environmental changes I could not get "lucky" and not see failures. I also had 4 rooms to play with at one point. Ignoring the failures due to components on the edge of failure the run rate of the failures went up directly with the temperature once you got over about 75, below that it was more random.

Air flow containment is my #1 priority, then fiddle with supply temps to tune to a sweet spot. I used processor temp as my prime benchmark not inlet temp. They don't always track together depending on the various server types and quality levels.

The size of the installation makes a huge difference on whether the high temp inlet strategy makes sense. Also the exact type of cooling system, local climate etc..

Seems like there is a tendency to believe that servers can just be adjusted to meet the new inlet specifications, in reality its non trivial to redesign from the silicon level up to keep the electronics healthy and meet the new guideline.

Just my take on it...

Dave, that's excellent information! Any chance you've ever given a formal presentation on your result? There's a number of people and institutions which would love to see hard data like yours.

You can save a lot more energy by eliminating the fans entirely. Again, as Dennis points this is a go-forward strategy with new equipment, most likely deployed in clusters, using directly connected cooling systems, like the Liebert XDS or Cray/IBM pumped liquid, to reject the heat at a lower total energy cost than "free cooling".

This approach greatly reduces the total IT connected load AND cooling plant load producing a double benefit. More power now available at the UPS to support a real workload instead of fans and significantly more total power available at the building entrance. Enough in some scenarios to land a few more UPS systems and expand the IT connected load beyond the initial design. (Note, this will require some serious rework to the switchgear and distribution system but it beats building a new data center!)

Can currently available technology realize 100% rejection through circulating liquid, or does it just get us back down to a few kW per rack that needs to be handled by air? I've seen a few of the IBM solutions that were liquid cooled at the processor level, but drives and other components were still air cooled. They handled that load through a rear door heat exchanging system if I recall, or allowed the user's room to handle the rest. It was still roughly a 5kW heat load to cool with air, but something over 20kW went out with the water.

Anything that makes the space more efficient would be great, and anything that allows the user to reclaim lost capacity, while mantaining reliability is key as far as I can tell.

The real question is that is is it worth the sunk cost in all the fancy wet systems?

I've seen a couple really neat systems, but when you add up the maintenance the special parts the non standard chassis etc, do we really win cost wise? I have not seen a really good top down detail cost analysis yet that makes me jump at it hard. Logically it makes sense but I haven't found the proof that just makes me ready to make it my #1 option, or an option that I can sell to a client with minimum energy.

I know that in the room I had we had very solid hot isle containment, the supply air was running at 70 degrees and the outlet air temps on the blades were about 110 (Servers CPU's ran at 90+ % doing numerical analysis). The fans in the systems were not lost energy because they were helping the air move, they became a component of the AC system, if I turned on 5 or 10 racks at once you could watch the load on the building fans drop to maintain the same pressure set-point and fan RPM. The server fan control and the building fan control naturally came to a balance.

From a TCO point of view the biggest advantage was that I didn't have to buy any thing special, any server worked, as long as the fans were pulling air through the server and not just spinning internally to service a CPU heat-sink for instance. If the unit was breathing front to back I was good to go. We had a lot of churn in servers due to the type of work they were doing so removal and installation were big for me. Our service life was about 2.5 years for servers. I'd swap about 3500 out at a time. With liquid cooling that would have been ugly, depending on the plumbing and piping system.

Unfortunately I left that company to try something new and don't have access to all the data any more... I can only describe the trends I got to play with.

Hello David,

You might want to consider building a simulation model of your data center before taking the drastic step of installing containment. With a model you can visualize airflow and the cooling performance of the data center system (facility and IT equipment working together) from a fundamentals standoint. The model can be used to assess the impact of radical changes (containment) or small changes (blanking, re-distribution) on financial and cooling performance metrics.

ASHRAE & the manufacturer in their recommended T&H standards are opening the door to imagination when it comes to energy reduction in a data center. Yes with the rise of temperature you can save chiller kW/T and possibly equipment sizes while the fan energy may or may not go up however, and based on the location, free cooling hours may extend and provide the most savings so why limit your options when looking at the best approach, you be within the allowable limits and possibly improve the efficiency and the reliability. It is possible to operate at the lower part of the quadrant during winter and free cooling energy while using the upper quadrant during those times when the chiller energy is to be used. It was a great decision by ASHRAE to open the envelope.

You may think of a VFD for fan modulation, it depends on the application and how the system is configured or, need an upgrade if it is existing, or the type of system (e.g. chilled water, DX). Everything will need controls to modulate the system for better performance. Air velocities at the coils are important, so it all depends on the application.

The main center I ran was 100% VFD, used water side economizer and was controlled by a temperature set-point AND pressure feedback. We ran the room initially without the containment being complete but once the containment was in place the savings were bold. We had done a full finite analysis before design was complete. Overall the cooling design was a extension of the fab style air flow used in the chip factories.

Adaptability and easy deployment were requirements of the design. We also already had enough gear to substantially fill the facility so we did not have to worry about overage on capacity, a luxury most likely would not have at the design phase.

Dave

Increase the inlet temp, control the cold aisle air velocity by VSD on the CRACS directly proportional to the inlet temp.....disconnect the server fans!! QED