The FedEx green data center’s P.U.E. stands at 1.28. Google tallies an average P.U.E. of 1.19, while Facebook expects its Prineville, Oregon facility to earn a P.U.E. of 1.15.

Not bad... But not good enough. SGI's ICE Cube Air to run with outside air and evaporative cooling in most climates, achieving a PUE of 1.06 or less. Containerized Data centers are 1/10th the cost of fixed data centers. That means I can purchase 10 equally equiped data centers for the cost of the one in Colorado Springs.

I think it's all in the education. Many people beleive fixed data centers are the defacto standard. Microsoft doesn't think so. There containerized data center in Illinois is very efficient and many times more affordable than fixed data centers. I give Microsoft accolades for leading the charge for a greener change.

Recycling the data center. Can Microsoft recycle its data center never letting its infrastructure get out of date... Yes, Microsoft had the insight to see that moblie containerized data centers were smarter and greener than fixed data centers whose infrastructures are far less flexible. Plug in a new container... Recycle the old container.
I my humble opinion Microsoft got it right.

http://www.microsoft.com/showcase/en/us/details/bafe5c0f-8651-4609-8c71-...

Great post Karl - thanks. I wonder what LEED certification they are going for, and what FedEx will achieve.

Hi Ron,
I'd be interested to hear some figures behind your statement "Containerized Data centers are 1/10th the cost of fixed data centers."

Ron, I've always wondered about those stated PUE's. Perhaps FedEx's data center's PUE is the expected annual average, not a single shot value in the best possible operating mode? A 1.06 is possible, I've seen in done, single shot, with no UPS factor in, no accounting for operation outside of the economizing window, and sometimes no lighting. Most of the numbers only account for fan energy usage to move the air through the box, and even that seems pretty agressive. (we are after all only talking about 60 watts of fan overhead for every 1000 watts of IT usage)

Would it be possible for you to lay out the overhead to reach the PUE number for an SGI Ice Cube?

Dan, you can ask SGI for their white paper or better yet ask one of their sales guys who will give you the information. I know it is not single shot. Systems thermal was not my gig at SGI. I did the majority of design. Of the Rackable Systems branded products, I design 70% of what shipped between 2003-2009.

How do they achieve the power numbers ? I know that in DC applications they are using a cluster of 3 x n Emerson high efficiency power supplies which can be strapped without the use of steering diodes. This is done by syncing the PWM's. What you end up with is perfect phase balance and much smaller transformers. The Emerson AC/DC supplies at the rated loads are ~96% efficient. This single conversion to 12 Vdc distributed to each server provides the lowest power dissipation in the industry. As far as lighting and the higher level stuff, I know they used the low wattage high frequency florescent lamps. Because of the density of the ICE Cube they use a 1/5 as many lights. I also know that the Cube blowers they use provide much lower power usage than traditional data centers. Short runs means they don't use nearly as much power.

Why would Microsoft who has so many fixed data centers go showcase the most efficient data center in the industry, their words not mine. They made a huge deal out of the fact that there new data center was cutting edge. This was due in part to Chicago not wanting a huge data center that could not be retrofitted, which would spawn other data centers down the road. They wanted them not to spread out. So with the modularity they can upgrade the containerized data centers and completely recycle / refurbish them. A much better way to go IMHO.

Andy, Just contact SGI and they will provide you with white papers. FYI, a standard 40ft container can be purchased here in California for $2500-.... depending upon whether it is just a bare container or kitted. How much is it that they spend per square foot for traditional data centers... Just the building ? The numbers are staggering.

Ron. You mention Microsoft and containers, I have looked at the ITPAC, and indeed its radical, but look at how many clients have SLA's.......not many.
Also, only a small % of any containerised solutions have any form of redundant power, the majority at best meeting tier 1, so a like for like PUE stating containers is a myth.
I think we have to go back to the bare bones and consider whats included in the PUE and provide annualised figures, and not a snapshot's in time in the coldest winter days/months.
I believe containers have there place, but depending on client demands, security, redundancy, etc, fiixed DC's are here to stay IMHO.

Kevin, you nailed it more succinctly than I did. They are going to complement, but likely never replace fix data centers.

The only other place I have seen direct rack level cooling which gives Containerized Solutions the cooling advantage is at Yahoo where they use Over the Cabinet crack units. I know that Leibert and Dell sell cabinets configured in this manner.

I like the term radical as if "change" is a scary dangerous thing. When someone says "radical" and lower cost of ownership, lower operating costs then radical is truly apro pos.

In my prior post, I gave away the keys to the kingdom when I wrote about the Emerson Rack Level 12 Vdc single conversion technique. Single conversion should absolutely replace double conversion and the losses associated with the arcane double conversion techniques for power distribution. Phase balance / equal loading at the rack level was always ad hoc at best. No with the advent of Rack Level Power Conversion a new "RADICAL" paradigm has been introduced.

So lets talk more about "RADICAL" Rack Level Power Conversion and the MTBF and multiple points of failure. In traditional cabinets multiple points of failure is the reason for not using centralized power conversion. The trouble with this thinking is they miss the MTBF numbers of the larger power Emerson Power converters and the typical power supplies used in servers. At 96% efficiency the amount of heat that is dumped by the Emerson supplies is very small. Let us further consider that the Intel SSI specification says 75-85% efficiency. So we are dumping typically 15-40 Watts more per Rack Unit Space than the Emerson single conversion Power Supplies. 1U power supplies are choked for air. This means that the capacitors which are used to filter the DC power for ripple are dried out and then fail. 1U power supplies are the highest point of failure in cabinets today with HDD's that are less than 10K Rpm.

Understanding Why Containerized Data Centers are more efficient is key not to call the power saving idea "RADIAL". Lower heat dump per cabinet is not radical, It is responsible. Responsible people make responsible decisions. One of them is not to cling to tradition because "that is the way we have always done it".

Other technology is the use of chilled water from the environment then the "responsible" method of using settling ponds which sink the heat before the water spills back into the water source.

To me huge data centers are wasteful. They have long haul HVAC systems which push / pull air which is $$$ thrown away. Going back to the Yahoo crack method. Yahoo was responsible for installing crack units above the Cabinets.. Not radical...

I suppose ones perception is really about ones experiences and understanding. This is why I have taken the time to give.. what I know.. and why it offers the benefits it does.

Ron.
Radical, as its a change from the norm....I never said it was bad....only it serves a purpose, but not everyones.
Indeed, in mission critical services, of which, Yahoo, Microsoft, Google are not in the main part.Redundancy, back up power, etc are required and always will be.
We all strive to make efficiency gains for all the right reasons, I have been labelled radical in my approach to cooling at Parthenon, no chillers, 100% free cooling, deep water heat storage fro reuse, etc, etc etc, etc.
But its for mission critical services, indeed take out what you can, great, all good what you are saying, but when the customer asks for the availability calcs, it has to make sense, as signing an SLA is serious stuff!!
Ask the aforementioned companies, I have, they dont sign too many, indeed, when it says on Google, oops retry in 48secs, I dont throw a wobbly....my clients will!!!

So one trend I've been noticing with cloud type providers, is that the infrastructure redundancy isn't important, but having redundancy in compute capacity at other site is. It seems many of the providers are building DC sites all over, and if one goes down, or is operating above it's thermal limits due to outside air temps, it's no big deal as the load can be shifted to other DC that are operating cheaply. (air side or water side economized) Organizations like FedEx in this example can't do that right now, hence the focus on reliability, and the higher PUE values that come with it. That trade off is more than acceptable to enterprise data centers it seems for the time being. In the end, comparisons between a Facebook DC and a FedEx DC aren't valid, as the mission they have is different.

Daniel. Exactly, but does not mean its wrong!!
Indeed, that seems to be the trend. Tier 1 facilities, using other Tier 1 facility's to provide the redundancy.
Its interesting when selecting cloud for example with Amazon, no mention of where your data is stored, only a page saying that all facilities have the green light, and indeed, no SLA.
Outages have occured, and then some corporates have shouted saying they went down, but shame on them if they have no idea where the data is stored, or take the time to investigate the infrastructure.
The word cloud is a hype, but its real, its a data center......but where, thats the question, and until some of the big boys start to say what they have and where, then the uptake of cloud will be slower than expected.
There is a place for every type of service in this world, the more conservative will always want proof,and demonstration to protect services requested in SLA, others will have less stringent requests, and can take an outage!!
Pay your money and take your choice.

Kevin what method of humidity control do you use with your 100% Free Cooling?

Dave. Ultrasonic. No dehumidification is used. We analyise external ambient and water vapour levels and when required we shut down the IT space ventilation.
This only occurs at 11gr/kg.........very, very rarely.

Thanks Kevin, what operating temperatures and RH within the facility, and where it is located. We have designed and installed a Free Cooling system for one of our customers, we use the ambient air backed up with chillers for the odd days when the outside ambient is above 25% (not that often in the north of the UK) but use adiabatic to maintain RH, we did look at Vapac but felt it was to much of an overhead We are maintaining 22 degrees plus or minus 2 and RH of 40 - 55% but i am still looking for a better way of controlling RH?

Dave, if you look at an indirect approach (airside economiser) to using outside air, one that offers the ability to use the low temperatures but does not transfer humidity levels from outside into the room you will find that, in many climates, you will get many more hours of free cooling than you do by using direct airside economisers. Doing it this way makes sure you don't take the risk with gas and particulate contamination and your humidification overhead, both in terms of equipment size and running costs, is very small.

Dave.
Tony is right. I hope I did not confuse you with saying free cooling as opposed to free air cooling. We use indirect method of free cooling. Humidity is no problem, but have the ultrasonic if needed.

Kevin/Tony thanks for the replies

Andy, The 40 ft container I designed was ~ $100-$150 k fully kitted minus transformer(s). That's lighting & electrical, chill water, two post racks, flooring, Insulation, and a partition. The total rack space was 800-1000 Rack Units. I am a big fan of 480 / 277 than 208. I know I will be called radical but all modern building lighting today uses 277 because there is 60-70% savings in copper. Also transformers are much smaller and cost up to 1/2 as much as 208 trannies.. If one were to employ direct conversion 277 to 12 Vdc this would also yield a aggregate cost savings of $750-$1500 per 40 Rack Units, depending upon the loads. Server depths should be relegated to ~ 20" and use 2.5" HDD's.

Compare this to data center and we are talking up to 10 times as much money for a similarly kitted data center. Lets say for the above container with average 50 Watt processors because this gives the best MIPS per $$ we are talking about $5 million customer cost. How much is it for a similarly kitted data center... ?

One of the best design features of the mobile data center is it is completely recyclable / upgradable. The data center can easily expand if necessary. It can expand up or out. Because of it's inherent modularity and flexibility data center managers need not be as worried about growing of of their data centers every 3-5 years. With the rising energy costs, they provide the densest cooling solution which saves money. The more air you have to push over a distance costs lots of $$$.

Lastly, I would like to add, even though it not be part of this thread, instant on servers. I know that most everyone has seen or heard of the Macbook Air which uses flash to save the state of the machine so it can be literally turned on in an instant. Boot times cost lost of money to handle data center loads. A Software industry has sprung up to manage this to effectively handle this problem. I believe there is no reason why all servers today shouldn't be "state" machines. This is responsible computing. Instant on / instant off. Saving $$$ in power.

Wondering what kind of ROI was realized, what type of funding realized (gov stimulus, green credits, tax credits, etc.)? It seems going green may be very expensive unless the tax payer funds most of it.

Ron, those are good numbers, that's around $320-475 a square foot best I can tell. Now I normally see total data center builtouts running around 1.2K a square foot:

http://www.datacenterknowledge.com/archives/2010/03/26/cloud-economics-b...

This includes chillers, generators, UPS systems, transformers etc in the number from data center knowledge above, I'm not sure how those numbers come in when you add them to the container numbers, but I'm betting they are close. There maybe other advantages, just not sure what they are at this point.

I agree entirely on instant on serves. Stated servers will liekly be the next big thing, scaling your computing requirements seemlessly to your hardware would allow for savings that are far greater than gains that can be made on the cooling/infrastructure side. That means a total cloud computing infrastructure though at some point I would imagine, and after the Gmail hiccup this weekend, is the adoption rate going to keep up it's steam?

Ron, For $150 K what are you missing? Does this include any mechanical loop to support cooling medium? I assume you would require ancillary chilled water or pumped refrigerent cooling since free air method is rather limited. Did you include the rack based cooling in this price? How about insulation, remote monitoring? At what capacity is your design? Seems really low. I assume you are not offering a turn key solution at this price.

The chill water / heat exchanger is external. I gave a price range for that being $35 k + depending upon a number of factors. I would much rather keep the Containers close to sreams / rivers with cold water. A settling pond would be required. The downside is being close to fiber.

As far as the $150 Container price. This is for all the bits. It is not a selling price. It is for your cost if you were to procure them yourself. It is not the cost of them fully assembled. What is the caveat ? Many 1U servers have a depth which is over 20". 26-28" are typical. A 20" server allows only 2.5" HDD's to be used. So consider this when we are talking about 800 Rack Units. Designing a custom 2U half depth server would be cosidered non standard but their are folks who do these shallow depth servers like SGI and IBM.

Also, I like to err on the side of cautious and I would not design a container with more than 800 Rack Units. This would give me ample space for the heat exchangers and lower the rack / chiller fan speeds.

800 RUs, even at only 500 watts per U is going to 400kW or north of a 100 ton chilled water plant / HEX. That's going to add around 100k to the price tag based on an air cooled chiller design, perhaps half that if it was only a component of a larged water coold plant. Add generators for 400kW and the UPS system to support, and your at the cost quoted by Terremark for a traditional build out, perhaps more.

I've never seen a cost model for containers vs. traditional build out. Anybody have access to a whitepaper, or know where to look?

I would say 100K for the chilled water plant would be low. There are many factors to consider with chiller plant for a modular solution. Most companies deploying such solutions require redundancy on the chiller system. Also, would not suggest pond or stream water for a high efficiency chiller. The mechanical system would be at higher risk for failure. Also consider if your environment is low ambient and water can be much lower than 45 degrees you would require at least a 1500 gallon stroage tank to heat the water to minimum entering temp and cool 400 KW with thermal dissipation if power were cut and pumps were on UPS.

$35k for a heat exchanger / pump hoses.. This takes into consideration that you already have access to chill water... from lake / stream..

I find it interesting how Ron's brief reference to Containers took everyone into a discussion on C/MDC (Containerized/Modular Data Center) solutions. Kevin I agree with ypu that C/MDC is nt for everyone but you are dead wrong about the redundancy.
I have yet to find a suplier where full "A" & "B" circuiting is nt availabe.
All that being said I invite you all to:
1. Join my C/MDC (Containerized/Modular Data Center) discussion group on LinedIn
2. Read Bruce Myatts column in the Jan/Feb issue of Mission Critical Magazine (p14)
3. Watch for my follow-up in the May/June edition of Mission Critical Magazine

Lastly, get ready for the explosion in C/MDC offerings. As of today there are over 20 vendors with product in the market. i/o just built a factory that can pump out 48 units a month - that's 576 a year from just one vendor.
There is a heck of a lot interest in this Data Center sub market. Even Colo's are getting into it. So stay tuned and be ready as your next data center just might be factory built!

Dennis.
I think I said, ''a small % has redundancy''......!
But always happy to be proved wrong.

Dennis.
I feel that we also have to make a distinction surrounding containerised solutions and POD technology.
I have seen containers dropped in the middle of a field, and PODs being built on site, inside and/or outside a secure environment.
Indeed, we are all looking at a fast time to market with short build times, and POD type building is being carried out by most, and this is not a knew thing, its a technology thats been around for decades in other industry.
For example, I have seen industrial freezers of approx 300m2 built in 5 days, using coated aluminium, interlocking sheets with 5cm of compacted insulation between, ideal build type for a data center.
There are many more examples of fast builds, we just have to embrace all these ideas and have an open mind when some would say, its a freezer building not a data center....why??
Technical fit out is then client preference, as much or as little as the they want.

Kevin, we have used the composite panels for a rapid deployment in a warehouse environment to build a data centre. Not only was it a quick deployment it also provides a relatively easy expansion path due to the interlocking method of securing the individual sheets with no dust generated by materials such as plasterboard.

If you see an opportunity, we would like to offer our LED Lighting solutions for your Data Center - Energy Efficient program. You can reach me at stevan.bratic@bratic.net with any questions.

Excellent article today in DCDFocus on the subject at hand

http://www.datacenterdynamics.com/focus/archive/2010/10/what-will-modula...