Husam Alissa

Cooling failure in data centers (DCs) is a complex phenomenon due to the many interactions between the cooling infrastructure and the information technology equipment (IT). To fully understand it, a system integration philosophy is vital to the testing and design of experiment. In this paper, a facility-level DC cooling failure experiment is run and analyzed. An airside cooling failure is introduced to the facility during two different cooling set points as well as in open and contained… Show more

Cooling failure in data centers (DCs) is a complex phenomenon due to the many interactions between the cooling infrastructure and the information technology equipment (IT). To fully understand it, a system integration philosophy is vital to the testing and design of experiment. In this paper, a facility-level DC cooling failure experiment is run and analyzed. An airside cooling failure is introduced to the facility during two different cooling set points as well as in open and contained environments. Quantitative instrumentation includes pressure differentials, tile airflow, external contour and discrete air inlet temperature, intelligent platform management interface (IPMI), and cooling system data during failure recovery. Qualitative measurements include infrared imaging and airflow visualization via smoke trace. To our knowledge of current literature, this is the first experimental study in which an actual multi-aisle facility cooling failure is run with real IT (compute, network, and storage) load in the white space. This will establish a link between variations from the facility to the central processing unit (CPU). The results show that using the external IT inlet temperature sensors, the containment configuration shows a longer available uptime (AU) during failure. However, the IPMI data show the opposite. In fact, the available uptime is reduced significantly when the external sensors are compared to internal IT analytics. The response of the IT power, CPU temperature, and fan speed shows higher values during the containment failure. This occurs because of the instantaneous formation of external impedances in the containment during failure, which renders the contained aisle to be less resilient than the open aisle. The tradeoffs between PUE, OPEX, and AU are also explained. Show less

This article reports experimental and numerical testing performed to characterize the operation and reliability of the open compute (OC) storage system in contained environment from server to aisle levels. The study is comprised of three parts. The first part is an experimental analysis of the high density (HD) 3D array storage unit thermal and utilization responses during airflow imbalances. This is done with the stress test proposed for IT in containment to mimic possible mismatch and cascade… Show more

This article reports experimental and numerical testing performed to characterize the operation and reliability of the open compute (OC) storage system in contained environment from server to aisle levels. The study is comprised of three parts. The first part is an experimental analysis of the high density (HD) 3D array storage unit thermal and utilization responses during airflow imbalances. This is done with the stress test proposed for IT in containment to mimic possible mismatch and cascade failure scenarios. It is found that downstream HDDs are most prone to overheating and loss in utilization during an airflow imbalance. This was proven to undermine the storage capacity of the hard disk drives. An IT level airflow prediction model is discussed for the storage unit and validated for different fan speeds. In the second part, a computational fluid dynamics model is created for a high density open rack based on the active flow curve method. Here, the measured airflow response curves for the open compute IT(storage and compute servers) are used to build compact models and run rack level testing for IT air systems sensitivity and create a rack level AFC (active flow curve) airflow demand prediction model. Finally, the experimental characterization data is used to build an aisle level model (POD) that incorporates IT fan control systems (FCS). This modeling approach yields shorter uptime during chiller failure due to increased recirculation induced by increased IT airflow demand during cases such as chiller failure or high economizer temperatures. Show less

We developed a generalized testing methodology in which IT fans' internal and external impedance effects are collapsed into a single curve (i.e., flow curve). Such curves can give accurate flow rate predictions of the IT equipment based on RPM logging. Flow curves cover the three possible airflow regions for IT in a contained environment. The experimental procedure is introduced to rank IT equipment based on its air systems, and the flow regions are correlated with the CPU performance. The… Show more

We developed a generalized testing methodology in which IT fans' internal and external impedance effects are collapsed into a single curve (i.e., flow curve). Such curves can give accurate flow rate predictions of the IT equipment based on RPM logging. Flow curves cover the three possible airflow regions for IT in a contained environment. The experimental procedure is introduced to rank IT equipment based on its air systems, and the flow regions are correlated with the CPU performance. The strength of the IT air system is characterized by the free delivery and the critical pressure points. In addition, the experimentally obtained active flow curves can be used as a numerical boundary condition in various containment designs. Accordingly, for the first time in the literature, the aerodynamic interaction of IT with different air systems in the containment (i.e., BladeCenters, servers, and switches) can numerically be investigated. It is found that stacking IT in a contained environment with no physical consideration can lead to radical reliability problems. The region of reverse flow is discovered to be very possible upon events in the data center. In such a case, the weak IT (e.g., 1U switch) behaves as an open leakage into the sealed containment. On the other hand, at the critical pressure point, minimum forced convection occurs in the weak IT, leading to CPU overheating. Show less

In a previous study, the interaction of perimeter and row based cooling were contrasted in a CAC environment, however, emphasis is needed still on how each system behaves in entrainment conditions due leakage. Also, benefits of containing a row based cooled aisle has not been compared with the aisle in open conditions. Sets of IT that differs in air system strength are used as load to be cooled by both cooling systems. Imperially measured flow curves of IT equipment are employed to simplified… Show more

In a previous study, the interaction of perimeter and row based cooling were contrasted in a CAC environment, however, emphasis is needed still on how each system behaves in entrainment conditions due leakage. Also, benefits of containing a row based cooled aisle has not been compared with the aisle in open conditions. Sets of IT that differs in air system strength are used as load to be cooled by both cooling systems. Imperially measured flow curves of IT equipment are employed to simplified IT equipment model in the CFD code. This study numerically investigates the performance of in-row coolers in both opened and cold contained aisles. The steady state analysis includes manufacturer’s specifications such as heat exchanger performance. The CFD model was validated using a new data center laboratory at Binghamton University with perimeter cooling. A Libert® CW114 is the sole cooling unit in the data center. In this study, a sliced part of the data center is studied in order to generalize the results and reduce the computational time. 16 racks are distributed on two sides of the cold aisle that contains 2 power distribution units and built in full details in the CFD code. This study elucidates on the particular behavior of airflow in the cold aisle when row-based cooling is utilized. This is mainly attributed to the change in the location of the airflow source. Also, correlations are derived to predict the ability of ACUs to maintain set points at different air flows. Show less

In cold aisle containment (CAC) the supply of cold air is separated within the contained volume. The hot air exhaust leaves the IT and increases the room’s temperature before returning to the cooling unit. On the other hand, hot aisle containment (HAC) generates a cooler environment in the data center room as a whole by segregating hot air within the containment. Hot air is routed back to the cooling unit return by a drop ceiling or a chimney. Each system has different characteristics and… Show more

In cold aisle containment (CAC) the supply of cold air is separated within the contained volume. The hot air exhaust leaves the IT and increases the room’s temperature before returning to the cooling unit. On the other hand, hot aisle containment (HAC) generates a cooler environment in the data center room as a whole by segregating hot air within the containment. Hot air is routed back to the cooling unit return by a drop ceiling or a chimney. Each system has different characteristics and airflow paths. For instance, leakage introduces different effects for CACs and HACs since the hot and cold aisles are switched.
This article utilizes data center measurements and containment characterization carried out circa April 2015 in the ES2 Data Center lab at Binghamton University. Details on the containment model include leakages at below racks, above racks, below CAC doors, between doors, and above doors. The model deploys the experimentally obtained flow curves approach for flow-pressure correlation.
Data center operators rely on the pressure differential to measure how much the IT is provided. Hence, in this study the level of provisioning was expressed in terms of pressure differentials between the hot and cold aisles. In this manner the model reflected real-life DC thermal management practices. This was done by integrating a pressure differential based controller to the cooling unit model. Leakages in each system are quantified and ranked based on a proposed LIF (Leakage Impact Factor) metric.
The LIF describes the transport contribution each leakage location has. This metric can be used by containment designers and data center operators to prioritize their sealing efforts or consider deploying the containment solution differently. Finally, a systematic approach is shown in which containment models can be used to optimize operations at the real-life site. Show less

As the number of data centers is exponentially growing globally, pragmatic characterization schemes are considered to be a necessity for measuring and modeling the load capacity and flow pattern of the facility. This paper contains experimental and numerical characterization of a new data center laboratory using practical measurements methods, including tiles and CRAH flow measurements. Then a full physics based CFD model is built to simulate/predict the measured data. A rapid flow curve method… Show more

As the number of data centers is exponentially growing globally, pragmatic characterization schemes are considered to be a necessity for measuring and modeling the load capacity and flow pattern of the facility. This paper contains experimental and numerical characterization of a new data center laboratory using practical measurements methods, including tiles and CRAH flow measurements. Then a full physics based CFD model is built to simulate/predict the measured data. A rapid flow curve method is used showing high accuracy and low computational expense.
Detailed descriptions of the data center structure, dimensions, layout (Appendix, A-1) and flow devices are given. Also, modeling parameters are mentioned in details to provide a baseline for any investigative parametric or sensitivity studies. Four experimental room level flow constraint scenarios are applied at which measurements were taken, (Appendix, A-2). The model is then built and calibrated then used to predict measurements.
Measurements of the cooling unit were performed using hot wire anemometry with a traverse duct installed at the top of the CRAH. The tiles measurements were carried out using a flow hood with back pressure compensation. A detailed CFD model is constructed to predict the four experimental cases. For modeling the interdependency between the flow and pressure in flow devices flow curve approach is used. This is a rapid modeling technique that relies on experimentally measured (for IT) or approximated (for CRAH) flow curves. Applying the operational flow curves boundary conditions at the vents of the flow device results in a very accurate simulation model. It is also shown that the flow curves can be used to predict the real-time flow rate of servers at known RPM. This greatly simplifies flow rate measurements of IT in the data center. Show less

Because of the rapid growth in the number of data centers combined with the high density heat dissipation in the IT and telecommunications equipment, energy efficient thermal management of data centers has become a key research focus in the electronics packaging community. Traditional legacy data centers still rely largely on chilled air flow delivered to the IT equipment racks through perforated tiles from the raised floor plenum. When there is large variation in the amount of heat dissipated… Show more

Because of the rapid growth in the number of data centers combined with the high density heat dissipation in the IT and telecommunications equipment, energy efficient thermal management of data centers has become a key research focus in the electronics packaging community. Traditional legacy data centers still rely largely on chilled air flow delivered to the IT equipment racks through perforated tiles from the raised floor plenum. When there is large variation in the amount of heat dissipated by the racks in a given aisle, the standard air cooling approach requires over-provisioning.
Localized hybrid air-water cooling is one approach to more effectively control the cooling when there is wide variation in the amount of dissipation in neighboring racks. In a closed hybrid air-water cooled server cabinet, the generated heat is removed by a self-contained system that does not interact with the room level air cooling system. In this study, a comprehensive procedure for CFD validation in a close coupled hybrid cooled enclosed cabinet is described. The commercial enclosure has been characterized experimentally in an earlier study, where the effectiveness values were applied as boundary conditions to the compact heat exchanger model.
Here, the previously obtained experimental data are used to validate the results from computational modeling. Two cases with different air flow rates are compared. Very good agreement is achieved, with the maximum overall average error less than 4%. Due to relatively high pressure inside the cabinet, it is possible that air leakage from the cabinet may be responsible for the discrepancy between the model and experimental results. A sensitivity study was applied to the validated model to investigate the effect leakage had on the cabinet’s performance. Show less

A brief failure scenario is then run, and duration of ride through time in the case of row-based cooling system failure is compared to raised floor perimeter cooling with containment. Non-raised floor cooling schemes will reduce the air volumetric storage of the whole facility (in this small data center cell it is about a 20% reduction). Also, the varying thermal inertia between the typical in-row and perimeter cooling units is of decisive importance.
The CFD model is validated using a new… Show more

A brief failure scenario is then run, and duration of ride through time in the case of row-based cooling system failure is compared to raised floor perimeter cooling with containment. Non-raised floor cooling schemes will reduce the air volumetric storage of the whole facility (in this small data center cell it is about a 20% reduction). Also, the varying thermal inertia between the typical in-row and perimeter cooling units is of decisive importance.
The CFD model is validated using a new data center laboratory at Binghamton University with perimeter cooling. This data center consists of one main Liebert cooling unit, 46 perforated tiles with 22% open area, 40 racks distributed on three main cold aisles C and D. A computational slice is taken of the data center to generalize results. Cold aisle C consists of 16 rack and 18 perforated tiles with containment installed. In-row coolers are then added to the CFD model. Fixed IT load is maintained throughout the simulation and steady state comparisons are built between the legacy and row-based cooling schemes. An empirically obtained flow curve method is used to capture the flow- pressure correlation for flow devices.
Performance scenarios were parametrically analyzed for the following cases: (a) Perimeter cooling in CAC, (b) In-row cooling in CAC. Results showed that in-row coolers increased the efficiency of supply air flow utilization since the floor leakage was eliminated, and higher pressure build up in CAC were observed. This reduced the rack recirculation when compared to the perimeter cooled case. However, the heat exchanger size demonstrated the limitation of the in-row to maintain controlled set point at increased air flow conditions. For the pump failure scenario, experimental data provided by Emerson labs were used to capture the thermal inertia effect of the cooling coils for in-row and perimeter unit, perimeter cooled system proved to have longer ride through time. Show less

This study illustrates basic guidelines for integrating
empirical measurements to state of the art modeling
approaches; that can be used to improve the accuracy of
physics based data center CFD models. Experimental
measurements gathered in a new research dedicated data
center laboratory are used to guide an advanced approach to
CFD modeling, the intent is to come up with a validated
predictive model of the data center. One of the goals of this
study is to provide… Show more

This study illustrates basic guidelines for integrating
empirical measurements to state of the art modeling
approaches; that can be used to improve the accuracy of
physics based data center CFD models. Experimental
measurements gathered in a new research dedicated data
center laboratory are used to guide an advanced approach to
CFD modeling, the intent is to come up with a validated
predictive model of the data center. One of the goals of this
study is to provide scientists and engineers with a clear set of
design guidelines; that allow for the introduction of important
physical details into the model which have previously been
largely overlooked. This approach will establish a valid
baseline for future designs of data centers as well as future
research projects in the data center laboratory. The current
CFD model incorporates important physical parameters that
affect the accuracy of the results. These include the plenum
floor jacks’ hydraulic resistance, the impact of cutouts, exact
locations of supply vents, CRAH (computer room air handler)
unit blower jet effects and momentum transfer through the
perforated tiles. Other details of the CRAH unit such as the
detailed supply vents, blower specifications, turning vent and
angle of flow are included; also rack detailed structure
including the mounting rails, cable penetration and PDUs
(power distribution units). CFD modeling results are
compared to the measured tiles flow rates from several
different experimental conditions. The models were refined
based on the experimental data and comparisons of the CFD
versus experimental tiles flow data showed an overall average
error of ~3%. Show less

The purpose of under floor plenum in a typical raised floor
data center is to route the supply of cold air to perforated tiles
in the cold aisles, and hence, to the racks. However, the
presence of under floor chiller piping and various wiring may
have an adverse effect on flow rates if not placed based on
physical considerations; the pressure drop caused by chiller
piping and under floor blockages has not been investigated
thoroughly in modeling of a fully representative… Show more

The purpose of under floor plenum in a typical raised floor
data center is to route the supply of cold air to perforated tiles
in the cold aisles, and hence, to the racks. However, the
presence of under floor chiller piping and various wiring may
have an adverse effect on flow rates if not placed based on
physical considerations; the pressure drop caused by chiller
piping and under floor blockages has not been investigated
thoroughly in modeling of a fully representative real life data
center, and in particular, in applications where some or all of
the cold aisles may be contained. This effect on flow rate is
expected to be even more profound in contained systems; this
study aims to address the effect of under floor obstructions on
data center performance.
It was shown that when having blockages in critical
locations, containment can act as a solution for the inlet
temperatures at the racks, however, the blockage effect can
still be seen on the racks outlet and the CRAC return
temperatures. It was also observed that inadequate distribution
of those blockages led to a change in the operating point of
both the CRAC and severs fan curve reducing the flow being
fed to the IT equipment, and hence the chiller cooling load is
expected to increase, which results in a thermal deficiency of
the data center. Show less