Data center co-location electrical system design

Learning objectives

• Learn the differences between enterprise, co-location, and cloud data centers.
• Understand co-location electrical systems and factors that determine how they are configured.
• Review the benefits and considerations of co-location data centers. Compare against traditional enterprise data centers and cloud data centers.

In an ever-evolving data center market, owners find it very difficult to predict future information technology (IT) needs. How much white space—or equipment space—will be needed? What power density should be designed for? What is going to happen in the future? This is how every data center design conversation starts, followed by “How much is this going to cost?”

Co-location data center providers offer a unique solution to those questions about the future by allowing the customer to lease space for IT equipment as well as power, cooling, and communications bandwidth. If a company’s IT needs to grow, and it needs more data center space, the owner can lease it. If the company’s IT needs to shrink and doesn’t need as much equipment, the owner can always renegotiate at the end of the lease and give space back. This flexible, “future-proof” solution is one of the reasons why co-location data centers have become so popular and continue to grow. When is a co-location facility the right choice?

Enterprise data centers

Traditionally, corporate/enterprise data centers and data center operations were kept in-house. The land, building, and all equipment were owned, maintained, and operated by the enterprise company. These in-house-type of data centers are what the industry now refers to as enterprise data centers. For some companies, the enterprise data center can be a small server room. For others, it can be a large multibuilding facility.

The design and size of the enterprise data center is based on the needs of one particular business/end user. The electrical distribution system for an enterprise data center follows that same principle where the availability and reliability are dictated by the business case of that specific company. What is an acceptable level of risk? How much protection does the company need against failures in the system? How much downtime is allowed for maintenance?

Because of the wide range of answers to these questions, enterprise data centers come in all different shapes and sizes. In data center parlance, N refers to the number of components needed to meet the load. For example, an 1,800 kW data center using 1,000 kW uninterruptible power supply (UPS) modules needs 2 UPS modules to meet the load requirement (N=2). Smaller companies that can’t afford multiple levels of redundancy tend to have a single UPS backed by a single generator or the minimum equipment needed to meet the load requirements (N). Larger corporations that highly depend on connectivity will have fully redundant or double the equipment (UPS and generator) needed to serve the load (2N)  and  dual pathways throughout the distribution system to allow for concurrent maintenance, and isolated systems that automatically respond to failures.

For the purposes of this article, we will look at a typical electrical distribution system used in the financial sector (banks, investment firms, insurance companies, etc.) that still tend to use enterprise data centers for regulatory and security reasons. The electrical distribution systems for enterprise data centers usually comprise the following:

•  2N utility services: Fully redundant utility service with the capability to serve the critical load from either service.
•  2N UPS systems: Fully redundant UPS systems where the failure of one system will result in the critical load transferring to the other UPS system to remain protected.
• N+1 or N+2 generators: Due to the expense of a fully redundant generator plant, generators are typically provided as Need (N) plus one or two to allow for one generator to be out for maintenance and still survive a failure. Oftentimes, there is also a means to connect a portable generator to the system.
• 2N distribution: Fully redundant distribution pathways from the utility service to the server cabinet.
• Maintenance bypass: Distribution pathways that bypass major equipment like a UPS to allow for its maintenance without shutting down the downstream critical IT equipment (concurrent maintainability).

Co-location data centers

By definition, the prefix “co” means together, in common, and mutually. A co-location data center is a data center located in a common facility that is shared mutually by multiple tenants. A co-location or retail-type co-location data center also can be thought of as a hotel for server cabinets. The land, physical building, and all electrical and mechanical infrastructure equipment are owned by the co-location developer/landlord.

Business customers rent space within the facility to house their servers and other IT equipment. In this scenario, the landlord provides and maintains the electrical power, cooling, and internet bandwidth, and the customer/tenant provides and maintains the IT hardware within the space. A service-level agreement (SLA) defines the criteria for reliability/resiliency of the facility and its mechanical/electrical infrastructure.

There are other types of co-location, such as wholesale, where the entire building or infrastructure is built specifically for one client but operated and run by the provider. This article will focus on the retail type of co-location facility.

Unlike enterprise data centers, where the facility is designed to meet the needs of one end user, the co-location data center is designed to meet the needs of multiple end users/customers. Instead of the availability and reliability being dictated by a specific business case, they are based on the provider’s business model, which uses a specific algorithm, or set of calculations, designed to maximize reliability and availability while minimizing capital expense.

Co-location data centers earn profit by selling space. The lower the capital and operating costs, the better the profits. However, the co-location data center also must maintain a high level of availability and reliability. An unreliable data center that has failures or an unavailable data center that has potential for downtime will not be able to lease space to certain clients. Maintaining that fine balance between cost-efficiency and reliability is a market differentiator for co-location providers.

The electrical distribution systems for co-location data centers usually include the following:

• N utility service: Utility-service redundancy can vary between providers and locations. Due to the expense, it is common to see a single utility service with capability to add a second in the future. In an area where a second utility source is readily available or inexpensive, the provider may add the 2N utility service for increased availability.
  
• N+1 or N+2 UPS systems: UPS systems usually are arranged in some type of isolated redundant configuration (“catcher system”) where one or two UPS systems back up multiple units or a distributed redundant system where they use three systems to make/serve two.
• N+1 or N+2 generators: Due to the expense of a fully redundant generator plant, generators are typically provided as Need (N) plus one or two to allow for one generator to be out for maintenance and still survive a failure. Oftentimes, there also is a means to connect a portable generator to the system.
• Redundant distribution: This characteristic/approach varies between providers. Some providers use a full 2N redundant distribution topology. Others use a combination of 2N and distributed redundant architectures, otherwise known as “3 to make 2.” Both of these system topologies can provide for concurrent maintainability of equipment, but they vary in their level of stranded (unusable reserve) capacity and the level of operational management required.
• Maintenance bypass: Distribution pathways that bypass major equipment like a UPS to allow for its maintenance without shutting down the downstream critical IT equipment (concurrent maintainability).

One big difference between enterprise and co-location data centers is the shared infrastructure. With enterprise data centers, the main/primary electrical and redundant electrical systems serve only one end user’s equipment. In the co-location data center, the electrical infrastructure and redundant equipment is shared among multiple customers. If a tenant or data hall is using the redundant equipment during maintenance of its equipment, it may not be available for another tenant’s equipment in a failure scenario. Because of this shared infrastructure, some co-location providers add additional redundancy (N+2) to the facility for added reliability and availability.

Another difference is that co-location providers are in the business of leasing space to customers so they can determine the size of data center and when to build a new one. This allows them to set the need based on their specific business model and algorithm. Some providers will limit the need to four systems, meaning there is one redundant unit for every four base units (one redundant generator backing up four base generators). Other providers may expand that to save cost and set the need to 10 systems, meaning there is one redundant unit for every 10 base units (one redundant generator backing up 10 base generators).

These are both important aspects of the electrical distribution system to review when deciding upon a co-location provider. Having additional redundant equipment (N+2) and sharing infrastructure with fewer tenants or across fewer systems can help ensure that the equipment will be available when a tenant needs it.

Another component of co-location is how rental rates are structured. Some co-location providers establish rent based on physical space and a power-capacity allotment. Others actually submeter the electrical consumption and charge based on the power used. In either case, metering is very important to co-location providers and customers.

Cloud data centers

Cloud data centers are similar to the co-location data centers in that they serve multiple end users. However, unlike co-location data centers where the IT equipment is owned and managed by the customers, in cloud data centers the servers and hardware are owned and operated by the cloud service provider (CSP). The customer does not have direct access to the space or equipment and should not need to enter the physical data center(s) at all. Customer access to the server equipment is remote via the internet.

One unique aspect of cloud data centers is that the redundancy is provided at the software level and not the infrastructure level. Applications and storage are set up virtually so that any failure to one device can be remapped through the software to another device. In cloud data centers, redundant components/equipment typically are provided for availability (concurrent maintenance) and not reliability (protection against failures).

The electrical distribution systems for the server and storage applications in a cloud data center include:

• 1N or 2N utility services: Where available, fully redundant utility service with the capability to serve the critical load from either service is preferred. Cloud data centers can be very large in size, however, making it difficult to obtain full 2N redundant utility feeders.
• N or N+1 UPS systems: UPS systems can vary between cloud providers. Larger cloud service providers use battery backup built into the server and avoid a central redundant UPS system. Others that do not have that technology will use an N+1 isolated redundant-type UPS configuration.
• N+1 generators: Generators typically are provided as need plus one to allow for one generator to be out of service for maintenance. Oftentimes, there also is a means to connect a portable generator to the system.
• Single distribution with maintenance bypass: Distribution pathways that bypass major functions to allow for maintenance without shutting down the equipment (concurrent maintainability).

Cloud data centers are similar to co-location in that they can determine the size of the data center and set the Need (N) based on their specific business model. Because the redundant equipment in cloud data centers is more for availability and maintenance, the need or base systems in the N+1 configuration is set much higher in cloud data centers (12 to 16) than for co-location (4 to 10) or enterprise data centers. Meaning there is one redundant unit for every 12 to 16 base units (one redundant generator backing up 12 or 16 base generators).

Because customer access to the server equipment is done remotely, through the internet, connectivity is very important in cloud data centers. Therefore, the portion of the electrical distribution system that serves the network systems in a cloud data center tends to have full 2N redundancy (2N UPS systems, 2N generators, and full 2N redundant pathways).

Benefits of co-location

Capital cost: New enterprise data centers can have relatively large capital costs associated with the land, building, and infrastructure. In a co-location data center, this capital cost is borne by the co-location provider. That large capital cost is spread across all customers and built into the lease payments so that it is not an upfront cost. Many older, larger enterprise data centers with equipment reaching end-of-life face significant capital costs for replacement. If the data center was not originally designed to facilitate equipment replacement, potential disruption could also result.

Lower operating cost: Cooling and power costs are spread across multiple customers. With this volume of customers, co-location providers can negotiate better equipment, utility, and maintenance contracts than a single company can. To minimize capital costs, co-location providers tend to use equipment to its maximum capacity. Customers may pay to reserve capacity but not use it. Co-location providers will share that unused capacity with other customers until the particular customer is ready to use it. By operating equipment closer to full capacity, the operating efficiency is improved to further reduce operating costs.

Lower maintenance costs: Maintenance costs and the cost of dedicated personnel to maintain the equipment is shared between multiple customers.

Quantity of scale: Co-location providers with many facilities get better pricing on equipment, such as UPS, generators, and switchgear, due to the quantity of equipment they are purchasing.

Redundancy: Smaller enterprise customers may not be able to afford redundancy and multiple distribution paths for concurrent maintenance. Sharing the redundant equipment also means the cost is shared between customers. This gives the smaller companies some level of redundancy at a more reasonable cost.

Rightsizing: Customers can lease the correct amount of space without the fear of running out of space in the future. If they need more IT space, they can always lease more space. Co-location also mitigates the risk of overbuilding a large facility and not using the space or utility capacity. In addition, many larger Fortune 500 IT users require a mix of reliability levels where some applications do not require the same high level of resiliency as others. Those less critical applications can be moved to a co-location facility, and the enterprise data center can be rightsized for just the critical load that requires full 2N redundancy.

Speed to market: Co-location providers with multiple facilities can standardize their major equipment types and have equipment in the pipeline manufactured before it is needed. That equipment can be shipped and installed wherever there is a need to expand without long lead times. This can be beneficial to larger Fortune 500 companies that can’t wait for the time required to build a new enterprise facility.

Change in reliability requirements: There also are many cases where the reliability requirements for the larger enterprise user change over time. In the past, they may have built large facilities with full 2N or 2(N+1) redundancy, but the 2N requirement and the projected load capacity are no longer required. Often times this change happens at the same time the equipment in the facility needs to be replaced due to it age. In this scenario, co-location can provide a great opportunity to downsize without interruption or overspending on replacement equipment.

Things to consider

The ideal data center includes electrical and mechanical infrastructure that meets the following criteria:

• Reliable: Keeps the critical load operating though maintenance activities and equipment failures.
• Modular: Alignment of system capacities for cost-effectiveness and maximum use.
• Scalable: System set up to be readily expanded without interruption to ongoing operations.
• Flexible: Provide for a range of power capacities and customer requirements across the space.
• Adaptable: Ability to transition to alternative emerging technologies.

For some companies, co-location data centers can be the answer to help to check off all the above boxes. For others, co-location may be just one element of an overall hybrid strategy. A co-location data center may provide disaster-recovery space away from their primary enterprise data center or a quick expansion space to accommodate unforeseen system growth or a new technology. Co-location may provide space for less critical applications, creating room in the enterprise data center for higher-reliability applications or applications with stricter security requirements.

Not all co-location sites are the same. Make sure to closely review the electrical distribution system to understand redundancy levels and how the IT equipment will be protected during maintenance activities and during a potential failure. Make sure to review the co-location provider’s lease agreement and SLA to ensure there are no surprises or hidden charges. Do they allow for a certain number of outages per year, or do they guarantee uninterruptible service regardless of the outage? Are there any penalties if the SLA is not met? Review the setup and licensing fees.

Be sure to perform due diligence, do your homework, and read the fine print to know exactly what you are getting from the co-location provider. This will help you determine if co-location makes sense and which co-location facility/provider is the right one for your application.

Kenneth Kutsmeda, PE, LEED AP, Jacobs, Philadelphia

Author Bio: Kenneth Kutsmeda is an engineering design principal at Jacobs. For 20 years, he has been responsible for engineering, designing, and commissioning power distribution systems for mission critical facilities. He is a member of the Consulting-Specifying Engineer editorial advisory board and a 2010 40 Under 40 winner.