Previously Published in LAN M agazine, November, 1994

© 1994 Douglas E. Welch

Just because many companies are moving away from mainframe systems and into client/server technology it doesn't mean they should throw out everything they learned from their mainframe years. Consolidating servers in a centralized location with specialized power, security and cooling systems is more than a good idea. It can prevent costly downtime, increase productivity and protect the data that keeps companies profitable.

This article will outline the basics of setting up a LAN server room from the ground up. It makes good business sense to operate LAN servers in an environment specifically designed for them. Centralized servers are easier to install, maintain and service. Too many servers spend their lives in cramped closets or break rooms exposed to extreme heat fluctuations, dust and spills and surrounded by users unaware of the systems' importance. Physical security for most of these servers is nonexistent and network support staff are often forced to physically visit a site to repair hardware or software problems.

As with any project, planning is the most important part. Good planning will prevent problems throughout the entire process including design, budgeting, sign-off and installation. Bad planning can produce an unworkable server room or prevent it from being built in the first place. Throughout this article this rule will be repeated many times: plan for more capacity than the company may require in the future otherwise many sites will find their server rooms running low on space, power and cooling long before it was expected.

The Building

Selecting a site for your server room is fairly straight forward. The site should be centrally located to any other company facilities and as close as possible to any existing power and communications conduits. A new space is obviously superior to an existing site as the new design will not have to adapt to previous designs for the space. If the server room is being created out of an existing building it is best to plan on gutting the area entirely. In some high security cases such as banking companies it might be preferable to locate the server room or even an entire computer center far from the central offices. If this is the case, be aware that there will be additional cost involved in connecting the sites together with high-speed communication lines.

Any LAN server room should be an island within its enclosing building. It should have its own security system, power supply and cooling. This allows the server room to continue operation even if the standard building systems fail. If an employee short circuits an electrical outlet in a break room it shouldn't bring down the entire network. The nature of the server room also dictates specialized services for water detection and fire suppression.

One of the first decisions to be made after selecting a location relates to the actual usage of the room. Server rooms are designed for the comfort of the equipment they contain and not the comfort of humans. Don't try and share server room space with help desk operations or other services that require constant staff attention. Workers will soon tire of the cold and noisy environment and valuable equipment space will be sacrificed to non-network related use and storage.

A decision must also be made regarding the life span of the site. It doesn't make sense to spend large sums of money on a room that will be abandoned 1 or 2 years down the road. The most realistic payback on a server room of any consequence is somewhere between 3 and 5 years. If the company is planning on a move or a major change in business a server room may not be a viable investment. It would make more economical sense to establish better conditions for servers that are distributed throughout the company. Get them out of their dusty closets, provide them with backup power and train someone on-site in server operations.

The life span also effects decisions regarding the size of the server room and all its related services. Planning must allow for any equipment that might conceivably be placed in the room over its life span. While the initial plan may call for 4 servers it is very possible that the company may end up with 8 or more servers over several years. Considerations also need to be made for related equipment as well. There are gateway machines, tape backup units, routers, hubs and switches that will take up space, generate more heat and require even more network and power cabling. Design the server room so that it can provide for any future plans and then add another 15% to 20%. While that may sound outrageous remember the old rule that "data expands to fill all available space" and network equipment seems to follow the same rule. Even the most liberal views of future requirements will probably fall short of reality.

Flooring

Most mainframe installations made use of some form of raised flooring and it provides many benefits for a server room as well. First and foremost it allows for easy management of the miles of cable that will eventually be installed. Cables can be easily accessed and routed merely by pulling up a few tiles at a time. Raised flooring also provides for easy reconfiguration of the room. The arrangement of equipment in a server room will change quite regularly and a raised floor will make it less painful to move systems or group them in a more logical manner.

Finally, a raised floor provides for cooling all equipment in the room. The most important part of planning for a raised floor is to ensure that it is deep enough to hold all the cable and still provide 6 to 8 inches of space for air flow. Type 1 cabling is much larger than 10Base-T and will take up much more space. Don't cramp your raised floor as it will only create problems with cooling later. If the site is in an area prone to earthquakes there may be some restrictions on the height of the raised floor. Floors over a certain height will require cross-bracing to insure stability and other restrictions may apply as well. Check with local building departments to insure that the site is installed to code.

Cable trays

Network cabling should not just lie on the sub-floor of the server room beneath the raised floor. Cable trays should be provided throughout the room to provide standard paths for all cabling in the room. These standard trays should be arranged in a flexible grid that can be adapted as needs change. Figure 1 shows a sample arrangement of cable trays for a rectangular server room.

***(Cable tray figure here) ***

The cable trays should be spaced off the sub-floor by at least 1/2" to allow for any minor water spills that might occur from air conditioner condensation or minor wall or ceiling leaks. These trays should be connected to the same ground as the server room power system to reduce any shock hazard since the electrical wiring will also run through these trays.

Cable trays provide for "good housekeeping" within the server room. Nothing is more confusing than a rat's nest of cable strung diagonally over each other from unit to unit when a network staffer is trying to troubleshoot a problem. A server room is supposed to make network management easier. Those sites that don't install cable trays or fail to use them properly will find that decision come back to haunt them in the future.

Cabling

Regardless of the type of cabling that will be used there are some general rules to guide the cabling of the server room. After the cable trays are installed but before the raised flooring, racks and equipment are put in place the entire room should be pre-wired. Pre-wiring the room provides the wiring contractor a wide open space in which to work and to logically lay out the wiring in the room without trying to work around existing equipment and obstructions.

The initial arrangement of the room, such as where the racks will stand and where servers will be located, should already be completed. Take the time to pre-install cabling for each server, hub and router that might eventually be installed in each rack or work area. Wire each rack and work area to the total planned capacity. Even if there will only be 1 server in a rack that can hold 4 servers install the cable for all the servers. Extra wires can be bundled together in the racks and in the cable trays and placed out of the way until needed. Installing all the wiring now prevents the major problems associated with running cable after equipment is installed.

If fiber optic cabling is to be used in the server room there are a few special considerations. Due to the somewhat fragile nature of fiber optic cabling it is best to run any fiber through a conduit that has been run inside the cable trays in the floor. This will prevent the fiber from being buried under large bundles of copper cabling and protect it from being twisted together with all the standard network cables. Fiber is a major investment and should be protected as much as possible. Using conduit for fiber optic cables is just one more step towards good housekeeping as described above.

Another consideration is the installation of a high-speed backbone specifically for the server room. High-speed protocols and equipment can provide a high-bandwidth path for server to server, gateway and backup communications. A server room backbone helps to isolate the high traffic environment of the server room from the rest of the network. Nightly backups, mail propagation and database updates can create large amounts of traffic. If this traffic was allowed to propagate out to the normal user network it is possible users would see slowdowns in their daily work. No one likes a slow network and a server room backbone can help prevent user complaints and management questions while making the operations as fast as possible.

A server room backbone can take many forms. Some sites might select fiber connections and the FDDI protocol while others may decide on Fast Ethernet and switched hubs to provide this connection. Each site should decide what is best for their environment. Details such as integration with existing equipment, internal research and future planning will dictate what system should be installed.

Racks

The decision to use equipment racks is often a hard one. They can be unwieldy if equipment is not specifically designed for them and they may not come in the right configuration for the equipment to be installed. While it is possible to add racks at a later time it will almost certainly require a re-wiring of all power and network cabling to fit the new configuration.

It also depends on how much equipment is to be installed in the server room. If every square inch is required, racks may be the only way to go. While they save space they do tend to make it harder to install, maintain and service equipment. Make sure you leave enough room both in front and behind racks to allow for easy access. If the only way to get all the necessary equipment into the room is through the use of racks it would be wise to reconsider the design and perhaps increase the size of the room. Remember the 15%-20% rule. It is very rare that equipment will be removed from the room but very likely that equipment will be upgraded and new equipment will be added.

If racks are used it is worthwhile to look for equipment that is specifically designed to be rack mounted. This will save precious rack space and often includes features which will improve their ease of use such as connector placement, case design and ventilation.

A side note involves the use of monitor and keyboard switches. These switches allow one keyboard to be used for a number of different systems. Be sure to leave space for these items when configuring your racks. Despite the level of automation in today's network world there are still times you need to have direct access to the server.

Power

When dealing with electronic equipment nothing is more important than the quantity and quality of the power delivered to the server room. Computer systems and their components are very sensitive to fluctuations in power such as surges or brown-outs. If there is not enough power delivered to the server room network staffers will constantly be juggling systems to prevent breakers from tripping due to excessive loads. As mentioned before, plan ahead for the future power requirements of the server room so that it can grow with the company.

Deciding on the amount of power is fairly straightforward. Each device in the server room will have a power rating in amps. Usually this is stamped on a label on the device or listed with other specifications in the documentation. Add together the amp ratings of all the equipment that will be installed in the server room. Next, add together the amp ratings of all the non-computer related equipment that will be placed in the room. This includes fax machines, lighting and cooling equipment. Add these two numbers together to get the total number of amps used by the server room. The electrical service for the server room should then be 15%-20% above this number. Electrical contractors can assist you in making these calculations and recommending appropriate electrical service.

Remember, these calculations should be made considering any equipment that may be placed in the room in the future. There may be only 4 servers today but when the next 8 servers are installed the power will need to be there to support them. Don't scrimp on power. Doing so could limit the future capabilities of the server room and remove the flexibility that will allow it to grow more useful over time.

Backup Power

Backup power is also an important concern. Most computer equipment cannot be turned off without first going through some form of shutdown procedure. Power failures can damage equipment and cause the loss of data. While some companies may have backup power units, usually called UPS (un-interruptable power supplies) for each piece of equipment, a server room should have one central backup power unit for the entire room.

Calculating the size of a server room UPS relates back to the electrical load calculations mentioned above. The one exception is that all non-essential loads such as lights will not be connected to the backup power. UPS's are rated according to the amount of load they can provide and how long they can provide it. For example, the UPS may provide 100 amps of power for 30 minutes. Non-essential systems would only increase the requirements and wouldn't provide any major benefit during a crisis. Again, make sure that any backup unit can support the highest load that it might be subjected to in the future. If there are 200 amps of load in the room with the UPS mentioned above then this UPS would possibly only give 15 minutes of service or less. In fact, it might not even operate at all if the excessive loads trip circuit breakers.

Backup power is usually installed so that any major fluctuation in regular power causes the UPS to engage. There are two different scenarios though when it comes to wiring the UPS into the server room electrical service. In one arrangement all the equipment in the server room is run directly from normal power. If normal power fails for any reason the UPS switches in and starts providing power for the essential units. In the second arrangement the server room runs on battery power that is constantly being recharged by normal power. This method insures that the UPS batteries are always operating properly. It will be known immediately if they fail as the server room will automatically switch over to normal power service. This prevents an embarrassing and potentially damaging situation where backup power is suddenly required but the UPS batteries are flat dead due to an undiscovered problem.

(*** UPS wiring diagram ***)

A high-end solution for those sites in areas with chronic severe weather, earthquakes or other natural disasters is the installation of a backup generator. Major disasters can knock out regular power for long periods of time and a generator is almost a requirement for mission critical systems. If management will not approve a generator in the initial design it is best to at least provide for the automatic transfer switch so that a generator could be added at a later date.

Whatever type of backup power is used it is very important that the UPS be able to communicate its state to the various network equipment. Most servers provide for automated shutdown when they receive a notice from the UPS that it is almost out of power. This shutdown procedure protects these systems from the damaging effects of a power failure. Make sure the UPS can provide this communication to all the necessary devices in the server room. This will provide for maximum protection even when the server room is not manned, such as evenings and weekends.

Cooling

The most important aspect of any server room, after power, is cooling. The equipment in a server room generates a large amount of heat. With no cooling the temperature in the server room would quickly exceed the operating temperature specifications of most computer equipment. Cooling systems counteract this heating and provide an optimal operating environment for this equipment.

Once again, it is important to overestimate the cooling needs for the server room. Make sure that there is enough cooling capacity to provide for any future equipment and then add an additional 15%-20% of capacity on top of that. Inadequate cooling can do as much damage as inadequate power but there are no automated breakers that will protect the equipment. It is up to network staffers to realize when there is a cooling problem and work quickly to correct it.

When possible, it is in everyone's best interest to install a 1/2 capacity backup cooling unit as well to provide for minimal cooling in the event of main unit failure. This will allow enough cooling to keep essential systems running and provide a minimal level of service. All cooling systems should also be connected to the backup power system. It will do little good to keep systems running if they are only going to overheat due to lack of cooling. Therefore the power requirements of the cooling system must also be included in the calculations for the backup power unit as mentioned above.

Money spent on the proper cooling and power installations can and will save time and money in the future. They are the most important part of any server room installation. The investment made here is easily justified when compared with the costs of damaged equipment, lost data and lost productivity.

Fire suppression

The most popular fire suppression system for computer centers, whether they contain mainframes or network servers has been Halon Gas. This gas combines with the oxygen in the server room and the fire smothers. Since it is a dry system it has been widely used and praised. Unfortunately, Halon has been found to damage the ozone layer and its use and production has been banned in the United States. Replacement systems such as Dupont's FE-13 are available but experts are unsure whether they will provide the same level of protection.

An alternative is a dry standpipe system. This type of system has two alarms. The first alarm will arm the system but not actually release any water into the room for a specified time. This provides staffers some time to control the fire using standard electrical fire extinguishers. Staffers can then prevent the sprinkler system from engaging if the fire is under control. Otherwise the system will trigger the sprinklers and extinguish the fire. This provides some protection against water damage to the server room equipment unless it is absolutely necessary.

Finally, if there are sprinklers installed in the server room it is important they be installed both in the ceiling and under the raised flooring. Since all the cabling for power and network are under the floor it is important that the water is able to reach the fire area immediately.

Justification

Despite all the technical hurdles to be crossed in designing a server room the highest hurdles will be justifying the expense to upper management. While a server room is considered a technical necessity by most network managers, proving this fact to their managers can be troublesome.

Quantify losses due to computer failures and downtime. Search out stories about other companies successes and failures with server rooms. These stories help to illustrate the point more clearly than facts and figures alone. Stories from personal experience were the basis for several topics in this article. If the server room solves problems that have occurred in the past, be sure to address them. A case study could help by giving a detailed analysis of one of these problems and how the new server room would prevent it from happening.

In most cases it is best to avoid too much technical talk. Management needs to understand the business needs for the server room, not some obscure technical reason for choosing fiber over Fast Ethernet or ATM. If there is a knowledgeable technical person involved then the scope may broaden but even then it will probably not encompass discussions of every hub and router.

Above all else, the one rule to remember when designing a server room is to provide more capacity than will be needed for the life of the site. When dealing with computers and computer equipment and the extremely fast evolution of the computer industry it is almost impossible to over-estimate. Computer equipment almost seems to breed in most companies and server room equipment seems to follow the same rule. Server rooms are just as important today as mainframe glass houses were in their day. Companies have made extremely large investments in computer equipment and it only makes sense to make a smaller investment to protect this equipment.