Previously Published in LAN M agazine, August, 1995
Scientists are fond of saying, "if you didn't write it down, it didn't happen." The same could also be said about the corporate LAN. With staff running in all directions trying to keep the LAN running, documentation is the only way to make it all work. Although it is easy to document network systems, very few network groups take the time to document how their LAN works or why it works at all.
Change is the essence of business. Employees move on, systems evolve and hardware and software are constantly being upgraded. Losing a staff member to layoffs or career moves can be just as disastrous as any hardware or software failure when that person is the only one who knows how to keep a particular system running. A little documentation can go a long way towards alleviating the pressure when staffers move on. Network managers need to insure that despite all this change the network can still continue to operate. Plan for the loss of critical staff by insuring that systems are well documented and more than one person understands their operation. Tighter budgets mean smaller network groups who have to work harder. Everyone has to know more about everything and be able to make the best use of the knowledge.
Horror Stories
At one major company, the electronic mail administrator, some would call him a wizard, was promoted to another division of the company. He had succeeded in keeping the company's 12 mail gateways running, remote access hubs churning and the mail flowing for almost 3 years. Despite weeks of effort in training other staff members in the arcane science of keeping all this operating, the first week the administrator was gone the entire system started to fail. Like a living example of entropy one system after another went off-line and no matter how hard the staffers tried to put it back together it continued to fall apart. After hours of phone calls and long nights the system was band-aided back together and scheduled for a complete overhaul.
In another case, only one network administrator knew the special settings that needed to be entered to restart a file server after a power outage. Despite an hour of trying every combination under the sun the other staffers had to wait for the administrator to return from a training class before the server could be restored. Had the settings been placed near the server, it could have been restored much more efficiently. Instead, user productivity dropped to nil and several hours of work time were lost. These stories illustrate the reasons for requiring documentation in your network group. Below are detailed several benefits of documenting your system and some policies that can help you start the documentation process today.
Side Benefits
Besides heading off disasters like those mentioned above, documentation can provide many side benefits. One of the banes of network groups today is excessive specialization. Each staff member has their area of expertise but very few are able to cover for another staffer should the need arise. Network staffers should endeavor to become computer generalists and documentation can be very helpful towards this end. It can create a staff that is cross-functional and can easily move from one operation to another. This doesn't mean that staffers should have no specialization. In fact, it will be their job to train others in their specialty so that all the staff becomes well-rounded in the diverse areas of network management.
Staffers can be out sick, go on vacation or attend a training class and without thorough documentation entire operations can grind to a halt. While it is difficult to cross-train the entire network staff, documentation can provide at least a modicum of information the staffers can build on to solve a problem even if the main guru is unavailable. The goal is not to replace the specialist; it is to disseminate the specialist's knowledge to the rest of the network staff. Specialists become mentors to less experienced staffers and assist them in becoming fully functioning members of the team.
It is not pleasant to have to worry about losing staffers but it is necessary to prepare for the eventuality. Planning can help remove the sting of unexpected departures and allow the network group to get back to work as quickly as possible.
It is important to give every staffer the ability to learn and grow. This allows them to develop skills that will serve the network group better in the future. Preparing documentation helps staffers increase their understanding of various systems and reading documentation makes each network staffer more useful to the firm and to their fellow staff members.
Another effect of documentation is the creation of a free flow of information between staff members. Staffers can research other systems first through the documentation. Then they can take the more advanced questions to the specialist who currently maintains the system. The documentation answers the basic questions and provides a common background so that the specialist and the beginner have a place to start their conversation. The expert doesn't have to cover the basic ground before they can get down to the nuts and bolts of running the system. The documentation does that for them.
It is time to break the cycle of job security through obscurity where staffers protect their jobs by keeping others ignorant. The process of creating documentation will quickly point out those people who are hoarding information for their own purposes. Sharing the wealth of information makes everyone more important to the company and insures that everyone has an equal chance at excelling in their job.
Getting Started
It doesn't take much to get started. All it requires is a desire, some planning and a few minutes of time. The question is often asked, "Where do you start?" The answer is _now_ and _anywhere._ Here are a few recommendations though. One of the easiest places to start it with projects that are currently in progress or just beginning. If the project plans do not include time and budget for documentation, add it now. In fact, any project from this point forward should include documentation as part of its original plan. Otherwise the network group will merely have to document these systems sometime in the future. Producing documentation during a project is much simpler and results in better quality information. It can even help point up problems with the new system before they turn into end-user nightmares. As older, undocumented systems are dismantled the new, fully documented ones can take their place.
It is also a good idea to start with the most troublesome systems and procedures. This shows immediate results and proves to both staff and management that documentation is a worthwhile endeavor. Often the time spent on documenting these systems can be recouped with weeks due to more efficient operations and better productivity. If a system is crashing on a regular basis, documenting that system could provide some answers. It at least provides a scheduled block of time to research the problem and allows the creation of procedures to insure the quick recovery of the system when it does crash.
The next systems to tackle are those that are mission critical to the company. Any system whose outage would effect more than 10% of the companies productivity should be documented early in the cycle. Nothing is more embarrassing than having to explain to an executive that the company can't send out bills today because the person who knows the recovery procedure is out sick. Use documentation as a weapon to keep mission critical systems performing their best and show executives that documentation is what allowed these systems to remain on-line.
Procedures that are performed daily or weekly are better candidates for early documentation than those that are only performed once a month or quarter. All these procedures should be documented eventually but concentrating on those that occur frequently shows results immediately. An exception to this are procedures that are performed perhaps once a year. These need to be well documented since it is possible that the staffer who last performed the procedure will have moved onto a new position or even left the company. As always, consider the needs of the particular application and the needs of the company.
One additional benefit of starting with the above systems is that it allows the common problems to be resolved by help desk staff instead of network staff. This allows more time for network staff to develop and install new systems instead of performing tasks, like rebuilding mail databases or routing tables, that could easily be performed by the help desk. Help desk staff shouldn't have to go hunting for a specialist to reset a gateway that is in the same room. Help desk personnel require the proper tools to react quickly and accurately to system problems. Documentation is one of the strongest tools they can have at their disposal. This allows the help desk to react more quickly, reduces down time and increase everyone's productivity.
Do it with style
When building in-house documentation it would be best to avoid the dull, dry and bloated examples given by most software manufacturers. In-house documentation should be considered a living and growing document just as the network sometimes takes on the feel of a living growing entity. Think clear, concise and, above all else, practical. Nothing is more useless than unusable or inaccurate documentation.
Documents should be more than lists of software and hardware. Include detailed procedures which allow nearly anyone to restart or recover the system. Don't be afraid of a little humor if it gets the point across. Just as every company has a style so should the documentation. Include diagrams from the manufacturers documentation when needed. Nothing is worse than being presented with a row of 8 DIP switches and having no idea which switch does what. Maybe a staff member can draw appropriate diagrams when no other is available.
Include flow charts showing the flow of data into and out of the system. Make it clear where problems can occur and what symptoms indicate a problem at each of the locations. Include pointers to other documentation that might be useful. The general rule is that you cannot be too thorough in documenting a system. More information, as long as it is well formatted, is always more useful than less.
Standard documentation forms should be developed early on in these efforts. Software templates should be developed that allow any staff member to create a clear and straightforward explanation of a system. This will also insure that the documents will look the same and can be easily scanned for the needed information. Most word processors allow the easy addition of a table of contents and index to any document. Each document should also include a version number and update history so that staffers can be sure they have the most current procedures and configuration information.
Programs, batch files and configuration files should be self-documenting whenever possible. Include pertinent information in program source code so that any staff member can open the file and immediately understand it. Configuration files should include notes explaining when a change was made, who made it and the reason for the change. A small batch file could be written that gathers up all the comments and combines them into a weekly or monthly report so that all staff members can be easily made aware of the status of any system.
All documentation should be stored in a central repository in both paper and digital formats. Set aside a directory on the main file server to hold documents as they are created. It might even be possible to use the document summary features of some word processors to allow the searching of the documents to provide direct access to specific information. A high-end solution is to build an electronic support database that includes the most common questions and their resolutions. Don't let the technology get in the way though. The end goal is to have good documentation. The technology used to accomplish this goal is secondary.
Documentation is very good at pointing up design flaws or procedures that are too complex. No procedures should be more than 3 pages long. Any procedure that runs longer than this points to a system that needs to be re-designed or, at the very least, automated more fully. Don't make staff type in 20 lines of cryptic commands to restart a system. Provide a batch file or other macro that performs the repetitive or time consuming work for them. Perhaps the system might need to be broken down into various operation instead of trying to document the system as a whole.
Make sure there isn't any "just because" reasoning in your documents. If a system is that way "just because" then find out the reason why. Perhaps there is a better way of configuring the system but no one has taken the time to find out how or why. If there is a specific reason, state it. Don't leave staffers questioning the sanity of the person who developed the system. Explain cumbersome procedures or odd configurations. If you can't explain them, get rid of them.
Getting practical
While technology can be of great use in any documentation effort, low tech solutions are often the best way to get started. Paper-based systems can give way to digital ones once the basic outlines of the documentation effort have been established. Once again, it is the documentation that is most important, not the technology used to create it.
Firsts and foremost, create a change log and attach one to each piece of network equipment in the LAN. This log should include date, time, operation performed and name of the staffer who performed. Like Hansel and Gretel leaving bread crumbs along the trail these logs will let staffers follow the sometimes confusing trail of additions, deletions and modifications that accompany every network. These logs must be easy and quick to fill out. They are meant to decrease work, not increase it.
The next step is to label every thing in sight. Server and PC's should be labeled with their name and any other configuration information that might be relevant. Staffers shouldn't have to search through 20 gateway machines to find the one that needs to be restarted. Make it easy for staffers to quickly locate the machine they are looking for and solve the problem with the least possible effort. It might also be useful to print configuration sheets available from a variety of utilities and attach them to the computer as well. Give staffers all the information they might need to solve any problem. This becomes especially important when dealing with the unseen traps of interrupt settings, memory addresses and network node addresses. Network software and associated configuration files are even easier to document. A log should be created at the top of each file that contains much of the same information as the paper logs above. The ease of updating allows for even more detailed descriptions of any additions, deletions or modification. Automated systems can be designed to collect these comments and format them into reports that can be delivered via electronic mail to all network personnel.
If the network hubs and concentrators look like a rat's nest it is time to strip them apart and instill some method on the madness. Make sure that each cable is labeled clearly and correctly. Staffers shouldn't have search through a mass of cabling to patch in a new system. Systems which do not move frequently, such as servers and gateways, should be labeled at the hub where they are connected. Make it easy to trace cabling so that cabling problems can be easily diagnosed. Entire hubs or concentrators don't need to be organized at one time. Tackle it 8 ports at a time if necessary but do it. Once each hub is organized set up procedures to keep it that way.
Once everything is labeled, document it further by noting the interconnection between all the different systems and hubs. Many software products exist for mapping networks but again, a simple paper diagram might be the first place to start. Be sure to include the model and type of each device and any configuration information that might be important. This could include network addresses and numbers, types of interfaces and, if the system serves a particular group of users and denotation of specific group, building or office. Each area, whether one floor of offices or an entire building, should be diagrammed. These diagrams should show the location and number of each network jack as it is labeled in the network patch bay. The goal is to provide anyone with the ability to envision the network while sitting in the comfort of their office. The change logs mentioned above can be very useful in creating and updating these diagrams.
Since no network is complete without workstations to access the servers, network groups need to concern themselves with the documentation of some aspects of these workstations. Each PC should have a label indicating the type and configuration of the installed network interface card (NIC). If special software is required to activate the NIC then a floppy disk should be included in a package attached to the PC. Each network connection box should be clearly labeled with a number or name just like the cable in the network patch bay. If desks or book shelves will be placed against the jacks, arrange a method to place the jack numbers in an easily viewable area. This will allow for easier reconfiguration and repair in the future.
It doesn't end here
Once the process of documentation is started, the results should begin to appear almost immediately. It is important to remember though that this is the beginning of a process and not the end. In fact, the process never really ends. The final step in the success of any documentation process is the ongoing efforts to create new documentation as needed and maintain those documents that have already been created. Inaccurate documentation can be worse than no documentation at all. Documentation should be updated whenever hardware is replaced or software is updated. These changes could be small or large but it insures that each document has a chance to be updated on a regular basis. This also means that specific documentation can be abandoned once the system itself is removed or replaced. There is no need to keep dead wood around cluttering up the system.
If possible, a yearly review of all documentation should also be scheduled. This will point out new systems that might have been missed or changes that might not have been included. Everyone should have a chance to add their knowledge to the document and the year-end review provides a time to do this.
Since documentation is so important to the operation of a LAN it makes sense to make it part of each staffers yearly review. It should be noted how well the staffer contributed to the documentation efforts and also how well they transmitted their knowledge to the other members of the network group. This might even be part of a specific "mentoring" program where veteran staffers are made responsible for the education and elevation of incoming staff members.
The task of starting and maintaining network documentation might seem like a daunting task but in reality it is the only way that a network group can hope to maintain there growing systems and still find time to develop new systems that will directly effect the companies goals. Too many network groups find themselves falling into patterns of "fire-fighting" where all their attention goes to keeping the LAN running instead of developing new systems that keep the company on the cutting edge of technology.
Not only is this damaging to the company, it can be damaging to the careers of network staffers. They will find their skills degrading and their jobs will come under more and more scrutiny during each budget cycle. Documentation helps reduce the time spent maintaining systems and provides staffers the time to build their skills, research new technologies and build a better network.
Network groups only need to open the nearest wiring closet or server room to find a starting place for documenting a network. Don't be overwhelmed by the job; start with one small area and grow from there. It might seem like a lot of extra work now but, in reality it might just be the key to saving the network, staffer's sanity and maybe even their jobs.
** SAMPLE SERVER LOGIN ** ** SERVER LOGIN SCRIPT 4/14/93 DEW ** * 01/03/94 18:30 DEW - Modified section for running TCP*.BAT files. * 02/07/94 15:15 DEW - Added Oracle Office section * 02/16/94 11:30 DEW - Modified section for running TCP*.BAT files. * 05/27/94 09:00 DEW - Added Capture statement for Q_SERVER_TOPSIDE * 08/23/94 07:04 RBT - Renamed SBS group, changed sys login to reflect name * 08/25/94 01:12 RBT - Added UP2 Login Configuration collector * 09/15/94 11:28 RBT - Removed Pause after Timesheet Reminder * 11/04/94 03:27pm RBT - added UP2 include file stuff * 12/16/94 09:00am RBT - added win95 group mapping * 01/30/95 04:15pm RBT - Comment out Oracle Office Group * 04/13/95 09:15pm RBT - Removed M: drive mapping for ACCESS_USERS"Fire-fighting" is a problem that arises when network groups are spending so much time maintaining old systems that they have no time to research new technologies or develop new systems. They are constantly "putting out the fires", hence the name. This can cause a company to lose its competitive edge since new technology is not be applied to business problems. 'Fire-fighting" can also lead to staff burnout, job dissatisfaction and high employee turnover.
Douglas E. Welch is a Creative Technologist for Entertainment Communications Network in Studio City, California. He can be reached at dewelch@pop.com or 76625,3301 on Compuserve or via the World Wide Web at Douglas E. Welch Studio