Disaster Recovery in Fiber Optic Networks
In the past decade our globe has been battered with one disaster after another. There have been hurricanes, floods, ice storms, fires, earthquakes and volcanoes. The list seems endless. Unhappily, climate change is likely to worsen this situation as the 21st century progresses, which will cause enormous problems for companies deploying modern fiber-based communication infrastructure.
Climate change models advise us to expect more precipitation for the northern United States and more droughts in the Southwest. Heat waves will become more intense; once-in-20-year extreme heat events are projected to occur every two or three years. The intensity of hurricanes is projected to increase while their speeds will slow, exacerbating the damage they cause. Global sea levels will rise another 1 to 4 feet by 2100, increasing urban flooding around the globe.
In terms of preparation and recovery, it is less useful to categorize by type of catastrophe—flood, tornado, fire, earthquake—and more useful to look at their predictability. Unpredictable disasters can be devastating, whether it be a magnitude-9.0 earthquake off the coast of Japan that produces a huge tsunami, or a tornado in the U.S. Midwest. Dams break, bridges collapse and wildfires burn, often resulting in subsequent mudslides.
In contrast, predictable disasters can be statistically analyzed and forecasted. We know there will be more blizzards and ice storms. There will be another hurricane with both flooding and wind-driven network damage. With predictable disaster preparation and recovery procedures, including those for rebuilding fiber-based communications networks can be developed and practiced well in advance.
The good news is that passive fiber is usually rugged and damage-resistant. In some cases, it can even be submerged and as long as the endfaces remain mated, the damage can be minimal. Of course, each disaster will require a different response to repair, restore and refurnish the fiber networks upon which we all rely. Let’s take a look at the procedures and tools we can deploy to keep our networks running.
Prevention or planning?
In terms of prevention, there’s not much you can do. If the cable close tor communications rack is in the basement, it’s going to flood. But the best disaster recovery plans are established and rehearsed long before the disaster hits. That means thinking about the possibilities, and pre-positioning supplies to be deployed easily.
It is essential that repair techs be provided the proper tools and the proper training in advance of the event.
If a tech fails to effectively inspect and clean during the initial repair, it could cause system failure, ultimately resulting in more time and costs to rework the repair for a second time. Better cleaning up-front is like a good insurance policy, which minimizes the cost of a disaster.
During fiber network disaster recovery, the first challenge is access. Avoid downed power lines and flowing flood waters. If water cannot be avoided, waist-high waders are crucial tools. In addition to the obvious risks of being swept away, flood waters are nasty and polluted. They carry sewage, disease, live animals including snakes and fire ants, dead animals, and a wicked brew of household and industrial chemicals.
Once at the work site, it is highly recommended that every tech first makes sure everything is safe. Follow OSHA (Occupational Safety and Health Administration) safety standards for all procedures. The building, tower or manhole should be structurally sound, free of vermin and more-or-less dry. Electrical systems should be powered off. Backup batteries should be disconnected. Be cautious for electrocution hazards, toxic hazards, and leaking fuels (like improperly stored gasoline or damaged containers of alcohol). Don’t expect the lights to be working, so us an LED flashlight, which provides strong, reliable lighting without sparks.
Another worry is air quality. Outside-plant cables are made of rubber and polyethylene, which is very toxic when burned, so good ventilation is essential. Never enter underground or enclosed spaces without ventilation and supervision. Follow OSHA “enclosed space” rules with which everyone should be familiar.
You can expect the facility to be a mess. Start with presaturated alcohol wipes to remove grime, soot and debris. These wipes have an added benefit of disinfecting surfaces. In addition to cleaning the fiber, the wipes can be used for cleaning active components. Nonflammable, high-pressure contact cleaner aerosols are very useful to rinse particulate from hard-to-reach places. You might be able to clean the surviving hardware, workbenches, keyboards and racks with optical-grade nonflammable dusters.
Now, as to the fiber itself, assess the damage first. Use your test equipment to troubleshoot links. Remember not to look directly into the connectors, as human eyes can’t see the light produced by fiber-optic transducers and the risk of eye damage is real and permanent.
There will be no shortcut to the restoration process. Every endface needs to be opened, tested and inspected. Don’t assume mated cables and patch cords are bad; fiber is surprisingly resilient to flooding because the physical contact keeps water and debris out of the central contact zone. Nonetheless, bring replacements and be ready to test them and clean them.
Mechanical “push-to-clean” tools generally won’t get the job done. While they’re handy for quick-and-dirty cleaning of lightly contaminated end-faces, you’ll need sturdy cleaning sticks for the more thorough and rigorous cleaning required in this environment. Quality cleaning sticks are helpful in two ways: they clean the entire endface (not just the contact zone), and they are able to clean the sides of the adapter sleeves.
Be careful as to which cleaning sticks your company selects. Some companies try to use cotton swabs, but they will fray on the sharp vertical edges of the adapter sleeve and leave debris entrapped in the adapter. Another weak choice is an inexpensive reticulated foam swab. By its very nature, foam is a dirty cleaning material and is doubly cursed because it generates static electricity. The sharp edges of the adapter will shave chunks off foam swabs and deposit them on the endface, while the static will cause the particulate to stick to the endface.
In our experience, wet-dry cleaning works best on endfaces. A fast-drying solvent will help dislodge contamination and remove oily residues. Slow-drying fluids, such as water or even the popular alcohol cleaners, are sub-optimal and will slow the repair. Be sure to select a cleaner in a hermetically sealed, leakproof container. Refillable pump-bottles will not keep the solvent clean enough for proper repairs and may just spread contamination around.
Lastly, when trying to restore a network damaged in a disaster, it is absolutely essential techs are provided with high-quality, optical-grade lint-free wipes. These will be used to clean jumpers and can remove soot, grease, silt, and particulate from endfaces. Quality wipes can serve double-duty and be used to wipe cabinets, racks, workbenches and other surfaces on the first try, speeding the restoration process and simplifying inventories.
Disasters, whether predicted or from out of the blue, will continue to wreak havoc on fiber networking systems. The best approach to manage them is to be prepared. Do some preplanning and have a good solid recovery and repair strategy in place. Be proactive by assembling the recovery tools needed and providing repair techs with the knowledge and training to make fiber disaster repairs both safe and successful.
Disaster recovery tools for fiber network
What tools should a repair tech have? A good list might include the following:
- Portable lighting with extra batteries
- Replacement patch cords
- A low-power inspection scope, for inspecting the connector endfaces. Be sure to include adapters for the most popular configurations of ports and jumpers
- Hand tools like strippers, cleavers, gloves, pliers and screwdrivers, because anything prepositioned will be lost, destroyed or covered in grime. Special tip: Avoid cheap tools because this is a time-sensitive situation, so don’t cut corners.
- Assume the labels on the fiber bundles will be destroyed or unreadable. Bring sticky-stuff remover and a label-making machine.
- Visual fault locator and continuity tester, because if the fiber racks have been toppled or damaged the fibers may have been actually broken or crushed.
- Presaturated alcohol wipes in rugged plastic containers
- Nonflammable, optical-grade dusters to blow dry debris from surfaces
- Nonflammable contact cleaner sprays, to flush dry debris from surfaces
- Nonflammable, fast-drying, optical-grade fiber-optic cleaning fluids
- Painter’s tape, to pick up glass shards
- An optical time-domain reflectometer is essential to isolate where problems are
- Optical-quality lint-free wipes, not the cheap ones packaged in a cardboard box
- Battery-powered light source for troubleshooting
- Drinking water
- All of this equipment should be in a hardened case suitable for harsh environment