How to Prevent Network Downtime

How many hours of downtime did your dealership experience last year due to Information Technology (IT) network disruptions? Does your dealership track these lost hours of productivity? Network downtime is extremely prevalent, according to hosting provider Cogeco Peer 1, with an estimated one-third of businesses dealing with downtime every month.

The costs associated with network downtime are significant, and for dealerships, can add up to hundreds of thousands of dollars per year. When calculating the cost of downtime, you need to factor in more than just the cost of fixing the problem. The real impact on the bottom line stems from:

Loss of data:50 percent of employees report losing access to critical data during outages.
Risking a security breach:Employees often turn to unsecured devices during downtime, which could cause sensitive company data to leak.
Loss of Productivity:Downtime can result in 30 to 40 percent reduced productivity for employees.
Loss of Customer Goodwill: If the downtime occurred during peak business hours, it can substantially hurt your dealership’s reputation.
Loss of revenue: all of the above factors add up to substantial costs; including the revenue that your dealership could have generated during the downtime.

An Ounce of Prevention
While eliminating downtime is virtually impossible, there are prevention measures that your dealership can put in place to reduce the number of incidences, time and associated costs. The first step is to know what the most common causes of downtime are, and to develop prevention plans for each.

Human Error. One Gartner study projected that 80 percent of outages are caused by people and process issues, and more than 50 percent of those will be caused by change/configuration/release integration and hand-off issues.

It’s difficult to prevent human error, but proactive network monitoring can mitigate the effects of any errors that do occur.

Server Room Conditions. If your HVAC blows a fuse and your computer room becomes overheated, your equipment may suffer tremendous damage and failure. An event like this can be avoided by having environmental sensors in your server room that alert you when temperature thresholds are breached.

Adequate ventilation is critical; I have opened up servers to find layers of dust inside that needed to be vacuumed out. Appropriate shelving allows space between equipment and air to circulate freely.

An appropriate fire suppression system is also important as you don’t want water from a sprinkler system spraying all over your computer equipment.

Natural Disasters. You can’t control Mother Nature, but every dealership should have a disaster recovery plan. I get calls from dealerships several times a year when lightning, snowstorms or windstorms cause devices to fail. Sometimes this can’t be prevented but one thing that helps is having servers and PCs plugged into surge protectors.

Third-Party Outages. When your Internet Service Provider (ISP) has an outage, or if there’s a power outage, or your vendors experience application problems, these issues are out of your control.

That’s why it’s important to have back-up and redundancy. We recommend that dealerships have two Internet Service Providers, so if one goes down the other is still operating. A generator can supply power in the event of a power outage.

Best Practices
In addition to the prevention measures listed above, it’s important for your IT team to implement best practices in the every-day management of your dealership’s network. These include:

Have control of a primary firewall at all times. Ensure that firewalls are configured properly and systems are patched with the latest security updates. Will this prevent an outside attack from hackers? There’s no guarantee, but ignoring these steps leaves you vulnerable. Firewalls should also be monitored so you can be alerted to abnormal events, such as high connection counts and high bandwidth utilizations.

In the server/network room, label as many ports and cables that you can. That way if your primary IT support person is unavailable or leaves suddenly, the next person will be able to quickly identify the source of the problem.

Explain the server/network room to every manager in the dealership so if the network goes down at 7AM or 7PM you have a staff member with at least some knowledge that will know where to find the server/firewall.

Post somewhere visible the names and contact information for IT staff, vendors and other outside consultants to contact in case of a network emergency.

Secure all networks. One of the most dangerous practices I see in dealerships is the setting up of unsecured wireless routers in dealerships, or allowing customers to access the same wireless network that your business applications are running on. Customer WiFi networks should be separate and secure, or you risk a data breach that can be very expensive to deal with.

DMS Considerations
Some dealerships are limited in the amount of changes they can make to their own network due to the contracts they signed with their dealership management system (DMS) providers.

I have witnessed scenarios where the dealership IT staff has a firewall they are not allowed to access or configure, a switch that is fully managed by someone else and printers that are embedded with a chip that monitors every page printed.

Some DMS providers insist on controlling the dealer’s network in the name of security. That may sound good, but when your network is down and you need to make changes this can be problematic. Every hour of downtime costs thousands of dollars, so having immediate access and control is important.

If you are considering a new DMS Provider, or if it’s almost time to re-negotiate a contract with your existing DMS provider, it’s important to ask questions related to how much control you have over your own network. It’s your IT network, after all, and you are paying for it. You should have 100 percent access and the autonomy to make any changes necessary.

If you need to make configuration changes to your firewall, will your DMS provider allow access to the device? I have seen some DMS vendors charge dealers as much as $500 for something as small as adding an IP address to a network. Adding an IP address takes an experienced IT person about three minutes and shouldn’t cost anything, but it can’t be done if your vendor is blocking access to your firewall.

Another issue with DMS providers is network segmentation. Ideally your DMS provider will be willing to work around your existing infrastructure instead of creating an entire new IP address scheme. Some DMS providers may want to create a whole new network separate from what you already have.

Unfortunately, this creates a situation where you rely on both networks communicating efficiently to each other every day. I have witnessed as many as four separate networks under a single rooftop, which substantially raises IT costs and increases the chance of miscommunication, which leads to more downtime. Your dealership should have just one primary network for business operations and a separate WiFi network for customer access.

The costs associated with network downtime are significant, so it’s important to have prevention measures in place. Additionally, best practices in the every day management of your network can help to mitigate the negative effects of downtime. Dealerships need to insist on having access to and control of their own network. Combined, these factors could save your dealership hundreds of thousands of dollars per year.