In my full-time job, this week has been a plethora of lessons learned. The story I tell here applies to anyone that is in charge of networks… or more importantly, your admins on that network.

Earlier this week, I received a disturbing email letting me know that certain services were down on our network. Specifically, some vital web pages used for specific things were not working, and troubleshooting was taking place. The webpages in question were owned by a particular shop of mine that was in the middle of development for a project that was about six months in the works.

Normally, this would not be an issue because I have several network and system admins in charge of troubleshooting situations just like this. The problem was that the development server was configured a bit different than the other normal servers. In this case, both IIS (Windows Web Server) and XAMPP (Linux Web Server) were installed on the same machine to test which platform was the most useful for this developer. The decision was made by that team to go ahead with XAMPP; however, IIS was still installed and on by default.

When I got the email that they were troubleshooting, I made the assumption that the server had rebooted, and both of the web server components were fighting over port 80. In this case, IIS would win over XAMPP every time after a reboot. So the answer is simple… either turn off IIS or uninstall it as your first measure of troubleshooting.

Needless to say, I’m writing this for a reason. Instead of troubleshooting properly with a 50/50 method, the admin simply did a re-install of XAMPP.

Before I mention what happened with this reinstall, I would like to inject here to say why I titled this Train Your Admins. This development team was made of developers, not systems administrators, and they didn’t understand that BACKUP and REDUNDANCY are your lifeblood when you’re developing code worth over a hundred thousand dollars. In this vein, they had been backing up their data within the XAMPP folder structure, but failed to continuously place it eslewhere. I created a different location for them when the project started, and unfortunately assumed that they would continue to use this other location for safety; but alas, they did not.

So now onto what happened during the reinstall. An administrator, during their troubleshooting, decided that reinstalling XAMPP would bring the services up again and all would be well. They had not touched the port configurations because there were no Windows logs that implied that there was a port problem… those logs were in the Apache folders on XAMPP, which were crying at the top of their lungs saying that port 80 was already taken.

When the crew decided to reinstall XAMPP, they didn’t realize that ALL folders and files within the XAMPP structure would be replaced with the EMPTY folders and files that are shipped with the program. In one swoop, and a couple clicks, everything was lost! The original backup that I created had files from five months ago, but the server presented no options to bring back that which was so precious… overwritten folders.

This story is important for so many reasons. The obvious reason is the story that I’ve written. The not so obvious reason is the trust that you place in your admin’s hands when you sign them on to manage your network. It’s wonderful to have people you trust, and sleep at night knowing that any red line on a network diagram can be changed to green because they know what they’re doing. It’s different when you give experts a situation that wasn’t documented, and expect them to be on point with everything they do. The responsibility was mine to make sure that everyone was trained, and they absolutely knew every aspect of what was going on. Perhaps I over trusted my crews… but I don’t think that was the case. I simply didn’t arm them with the respect for the system with which they were entrusted.

Needless to say, we’ll get up and running again. My message to them was that they can take all that they’ve learned, and apply it to the rebuild. In my experience, the second go at things tends to yield awesome results! But the time that we’ll lose on this one can’t be quantified… and certainly could have been avoided.

Learn this lesson now! Don’t take anything for granted, and even though your plate is full with multiple parallel projects, don’t lose sight on any of them. If you feel yourself falling behind because there’s too much going on, speak up and get some help; or better yet, walk around and get to know each shop over again because it will spark questions that will save you.