Updated on Friday June 26th – Please scroll down for the update.
As Murphy’s law would have it, just days after we’ve launched our first commercial offering we’ve suffered a complete network outage for 12 minutes. This happened about 10 minutes ago. I’m currently working with our network provider Netriver to find out exactly what happened and to prevent it occurring in future.
For most web companies this would not have a significant impact. But because we provide real-time stats, there is currently a hole in your graphs for the last 60 minutes that lasts about 12 minutes.
This outage did not affect the ability of our customers websites to load. It simply means that your stats were not logged for 12 minutes.
Our sincerest appologies for any inconvenience this has caused.
Mark.
Update:
Netriver has been working with Cisco to resolve this issue. What happened is that Netriver has multiple redundant connections to the Net and has two redundant routers that route their traffic. In a rather unusual incident, both routers went down simultaneously. Here is the incident log:
Network Event times are approximate:
· 6:15 one of our core routers rebooted.
· 6:18 the router was back online.
· 6:25 we discovered the other core router was hung with a memory issue.
· 6:30 – 6:40 Tried to correct the memory issue unsuccessfully. We then made the decision to copy off the logging on the router and reboot.
· 6:43 rebooted router.
· 6:48 router was back online.
Netriver has multiple tickets open with Cisco to try and resolve the issue and Cisco have already responded with a diagnosis. Netriver is implementing Cisco’s recommendations to prevent this happening in future.
Feedjit, Netriver and Netriver’s upstream providers run redundant systems at every level from the application to infrastructure. This incident is highly unusual and the impact was minimal thanks to the quick reactions of the Netriver team. Both Feedjit and Netriver are taking steps to ensure this doesn’t happen again in future. Once again we apologize if the 12 minutes of down-time caused you any inconvenience.
Mark Maunder – Feedjit Founder & CEO.
