Middle of the night Update:What happened?
Around 11:06 AM on Saturday (3/31) we started receiving our first error reports on the VFW WebCOM Network. The network team quickly responded by attempting to determine what was causing the problem. We soon realized that the errors we were receiving had moved beyond the typical quick-fix type problems we occasionally receive. So we immediately contacted our dedicated server provider who quickly moved into action. It was determined in a short period of time that the main hard drive on the server was rapidly failing...and, in fact, did fail just a few short moments later before any data could be safely retrieved.
The server farm's Network Operations Center (NOC) immediately took the server offline and began replacing the main hard drive. In the mean time, our team and our service provider began examining the backup files for the system. Upon closer examination of the backups, we soon realized that the slow downs many of you experienced earlier in the week were actually early precursors to the drive failure.
Our server normally makes three backup copies of the entire network...a daily copy in the early hours of each morning, a weekly copy and a monthly copy. Unfortunately, in this instance, the problems earlier in the week and the drive failure at the end of the week in a month where all three happen at the same time created, for lack of a better term, A Perfect Storm for corrupted backups. With the daily, weekly and monthly backups all occurring within a 48 hour time span on a drive that was already beginning to fail, most of the database information was corrupted in rapid succession on all three.
When we hit strike three on the backups, we hoped for the best but began preparing for the worst. Our service provider and the folks at NOC worked thru the night attempting to remount the failed hard drive and recover as much data as possible. But around 2:20 Sunday afternoon, we received word that the drive was too corrupted to retrieve any usable data.
By that time, we had already begun rebuilding work on the server to get it operational with the new drive. Our first priority was to get this forum back online so that we could keep you all informed as to what was happening.So what's the plan?
What was lost? Well, at first glance, seemingly everything. And in some cases, that would be the truth. The primary example would be this forum. Activity here in the forum had been steadily climbing in recent weeks as we had reached over 2000 posts from over 400 members. Unfortunately, most of that is gone...but may not be entirely lost. I'll explain more in a moment. If you had an account here in the forum, you will need to re-register to be able to once again take part in the conversations.
For those posts and districts who had claimed and began using their weblogs, unfortunately, we were not able to recover the previous entries that you had made. However, all hope is not lost. Thanks to the miracle of search engine cacheing, many of your entries will be able to be recovered...although we may need your help if you still have any photos or files that were included in the stories.
Here's the reconstruction plan:
1) Rebuild the forum as a central receiving point. As of right now, anyone attempting to reach just about site on the network will automatically be re-directed here to this forum. This redirection will stay in place until we can rebuild the rest of the network.
2) Reconstruction of the weblog network is already underway. Thanks to many of the programming tricks we've learned over the past several years in constructing the network, this should move along fairly rapidly. You could expect to see the department-level sites back online sometime on later today. The post-level weblogs will follow shortly thereafter. Order of priority will be determined by those states who have the most active posts.
3) While the post-level re-construction continues, other members of our team will begin re-populating the content on as many of the previously active sites as possible with as much information as we are able to retrieve.
4) Completion of the re-build and troubleshooting.
5) Take further steps to ensure data redundancy.The Good News
The good news is that some parts of the network have been recovered quickly.
For those users of the VFW WebMail system in Virginia and select other departments, your email has been fully functional since around 5:00 PM Sunday. Since your mail is stored on a different server, you did not lose any of your existing emails or address book entries in your mailbox.
The news aggregator at http://vfwwebcom.org/news.php
is again functional as well as the Google network search feature found near the top of most pages throughout the network.
It also appears that we "may" fully be able to recover all of the post information in the Department of Alabama. We have been able to locate a possible backup of the entire Alabama post network and if we are able to successfully restore that database, much of the content stored there can be quickly replicated to the other departments.What can you do to help?
1) Be patient. We are rebuilding as quickly as we can and there may be some bumps in the road with software and other unforseen situations.
2) We will be posting messages here regularly to update the progress. If you had already claimed your weblog in one of our active departments, please re-claim your site as soon as possible so that we will know where to focus our attention in recovering lost data. If you had posted pictures or uploaded files, try to gather them where they can easily be retrieved when ready. To assist us, you can also begin retrieving your data thru the cache systems on Google and other search engines. (Hint: start by downloading and using the search box on your VFW Department Toolbar
and clicked the "Cached" link for the listing you want to see)
3) To our forum users: We will be having a large amount of traffic flowing through this forum for the next few days and weeks to come. Please re-register and help us welcome new members to the forum and potential new membership for the organization.
Beyond that, we'll keep you updated as often as we can. Again, we appreciate your patience while we work through this difficult time and please accept our sincere apologies for any inconvenience.