MyVFW Social Network    Latest MyVFW Photos   Latest MyVFW Videos    MyVFW Groups    Latest Network News

Author Topic: Server Crashed- Back-Up Corrupted  (Read 15232 times)

0 Members and 1 Guest are viewing this topic.

Offline Ross Myers

  • Administrator
  • Sr. Member
  • *****
  • Posts: 267
  • Serving those who Served....
    • Email
Server Crashed- Back-Up Corrupted
« on: April 01, 2007, 06:06:49 pm »
Hello all,

As you may have noticed there have been server issues starting with the slow down stutter step sa couple days ago.  Since then the server crashed and the back-up was corrupted.  Our server guys have been extremely responsive and the new server is up and running. We are in the process of bringing things back online now. 

Check back for more information.

Unfortunately, this is not an April Fools joke.

Offline IJK3770

  • Dept of Mo Webmaster, QM Post 3770, Mtn Grove, Mo
  • Administrator
  • Hero Member
  • *****
  • Posts: 3370
    • Mtn Grove VFW Post 3770
Re: Server Crashed- Back-Up Corrupted
« Reply #1 on: April 01, 2007, 10:04:02 pm »
Progress as promised.
Cheerily
IJK
All-State Post QM :
97-98,98-99,99-00,00-01,01-02, 03-04,04-05,06-07,08-09, 09-10,10-11,11-12,12-13, 14-15, 16-17

Assistant Inspector General 2008-09
National Aide-de-Camp 2009-10
National Aide-de-Camp 2010-11
Assistant Inspector General 2011-2012
National Aide-de-Camp 2013-2014

ShapeShifter

  • Guest
Re: Server Crashed- Back-Up Corrupted
« Reply #2 on: April 01, 2007, 11:59:12 pm »
Sounds like a clean start for all. Did the crash wax the blogs as well? If so not a problem, I was not keeping mine up to date as I should have been. Thanks to blogmaster for spending his weekend getting it back up and running.

The latest report is that the Britons were ready to fight off their abductors. Certainly their escorting ship, HMS Cornwall, could have blown the Iranian naval vessel out of the water. However, at the last minute the British Ministry of Defense ordered the Cornwall not to fire, and her captain and crew were forced to watch their shipmates led away into captivity. WTF over?

ShapeShifter

  • Guest
Re: Server Crashed- Back-Up Corrupted
« Reply #3 on: April 02, 2007, 12:02:27 am »
I just tested the lost username/password, it works.

Offline TheShu

  • Administrator
  • Hero Member
  • *****
  • Posts: 2350
    • The Other Shu
    • Email
Re: Server Crashed- Back-Up Corrupted
« Reply #4 on: April 02, 2007, 06:49:33 am »
Middle of the night Update:

What happened?

Around 11:06 AM on Saturday (3/31) we started receiving our first error reports on the VFW WebCOM Network.  The network team quickly responded by attempting to determine what was causing the problem.  We soon realized that the errors we were receiving had moved beyond the typical quick-fix type problems we occasionally receive.  So we immediately contacted our dedicated server provider who quickly moved into action.  It was determined in a short period of time that the main hard drive on the server was rapidly failing...and, in fact, did fail just a few short moments later before any data could be safely retrieved.

The server farm's Network Operations Center (NOC) immediately took the server offline and began replacing the main hard drive.  In the mean time, our team and our service provider began examining the backup files for the system.  Upon closer examination of the backups, we soon realized that the slow downs many of you experienced earlier in the week were actually early precursors to the drive failure.   

Our server normally makes three backup copies of the entire network...a daily copy in the early hours of each morning, a weekly copy and a monthly copy.  Unfortunately, in this instance, the problems earlier in the week and the drive failure at the end of the week in a month where all three happen at the same time created, for lack of a better term, A Perfect Storm for corrupted backups.  With the daily, weekly and monthly backups all occurring within a 48 hour time span on a drive that was already beginning to fail, most of the database information was corrupted in rapid succession on all three.

When we hit strike three on the backups, we hoped for the best but began preparing for the worst.  Our service provider and the folks at NOC worked thru the night attempting to remount the failed hard drive and recover as much data as possible.  But around 2:20 Sunday afternoon, we received word that the drive was too corrupted to retrieve any usable data.

By that time, we had already begun rebuilding work on the server to get it operational with the new drive.  Our first priority was to get this forum back online so that we could keep you all informed as to what was happening.

So what's the plan?

What was lost?  Well, at first glance, seemingly everything.  And in some cases, that would be the truth.  The primary example would be this forum.  Activity here in the forum had been steadily climbing in recent weeks as we had reached over 2000 posts from over 400 members.  Unfortunately, most of that is gone...but may not be entirely lost.  I'll explain more in a moment.   If you had an account here in the forum, you will need to re-register to be able to once again take part in the conversations.

For those posts and districts who had claimed and began using their weblogs, unfortunately, we were not able to recover the previous entries that you had made.  However, all hope is not lost.  Thanks to the miracle of search engine cacheing, many of your entries will be able to be recovered...although we may need your help if you still have any photos or files that were included in the stories.

Here's the reconstruction plan:

1) Rebuild the forum as a central receiving point.  As of right now, anyone attempting to reach just about site on the network will automatically be re-directed here to this forum.  This redirection will stay in place until we can rebuild the rest of the network.

2) Reconstruction of the weblog network is already underway.  Thanks to many of the programming tricks we've learned over the past several years in constructing the network, this should move along fairly rapidly.  You could expect to see the department-level sites back online sometime on later today.  The post-level weblogs will follow shortly thereafter.  Order of priority will be determined by those states who have the most active posts.

3) While the post-level re-construction continues, other members of our team will begin re-populating the content on as many of the previously active sites as possible with as much information as we are able to retrieve. 

4) Completion of the re-build and troubleshooting.

5) Take further steps to ensure data redundancy.

The Good News

The good news is that some parts of the network have been recovered quickly.

For those users of the VFW WebMail system in Virginia and select other departments, your email has been fully functional since around 5:00 PM Sunday.   Since your mail is stored on a different server, you did not lose any of your existing emails or address book entries in your mailbox. 

The news aggregator at http://vfwwebcom.org/news.php is again functional as well as the Google network search feature found near the top of most pages throughout the network.

It also appears that we "may" fully be able to recover all of the post information in the Department of Alabama.  We have been able to locate a possible backup of the entire Alabama post network and if we are able to successfully restore that database, much of the content stored there can be quickly replicated to the other departments.

What can you do to help?

1) Be patient.  We are rebuilding as quickly as we can and there may be some bumps in the road with software and other unforseen situations.

2) We will be posting messages here regularly to update the progress.  If you had already claimed your weblog in one of our active departments, please re-claim your site as soon as possible so that we will know where to focus our attention in recovering lost data.  If you had posted pictures or uploaded files, try to gather them where they can easily be retrieved  when ready.  To assist us, you can also begin retrieving your data thru the cache systems on Google and other search engines. (Hint: start by downloading and using the search box on your VFW Department Toolbar and clicked the "Cached" link for the listing you want to see)

3) To our forum users: We will be having a large amount of traffic flowing through this forum for the next few days and weeks to come.  Please re-register and help us welcome new members to the forum and potential new membership for the organization.

Beyond that, we'll keep you updated as often as we can.  Again, we appreciate your patience while we work through this difficult time and please accept our sincere apologies for any inconvenience.

Offline easingwr670

  • Hero Member
  • *****
  • Posts: 850
  • 1st Bn 505th PIR, 3rd Bde, 82nd Abn RVN 69
    • Email
Re: Server Crashed- Back-Up Corrupted
« Reply #5 on: April 02, 2007, 07:09:40 am »
Since I do a Post and District blog in NC, I wish you a SPEEDY recovery ;D
Richard C. "Dick" Easingwood
Past NC Administrator
VFW Member Since 1969
Life Member Since 1989(Gold Legacy)
District QM (6 Yrs Total)
District Inspector
Past Post QM (12 Yrs Total)
Cootie CCDB 24 Yrs Total)
Asst Inspector General (2007 til     )
http://www.myvfw.org/nc/dist8

Offline Redmaxx

  • Administrator
  • Hero Member
  • *****
  • Posts: 2963
  • Past Department of Michigan District 4 Commander
    • Email
Re: Server Crashed- Back-Up Corrupted
« Reply #6 on: April 02, 2007, 09:09:59 am »
Thank you for the help in getting re-registered. If you are having trouble logging on after you register please delete your cookies and there shouldn't be a problem.
Department of Michigan
District 4 Commander 11-12
National Aide-de-Camp 10-11, 14-15
National Veterans and Military Services Committee 2014-Present
District 4 Chaplain 2016-17
Candidate for Department of Michigan Jr. Vice Commander 2017-18

Offline DoggyDaddy

  • Administrator
  • Hero Member
  • *****
  • Posts: 5512
  • Have Dog Will Travel
    • VFW Post 1716, Freedom CA
    • Email
Re: Server Crashed- Back-Up Corrupted
« Reply #7 on: April 02, 2007, 01:40:46 pm »
 ???  Has the server crashed before with data and registration profiles being lost?  Are we to expect that it will happen again?   :(   Sadly there was a lot of great information and advice lost for ever. 
Joe Kleinsmith
All State VFW Post 1716 Cmdr (1998-2000)
Cpt, VFW Post Honor Guard, Retired (1991-2009)
SC-SB County Council Cmdr (1996-1997)
SFC, US Army, Retired (1971-1991)
Full Time RV'er
www.vfwwebcom.org/ca/post1716
http://vfwwebcom.org/ca/Post1716HonorGuard/

Offline TheShu

  • Administrator
  • Hero Member
  • *****
  • Posts: 2350
    • The Other Shu
    • Email
Re: Server Crashed- Back-Up Corrupted
« Reply #8 on: April 02, 2007, 01:52:00 pm »
Yes, there was one previous instance where we had a hard drive failure on a different server a little over a year ago...back before this forum was even started and there were only a few state VFW departments online.  Fortunately, in that instance, the backup files were recoverable and the network was only down for a few hours with almost no loss of data.

As a result of recent events, we will be taking even further steps to add backup redundancy to the system as well as further measures to keep the system online in the event of a server failure.  We are currently reviewing options and will have a plan in place in the very near future.