Database Loss (Updated 22th of June)

Home

Database Loss (Updated 22th of June)

Posted June 8, 2012 09:14 PM by Webmaster

Update

Most of the data have been recovered, please use the site as normal while we gradually merge in the missing data. If you were a newly registered member but your account was lost in the crash, then please register again with the same email and your collection will be automatically restored. We deeply apologize for all the inconvenience.

This news post has now been unstuck and closed for comments, we refer to this forum thread for further details and questions.


8th of June

We are extremely sad to let you know that we've experienced 7 weeks of database loss. 7 weeks ago we moved to a new much improved server, but unfortunately earlier today the hard drives of the database crashed (was using RAID). On the old server we did daily backups, but since we changed server and setup, the old server backup solution didn't work anymore. We have been discussing the new backup system on a daily basis, but hadn't yet implemented it, so the timing couldn't have been worse.

What is missing the last 7 weeks

  • HT Gallery updates and comments
  • All forum activity
  • New user registrations
  • Cast & crew
  • Any other database data other than mentioned below
So far we have managed to recover

  • The complete Blu-ray database
  • The complete Blu-ray review database
  • The close to complete DVD database
  • The majority of the theatrical database
  • The complete movie ratings
  • The complete price trackings
  • The complete price history
  • The complete Blu-ray user reviews
  • The complete theatrical, hardware and games user reviews
  • A few weeks of news
  • Most of the collections (work in progress)
  • A few weeks of news
We are so terribly sorry about this and we feel the worst we have done in years, we've been literally sick to our stomachs and working non-stop when we realized we couldn't recover the hard drive and only had an old backup.

We would like to give our deepest apologies to our members, because it's also your data we have lost and what we feel the absolutely worst about. Blu-ray.com is built as a community and wiki and is nothing without its members, collectors and contributors. We basically beg on our knees that you update your last 7 weeks of collection updates and other things you have lost. New Android collectors will have to register again and re-add their complete collections :(
We will try to make it up to you the very best we can.

We have a complete image dump of the crashed hard drive that we are trying to dig information in and we will do our best to try to recover more data, but it doesn't look very good. Because we deem the chance of recovering more data low, the site is in a live state and all changes you make will stick. If we find more data we will try to append it, but as we said, the chances seem very slim.

Again, our deepest apologies.

Update 9th of June

We have managed to restore most of the theatrical movie database and we can restore more of it, so it should be almost complete when we are done. We are doing backups and also working on the hard drive image to see what (if any) we can rescue. We think we can restore at least some of the collections (those members who were logged in 24h prior to crash). This has 100% priority of course.

Update 10th of June

The majority of the movies/theatrical database have been restored. The complete (or almost complete) DVD database should have been restored. We should also be able to restore at least a few thousand member collections and we are analyzing the crashed hard drive image and there seem to be at least partially intact information, so we are trying to see what we can restore.

Update 11th of June

The forum search function is back online now that we've had time to fully rebuild the post and thread search indexes. As for a recovery progress report, a lot of data seem to be intact on the old hard drive image, the problem is that the file system is corrupted and it seems impossible to repair, so extracting the old data is like one gigantic puzzle and a quick solution doesn't seem like an option for now. Besides regular backups we are also working on setting up several mirrors for redundacy so this will never happen again. Theatrical posters will be restored tomorrow and also about 2500 collections, we apologize again for not fixing that sooner.

Update 12th of June

We have restored one thousand theatrical posters, unfortunately the meta data with member credits couldn't be restored, but we have assumed that whoever created the theatrical entry also uploaded the poster (which is correct most of the time) and has such been assigned the credit for the poster as well. To make sure we don't make any mistakes, we want to do some more testing before inserting the 2500 collections, so that will unfortunately have to wait until tomorrow. We have also worked more on the backup routines and the hard drive image. Within 3-4 days we should know better if/how much we can restore from the image.

Update 2
We've had a possible breakthrough and it might actually be possible to recover a lot of the old data, including all collections, movie reviews, price trackings etc. No promises, but it looks quite good. We will not do a simple rollback to the state of the 8th, so you will not lose any information you've updated the last few days. We will get back with more updates.

Update 3
If you were a newly registered user the last 7-8 weeks and your account was lost, then please register again with the same email and there's a good chance we can restore your entire Blu-ray collection and price trackings.

Update 13th of June

We've been able to recover several databases, among them is the complete collections database. We are in the process of merging the current state of the collections with the old state, so you don't lose any recent updates and at the same time getting back your old collection including old meta data (comments, prices etc). We will first merge members who did not lose their accounts in the crash and then see what we can do about new members.

Update 2
The majority of the Blu-ray collections have been restored. We have not yet restored lost member's collections and not yet titles where the collection category was changed during the crash window. We are working on this. If you had a registered account prior to crash but it was lost after crash, then please register again with the same email address and there's a good chance we can recover your collection.

Update 14th of June

You can now see exactly which titles have been recovered by clicking on "Recovered" in red in your collection. For those of you who deleted your complete collection in anger because of the 7 week window loss, please let us know and we can restore that part as well ;)
We are working on restoring some lost meta data and also if you changed any custom collection categories in the 7 week window. New members who have registered again with the same email should now have their complete Blu-ray collection back. We are also working on restoring DVDs, theatrical movies and games, but you will unfortunately have to wait a bit longer. We will restore part for part of the 7 weeks of missing data, so please continue using the site as normal and we will merge in missing data gradually.

Update 15th of June

We are almost done with the Blu-ray collection meta data and some collections with custom categories that are in limbo. The reason for delay is that the merge is somewhat complex and we don't want to mess things up (even more). The preliminary restore schedule/priority is: Blu-ray collections, DVD collections, theatrical collections, PS3/Xbox360 collections, price trackings, movie ratings, Blu-ray user reviews, PMs. After that we will continue with HT galleries/comments, favorites, some more theatrical/DVD fixes etc. We are trying to restore personal user data first and in the order we think members want them the most. If you for example have readded collections or price trackings after the crash, we will naturally not overwrite them, everything will be an unique merge with your personal data before and after crash. We have downgraded the priority of forum posts and forum threads, because they are somewhat more complex to merge well and those databases were also unfortunately a bit smashed up in the crash and are incomplete at the moment (we will try to fix them as well, but priority is downgraded). We will try to recover things first in a combination of what we think members want the most and what we think we can fix within an acceptable time frame. We will work on this more or less over the whole summer, but unfortunately we won't be able to do a perfect restoration, but we will do our utmost to make it an acceptable restoration.

Update 18th of June

  • All price trackings have been restored (including lost members')
  • All movie ratings have been restored (including lost members').
  • All price history information/graphs for Blu-ray movies, DVD movies and all other categories have been fully restored.
  • Lost members' collections have been restored (including BD, DVD, PS3, Xbox360 and custom categories).

  • We are working on meta data for collections-, custom categories- and DVD/PS3/Xbox360 collections for old members.

    Update 19th of June

  • Blu-ray user reviews have been restored in full for all members.
  • BDs in custom categories, DVDs, PS3 and Xbox360 collections have now been restored for all members. Unfortunately some of the newly added DVDs, PS3s and Xbox360 games are still lost and such we haven't been able to re-add them to your collection.

  • Meta data for collection updates in the 7-week window should be fixed tomorrow.

    Update 20th of June

    In addition to all Blu-ray user reviews, all user reviews (theatrical, hardware, games etc) have now been restored. Unfortunately it looks like we need one more day for the collection meta data to be restored.

    Update 22th of June

    The collection merge/update function is ready, but we also want to make it 100% reversible just in case, so unfortunately it has been delayed further. At least it should be done sometime next week. We are also working on restoring more of the DVD database including credits and the HT galleries.