• filter:
  • Page :
  • 1
  • Text Only
  • Search this Topic »
rated:
Gitlab.com had this happen to them recently.

A tired sysadmin typed rm -rf (Which is remove all including subdirectories and don't ask me, just do it!) from the 'root' directory...oops

https://www.theregister.co.uk/2017/02/01/gitlab_data_loss/?utm_source=fark&utm_medium=website&utm_content=link&ICID=ref_fark

Gitlab just found out that no one had bothered to check backups.

Member Summary
Staff Summary
Thanks for visiting FatWallet.com. Join for free to remove this ad.

rated:
There's nothing quite like that feeling at the exact moment that you realize just what you've done...

rated:
If these were physical servers, I'd shutdown and run a file recovery on the harddrives. I don't know if that's possible to do at all with an Azure setup, which is apparently what this was. 

rated:
And a followup

https://developers.slashdot.org/story/17/02/02/1453229/gitlab-sa...

Ouch, 6+ hours of updates...gone

rated:
If you aren't testing your backups, you don't have backups.

rated:
One co I worked at would test their backups twice a year where they'd restore their data from the tapes at an offsite location and have the users verify that the data are accurate. Oh, and the data has to be restored in a timely fashion as well.

rated:
ach1199 said:   One co I worked at would test their backups twice a year where they'd restore their data from the tapes at an offsite location and have the users verify that the data are accurate. Oh, and the data has to be restored in a timely fashion as well.
  I can't imagine testing that infrequently.  I get worried when I'm only testing monthly.  Maybe I'm just paranoid, but to me it seems crazy to take a chance like that.  6 months before finding out that my backups are bad?  No thanks.

rated:
minidrag said:   
ach1199 said:   One co I worked at would test their backups twice a year where they'd restore their data from the tapes at an offsite location and have the users verify that the data are accurate. Oh, and the data has to be restored in a timely fashion as well.
  I can't imagine testing that infrequently.  I get worried when I'm only testing monthly.  Maybe I'm just paranoid, but to me it seems crazy to take a chance like that.  6 months before finding out that my backups are bad?  No thanks.
 

  This was a financial institution that had redundancy at a secondary site and pretty solid backup system.  The twice a year recovery from tapes was just to prove that we can successfully recover the data from tapes in case the two main redundancy systems failed (mirrored cluster at an offisite and SAN backup as a daily backup). 

rated:
Ah, I see. That makes a bit more sense.

  • Quick Reply:  Have something quick to contribute? Just reply below and you're done! hide Quick Reply
     
    Click here for full-featured reply.


Disclaimer: By providing links to other sites, FatWallet.com does not guarantee, approve or endorse the information or products available at these sites, nor does a link indicate any association with or endorsement by the linked site to FatWallet.com.

Thanks for visiting FatWallet.com. Join for free to remove this ad.

While FatWallet makes every effort to post correct information, offers are subject to change without notice.
Some exclusions may apply based upon merchant policies.
© 1999-2017