Computer Science Canada

Data Backup
Author:  md [ Mon Apr 02, 2007 12:57 pm ]
Post subject:  Data Backup
Ok; so as everyone who lurks on IRC knows I have a rediculous amount of storage. The problem is that currently it's all in a raid formated with XFS. For you windows users that would be a linux/unix file system.

My problem is that XFS seems to be slowly corrupting files, and in such a manner that there is almost no way to tell that it's happening without opening every file (there are too many to do that). My solution is just to delete/replace corrupted files as I find them, but that still leaves XFS corrupting my files.

Now then... what I'd like to do is convert the filesystem to ReiserFS, which has never caused me any troubles at all. That means that I need to backup all my data elsewhere first; specifically I need to backuo 710GB of data. Currently I don't have a dvd burner but I am strongly inclined to invest in one if that's what it takes to backup everything.

Can anyone else think of a good way of backing things up? Preferably without spending oodles of money Razz
Author:  haskell [ Mon Apr 02, 2007 1:29 pm ]
Post subject:  RE:Data Backup
Abuse free hosting?

They are asking for it...
Author:  Clayton [ Mon Apr 02, 2007 2:38 pm ]
Post subject:  RE:Data Backup
For 710GB of data? That's ridiculous Confused I told you this would come back and bite you in the ass one day md Razz

A DVD burner is about the only thing I can think of right now (at least for that amount of data).
Author:  md [ Mon Apr 02, 2007 3:16 pm ]
Post subject:  RE:Data Backup
I wonder if there is a way of shrinking linux software raids... because I would prefer not to buy ~180 DVDs and a dvd burner solely to back things up.
Author:  Drakain Zeil [ Mon Apr 02, 2007 3:27 pm ]
Post subject:  RE:Data Backup
portable 100+gb hd
transfer to another hd/computer(over ethernet).
Author:  md [ Mon Apr 02, 2007 3:50 pm ]
Post subject:  Re: RE:Data Backup
Drakain Zeil @ 2007-04-02, 3:27 pm wrote:
portable 100+gb hd
transfer to another hd/computer(over ethernet).


See 100GB is not even close to what I need to backup Wink

I have a solution though... I think. Deleted 100GB of movies that were crap and now if I buy another 320 drive I *should* be able to manage some combination of failing drives in my raid and copying that'll allow me to back everything up... I hope.
Author:  Clayton [ Mon Apr 02, 2007 4:35 pm ]
Post subject:  RE:Data Backup
md wrote:

if I buy another 320 drive I *should* be able to manage some combination of failing drives in my raid and copying that'll allow me to back everything up... I hope.


You like failing drives? You sir, have a strange taste in computer hardware.
Author:  md [ Mon Apr 02, 2007 4:47 pm ]
Post subject:  RE:Data Backup
By fail I mean flag it as failed in the RAID5, which means I can remove it and use it for something else; although that does mean my RAID5 will be dangerously close to complete collapse.
Author:  Clayton [ Mon Apr 02, 2007 4:49 pm ]
Post subject:  RE:Data Backup
Embarassed yes.... I uh.... there's uh... no excuse... I'll just go sulk in the corner now
Author:  klopyrev [ Mon Apr 02, 2007 5:13 pm ]
Post subject:  Re: Data Backup
Forgive me for interupting your conversation, but what exactly do you need so much harddrive space for?
Author:  Clayton [ Mon Apr 02, 2007 6:07 pm ]
Post subject:  RE:Data Backup
Music, Video, documents personal stuff, etc.
Author:  Mazer [ Mon Apr 02, 2007 7:36 pm ]
Post subject:  RE:Data Backup
Video editing can take up crazy huge amounts of space.
Author:  TheFerret [ Mon Apr 02, 2007 8:33 pm ]
Post subject:  RE:Data Backup
Your best way would be to resize you raid down to the smallest size it can go then create the new partition then move some data over, and repeat resizing the partition smaller and smaller until its done...
Author:  md [ Mon Apr 02, 2007 11:05 pm ]
Post subject:  RE:Data Backup
XFS cannot be shunk; another reason I am moving away from it.

Mostly I use my storage for media; music is near 100GB, movies is about 200, TV shows are another 300. Documents, in progress downloads, projects, and pictures make up the other 100GB.
Author:  Mazer [ Tue Apr 03, 2007 11:12 am ]
Post subject:  RE:Data Backup
Why did you pick XFS in the first place? I don't know anything about it, and I hardly know anything about ext3 or reiserFS either, I'm just curious. Does it have any advantages?
Author:  rdrake [ Tue Apr 03, 2007 12:06 pm ]
Post subject:  Re: RE:Data Backup
Mazer @ Tue Apr 03, 2007 11:12 am wrote:
Why did you pick XFS in the first place? I don't know anything about it, and I hardly know anything about ext3 or reiserFS either, I'm just curious. Does it have any advantages?
It's very high performance. Trade offs being of course that it cannot be shrunk and it apparently corrupts files. People would use it where their partition layouts wouldn't change, and they need fast performance.

Oh, and never power down improperly when you have an XFS file system. Also a UPS would be a good investment. Random power outages and the like corrupt files like md is experiencing.
Author:  md [ Tue Apr 03, 2007 1:58 pm ]
Post subject:  RE:Data Backup
XFS has support for extended attributes; as well as being slightly faster for reading and writing large files (movie sized).

ReiserFS is almost as fast as xfs; reiser4 is faster. Both are much faster at deleting files, but slightly slower at accessing large files.

At the time I was not worried about things crashing because I didn't think it would be such a big problem. Unfortunately I have had more then a couple power failures since then and some random crashes due to some unknown cause. That combined with XFS's aggressive memory caching is what lead to the corruption.
Author:  Bobrobyn [ Wed Apr 04, 2007 2:47 am ]
Post subject:  Re: Data Backup
I'd say the best thing to do to back up that much media is go to your friends and beg. I happen to have a friend with a 5 TB fileserver, which would have lots of room for me. But yeah, throwing 30 GB's here, and there, throughout your friends harddrives always works. Especially if you offer them any of your media for the help.....or you can go with the expensive solution. Whatever floats your boat.
Author:  md [ Wed Apr 04, 2007 11:51 am ]
Post subject:  RE:Data Backup
Well I've managed to back things up with only minimal deleting and an acceptable amount of risk. Ended up getting another drive and manipulating my current storage setup some.

Now I get to have 1.4T of space; 1.6 if I replace some old drives with larger ones.

Spreading data out among friends really doesn't work when they are no where near close by (most are in waterloo) or if htey are here are not very tech savvy. Also... who is this friend who has 5TB of storage; and how did he manage to get that?

I have 4 320s and 2 250s as well as a 120 for my root; that fills my case. How did he manage to get that much storage?
Author:  Bobrobyn [ Wed Apr 04, 2007 12:13 pm ]
Post subject:  Re: Data Backup
He researched it for years...it's basically a 5 TB file server, in a RAID set up. I'm not sure the EXACT amount of space, but I know it's around 5 TB. I know it involves 16 300 GB hard drives (So yeah, at least 4.8 TB's). I'll give you the website where he writes about how he did it, sometime, when it goes back up.
Author:  md [ Thu Apr 05, 2007 1:52 pm ]
Post subject:  RE:Data Backup
Thanks all for the ideas; but as always throwing money at the problem solved everything.

New disk stats for anyone curious as to just how much storage exists on this computer...

Quote:
[john@saxifrage ~]$ df -H
Filesystem Size Used Avail Use% Mounted on
/dev/hda3 60G 6.7G 54G 12% /
udev 662M 3.0M 659M 1% /dev
/dev/mapper/raid--lvm-home
1.4T 561G 830G 41% /home/john
none 2.2G 8.2k 2.2G 1% /tmp
none 3.3G 0 3.3G 0% /var/tmp/portage
shm 662M 0 662M 0% /dev/shm
192.168.1.5:/home/storage
21G 5.7G 15G 29% /home/storage
Author:  Bobrobyn [ Fri Apr 06, 2007 10:35 am ]
Post subject:  Re: Data Backup
That's awesome. Enjoy your disk space!
Author:  bugzpodder [ Fri Apr 06, 2007 11:44 am ]
Post subject:  RE:Data Backup
reiser4 is buggy and the lead programmer hans reiser is charged for murdering his wife.
Author:  rizzix [ Fri Apr 06, 2007 12:41 pm ]
Post subject:  RE:Data Backup
ZFS! Don't make the same mistake again. If necessary install Solaris (10 or OpenSolaris) as as your dataserver.
Heh.

Oh and I heard bad stuff about Reiser4 too.
Author:  md [ Fri Apr 06, 2007 8:38 pm ]
Post subject:  Re: RE:Data Backup
bugzpodder @ 2007-04-06, 11:44 am wrote:
reiser4 is buggy and the lead programmer hans reiser is charged for murdering his wife.


Reiser4 is definitely buggy; I tried it a year ago and it was fast as hell... but had issues. Hans' being charged with murdering his wife (really from what I have read it's hard to prove at all) has absolutely nothing to do with the filesystem on the other hand. Insinuating a relation where none exists is silly Wink

ZFS is SLOW, very slow. And there is no linux port. So for all intensive purposes it sucks too.
Author:  tro [ Mon Apr 30, 2007 12:19 am ]
Post subject:  RE:Data Backup
You could have also used http://mozy.com/ , but their client is windows-only right now, I believe, so you'd have to throw wine or vmware into the mix, but it's cheap.
Author:  md [ Mon Apr 30, 2007 12:47 am ]
Post subject:  RE:Data Backup
Pay money? Eww! Besides; backing up 600GB over the internet is never a good idea, WAY too slow; even over a 100mb lan.

:


You can syndicate this boards posts using the file backend.php or view the topic map using sitemap.php.

Terms of Use | Privacy Policy