How does everyone store their bulk data these days? Was a time I could just have a monster drive for "data" mount the smaller drives off that and let he rip, and that kept me going for years...
How does everyone store their bulk data these days?
Also thought about RAID for a bit, but I really don't have enough
similar capacity devices of sufficient size to make it all worthwhile.
And this is the problem I find myself with now. I haven't managed a
good run with drives over 2TB. They tend to like to fail for me :(
I've been handed a 2.0TB drive which is mostly full, and find myself unable to copy the whole thing.... <sigh> Hence shuffling... and boy
does it take time...
How does everyone store their bulk data these days? Was a time I could just have a monster drive for "data" mount the smaller drives off that and let her rip, and that kept me going for years...
hear bad things about MTBF and monster sized RAID arrays. If you have enough of an error on one drive, chances are theres another one somewhere due to the amount of space in use rendering your RAID array dud.
If you are really dead serious you build a NAS (for home storage). Something like RAID 10 should do for monster deployments.
FreeNAS with ZFS is a popular option. Prepare to spend kilobucks.
I do IT all day everyday & didn't want to have to end up "fixing" something at home or frankenstein'ing something together, so I ended
up buying a Drobo 5N NAS. The cool thing with Drobo is you can mix
& match different sized drives and it'll "just work". I started
out with three drives (2 1TB WD Black drives and one other Seagate
1TB I had laying around) - as needs changed I bought two more 4TB
WD Red drives to fill the two remaining bays and (over time) swapped
out the original 3 drives for 4TB Red's as well. (5 x 4TB = 20TB
w/ 14.35TB usable after redundancy).
Arelor wrote to Spectre <=-
If you are really dead serious you build a NAS (for home storage). Something like RAID 10 should do for monster deployments. FreeNAS with
ZFS is a popular option. Prepare to spend kilobucks.
Warpslide wrote to Spectre <=-
There is an option to turn on dual drive redundancy if I wanted to be
able to loose two drives at once, costing more free space, but I
haven't seen the need to do that.
All in, I don't think I have enough of the same size drives to make RAID anysort of real possibility and although I like th
idea, I tend to try and maximuse the capacity over redundancy.
I spose I'm looking for ideas as much as anything. I'm presently poking about trying to organise 3 storage options.
There's a lot to be said for staggering drive build times and lot
numbers. I built my home RAID 1 with two drives from the same lot,
and they both failed within weeks of each other - albeit after 3
years.
to get around to shipping that fucking thing back to Seagate. Yeah - it
was a Seagate. Fucking piece of garbage.
I also recently sync'd a third drive and drove it a couple of hundred miles to my folks house and plugged it in to a Pi there. Now each night it also syncs over a VPN. Having offsite backup is quite important to
me in case of fire, theft etc.
Switching over to FreeNAS and ZFS, the array is done at the file placement level (I believe), so a failed drive only needs to reassemble the array to the utilised capacity, not the capacity of the drive. Plus (IIRC), ZFS regularly checks write stripes, yells when it senses problems and moves
the data while it can still be read with the other drives.
RAID 10 seems like overkill with SSDs. I started building RAID 5
arrays with SSDs to maximize disk space and use RAID 10 for the
larger SAS arrays to give them a bit of a speed boost. Raid 5 with
SSDs seems plenty fast. But, I'm just supporting development work,
nothing production or revenue generating.
On Jun 28th 4:04 am Spectre said...
I spose I'm looking for ideas as much as anything. I'm presently poki about trying to organise 3 storage options.
I have a RAID 1 mirror on one server at home but I almost prefer my little raspberry PI NAS for it's simplicity. Two 2.5 inch USB drives plugged in an
to get around to shipping that fucking thing back to Seagate. Yeah - i was a Seagate. Fucking piece of garbage.
I have owned lots of hard drives over the years - all different manufacturers. Lots of years/decades of running lots of different drives and measuring how long they last (both at home and work). Mostly
running them 24x7, no off/on.
Seagate drives that I have owned generally last 4-5 years at the most.
I had built several RAID arrays using Seagate drives and after 4-5
years, had to replace 85% of them due to errors.
The best hard drive (spinning disk) that money can buy is Hitachi HGST drives. The enterprise drives, even better. They almost never fail.
That is all I buy now - nothing else. One of my HGST drives is almost
14 years old, still works perfect with zero errors. I also have a huge
36 drive array of 2TB HGSTs that I use for temporary backups and all
those drives are 9 years old and work perfect.
On 28 Jun 2020, pokeswithastick said the following...
I also recently sync'd a third drive and drove it a couple of hundred
miles to my folks house and plugged it in to a Pi there. Now each night
it also syncs over a VPN. Having offsite backup is quite important to
me in case of fire, theft etc.
That's actually a clever idea. My Mom now has fibre to the home provided by the local hydro company. She pays a lot less for 1Gb up & down than I do for 1Gb down/30Mb up over coax. I wonder if I could convince her to
let me set something like that up.
I've got a few hitachis and they're very good. Reliable, not too slow,
not to mention affordable. I may forgo that RMA I've filed and just buy a 4TB Hitachi.
miles to my folks house and plugged it in to a Pi there. Now each night it also syncs over a VPN. Having offsite backup is quite important to
me in case of fire, theft etc.
Is the "fibre.. by the local hydro company" in Ontario?
Fun fact, I once considered doing something like that. Howeve, the PI has very slow I/O. If you want to access a big ammount of data at once I suspect you will be up for some waiting.
do for 1Gb down/30Mb up over coax. I wonder if I could convince her to let me set something like that up.
I'll put in a plug for Resilio Sync, which is a program that you run on a couple (or more) computers, and share a directory.
Mind you, it currently doesn't work for me because I can't get my brother to keep my other computer on, but if you can find someone to host a
On 28 Jun 2020, Ogg said the following...
Is the "fibre.. by the local hydro company" in Ontario?
Yup, Lakeland Networks in Muskoka:
https://www.lakelandnetworks.com
It's even available at my Dad's house now "in the middle of nowhere". Not that he'd ever get internet. Up until this, his only option was a
wireless solution, whether "cottage" internet (LTE) or a fixed tower based setup.
Sounds like you need something that can work with a 'sometimes on' peer.
I would have thought that if Resilio is built around the Bittorrent protocol that this should in theory be possible.
It is not clear on that site that that is a HydroOne initiatiative (or,
I can't spot it).
It sounds like they are offering leasing opportunities to resell data transmission via their established fibre networks. Maybe *that* is how LakelandNetworks is doing it?
Weatherman wrote to Poindexter Fortran <=-
One of the new home servers I built using (2) 1TB SSD drives in a
mirror (for the server OS boot drive). Eventually the plan is to get
(8) more Samsung 1TB SSDs in a RAID5 for my new ESXi datastore. Maybe
go with larger SSDs if they price makes sense.
They are getting cheaper.
I get 60/20 over powerline, but it drops out, making a NAS a problem.
Would love to get all of my storage off of my desktop and onto the
network, but need a proper network first.
Weatherman wrote to Poindexter Fortran <=-
Wired is a must for solid performance. Yesterday evening I spent some time getting my new servers ready with the 10G fiber NICs. Those won't
be able to get maxed with the hardware I have in place, even with SSD drives due to the controller speed limitation.
Adept wrote to pokeswithastick <=-
I'll put in a plug for Resilio Sync, which is a program that you run on
a couple (or more) computers, and share a directory.
I know Lakeland Power is the local utility in Bracebridge, but my Dad who lives outside of the town is with Hydro One. Could be a bit of both? In town has cheaper pricing than out of town, maybe because they're using leased fibre?
What was a nice surprise was finding the btsecrets subreddit, where
people were sharing a ton of stuff.
The only problem? I think I have a skunk living under there... We've
had an uneasy truce so far.
I kinda wish there was a similar initiative in my area for stringing up fibre on the exisiting poles so that I could get off my capped at 5GB/mo mobile ISP solution.
If you do decide to run this cable, PLEASE film it so we can can all see it on YouTube... ;)
I finally figured out a path from my downstairs router to my upstairs office. I'll need to go under the house, along the basement, through
a foundation vent, then up through the floor.
The only problem? I think I have a skunk living under there... We've
had an uneasy truce so far.
Yeah, so I can be known as that "got sprayed with a skunk, screamed like
a little girl, knocked himself unconscious on a overhead floor joist and ended up face down with plumber's butt" guy.
Then I think you can just create a volume group spanning all your spare drives using LVM. But that does not look very solid to me. I hope
you have a good backup plan if you try.
Sysop: | sneaky |
---|---|
Location: | Ashburton,NZ |
Users: | 31 |
Nodes: | 8 (0 / 8) |
Uptime: | 154:10:56 |
Calls: | 2,074 |
Files: | 11,137 |
Messages: | 946,945 |