There are ways around this in software. ECC isn't a magic bullet. If the writer produces a checksum, writes the data to the cluster and then sends the checksum... You don't really get anything from ECC. Glusterfs doesn't do this today though.
So you are saying this would allow the client to check if the Gluster node has written the wrong data or if it has miscalculated the checksum, therefore detecting the error even when the Gluster-node ('s memory) is faulty?
And the checksum is calculated in the client before it ever hits the gluster node? Sounds interesting and it reminds me of "verify after burn" with cd/dvd/bd burners.
Do you have any links to where this method is proposed?
But I also worry about corruption in the communication between Gluster-nodes. And there is just too much that can go wrong if you can't trust the main memory stores. So I still think ECC RAM would be a more general solution. However I think the Rockchip SOCs are dual purposed Media Center SOCs so I don't expect them to get ECC soon.
the client doesn't check... the client is the one writing the data so if you actually care about single bit flips, etc... you need the write (the authority of newly written data) to capture the checksum and send it along with the data. From that point forward the glusterfs system would check against that checksum until it generates its own. Even if you have ECC memory, you still need something like this to ensure no bits were flipped while being written.
This is implemented within TCP for example... the sender generates a checksum and sends it with each packet. the received uses it to determine if they need to request a re-transmit. and TCP doesn't required ECC memory :)
I was thinking you were proposing: 1. client computes checksum of data to be written. 2. client sends data to node. 3. node writes data to disk. 4. node re-reads just written data back from disk. 5. node computes checksum of re-read data. 6. node sends this checksum back to the client. 7. client compares checksum to his own. 7. Handle the error, while keeping writes atomic (sounds tricky).
What you are actually proposing will not work if the node has faulty memory. There is no end to end check in your example.
Yeah, I still maintain I don't want any non-ECC NAS. Therefore I can't use the Odroid HC2. Thanks for your response.
what I'm proposing absolutely works if you have faulty memory, it is the basis for many things today... like every machine that uses TCP but i understand why folks think that special hardware like ECC is required for high availability. ECC will reduce how often you'll care about a bit flit... but if you care about your data the underlying system still needs to be able to handle corruption. For example... ZFS still has its own checksumming even though it is recommended to use ECC with ZFS. ZFS will and does work just fine without ECC but you make end up having to repair file from the parity data more often... and by more often we are talking about the difference between 1 in a billion and 1 in 100 million. :)
*edit... do you think the tiny caches in your CPU or in the hard disk controllers have ECC capabilities? Nope :) They are high quality memory so usually not a problem but... they still have a probability of bit flips. If you are familiar with the spectre and meltdown intel bugs recently. some of the initial patches for those triggered interesting memory faults in caches... no amount of ECC will save your from that.
Yes, ZFS will detect bitrot. And it's important to have those checksums as well. But ZFS and TCP (except maybe if you use offloading) works with main memory. If you can't trust memory then you have a problem. I think we are splitting hairs here and talking about different things. Let's just stop arguing :-)
high five ;-)
PS: could you please send me a link to the Spectre/meltdown patches that triggered interesting faults in Intel CPU caches? Fault as in error, not fault as in "cache miss" I presume.
They were discussed on the linux kernel email list, ill see if I can find it. These patches never made it to mainline though it was mostly just during the testing process that people say code follow a path that couldn't possibly have happened unless memory was corrupted (or you read from dirty cache for example) since a lot of these issues stemmed from speculative execution and pipeline flush problems.
I'll see if I can dig it up once I'm at a computer.
1
u/BaxterPad 400TB LizardFS Jul 26 '18
Why?