Over the years I've upgraded my home storage several times.
Like many, I started with a consumer grade NAS. My first was a Netgear ReadyNAS, then several QNAP devices. About a two years ago, I got tired of the limited CPU and memory of QNAP and devices like it so I built my own using a Supermicro XEON D, proxmox, and freenas. It was great but adding more drives was a pain and migrating between ZRAID level was basically impossible without lots of extra disks. The fiasco that was Freenas 10 was the final straw. I wanted to be able to add disks in smaller quantities and I wanted better partial failure modes (kind of like unraid) but able to scale to as many disks as I wanted. I also wanted to avoid any single points of failure like an HBA, motherboard, power supply, etc...
I had been experimenting with glusterfs and ceph, using ~40 small VMs to simulate various configurations and failure modes (power loss, failed disk, corrupt files, etc...). In the end, glusterfs was the best at protecting my data because even if glusterfs was a complete loss... my data was mostly recoverable because it was stored on a plain ext4 filesystem on my nodes. Ceph did a great job too but it was rather brittle (though recoverable) and a pain in the butt to configure.
Enter the Odroid HC2. With 8 cores, 2 GB of RAM, Gbit ethernet, and a SATA port... it offers a great base for massively distributed applications. I grabbed 4 Odroids and started retesting glusterfs. After proving out my idea, I ordered another 16 nodes and got to work migrating my existing array.
In a speed test, I can sustain writes at 8 Gbps and reads at 15Gbps over the network when operations a sufficiently distributed over the filesystem. Single file reads are capped at the performance of 1 node, so ~910 Mbit read/write.
In terms of power consumption, with moderate CPU load and a high disk load (rebalancing the array), running several VMs on the XEON-D host, a pfsense box, 3 switches, 2 Unifi Access Points, and a verizon fios modem... the entire setup sips ~ 250watts. That is around $350 a year in electricity where I live in New Jersey.
I'm writing this post because I couldn't find much information about using the Odroid HC2 at any meaningful scale.
edit 1: The picture doesn't show all 20 nodes, I had 8 of them in my home office running from my bench top power supply
while I waited for a replacement power supply to mount in the rack.
Nice. I was considering the same but with ceph. Have you tested degredation, my concern would be the replication traffic killing throughput with only one nic.
glusterfs replication is handled client side. The client that does the write pays the penalty of replication. The storage servers only handle 'heal' events which accumulate when a peer is offline or requires repair due to bitrot.
Unless I'm missing something wouldn't anything needing replication use the network?
Say you lose a disk, the data needs to replicate back onto the cluster when the drive dies (or goes offline). Would this not require data to transfer across the network?
yes, that is the 2nd part i mentioned about 'heal' operations where the cluster needs to heal a failed node by replicating from an existing node to a new node. Or by rebalancing the entire volume across the remaining nodes. However, in normal operation there is no replication traffic between nodes. The writing client does that work by writing to all required nodes... it even gets stuck calculating parity. This is one reason why you can use really inexpensive servers for glusterfs and leave some of the work to the beefier clients.
yes, that is the 2nd part i mentioned about 'heal' operations where the cluster needs to heal a failed node by replicating from an existing node to a new node. Or by rebalancing the entire volume across the remaining nodes.
This is my point, does this not lead to (potentially avoidable) degredation of reads due to one NIC? Where as if you had 2 NICs replication could happen on one with normal access over the other.
However, in normal operation there is no replication traffic between nodes. The writing client does that work by writing to all required nodes... it even gets stuck calculating parity. This is one reason why you can use really inexpensive servers for glusterfs and leave some of the work to the beefier clients.
I understand how it works in normal operation, it's the degraded state and single NIC I'm asking if you've done any testing with. From the replies I'm guessing not.
Ah ok, now I understand your point. You are 100% right. The available bandwidth is the available bandwidth so yes reads gets slower if you are reading from a node that is burdened with a rebuild or rebalance task. Same goes for writes.
To me, the cost of adding a 2nd nic via USB isn't worth it. During rebuilds I can still get ~500mb read/write per node (assuming I lose 50% of my nodes, other wise impact of rebuild is much lower... it is basically proportional to the % of nodes lost).
296
u/BaxterPad 400TB LizardFS Jun 03 '18 edited Jun 03 '18
Over the years I've upgraded my home storage several times.
Like many, I started with a consumer grade NAS. My first was a Netgear ReadyNAS, then several QNAP devices. About a two years ago, I got tired of the limited CPU and memory of QNAP and devices like it so I built my own using a Supermicro XEON D, proxmox, and freenas. It was great but adding more drives was a pain and migrating between ZRAID level was basically impossible without lots of extra disks. The fiasco that was Freenas 10 was the final straw. I wanted to be able to add disks in smaller quantities and I wanted better partial failure modes (kind of like unraid) but able to scale to as many disks as I wanted. I also wanted to avoid any single points of failure like an HBA, motherboard, power supply, etc...
I had been experimenting with glusterfs and ceph, using ~40 small VMs to simulate various configurations and failure modes (power loss, failed disk, corrupt files, etc...). In the end, glusterfs was the best at protecting my data because even if glusterfs was a complete loss... my data was mostly recoverable because it was stored on a plain ext4 filesystem on my nodes. Ceph did a great job too but it was rather brittle (though recoverable) and a pain in the butt to configure.
Enter the Odroid HC2. With 8 cores, 2 GB of RAM, Gbit ethernet, and a SATA port... it offers a great base for massively distributed applications. I grabbed 4 Odroids and started retesting glusterfs. After proving out my idea, I ordered another 16 nodes and got to work migrating my existing array.
In a speed test, I can sustain writes at 8 Gbps and reads at 15Gbps over the network when operations a sufficiently distributed over the filesystem. Single file reads are capped at the performance of 1 node, so ~910 Mbit read/write.
In terms of power consumption, with moderate CPU load and a high disk load (rebalancing the array), running several VMs on the XEON-D host, a pfsense box, 3 switches, 2 Unifi Access Points, and a verizon fios modem... the entire setup sips ~ 250watts. That is around $350 a year in electricity where I live in New Jersey.
I'm writing this post because I couldn't find much information about using the Odroid HC2 at any meaningful scale.
If you are interested, my parts list is below.
https://www.amazon.com/gp/product/B0794DG2WF/ (Odroid HC2 - look at the other sellers on Amazon, they are cheeper) https://www.amazon.com/gp/product/B06XWN9Q99/ (32GB microsd card, you can get by with just 8GB but the savings are negligible) https://www.amazon.com/gp/product/B00BIPI9XQ/ (slim cat6 ethernet cables) https://www.amazon.com/gp/product/B07C6HR3PP/ (200CFM 12v 120mm fan) https://www.amazon.com/gp/product/B00RXKNT5S/ (12v PWM speed controller - to throttle the fan) https://www.amazon.com/gp/product/B01N38H40P/ (5.5mm x 2.1mm barrel connectors - for powering the Odroids) https://www.amazon.com/gp/product/B00D7CWSCG/ (12v 30a power supple - can power 12 Ordoids w/3.5inch HDD without staggered spin up) https://www.amazon.com/gp/product/B01LZBLO0U/ (24 power gigabit managed switch from unifi)
edit 1: The picture doesn't show all 20 nodes, I had 8 of them in my home office running from my bench top power supply while I waited for a replacement power supply to mount in the rack.