Distributed - but when do we over-react

[DRAFT!] 

TL;DR: Distributing a Database helps with Sharding and Resilience. But do I really want my data spread out over 10s or 100s of "nodes" ? 


Background.

Been investigating Distributed Databases.


Finding: Scale out means Fragmentation of Data.

Adding nodes to a cluster can help to improve both 

Resilience, by having at least 2 spare copies and a voting mehanism, and 

Scalability, by adding more workers, more CPU, more memory.

But the downside that I see (or imagine) is that my data gets too fragmented, too much spread out. 


Solution: Shared Disks ? 

For Resilience, I would prefer to keep 3, 5, or maybe even 7 copies of my data. But not more.  Too many copies mean too much overhead on writing and voting.

For Sharding, I want the data spread, but not fragmented over separated storage components any more than necessary.

For some "scale out" situations, I may want more CPU and more Memory, but maybe not a further fragmentation of my data on (non-shared) disks.

Under sudden increase of traffic, load, I may want to scale up (quickly!) to 37 nodes, to add CPU and memory to my system, but I dont want to have to look after 37 storage-partitions. And I dont want to invest effort in re-distributing shards over 37 disks, especialy not under load.

Alternatively, when scaling back down from 37 nodes to, say, 9 (3 regions, 3 zones), I dont want the system to have to re-consolidate my data from 37 back to 9 storage-partitions.

Ideally, Scaling up is a matter of CPU and Memory, not for Storage, at least not by moving data around to occupy new storage-partitions. And scaling down is actually quite difficult if you have just spread out over multiple non-shared storage partitions.

Note: On scale-down, I can also imagine re-mounting 37 storage-partitions to just 9 nodes, but that would be complicated in the databases I have observed so far.


In Short:

Distributed Databases use multiple nodes (instances, shard) to achieve both Resilience and Sharding. But I hesitate to spread my data out over too-many pieces of storage.


Additional thoughts

Scale-down in both CRDB and YB seems to involve manually moving data back to the remaining nodes.


Comments

Post a Comment

Popular posts from this blog

yugabyte : Finding the data, in a truly Distributed Database

Image
 TL;DR: I dig deeper into yugabyte to find/verify how the various components work, and where the data is kept. I find, with a little digging, that the data is Indeed Distributed and Replicated. Nice! Background In a previous post, I did some RTFM and identified various components that make up a running yugabyte "universe" or a cluster. Now it is time to find the actual data, and explore some of the tools.  hidden-agenda, wishlist: I would really like to be able to "query" this data straight from psql or cql, but that would require additions by yugabyte similar to how postgres exposes its catalog. In a distributed  environment, this is more of a challenge than in a "monolith" where everything is nicely in 1 place. Setup and Tools My "cluster" consists of 4 to 6 nodes running in docker. The setup is described in an earlier blogpost, and was a good starting point. But by now I begin to see I might want to use specific nodes for master and for tserve
Read more

Testing Resilience of CockroachDB - Replacing 1 node.

Image
 TL;DR: I'm using a multi-node cluster to see what happens if I remove and add nodes. Spoiler: it is Resilient, but within limits. The Replication-Factor determines how many nodes you can loose without impacting availability. And there is a more or less long period of vulnerability after losing a node: Some ranges will run witout their "3rd replica" for a while, which makes the system vulnerable until all the under-replicated ranges are fixed. Background: A distributed database uses Replication of data to protect against failure. In the case of cockroach, the default replication factor is 3 and it uses the raft-protocol to determine who is the "leaseholder" and whether a large-enough (majority) of replicas is available to continue, the Quorum. Cockroach also features a more or less "automatic repair" to cover situations where a node disappears. Testing: 3 nodes, a ha-proxy, and a running insert. Nodes are called roach1 ... roach3, and additional nodes
Read more

yugabyte terminology and related information

Image
[Work in Progress - notably needs an ER-diagram to help visualize]  TL;DR: This text will Identify the parts (the objects, entities) that make up a YB-universe and try to clarify their function.  In a later post, I will try to use the web-consoles at ports 7000 and 9000, and experiment with yb-admin and the yugatool to find some of the available information about these objects. For the momet this post will collect my notes, and hopefully result in a readable article at some point. Background When thinking about a system, it helps me to have an ERD-view of the components. In this post I will try to find the yugabyte terminology, and try to interpret it from what I see and what I can peek out of the yb-tooling. I'm keeping the xCluster replication out of scope for the moment, but that should be included ASAP, as I think it is important.  Although the web-exposed pages contains some performance-data, I will also keep that out of scope for the moment (check ports 7000, 9000 and 15433,
Read more