Predis claim to have Client-side sharding (support for consistent hashing of keys). http://github.com/nrk/predis
I can do sharding using connect to an array of profiles (nodes) but it isn t consistent hashing. When I add another node to the pool, some of the keys can t be found. Anyone has any experience on this?
Using php 5.2 (and redis s php 5.2 version).
The official Redis site says "Redis supports client-side sharding via consistent hashing. Currently there is no support for fail tolerance nor to add or remove clusters at run time."
From what I understand at the moment this kind of sharing is not fault tolerant, and all keys stored on a failed node will be lost. Equally if you add a new node, some portion of the key space will now be lost (as the keys will be stored on the wrong node). Normally in a consistent hashing system, when a new node joins it copies all the keys which now map to it from its neighbours. There is no support in the Redis server to do this.
So the consistent hashing works fine if you are using Redis as a cache, where the actually data is stored behind Redis, but for the moment don t expect your data to not go missing.
UPDATE: It is possible to implement real sharding via a consistent hashing library called ketama.
The solution is to use virtual sharding. I don t know Predis framework works but I predict that it uses some kind of array - you probably fill it with information about each shard at start-up.
Assume that you will have maximum of 10 shards (this number should be unlikely to be reached). Then, create sharding array that points to only 3 real servers. In the future when you will add new nodes, you will migrate related data to new shard and change mapping. This approach preserves form changing hash function.
Initial mapping:
0 => 0 //node #0
1 => 0
2 => 0
3 => 1 //node #1
4 => 1
5 => 1
6 => 2 //node #2
7 => 2
8 => 2
9 => 2
When you adds new node you only change mapping:
0 => 0
1 => 0
2 => 3 // new node #3
3 => 1
4 => 1
5 => 3 // new node #3
7 => 2
8 => 2
9 => 3 // new node #3
so you have to move data with h(x) = 9 or 5 or 2 to node #3.
Say I have a "message" table with 2 secondary indexes: "recipient_id" "sender_id" I want to shard the "message" table by "recipient_id". That way to retrieve all messages sent to a certain recipient ...
When performing many inserts into a database I would usually have code like this: using (var connection = new SqlConnection(connStr)) { connection.Open(); foreach (var item in items) { var ...
Does anyone have any good sources of information of using NHibernate with Sql Azure with the implications of sharding (because of the 10gb cap)? I know there are posts on the internet that reference a ...
I have a problem where I need to assemble a Map whose eventual size is in the GBs (going on past 64GB) and I cannot assume that a user of the program will have this kind of monster machine hanging ...
Predis claim to have Client-side sharding (support for consistent hashing of keys). http://github.com/nrk/predis I can do sharding using connect to an array of profiles (nodes) but it isn t ...
I m trying to run a sample program for hibernate shards. I m all done with it but whenever I run the test program I get an Exception javax.naming.NoInitialContextException: Need to specify class ...
I m new to both memcached & sharding. I gone though some articles on both. I need to implement both in my application. Articles I d gone through were good but none of them gave me how to use ...
Looking to shard a database and to assign different users to different home servers based on their user id. User IDs are 10 character strings, e.g., "f4gKUKkj91" ... each server has an ID of 1 - 1000....