random($foo) - Latest Comments in Some Notes on Distributed Key Stores

Re: Some Notes on Distributed Key Stores

lhl — Tue, 27 Apr 2010 02:58:08 -0000

Well, you could try something like repcached I suppose, which as long as you can repopulate from a canonical store if everything goes down, should work.

Re: Some Notes on Distributed Key Stores

GPN — Wed, 21 Apr 2010 16:01:23 -0000

Thanks for the useful write up. Certainly helped me go through a lot of options fast.

I am facing kind of an different angle on the distribution problem, was wondering what you may recommend for it. Description:

A SMALL set of data that is both read & write. Each server holds a FULL copy of the data (since its small). However must be distributed between MANY MANY servers. When one server updates its local copy, the update must be propagated to the others. "Eventual consistency" is good enough but should be seconds not hours. Nice to have: bindings to PHP, C/C++.

The catch - do not want to add a data layer (no NFS, no DB) but rather keep it all between the symmetric identical "many many servers".

Any suggestions other than "roll your own" ?

Re: Some Notes on Distributed Key Stores

lhl — Thu, 08 Apr 2010 23:18:47 -0000

The client wasn't considering commercial systems - but that's probably a bias of most people building things in the web startup space - a bias which I also happen to share.

But not *entirely* w/o good reason. For example, I've never heard of Caché (nor has it come up in any of the NoSQL conversations in the community I track until now) - the chances are that any startup will be able to hire someone w/ expertise in that system is probably pretty low. I ended up putting together a system in less time that it would have taken to schedule a sales call, much less go through the evaluation process (built and launched for client over the weekend). Certainly for less cost than the evaluation time as well. Not to mention the inevitable incremental costs for upgrades, additional licensing, and support that comes w/ enterprise software.

In fact, I can't think of any large internet companies that isn't based primarily on an open source infrastructure supplemented by their own code (MS is an exception, but not in terms of depending on 3rd party technology, just avoiding most of the open source stuff).

I think for most startups the risk/expense/agility curve is just isn't a good fit for most niche 3rd party software infrastructure.

(Oh, also, peeking at the docs Caché doesn't support language bindings for the languages that the startup used, so that's another strike.)

Re: Some Notes on Distributed Key Stores

Gandalf62 — Sat, 27 Mar 2010 13:07:28 -0000

Was your client only considering open source databases, or was that more your choice? There are other very mature products out there that might be a better option... consider the Caché database (by InterSystems) for example.

Re: Some Notes on Distributed Key Stores

findchris — Mon, 08 Feb 2010 21:24:51 -0000

Any luck getting a master master config running?

Re: Some Notes on Distributed Key Stores

findchris — Mon, 08 Feb 2010 21:23:01 -0000

Are those redis numbers local or over a network?

Re: Some Notes on Distributed Key Stores

Bela Patkai — Thu, 24 Sep 2009 08:31:04 -0000

Thanks for the great write-up, we need more of these. I started using #cloud_nohype on twitter to tag posts that are realistic and have hard work behind them. In a new project I need a key-value store - or a two column table - with very high and variable demand both in terms of size and queries. Your post made me reconsider starting S3 testing, but also made me reconsider an earlier idea of a HSQLDB cluster on EC2. MySQL Cluster sounds good too, but it is not marketed well so I didn't know about it, even if I go to the mysql site sometimes. The sad fact is that sql dbs are not very exciting anymore - we were using them too long :)

Re: Some Notes on Distributed Key Stores

lhl — Mon, 03 Aug 2009 05:54:22 -0000

kvstores are particularly good for anything where you want to pull stuff quickly and randomly by id - canonical storage for documents perhaps, or pointers to media files. also, just about anything you would use something like memcache for, but that requires persistence.

Re: Some Notes on Distributed Key Stores

CaptTofu — Wed, 08 Jul 2009 10:30:42 -0000

Wow! Thank you for this write-up and all the various comments. I'm curious - what kind of application is this that the kv store is required for? What do you use for your various tests?

Re: Some Notes on Distributed Key Stores

Chris — Tue, 19 May 2009 00:28:33 -0000

Great write ip, chocked full of good insights.

I know the pain of having to build my own fundamental library as what existed is just not quite what I needed.

On the topic of key/value stores you missed one which is a little more obscure but I like it a lot. SkipDB from the author of IO.

Re: Some Notes on Distributed Key Stores

Przemyslaw — Fri, 08 May 2009 13:10:22 -0000

Let me be the next to announce another K/V store ;) http://www.subrecord.org
From what I have just learned seems I have been working on kind of Voldemort clone. I mean, working in isolation I've come to very similiar concept/architecture however still different. Nice :)

Re: Some Notes on Distributed Key Stores

schemafree — Fri, 08 May 2009 10:14:36 -0000

Hi everyone, here is yet another key/value store: Schemafree.

http://code.google.com/p/sc...

It uses Mysql as storage, has key based distribution, versioning, being able to make incremental changes to lists and other features. It has built-in integration with Memcached.

Re: Some Notes on Distributed Key Stores

lhl — Mon, 04 May 2009 01:10:05 -0000

Fair enough on making sure the data is accurate. Still, the hyperbole ("has no idea what he's talking about" or "spewing misinformation") does a disservice if free information flow is your goal. And from your tone and borderline ad hominem attacks, it sounds you have an axe to grind (unless your last comment is simply confusion- ie, I wasn't referring to his writing simplejson, but rather writing voldemort_client).

Personally, not knowing Bob at all, I found his presentation to be useful as a good overview for people that haven't been playing around with the various packages (and it jibed well enough with my own experiences) - I think he was pretty up front about where he was approaching it from (as someone who needed a solution that worked and his experiences - not as any domain expert, whatever that means). Most of the data is going to be out of date anyway since projects have been moving pretty quickly. And the plain fact of the matter is that his presentation has gotten attention precisely because there's so little published out there. In that respect, I think that it's a pretty big contribution to the community and I wish there was *more* of that out there, not less.

Anyway, if want to correct the errata in the presentation, why not just drop a line and it'd get fixed? If it just offends your sensibilities that "anyone" can go around, test some stuff and talk about it... well, that's err, usually how that works. At least he's put his real name to it (that's what I'm a firm believer in).

Anyway, since we've all said our pieces, I'd like to consider this conversation closed unless Bob jumps in. Life's too short.

cheers,

Re: Some Notes on Distributed Key Stores

coder — Sun, 03 May 2009 14:06:03 -0000

Hi; I am a firm believer in "knowing what you don't know". Getting up there and talking in front of people and spewing bunch of misinformation is not cool. I don't think Bob did the research, to his credit, I don't think any one man could do it given the wide range of products he was looking at; he was looking at bunch of stuff, and he confused things, and wrote some wrong stuff. I understand this. But I also want to set the record straight. Writing some JSON library does not make you an automatic authority on all things IT.

Re: Some Notes on Distributed Key Stores

Colin Howe — Wed, 29 Apr 2009 11:11:27 -0000

Great post!

Rough comparison of MySQL performance against Redis performance
http://colinhowe.wordpress....
Probably similar numbers for other KVS... but yet to find out.

Will be looking at Tokyo Cabinet later and adding in a similar test :)

Re: Some Notes on Distributed Key Stores

Wil Tan — Wed, 29 Apr 2009 06:13:07 -0000

Thanks for the nice practical roundup.

For the idle socket hanging issue, do you think it's an issue with pytyrant or tyrant itself?

I submitted a patch to pytyrant that could potentially be related to it, basically the client hangs when the socket is closed (which could happen on idle connections.)

http://code.google.com/p/py...

See if it fixes your issue?

=wil

Re: Some Notes on Distributed Key Stores

lhl — Wed, 29 Apr 2009 02:58:03 -0000

coder, that's a useful link, however saying that Bob has no idea what he's talking about is going a bit too far I think, seeing as he did explore Voldemort enough to write (the only) python binding for it... (your post btw also nears the line where I start with comment smackdowns - if you're gonna blast people, you need to man up and put your Real Name and Reputation on it; I find that helps to keep conversation constructive and civilized),

Both the partition-rebalance2 and protobuf branches look promising, so we'll just have to see.

Re: Some Notes on Distributed Key Stores

terrycojones — Tue, 28 Apr 2009 01:56:23 -0000

Thanks for the fast-paced, interesting, and amusing write-up. Sounds like a fairly intense weekend :-)

Terry

Re: Some Notes on Distributed Key Stores

Frazer — Thu, 23 Apr 2009 09:00:36 -0000

Hi,
Yes, MySQL Cluster is the name we give a system of MySQL servers connected to an Ndb Cluster.
I think there's some confusion with the definition of disk-persistence and durability.
MySQL Cluster has always had disk-persistence. All changes are redo-logged to disk and checkpoints to disk are used to allow the Redo log to be trimmed. Checkpointing has a few percent impact on achievable throughput - the disk write bandwidth used can be traded off against checkpoint duration and hence redo log size. The redo log is not fsynced at every transaction commit, but periodically - usually every 2s, and down to every 100millis. This tradeoff allows high throughput on Cluster's internal 2PC.
This window means that committed transactions are not immediately disk-durable, but when running with 2 or more replicas, all data is synchronously replicated at commit time, so committed transactions are machine-failure durable, and become disk durable (on all replicas) within ~2s. This is a three-way trade off between tolerance to total cluster failure (requiring disk durability), tolerance to machine failure (requiring machine-failure durability) and throughput (requiring control of fsyncs/s).

Prior to MySQL 5.1, all data was held (and had to fit) in memory.
From MySQL 5.1, non indexed data (i.e. the values in a kvp) can be stored on disk. This means that when they are read/written they are fetched from disk into an in-memory LRU cache in the same way as most databases. This allows data sets larger than the memory size to be handled by a single cluster node, at the cost of some performance. Persistence/Durability is the same, with Redo log flushed periodically etc.

Over time we will add support for disk-storage of indexed data (keys in a kvp), disk-durable transactions etc.

I take your point about complexity. Getting a system that has 'just enough' complexity to meet your needs is always hard. I think MySQL Cluster could suit some folks but it's not the simplest system out there.
Frazer

Re: Some Notes on Distributed Key Stores

xian — Thu, 23 Apr 2009 01:09:08 -0000

99% of it was over my head but reading you Leonard always makes me *feel* smarter.

Re: Some Notes on Distributed Key Stores

lhl — Wed, 22 Apr 2009 17:07:26 -0000

Frazer, I've played around w/ NDBCLUSTER a bit, which is what MySQL Cluster is running on, right? Does it have durability now? My understanding at the time I played w/ it was that it was neat but didn't have storage - for the disk-persistence you mention, how does check-pointing affect performance? It sounds interesting, although one of the appelas of running a "simple" system is not needing a dedicated DBA or data-wrangler...

Hopefully someone gives MySQL Cluster a spin, would love to see how it compares.

Re: Some Notes on Distributed Key Stores

lhl — Wed, 22 Apr 2009 14:17:26 -0000

Here's your chance EllisGL, do some tests and post some numbers.

Re: Some Notes on Distributed Key Stores

coder — Wed, 22 Apr 2009 14:12:56 -0000

On Voldemort store; Bob Ippilito has no idea what he is talking about. I asked VM folks about his claims on how VM leaves deleted objects around, and they flatly denied his claims. See

http://groups.google.com/gr...

Rebalancing, lI believe, is the next feature to be released.

Re: Some Notes on Distributed Key Stores

Parand — Wed, 22 Apr 2009 13:31:41 -0000

I haven't personally used HBase, but a friend is using it in production in a fairly large site and tells me query speed is definitely not fast enough to be user facing (they have a huge memcached farm in front of it).

Also, my understanding is that HBase mostly uses HDFS (the distributed file system) as opposed to Hadoop.

Re: Some Notes on Distributed Key Stores

scoop — Wed, 22 Apr 2009 12:21:40 -0000

Hey Leo, thanks for the great write up. There is a lot of hype around these k-v dbs. By the time you write a serious domain application around most of them, you begin to understand why "traditional" persistent stores are not as fast. If you want to use these as the primary persistent back end for a domain app, you'll soon realize that most of these "databases" push the messy details to the programmer.

"Partionable", and "distributed" are also tall claims for most of them. I looked at redis too and can't understand where the distributed part comes in.

"Based on the maturity of projects out there, you could write your own in less than a day. It’ll perform as well and at least when it breaks, you’ll be more fond of it. Alternatively, you could go on the conference circuit and talk about how awesome your half-baked distributed keystore is"

Completely agree. At the end of the day, its not rocket science to write your own memory hash-map and have a thread write backups to a disk file or just embed BDB and be done with it. And you can tune it to do exactly what you need for your own domain, including managing relationships if necessary.