random($foo): Some Notes on Distributed Key Stores
amix
· 8 months ago
Using consistent hashing and using Tokyo Tyrant as the backend _is_ basically what LightCloud does...! The only reason why you need two hash rings is if you want to dynamically add and remove nodes to the system.
Some issues with your system: - what happens if a node fails? Without using Tokyo Tyrant's master-master replication you are pretty doomed - what happens if you need to scale beyond 150M keys?
lhl
· 8 months ago
Hey Amir, I'm aware that what I built is similar (but simpler) to LightCloud - by the time that I went through testing multinode TT performance and figured out the setup I had built the thin layer that we needed. My understanding is that the addition of the lookup ring means that you end up with a lookup record for every single key?
As I mentioned given the tradeoff in write i/o, snapshotting was acceptable for the client, so not doom, but more like minor inconvenience w/ acceptable data loss.
Updated: For scaling, lookups for collisions are O(log n), which seems acceptable while nodes are added and data redistributed.
Emil Eifrem
· 8 months ago
Great writeup. Did you consider a graph database like Neo4j? If so, why did it fall short? In Neo4j, you can set arbitrary key-value properties on both nodes and the relationships between them. Neo4j ships as an embedded db but if you add a thin REST layer, you basically get a key-value store but with full-blown support for relationships. In my experience, how entities are related is often a very important part of a domain.
Nope, I'd actually looked at Neo4j in the past but didn't actually even think about it for this. Hopefully someone takes a look at it and posts some results.
anildash
· 8 months ago
70% of this was above my head but all of it was entertaining to read. :)
lhl
· 8 months ago
Well, at least some good has come out of this then. ;)
Parand
· 8 months ago
Any thoughts of open sourcing the thin layer of consistent hashing business? Will save me writing my own.
lhl
· 8 months ago
Sorry Parand, as this was work for a client (so not actually mine to open source). Like I mention though, the actual coding for the thin layer was literally hours. In this case, I used the a custom hash implementation, but you can check out Amir S's hash_ring implementation (although I was pretty meh about LightCloud, I'm actually a big fan of Amir's blog and what he's been up to).
I may do a clean-room rewrite and add in the dynamic expansion features and all that, but based on the ridiculousness of my near-term schedule... that probably wouldn't be anytime soon.
jmason
· 8 months ago
Thanks -- great writeup! I hadn't paid much attention to Tokyo Tyrant, but I'll be changing that.
Were you looking for any backup capability? can you snapshot the state of the Tokyo Cabinet store to take a backup of that? or are you just relying on doing that via EBS?
(also: S3 as a k-v store: slooooow)
lhl
· 8 months ago
Yeah, for backup the decision was that it'd be good enough to snapshot (TC atomic copy and EBS snapshot) - for the Master-Master replication, maybe someone who's done more w/ it can chime in w/ when the ulogs can be removed?
ie, for people that couldn't afford the data loss, I'm assuming that they can run M-M (w/ ulog'ing on another EBS volume if disk I/O is write-limited) and create a cleanup daemon that will check the rts's and delete expired ulogs? Based on my understanding, anything older than the rts timestamp on the corresponding master could be safely dispensed of? I didn't really test and I couldn't find that in the documentation so I punted. But if there are any Tokyo Cabinet experts reading this (or people that have tested) it'd be great to hear.
Also: agrreeeeed. :)
zzgavin
· 8 months ago
Like Anil, much of this was uncharted territory to me, but thanks for writing such a helpful overview. I know a lot more about non-rdbms than I did 20 minutes ago. cheers Gavin
jay
· 8 months ago
Hi Leonard,
WRT voldemort. You are correct that all of these systems are very alpha. However, you should not see an decrease in performance based on the number of nodes. We are using a number of voldemort clusters at LinkedIn and we have not seen this problem, it sounds like a bug. Could you send me a little more information about your test setup so we can try to reproduce?
Thanks!
Jason Dusek
· 8 months ago
I would really like to know how this discussion turns out.
lhl
· 8 months ago
FYI, I sent Jay some details on my setup and dataset. Hopefully that's enough to help replicate, otherwise may be a bit slow going since I'm juggling a few other balls atm.
Abraham
· 8 months ago
Its funny you say redis is way to alpha so you reinvented the wheel instead, like that you built is not alpha :)
lhl
· 8 months ago
I know that's slightly tongue and cheek, and there's some truth to that, but I think it's worth pointing out the difference between the solid data storage and retrieval part and the distributed part. In a comparison between redis and tc/tt for the former, I don't think there's any question (certainly not in my mind) which one is more battle-tested. So it's not like I went out and built my own keystore. For the distributed part, it was a matter of putting together the simplest thing that could work after it turned out that I there wasn't a black box solution to be had.
Bojan
· 8 months ago
Thank you for this post, I'm dealing with the exact same problem now, it seems. Although the amount of data (and the bandwidth I expect on write) is probably one step above compared to yours. interestingly enough, i came to the exact same conclusion - Tokyo Cabinet/Tyrant with custom routing to multiple nodes is the only available solution (that doesn't cost an arm and a leg) to be fast enough and rock-solid.
csb
· 8 months ago
Which persistence backend did you use for your Project Voldemort testing?
antirez
· 8 months ago
Hello! (disclaimer I'm the author of Redis).
Redis is surely a beta product, there are people using it in production but still we are entering in few days the feature-freeze stage now that Redis-git includes non-blocking replication. After we enter the feature freeze stage Redis will be stress-tested for weeks, then 1.0.0-rc1 will be released. My goal is to provide a rock solid product to the market.
So my hint is: handle with care since it's young code, but we are moving very fast, and feature-freeze stage is near. Also to make people safer Redis 1.0 will include a tool to dump a Redis DB into SQL format.
Redis apart, 1600 inserts / second are very poor performances. I think Tokyo cabinet is ways faster, probably it's the networking layer that is slow? Even MySQL is capable of 1600 inserts/second so if you really care about stability, replication, and things like this and you can live without very fast performances a table with an unique key ID and a blob value can really be a good alternative, especially in contexts where all you need is a plain key-value DB like Tokyo.
This is why Redis is stressing a lot on the data structures bit, that are things that are hard to model otherwise.
lhl
· 8 months ago
Hey antirez,
Redis looks cool so it'll definitely be something I'll keep an eye out in the coming months. The data structure approach is interesting...
You're correct that the insert/s numbers are much lower than the typically published numbers. Part of it is that it is going over the network, another part is that the items sizes are much bigger than those typically used in the benchmarks published. And it's EC2, so the I/O is crap. You're right that MySQL is the baseline there - I think lots of people don't know how fast it can be w/ simple queries -- although it tends to like lots of spindles. Lots and lots of them.
antirez
· 8 months ago
I can confirm that EC2 performed poorly compared even to low-end linux boxes when we run tests against EC2. In the smaller instances the redis benchmark returned 15000 queries/second, with the largest instance they provide it was 50000 q/s that is a number that's trivial to get with any kind of old Linux box.
Btw thank you for this article, I understand your findings can be complete or always accurate, but to find non biased data on this stuff is really hard. I hope many other guys in the field will try different key-value stores under real world load and publish their findings. This is the only way all this projects can mature faster, start to be more reliable, and understand what the real user feeling is.
scoop
· 8 months ago
Hey Leo, thanks for the great write up. There is a lot of hype around these k-v dbs. By the time you write a serious domain application around most of them, you begin to understand why "traditional" persistent stores are not as fast. If you want to use these as the primary persistent back end for a domain app, you'll soon realize that most of these "databases" push the messy details to the programmer.
"Partionable", and "distributed" are also tall claims for most of them. I looked at redis too and can't understand where the distributed part comes in.
"Based on the maturity of projects out there, you could write your own in less than a day. It’ll perform as well and at least when it breaks, you’ll be more fond of it. Alternatively, you could go on the conference circuit and talk about how awesome your half-baked distributed keystore is"
Completely agree. At the end of the day, its not rocket science to write your own memory hash-map and have a thread write backups to a disk file or just embed BDB and be done with it. And you can tune it to do exactly what you need for your own domain, including managing relationships if necessary.
Zak
· 8 months ago
What an awesome post! Thanks very much for writing it.
klaus
· 8 months ago
Is there a stable Python client for TokyoCabinet?
lhl
· 8 months ago
Yeah, I'm using Bob Ippolito's (he's everywhere :) pytyrant, a pure python implementation that's *very* active. There is also a wrapper for the C API: python-tokyotyrant.
thanks for writing this up. Can you elaborate on the CouchDB vaporware bit, though?
Cheers Jan --
lhl
· 8 months ago
Jan, if I recall correctly, when I tried out CouchDB last year, Lucene wasn't in releases or trunk (and the branch didn't build) and the replication was a joke. While there has been consistently big talk about CouchDB scaling, I could not find any actual distributed features for dealing with large datasets. CouchDB choked on a relatively modest data set when generating views - many minutes to generate one on a small 100K item/2GB data set. It also took 5+GB of storage for that.
I also couldn't really wrap my head about the benefit of not having indexes but having to recalculate a view anytime the data changed, but I'd say mostly that at the time (and based on the Q&A at Bob Ippolito's talk maybe still) that CouchDB fanboys and developers were all over the Internet taking up oxygen about Couch while like I mentioned, what I assumed were core components didn't exist workably, much less being suitable for anything but the most toy test projects.
evgen
· 8 months ago
CouchDB does take a long time to generate the view the first time you access a view, but each subsequent access uses the pre-generated version and returns quite quickly. When the dataset from which a view is created gets updated the view is also updated with the new data.
Next time, when you have no clue how a system works it would be best to refrain from talking trash about it and revealing the depth of your ignorance. CouchDB is not the greatest thing since sliced bread, is not a key-value datastore (which makes me wonder why it was even in your list other than to justify your petty little rant), and has a ways to go to meet some of its design goals, but within its niche it is a rather interesting tool that people should pay attention to.
lhl
· 8 months ago
Oh BURN! (not) I'm perfectly happy to reveal the depth of my ignorance if the ensuing discussion can help shed light on it (although my patience for other people's asshattery on *my blog* is finite). You're right that CouchDB isn't a kv store, but since every conversation about any of these subjects ineveitably brings up the "What about x?" where x invariably includes Couch, it'd be worth pre-empting. Personally, I think the critique I give in the posting (one line != rant) is pretty valid. The detailed response to Jan was because he asked. I'm sure he and the rest of the Couch team are good peoples, but my sentiments aren't unique - some have suggested that I should have made the CouchDB line it's own bullet-point.
As for trash talk, I have to say that you've been engaging in a fair amount of it. I'm posting my experiences (and I don't claim it to be anything more than that). It's not rocket science, but it's real data w/ real world usage in an area where there's significantly more smoke than fire (or published results). So, what's your skin in the game, and what's your contribution?
Jan Lehnardt
· 8 months ago
Hey, thanks for the feedback. CouchDB documentation is still coming along and there's a lot of things you can do wrong if you don't understand it as it is different in a lot of ways. So lucene is the only "vapor that's on the frontpage", by now there's a decent GitHub branch that we're looking to integrate. Replication is solid and has been for quite some time, I wonder what didn't work for you. View Indexes only recalculate what changed in a DB, it's incremental, they do not reindex all data when things change. Also, the view behaviour you saw is most likely wrong usage or outdate dependencies. We're happy to address any issues on the user@ mailing list, but I understand that getting under the skin of every project out there is not your priority.
Jan Lehnardt
· 8 months ago
In addition, the CouchDB devs never did any "big talk about scaling" auto-sharding et.al is a future feature. Just by the fact that it comes with a HTTP interface makes key-based partitioning a snap (see couchdb-lounge on Google Code for a 3rd party project that does just that). CouchDB has "alpha software" written all over the place :)
lhl
· 8 months ago
No worries, will continue to keep an eye out. I think part of the problem is the cool stuff CouchDB is tackling (non-relational, document-based, built w/ erlang, map-reduce processing) is catnip for devs and tends to make them forget about the "alpha software" bit unless it's big and blinking. The inevitable backlash/eye-rolling when it gets brought up everywhere isn't necessarily your fault, but something to be aware of.
Cheers!
Jonathan Ellis
· 8 months ago
Yes, Cassandra is starting to get its act together under the Apache umbrella now.
What problems did you run into? We recently fleshed out the docs at http://incubator.apache.org/cassandra/; we'd appreciate feedback as to what needs to be added.
lhl
· 8 months ago
Jonathan, the Getting Started Guide was useful, but I'd recommend a more comprehensive step-by-step? I know that's a PITA to document, but I suspect would really help in both getting people up and running and getting specific feedback on problems people are encountering (eg, in Step 12 on RHEL5 I had to do x).
If you'd like to see what I ran into, you could spin up an EC2 instance running a Rightscale Debian or Alestic Ubuntu instance and make sure that a user is able to get a blank system up and running. I was able to compile Cassandra, but the Thrift bits gave me some trouble. Once that was supposedly all up and running, I couldn't actually talk to the Cassandra server to test, so my assumption was that I'd missed something in setup.
I got distracted by some other tests at that point and ended up never pushing further having found a better solution. Also, I'm not sure if your intentions are for widespread production use at this point, but if so, I think that I'm like most sane people in getting totally skeeved out by running trunk checkouts in prod. Some packages/releases would probably also be really helpful in that regard.
Jonathan Ellis
· 8 months ago
Good feedback, thanks.
We're about to turn the corner from mostly-developer-focused to trying to get something that works out of the box for most people. Getting a release out is part of that. Thrift is a bitch and there's unfortunately not much we can do about that, but maybe providing a vmware image with sample single-node and 3-node configurations (for instance) would help there.
brendano
· 8 months ago
Why aren't there Debian packages for Thrift? Yes, packaging is a pain, but Thrift is intended to be a lowish-level service that many different apps use; therefore having .rpm's and .deb's makes a lot of sense. That would certainly make Cassandra installation easier. (I had similar issues.)
Jonathan Ellis
· 8 months ago
My understanding is that debian prefers to wait until a project has an official release and then package that rather than a svn snapshot. I work with a debian developer and he has a deb ready to roll as soon as Thrift gets out their 0.1 or whatever they are going to tag it. (They are actually making an effort towards that now, so hopefully soon.)
Of course then there's the whole RPM side of things, not to mention things like Gentoo or even (shudder) Windows. :)
lhl
· 8 months ago
Sample single-node and multi-node AMIs would be *huge*. I think that that, and some sample schemas would be great. If you created an empty table on the wiki for people to post up their testing results, I'd have to believe that it'd also fill up pretty quickly. I think there are a lot of people that are reviewing these things, but probably getting hung up getting started/wrapping their heads around deploying.
Hey, I just wanted to say, thanks for doing all this legwork and posting the results. It's very hard to find evaluations of these systems that weren't done by their authors.
lhl
· 8 months ago
Yeah, I'll be happy if this helps some people getting started, but even happier if this encourages more people out there to publish their findings/results, even if it's like mine where I could only get a few of them running (that in itself maybe a useful datapoint).
It'd be nice to get the s/n ratio up a bit for people that actually need to run something into production (I mean sure it's the Interweb, but the amount of fanboy/hater hot air has been pretty insufferable in this area).
I'd be interested in seeing how Bycast's StorageGRID product stacks up. It's a proprietary solution sold to hospitals through HP and IBM, as well as through direct sales. It isn't being marketed as a keystore, but that's what it is.
I used to work there, so I know the product intimately. But I haven't compared it to the open source stuff out there. I doubt any of the free solutions are the type of thing you'd run a hospital on (due to support, documentation, etc.)
ilya haykinson
· 8 months ago
I'm surprised that you didn't try HBase. It uses Hadoop as the backing store, and has both serious production use as well as high performance going for it. While its native API is Java, the thing does come with a Thrift-based interface that I found to be just as fast.
lhl
· 8 months ago
Ilya, do you have numbers on your HBase setup? How's the latency for queries? My understanding (and this applied to Hypertable too if I would've been able to get it up and running) that as BigTable clones, they're oriented about fast sequential requests, but not as good on the random. Would be interesting to get actual #s from your testing (ms latency, qps, on #/kind of nodes, w what kind of data set).
Parand
· 8 months ago
I haven't personally used HBase, but a friend is using it in production in a fairly large site and tells me query speed is definitely not fast enough to be user facing (they have a huge memcached farm in front of it).
Also, my understanding is that HBase mostly uses HDFS (the distributed file system) as opposed to Hadoop.
Frazer
· 8 months ago
Hi, Good writeup. Just to add to the list of 'what about X' posts : You mention MySQL, but not MySQL Cluster. (I am a MySQL Cluster developer btw.) MySQL Cluster is at heart a distributed key-value store using hash based partitioning. It supports in-memory or disk storage of key-value pairs as primary key and attributes. Additionally it supports : Multi kv pair transactional reads/updates Synchronous replication of updates within a cluster Disk-persistence of in-memory data via Redo and checkpointing. Automatic node failure and recovery handling Asynchronous replication of updates to other clusters/MySQL databases, including Master-Master with conflict detection/resolution Online addition of storage nodes and data repartitioning (from version 7.0) Secondary indices on data (unique, ordered). SQL access supporting MySQL SQL syntax Access from all MySQL supported connectors (JDBC, PHP, Perl, etc..) Latency+throughput optimised API for remote clients Online snapshot backup, optionally compressed e.t.c...
It is open source, licensed through GPL, with support available if required. I suspect that MySQL Cluster could meet or beat the latencies and throughputs of the other systems discussed here, especially when accessed via a native API rather than through MySQLD. Internally it uses a message-passing state machine architecture (similar to the CSP style of Erlang) which gives really nice properties w.r.t. latency, throughput and system efficiency.
Perhaps because MySQL Cluster is associated with MySQL it appears to be 'relational' and therefore does not get included in open-source kv store comparisons? Hope this doesn't sound too much like an advert :), Frazer
lhl
· 8 months ago
Frazer, I've played around w/ NDBCLUSTER a bit, which is what MySQL Cluster is running on, right? Does it have durability now? My understanding at the time I played w/ it was that it was neat but didn't have storage - for the disk-persistence you mention, how does check-pointing affect performance? It sounds interesting, although one of the appelas of running a "simple" system is not needing a dedicated DBA or data-wrangler...
Hopefully someone gives MySQL Cluster a spin, would love to see how it compares.
Frazer
· 8 months ago
Hi, Yes, MySQL Cluster is the name we give a system of MySQL servers connected to an Ndb Cluster. I think there's some confusion with the definition of disk-persistence and durability. MySQL Cluster has always had disk-persistence. All changes are redo-logged to disk and checkpoints to disk are used to allow the Redo log to be trimmed. Checkpointing has a few percent impact on achievable throughput - the disk write bandwidth used can be traded off against checkpoint duration and hence redo log size. The redo log is not fsynced at every transaction commit, but periodically - usually every 2s, and down to every 100millis. This tradeoff allows high throughput on Cluster's internal 2PC. This window means that committed transactions are not immediately disk-durable, but when running with 2 or more replicas, all data is synchronously replicated at commit time, so committed transactions are machine-failure durable, and become disk durable (on all replicas) within ~2s. This is a three-way trade off between tolerance to total cluster failure (requiring disk durability), tolerance to machine failure (requiring machine-failure durability) and throughput (requiring control of fsyncs/s).
Prior to MySQL 5.1, all data was held (and had to fit) in memory. From MySQL 5.1, non indexed data (i.e. the values in a kvp) can be stored on disk. This means that when they are read/written they are fetched from disk into an in-memory LRU cache in the same way as most databases. This allows data sets larger than the memory size to be handled by a single cluster node, at the cost of some performance. Persistence/Durability is the same, with Redo log flushed periodically etc.
Over time we will add support for disk-storage of indexed data (keys in a kvp), disk-durable transactions etc.
I take your point about complexity. Getting a system that has 'just enough' complexity to meet your needs is always hard. I think MySQL Cluster could suit some folks but it's not the simplest system out there. Frazer
Here's your chance EllisGL, do some tests and post some numbers.
coder
· 8 months ago
On Voldemort store; Bob Ippilito has no idea what he is talking about. I asked VM folks about his claims on how VM leaves deleted objects around, and they flatly denied his claims. See
Rebalancing, lI believe, is the next feature to be released.
lhl
· 7 months ago
coder, that's a useful link, however saying that Bob has no idea what he's talking about is going a bit too far I think, seeing as he did explore Voldemort enough to write (the only) python binding for it... (your post btw also nears the line where I start with comment smackdowns - if you're gonna blast people, you need to man up and put your Real Name and Reputation on it; I find that helps to keep conversation constructive and civilized),
Both the partition-rebalance2 and protobuf branches look promising, so we'll just have to see.
coder
· 7 months ago
Hi; I am a firm believer in "knowing what you don't know". Getting up there and talking in front of people and spewing bunch of misinformation is not cool. I don't think Bob did the research, to his credit, I don't think any one man could do it given the wide range of products he was looking at; he was looking at bunch of stuff, and he confused things, and wrote some wrong stuff. I understand this. But I also want to set the record straight. Writing some JSON library does not make you an automatic authority on all things IT.
lhl
· 7 months ago
Fair enough on making sure the data is accurate. Still, the hyperbole ("has no idea what he's talking about" or "spewing misinformation") does a disservice if free information flow is your goal. And from your tone and borderline ad hominem attacks, it sounds you have an axe to grind (unless your last comment is simply confusion- ie, I wasn't referring to his writing simplejson, but rather writing voldemort_client).
Personally, not knowing Bob at all, I found his presentation to be useful as a good overview for people that haven't been playing around with the various packages (and it jibed well enough with my own experiences) - I think he was pretty up front about where he was approaching it from (as someone who needed a solution that worked and his experiences - not as any domain expert, whatever that means). Most of the data is going to be out of date anyway since projects have been moving pretty quickly. And the plain fact of the matter is that his presentation has gotten attention precisely because there's so little published out there. In that respect, I think that it's a pretty big contribution to the community and I wish there was *more* of that out there, not less.
Anyway, if want to correct the errata in the presentation, why not just drop a line and it'd get fixed? If it just offends your sensibilities that "anyone" can go around, test some stuff and talk about it... well, that's err, usually how that works. At least he's put his real name to it (that's what I'm a firm believer in).
Anyway, since we've all said our pieces, I'd like to consider this conversation closed unless Bob jumps in. Life's too short.
cheers,
xian
· 8 months ago
99% of it was over my head but reading you Leonard always makes me *feel* smarter.
terrycojones
· 7 months ago
Thanks for the fast-paced, interesting, and amusing write-up. Sounds like a fairly intense weekend :-)
Terry
dready
· 7 months ago
Thanks for the nice practical roundup.
For the idle socket hanging issue, do you think it's an issue with pytyrant or tyrant itself?
I submitted a patch to pytyrant that could potentially be related to it, basically the client hangs when the socket is closed (which could happen on idle connections.)
It uses Mysql as storage, has key based distribution, versioning, being able to make incremental changes to lists and other features. It has built-in integration with Memcached.
Przemyslaw
· 7 months ago
Let me be the next to announce another K/V store ;) http://www.subrecord.org From what I have just learned seems I have been working on kind of Voldemort clone. I mean, working in isolation I've come to very similiar concept/architecture however still different. Nice :)
Chris
· 7 months ago
Great write ip, chocked full of good insights.
I know the pain of having to build my own fundamental library as what existed is just not quite what I needed.
On the topic of key/value stores you missed one which is a little more obscure but I like it a lot. SkipDB from the author of IO.
CaptTofu
· 5 months ago
Wow! Thank you for this write-up and all the various comments. I'm curious - what kind of application is this that the kv store is required for? What do you use for your various tests?
lhl
· 4 months ago
kvstores are particularly good for anything where you want to pull stuff quickly and randomly by id - canonical storage for documents perhaps, or pointers to media files. also, just about anything you would use something like memcache for, but that requires persistence.
Bela Patkai
· 3 months ago
Thanks for the great write-up, we need more of these. I started using #cloud_nohype on twitter to tag posts that are realistic and have hard work behind them. In a new project I need a key-value store - or a two column table - with very high and variable demand both in terms of size and queries. Your post made me reconsider starting S3 testing, but also made me reconsider an earlier idea of a HSQLDB cluster on EC2. MySQL Cluster sounds good too, but it is not marketed well so I didn't know about it, even if I go to the mysql site sometimes. The sad fact is that sql dbs are not very exciting anymore - we were using them too long :)
All Comments
Some issues with your system:
- what happens if a node fails? Without using Tokyo Tyrant's master-master replication you are pretty doomed
- what happens if you need to scale beyond 150M keys?
As I mentioned given the tradeoff in write i/o, snapshotting was acceptable for the client, so not doom, but more like minor inconvenience w/ acceptable data loss.
Updated: For scaling, lookups for collisions are O(log n), which seems acceptable while nodes are added and data redistributed.
http://neo4j.org
http://github.com/andreasronge/neo4j/tree/master (Ruby bindings with REST support)
(Disclaimer: I'm involved.)
-EE
I may do a clean-room rewrite and add in the dynamic expansion features and all that, but based on the ridiculousness of my near-term schedule... that probably wouldn't be anytime soon.
Were you looking for any backup capability? can you snapshot the state of the Tokyo Cabinet store to take a backup of that? or are you just relying on doing that via EBS?
(also: S3 as a k-v store: slooooow)
ie, for people that couldn't afford the data loss, I'm assuming that they can run M-M (w/ ulog'ing on another EBS volume if disk I/O is write-limited) and create a cleanup daemon that will check the rts's and delete expired ulogs? Based on my understanding, anything older than the rts timestamp on the corresponding master could be safely dispensed of? I didn't really test and I couldn't find that in the documentation so I punted. But if there are any Tokyo Cabinet experts reading this (or people that have tested) it'd be great to hear.
Also: agrreeeeed. :)
I know a lot more about non-rdbms than I did 20 minutes ago.
cheers Gavin
WRT voldemort. You are correct that all of these systems are very alpha. However, you should not see an decrease in performance based on the number of nodes. We are using a number of voldemort clusters at LinkedIn and we have not seen this problem, it sounds like a bug. Could you send me a little more information about your test setup so we can try to reproduce?
Thanks!
interestingly enough, i came to the exact same conclusion - Tokyo Cabinet/Tyrant with custom routing to multiple nodes is the only available solution (that doesn't cost an arm and a leg) to be fast enough and rock-solid.
Redis is surely a beta product, there are people using it in production but still we are entering in few days the feature-freeze stage now that Redis-git includes non-blocking replication. After we enter the feature freeze stage Redis will be stress-tested for weeks, then 1.0.0-rc1 will be released. My goal is to provide a rock solid product to the market.
So my hint is: handle with care since it's young code, but we are moving very fast, and feature-freeze stage is near. Also to make people safer Redis 1.0 will include a tool to dump a Redis DB into SQL format.
Redis apart, 1600 inserts / second are very poor performances. I think Tokyo cabinet is ways faster, probably it's the networking layer that is slow? Even MySQL is capable of 1600 inserts/second so if you really care about stability, replication, and things like this and you can live without very fast performances a table with an unique key ID and a blob value can really be a good alternative, especially in contexts where all you need is a plain key-value DB like Tokyo.
This is why Redis is stressing a lot on the data structures bit, that are things that are hard to model otherwise.
Redis looks cool so it'll definitely be something I'll keep an eye out in the coming months. The data structure approach is interesting...
You're correct that the insert/s numbers are much lower than the typically published numbers. Part of it is that it is going over the network, another part is that the items sizes are much bigger than those typically used in the benchmarks published. And it's EC2, so the I/O is crap. You're right that MySQL is the baseline there - I think lots of people don't know how fast it can be w/ simple queries -- although it tends to like lots of spindles. Lots and lots of them.
Btw thank you for this article, I understand your findings can be complete or always accurate, but to find non biased data on this stuff is really hard. I hope many other guys in the field will try different key-value stores under real world load and publish their findings. This is the only way all this projects can mature faster, start to be more reliable, and understand what the real user feeling is.
"Partionable", and "distributed" are also tall claims for most of them. I looked at redis too and can't understand where the distributed part comes in.
"Based on the maturity of projects out there, you could write your own in less than a day. It’ll perform as well and at least when it breaks, you’ll be more fond of it. Alternatively, you could go on the conference circuit and talk about how awesome your half-baked distributed keystore is"
Completely agree. At the end of the day, its not rocket science to write your own memory hash-map and have a thread write backups to a disk file or just embed BDB and be done with it. And you can tune it to do exactly what you need for your own domain, including managing relationships if necessary.
thanks for writing this up. Can you elaborate on the CouchDB vaporware bit, though?
Cheers
Jan
--
I also couldn't really wrap my head about the benefit of not having indexes but having to recalculate a view anytime the data changed, but I'd say mostly that at the time (and based on the Q&A at Bob Ippolito's talk maybe still) that CouchDB fanboys and developers were all over the Internet taking up oxygen about Couch while like I mentioned, what I assumed were core components didn't exist workably, much less being suitable for anything but the most toy test projects.
Next time, when you have no clue how a system works it would be best to refrain from talking trash about it and revealing the depth of your ignorance. CouchDB is not the greatest thing since sliced bread, is not a key-value datastore (which makes me wonder why it was even in your list other than to justify your petty little rant), and has a ways to go to meet some of its design goals, but within its niche it is a rather interesting tool that people should pay attention to.
As for trash talk, I have to say that you've been engaging in a fair amount of it. I'm posting my experiences (and I don't claim it to be anything more than that). It's not rocket science, but it's real data w/ real world usage in an area where there's significantly more smoke than fire (or published results). So, what's your skin in the game, and what's your contribution?
Cheers!
What problems did you run into? We recently fleshed out the docs at http://incubator.apache.org/cassandra/; we'd appreciate feedback as to what needs to be added.
If you'd like to see what I ran into, you could spin up an EC2 instance running a Rightscale Debian or Alestic Ubuntu instance and make sure that a user is able to get a blank system up and running. I was able to compile Cassandra, but the Thrift bits gave me some trouble. Once that was supposedly all up and running, I couldn't actually talk to the Cassandra server to test, so my assumption was that I'd missed something in setup.
I got distracted by some other tests at that point and ended up never pushing further having found a better solution. Also, I'm not sure if your intentions are for widespread production use at this point, but if so, I think that I'm like most sane people in getting totally skeeved out by running trunk checkouts in prod. Some packages/releases would probably also be really helpful in that regard.
We're about to turn the corner from mostly-developer-focused to trying to get something that works out of the box for most people. Getting a release out is part of that. Thrift is a bitch and there's unfortunately not much we can do about that, but maybe providing a vmware image with sample single-node and 3-node configurations (for instance) would help there.
Of course then there's the whole RPM side of things, not to mention things like Gentoo or even (shudder) Windows. :)
It'd be nice to get the s/n ratio up a bit for people that actually need to run something into production (I mean sure it's the Interweb, but the amount of fanboy/hater hot air has been pretty insufferable in this area).
http://www.bycast.com/
I used to work there, so I know the product intimately. But I haven't compared it to the open source stuff out there. I doubt any of the free solutions are the type of thing you'd run a hospital on (due to support, documentation, etc.)
Also, my understanding is that HBase mostly uses HDFS (the distributed file system) as opposed to Hadoop.
Good writeup.
Just to add to the list of 'what about X' posts : You mention MySQL, but not MySQL Cluster.
(I am a MySQL Cluster developer btw.)
MySQL Cluster is at heart a distributed key-value store using hash based partitioning.
It supports in-memory or disk storage of key-value pairs as primary key and attributes.
Additionally it supports :
Multi kv pair transactional reads/updates
Synchronous replication of updates within a cluster
Disk-persistence of in-memory data via Redo and checkpointing.
Automatic node failure and recovery handling
Asynchronous replication of updates to other clusters/MySQL databases, including Master-Master with conflict detection/resolution
Online addition of storage nodes and data repartitioning (from version 7.0)
Secondary indices on data (unique, ordered).
SQL access supporting MySQL SQL syntax
Access from all MySQL supported connectors (JDBC, PHP, Perl, etc..)
Latency+throughput optimised API for remote clients
Online snapshot backup, optionally compressed
e.t.c...
It is open source, licensed through GPL, with support available if required.
I suspect that MySQL Cluster could meet or beat the latencies and throughputs of the other systems discussed here, especially when accessed via a native API rather than through MySQLD. Internally it uses a message-passing state machine architecture (similar to the CSP style of Erlang) which gives really nice properties w.r.t. latency, throughput and system efficiency.
Perhaps because MySQL Cluster is associated with MySQL it appears to be 'relational' and therefore does not get included in open-source kv store comparisons?
Hope this doesn't sound too much like an advert :),
Frazer
Hopefully someone gives MySQL Cluster a spin, would love to see how it compares.
Yes, MySQL Cluster is the name we give a system of MySQL servers connected to an Ndb Cluster.
I think there's some confusion with the definition of disk-persistence and durability.
MySQL Cluster has always had disk-persistence. All changes are redo-logged to disk and checkpoints to disk are used to allow the Redo log to be trimmed. Checkpointing has a few percent impact on achievable throughput - the disk write bandwidth used can be traded off against checkpoint duration and hence redo log size. The redo log is not fsynced at every transaction commit, but periodically - usually every 2s, and down to every 100millis. This tradeoff allows high throughput on Cluster's internal 2PC.
This window means that committed transactions are not immediately disk-durable, but when running with 2 or more replicas, all data is synchronously replicated at commit time, so committed transactions are machine-failure durable, and become disk durable (on all replicas) within ~2s. This is a three-way trade off between tolerance to total cluster failure (requiring disk durability), tolerance to machine failure (requiring machine-failure durability) and throughput (requiring control of fsyncs/s).
Prior to MySQL 5.1, all data was held (and had to fit) in memory.
From MySQL 5.1, non indexed data (i.e. the values in a kvp) can be stored on disk. This means that when they are read/written they are fetched from disk into an in-memory LRU cache in the same way as most databases. This allows data sets larger than the memory size to be handled by a single cluster node, at the cost of some performance. Persistence/Durability is the same, with Redo log flushed periodically etc.
Over time we will add support for disk-storage of indexed data (keys in a kvp), disk-durable transactions etc.
I take your point about complexity. Getting a system that has 'just enough' complexity to meet your needs is always hard. I think MySQL Cluster could suit some folks but it's not the simplest system out there.
Frazer
http://www.persvr.org/
http://groups.google.com/group/project-voldemor...
Rebalancing, lI believe, is the next feature to be released.
Both the partition-rebalance2 and protobuf branches look promising, so we'll just have to see.
Personally, not knowing Bob at all, I found his presentation to be useful as a good overview for people that haven't been playing around with the various packages (and it jibed well enough with my own experiences) - I think he was pretty up front about where he was approaching it from (as someone who needed a solution that worked and his experiences - not as any domain expert, whatever that means). Most of the data is going to be out of date anyway since projects have been moving pretty quickly. And the plain fact of the matter is that his presentation has gotten attention precisely because there's so little published out there. In that respect, I think that it's a pretty big contribution to the community and I wish there was *more* of that out there, not less.
Anyway, if want to correct the errata in the presentation, why not just drop a line and it'd get fixed? If it just offends your sensibilities that "anyone" can go around, test some stuff and talk about it... well, that's err, usually how that works. At least he's put his real name to it (that's what I'm a firm believer in).
Anyway, since we've all said our pieces, I'd like to consider this conversation closed unless Bob jumps in. Life's too short.
cheers,
Terry
For the idle socket hanging issue, do you think it's an issue with pytyrant or tyrant itself?
I submitted a patch to pytyrant that could potentially be related to it, basically the client hangs when the socket is closed (which could happen on idle connections.)
http://code.google.com/p/pytyrant/issues/detail...
See if it fixes your issue?
=wil
Rough comparison of MySQL performance against Redis performance
http://colinhowe.wordpress.com/2009/04/27/redis...
Probably similar numbers for other KVS... but yet to find out.
Will be looking at Tokyo Cabinet later and adding in a similar test :)
http://code.google.com/p/schemafree/
It uses Mysql as storage, has key based distribution, versioning, being able to make incremental changes to lists and other features. It has built-in integration with Memcached.
From what I have just learned seems I have been working on kind of Voldemort clone. I mean, working in isolation I've come to very similiar concept/architecture however still different. Nice :)
I know the pain of having to build my own fundamental library as what existed is just not quite what I needed.
On the topic of key/value stores you missed one which is a little more obscure but I like it a lot. SkipDB from the author of IO.