It's almost scary to see how productive, focused and well versed Salvatore Sanfilippo is.
Most of the developers on my team struggle to build simple CRUD web apps, wasting long hours complaining about how wrong the Domain models is or how why this things is not in that place.
The simplistic low-level nature of redis probably helps avoid fatigue. Many of the data structures implemented are well defined in their purpose, it's more a matter of implementation & optimization. The same is not true for higher-level projects, such as CRUD apps, where project structure and project goals can often be vague and are likely to change during the course of the project.
This isn't to take away from Salvatore, who is obviously talented and seems to have a knack for researching and implementing the data structures in an efficient manner.
I didn't say it was easy. redis is full of reinventing the wheel in the name of improved performance. This is something developers often do, but are often told it's pointless, just use built-ins.
Salvatore has found a niche where he can see these optimized implementations to their end. I would imagine there is a certain level of satisfaction that comes from thoroughly researching a topic, implementing it as best you can, and deploying it to the public you're popular with. It probably helps having firm control of the project with less red tape.
It's like a comparison between a craft woodworker and someone who would like to play around with the concepts, but doesn't have the tools, time or know how, or they're told there is no point in building your own furniture from scratch, so they are instead building Ikea furniture.
Maybe it's a case of "developer interest dissonance". Technical people digging technical problems (cache hit issues) versus business logic... which might be why they "got into computers ".
I get this as well. In my experience these are people who just don't like what they're doing (and are also often bad at it), so they try to undermine every decision. I haven't found a good way of dealing with this so I'd love to hear some pointers.
Scalable bloom filters! [1] Drop size configuration and just have a maximum false positive rate. FWIW I built one by wrapping Guava's, you can take an existing bloom filter implementation and wrap auto-scaling around them.
Native rotating bloom filters, fantastic! Sort of thing I've had to do myself more than once - great efficient way to be able to ask "Have I seen this recently?".
Has Salvatore ever considered forking redis into a legal entity and providing official commercial support? I'm willing to bet alot of startups and enterprises don't use redis because of the lack of this.
Hello, I considered it, in some way this could be good, allowing me to have a team of developers to build a better Redis in less time. On the other side there is always the risk of ruining the project a bit, when you take external money and need to be profitable and so forth. But probably the biggest reason is that I don't feel like that currently is a good idea to travel more, work long hours, and so forth. I try to work my 40 hours and use the remaining time for family and other activities. So far the sponsorship model has worked well, even if there are rooms for improvements for sure.
According to the Redis support page, Pivotal sponsors Redis and they provide commercial support services for it.
Although, to be honest, the page linked from the Redis site (http://www.pivotal.io/big-data/redis) doesn't seem to have much to say on the matter. Looks like an ad for their cloud platform, and I wouldn't have guessed they offer support for Redis itself.
Ah yes, our brochureware site is too vague for my tastes, but Pivotal-the-company is mostly pitched at F500s, not engineers per se.
At the coalface, Pivotal also includes Pivotal Labs (for whom I work and from which the larger company took its name), which includes people who've done a lot of production Redis work.
And we can always ask Salvatore for a tip if we get stuck.
Similarly, I have access to Hadoop, Gemfire, Greenplum, Cloud Foundry, RabbitMQ or Spring devs for help. Being inside the same tent is nifty.
> "brochureware site" <- don't be too hard on yourselves, there's plenty of goodness there too :)
Besides Pivotal's commercial support for Redis (which, from what I've heard, is superb) there are also at least two hardcore Redis community members who're providing professional support for Redis deployments - for more details, contact Michel Martens and Damian Janowski.
Redis Labs, my employer, is also heavily into Redis (duh) so naturally we give support for our product line that consists of a Redis on-prem solution and a cloud service.
Our designers, no lie, are awesome. But we are definitely writing copy aimed at the C-suite right now, on the developer front it's a bit scantier but improving.
I see you guys have a service offering on PWS. Nifty.
Salvatore is just one guy, and he probably doesn't want to waste his time managing a customer support organization.
Meanwhile, according to the official website [1], somebody else is already doing it:
> Pivotal, the official sponsor of Redis, offers Redis support for developers and 24x7 production Redis deployments.
And of course nothing prevents any other company from competing in the same space. Perhaps one of the startups that use Redis a lot could "pivot" into a Redis commercial support company.
Here's Redis Labs' (my employer) comparison table of cloud services for your reference: https://redislabs.com/redis-comparison. It's only slightly outdated (missing the MongoHQ-cum-Compose.io offering) and we'll probably add it in the near future.
Redis uses asynchronous replication, in order to have a latency you expect for most use cases where Redis is used, however after the tests Aphyr performed we tried to improve what was possible to improve without transforming Redis into something completely different. Basically even with asynchronous replication and a drastic merge function (last failover wins), you can still do things to improve the real world behavior of the system, and this is what we did:
1) Improve Redis master-slave protocol in order to have asynchronous acknowledges, at least. So it is now possible, during partitions, to have a bound data loss: after some time the isolated master stops accepting writes. This is implemented both in Redis Cluster (natively) and in Redis vanilla replication (so can be used with Sentinel or other failover mechanisms).
2) Use a replication offset in order to try, during failovers, to pick the slave with the most updated state.
Currently it is just an idea but in the future for certain data types and operations, Redis Cluster may take a log of the operations that were not acknowledged by all the slaves of a given master, and when the master is turned into a slave, it would re-play the log into the new master, at least for operations which are idempotent and commutative. This would allow for safer writes. The other (more used) approach of merging values directly is not very easy to perform with Redis for a number of reasons.
That post was a bit before the new redis clustering in 3.0 (http://antirez.com/news/79 -- Redis cluster, no longer vaporware). Wonder if anything has changed.
Aside from the fact that Aphyr showed basic replication to be flawed, as far as I know Cluster isn't even designed to be consistent across partitions.
The announcement page you linked to doesn't let me copy any text on iOS, but he's basically saying that if you get a partition and resolve it, the merge algorithm used is prone to data loss.
Most of the developers on my team struggle to build simple CRUD web apps, wasting long hours complaining about how wrong the Domain models is or how why this things is not in that place.