April 4, 2011

I’m proud to announce the Ripple family of gems (riak-client, ripple, riak-sessions) version 0.9.0 were released yesterday. This is a huge leap forward from the 0.8 series, last release of which was in December. I’m going to highlight some of the best new features in this blog post.

HTTP+SSL Support

Adam Hunter did an awesome job implementing support for HTTPS, including navigating the various idiosyncracies of HTTP client libraries. If you’ve got HTTPS turned on in Riak, or a reverse-proxy in front that provides SSL, it’s easy to set up.

“`ruby
# Turn on SSL
client = Riak::Client.new(:ssl => true)

# Alternatively, provide the protocol client = Riak::Client.new(:protocol => “https”)

# Want to be a good SSL citizen? Use a client certificate.
# This can be used to authenticate clients automatically on the server-side.
client.ssl = {:pem_file => “/path/to/pemfile”}

# Use the CA chain for server verification
client.ssl = { :ca_path => “/path/to/ca_cert/dir” }

# All three of the above options will invoke “peer” verification.
# Use “none” verification only if you’re lazy. This is the default
# if you don’t specify a client certificate or CA.
client.ssl = { :verify_mode => “none” }
“`

Adam also added HTTP Basic authentication for those who use it on their reverse-proxy servers. It can be set with the :basic_auth option/accessor as a string of “user:password”.

Protocol Buffers

Riak has had a Protocol Buffers-based client API for a long time, but the state of Protocol Buffers support in Ruby has been very bad until recently. Thanks to Blake Mizerany’s “Beefcake” library, it was really simple to add support in a cross-platform way. While not insanely faster, the decreased overhead for many operations can make a big difference in the long run. Check out these benchmarks (run on MRI 1.9.2, comparing against the Excon backend):

                               user     system      total        real
http  ping                 0.020000   0.010000   0.030000 (  0.084994)
pbc   ping                 0.000000   0.000000   0.000000 (  0.007313)
http  buckets              0.010000   0.000000   0.010000 (  0.894827)
pbc   buckets              0.000000   0.000000   0.000000 (  0.864926)
http  get_bucket           0.480000   0.020000   0.500000 (  1.075365)
pbc   get_bucket           0.170000   0.030000   0.200000 (  0.271493)
http  set_bucket           0.060000   0.000000   0.060000 (  0.660926)
pbc   set_bucket           0.030000   0.000000   0.030000 (  0.579500)
http  store_new            0.710000   0.040000   0.750000 (  2.443635)
pbc   store_new            0.630000   0.030000   0.660000 (  1.382278)
http  store_key            0.730000   0.040000   0.770000 (  2.779741)
pbc   store_key            0.580000   0.020000   0.600000 (  1.539332)
http  fetch_key            0.690000   0.030000   0.720000 (  2.014679)
pbc   fetch_key            0.410000   0.030000   0.440000 (  0.948865)
http  keys                 0.300000   0.090000   0.390000 ( 78.455719)
pbc   keys                 0.530000   0.020000   0.550000 (  0.828484)
http  key_stream           0.200000   0.010000   0.210000 (  0.689116)
pbc   key_stream           0.530000   0.010000   0.540000 (  0.833347)

Adding Protocol Buffers required a breaking change in the Riak::Client class, namely that the port setting/accessor was split into http_port and pb_port. If you still have this setting in a configuration file or your code, you will receive a deprecation warning.

“`ruby
# Use Protocol Buffers! (default port is 8087)
client = Riak::Client.new(:protocol => “pbc”, :pb_port => 8087)

# Use HTTP and Protocol Buffers in parallel, too! (Luwak and Search require HTTP)
client.store_file(“bigpic.jpg”, “image/jpeg”, File.open(“images/bigpic.jpg”, ‘rb’))
“`

**Warning**: Because some operations (namely get_bucket_props and set_bucket_props) are not semantically equivalent on both interfaces, you might run into some unexpected problems. I have been assured that these differences will be fixed soon.

MapReduce Improvements

Streaming MapReduce is now supported on both protocols. This lets you handle results as they are produced by MapReduce rather than waiting for all the results to be accumulated. Unlike the traditional mode, you will be passed a phase number in addition to the data from the phase. Like key-streaming, just give a block to the run method.

“`ruby
# Make a MapReduce job like usual.
Riak::MapReduce.new(client).
add(“people”,”sean”).
link(:tag => “friend”).
map(“Riak.mapValuesJson”, :keep => true).
run do |phase, data| # Streaming!
puts data.inspect
end
“`

MapReduce key-filters were available in beta releases of 0.9. This is a new feature in Riak 0.14 that lets you reduce the number of keys fed to your MapReduce query by some criteria of the key names. Here’s an example:

“`ruby
# Blockish builder syntax for key-filters
Riak::MapReduce.new(client).
filter(“posts”) do
tokenize “-“, 1 # Split the key on dashes, take the first token
string_to_int # Convert the token to an integer
eq 2011 # Only pass the ones from 2011
end

# Same as above, without builder syntax
Riak::MapReduce.new(client).
add(“posts”, [[“tokenize”, “-“, 1],
[“string_to_int”],
[“eq”, 2011]])
“`

Ripple::Document Improvements

Ripple::Document models got a lot of small improvements, including:

• Callback ordering was fixed.
• Documents can be serialized to JSON e.g. for API responses.
• Client errors bubble up when saving a Document.
• Several association proxy bugs were fixed.
• The datetime serialization format defaults to ISO8601 but is also configurable.
• Mass-attribute-assignment protection was added, including protecting :key by default.
• Embedded documents can be compared for equality, which amounts to attribute equality when under the same parent document.
• Documents can now have observer classes which can also be generated by the ripple:observer generator.

Testing Improvements

In order to make sure that the client layer is sufficiently independent of transport semantics and that the lower layers comply with the “unified” backend API, there is a new suite of integration tests for riak-client that covers operations that are supported by both transport mechanisms. This should make it much easier to implement new client backends in the future.

The Riak::TestServer was made faster and more reliable by a few changes to the Erlang bits that power it.

Onward to 1.0

Recently the committers and some members of the community joined me in discussing some key features that need to be in Ripple before it reaches “1.0” status. Some of them will be really incredible, and I’m anxious to get started on them:

• Enhanced Document querying (scopes, indexing, lazy loading, etc)
• User-defined sibling-resolution policies, with automatic retries
• Enhanced Riak Search features
• Platform-specific Protocol Buffers drivers (MRI C, JRuby, Rubinius C++)
• Server templates for creating development clusters (extracted from Riak::TestServer)

To that end, I’ve created a 0.9-stable branch which will only receive bugfixes going forward. All new development for 1.0 will be done on master. We’re likely to break some legacy APIs, so we will try to add deprecation notices to the 0.9 series where possible.

Enjoy this latest release of Ripple!

— Sean and contributors