Mariano Guerra

event-fabric.com

• Namespaced Key-Value Store

• Replicated to all nodes

  • Eventually

  • Asynchronously

• In memory (ETS)

• On disk (DETS)

• Local Read/Writes

Make riak_core run on windows

• Per user Rate Limiting

• Per Bucket/Key Eviction

• Grep?

• Follow calls?

• Visualizations helps

  • Yak shaving: erlplorer

• Problem

• Vocabulary

• Alternatives

• Previous Work

• Related Theory

• Tradeoffs

• High Level Design

• "Benchmarks"

99 papers on the pile,
take one,
read it,
follow the references,

123 papers on the pile...

Alvaro Videla's Syndrome

Achievement Unlocked

rebar3 new rebar3_riak_core tanodb
# add recon dependency to rebar.config
make devrel

# in different consoles
make dev1-console
make dev2-console
make dev3-console

make devrel-join
make devrel-status
make devrel-cluster-plan
make devrel-cluster-commit
make devrel-status

Using recon on the 3 nodes:

• Trace all riak_core_metadata modules

• Ignore lazy_tick messages

• Log call params and returns

ReturnTrace = fun(_) -> return_trace() end.
% at most 1000 calls per second
Rate = {1000, 1000}.
recon_trace:calls([{riak_core_broadcast, '_',
    fun ([A, _]) when A /= lazy_tick -> return_trace() end},
       {riak_core_metadata_hashtree, '_', ReturnTrace},
       {riak_core_metadata_object, '_', ReturnTrace},
       {riak_core_metadata_manager, '_', ReturnTrace},
       {riak_core_metadata_exchange_fsm, '_', ReturnTrace},
       {riak_core_metadata, '_', ReturnTrace}], Rate).

Write something in riak_core_metadata:

FullPrefix = {<<"tanodb">>, <<"mymeta">>}.
MDKey = my_key_1.
MDValue = <<"my metadata value">>.
riak_core_metadata:put(FullPrefix, MDKey, MDValue).

When the dust settles:

recon_trace:clear().

• Store logs from 3 nodes

• Annotate trace calls and returns

• Follow code from the traces

• Copy code simplify and inline it

• Draw sequence diagrams

• Draw state diagrams (for FSMs)

• Understanding code is hard

• Switch strategies when stuck

• Approaches reinforce each other

• Be organized with your notes

• Try your own

• We need more tooling

• Implementation of paper Epidemic Broadcast Trees

• Known as Plumtree

• Eager Set

• Lazy Set

• Seen Msgs

How does node 3 gets the values broadcasted while he was down?

Tree in which every non-leaf node is labelled with the hash of the labels or values (in case of leaves) of its child nodes

• Git

• IPFS

• ZFS, Btrfs

• BitTorrent

• Bitcoin, Ethereum

• Cassandra, Riak

• Insert

• Update

• Compare

hashtree:compare(Tree, RemoteFun).
hashtree:compare(Tree, RemoteFun, AccFun).

Change in riak_core_metadata_manager from:

riak_core_metadata_hashtree:insert(PKey, Hash),
ok = dets_insert(dets_tabname(FullPrefix), Objs);

To:

PersistenceType = proplists:get_value(persistence_type, Opts, disk),
case PersistenceType of
    disk ->
        riak_core_metadata_hashtree:insert(PKey, Hash),
        ok = dets_insert(dets_tabname(FullPrefix), Objs);
    memory ->
        ok
end,

Implemented my own riak_core_broadcast_handler

Add it to advanced.config in the riak_core section:

{riak_core, [
  {broadcast_mods, [riak_core_metadata_manager,
                    tanodb_broadcast_handler]}]}

Try it:

tanodb_broadcast_handler:start_link().
tanodb_broadcast_handler:put({{<<"tanodb">>, <<"memmeta">>},
                             mem_key_1}, <<"my value">>).

It works!

Dotted Version Vectors:

• Time, clocks and the ordering of events in a distributed system

• Virtual Time and Global States of Distributed Systems

• Dotted Version Vectors: Efficient Causality Tracking for Distributed Key-Value Stores

Gossip/Broadcast:

• HyParView: a membership protocol for reliable gossip-based broadcast

• Epidemic Broadcast Trees

• riak_core

• plumtree

• partisan

@warianoguerra

github.com/marianoguerra