Implement Databricks-Style Key Execution

[splinterhq]

Data Bricks has this really cool way of letting you begin a key with a special prefix like %lua:, %wasm: or even %python3, %m4 (you get it). So the key:

%lua:normalizeFieldVectors

Would treat whatever's in the corresponding value arena offset as runnable Lua code (it could validate first, if desired).

We can implement WASM just like Lua (I'm working on that now) and for non-embeddable runtimes, we can pass the value region contents as a file or argument and return it. To be clear this would happen on the client side only, core storage remains passive.

We can also, once I get the completion inference shard pushed, allow for %sql or other transformations using a local code model (or even third-party remote API model, or pipe through Claude code somehow).

The same sidecar methodology applies, and the results can be tied to the signal arena.

So this would make Splinter an AI "BusyBox" because it has:

• Cache
• Vector Storage
• Inference
• Executable keys
• WASM for SIMD loads
• Tandem slots for multi-modal vectors

That's cool!

[splinterhq]

I'm scoping this to only keys there are runtimes in the CLI to back (no external execs) because I don't want to implement process life cycle management in the client without a major compelling reason to have it (I always tend to go with dynamic loading wherever possible, so it would have to be impossible via that, and urgent to implement).

It will go out with 1.2.0 👍

[splinterhq]

The way that this is going to come together:

Adding splinter_run()

This will avoid conditional compilation checks in splinter_get(). We also might want to get the contents of a key (the code) without actually invoking it. splinter_run() will behave much like splinter_get(), except:

• It fails if there's no %lua or %wasm prefix
• It invokes the intepreter as defined in the CLI to run the code
• it returns the result from the runtime as well as any stdout/stderr writes

Adding 'run' to the CLI

run <key> [with_options=options to pass to the interpreter] [options=options controlling this run]

Where run_options are (for now) --unstable (enable unstable run features)

And with_options are options that can be passed to the language interpreter, currently supported should be "test" with "true" being the only possible value. This will allow us to see what we have to actually use to coax Lua and WASMEdge into the limits we want (max allocation, mostly).

De-coupling the Lua / WASM so they're reusable

We may end up making an embedded language provider module (splinter_cli_languages.c) that has the WASM and LUA code in it, which the respective commands will link against anyway. If visibility is an issue we may go this route to avoid spaghetti.

There is no "return" value other than failure or success from the interpreter

We should just pass stdout / stderr from the execution right to the console. If someone runs this, it's because they have a data payload and they want to use the built in LUA or WASM bindings to read and write to keys in the store with as it runs over that data. SIMD workloads are expected.

At most, we just have log chatter from whatever encoded process the code has.

This will need a CMake flag

If we don't spell out to not run untrusted code in keys, someone will do it, and then encourage other people to do it. This is a great convenience but it comes at a risk. --enable-executable-keys to configure should enable HAVE_EXEC_KEYS to gate the feature.