Just finished listening…great!

Some thoughts on the whole “drawing the interface” vs. “building a machine arranging widgets using a grammar”:

• Widgets vs. drawn UI

We used to actually be much more in the “draw the interface” space, basically Views that draw what they have to display. Widgets were added later on for standard interactive elements such as text boxes and sliders etc.

And there was a clear distinction, in that for most applications, the dynamic content (document) would be drawn dynamically using a custom view, whereas ancillary/auxiliary information (inspectors etc.) would be fixed layouts of widgets, which might be dynamically enabled or disabled.

But the whole structure of the UI was largely fixed (dynamic content inside the views + widgets in static layouts).

Although there was a bit of a trend towards widgets, that trend really took off with the Web and the iPhone.

With the iPhone, the content both got more dynamic, partly due to latency hiding with animations, partly due to the small screen making it necessary to hide unused UI, rather than juts disable it, and at the same time our tooling got more static, with UIViews that have dynamic content discouraged and preference given to static layers that are moved in and out and around.

With the DOM, you really don’t have many options except to change the structure if you want dynamic content (Canvas notwithstanding).

So we’ve been moving more and more towards a situation where even purely informational applications display their information via these static/rigid widget sets, at the same time that they information got more dynamic.

That looks very promising! Types are statements about code, so it makes sense to move them to an annotation layer and have tool support for that layer.

Next: make that layer optional (perhaps it already is). Then: allow multiple type annotation layers for the same codebase. Allow programmers to add their own domain-specific type systems.

Great stuff!

My favorite part, and I wonder if this would be the selling point for structure editors for pro-coders (I don't think it will ever be "avoiding syntax errors" etc.):

the compiler is involved in the entire process of development, responding to programmer feedback, and giving context-aware feedback in return. In this model, the source code is enriched with the compiler's inferences along the way, which the programmer can inspect & correct (but can also be elided from display for a more compact presentation).

About this:

When typing an identifier in Jerd, if you select something from autocomplete, the ID of that term is stuck onto the identifier node, and then the compilation pass doesn't need to do any name resolution -- it knows exactly what is being referenced. And, if you later change the name of a variable, none of the references are broken.

How do you plan on handling the case where a new higher precedence function overload is added after a id+arguments code has been 'bound' to an overload that should now be preceded by the new overload for the given id+arguments? So you could end up in a situation in which exactly the same function call expressions bind to different overload depending on which overloads existed at the time of writing (binding) of the call. I guess you could just run the overload resolution again for all the calls, but what if the result for a call depends on a (now hidden) user defined type annotation. You could still end visibly exactly the same function calls binding to different overloads. How would possible implicit casts/conversions fit in?

Glad to see you working on interactive approaches to this problem. 🙂 If you're looking into interesting prior art in the space, I strongly recommend this paper: plover.com/~mjd/misc/hbaker-archive/TInference.html

Konrad Hinsen 🤯 domain-specific type systems? Do you have any more thoughts on what that would look like? As far as whether the layer is optional, it very much isn't as knowledge of types informs code generation 🤷 .

@Jarno Montonen Thanks! So, in this system, we no longer have a notion of "precedence" in function overload resolution. If there is any ambiguity (multiple functions w/ the same signature), the answer is to always ask the user. You could still end visibly exactly the same function calls binding to different overloads. So that's already the case with a + b and c + d , if a & b are ints, but c & d are strings, right? Now I actually solve this by coloring all identifiers according to the hash of the definition, so the two function calls would very likely have different colors, allowing you to immediately recognize them as distinct.

Jack Rusher thanks for the link! fascinating read.

Okay, so say there exists a 'myfunc (double x, double y) -> double' and an implicit conversion from int to double. the user writes the call 'myfunc(1, 2)' which binds to the function. now user adds 'myfunc (int x, int y) -> int' and writes another call 'myfunc(1, 2)'. Do you ask the user which overload to use or bind to the int version? Do you ask the user to re-evaluate which overload to bind the first call to? Either way the user could end up in a situation that exactly the same calls (down to argument types) bind to different overloads, possibly without the user ever choosing that the first call should bind to the less-specific double overload. Of course if the only difference in the signature is numeric types, it's likely that the functions do the same thing, but you could end up with a more problematic case with user defined types.

@Jared Forsyth My favorite example of a domain-specific type system is dimensional analysis in science and engineering (en.wikipedia.org/wiki/Dimensional_analysis). It illustrates the general characteristics of domain-specific typing: (1) types apply to some but not all data/variables in a program and (2) each piece of data and each variable can have types from another type system as well.

Regarding:

Prior Art

…

I don’t know if they [Lamdu] store the inferred types in the syntax tree, or if they re-compute them on every edit.

Lamdu does a combination:

• For every definition (global variable/function) it stores the types of every definition that it used, so that if that definition changes its old type is still used for inference until a manual (and informative!) “update type” action
• Inside the definition it does re-infer it on every edit, and an edit that would had resulted in an error in a normal language gets inserted as wrapped in a “fragment” (aka “non-empty hole” which was our old term and was also adopted by Hazel)

And btw, Lamdu also has the option to mark the new usage with fragment as the correct one and push the error to other parts of the code.

It’s all explained/demonstrated in more detail here: youtu.be/viF1bVTOO6k

oh fantastic, thanks Yair Chuchem for the more info! exactly what I was hoping for 🙂

@Jarno Montonen If there's the possibility of ambiguity, always ask the user! Also, adding a new function never changes previously defined code.

So in my example, user would have to go through all the calls to a previous overload and select the correct one? Could be quite a bit of work in a big codebase.

If you actually want to change the behavior of all of your code, it will take some work, yes 😄 there'll be automated assistance however, that will allow you to "re-base" existing functions onto a new function (or indeed a changed definition of a previous function), while running any relevant tests. hash-based identification is definitely a different paradigm. By default, the behavior of a function is deeply immutable, but there are tools to migrate changes broadly if that is what you want.

You can think of it as pull-based vs push-based. The current general mode is pull-based (everything always auto-updates dependencies), but jerd (which takes inspiration from unison) is push-based (dependency updates are explicit). We're used to that pattern in external dependencies, and here it's applied to internal ones as well.

(this is all experimental of course, it may well turn out that operating this way is prohibitively annoying 😅 )

I don't understand. What does this tool do?

Live collaboration over a graph, like Google Docs allows over a text document?

It's a node-based wrapper around javascript. In this example, you can see the html_element changing in the bottom left. There are also threejs and other library wrappers, but ultimately it can do whatever is possible in the environment it's running (in this case a browser).

Yup! Live collaboration on a graph, but that graph represents code which is running in both browsers.

Using yjs, a crdt algorithm under the hood.

brilliant talk/demo! it seems like those rules work really well on the GPU (although the cell splitting would take some gymnastics) -- is that what you're doing / have you tried that?

Thank you very much!

Yes I reckon there's some scope to run them on the GPU, but it might be harder than expected.

So, just for context - the engine's main bottleneck currently is rendering, because it's just using javascript canvas. The renderer could be sped up loads by using webgl instead, and then the bottleneck would become the computation behind it.

At that point, GPU would be a consideration! Cellular automatons are usually great for GPU optimisations, because all the classic (like game of life) involve rules where only the 'middle' of an 'event window' gets edited. Which is perfect for shaders.

One of the challenges of an engine like CellPond (which stems from Dave Ackley's Moveable Feast Machine engine) is this: When a rule gets applied, it can affect any cell in the event window. This doesn't work so well with shaders. However, with the rise of GPU compute, eg: webgpu, it might become easier to leverage the GPU for this kind of thing! I need to look into it more!

Another challenge is this: Most cellular automatons are deterministic. But CellPond (and the Moveable Feast Machine) are non-deterministic. So parallelising 'events' involves having to make sure that any randomly-picked locations don't overlap with each other. You can't just get away with a classic block cellular automata approach.

I did manage to do a dodgy solution to that once on the GPU with this experiment though. And I made two videos about it:

1. The initial idea of using GPU to speed up one of these engines.

2. Taking the idea to its logical conclusion.

Big answer but it's an interesting topic, thanks for asking :)

Oh fascinating, thanks for the details response! So these nondeterministic event windows; do they pick a single rule to evaluate at random? Or do they evaluate all available rules? And is it critical that the window be a fixed size, or could it adapt to the size of the largest rule?

(I see movable feast has some papers written,I can just go read those as well)

In the London creative coding video, say at one point that cell splitting was a big leap forward, because it allowed you to encode direction, which was difficult before. How is it different from encoding direction as a color (blue goes up, red goes down) or pairs of colors in a non-split grid? (For that matter, I bet split grids make event windows quite complicated 😅)

So, in CellPond, I was hoping to do this:

• Pick a random position (within the 0.0 -> 1.0 range of x and y)
• Apply a random rule (if it matches).

However, for some performance reasons, it actually does this:

• Pick a random position.
• Shuffle all rules.
• Apply the first rule that matches.

The 'variable event window size' was a key new thing for the engine to do! The Moveable Feast Machine (MFM) has a fixed diamond-shaped window. SandPond has a square shaped one.

CellPond needs to be flexible. The way it gets around this is...

When you make a rule with the drag-and-drop interface, it actually gets compiled down into multiple rules -- one for each cell in the diagram silhouette.

The classic sand rule...

:large_yellow_square:➡⬛

⬛➡:large_yellow_square:

... would produce two rules on the virtual machine.

One where the top cell (yellow) is the 'center/origin' of the event window. And one where the bottom cell (black) is the origin.

Regarding the encoded direction... You certainly could encode direction as colour! And I did it a lot in this video!

However, this isn't a very useful abstraction when it comes to writing rules. You'd need a rule for moving up, a rule for moving down, and so on.

By encoding direction as a spatial relationship, we can just make one rule: Move 'forward'.

Other examples of this include:

Using a split cell where one cell indicates an element type, and the other cell indicates its temperature. You could make water that heats up and does something (eg: freeze/evaporate) when it hits a certain temperaure.

Cold water

:large_blue_square:

⬜

Hot water

:large_blue_square:

⬛

Tepid water

:large_blue_square:

(grey)

It's nice that it lets you 'quote' extra data in a way! We can use the black colour to mean temperature when attached with a blue cell, but it might mean different things in other places.

You could use a similar thing to make the 'demon horde sort' that Dave Ackley uses to exemplify the MFM too by the way!

hmm yeah that's neat! I wonder how it would be to have one (or several!) "hidden" or "background" layers for metadata, and rules that could take them into account, but the fore-most layer is the one that gets rendered 🤔

Yes that would be great to try out! I was originally planning for every cell to be controlled by its own mini 9x9 cellular automata world, and the middle cell was the only one that got rendered when zoomed out. But when I started trying it out, I got tempted by the dynamic splitting/merging idea 🙂

Very interesting. I had no idea how large the influence of Dave Ackley and his MFM was here.