I'm aware, thanks 🙂

Here's a Twitter thread relevant to this topic: https://twitter.com/codexeditor/status/1302585628178051075 Many of the features listed there should be implementable via multi-layer text or a simple extension of the idea.

The concept here of standoff annotations is similar to what I've been improving in Zep, with its 'RangeMarkers' as I call them. Originally these were for annotating source code with compile errors:

The range marker is locked to the text so that edits above/below and undo/redo will keep it consistently in the same place. This one has an associated mark in the number column, and a tooltip. But I have several different kinds of range marker, including simple underlines, bounding boxes around text, etc.

My markers can also introduce widgets into the text, such as this color swatch; in which case the text flows around them and they are effectively invisible to edit operations on the text.

Like the standoff annotations, markers also overlap; but it is a work item to make that visibly clearer!

There are 2 paths forward from this point:

• Have shader-specific parsers automatically inject markers at the right point (like compile errors, or color pickers on color variables)
• Add additional annotation information to the marker other than the tooltip.

The second part is where things get interesting, and where the layer system would help a lot. For example, if the compile error was on a layer, I could choose to show the error layer, effectively embedding all compile errors inside the text.

My first priority is to get easy edit widgets like the color picker to work well, but I'm spending some brain power at the moment trying to figure out how to get a layer system that works well into the editor....

update: https://twitter.com/crabl/status/1303193386359291904?s=21 starting to explore the TextKit stack a little more since it offers a pretty clean path to implementation. that said, if anyone has experience with implementing a custom NSTextStorage, i'd love to pick your brain for an hour!

Chris Rabl Looks great. Not an expert on Text Kit, and you might already be well aware of it, but sometimes these older but still relevant docs are hard to find: https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/TextStorageLayer/TextStorageLayer.html#//apple_ref/doc/uid/10000087i

Chris Rabl (But if you have specific questions, please send them my way — probably via private message — and I’ll see if I can help.)

Chris Maughan indeed, we find that there is a ton of possibilities when the editor is able to host dynamic adornments. For example:

https://twitter.com/feenkcom/status/1303213233441067009

https://twitter.com/feenkcom/status/1303221842824626176

Tudor Girba My #two-minute-week video has some of that in it. I haven't played with Glamarous toolkit yet, but it is on my list 🙂

I was implementing this and got stuck part way with copy paste. What should happen with cut/copy/paste? Say you hide a few layers, then cut a visible chunk that surrounds some hidden text and paste it elsewhere. Does the hidden text:

• Vanish
• Travel with the cut/pasted text
• Move to the end of the original cut locationNote that you may cut but never paste the text. Or you may copy/paste.

I think that you should always favour preservation; so my intention here is to have it move to the end. But I'm not certain, for sure. Travelling might make sense too; but perhaps a that becomes a 'deep copy' operation with associated key stroke, etc.

Yeah I'm thinking if something is hidden (like a hidden photoshop layer) you shouldn't be able to affect it - neither copy nor cut. So it would just drop to the end of the original cut location. Side note: traveling with the cut is also harder to implement for me because the clipboard needs to be modified to hold the hidden text. Finally if you cut, but never paste, you don't want to lose what you didn't see when cutting.

OTOH, if you 'show' a layer and cut/paste, you want the layers to be preserved, not flattened, so we probably need to modify the clipboard to hold layered text anyway..

Agreed; I was thinking about having a secondary buffer containing the layer for each character and storing/retrieving that with the text itself. I have quite a narrow undo/redo layer; essentially there are only 2 commands that can modify text - Insert/Delete. In these commands I already store the location of adornments/markers so that they can be replaced after a paste. So that isn't hard; but it is hard to know what to do in each circumstance.

Shalabh Chaturvedi you make a good point about the principles involved in implementing this (that if a particular layer is hidden, actions which operate on other layers shouldn't be able to affect it): my intuition was almost the complete opposite in the sense that clipboard operations would need to operate on all the layers at once, regardless of their visibility (which was part of the reason I opted to explore the "always show invisibles" route rather than attempting to implement some sort of position-preserving operations or other magic). I suppose this is a consequence of the serial nature of text, its an inherently linear medium...

Chris Rabl Yes I can see if I have one hidden sentence in the 'draft' layer, it clearly belongs in the surrounding context and should travel with it. In fact most layer use cases might be like this. I've implemented layers by 'collapsing' instead of hiding - the layer collapses into a single character. Maybe this sets the expectation wrt moving and deletion?

Interesting Shalabh Chaturvedi - I had been intending to do the same thing; collapse to an 'indicator' for the hidden layer points, but it didn't occur to me that this helps the problem of working on hidden layers because you can see what you are copying, etc. I wonder if that is more of an argument for destroying layer content if it is visibly present and collapsed. Perhaps the 'transport' or 'delete' option could be a user option. And perhaps deleting hidden layer content could move that content onto a special 'deleted' layer....

To counter what I was saying earlier (hidden layers should not be affected), one observation is that there is no absolute coordinate system in layered text. In 2d graphics, each layer is independently grounded to a stable absolute coordinate system. All movement is relative to the absolute coordinate system. (Is this true? I haven't used 2d drawing that much)

In text all we have are relative coordinates. So when moving text chunks, we're not just moving the content, we're also possibly moving the coordinate system - the ground itself is moving. In other words, the principle 'hidden layers should not be affected' seems to be on shaky ground.

Chris Maughan the 'deleted' layer idea is brilliant.

I have a feeling that part of the difficulty is in porting a layer-oblivious concept of "selection" to a layered context. Selections have to know what layers they contain and maybe which they skip, and make it clear to the user. If you're writing a long text, different use cases will call for including the outline layer or just the "final text" layer in a copy-paste.

Idea 1: only show placeholders for layers when they are in or next to a selection/the cursor. Idea 2: part of the UI display of a selection is a layer-selector similar to the one for the whole document, so you can choose to exclude layers within the range defined by the selection (at which point it starts acting like a multi-selection.)

Text is a linear sequence of characters and selections are just ranges in that linear sequence. Introducing different layers means at least introducing some form of hierarchy that breaks all the simple algorithms that work well with ranges in sequences.

Cutting a range out of a linear sequence only makes sense if the sequence is the ultimate truth — but then you can’t model different layers with it. Adding layers means you fundamentally change the core data model from something as simple as a sequence to something more complex like a tree or worse.

But that’s a great direction to think in. We need to figure that out for structural editing too.

I don't see any potential for recursive hierarchy in the mockup that kicked off this thread. I think there have been a couple conflicting models discussed here, but that one is pretty flat AFAICT. I think it's basically a sequence of (layer_tag, text).

As long as all the layers need to be contiguous but non-overlapping, i.e. perfectly cover the range [0,n), then the linear sequence with layer metadata is an equally valid ultimate truth (in a certain sense anyway?) to a sequence of layer segments. I don't necessarily want to get bogged down in the representation of a selection, but I think an index range is not crazy. Algorithms on pointers into this structure are going to be tricky regardless.

@Andrew F I was thinking about cut & paste with overlapping ranges (which I interpreted as something we talked about above), considering the selection to be one of the ranges. My comment was meant conceptually, not talking about implementation. There are many different ways to implement it, and it doesn't have to end up in an actual tree data structure, but conceptually it seems to me that there is a line that's crossed when you need to deal with overlapping ranges where you cross over into a different complexity class.

If I'd be better at math, I'd probably be able to explain that better with algebraic structures and the operations defined on them. I think it has to do with the total order requirement on the character sequence, which makes ranges work, but then as soon as you start handling ranges as their own entities with their own operations defined on them, you are introducing another… hmm… layer that has different requirements. And that's what I mean by hierarchy — layers are above the sequence conceptually and allow certain operations, doesn't matter how you implement them.

Ah, well, that probably doesn't help, sorry. I wish I could explain it better. Right now it just feels to me that there are some use cases, like cut & paste of arbitrary ranges which don't correspond exactly to layer range boundaries, that can't be represented unambiguously with a pure linear sequence + a set of ranges into that sequence. It could probably be worked around though with some heuristics that'll translate the edge cases into weird/unintuitive behavior from the user's perspective, and perhaps that's totally acceptable.

But I'd love to be proven wrong and maybe there's a better way! This would solve one of my current problems too…

The simplest way I'm thinking about this conceptually is that each character has attached metadata - the layer name. It travels with the character where ever it goes. So editing as usual just moves the characters around. Contiguous is more of an incidental feature - many consecutive characters have the same layer. As an optimization, you can only store entire spans as Andrew suggested above. The only new features, which are outside of editing the text, are 'attach layer to character (or span.. as optimization)' and 'show/hide layer'.

I was just reading this clever guy's idea about that -

"What if plain text had included special invisible characters that could indicate nesting? After all, aren’t all control characters arbitrary anyway?"

https://shalabh.com/programmable-systems/plain-text.html

(Based on the comments I got) That essay is often taken literally, but it was meant more of an appetizer to start looking at how the design of a medium affects all things built upon it. Layered text would also be a legit proposition in those terms. BTW layering seems richer/more flexible than nesting for some reason - can't put my finger on it yet.

Quick update, I actually got in the Horizon beta and tried it out. The final UI is pretty close to what we discussed before I left and some of my early ideas. Specifically 2.5D with nested expressions, drag and drop from categories

so a lot of it is a pretty straight forward "scratch in VR" type system or a basic visual lisp editor, but it works pretty well. Duplicate and move work using the same UI as the world building tools

So mainly what got removed and paired back was the actor model inspired message passing system, which is a shame because it would have made object sharing easier, and certain types of decoupled state changes easier

Message passing still exists, but the only thing that remains is a single send to an explicit object reference. They also kept connect (probably the only concept that I didn't introduce) which is basically a delegate system.

I had other ways of doing this built into my initial implementation. Specifically tag messages, which allowed the user to add multiple takes to an object, and send to a tag. Another thing that they maybe misunderstood? was there is already support for multiple handlers for the same event

names didn't have to be unique, that might have been a result of poor understanding of the system or something else. I had to fight a lot of programmer assumptions about... well programming languages. Like the idea that method names have to be unique in most OO languages. That constraint doesn't exist in the Horizon language but oh well 🙂

Not sure what NDA allows but I might share some screen shots and videos later

Thanks for sharing this, it seems very interesting! Of course, I'd love to see screenshots and videos if this is possible! 🙂

Cool project! What were your reasons for forking Scratch as opposed to e.g. using ScratchX or Blockly?

I have created ScratchX extensions in the past, and in fact it is that experience that made me think that writing Scratch extension is an "easy" way to explore new end-user programming use cases. But ScratchX is based on the "old" flash implementation while Scratch 3 is a pure HTML/JavaScript/CSS which I know better.

And why not Blockly? Well I thought of using Blockly, particularly because you can design any block programming system with it, which opens more possible experiments than with Scratch. But for now I wanted to have a complete environment set up quickly, Scratch is usable now for project useful to users. With Blockly, you have to implement lots of things.

This example is extremely thought-provoking: https://create.adacraft.org/?view=63e29a12

In what way?

This looks interesting from a conceptual point of view, but I can't bring myself to look at any example in detail for the same reason that I have never seriously played with Scratch: visual clutter and bad contrast makes the code very hard to read. White text in a small font on a yellow background in fancy shapes is not my idea of an ergonomic interface.

Thanks for sharing your thoughts on that, Konrad. I totally agree!

I think one of the UX "regressions" from Scratch 2 with the new Scratch 3 design, is less readable block texts. And I find that the new shapes, even if more eye candy, add too much visual clutter. The same with the colors.

I am very tempted to try to change all this with adacraft. I.e dark text on light block background, remove unnecessary part of shapes (like the small "slots") or make them less present. Though I don't know how difficult it will be to change all that...

Nicolas Decoster I meant the juxtaposition of what I think of as 'advanced' ML software in pretty much the most beginner-friendly language.

I'm also not sure how I feel about ML being such a black box. It feels like all the interesting stuff is hidden away. This risks falling into the worst of both worlds.

In fact, ML is a a black box for most people, even for those AI experts! 😅

And, on the contrary, I find that bringing people to create their own image classification models with Teachable Machine and making them use it from adacraft (or any Scratch with AI extension, there are others) helps them figure out what ML means, and particularly in terms of bias. I have done it several times since one year during workshops and it is very demonstrative.

Ah, you're getting out of the building! That certainly trumps any knee-jerk opinions I have.

I meant the juxtaposition of what I think of as 'advanced' ML software in pretty much the most beginner-friendly language.I agree this is really an exciting feature.

And I can't wait to create blocks for all the https://ml5js.org/ library. With this, there will be really interesting new creative possibilities. Adding blocks based the PoseNet, FaceAPI and other interactive stuff will open to new tangible experiments.