guys, there's a spectrum ... I'm confused by the black and white answers I sometimes get from the "Future of Coding" forum. It seems the future is black and white...

at one side, there's "just use maps". At the other, there's Transit. What's in the middle?

I'm just curious how other people solved this issue, seems like a problem a lot of people may have tackled. For instance, if I "just use maps", then if I want to encode that map verbatim, I need to escape somehow. I'm curious how other dealt with this problem. Maybe the answer is to not worry about that case, reversibility, etc.

Didn't intend to sound black and white! Just trying to answer the question to my knowledge. I'm not an expert here, so it's hard to be dogmatic 😄

Encoding maps: just put them in quotes, use backslashes for escaping? I'm not familiar with Transit, unfortunately.

From the Transit page:

Users of data formats without such facilities must rely on either schemas, convention, or context to convey elements not included in the base set, making application code much more complex. With Transit, schemas, convention, and context-sensitive logic are not required.

I suppose conventions and schemas is what we're both reaching for. I just haven't felt this particular pain point yet..

You'll always run into escaping issues in principle, the trick is finding the way to avoid them in practice. Depending on your context, the "~#" prefix might be enough, you might not even ever need to escape them. Not if "~#" doesn't occur naturally.

Using a hash with a "type" key or perhaps a "type" key for extra safety is perhaps a more versatile (and verbose) solution.

The only downer of using extra ~s to meta-escape is that it can get silly sometime. I vaguely recall a system wherein to write backslash I had to use the string "\\\\\\". So having another way to say backslash would have worked out better.

See also Quining https://en.wikipedia.org/wiki/Quine_%28computing%29.

here’s an off the wall idea: you could have a companion “types” array whose keys are access instructions and whose values are enums representing the type. eg “person.birthday”: “date”. you could make it ad-hoc with relative positioning with a jq-like syntax, eg user: { createdAt: tel:17261669, “.createdAt”: “date” }. idk if that’s too wacky

Hungarian notation for the key names

This may still be too heavy weight, but why not implement just a subset of transit? Iirc it should be possible

A subset of transit could work, I was just curious about other options.

Actual transit ends up looking like a lot of nested arrays, something like:

["~#array", [1, 2, [~#date 1929292929], [~#json, [1, 2, 1929292929]]]

I like Garth's idea. Send complementary JSON with the same structure, with the type names in the place of the values (you can omit unnecessary parts)

What if I wanted to encode the "~#date:1929292929" string verbatim? Now I have to come up with some escaping mechanism, say, ~~

Or you could use the same mechanism: "~#string:'~#date:...'"

https://github.com/edn-format/edn is a simpler textual representation (Clojure's version of sexps) that includes reader extensibility ("tagged elements"). It's easy to emit and there are a fairly large number of reader implementations out there, including https://github.com/edporras/edn_turbo embedded into Ruby that we built at my last company. Would that work for you?

hey Jack, I think EDN is a good inspiration too, but is mostly a stand-alone textual format, I was thinking more of something more directly built on top of JSON

What would the layer above look like/seek to accomplish? That info might lead to a better suggestion 🙂

if you end up going with my idea, please let me know how it works out for you! it’s basically half-assing infra’s method: http://www.christopherkhall.com/research.html

The details are a bit hazy ... very roughly, I've a PoC that takes https://gist.github.com/EmmanuelOga/0d626107e5ed2bc6f5e7bf3925d032fd and turn them into HTML. It works but it has quite some layers of complexity (RDF+XML/XSLT).

It uses Saxon/JS. Saxon does what I'm asking: it expresses the full XSLT type system on top of JSON, it is just more than a bit hairy, as you can imagine... At the same time Saxon/JS lacks basic things like supporting async extension functions.

It occurred to me I could create a format to express both markup and graph data with an eye on building on top of JSON from the start, as opposed to the Frankenstein of tech I currently use.

Something like https://github.com/thi-ng/umbrella/tree/develop/packages/egf comes close but it is missing the markup part (mixed content).

For now I'm just looking at "prior art", specially if I can find very minimal formats that build on top of JSON without adding too much complexity to ... steal 🙂 .. some of the patterns people use when building on top of JSON.

https://concise-encoding.org/ is able to express both markup and graphs and apparently has a JSON representation as well, so I'll be looking at it too. C/E's scope seems a lot larger than what I'm looking for and I don't necessarily like its grammar.

Not sure if this fits your use case, but there is also https://dhall-lang.org/

looks like dhall can tx to json but not the other way around, so given a piece of json you don't necessarily know the type of a k/v

Emmanuel Oga So, after sleeping on this, I kept coming back to the feeling that what you really want is a lisp and s-expressions, but barring that maybe your life would be simpler of you used something like Karsten's https://www.npmjs.com/package/@thi.ng/hiccup for the markup instead of including HTML strings?

hiccup looks like a possible building block for the markup, not sure if I will be able to use it directly but maybe I can use some of its ideas

But we need the output to be actionable by a student.I like their focus.