You are viewing archived messages.
Go here to search the history.

Emmanuel Oga 🕰️ 2020-12-20 19:10:24

I'm looking for a format to add a bit of semantic meaning to JSON (as minimal as possible).

https://github.com/cognitect/transit-format#transit is such a format, but I'd like to know if there are others.

For instance, imagine trying to represent JSON+Dates, and no other additional type.

From: ["a", 123, ..., "whatever", "~#date:1929292929"] ...

... a JS decoder would see "~#date:1929292929" and convert that to ["a", 123, ..., "whatever", new Date(1929292929)] .

What if I wanted to encode the "~#date:1929292929" string verbatim? Now I have to come up with some escaping mechanism, say, ~~ : "~~~#date:1929292929" , etc.

So I could try to come up with a format to do what I need, but I wonder if there are already specs out there doing something similar other than transit.

I want something a bit easier to implement than Transit... it has a bit more features than I need (for instance, I don't really need caching or user defined types).

Emmanuel Oga 2020-12-20 23:30:08

guys, there's a spectrum ... I'm confused by the black and white answers I sometimes get from the "Future of Coding" forum. It seems the future is black and white...

Emmanuel Oga 2020-12-20 23:30:24

at one side, there's "just use maps". At the other, there's Transit. What's in the middle?

Emmanuel Oga 2020-12-20 23:32:46

I'm just curious how other people solved this issue, seems like a problem a lot of people may have tackled. For instance, if I "just use maps", then if I want to encode that map verbatim, I need to escape somehow. I'm curious how other dealt with this problem. Maybe the answer is to not worry about that case, reversibility, etc.

Kartik Agaram 2020-12-20 23:45:14

Didn't intend to sound black and white! Just trying to answer the question to my knowledge. I'm not an expert here, so it's hard to be dogmatic 😄

Encoding maps: just put them in quotes, use backslashes for escaping? I'm not familiar with Transit, unfortunately.

Kartik Agaram 2020-12-20 23:55:19

From the Transit page:

Users of data formats without such facilities must rely on either schemas, convention, or context to convey elements not included in the base set, making application code much more complex. With Transit, schemas, convention, and context-sensitive logic are not required.

I suppose conventions and schemas is what we're both reaching for. I just haven't felt this particular pain point yet..

William Taysom 2020-12-21 01:35:25

You'll always run into escaping issues in principle, the trick is finding the way to avoid them in practice. Depending on your context, the "~#" prefix might be enough, you might not even ever need to escape them. Not if "~#" doesn't occur naturally.

Using a hash with a "type" key or perhaps a "type" key for extra safety is perhaps a more versatile (and verbose) solution.

The only downer of using extra ~s to meta-escape is that it can get silly sometime. I vaguely recall a system wherein to write backslash I had to use the string "\\\\\\". So having another way to say backslash would have worked out better.

See also Quining https://en.wikipedia.org/wiki/Quine_%28computing%29.

Garth Goldwater 2020-12-21 03:17:04

here’s an off the wall idea: you could have a companion “types” array whose keys are access instructions and whose values are enums representing the type. eg “person.birthday”: “date”. you could make it ad-hoc with relative positioning with a jq-like syntax, eg user: { createdAt: tel:17261669, “.createdAt”: “date” }. idk if that’s too wacky

Christopher Galtenberg 2020-12-21 03:20:15

Hungarian notation for the key names

Jarrett Atkinson 2020-12-21 04:00:25

This may still be too heavy weight, but why not implement just a subset of transit? Iirc it should be possible

Emmanuel Oga 2020-12-21 04:48:58

A subset of transit could work, I was just curious about other options.

Actual transit ends up looking like a lot of nested arrays, something like:

["~#array", [1, 2, [~#date 1929292929], [~#json, [1, 2, 1929292929]]]

Chris Knott 2020-12-21 06:36:14

I like Garth's idea. Send complementary JSON with the same structure, with the type names in the place of the values (you can omit unnecessary parts)

Shalabh Chaturvedi 2020-12-21 07:26:28

What if I wanted to encode the "~#date:1929292929" string verbatim? Now I have to come up with some escaping mechanism, say, ~~

Or you could use the same mechanism: "~#string:'~#date:...'"

Jack Rusher 2020-12-21 08:35:20

https://github.com/edn-format/edn is a simpler textual representation (Clojure's version of sexps) that includes reader extensibility ("tagged elements"). It's easy to emit and there are a fairly large number of reader implementations out there, including https://github.com/edporras/edn_turbo embedded into Ruby that we built at my last company. Would that work for you?

Emmanuel Oga 2020-12-21 08:39:06

hey Jack, I think EDN is a good inspiration too, but is mostly a stand-alone textual format, I was thinking more of something more directly built on top of JSON

Jack Rusher 2020-12-21 08:42:56

What would the layer above look like/seek to accomplish? That info might lead to a better suggestion 🙂

Garth Goldwater 2020-12-21 17:21:06

if you end up going with my idea, please let me know how it works out for you! it’s basically half-assing infra’s method: http://www.christopherkhall.com/research.html

Emmanuel Oga 2020-12-21 22:20:15

The details are a bit hazy ... very roughly, I've a PoC that takes https://gist.github.com/EmmanuelOga/0d626107e5ed2bc6f5e7bf3925d032fd and turn them into HTML. It works but it has quite some layers of complexity (RDF+XML/XSLT).

It uses Saxon/JS. Saxon does what I'm asking: it expresses the full XSLT type system on top of JSON, it is just more than a bit hairy, as you can imagine... At the same time Saxon/JS lacks basic things like supporting async extension functions.

It occurred to me I could create a format to express both markup and graph data with an eye on building on top of JSON from the start, as opposed to the Frankenstein of tech I currently use.

Something like https://github.com/thi-ng/umbrella/tree/develop/packages/egf comes close but it is missing the markup part (mixed content).

For now I'm just looking at "prior art", specially if I can find very minimal formats that build on top of JSON without adding too much complexity to ... steal 🙂 .. some of the patterns people use when building on top of JSON.

Emmanuel Oga 2020-12-21 22:57:34

https://concise-encoding.org/ is able to express both markup and graphs and apparently has a JSON representation as well, so I'll be looking at it too. C/E's scope seems a lot larger than what I'm looking for and I don't necessarily like its grammar.

Roman Sushkov 2020-12-22 16:24:41

Not sure if this fits your use case, but there is also https://dhall-lang.org/

Emmanuel Oga 2020-12-22 22:18:55

looks like dhall can tx to json but not the other way around, so given a piece of json you don't necessarily know the type of a k/v

Jack Rusher 2020-12-23 08:34:29

Emmanuel Oga So, after sleeping on this, I kept coming back to the feeling that what you really want is a lisp and s-expressions, but barring that maybe your life would be simpler of you used something like Karsten's https://www.npmjs.com/package/@thi.ng/hiccup for the markup instead of including HTML strings?

Emmanuel Oga 2020-12-23 11:29:56

hiccup looks like a possible building block for the markup, not sure if I will be able to use it directly but maybe I can use some of its ideas

Breck Yunits 2020-12-24 01:04:04
William Taysom 2020-12-24 02:16:47

But we need the output to be actionable by a student.I like their focus.