I think the fundamental mistake that people seem to be making with LP is using it to describe algorithms rather than code history, code inspirations, bug stories, features and any other context surrounding code like tests / security.

This sounds a bit strange to me. The original goal of literate programming, as introduced by Donald Knuth, was to formulate algorithms in a pedagogical way. I wouldn't call that goal a mistake. Maybe literate programming is not the best way to achieve it, but then... what is?

That said, I am perfectly fine with your approach, which is more in line with the various notebooks that have appeared over the last years. From my perspective as a long-time user of Emacs-plus-org-mode, I'd say you are doing org-mode with Web technology.

But I think this approach misses something essential: the algorithms on which all the high-level stuff ultimately depends. I want those algorithms to be just as accessible to readers as the high-level outline. With notebooks and friends, they disappear in a black box instead.

I am super into LP too, I use Observablehq, and I love mixing in tweets into my notebooks. You can tell a whole story, so you can connect a LP story to internet media (or in my opinion you should do). I also dogfooding my experience here 💬 #share-your-work@2022-03-21T10:38:10.612Z

I also think git is less needed in LP situation, you just need the rollback safety features, and merging for team workflow. Funny enough we discussed that organically last week!

Tom Larkworthy

Yes its about the story! Does Observablehq allow you to embed content from other blocks, like noweb ? Jupyternotebook can't really do that and it is just linear. It is crucial to have that so that you can explain code in any order.

Konrad Hinsen

To take quick sort as an example, I would

• show the video of quick sort first
• show the unit test second
• explain the history of the algorithm
• explain any trivia
• give a brief explanation of the algorithm along with common bugs
• put up personal anecdotes
• put the complete quick sort code rather than go into details like declarations, loops, recursion ...

I find examples going into too much details while completely missing 1-6

In the LP wiki quick sort is described as

<<public declarations>>=

void quicksort(void * base, size_t num_elements, size_t element_size,

        int (*comparer)(const void *, const void *));

<<quicksort>>=

void quicksort(void * base, size_t num_elements, size_t element_size,

        int (*comparer)(const void *, const void *))

{

  <<quicksort declarations>>

 <<check termination condition>>

  <<select a pivot>>

  <<partition the array>>

  <<recursively sort each subpartition>>

}

I would basically avoid declaration chunks like the above snippet ... unless its really a complex algorithm. Most everyday code in apps is far too simple to warrant such detail.

yes, observable is dataflow driven and features non-linear (topological) execution order https://observablehq.com/@observablehq/observables-not-javascript

xyzzy I think the challenge is to link all that together.

Example: I have a notebook that calls a function called quicksort. As a reader, I want to figure out what it does.

First step: look up the code that is actually called. That's your step 7, and should be trivial, which it is in a good IDE but (so far) it is not in notebook-like environments. The only other piece that a code analysis could lead to is unit tests (your step 2).

Everything else would be annotations added by the developer, so now we are in the "documentation" category. Donald Knuth's prose gets complemented by other media, links to Wikipedia, etc. All the niceties that didn't exist at Knuth's time. But that's just a technology update, not a fundamental improvement compared to the LP of the past.

Roughly speaking what I propose is not an update on LP, anything that implements tangle and mangle is LP, but an update on the code documentation genre / style. The final LP document should read like a history book as opposed to technical description of the code. My goal is to capture the evolution of all technical decisions made in a project, algorithms unless they are critical ... like say complex signal processing are of secondary importance.

This one was a style I liked https://observablehq.com/@tomlarkworthy/mip. BIG intro to topic, executable tests, actual implementation was really done elsewhere. Sorta looks more like a text book in the end. I have a change log for the history, though I tend to be quite terse on on that bit. I don't think history is very interesting if you end up reading about doing something that 3 months later is deleted and replaced with a different way. I use history to answer... did anything change since I last looked?

If you want an overview of how a whole community is using LP (which is different to how I do it) I have a 100 of the best here:- https://observablehq.com/@tomlarkworthy/notebooks2021

I like obervablehq's style! What a difference it makes to show examples and tests first. In many ways blogs are literate programs too.

While a history approach may not seem useful for discussing math heavy topics, it becomes very useful for showcasing project evolution.

Right now I am evaluating DuctapeJS and Lua. Instead of throwing the final code away, I will just add a section on scripting engine under early prototypes chapter for example.

Let us say a project is changing from apache to Nginx or from REST to Graphql, all these can be discussed in github issues and then the code changes can be documented. Heck, you can even have a toggle button to show old code - as I am using web stuff for doing LP, this is entirely possible. I right now use a "DEV" flag to hide stripe API keys.

Many decisions like framework choice, architecture choice, requirements can go into LP. Why are you building the damn thing in the first place ? When a function is deprecated I would like that to be documented too. Right now this is scattered in pm tools, github issues, mails, ad-hoc documents and commit messages. Apart from examples and tests, add a tutorial and user guide if you are building a reusable library. I am using wheel to build everyday polyglot webapps, where most decisions are actually related to UI, so I am also documenting colorschemes 🙂

I always love little gems hidden in source code

Crosscut ran into an error.



 Please try a 10-finger reload.

 If that proves unsuccessful,

 you must delete Crosscut,

 and lose all your work.



 Would you like to say a few words?



No one will read them, but your feelings will be set free.

Backstory — we had a retreat for all the lab folks back in December, and we were going to do some user testing of the prototype. But there were a bunch of really easy-to-hit crashes. (Still are — yay, prototypes). So as a quick hack, I added an input handler for 10 fingers that'd just run location.reload() or whatever. Of course, that only worked for some of the crashes.. so I wrote this error message as a prompt() so that people could.. vent.

Turns out, people hit this error a lot, and there was a lot of venting. But it had the intended effect — laughter overcame the frustration of losing your work.

The way Excel does "loops" it by smart autofill from examples. I feel like that would be the most natural fit...?

I don't know if this is impossible to implement, but I think my dream would be to just copypaste as many as I think I need, box-select them, then be able to edit the 'count' box to adjust dynamically.

(obviously there's a lot to love here, I skipped straight to criticism because it was presented as a FoC-nerdsnipe)

copypaste as many as I think I need, box-select them, then be able to > edit the 'count' box > to adjust dynamically.

This would require 'count' to be more semantically aware, and count copied items as single entities.

Good question about production errors... and timely. I had an issue popup. I thought the fact I can debug and step through the server code would be enough, but retrospectively obvious in hindsight, of course that does not help for those heisenberg bugs that happen in production for unexpected situations. I had just such an issue occur.

Anyway, the bug was that the server would break against malformed requests. A classic case of not checking my preconditions. It was quite easy to generate a malformed request as anyone who clicked the API link in the backend notebook source would generate such a request. But all I saw was the server would break after a while randomly. Turns out even though I had Sentry setup, it does not notify caught exceptions and Observable's runtime catches errors.

To find the error I resorted to exporting a notebooks state using flatdata. I had to create a method for exporting a notebooks state first (https://observablehq.com/@tomlarkworthy/notebook-snapshot)

Once I could see the cell "validatedConfig" was unexpectedly throwing it was fairly obvious what the programming mistake was. Then it became obvious why Sentry was not detecting it. So I created a generic catchAll method that creates a global notebook error handler (https://observablehq.com/@tomlarkworthy/catch-all), and from there I could upgrade my Sentry integration to capture notebook runtime exceptions too (https://observablehq.com/@endpointservices/sentry)

So now those random errors in prod should be picked up by sentry, and sentry is awesome because it includes the context and line numbers (pictured)

So now the answer to detecting bugs in prod is to just use Sentry! What's cool about using a browser as the backend is that a single Sentry installation covers both frontend AND backend monitoring. That same sentry setup is present and active regardless of whether using the whitelabel domain (https://thetarot.online) the front end notebook (https://observablehq.com/@tomlarkworthy/tarot) the backend notebook (https://observablehq.com/@tomlarkworthy/tarot-backend), the webcode API endpoint (https://webcode.run/observablehq.com/@tomlarkworthy/tarot-backend;api) or the application embedded in Medium (https://medium.com/@tom-larkworthy/gpt-3-knows-tarot-385d935662a3). You integrate a single state-of-the-art production monitoring solution and cover all bases with a single tool.

A small video where you edit some things and talk us through the process would be nice 🙂

Second this, a video would be helpful!

That is an excellent point, I shall deliver a video in my next side project time slot