This is, I think, an excellent book on end-user programming. https://www.amazon.com/Small-Matter-Programming-Perspectives-Computing/dp/026228040X It includes a significant discussion of the role of peer support

Indeed. Searching http://history.futureofcoding.org for 'nardi' brings up some good past threads.

Here's a way to put it... If you're making sufficiently rich software (from the end-user perspective), they are going to need help using it. You can choose to assist them in getting help, or you can hope that users will self-organize a forum for helping each other. Either way, help will be needed in order to get the most out of your software.

This is a frequent situation in computational science as well. It's not quite end-user programming, but a mixture of next-to-end-user and end-plus-next-to-end-user programming. Typically collaboration of a domain expert with some computing knowledge and a software professional with some domain knowledge.

I think there are increasing levels of end-user programmability:

• adjusting an app's configurations and settings, customising in your settings
• simple rules: email filters, IFTTT
• spreadsheet formulae; box-n-wire dataflow through function boxes
• copying someone else's code and changing the obvious parameters, etc
• doing lots of the above and realising it's become a massive program, then panic!

So I think we're talking about the latter here? Where an EUP system is very easy to get into, but then as a result it's very easy to create a huge blob of dense, opaque programming?

I think it's important that the EUP system offers a model that is structured from the start, and where the normie programmer can draw on the work of others in a structured way, not just copy-pasting.

They should be able to build a complex system incrementally and safely because the structure enables them to see how both their own constructions come alive and also how the work of others operates, and to see the effects of their changes immediately.

(needless to say, I'm designing Onex this way, in the hope of addressing this issue!)

Of course, all this tech is irrelevant if you can't nurture a vibrant open source community that is willing to create examples and to help normies!

Thanks Duncan Cragg for your comment! You are right: an important thing I want to address is how to design a system that can manage the "panic" step. And you point that one way to do it, is to try to propose tools that limit the risk of occurrence of the panic, by offering a safe environment where complexity doesn't "hit" the programmer to late.

But what I also want to address, is even if the programmer doesn't panic, there will be some points where obviously he misses some knowledge/experience/etc. that prevent him to go forward. And to keep going, some external help is needed.

So I would like to take this into account in the design of programming environments. This situations will always happen and how to improve the experience for both the one(s) that need/get help and the one(s) that give it.

@William Taysom:

You can choose to assist them in getting help, or you can hope that users will self-organize a forum for helping each other.Here, I have already "choose to assist them in getting help", and moreover, the environment must be "between-users-help aware".

Well that is perhaps leading to an idea of a distributed, semantic (AST not text) IDE?

It's also worth noting that decreasing scope can often reduce essential complexity; for example we could probably write some crazy huge language that can compile to all other languages but it'd likely be more useless than any of the others too. In the past I've seen a simple core, with pattern-based replacement for "optimization" work well, though the replacements often carry inexpressible constraints (leaky abstractions) and can increase complexity :/

Here's a doc on my site talking about this: http://object.network/onex-app.html

In Onex, you already get a distributed "AST" graph for both rules and data, both of which can be edited by multiple users

Reviving this thread 👋

One thing I’ve found helpful is to consider: What sorts of things do you want people to be able to do with your system? What goals can they accomplish? How will their thinking be changed by using your system? Why should they turn to it, instead of something else? Don’t think about how you’ll do this yet, just focus on human needs that need meeting.

From there, you can start to sketch out ideas (I’d start on paper, as it’s the most free form). Remember the needs and contexts of the people you’re trying to help! Are they best met with text files (and things like git, other source code tools) or are they better met with something more graphical? (Or really far out: are they best met with an entirely new kind of computer??)

At this point, I like to move into something more like a drawing tool (I use Sketch on the Mac) that lets me mock things up and write notes, spatially. I like to imagine different kinds of UIs for solving issues, and then I can write notes along side them, etc.

As far as actually prototyping these things, that’s where I struggle to do so rapidly (as far as real working systems go). But I think there’s something very powerful (and cheap!) about mocking things up in a drawing tool (or in a text file — but beware, if you do everything in a text file, you’ll probably narrow your ability to work on programming UX things provided by your environment)

@Jason Brennan the original thread seems to be about designing languages. I interpret that as about designing textual programming languages. Does prototyping those still have a limitation with text files? (Earlier phases certainly benefit from the ability to draw lines and boxes and so on.)

In general I'm a bit bemused by this thread. As a programmer it all feels quite waterfall-y. If you try to "mock up" a language before giving it a grammar you're quite likely to end up introducing ambiguities that need modifying the language to resolve. Similarly with many other aspects of the activity. If an activity eventually needs to be mapped on non-linear primitives (as computer programs do), it seems to me that it benefits from lots of feedback loops.

Funny story: the original waterfall diagram [1] had feedback loops between phases, so even waterfall practiced right can be quite effective. Of course, that's not what anybody means by the term anymore..

[1] http://www-scf.usc.edu/~csci201/lectures/Lecture11/royce1970.pdf, figure 3

Yep! I still think there’s a ton of benefit to mocking this stuff up in a drawing of some kind, even if you’re doing a textual language.

There’s lots of “programming experience” I hope everyone in here aims to include for their languages, like auto-complete, debugging tools, visualizations, etc. In my view, it’s better to design these things along with the language itself (not as goodies added on later), as they’ll help you steer how you want the language to work.

Of course, as you say, it’s still important to have a somewhat working implementation to know the limits of what can be reasonably expressed in your language, but I look at that more as a limitation than a guiding tool

The tendency of grammars to become ambiguous is not a limitation of some single tool, it's more like a limitation of the universe you live in. It's a little bit like the speed of light. I suppose you could see it as a technical limitation of current technology rather than a property of the universe, but either way it seems as unfair as asking a carmaker why they haven't managed to add a fusion drive yet to their creation. If you create an ambiguous grammar, all your tools like autocomplete, visualizations, etc. have to now deal with the grammar. And the drag (amount of implementation needed) compounds over time.

Perhaps this is getting off-topic. I'll just reiterate that programming isn't just something you do once you know what you want. It's a tool in the toolbox for arriving at good designs.

"Writing doesn't just communicate ideas. It generates them." -- Paul Graham (http://www.paulgraham.com/writing44.html)

Ohhh, Kartik Agaram, have you read A Timeless Way of Building? If you haven’t, you’d probably enjoy it.

I've read bits and pieces of it as others have pointed them out. But I really ought to. Particularly since I cited it in http://akkartik.name/akkartik-convivial-20200607.pdf 😕

I never really got that Alexander had generative grammars in mind for what he calls pattern language. But that becomes quite clear in ATWoB. That elevates the concept to pretty much your description above.

Obviously, I can’t make you read a 500 page book, but please do and then tell me (us) what you think. :-)

I think differential dataflows loops might be useful here. They allow having lexically scoped areas of ordered time, rather than having to try to mash everything into a single global timestep.

Yeah with DD, you’re bringing more imperative control to the declarative-ness so you can try to manually resolve some of the semantic issues that arise (via a combination of careful ordering and explicit branching)

We first need to articulate what is meant by "programming." Without a coherent definition, we will be at a loss to come up with some modified form of it called "end user programming." My view on this is that "telling a computer what to do, later" is a reasonable enough definition, and from that it's easy for us to see that most things done using a computer are "programming-like" already.

The next stage is then to divorce the contemporary association of "programming" with languages and text editors

Using a computer involves using a computing environment, not a language

That environment can certainly prefer a language, or even deeply support a single language (see: older systems that boot into BASIC, etc)

But what is important for everyone -- whether you are the power user called a "programmer" or the regular user today -- is the environment

What I mean by this is that there is not going to be a compelling "end user programming" experience in contemporary computing environments

I mean, many people work at companies on products they don’t themselves use. Part of the rise in the profession of Product Managers and Customer Success is to help bridge the gap to the end-users and their goals.

Programmers ALONE sitting in a cave probably can’t create programming tools / interfaces / languages that accidentally empower end-users. I would agree there

They are not designed with that goal in mind, and are in fact actively hostile towards it

yeah I broadly agree with @Eric Casteleijn

the current software ecosystem is too complex and instills values that are somewhat counter to the goals of EUP

even if you built an awesome EUP tool like Excel, it will need to interface with a MySQL database or hosted in a cloud server somewhere or have complex permissions that enterprises need

Another way to look at this is to look at compelling examples from the past of end user programmable systems

thats why new mediums / platforms entirely are interesting. Like Dynamicland or VR / AR or some new platform entirely

there’s an opportunity to start greenfield on a hardware + software platform that’s uninteresting to existing programmers and build strong values & cultural ideals around observability, understandability, and accessibility \

but thats not easy!

Hypercard is a good example. It was great, and extremely popular in its own day. There are clones today -- why are they not as popular? My answer is because HC fit holistically into the OS of its day. It could do a large part of what the whole OS could do, but within its own environment

Srini Kadamati yeah I agree

Here's an analogy: why is it so hard to navigate the streets of older world cities? Because they've built atop themselves over the centuries, and were designed before mass transit, cars, etc

You couldn't build an NYC style grid layout on top of an old European city without razing the whole thing to the ground first

Likewise, a teletype-based operating system for time-sharing computing systems isn't going to easily "evolve" towards something that is amenable to end user programming

all good analogies. Here’s a more computing related one… Microsoft was investigated for anti-trust b/c of Windows dominance and their shenanigans with IE. Google ended up being more important b/c they were kinda the homepage for the internet after 2000. Then, Apple ended up becoming super important b/c of their control of mobile.

But neither of these companies threatened Windows by fighting directly. They would have lost! Google tried to commodotize / trivialize the OS by focusing on the web (Search + chrome). Apple built a parallel platform that had nothing to do with Windows

The main reason I don't solve more of my problems with programming is that the tools for doing so are so clumsy that it's only worth it for really big jobs. There's an aspect of learnability (bash language is a nightmare) and of integration with the system (hooking most applications is a nightmare AFAIK, especially GUI)

Most of these comments are (interesting) observations about why it’s hard for end-users to use existing environments for programming. Any comments specific to why programmers don’t use them for programming? (E.g., they write code as their job, but don’t write much code to do their job more effectively.)

I'm probably not the only programmer who started off trying to use programming to solve things, but got burned because it was just way more efficient to do it manually.

Roben Kleene it's the same reason

It's too complicated and takes too much time

Yeah totally makes sense, things like Bash seem incredibly clumsy until you try to use something else, and then realize it’s the task itself that’s clumsy (my interpretation)

If you are a professional programmer, someone is paying you to do all that complicated stuff

But personal time is more valuable

That said, bash scripting and things of that sort definitely constitute programming, and developers do it for themselves all the time as far as I've seen

I do a ton of this kind of thing on company time, I usually automate all the things everyone is doing manually. But the people who do this work are maybe 1 out of 10 programmers in my experience.

I think Bash is the most popular, but I still find it to be rare, but that would be an interesting data point if others find it less rare

The task is usually more burdened by accidental complexity than actually clumsy. That's the system integration aspect a couple of us have mentioned. Doing everything in plain text was cute but causes a lot of friction when working with structured data, especially when you have to convert between formats.

Roben Kleene can you clarify what you mean by:

For a long time, I’ve been asking myself the question about why more programmers don’t use programming to solve their own problems,

No Roben Kleene I think you are right; most paid people are not doing it

what do you mean by “their own problems”?

But most paid people are working in environments that are clumsy, archaic, and in many ways hostile

problems b/c of programming / their code? Or like problems in their life (e.g. too much time spent running errands)

or like “I have too many meetings”

personally I don't write too many bash scripts because I'd rather shove a meat thermometer into my ear and wiggle it around. And if I'd rather to that, then I'm always up for just manually doing whatever the task is

yeah I do 0 bash scripting myself

often end up doing a lazy python script

Writing a bash script to do a huge find and replace programmatically, making scripts to quickly open projects they open over and over again, customizations, e.g., I always adding a shortcut to open hyperlinks from my text editor. It’s more lots of small things rather than one big thing.

Then I think "maybe there is some linux program that does this" and I'm off to the races trying to find a command name that makes sense, usually dropping vowels for no reason

Yeah I think you'll find among Emacs users a higher concentration of people that do what you are suggesting. And I'd say part of the reason for that is the quality of the environment

Jeez, this thread is going fast. I'll wait for it to slow down, except to say: I don't understand Roben Kleene's original question, and why it assumes customizing is not programming, and why extensions are not programming. Why is only writing new full programs considered programming? Most of us only modify existing programs in our day jobs.

A founding ethos of programming is to do as little of it as possible. That's broadly true even in my vision of utopia.

I always think of @Steve Krouse example he talked about a “email construction kit”, I keep thinking, wait a minute programmers already have that (Emacs) and they barely use it, and they already know how to program...

Writing a bash script to do a huge find and replace programmatically, making scripts to quickly open projects they open over and over again, customizations, e.g., I always adding a shortcut to open hyperlinks from my text editor. It’s more lots of small things rather than one big thing.Now I see what you mean, kinda like ‘automate the boring things’.

I'm with Kartik Agaram here -- configuration is also "programming"

These seem worth reflecting on:

• https://xkcd.com/1205

• https://xkcd.com/1319

Kartik Agaram Those are all programming! My point is mainly that they stay low on the hierarchy, and don’t progress to writing their own customizations. E.g., when we talk about end users programming, presumably that’s more than cut and pasting a few common snippets online, and that’s about as far as most programmers go with their own environment (while I agree that 100%) qualifies as programming.

@Eric Gade I agree regarding Emacs, part of my inspiration for this question is why isn’t it more popular? And, furthermore, VS Code, which is arguably the most popular text editor ever, specifically makes these types of customizations harder (there’s no .emacs equivalent).

Emacs is not more popular because it is designed to be a system that is text only

But the systems everyone uses -- even programmers -- have robust GUIs

Re the xkcd, if we haven’t been able to make automation a worthwhile time investment for experts, then what chance is there we'll be able to make it worthwhile for non-programmers?

I think your question is a very good one actually

But I'd say that perhaps professional programmers are "experts" in many of the wrong things

Frankly if I was building a new computing system from the ground up, I would not want to work with most people trained in CS these days, or anyone who has extensive software development experience

My litmus for a computing system in 2020, both as a professional programmer and as a user, is this: can I easily create a button that does some task I want when I click it?

The answer is no, both for the programmer and for the user

On some personal computing systems this used to be quite easy for both

For me the answer is that nothing feels like a stable foundation on which to build. If somebody wants to pay me to spend a majority of my time fixing previous code I and others wrote, that’s fine, but there’s something deeply frustrating about it when you see the extent to which your tools get in the way of your own vision. I don’t want to live in the command line, but that’s about the only place I feel confident that what I build can persist.

... if we haven't been able to make automation a worthwhile time investment for experts, then what chance is there we'll be able to make it worthwhile for non-programmers?

"There never was much hope, just a fool's hope"... but for some reason we're all in this Future of Coding thingy anyway. I see this as the exact problem we're here to solve.

I actually do automate a lot of things (and I use Emacs). Mainly because the emacs makes it easy to do.

For me, the important aspect is extensibility. In many cases there is an almost good solution but with one or two things I need to do differently. And too often the cost to modify the solution is about as hard as to reimplement it from scratch. Especially true for full-blown GUI and android apps.

For emacs, you can always take a half-baked solution, modify its code on the fly and you're done.

If I were to speculate about why others don't do the same and automate easy tasks, I would say that the tools have high learning curve. It took me quite a while before I started to feel proficient scripting Emacs. There is also an attitude problem—not all programmers know they can extend their editor and what's possible, so they don't even look for opportunities.

in particular, i think that the majority of the “old european cities” we’re living in we’re built for batch processes. most of the tasks i’d like to automate have to do with user input (the click a button example is really stark and appropriate). and that user input isn’t the start or end of the program. id really like pretty much any environment that gave me live insight into how eg keyboard shortcuts and button presses were flowing across the system, and similar insight into graphical primitives and responses. i can build data abstractions pretty much anywhere but gui and input processes always feel like i’m running a marathon with petulant snakes around my ankles

Another relevant xkcd;

https://xkcd.com/974/

I think currently automation appeals to people who actually enjoy the problem solving aspect. The fact that it may or may not be massively more efficient is secondary.

I wrote a python script to combine image files into 2x2 combined images of 4, because it was slightly cheaper to print those as "single" images for my wedding than use the company's own quarter size print option. Probably only saved a few quid in total but I enjoyed the task.

This instinct is kinda orthogonal to people who like programming/computers.

I know people who come up with elaborate schemes for making sandwiches/wrapping presents/stuffing envelopes etc en masse. These type of people are the ones who will jump on end user programming when it becomes easier.

A lot of users though, aren't even using copy-paste, select all, find-replace etc yet

Maybe because we are programmers, we know the cost of creating/maintaining the things we make, so we rather spend our time doing other things. One quote that has stuck recently is, “In software, anything is possible, but nothing is free”. Even though, I can certainly optimize some of the things I want to do, I don’t want to pay the time and effort to do so.

Garth Goldwater I will add that when I'm working on Smalltalk projects, I tend to make those kinds of buttons all the time, and just pop them out onto the desktop. I wish I could do it in macOS

Chris Knott There is lots of evidence from the past that the kind of division you are describing is more of a gradient. Apple used to be really good about this, not only with Hypercard, but also with Applescript, which they have shamefully allowed to die on the vine. Macs were the desktop publishing platform of choice because "users" could write Applescripts for performing tasks across applications like Illustrator and Quark etc

What I think is important in future systems is that they should be discoverable in the sense that you can "peel back a layer" and see how things work in some slightly more complicated context, and then when and if you are interested, peel back a subsequent layer. Ideally this would have manifested itself in, for example, a MacOS where the "top layer" was all described in HyperTalk/Applescript

But they never went that far

Now if you want to make your own buttons in macOS, say, you have to download xcode, learn about unix, build processes, and a full on programming language

There is nothing in between, and for no good reason!

Roben Kleene I eventually came to believe that it’d take a pretty significant societal shift for it to be realistic now. The time that many of us base our dreams on (the days of hypercard and VB6) just doesn’t exist anymore - there are now endless distractions and more apps to do what you want than you could ever begin to look at. In the earlier days of computing there was just.. less stuff. And it was more obvious that if you wanted something maybe you’d put it together yourself. Even with the perfect toolset, most people aren’t aware that they even could change things and even when you tell them they can, they don’t know what to do with it.

Fortunately if you take a longer view, there is one clear exception to that: contexts where you can use programming to build stories/worlds. Games like Minecraft, Roblox, etc, expose kids to programming in a way that isn’t “let’s do this faster,” but instead through the joy of making universes to explore. If I were going at this now, I’d be focused on building a path for the folks who grow up in those worlds to apply what they learned to the more mundane “adult” life.

We can definitely continue to lower the floor of programming and open it up to more people, but to achieve the true “end-user programming” goal, we’d have to significantly change people’s relationship to computing. Realistically I think that’d take growing a generation.

to second Chris Granger in the early days, there weren’t these dominant platforms / models that everyone was already used to having. I think there’s optimism here if you look at other areas entirely where the experience & tooling is very different for an end-user vs a professional.

Cooking is the best example I keep coming back. Home chefs / ’amateur’s use scaled down pots & pans. Industrial kitchens are more like factories and use giant equipment that an ‘end-user’ would never really consider buying. But both of these are 100% valid industries and crafts. There’s more content targeting end-user home-chefs than there is for elite chefs. In theory, restaurants could provide 99% of the meals that all humans need but cost, cultural traditions, etc are still big barriers.

This has preserved home-cooking for literally centuries! If restaurants / central kitchens could provide all meals at the same cost as you making it, I predict we’d see a huge dip in home-cooking. Most people don’t make their own furniture anymore but pretty much everyone still cooks to some degree (even if you’re just microwaving food). Necessity is really the driver / mother of invention here. Home-cooking is still the best hammer for most people’s food problems

This just showed up on my feed and seems relevant.

https://twitter.com/tayroga/status/1296538378255491072

Essentially programmers (and myself) do some programming for our own problems. But the systems we have aren't fundamentally designed for end user programability. So the burden is large. We do it more where some software has been designed for extension in some respects only - e.g write a plugin for your editor.

I like Alan’s heuristic of fist / hand of code, and then a “page of code”. If adding something takes more than these… the burden might be too much. Anyone (even kids) can quickly load code-context into their short-term memory and extend / modify software. Past that, you’re closer to a programmer

Speculation: The vision that Alan Kay describes in this talk hasn't been realized in mainstream software because it doesn't fit with how industrial societies see production as separate from consumption. We even use the economic link between the two (money flow, GDP) as the measure of collective wealth. People solving their own problems don't contribute to GDP, so their work has no value for the economy.

If there is some truth to this, the good news is that industrial societies are slowly realizing that maximizing GDP isn't such a great idea. And in a way the FOSS movement is a first sign of this in the software world.

Konrad Hinsen I agree but I think my comment from earlier adds another tidbit here:

https://futureofcoding.slack.com/archives/C5T9GPWFL/p1599602445178200?thread_ts=1599588394.135900&cid=C5T9GPWFL

There’s no economic story around end user programming. In my cooking analogy I give, making food yourself has pretty much always been cheaper than buying restaurant or mass produced food. There’s also been strong cultural ideals around cooking. But if it was ONLY the culture, I’d suspect very few people would cook.

Commercial software is so so cheap because of the 0 distribution cost of software. The ‘junk food’ / ‘processed food’ of software is good enough for most people most of the time

I think its interesting to think about areas, users, personas, use cases, etc where customization, culture, and cost are aligned towards EUP.

• Creative art. Sure many folks use Photoshop and what not, but people are willing to experiment with new tools if they can express themselves in new ways.
• Kids / K-12. There’s definitely pressure to “teach Python for jobs” or w/e but many kids intrinsically don’t yet care about jobs and more about creative activities / things that solve problems now for them (e.g. setting up Minecraft servers or making their own games)
• Non-technical users that dream of making technical things. I found out from this Slack that a lot of the award winning game Hollow Knight was created using Playmaker (no-code / wires & boxes editor for Unity). The creators didn’t let their lack of deep coding skills stop them!
• Data science. This is my world, and the trend here is to make everything work via a SQL interface, literally. SQL is learnable by pretty much anyone IMO (my mom struggled with Java and Python but is better at SQL than me!). Getting analysts to learn Python is a big ask and people worry about mistakes that could be make (plus that type of code is harder to audit). Also, I’ll throw in obligatory popularity of Excel here! You still got the ML in Python people but most analysts and less technical people don’t necessarily want Python / R right away.
• Note taking / personal organization. There’s a subset of note-taking (2nd brain, Roam, Digital Garden etc) who are interested in adapting their current tools even more to their personal workflows & preferences and don’t mind simple scripting.

When Chris Granger says we need a cultural change, he’s right. Our culture is utilitarian and wrapped in short-termism. This is reflected in computing as well.

FOSS is good in some ways, but kind of counterproductive structurally: we know from the 60s and 70s that in order to get truly qualitative leaps in computing, we need to fund people for longer periods of time with not so many restrictions. The current economic thinking precludes that kind of funding model

Additionally, as Srini Kadamati has pointed out, our education system has also become a victim of this cultural shift. Students are not educated, but rather trained for jobs. Hence the obsession with “teaching kids to code”

FOSS kind of reinforces all this by providing increments upon the leaps of the previous generation, without providing a means for future leaps. Corporate sponsorships and people doing projects in their free time isn’t going to cut it. They need the means to follow sometimes errant paths for years at a time, and they need to be funded during that time so they can concentrate

I respect the whole free software / libre movement, but if there is not a change to the greater political economy, it’s going to be a movement that noodles around in unix and other partial techs from the 70s until the end of time

I share all of your thoughts @Eric Gade and sentiments. There’s a pessimistic and an optimistic view, but the nature of the change of revolutions seems to be about the niches that adopt these ideas.

The good thing with some of the areas I listed above is that I feel / think that those end-users are a bit more patient. They want to do more with less and they’re willing to try tools that are different but can enhance their workflow.

There’s still the “integration with existing tools / workflow” problem though. The data science one is that ultimately any fancy EUP tooling still has to talk to a clunky big database. With art, people expect similar formats as the output (SVG, PNG, MP4, etc).

Srini Kadamati Great examples! I'm particularly interested in the Hollow Knight story, if you happen to have a link or any other source I'd love to read more about it.

The theme of the responses here seems to be: Only a small percentage of people are interested in using programming to improve their own workflow, but much more people are interested in using programming to build things to share. Which I think resolves my initial conundrum about programmers not using programming to solve their own problems: I was looking at scripting and customization, but what I probably should have been looking at is things like side projects and personal sites. Which to me seem much more popular than scripting/customization?

There might be a lesson for end-using programming here too: It's probably better to focus on tools that let people create things for other people than it is to focus on anything that you'd call "automation". For whatever reason most people aren't interested in automation (maybe just because it's not worth the time, e.g., the relevant xkcds)? But they are interested in building things to share, e.g., see the Hollow Knight example above. This seems consistent with the no code movement going on as well.

Perhaps expressiveness > utility

Lots of good contributions here... just one more comment on the "most people don't need more than ready-made apps" argument: that's really a cultural issues. People need food and shelter, plus whatever it takes to become a worthy and respected member of their society. Software is new, it appeared when our societies were already well into an industrial mindset of production and consumption. Other DIY activities, including cooking, have been part of our culture since long before the industrial era. One reason people cook at home is that they grew up seeing other people do it, and they have seen the advantages (economic, social, etc.) it brings. In a culture without end-user programming, it's not surprising that few people miss it.

Roben Kleene https://twitter.com/AsherVo/status/1293653951309426688

Thanks! Made in Unity with Playmaker (which I'd never heard of) https://assetstore.unity.com/packages/tools/visual-scripting/playmaker-368 Just amazing!

The theme of the responses here seems to be: Only a small percentage of people are interested in using programming to improve their own workflow, > but much more people are interested in using programming to build things to share> . Which I think resolves my initial conundrum about programmers not using programming to solve their own problems: I was looking at scripting and customization, but what I probably should have been looking at is things like side projects and personal sites. Which to me seem much more popular than scripting/customization?

I would nuance this more. Clayton Christenssen / Jobs-to-be-done framework and all these other stuff from product management land (disclaimer: I used to be a PM) emphasize that people don’t care about your product. They care about making progress towards a problem they have and they may hire your product / service or another one based on how well it solves their needs.

Cooking food solves many problems for people (cost to feed family, taste — restaurants can’t quite replicate home food taste for many, convenience once you know how to cook, feeling of self-sustenance, social — cooking for others, and probably 5 more). Getting food delivered is a worse proposition in many cases (except when you’re busy or money isn’t a concern or you want specific cuisine). Right now, most programming doesn’t solve problems people have today. It’s not ‘economically’ a better solution for anything for most people.

using programming to improve their own workflow, > but much more people are interested in using programming to build things to share> .

There ARE people who care about improving their workflow. Bankers and traders 🙂 they learn the shit out of Excel and use it to automate their workflow, check for errors, etc. B/c it literally saves them time and helps them make more money or get an advantage in the market.

Regarding bankers and traders, do you think a higher percentage of those groups customize their workflow with scripts than programmers customize their workflows? Sort of my starting theses is that for programmers, using programming to improve their own workflow is niche. I'd be very curious if there are other industries where using programming to improve their workflow is mainstream (for that industry).

no idea tbh, but I know a large % of them maximize the hell outta Excel. They really learn them well. Maybe programmers do this in other ways (customize their emacs setup, or their shell setup or w/e for little wins here / there).

I mean we have an eng team at work and they’re constantly looking for ways to improve the cloud deployment infrastructure, speeding up test suite, etc. I categorize those as ‘workflow improvements’ even though they accrue to all engineers in an org instead of just the engineer themselves

Re "FOSS being counterproductive strategically" - this twitter has interesting takes: https://twitter.com/jonathoda/status/1104522585092481024

Hypothetically lets say what's holding us back is poor forms of composition and the existing ontology of computing, so we need to design new forms that scale up better. However to be adopted we must be compatible with the existing world and so "compose" with it - so we perpetuate the existing composition and ontology models. This affects both commercial and free/open source software. Might affect side projects more because of a greater need to 'fit in'.

yeah I saw Jonathan’s tweet when it went out and it really made me ponder! At work, we’re commercializing / stewarding this viz / BI tool called Apache Superset - https://github.com/apache/incubator-superset but right now the #1 reason people use Superset and leave PowerBI / Tableau is because “we’re free” lol. We haven’t leveled up yet on the value front

I’ve described it this way. I think of FOSS and perhaps establishment computing today as being like the medieval scholastics. The certainly produced a lot of work, even original work in some sense, but it was all confined to Aristotelian thinking, and their contributions were increments on rehashing Aristotle. For centuries.

There’s lots of cool stuff out there in FOSS that makes a lot of different things possible in computing (though we should be honest that most of those things are commercially useful)

But that doesn’t mean FOSS isn’t rehashing unix forever. It seems that’s what is happening

Look on HN where any discussion of a “new operating system” is really just a new linux distribution

Regarding:

I mean we have an eng team at work and they’re constantly looking for ways to improve the cloud deployment infrastructure, speeding up test suite, etc.

Personally, I've struggled to get other engineers interested in working on these things, but of course that's extremely anecdotal. I'd love to hear perspectives from others on this.

Regarding Unix, the general pattern seems to be moving towards Unix system, e.g., Windows Linux Subsystem. My interpretation of that is that consumer computers seem to be going the way of locked-down smartphones, so traditional computer operating systems are mainly for programmers and other "heavy" workflows (media editing, etc...). There's just too much infrastructure on Unix systems to support those workflows to do anything else.

I mean we have an eng team at work and they’re constantly looking for ways to improve the cloud deployment infrastructure, speeding up test suite, etc.

Personally, I’ve struggled to get other engineers interested in working on these thingsI think when an org is big enough to have an internal products or platform team, they would work on these optimization/efficiency/infrastructure tasks.

In my experience, once an org reaches a certain size, there will be a dedicated team to maintain this stuff, but until then no one really wants to work on it. (Which I find so odd, because if there are any problems with it, if you're a dev, you're feeling those problems on a daily basis.)

I agree, based on working at a company with about 200 engineers. In a startup there is a lot of firefighting and pressure to launch / work on user-facing products, and infrastructure can be just good enough. When you have permission and management support (and even incentives) to make infrastructure really solid, then you can fix a lot of systemic problems.

At the opposite end, you can go on infrastructure vision quests, making something that other devs don't want to use, etc.

It's probably been said in the other 109 comments, but more people would shell script if the tool names and interfaces weren't so incredibly unmemorable. Is anti-memorable a thing? (I just decided it is.) The interface kinda stinks.

Git has an anti-memorable user interface, but by building GitHub and the GitHub desktop on top, it's now pretty easy to use without the fear of never seeing your code again. More people use git (i'm guessing) than shell scripts, even though its command line interface is irregular, dangerous, and unpredictable.

was just checking out some of bessemer’s recently released investment memos and i think an argument can be made that Shopify is actually a great example of an end-user programming success story: https://www.bvp.com/memos/shopify

yeah I agree. Shopify is a beast of a company and they aren’t interested in trying to centralize everyone onto a vague platform like Medium did. They embraced “the edges”

I might be missing it in the thread here, but one reason even programmers don't script things is that there's no good way to do it, a huge gap between interacting with a GUI and automating the interactions.

Coming back to this thread, I am wondering; what is that "programmer" category? Professional software developers, perhaps? Is that a sufficienly homogeneous group in their professional practices to discuss how much they develop software for their own needs?

My perspective: I did more "end-user programming" before I was a programmer, but knew enough programming to be useful. This was at an office job, where everything ran on Excel. I was doing lots of web scraping, data processing, making small tools for tedious tasks, etc.

Then I got a job doing programming, and didn't have any material problems to solve with programming any more!

So, to the question:

"Does end-user programming ever have a chance of succeeding for non-programmers to solve their problems if programmers themselves aren't using programming to solve their problems?"

my answer is:

"yes! Non-programmers have way more problems solvable with programming than programmers do"

To the specific point in the original post about some programmers not customizing their programming tools:

one hypothesis is that programming tooling is already pretty well optimized for the output that the industry wants from programmers. Programmers are customers too: It only takes a small amount of dedicated people making tooling like IDEs, text editors, plugins etc for the majority to benefit from the care and thought put into these tools.

My experience definitely matches yoshiki's (although I went the other direction, out of software development).

The amount of menial, manual use of computers that goes on in the world is a travesty. I'm talking about stuff like physically typing filenames into a Word doc.

It's long been recognised that a manager might send emails to her staff, instructing them to do some task on a computer, in a way that is almost pseudocode. e.g. "can you go through every sales report on the K drive for February and check if any of the unit codes have expired?".

End user programming should be looking to eliminate the middle human from this kind of human->human->computer situation.

This is the type of thing that SQL was meant to eliminate, but it didn't. I think the issue is solutions like SQL demand too much subservience in how information is managed, and they want it stored in a way that is different to how humans would naturally do it. The also demand labour up front (at the point of storing), for no immediate reward, which is always a foolishly optimistic thing to require.

I have high hopes for a system that approaches the OS from the same perspective as the user, for example all text OCRed, structure automatically inferred from physical layout etc.

Chris Knott Totally agree about the manager request example. I did a freelance job a couple of years ago for a non profit. They really just needed a specific way to process bank account information from several online services into an Excel template they used. The people working there knew exactly “what needed to happen,” but the tools on their system had no way for them to express it, and the services they were using were siloed from each other

In the end I had to make them a quick electron app that did it, and it was really eye opening how complicated the programming task was even though the real task was conceptually simple, and all the “parts” were already there in the system

As time has passed, I consider the design that got them (and myself) into that position to be hostile. But it gave me work, just like poor design gives programmers work every day across the world

Regarding:

one hypothesis is that programming tooling is already pretty well optimized for the output that the industry wants from programmers. Programmers are customers too: It only takes a small amount of dedicated people making tooling like IDEs, text editors, plugins etc for the majority to benefit from the care and thought put into these tools.

This is an interesting perspective, and I believe it's true to an extent, e.g., while almost every powerful application from Excel to Photoshop to DAWs has a cottage industry of extensions that surround it, it's certainly true that developer tools dwarf the others in quantity of existing customizations.

But I still see a lot of contrary evidence. E.g., it still seems that most programmers, when they encounter a problem with a programmatic solution, they'll tend to choose a manual solution. The canonical example is automated testing. While automated testing has certainly become popular, it still seems to be in the "eat your vegetables" category, instead of something programmers just do naturally. Which is odd if you think about it, because manual testing is just that, manual, and programmers are master automators, so...?

More personally, what I find most fulfilling about writing my own scripts and customizations is that I can make the software behave the way I want it to. When you're using someone else's customizations you're always at the mercy of the creator's decisions. I don't think that wishing your tools worked differently is rare, that's a sentiment that I feel almost every computer user has, but programmers are the only group that's empowered to change how their tools work today, and for the most part, they still choose not to.

I’ll challenge some of your assumptions here Roben Kleene not sure if this is useful, but could be another perspective!:

• Automating isn’t always better than manual. Clicking through a new UI sequence can give you the ULTIMATE end-user gut check. Things can still go wrong (caching, what have you) but less is likely to go wrong. You can build a burrito robot but unless you taste the burrito at the end with your own mouth, you just don’t know if everything worked as you thought it would.
• Automation is often less concrete / tangible. Similar to the first 1, but automation is also more abstraction. Abstraction is complexity. Even if the automation script is something simple, there’s overhead now to maintain a list of automations. Perhaps we need better ‘automation interfaces’ where the overhead is brought to 0. Analogy: I don’t think about not eating cookies in my day to day, I just don’t keep cookies in the house. 0 overhead!

I agree 100% with both of these points. I'd be the first to admit that I like automating things because I don't like doing them manually, not because I think it's objectively better by any other metric besides my own personal preference.

this is a long thread… it could be interesting to fork and start a new thread and this time include like 5-10 concrete examples in your life / things you’ve seen other programmers do Roben Kleene etc to spark the discussion!

I'd also say that "how can doing things manually still be better than automating them in 2020?" Because I believe that it's still better for many, possibly even most, cases is one of the central questions I'm grappling with.

Roben Kleene: Nooooooo! 😄 It seemed like you were moving towards the light here:

I'd be the first to admit that I like automating things because I don't like doing them manually, not because I think it's objectively better by any other metric..But then you immediately put your blind spot back on in the next comment:

how can doing things manually still be better than automating them in 2020?That framing is only going to lead you in circles. As Srini Kadamati pointed out above, and as I tried to say in the overflow thread ↪:

• The line between 'manual' and 'automated' is fuzzy on the computer. If I switch windows and type a command on the shell, I'm still making use of automation. Just less of it.

• Adding levels of automation always has costs. If it seems always a good idea to you, just wait a few years. We can improve lots of things here, but it's just not a reasonable goal to aim for "adding levels of automation should always improve life". There will always be situations where doing something manual is simpler, faster, less alienating. Start developing some warm and fuzzy feeling for doing things manually.

• Desire for automation is subjective to some extent, as you pointed out in your first comment.

I'd say join me over here where the goal isn't automation but comprehension. Practice throwing kicks not because kicks are always a good idea, but just so you build up judgment on when to use a kick, and so you can do a lot more with a single kick when the moment arrives.

(Movie recommendation: https://www.imdb.com/title/tt0061770)

Adding levels of automation always has costs. If it seems always a good idea to you, just wait a few years.I think you're misunderstanding me here, I'm specifically saying I don't always think automation is a good idea, I'm saying that it's the way I prefer to solve problems. It's an inclination that has more to do with me, than it does the problem. Correspondingly, I tend to choose to solve problems my approach is a good fit for. I do a lot of work with frameworks, where automated testing is more important. I tend to avoid issues in the UI layer, since manual testing is usually more efficient there. (I actually love working on design-system level UI stuff, but bugs that exist between the UI and the data layer are probably my least favorite thing.)

I would love to hear more about this "love of doing things manually" though.

I would love to hear more about this “love of doing things manually” though.I’m thinking if the environment you are in is introspective and malleable enough, the manual approach is just so easy

And the lines between manual / automatic become hazy

Sure... I agree that line can be fuzzy, but I guess I'm not sure how useful that distinction is, e.g., I'd put someone using a malleable environment / using automation on one side of the coin, but most programmers are still on the other side of the coin, where they're not using a malleable environment (or at least leveraging it), and they're not automating.

I think someone should do this as a real study

And collect background information about the programmers in question

Also if anyone wants to share there definitions or examples of malleable environments (i.e., environments that are so efficient that they remove the advantages of automation) I'd love to hear about that

The most obvious thing that comes to mind is text editor keyboard macros

You can record one, use it over and over, and save it if you want.

Importantly, you are personally in control at every step, and it is a very lightweight abstraction.

Ditto for doing things in Emacs lisp. Once you have the lisp made for it, turning it into a command is easy

Yeah, that's the first thing I thought of too. Keyboard macros seem pretty clearly automation to me? (And correspondingly, would be a feature most programmers don't use). It sounds like there's a concept of a malleable system that's not based in automation here, I'd love to understand what that is. (And I'd love to hear any more examples of both.)

While I object to needing your IDE to have thought of the things you want, JetBrains does a pretty good job of thinking of the things that my coworkers and I want, and I think we do leverage it to an extent that it is “automation”. Multi-cursor, regexp-replace and structural-replace are some examples that come to mind.

These text editor examples highlight something: good automation blurs the line between manual and automated action. Specifically, it makes the automation invisible by making it so easy it feels like manual work (this might just be my view of what you folks are talking about re malleable systems). Multi-selections are a great example. You can think of them as automating repetitive identical edits, but they feel like just cutting with a sharper knife, not programming a chopping machine.

Possibly one of the key things that makes them feel manual is that you can see the changes as they're made. Not exactly a new idea (oh hai Bret Victor), but maybe a new perspective on why it's important. If every bulk edit operation on a doc or database ran in a transaction with a live preview of the changes, people might feel more comfortable playing with them.

but programmers are the only group that's empowered to change how their tools work today, and for the most part, they still choose not to.Roben Kleene I don't think it's so much that they choose not to, it's often that they don't need to.

yoshiki Do you have any thoughts on preceding part?

When you're using someone else's customizations you're always at the mercy of the creator's decisions. I don't think that wishing your tools worked differently is rare, that's a sentiment that I feel almost every computer user has,

E.g., my observation is that developers tend to be frustrated with their tools (especially more experienced ones). Do you not find this to be the case?

(I guess I should add that VS Code appears to be the first very popular text editor that developers don't seem to be frustrated with, so perhaps my perspective is outdated.)

Don't you also see experienced developers being more resigned to their tools? In fact, that's one way to diagnose all of us pushing back on you in this thread: we've been living with our chains so long that we've forgotten about them. That would certainly explain why we don't customize more.

I had more customizations in 2001-2005, then I moved and changed a lot of stuff and just restarted my text editor settings from scratch. And they've never grown back to the sort of multi-megabyte state they were in then.

Backing up to your previous comment to me (I wrote a response yesterday that the Slack app ate):

I think you're misunderstanding me here, I'm specifically saying I don't always think automation is a good idea, I'm saying that it's the way I prefer to solve problems. It's an inclination that has more to do with me, than it does the problem.I don't understand the distinction you're making here. At least, what you're saying sounds like it answers your earlier question, "how can doing things manually still be better than automating them in 2020?"

Funny thing is, I do a lot to automate testing. In fact, one way to view my Mu project is as making the UI layer easy to test.

I would love to hear more about this "love of doing things manually" though.When I do something manually I sleep soundly at night that I haven't created new tech debt for myself. I don't have something new I have to maintain, or try to read later to try to understand just what I was thinking. Manual labor can be therapeutic, like gardening. All these things have nothing to do with the state of the underlying system. They're just about the mess I make, and about taking care of my own state of mind.

Sometimes I do things manually for a few days even when I'm sure I have to automate them eventually. Manual work keeps me close to the data and might give me some new insight. "Being the computer" helps me understand the problem before I try to solve it.

E.g., my observation is that developers tend to be frustrated with their tools (especially more experienced ones). Do you not find this to be the case?I see pockets of people feeling frustrated and other pockets of people who aren't. It's difficult to summarize since I think the landscape is complex(like you said, experience plays a part). I agree though that not everyone is satisfied. I'll have to revisit this later when I have more to say.

On automation: it has two very different costs that need to be weighed against the benefits. One has already been cited: the initial effort to put automation in place. The other one is less obvious: a fading understanding of what is really happening. It's much more pronounced if you run someone else's automation, but also happens when you run your own code for a long time without studying it from time to time. And if then you have to change something, it can be difficult.

So automation is an obvious win only if the two costs are low. Stuff like renaming hundreds of files: shallow but lengthy. The loop with a few-line body.

Kartik Agaram I really appreciate these comments, some responses below:

Don't you also see experienced developers being more resigned to their tools? In fact, that's one way to diagnose all of us pushing back on you in this thread: we've been living with our chains so long that we've forgotten about them. That would certainly explain why we don't customize more.

This seems to be saying the same thing I am, experienced programmers are unhappy with their tools, but accept them (that's the way I'm interpreting "resigned" at least?) The question is just why. You've given some great reasons, but I'm not entirely convinced (not saying I disagree either, just not sure either way), but I think the best response would be to give some concrete examples. That's also something a couple of others have requested too, and I also think it's a good idea, so I'm going to start a new thread soon with examples.

I had more customizations in 2001-2005, then I moved and changed a lot of stuff and just restarted my text editor settings from scratch. And they've never grown back to the sort of multi-megabyte state they were in then.

I used off-the-shelf software for the first half of my career (2002-2010), mainly with the defaults, then started customizing after that. I realized it solved a really really big problem for me: I used to keep changing software because an alternative would solve some problem I have, but after using the new software I'd realize the old software did some other things better. So my total number of problems would always stay the same.

Now that I customize, my total number of problems goes down. Now I think of applications like as a shell that I can customize to make it do the things that are important to me very quickly. And I'm so much happier as a computer user this way, because I'm not in a constant state of frustration because nothing works the way I want it to like I was before (this is is probably a core trait of customizers/automators, as well as contrarians in general).

I think you're misunderstanding me here, I'm specifically saying I don't always think automation is a good idea, I'm saying that it's the way I prefer to solve problems. It's an inclination that has more to do with me, than it does the problem.

I don't understand the distinction you're making here. At least, what you're saying sounds like it answers your earlier question, "how can doing things manually still be better than automating them in 2020?"

To me these are two separate thoughts: In the first, I'd guesstimate automators are about 1/10 of programmers, I'm just saying I'm in that 1/10. The second part is just surprise that using a computer programmatically is still so difficult in 2020, given it's a problem so many people have worked on. Not sure there's a contradiction here? I think you're saying #2 leads to the 1/10 in #1, which I agree with. But I don't see why that would change my personal preference?

When I do something manually I sleep soundly at night that I haven't created new tech debt for myself. I don't have something new I have to maintain, or try to read later to try to understand just what I was thinking. Manual labor can be therapeutic, like gardening. All these things have nothing to do with the state of the underlying system. They're just about the mess I make, and about taking care of my own state of mind.

I understand this in theory, but I just don't experience using computers this way. This reminds me of how I feel when I watch Gray Bernhardt type, I find him amazing to watch, but he has an affinity for the mechanical act of typing that I just do not share. I find just imagining doing what he does myself exhausting. (Also, doing that many small mechanical motions relative to his output seems like a recipe for RSI to me.) There is just no way I'm ever going to use a computer by typing everything out the way he does, the "cost of a key stroke" is just higher for me than it is for him.

Roben Kleene That really helps clarify things. I certainly agree that computers today fit better for people with a certain profile (who find certain things like typing cheaper to do), and that there are huge barriers to customizing them for other profiles. I think it analogous to the direction of a vector rather than its magnitude. The things you want are not that far from the current defaults, it's just that there's huge impassable mountains in the direction you want to take computers, whereas I[1] and Gary Bernhardt have the advantage of living in the direction of the flat plains. (And the poor laypeople are out in outer space with no service.)

[1] Though I see myself moving closer to you over time, what with my recent RSI troubles.

I’ll just add that I think both approaches work, John Carmack seems like a use the defaults type (https://twitter.com/id_aa_carmack/status/1302651878065475584?s=21). Linus seems like a customizer (maintaining his own Emacs, and git seems like explosion of Bash scripts, at least when I’ve looked at it under the hood).

I'm noticing an analogy here with lifestyle design. A few years ago there was a movement towards minimalism until people realized that not everyone is rich enough to afford minimalism[1]. Similarly, I think when you see someone who says they just use the defaults, you're really seeing someone privileged to be better suited to their environment.

[1] Arguably this pandemic has made us all poorer by forcing us to run less lean and maintain more inventory of more types.

As someone who comes from the antediluvian era when personal computers were personal, that was one of the most depressing threads I've read since joining this community. 😥

Yes! Sadly.

The more I learn about that "antediluvian era" the more I think we have this whole thing backwards. Back then, people like Alan Kay were trying to figure out how to make computers work for everybody. "Programmer" was just a synonym for "computer user".

But then we somehow accepted the idea that it's better to distinguish between so-called "experts" that can create all the software for "not experts" to use, even though these "experts" knew nothing of what the "not experts" really wanted to achieve. Primarily it seems, so we can sell a sh*tload of products and turn a planetary-scale nuclear attack-proof knowledge-sharing computer network into a shopping mall.

The result is that now everything is so complicated that not even the "experts" find it feasible to do most things with a computer themselves, and the "not experts" are now convinced that they can't possibly learn how to do anything useful with computers unless somebody else creates it for them.

We should've never stopped trying to make computers work for everyone.

And we should've kept talking about the "medium" aspect of computing more.

Ah, sorry, that turned into a not very constructive rant… I do appreciate that there are still a few people trying to make computers work for everyone, and in this community in particular.

This whole "end-user programming" thing (even though I've adapted to use the term because otherwise it just makes communication more complicated than it already is) just feels totally backwards to me — starting from a programmer mindset and trying to invent an easier way for non-programmers to do programmer things just seems wrong, wrong, wrong to me.

We need better options for non-programmers to teach us how to use computers in ways that are more useful and more relevant.

Stefan Lesser (last paragraph) have you expanded on it elsewhere?

Ricardo A. Medina I have a lot more thoughts about it than I have written down anywhere (yet?). Every once in a while I keep rambling about it here. What I did write down elsewhere is this: https://stefan-lesser.com/2019/12/13/democratize-programming/

Jack Rusher Regarding the thread being depressing, I personally am more hopeful after the realization I made here https://futureofcoding.slack.com/archives/C5T9GPWFL/p1599666723187800?thread_ts=1599588394.135900&cid=C5T9GPWFL That I think a better place to look at how programmers use programming for themselves is to create things to share with others. E.g., it seems to me that things are like personal sites and side projects (sharing) are much more common than using programming to improve personal productivity. Correspondingly, it seems to me that the most fruitful area in end-user programming would likely be in making it as easy as possible for non-programmers to share their creations with others. I think the history of the web also reflects this, with the move towards services that remove the technical hurdles to sharing (e.g., Twitter/Medium, for better or worse) and the current no-code movement.

I am from that antediluvian era as well, my first computer (https://en.wikipedia.org/wiki/Colour_Genie) was so personal that it didn't even have any connection to the outside world other than via physical artefacts (cassette tapes). But I never left the universe of end-user programming. All the software I write is for my own use. Often published for sharing with others, but never written exclusively for that purpose. From this perspective, the evolution of the last 30 years looks like computers becoming ever more powerful and at the same time ever more difficult to use. I spend more time today on administrative overhead (software updates, ...) than I did 30 years ago.

Related to Roben's point, I've always thought it should be easy to share workspace customizations and the like as well as code. Config packages should be easy to create, install, and uninstall.

@Andrew F I think the right way to share workspace customization is as "naked code". Copy-pasting fragments. One link from my OP above is relevant: https://mastodon.social/@akkartik/103994830568601931. My argument goes like this:

• Settings often grow in an unruly fashion compared to codebases. We're more careful about organizing modules in a repo than we are in adding knobs to a config file. Even though the modules are internal details and the config file is externally visible.

• Settings in config files often depend on each other in subtle, hard-to-debug ways.

• Creating packages to manage customizations requires dealing with these dependencies. That creates a lot of bloat to sense lots of different combinations of settings and behave appropriately.

• The bloat hinders further customization. People start to rely more on packages created by others, and the muscle of doing your own customization atrophies.

Bottomline: If you make something look polished, people will assume someone else should make it. If you make it look half-assed, it will encourage, even beg for, helping oneself. This is a case where worse is actually better, not just pragmatically but really.

Konrad Hinsen I, too, still write tools for myself constantly, including little 10-15 minute programs meant to be thrown away. All I could think looking at that thread is that if one's normal toolchain makes producing nonce personal code that difficult, something has gone terribly wrong.

Roben Kleene The first time we used Maria.cloud to teach a class of novices we added a feature where any "cell" in a notebook could be shared as a an "app" -- a view onto the whole notebook via an interactive graphical cell -- with a single mouse click. At the end of the first day everyone had made a game or some art, and they loved sending around their creations. It was great. So, yes, this is definitely something "non-programmers" (rather, not yet programmers!) respond to quite readily.

Aye aye aye, what a cool idea, a social networking platform based around sharing little interactive bits of code. Twitter x CodePen

Kartik Agaram you argument is compelling for text config files as they are today. However, looking to the future, blobs of text config can't be the only way people make customizations. The whole hidden dependency thing especially needs to die (that alone would take you a long way). Basically what I mean by "it should be easy" is that the obstacles in your post should be defeated. :)

Totally agreed. But it's not really about text. In fact, text helps here because it's still easier in 2020 to throw text into a random text box than it is to attach files to it.

The real problem is the indisciplined dependencies, and it's a devilishly subtle problem. For the obstacles to be defeated we need somehow for everybody to do "the right thing" -- even after we figure out the rules of engagement. I don't know how to even start attacking that social-organization problem.

In the meantime, it seems to me the best way to share customization is to encourage sharing what works for you, with the expectation that others will need to tweak it to get it working for themselves. It's fiddly and annoying, but still on balance better than the current world of pervasive learned helplessness.

I don't think the explanation even needs to be specific to programming. There has been a long-term trend towards people doing less and less for themselves, and instead satisfying more and more of their desires through market interactions. Cooking, cleaning, basic repairs, altering clothing, music, story-telling etc are all often outsourced now. End-user programming is swimming against that tide.

Maybe there is inspiration to be found by looking at surviving knots of diy culture, where people are still interested in learning new skills. Eg my old housemate used to hang out at a community carpentry workshop.

Kartik Agaram I disagree about one point: your argument is all about text. In a universe where people share small pieces of code for inspection and adaptation to their own needs, it is crucial to have a representation that is universally supported by communication media, and that means text. I actually guess we all agree that it means text for now, but my claim is that this is not going to change. Text is the dominant mode of time-delayed communication between humans, with pictures taking the second place and everything else (voice recording etc.) lagging far behind. Therefore text is going to be the preferred medium for the kind of interaction you describe for a long time. Perhaps we will develop picture-based communiction with machines to the point that it can complement text, but that's no more than a dream for now.

What made me realize the importance of text-based representations is my recent involvement with Pharo (i.e. Smalltalk). When interacting with a Smalltalk system, text is used only at the most fine-grained level, inside a method. But that means that discussing Smalltalk code by e-mail or in discussion forums is very cumbersome. People end up using constructs that are legal Smalltalk code but which nobody ever uses for code development, such as creating classes by sending a message to another class. This creates a serious disconnect between human-computer interaction and interhuman communication.

I've been trying to keep an open mind and work against my pro-text bias, but I'll be very happy if that's not needed 🙂

Re: DIY's passing, I sometimes wonder whether having an entire industry competing to building seductive simulations of personal development might be interfering with the actual personal development of human beings. Or, to put it another way, how many music lovers have spent more hours playing Garage Band than it would have taken to learn how to play and form an actual garage band?

The killer features of plain text are collaboration and sharing. One pet theory is that one reason for the incredible popularity of visual programming languages practically everywhere except for software development (e.g., https://twitter.com/robenkleene/status/1280182521796399106) is that most those are areas much less dependent on collaboration.

Jack Rusher I saw some back of the envelope math recently - if it takes about 1000 hours to become competent (not expert) at a skill, and the average american watches 4 hours of tv per day, then the opportunity cost of tv is 1.5 skills per year.

Jamie Brandon 1000 hours must be about deliberate practice, which is highly demanding and you can't put too much of it in a day. You can't easily replace 4 hours of watching tv with 4 hours of high-intense practice. But replacing 4 hours of tv with 30 minutes or an hour of practice should be fine.

But on the other hand, I don't believe you need a thousand hours to be competent either. My guess is 100–500 hours depending on the skill.

That should give you 1–3 skills per year. So yeah, tv is huge waste either way.

what a cool idea, a social networking platform based around sharing little interactive bits of code. Twitter x CodePen

You mean like http://www.bashoneliners.com?

Jamie Brandon It's funny I was actually talking about this with a few friends and we sort of all had the opposite experience. Not sure if we're part of the same generation. We found that the generation of our parents (born in the 50s early 60s) didn't have a DIY culture at all.

From real experience:

• If you're bike is broken :Parents => bring it to the bike shop

Us => Watch a youtube tutorial

• You want to organize a trip:Parent => Contact a local travel agent

Us => Do some research, go on Sky Scanner, try to find local guides, http://booking.com...

I was also talking to my step dad about it, who was also leaning to the same conclusion, He was amazed that kids of our generation all seem to cook. He is a city guy from Italy, he never cooked for himself. Being a bit counter-cultural he associated cooking yourself with the older more repressive generation (he was a bit of a hippy). And maybe we are doing the same things he did, we cook because our parents didn't.

I think there might be some vertical and also horizontal cultural differences. Most of my friends come from cities/big towns and we are Europeans. There might be different shift depending on your generation/location.

The closest thing to what I think you're asking about is using Logic Programming to do program synthesis. Here's a video demoing a system of this kind:

https://www.youtube.com/watch?v=5vtC7WEN76w

At risk of coming up as very naive: what about tests on the practical side and formal verification on the other side? Since those are allowing us to refactor code while ensuring (to varying degree) its correctnes and behaviour, how do those fit into what you are looking for?

I wonder how different proofs for the same theorem compare after resolving all abstractions and breaking them down to ZF. But I guess even proofs using the minimal amount of axioms aren't unambiguous, as two different axioms could share a common idea.

However, it is undecidable whether a set of axioms contains redundant axioms.

In general, deciding whether two programs are semantically equivalent is undecidable too. So there cannot be a universal tool that enumerates all alternative function bodys with the same behavior.

When dealing with undecidability, it's best to say, "well, we'll just handle the easy cases," and see how far you go. Compiler optimizations are certainly an exercise in factoring as is partial evaluation, which is a good deal more fun. Come to think of it, conventional refactoring is sort of the opposite transformation. Instead of removing indirection, add it so that the rest is more regular.

@Martin Sosic I appreciate the answer. Talking about testing is more concrete than talking about an abstract idea of behavior, and as you said the practice of testing is about manipulating code so the same tests (at least) still pass. I guess the picture I have in my mind is that when you are coding you are moving around a space of strings of symbols, and in that huge space is a space of valid programs, and once you write tests, then there is an even smaller space of programs that pass those tests. Is there any means of restricting code rewrites to discrete steps that take place entirely in the space of programs that pass the tests? Rewriting for-loops as while-loops wouldn't change any tests. Changing some variable names. Refactoring. These wouldn't change the tests at all, and in a specific context you could possibly have more. If you have various pieces of code that are interchangeable then you have something like an algebra where you can substitute equivalent expressions. Tests tell you when code is interchangeable so it gives you some algebra-like thing and how can you manipulate code at this level?

So tests are relevant for what I'm looking for because they let you say when two programs are equivalent. I don't think proof verification does the same in math.

@U01AN8DFFBN "In general, deciding whether two programs are semantically equivalent is undecidable too. So there cannot be a universal tool that enumerates all alternative function bodys with the same behavior."

Neat! Good to know there isn't a universal tool.

@William Taysom Easy cases is right! I haven't seen compiler optimizations before. I might take a peek into that area, but I have a feeling I might get scarred.

I agree with @William Taysom. This very much reminds me of compiler optimization.

Classical compiler optimization makes use of many transformations that maintain equivalence such as https://en.wikipedia.org/wiki/Static_single_assignment_form form. These sorts of transformations combined with heuristic based analysis are the bread and butter of compiler optimization. The GHC Haskell compiler is probably the epitome of the classical compiler optimization approach.

Then there are the recent results in applying Machine Learning and Genetic algorithms to compiler optimization. These compilers often produce extremely novel, unexpected, even bizarre, machine code, and yet the results have been shown to be "equivalent", and yet much more efficient in some dimension than known state of the art classical approaches (often either better runtime or less resulting machine code.) AFAIK, the way the ML compilers seems to work today is essentially unit testing and manual inspection against programs compiled with classical compilers.

https://arxiv.org/pdf/1805.03441.pdf

https://www.semanticscholar.org/paper/Compiler-Optimization%3A-A-Genetic-Algorithm-Approach-Ballal-Sarojadevi/6676a5489ced5412fa2ba3ecb76ca3e5ca2723e0

A key idea is that all these algorithms must have some notion of equivalence of programs. An optimization is only useful if the resulting behavior is equivalent.

The real heart of this is the Church-Turing thesis. Program optimization is essentially saying one program is equivalent to another program, but just computed in a different way. The fact that such equivalent programs can even exist is a direct result the Church-Turing thesis.

Your line of reasoning is related to some of the inspiration of the creators of logic programming and proof assistants like Coq and Agda. If you can encode a proof in a programming language, you have essentially shown that the proof is computable. If it's computable, then there are infinite equivalent programs (via Church-Turing). You can then apply all the known computable transformations like SSA, or graph pruning analysis, etc...

Wow, GCC flags Ray Imber? I do like the idea of ML being applied at high level "moves" than it often is. I mean, for instance, allow for only valid application of inference rules rather than random term rewrites. Let the system play over that space.

Nick Smith I think logic programming indeed began as identifying itself as a programming language: http://www-public.imtbs-tsp.eu/~gibson/Teaching/Teaching-ReadingMaterial/Kowalski74.pdf

The extant thread of how this plays out I think is in the work of John Reynolds in the form of logic relations right now. But the present day incarnations that have the major mindshare happens to be Prolog variety of languages and I think its a fertile ground for bringing in novel work with it.

http://akkartik.name/post/readable-bad

Oh, it's a local vs global argument? Interesting. The point being that locally readable code is not the bottle neck, but globally "readable" (comprehensible?) code is.

Locally-readable code is definitely a bottleneck. I dare you to try and tell my students otherwise after they’ve spent 12 weeks trying to learn Python fundamentals :/

But anyone defining “readability” as merely naming plus commenting is missing the obvious: the actual language that those names and comments are embedded within! The grammar and semantics of a language matters at least as much as the names that are used within it.

But yes, large-scale understanding is a separate issue requiring its own solutions.

global readability is a by product of global consensus. It is harder now days to achieve global consensus, because the skill and platform driving teams to be more and more narrowly focused. When everybody has ideas of how to organize, global readability is hard to achieve.

Code is pretty low level: it's hard to get an overview of it. Low level consistency, in naming for instance, will give you insight into what's going on, and I think more importantly, those who write locally readable code are also more likely have a good architecture.

I would say observability is equally important. It's nice and all if the code is readable, but many times the code goes slightly out of date with it's names. And, the way it works makes a lot more sense when you have good tools for inspection into what's going on or what happened.

I don’t know much about this and certainly a lot less than you do, but I am super interested in this and really appreciate you writing about it and putting together all these resources. So here’s my feedback: thank you and please keep going! :)

Thanks a lot for the encouragement! I just bumped into this after exploring random stuff related to PLT. A certain kind of recurrent idea here seems to me that there is something fundamental about computational structures (in a topology like sense) that we are all groping at in our daily lives.

Programming seems to me now to be an activity in creating some kind of <hypothesis>semiotic ground for manipulating Platonic forms</hypothesis>.

I am trying to see if those things can be made first class in a visual fashion and PLT semantics looked to me like the place where I can dig in as I have some idea of PLs and a bit of logic knowledge. Also, there are a great community of people studying this deeply and communicating about it in this field. But it is filled up with waaaay too much jargon to repel most programming practitioners as the notations are dense and require an understanding of mathematics before approaching it.

This sort of aligns with a direction of using computation graph as a first class entity which I think some people here and in Twitter is about to embark on. This can be seen in Twitter convos in the form of RoamOS or as Codex lately proposed: https://twitter.com/codexeditor/status/1303985191912747009

This would result in a computational would of structure, I think, close to the attached image. Where each context has a program inside it like in a graph editing tool which then becomes the ground for further computation. Think of modules referring each other and creating intermediate representations of computations in this stage, just before getting visualized out as UI elements in a pixel matrix.

But I think the links go deeper, and (un)fortunately semanticists, philosophers, and mathematicians working on specialized areas seems to be the ones with intimations of what these general structures could be like. Here is another image from Rocco Gangle’s work (He mixes Category Theory with philosophy to propose some kind of “diagrammatic immanence”). This but for programming is what I think is lurking inside the morphisms of control constructs.

Yes! I understand maybe half of what you’re saying, but that half feels true and important to me. I’ve been looking at linguistics, category theory, bidirectional transformations, parsers/grammars, complexity theory, and Christopher Alexander’s work on Wholeness/Life and his generative process of unfolding which I just recently started to understand as having parallels to Chomsky and grammars (search for Greg Bryant, if that sounds interesting).

All these things are very different disciplines but somehow I started to see strong parallels and connections which I totally believe can be expressed mathematically. And very likely we have already identified all the structures we need, we just need to make the connections explicit. Unfortunately, I often find it hard to describe and it makes total sense in my head, but then I’m struggling explaining it to others.

It looks like you’re onto something that ties into this and might help me find ways to explain some of these connections better.

And that’s not even touching on all the exciting ways this might help us find better visual representations for programming systems!

Exactly my thoughts! You put this better than I did. Christopher Alexander’s work was a big influence and he is a strong center of this kind of interdisciplinary thought. Much respect to him for his work. I need to look up Greg Bryant, thanks for the pointer!

I am working on and off on my mathematical skills to see how I can represent these ideas well. This is why I started the research on notation to see how historically they have helped us in expressing that other realm that we have inside us: https://github.com/prathyvsh/notation I am pursuing it under a feel that there is a good ROI from having a good fitness between the content/context relationship for the constructs you device to ground these forms.

Will definitely share when I have some clarity on this. Thanks for the good words ✌

Exciting. I’ll keep following you here and on Twitter.

Re Greg Bryant start here: https://www.youtube.com/watch?v=X-5KG73fzJ4

Then read this whole blog in chronological order (it’s not as much content as it seems): https://chomskyalexander.blogspot.com/?m=1

And feel free to ping me on anything related to this. I still have to dig through all what you posted but maybe there are chances to loosely collaborate.

i’m going to have to give this sincere focused time, but i wanted to reply before my initial reaction before i lost the references—i’ll come by and fill these out with hyperlinks later. i too feel like all of this programming language research is swimming in a dark cave, and in our hunting around i’ve got this suspicion that there’s a light switch not too far away. and that it has a lot to do with reifying the process of evaluation and letting users interact with it directly. in particular,

...there is something fundamental about computational structures (in a topology like sense) that we are all groping at in our daily lives...

I am trying to see if those things can be made first class in a visual fashion and PLT semantics looked to me like the place where I can dig in as I have some idea of PLs and a bit of logic knowledge. Also, there are a great community of people studying this deeply and communicating about it in this field. But it is filled up with waaaay too much jargon to repel most programming practitioners as the notations are dense and require an understanding of mathematics before approaching it.

and

All these things are very different disciplines but somehow I started to see strong parallels and connections which I totally believe can be expressed mathematically. And very likely we have already identified all the structures we need, we just need to make the connections explicit. Unfortunately, I often find it hard to describe and it makes total sense in my head, but then I’m struggling explaining it to others.

resonate really strongly with me—better-worded versions of stuff i’d tried to explain to other people in the past

the aforementioned references im working with off the top of my head:

• call by push value
• ohm/ometa
• kernel/vau/f-expressions
• partial evaluation
• programming should eat itself
• the work going on at red planet labs, alluded to by a few talks on the specter library for clojure
• meander, another clojure library
• the stuff rich hickey has started saying about functions having some knowledge about what they require to work properly (instead of specifying requirements on data structures directly)
• towers of interpreters
• f-algebras, recursion schemes
• scoping and binding in the rebol programming language, APL
• defunctionalization and refunctionalization
• concatenative languages, because: the current continuation for them is always the state of the stack plus the rest of the tokens in the source, so you can always split a program at any point, pause it and restart it, and composition instead of application is the default action of putting two words next to each other. another way of looking at it is that every concatenative program always carries with it the context it’s operating in/on. plus there’s this cool video i shared before: https://youtu.be/R3MNcA2dpts

Here’s something I’ve been looking into for a while that I didn’t see you mention specifically: the connection between

• (embedded) domain specific languages
• parser combinators
• abstract algebra
• transducers (you kind of mention that with generators/iterators, I think)
• transformation passes (towers of interpreters?)
• …it appears I’m going to list all of computer science if I keep going, but there are certainly some more items that belong in that list

There are quite obvious connections between some of them, for instance parser combinators are more or less directly applied abstract algebra. But there seems to be a more fundamental pattern that is reflected in all of them (and here we get to the part where I’m usually failing at describing it well enough — probably due to my lack of depth in mathematical understanding):

• They all involve (or at least can facilitate) transformations from a sequential to hierarchical structure.
• They all represent a set of well-defined composable entities that together form something like a grammar (some more directly than others).
• They all in a sense resemble words of a language, which can be combined to describe a lower level thing in more detail (-> Alexander’s Pattern Language).
• They all enable and/or are based on a fundamentally recursive pattern which allows them to be used on various levels of abstraction at the same time; they’re something like an abstraction of an abstraction, if that makes sense?

I mean, maybe I’m just looking at lambda calculus shining through in all of them (and in all of computing), and that’s that — and there’s nothing more to see here. But I don’t think that’s it.

I know, this is kind of weird, but maybe to some of you what I just wrote makes somewhat sense and you will have some comments that help me on the path to computational enlightenment… ;-)

Garth Goldwater pointed this out to me in a recent conversation: continuations could be seen as the same as monads. I dug up this little discussion with more discussion around this idea https://stackoverflow.com/questions/4525919/continuation-passing-style-vs-monads

https://www.schoolofhaskell.com/school/to-infinity-and-beyond/pick-of-the-week/the-mother-of-all-monads here’s the article i was referencing in that conversation

Semi relevant article https://medium.com/@old_sound/programming-languages-are-not-languages-c6f161a78c44

I suppose I'd it's important for comprehension, then the programming language environment might need to be more tolerant to potential error, right? For example if someone says "with each user online in slack, I want to ask them when their birthday is" I'd love for the computer to be able to help incrementally qualify that for us even if there are subtle "grammar fixes" that need to be applied. Then, programming with correct syntax would be a bit more similar to writing correct English with Grammarly.

Even for the professional programmer, a system that comes up with "but have you considered" questions could prove fairly useful. How many bugs are caused by odd interactions from otherwise fairly independent subsystems?

Though it would be interesting to consider talking to the computer and having it translate. I was thinking more about just a way to converse with other developers. The lack of humanity in the development process seems to lead to a lot of stress. People enjoy interacting with other people.

I do this now in clojure, but I'm curious if we developed shared rules around the speaking patterns.

This comes up a lot in mob programming. I'd love research in what gets named and if there are patterns for it. The GoF book also stresses the reason for naming the patterns and the follow up refactoring books use the new vocab with some additional verbs.

Also, programming languages hit the same areas in the brain as everyday languages... But the ambiguity restriction is very different (due to shared context)

Aha, indeed, I misunderstood the original idea.

It's tough to be able to verbally communicate anything involving multiple actors (DB, Load balancer, Http endpoints; Text buffer, RegExp, validation messages; etc.) That's why when we can't communicate it by writing the code, I tend to try drawing it out and annotating with different levels of annotation. So, if we were communicating how notifications work, I would start by writing out the high level actors involved in sequence of a notification being sent. Then, I'd incrementally add notes for where our business requirements are applied. Then, I might show where the data is stored, etc.

In this situation it is incredibly difficult to share knowledge only orally. It is far easier to write it out in a shared code editor, or through sending back and forth snippets, than it is to try to hold all the info in your brain as the conversation moves forward.

In some of the most complex topics, my peers and I will actually write out the questions we're asking in comments in the shared editor, so we can encode each other's thoughts carefully to save time from having to repeat ourselves and losing context.

It kinda sounds like as if a math teacher quizzed students orally without writing out the problem on the board. It would lead to too much repetition and clarifications, just so the student is able to write out the problem the teacher is asking...

would be really interested to know if there is any MRI research looking at how much overlap there is in which parts of the brain "light up" when speaking/writing/read code vs natural language

i think @Cole Lawrence is pointing at something important here, which is that the main affordance given by non-spoken language is asynchrony—being able to look at things in different orders, return to old topics, and read without listening or speaking. i think the idea of spoken language (and an interactive conversation) also probably have unexpected side benefits, but i’m struggling to think of them at the moment (probably because i don’t use spoken interfaces very often). the only one that comes to mind is that spoken interactions imply an awareness of context—think of how much more often you get to say things like “what’s this?” out loud rather than when you’re writing