What if everything came with batteries included? (eg. in a course, each word/term/concept/technique/obvious prerequisite, was clickable to show a similar explainer, thus limiting total prerequisites to the foundations of the english language?) (+ eg. docker/vm's for software)?

Interesting how almost everything fades/data-rot. Longevity requires continuous upkeep?

If the "batteries"/dependencies were built on a common foundation, the upkeep would be a collective effort that would benefit all. Eg. files: One org creates mapping from new dir-less/search-full system, to old directory structures, and the explainer becomes available for all?

Though that merely ensures a continuous thread to the current environment, but doesn't translate the course natively into the new age. Though better old and rusty than unresurrect-ably dead?

Re UI/data-shift: "who uses computer os:es nowadays, when you can do everything on your phone/browser?", "files is a concept of the past"; data should be where and when you need it; data doesn't belong to one specific file, to one specific directory - it may have multiple equally important places of relevance.

I struggle to believe this article isn't exaggerating the issue. I can accept that they might not have encountered the concept of a hierarchical filesystem on a computer but I don't accept that it's difficult to understand once explained.

It's a highly skeuomorphic design that is basically exactly analogous to boxes within boxes.

Are you saying that people genuinely struggled with that Nick?

They're getting used to it fairly quickly, but also I polled the class and most of them have still encountered a desktop UI before a phone UI. I don't think they're quite young enough to be the students in this article, where phone UI is the default and desktop UI a deviation from it

Also we're still in a situation where writing code pretty much still requires a desktop UI, with the important exception of JS livecoding sites a la Glitch

I feel like the pieces of the new paradigm are out there--maybe you write some JS in a browser tab over your low-latency 5G connection, publish it permanently for a one-time fee on IPFS, it runs brilliantly on a super-optimized ARM CPU and WebGPU

But that's going to take a while to gel

The phone / non-hierarchical UI and systems are designed for a world of constant connection. Maybe they are just better suited to that world, not sure

However, given what we know about the state of the world, do we expect an environment of constant connection to be the norm in the future?

If you want to use computing for something, and you have been acclimated to systems that assume constant connection, you are going to be totally screwed if the internet goes out

This is just amateur speculation about telecom vs. energy infrastructure, but...I imagine that if we got to widespread decentralized renewable power generation, the rosier edge-computing scenarios can basically free-ride on top of that

And if we don't get to that, then long term we're not gonna be doing much local computing anyway, outside of possibly a few fortress city-states far above sea level. (I've heard a convincing fan theory that the Capital of the Hunger Games is canonically Denver: a city high in the mountains defended by an airbase and ICBMs.)

It's borderline humorous/parody.

Though in my life, 90% of computers I've seen had disaster zone desktops. Files were probably too successful for everyone's good.

I play with the notion of "good riddance" – yes, files are as easy to teach as "see all those different photos in your photo app?" And, for me, files are little miracles.

But what if it's just better and sufficient to teach the skill of getting something onto your clipboard... Find the thing you want to work on, do what it takes to make it paste-able (or whatever), then open the place where you'd like to work on it.

Maybe work contexts shouldn't be file based.

@Nick Fox-Gieg There are many possible scenarios, I guess, but we seem to still be on track for the worst of them as of this writing

It's kind of hard to imagine The Internet balkanizing, or being widely unavailable

But then again in the 1950s it was probably hard to imagine not getting clean drinking water in the Greater Detroit area

Anyway what's needed are better computing metaphors. Files and folders were pretty good, but are perhaps insufficient at the volumes we deal with today

Of course the "Google" paradigm is to have no metaphors at all, which I think it a mistake. It does not lead to sophisticated end users

I mean I'm a pessimist, and I agree that North America specifically seems to be taking the Tokugawa Japan route of effectively banning new technology that threatens to alter the balance of power in local politics, at the cost of losing their advantage vs. external rivals

Christopher Galtenberg Yeah I think when iOS pioneered this approach, they demonstrated that most people are fine with a handful of silo'd databases for emails, pictures, and songs.

What I've learned from r/talesfromtechsupport is that there exist people who fundamentally cannot grok file hierarchies. Maybe it's an upbringing thing but some people apparently just can't do abstraction at all. I also learned that there are people who store things in the Recycle Bin and are confused and dismayed when it disappears.

So I find it easy to believe students are having trouble with it. File hierarchies are fundamentally a recursive concept, and those have a long track record of being tricky. I bet hands-on experience is the only way to really learn.

I do kind of like the sheer John Henry badassery of storing files in the Recycle Bin. Why design around a user story when you can have a user tall tale

Do data scientists use inferential statistics such as t-tests, ANOVAs (and related stats like p-values and confidence intervals)?

Yes

If so, is this done to compliment ML approaches?

Yes, but also a lot of people with that title are essentially just analysts, so they use them the normal way. So data scientists themselves are split between the ‘explanation’ and (ML-heavy) ‘prediction’ camps.

I think “data scientist” is a pretty roughly defined term and it’s best not to take it too seriously.

hey Allan, I’m knee deep in the data world so I can provide some color here!

• The higher the ‘stakes’ the more important the need for statistics and deeper modern understanding. Industries like insurance DO use things like machine learning, but statistics helps them actually better stay compliant (e.g. you REALLY need to make sure you aren’t discriminating on things like race and sex). But hey if you’re just predicting which vacation rental someone is more likely to rent, lower stakes (and revenue, A/B testing, etc can be used as a ‘process’ to improve the system thru space and time)

2. Statistics is especially used in controlled experiments. Things like A/B tests etc. ML is often used when you maybe can’t do that type of testing or the stakes aren’t that high

3. Definitely agree with Luke that “data scientist” is a very broad / poor name for a job. I have 2 mind about this. In one sense, better specialization and job titles makes roles clearer. On the other hand, we’ve seen what happens when people think about data in TOO narrow of a function (’just throw ML at it! YOLO!’) and there’s something to be said about full stack data people (even if they’re specialized in 1 or 2 things they should deeply understand the rest of the disciplines and the system I feel). But besides companies that are doing title inflation (https://medium.com/@chamandy/whats-in-a-name-ce42f419d16c#:~:text=At%20Lyft%2C%20we're%20rebranding,Science! for the sake of attracting candidates), usually Analysts are focused on understanding ‘the past’ / the current while ML Engineers, Data Scientists, etc are focused on trying to predict the future, provide more holistic / strategic advice, etc

Thanks a lot @Sophie Smithburg for this detailed explanation. No surprise so far - I think I have been led astray by focusing on "full-time", as opposed to for example "half-time". But that isn't the issue, it's really employment.

yea, i was using it as a shorthand for a specific employment arrangement

FTE is the shorthand for "employee" in the googleverse, and i still am stuck with a lot of that lingo even after some months away

This was a very thoughtful rant, Sophie. I was cheering along with:

with disagreements about such fundamental compound datatypes, the very idea of software correctness looks questionable at its philosophical foundations. correct for who? for what purpose? by what values?

I just finished an interview with Scott Anderson for the podcast, and one of the things we talked about (spoilers, haha) was the ethical / social / power dynamics in the emerging VR platforms.

This whole area weighs heavily on my mind. I really appreciate you bringing this up here — and I wish I had more to contribute, other than to say "this is important", but.. I'm really not sure what to do next.

I think by aligning distributed governance with distributed ontology development we can perhaps make substantial progress on solving wicked problems and making steady progress around essentially contested concepts.

Can you elaborate on "distributed ontology development"?

Thanks for the term “wicked problem,” that was new to me

just that ontologies are good for standardizing machine readable versions of social concepts, which all essentially contested ones will always be. by distributed development i just mean your standard open source collaboration.

I think this problem is much more endemic than just in tech. The infrastructure for cooperatively disagreeing at the societal level just does not exist. Bret Victor talks about model-driven discussion quite a bit (an example: http://worrydream.com/#!/ClimateChange ) which would be a significant step up in how we do things.

I agree about ontologies being useful, rather than the oft repeated maxim that systems should be value-neutral, I think they should be value-flexible. And more importantly, that flexibility should be usable at a local level, at the level of individual users and groups.

We are dealing with large problems of scale in society now, the size of society is larger than the buffers of the environment it operates in, and it is vital that we find more efficient, and humane ways to communicate and coordinate, because the fallback is always a boot to the head (sorry, couldn't resist), since the most efficient form of communication is no communication, and the most efficient form of coordination is top-down command. But of course neither is at all humane. So, we are under a timer to solve these issues.

I look forward to a future where I can talk to someone, have a debate based on actual data, with all the provenance information available, without having to think about all the technology that ties it all together. But this is a political problem as well, of course, even more than it is a technological one.

Just want to say that this conversation gets to the heart of so many things, important, persistent problems. I like S.M Mukarram Nainar’s comment about "value-flexible." There isn't going to be one source of truth. So let's accept that. And how much truth do we need? For the task at hand, is name relevant? Make it more operational, "name on birth certificate" or "name as printed on passport." (I feel this being in the process of immigrating. Oh man, I have a son who may have three last names depending, and my wife who has at least four first names depending. And, come to think of it, I have two entirely different full names.)

Ontologies, there's a word don't hear much. (I was adjacent to OWL back in the day.) These days though, I'm pleased when I see people using RSS rather than whatever siloed "top-down" app-driven garden walling fiefdom building technique is dreamed up.

Lastly, it's interesting to scale down from the disfunction of organizations to the disfunction of individuals: even when we're well rested, actively trying to communicate (rather than self-deceive), and have aligned goals — it can be tricky to get a handful, two, even one person pointing in a useful direction. (Think of a change to some coding convention, say the lifecycle of model objects in your software.) So I guess as a prerequisite, a position of patience without complacency is where I have to start when surveying coordination problems.

I really liked following the wiki trail that started with wicked problems, then led to contested concepts, and then to contestable concepts in a shared ethical framework. Basically, if I understand correctly, if we can find common goals, we can stop the attempt to resolve a disagreement with logic, and consider tactics

Value-flexible is a nice concept. We also need value-explicit. Much technology enforces values that its designers would perhaps be ashamed to discuss in the open. Political debate should be both model-driven (for the non-social aspects) and value-driven (for the social aspects).

There is one more aspect to political debates, which are related to technology as well but I don't see yet how they could be taken into account. In addition to having different values, and different information/knowledge about the outside world, people differ in attitude to risk-taking, novelty, etc.

Konrad Hinsen I like value-explicit. Reminds me of a great talk by Bryan Cantrill: https://www.youtube.com/watch?v=9QMGAtxUlAc

As for attitudes for risk-taking, etc, that is a good point. I think it indicates that a polyculture of communication/coordination methods are necessary. While there will always be failures in effective communication, I don't think it's the duty of us as toolmakers to design systems from the top-down to fix these problems entirely. But I do think we should give people the tools to communicate as they see fit. That said, I'd be interested to see if any research in psychology, economics, or something has some information about what the distributions for these things are emperically.

We just had an election where I live, so I've been thinking about this a bit and had this idea.

Political debates could be long-form with citations, on a mailing list or something similar, and happen asynchronously, which I think would work much better for actually debating policy. That said, some people just don't like reading long-form, so they could be summarized by third-parties (the moderating party, I guess) afterwards.

Finally, I think we should recognize one of the key types of tools people use and need are mental tools. We talk a lot about tools for thought, but I think tools of thought are sorely neglected. By which I mean education. We should seriously consider educational reform as something to be designed for. Not only can education change the energy barriers around communication, it can also change the meta aspects, such as effective risk tolerance via reducing uncertainty about the world, etc. The principle I have in mind is that access to information should be accompanied by the tools to interpret and understand it. Or more generally, that new capabilities should be accompanied by the capabilities necessary to regulate them. One innocent example is office tools. Stuff like Microsoft Word has awful defaults, and doesn't even try to provide some knowledge of typography to their users. The end result is everyone suffering awful design everywhere, which can actually have practical results—improper typography can be and often is hard to read, and likely wastes lots of time on the scale of society as a whole.

S.M Mukarram Nainar Your comment makes me wonder if wanting to fix problems is perhaps the ultimate problem with techies. It's of course a very good attitude for small-scale problems that you can fully comprehend. It's also a reasonable attitude for your own immediate problems. But on a larger scale, problem solving requires participation of all stakeholders, and tools that help people see, understand, and fix problems collectively, rather than being themselves the fix.

so, my technique is roughly this:

• use the AST as a guide for what to model. i'm using whitequark's parser gem for now because it's more at the level of abstraction users care about than the RubyVM::AST module, and i'm aiming for usability over exhaustiveness rn
• assume primitive datatypes exist and focus on modeling their semantics first, so, dual prioritization. AST nodes so we can tie things back to source locations trivially, and primitive datatypes because in dynamically typed languages with a propensity for duck typing, exhaustive exploration of the semantics of an AST node is intractable
• pick out examples from rubyspec for that AST node that imply some semantics
• either manually or by way of retooling the runner (more ideal) to do so automatically; symbolize the critical inputs
• take inputs and outputs over to z3 and find the implied minimal semantics (this part isn't in ruby rn bcz the best z3 gem is broken)
• manually, or with some tooling (obvs ideal) refine the implied minimal semantics. an example is that z3 couldn't infer boolean or semantics in a compact form at first, because it couldn't do some basic boolean rewrites that i expect something like coq etc would've been able to do. or at least, it can't do it as best i can tell yet.
• test the semantics against symbolic execution of MRI/etc using symbolic execution engines targeted at lower level semantics. probably going to use triton for this? maybe klee. depends. triton would be more portable to other implementations. s2e is an option too, but triton really feels easiest atm.
• then run the semantics against rubyspec to do a final test.

the end goal is to be able to infer semantics for all dynamic languages somewhat efficiently.

@William Taysom since you asked about this

(wall of text is A-ok)

In #thinking-together we're here for WoT 👏

Interesting stuff! I've been interested in using abstract interpretation for improving the programming experience (offload mental simulation, etc.). This seems to overlap, though I'm not sure I quite understand the difference between symbolic execution and abstract interpretation.

the end goal is to be able to infer semantics for all dynamic languages somewhat efficiently.

Is there a larger goal in mind here once you can infer these semantics?

yes, with said semantics in a separation logic friendly form, we can bring provable systems methods into the realm of dynamic languages with high confidence, i hope. also things like extracting TLA+ specs from implementations by way of rewriting/interactively abstracting with automated assistance away aspects of the separation logic spec of your program

abstract interpretation and symbolic execution aren't really fundamentally distinct, but i found one approach much easier to self teach, and also it's just easier to apply symbolic execution directly when there's no language spec, at least if you want to machine verify your spec.

at least as far as i could learn so far, that is

there are ways to exchange information between the abstract interpretation and symbolic execution, at least theoretically, but i don't know the details at all, and i'm not aware of any research exploring that in particular.

AI emphasizes AST and lattices under the hood more whereas SE emphasis bytecode or instruction level semantics a bit more, while representing things in a first order logic generally.

(technically i'm also drawing on counter example guided abstraction and there are other niches/terms some may not consider strictly SE that i'm including in it, such as dynamic symbolic execution, concolic testing, dynamic binary analysis. just in case anyone wants to follow up on any of this.)

This is way, way cool. And for "the end goal is to be able to infer semantics for all dynamic languages," Ruby is a great place to start since its semantics are nuts. (I say this as a proud Rubyist since the turn of the millennium.) Ruby is going to keep you honest. I mean the very fact that people are productive in Ruby shows that one can model its semantics "isomorphic up to shenanigans," but my, my what shenanigans are available!

Let me share a little bit about my interests adjacent to this: (1) treating execution traces as first-class objects for debugging, (2) refactoring.

(1) When dealing with a relatively big framework ( let's call it Rails), its nice being able to see how all the bits are dynamically connected. And one step removed form the execution itself, it's useful to ask the what-if at specific branches. This leads to...

(2) The most common useful refactor I see in practice is the business logic tangle of conditionals where a complicated joint space (let's say six booleans and four variables depending on them, so 256 cases to consider) can probably be factored in a reasonable way, but I'd really love tool support for navigating that space, making sure all the cases are accounted for.