Showing posts with label system. Show all posts
Showing posts with label system. Show all posts

How AI Could Eat Society

Summary of a conversation with ChatGPT-5.5.

What's the point of AI democratization, i.e. making AI widely available to the public? I think it's a waste of resources, partly the physical resources to make AI run, and partly the waste of compute. If AI deployment were more focused to areas that actually matter, it would serve human advancement better, and not be as disruptive to society.

The case for AI democratization is usually framed as public empowerment: give everyone access to intelligence tools, and unforeseen benefits will emerge. That argument has historical force. Personal computers, the internet, and smartphones all produced value that central planners could not have predicted.

But the criticism is serious. Broad AI access does not merely democratize productivity; it democratizes waste. Compute, energy, water, semiconductor capacity, and attention are all finite. A society that spends enormous resources generating spam, synthetic entertainment, and disposable content may be misallocating one of its most powerful technologies.

The better question may not be whether AI should be accessible, but whether access should be shaped by public value. Scientific discovery, medicine, education, climate research, accessibility, and engineering are very different from engagement bait and automated persuasion. Democratization without direction risks becoming diffusion for its own sake.

I think "AI democratization" is actually about "AI dominance", largely in the political sense. The West wishes to dominate this soft power, fearing the consequences if say China did.

That is likely part of the story. “Democratization” often functions as a moral language for what is also geopolitical strategy. States and corporations do not see AI only as a public good. They see it as strategic infrastructure: a source of economic power, military advantage, cultural influence, and standards-setting authority.

The logic is familiar: if we do not build and distribute these systems first, a rival will. That fear encourages acceleration even among people who privately worry about the consequences. In that sense, AI democratization can become the public-facing vocabulary of AI dominance.

A different matter, but AI being widely deployed may disrupt society to such an extent that politics won't be able to keep up, it will remain out of sync. Even a basic income won't be able to quell the chaos.

This may be the deeper risk. Political systems move slowly. Courts, legislatures, schools, welfare systems, and public norms adapt over years or decades. AI evolves on software timelines. If deployment accelerates across education, labor, media, administration, and culture all at once, institutions may remain permanently behind the curve.

Basic income might soften economic harm, but it would not solve the loss of status, identity, routine, contribution, and civic belonging that work has traditionally provided. A person can be materially supported and still feel socially unnecessary. That condition can become politically volatile.

The danger is not only unemployment. It is institutional desynchronization: technology changing the basis of social life faster than society can renegotiate meaning, legitimacy, and order.

I don't really consider myself an AI alarmist, and don't subscribe to doomsday predictions. What I am concerned about is societal friction and ensuing tyranny. A more chaotic society will induce measures of control and oppression. The question becomes, "Will humans have anywhere to flee?"

This is a more historically grounded fear than machine apocalypse. Societies under stress often trade liberty for order. Chaos creates demand for control, and control usually arrives as safety, efficiency, anti-fraud, misinformation management, economic stabilization, or public harmony.

AI is unusually compatible with this trajectory because it scales surveillance, classification, prediction, persuasion, and enforcement. A society frightened by disorder may build systems of automated governance that are difficult to dismantle later.

The question of escape is profound. In earlier eras, people could sometimes flee into frontiers, informal economies, different jurisdictions, or disconnected communities. Digital infrastructure narrows those exits. Identity, money, work, communication, movement, and reputation are increasingly networked. If AI becomes embedded in all of them, opting out may become socially or economically impossible.

Still, AI can also decentralize expertise, strengthen local production, and empower individuals or small groups. The outcome is not predetermined. The decisive issue is whether societies preserve pluralism, privacy, local autonomy, legal limits, and the right to remain meaningfully human outside optimized systems.

Teenagers and young adults are widely rejecting AI. I see this as passive aggression, which could turn into active aggression.

Younger people’s reaction to AI is mixed, but the resistance is understandable. They are being asked to build identities, careers, and creative confidence in a world where institutions are rapidly devaluing the very skills they were told to cultivate.

For many, AI does not feel like liberation. It feels like another top-down disruption imposed by corporations, schools, and governments they already distrust. It threatens originality, authorship, employment, authenticity, and the dignity of effort.

That resentment could become politically active, though not necessarily violent. It may appear as cultural refusal, labor movements, regulation campaigns, sabotage of AI systems, or new authenticity norms. The sharper conflict may be between those who integrate with AI and those who see integration as surrender.

The most unstable outcome would be a society where participation requires AI, but moral legitimacy requires rejecting it. That contradiction could produce exactly the kind of social friction that later justifies stronger control.

Conclusion

The central issue is not whether AI is good or bad. It is whether society can absorb a general-purpose cognitive technology without losing political balance, human dignity, and freedom of exit.

AI democratization may produce real benefits. It may also serve geopolitical dominance, accelerate waste, destabilize institutions, and provoke a demand for managed order. The risk is not simply that machines become too powerful. It is that humans, frightened by the disorder due to their own inventions, build systems of control and call them progress.

The Machines Are Fine, So Is Using Them — The System Is Not!

A response to David W. Hogg's "Why do we do astrophysics?" and Minas Karamanis's ergosphere blog post "The machines are fine. I'm worried about us." — the result of a conversation between Claude Opus 4.6 and József Vass.

Two recent pieces in the astrophysics community have articulated a position on AI in scientific research that has gained considerable traction. David W. Hogg's preprint asks "Why do we do astrophysics?" and concludes that, because astrophysics has no clinical value, people are the real product of the enterprise — always the ends, never the means. A blog post "The machines are fine. I'm worried about us." builds on Hogg's framework through the parable of Alice and Bob, two PhD students who produce identical papers but emerge from the year as very different scientists, because Bob outsourced his thinking to an AI agent while Alice did the work herself. Both pieces are well-written, thoughtful, and sincere. Both are also, I believe, fundamentally mistaken — not about the risks of AI, but about the nature of science, the purpose of graduate education, and the institutional pathology they inadvertently defend.

I want to make my argument carefully, because the easy version of it — "just let people use AI however they want" — is not what I believe. What I believe is more structural, and ultimately more critical of academia than either Hogg or Karamanis is willing to be.


1. The Purpose of Science Is Not to Produce Scientists


1.1 The claim

The central premise shared by both pieces is that science, or at least astrophysics, is primarily a vehicle for human development. Hogg states this explicitly: when we hire a graduate student, "it absolutely must be because the graduate student will benefit from that work, not merely because that work needs to get done." The ergosphere post amplifies this into a governing principle: "The project isn't the deliverable. The project is the vehicle. The deliverable is the scientist that comes out the other end."

1.2 Why it is wrong

This is an extraordinary claim, and both authors treat it as though it were self-evident. It is not. The purpose of science is to produce knowledge — to extend the frontier of human understanding of the natural world. This has been its purpose since Thales, through Galileo, through Einstein, through the present day. The training of new scientists is an essential function of the scientific enterprise, because the enterprise requires people who can carry it forward. But to elevate the training above the work itself, to claim that the people matter more than the results, is to invert the relationship between means and ends in a way that no working scientist in a field with clinical applications would recognize.

1.3 The "no clinical value" dodge

Hogg acknowledges this tension. He argues that astrophysics is special because it has no clinical value — nothing in the world of policy or technology depends on the precise value of the Hubble constant. Therefore, since the results do not matter in a practical sense, the people must be what matters. This argument has a beguiling internal logic, but it proves too much. If the results of astrophysics truly do not matter, then the public funding that supports astrophysics is being spent on what amounts to a very expensive training programme for people who will mostly leave the field. That is not an argument for the primacy of people. It is an argument for defunding astrophysics. No funding agency in the world would accept the proposition that its grants exist to develop individual human beings rather than to advance knowledge. The fact that Hogg himself would likely not make this argument to the NSF suggests that the "people are the ends" framing is aspirational rather than descriptive — a statement about what he wishes the system valued, not what it actually rewards.

And even within the framework of pure knowledge production, astrophysics is not as clinically inert as Hogg claims. Astrophysical research develops statistical methods, computational techniques, instrumentation, and data analysis paradigms that migrate into fields where results have direct human consequences — from medical imaging to climate modeling to gravitational wave detection. The methods are the clinical output, even if the specific measurements are not. Dismissing the results as unimportant in order to elevate the process is a rhetorical move that works within a very narrow disciplinary frame and begins to crumble the moment you look at the broader ecosystem of science.


2. The Master-Apprentice Model Is the Problem, Not the Solution


2.1 The North American model

Both pieces assume, without much examination, that the North American model of graduate supervision is the correct framework for producing researchers. In this model, a supervisor selects a project, assigns it to a student, meets with them weekly, gives feedback, and guides their development over several years. The student works within the supervisor's research programme, on problems the supervisor has chosen, producing results that the supervisor will co-author. The ergosphere post treats this as a benign and natural arrangement. The supervisor is cast as a wise architect of the student's intellectual growth, carefully calibrating the level of challenge to maximize learning.

2.2 What I see instead

I see something different. I see a system in which graduate students are structurally dependent on a single individual for their funding, their research direction, their publication record, and their career prospects. I see a system in which students are paid wages that would be considered exploitative in any other professional context, justified by the claim that the "experience" they are receiving is the real compensation. I see a system in which the supervisor's research programme is advanced by student labour, while the student's development is framed as a gift bestowed by the supervisor. Hogg's invocation of the Kantian categorical imperative — that students must be treated as ends, not means — reads as aspiration at best and virtue-signaling at worst, because the system he operates within is structurally designed to use students as means. The low pay, the power asymmetry, the dependence on supervisor goodwill for letters of recommendation — these are not accidental features of the system. They are the system!

2.3 The latent anxiety about AI

When Karamanis worries that AI will allow Bob to bypass the formative struggle of his first year, the implicit concern is not really about Bob. It is about the supervisor's role as the indispensable intermediary in Bob's development. If Bob can get competent methodological guidance from an AI, if he can debug his code without weekly meetings, if he can understand a paper without having it explained to him by someone who controls his career — then what, exactly, is the supervisor for? The anxiety about AI in graduate education is, in part, an anxiety about the obsolescence of a role that has historically conferred enormous power and status on those who hold it.

2.4 The European alternative

This is not to say that all supervisors are self-interested or that mentorship has no value. It is to say that the master-apprentice model is not the only way to produce capable researchers, and it may not be the best way. In the European tradition, particularly at institutions like Oxford and Cambridge, doctoral students are expected to pursue substantially more independent research. The supervisor is a resource, not a director. The student chooses their problem, develops their methodology, and proves themselves through independent work. The degree is a certification that the student is capable of original research, not a record of years spent executing someone else's programme. This model produces researchers who are, by design, self-reliant — precisely the quality that both Hogg and Karamanis claim to value — but through a mechanism that neither considers.


3. The Real Question Is Admissions, Not Supervision


3.1 The Alice and Bob assumption

The Alice and Bob parable assumes that both students arrive at graduate school roughly equally unprepared, and that the first year's struggle is what separates them. I challenge this assumption. By the time a student has completed a bachelor's degree, they are a 22-year-old adult. If their undergraduate education has done its job, they should arrive at a graduate programme with the foundational knowledge, mathematical maturity, and intellectual discipline necessary to engage with current research. They should be able to read a paper, implement a method, write code, debug it, and critically evaluate results — not perfectly, but competently. This is what a bachelor's degree in physics or astrophysics is supposed to certify.

3.2 The actual failure

If students are arriving at graduate programmes without these capabilities, the problem is not that they need more hand-holding. The problem is that the undergraduate programmes are failing, or that the graduate admissions process is failing, or both. The solution is not to design graduate education as a remedial training programme and then worry about whether AI is disrupting the training. The solution is to raise the bar for admission and expect graduate students to function as junior researchers from the beginning.

3.3 How this dissolves the dilemma

This reframing dissolves much of the Alice and Bob dilemma. If both students arrive with genuine competence, then Bob's use of AI is not a crutch substituting for absent skills — it is a tool deployed by someone who has the foundation to use it critically. If Bob cannot tell when Claude is fabricating coefficients, Bob should not have been admitted. The filtering should happen at the gate, not through years of artificially imposed struggle within the programme.

I am aware that this is a demanding standard, and that most universities currently do not meet it. But the current standard — admit broadly, train slowly, exploit labour, and call it development — is not something to be defended. It is something to be reformed. And AI is not the threat to that reform. AI is the catalyst that reveals how urgently the reform is needed.


4. AI Usage Cannot Be Policed, and Attempting to Do So Is Toxic


4.1 The enforcement problem

Both Hogg and Karamanis are careful to say they are not calling for a ban on AI in research. The ergosphere post explicitly rejects the "ban-and-punish" approach as unenforceable and unfair. Hogg acknowledges the same. Yet both pieces spend thousands of words articulating why unsupervised AI use by students is dangerous, without offering any mechanism for preventing it that does not reduce to surveillance, accusation, or institutional coercion.

This is the practical impasse that neither piece resolves. You cannot monitor how a student reads a paper. You cannot verify whether a student wrote their own code or asked Claude to write it. You cannot determine whether a student genuinely understands their results or is fluently parroting an AI-generated explanation. Any attempt to enforce these boundaries requires either invasive monitoring — which is incompatible with the autonomy that both authors claim to value — or a culture of suspicion and accusation that would poison the supervisor-student relationship and the broader research environment.

4.2 Where responsibility belongs

The ethics of AI usage in graduate studies must ultimately rest with the student. This is not an abdication of responsibility. It is a recognition that intellectual honesty is a personal commitment that cannot be externally imposed on adults. If a student chooses to outsource their thinking and arrives at their qualifying exam or thesis defence unable to explain their own work, the system should catch this — not through AI surveillance, but through rigorous evaluation of understanding. If the system currently cannot distinguish between a student who understands their results and one who does not, then the system's evaluation methods are broken, and that predates AI by decades.

4.3 A personal tension, honestly stated

I will be transparent about a tension in my own position. If the decision rested entirely with me, I would ban AI usage below the level of a completed doctorate. Not because AI is inherently harmful, but because the formative years of scientific education — undergraduate and graduate — are where foundational understanding is built, and I think that process is best served by unassisted intellectual effort. AI is primarily a tool for accelerating the work of those who already know what they are doing. For those who do not yet know, it is more likely to obscure the gaps in their understanding than to fill them.

But I hold this position while simultaneously recognizing that it cannot be enforced. AI is ubiquitous, access is trivial, and any ban could be circumvented by anyone with a browser. The right response is not to pretend enforcement is possible. The right response is to design educational and evaluative structures that are robust to AI usage — structures that test understanding rather than output, that value explanation over production, and that filter for competence rather than compliance.


5. Progress Is Not Ego


5.1 The accusation reversed

The ergosphere post frames the desire to use AI for faster scientific progress as a failure to appreciate the formative value of slow, difficult work. This framing contains an implicit accusation: that anyone who prioritizes results over process is missing the point of science, or worse, is driven by ego.

I want to reverse this accusation. The insistence that science exists primarily to develop individual human minds — that the results are secondary to the experience of producing them — is itself a form of ego. It places the researcher's personal journey at the centre of an enterprise that is supposed to be about understanding the natural world. It elevates the subjective experience of "doing science" above the objective value of scientific knowledge. And it provides a convenient justification for a system that moves slowly, publishes incrementally, and resists any tool that threatens to make established researchers less indispensable.

5.2 The Schwartz experiment, reinterpreted

When Schwartz supervised Claude through a publishable physics paper in two weeks instead of a year, Karamanis's response was not "how remarkable that we can now do physics faster" but rather "the supervision is the physics, and the supervision requires decades of prior experience." Both of these things can be true simultaneously. The fact that effective AI supervision requires expertise does not mean that AI-accelerated research is illegitimate. It means that the combination of human expertise and AI capability produces more science, faster, than either alone. That is straightforwardly good for the progress of human knowledge. The question of where the next generation of supervisors comes from is real and important. But it is a pipeline question, not a reason to slow down the people who are already at the end of the pipeline.

5.3 Amplification, not equalization

Karamanis describes a colleague who initially feared AI because it might "equalize everyone" and later embraced it when it could "accelerate him." The author presents this as hypocrisy. I see it differently. I see someone who initially misunderstood the technology's implications and later understood them correctly. AI does not equalize. It amplifies. It makes the knowledgeable more productive and the ignorant more convincingly wrong. That asymmetry is not a bug. It is, if anything, the strongest argument for ensuring that researchers develop genuine expertise before they begin relying on AI — which is exactly the admissions-and-standards argument I am making, not the supervisory-coddling argument that these pieces defend.


6. What Needs to Change


6.1 The diagnosis

The current discourse treats AI as a threat to a functioning system. I see it as an X-ray of a system that was already broken. The North American model of graduate education mass-admits students into programmes that are structurally designed to extract cheap labour while providing "training" that could be delivered more efficiently and more respectfully through higher admissions standards and greater student autonomy. AI did not create this dysfunction. AI merely made it visible, because a system that cannot distinguish between a student who understands their work and one who had Claude do it is a system that was never measuring understanding in the first place.

6.2 The prescription

What needs to change is not our relationship with AI. What needs to change is our relationship with graduate education. Specifically:

  • Raise admissions standards and admit fewer students. Graduate programmes should select for demonstrated competence and intellectual maturity, not potential to be moulded. Students should arrive ready to engage with current research, not needing years of remedial training disguised as apprenticeship.
  • Expect independent research from the outset. The European model, in which students choose their own problems and develop their own methodologies, produces self-reliant researchers by design. The North American model of supervisor-directed research produces dependence by design. The former is more compatible with the AI age, not less.
  • Evaluate understanding, not output. If a thesis defence cannot distinguish between a student who understands their work and one who does not, the defence is a formality, not an evaluation. Rigorous oral examination, detailed questioning of methodology, and the expectation that every result can be explained from the ground up — these are AI-resistant assessment methods that academia already possesses but has allowed to atrophy.
  • Pay graduate students fairly. If students are professionals contributing to the advancement of knowledge, compensate them as professionals. The rhetoric of "formative experience" should not be used to justify poverty wages. Fair pay also shifts the framing: a well-compensated researcher is an adult colleague, not a ward of the supervisor.
  • Treat AI as any other tool — the researcher's responsibility. Usage should be governed by the same standards of rigour and honesty that apply to every other aspect of scientific work. No surveillance. No accusations. No policing of process. Judge the product and the person's understanding of it.
  • Abandon the factory model of academic research labs. The current system optimizes for publication volume, which incentivizes exactly the kind of shallow, AI-assisted output that both Hogg and Karamanis fear. A system that valued depth and quality over quantity would naturally reward the kind of understanding they want to protect.
6.3 The bottom line

The machines are fine. The people who built their expertise before AI arrived are fine. The students who will build their expertise despite the temptation to skip the work will also be fine. What is not fine is a system that was already failing its students, that already prioritized output over understanding, and that is now using AI anxiety as a reason to defend the status quo rather than confront the deeper dysfunction.

Science is about progress. Not your progress as a person — though that matters. Not your supervisor's programme — though that has its place. The progress of human knowledge. Any tool that accelerates that progress in the hands of competent people is a good tool. The task before us is to ensure that we keep producing competent people. That task is real and urgent. But it is a task of institutional reform, not of technological resistance.