When AI does the work – what happens to our judgement?
When AI was used to help formulate both the questions and the answers in an interview about AI, the result seemed reasonable at first glance. But according to Professor of Design Johan Redström, the main point was lost.
The setup was fairly simple. I, a communications officer at HDK-Valand, was going to interview HDK-Valand Professor of Design Johan Redström about a research paper on design and artificial intelligence. As usual, I asked ChatGPT to summarise the paper and suggest some interview questions to work from.
When the questions reached the researcher, he asked his own AI to formulate answers. The model had access not only to the paper in question, but also to a larger body of research material and a prompt specifying how he wanted to express himself.
The result was, in a sense, an interview in which AI asked questions of AI. And the answers were good.
– Absolutely, they were good answers. The only problem was that some of the most important points in the paper disappeared, says Johan Redström when we meet for coffee a few days later.
What should we humans do?
He was particularly struck by the answer to the final question, which concerned what an AI service might look like if designers began to think of AI as something more than a conventional tool.
The AI suggested a search service that would make it clearer why an answer looks the way it does, which sources underpin it and which interests have influenced the result.
A reasonable answer, according to Johan Redström. The problem is that it is essentially the opposite of the idea he is trying to develop in his paper.
– The AI assumes that the question is about what the machine should do. But I think the fundamental question should be: what should we humans do? Only then can we ask what role machines should play in what we do.
What does my hammer do when I’m not there?
To understand why this distinction matters, we need to go back almost a hundred years. In his research, Redström has studied early Swedish industrial design. Designers in the 1930s were also facing a development in which machines were transforming society and people’s everyday lives.
Their response was to place the human being at the centre and turn everyday objects into useful things and tools. Tools were subordinate to human actions. It was the human who was to decide what should be done – not the machine.
That idea is still with us. We talk about AI tools in much the same way as we talk about other tools: humans decide what they want to achieve and use technology to do it. But the comparison does not quite hold.
– It’s not as if my hammer at home is doing something when I’m not there. But these things are doing an enormous amount all the time, without us even knowing what they’re doing.
How did I end up here?
AI systems anticipate, prepare and change things. They do not simply respond to our actions; they are part of technological systems that also influence which choices are presented to us. This becomes clear in an ordinary conversation with a chatbot.
– We write a question and get an answer. Then we are often given suggestions for the next step: would you like me to do this? Should I help you with that?
I have always thought of this as a service function. But the suggestions also do something to the conversation.
– Eventually, 45 minutes later, you find yourself wondering: how did I end up here? We are still acting. We write, click and choose. But the question is how obvious it really is that we are the ones determining the direction.
Who is doing the work?
Something similar happened with the interview responses in our experiment. The AI had been given an extensive body of material to work with. And yet something was lost along the way.
– AI is good at extracting what appears to be important and formulating a reasonable answer. But at the same time, it can miss what is significant to someone who really knows the material. And when research has a critical edge, there is also a risk that the edge gets blunted. Everything becomes diluted and smoothed out. Or, conversely, it becomes overly polemical.
Seen in this light, our little interview experiment becomes more than an amusing feedback loop. If AI can help the journalist decide which questions seem most important, and then help the researcher decide what should be said in the answers, the technology has taken on a role in two of the judgements that together determine what the interview will actually be about. To decide whether those suggestions are any good requires something else: judgement.
– Someone who knows their material can look at six AI-generated suggestions and realise that none of them quite captures what matters – or see that a seventh option would be better. But if you haven’t developed that judgement, suddenly you start seeing the world through what is presented to you.
If the machine does all this – what do we do?
This is one of the things that concerns Redström when it comes to today’s students. Engaging deeply with material is not simply about gathering information. Part of the work is learning to decide for yourself what is central, what is peripheral and what is worth investigating further.
If you hand over that judgement, you may also be handing over part of your own agency.
– Automation is nothing new, of course. When we write on a computer, we don’t have to think about how the letters are formed or about keeping the spacing between lines even. Technology has long taken over parts of the work. The difference is that automation is now beginning to reach further and further into areas that require creativity, critical judgement and reflection.
This raises a question that is bigger than whether AI can write a good text: if the machine does all of this for us – what do we do?
What is actually worth it?
Johan Redström tells me about a colleague who imagines a future in which researchers no longer write academic papers. Instead, they could produce research material and then let AI generate whatever text is needed at a particular moment: a conference paper, a report to a funding body or a text for the general public.
But something happens if writing is reduced to producing a finished text. Writing also consists of all the small decisions made along the way: what to include, what to leave out and how different things relate to one another. It is through making these kinds of decisions that judgement is developed.
Perhaps AI can therefore force us to ask a different question. Not simply how much work technology can do for us, but why we do certain things ourselves in the first place.
When a machine can produce music, images and texts that meet basic requirements in a matter of seconds, “good enough” suddenly becomes very easy. The interesting question then becomes what it takes for something to be more than that.
– What is it that makes something worth it? The time we spend writing, for example, rather than simply getting an AI to generate something for us.
He does not have a definitive answer. But at a time when general answers are easy to obtain, dwelling on the question seems more important than ever.
Text: Cecilia Köljing
1. The writer asked ChatGPT to summarise the researcher’s academic paper:
Prompt: Hi! What is this academic paper about?
The Art of Living with Machines
2. ChatGPT responded:
As machines and industrialisation transformed society, designers faced a question: What happens to human control and agency when machines are able to do more and more by themselves? The paper looks specifically at early Swedish industrial design and examines how designers historically dealt with the tension between technological development and human agency – that is, people’s ability to act, exert influence and make decisions for themselves.
According to the paper, one important response to industrialisation was for design to focus on “useful things” and tools. Technology was designed as something that people use: the human has the intention and control, while the object or machine is the tool that helps them accomplish something. This way of thinking has become a fundamental part of modern design.
The problem with AI is that this relationship is beginning to break down. Generative AI is not really a traditional tool that simply does exactly what the user tells it to do. Systems can, for example, generate content, suggest solutions, interpret instructions and influence outcomes in ways that mean agency is distributed between humans and technology– what the abstract refers to as distributed agencies.
The author’s argument therefore seems to be that design can no longer rely on the old model:
human → uses → passive tool
Instead, we need to develop new ways of thinking about situations that look more like:
human ↔ AI ↔ other people/systems, where multiple actors contribute to what actually happens.
When the abstract states that the previous “aesthetic and conceptual foundation has reached its limits”, this does not simply mean that AI needs better-looking or different interfaces. It is a more fundamental argument: the very idea of what we design, who acts and what constitutes a tool may need to be reconsidered.
In very brief terms
The paper essentially asks:
If design has historically made technology comprehensible by treating the machine as a tool under human control – what happens to design when AI no longer functions as a passive tool?
The author uses Swedish design history and industrialisation as a parallel to today’s development of AI and argues that generative AI requires new perspectives on design in which agency is not automatically understood as belonging solely to humans.
3. The writer then asked ChatGPT:
Prompt: The interview should become an article/feature rather than an academic interview. Can you help me select perhaps 5–6 questions in a good order, moving from an accessible opening question to more critical follow-up questions?
4. ChatGPT responded:
Absolutely. For a feature or journalistic interview, I would structure the conversation so that you begin with something concrete and accessible, move into the research, and finish by looking towards the future. That way, the interview will not feel like an academic seminar.
I would go with these six main questions, with possible follow-up questions in reserve:
- We often talk about AI as a tool. But you argue that this description is becoming inadequate – why?
Follow-up: When does AI actually stop being a tool? - In your research, you go back to early Swedish industrial design. What can that history teach us about the AI transition we are experiencing today?
Follow-up: Do you see any clear parallels between how people reacted to machines then and how they react to AI today? - You write about “distributed agency”. How does this manifest itself in practice when, for example, I use ChatGPT to create something?
Follow-up: Who is actually the creator of the result in such a situation? - If humans no longer have full control over what technology produces, what happens to the designer’s responsibility?
Follow-up: Can we still hold an individual designer responsible when the outcome is shaped by people, AI models and other systems together? - Is there a risk that we make AI out to be more revolutionary than it actually is? Why can’t we simply regard generative AI as a very advanced tool?
I would definitely include this one. It tests the researcher’s thesis and may produce a much more interesting quote than a question that already accepts the paper’s premise.
- If the old idea of the human as user and technology as tool has reached its limit – what comes after it? How should we design the future relationship between humans and AI?
Follow-up: Is there a concrete example of how an AI service would look different if designers began thinking this way today?
If you only have 10–15 minutes with the researcher, prioritise questions 1, 2, 5 and 6. Together, they create a strong narrative: What has changed? → What can history teach us? → Is the premise really true? → What happens next?
Another journalistic trick is to ask for examples whenever the answer becomes abstract. If the researcher starts talking about agency, autonomy, aesthetics, etc., feel free to interrupt with: “Can you give me a concrete example of what that might look like?” That is often where you get the material that actually works in an article.
5. The researcher gave ChatGPT the following:
Prompt: For this task I need you to be my communications manager, assisting me in sharing research results to a general audience. I have received a set of questions from a journalist about the paper "arts of living with machines" (in the source folder) that I need help answering. The questions are in Swedish and I need the answers in the same language, but keep my tone overall. I want the answers to be short, not much longer than the questions.
I also want the answers to bring out what is most important about the article.
6. ChatGPT responded:
- We often talk about AI as a tool. But you argue that this description is becoming inadequate – why?
A tool is something that waits for us to use it; intention and action are assumed to lie with the human. AI systems, by contrast, act partly autonomously, shape what we see and simultaneously collect data about us. This makes it unclear who is using whom, and the metaphor of the tool conceals more than it explains.
- In your research, you go back to early Swedish industrial design. What can that history teach us about the AI transition we are experiencing today?
It shows that we have been in similar situations before. Designers in the 1930s neither denied industrialisation nor made technology an end in itself; they tried to develop a new way of looking at everyday life and connect machines to how people actually lived. The lesson is that we, too, need a new conceptual framework – but not a new universal solution.
- If humans no longer have full control over what technology produces, what happens to the designer’s responsibility?
Responsibility does not disappear; it changes and becomes greater. The designer cannot simply be responsible for a finished form, but must also take responsibility for the relationships and conditions the system creates: what it enables, what it prevents, and which actors or consequences become invisible. Less direct control therefore does not mean less responsibility.
- Is there a risk that we make AI out to be more revolutionary than it actually is?
Yes. Much of today’s rhetoric around AI is familiar from previous encounters with machines: both promises of liberation and fears of subjugation. History helps us avoid treating AI as entirely unique. But it must not obscure what is new either: these systems act more independently and redistribute responsibility, resources and power.
- If the old idea of the human as user and technology as tool has reached its limit – what comes after it? How should we design the future relationship between humans and AI?
After “the user and the tool”, we need a more relational understanding: people, models, data, institutions and other affected parties all influence one another. The question is not simply whether something is useful, but for whom, at whose expense and with what consequences. We should design multiple responsible ways of living with AI rather than searching for a single optimal solution.
- Is there a concrete example of how an AI service would look different if designers began thinking this way today?
Take an AI-based search service. Today it appears to be a simple tool, while ranking, tracking, data collection, advertising and behavioural influence take place beneath the surface. A different service would show why the answer looks the way it does, which sources and interests have shaped it, and make it possible to influence the system’s priorities.