How do Designers Collaborate with AI?
Research Project
Role
UX Researcher
What I did
Research interview design, interview conducting, qualitative analysis, data visualization
Result
Findings on the relationship between regulative thinking and interaction with ChatGPT
Generative AI Tools in the Design World
As the demand for more designers grows across industries, how do we educate design students to meet this need? Specifically, how can design students learn to collaborate effectively? We can't reasonably provide every design student with an experienced designer to collaborate with, so how do we create collaborators for students?
To meet this demand, Dr. Ha Nguyen proposed the potential of ChatGPT as a collaborative designer. Dr. Nguyen hypothesized that ChatGPT could act as a collaborator, allowing design students to work with an "experienced designer". To test this hypothesis, we conducted design activity interviews and analyzed participant's thinking with qualitative coding. Our results were promising, showing a greater presence and diversity of reflective thinking ​
Methods
Participants:
We recruited 17 participants from a Mountain West University design program. Experience ranged from undergraduate students to advanced industry professionals.
The task:
Participants were asked to redesign Canvas, the learning management system familiar to users. They did one round of individual designing, brainstorming users and solutions on their own, and then another round of collaborating with ChatGPT.
-
Introduction to task
-
Individual design (10 min)
-
Collaboration with ChatGPT (10-15 min)
-
Design idea sketching (5-7 min)
-
Debrief
Background
The following research questions guided our analysis.
01: How do designers prompt ChatGPT in a design task?
02: How do designers engage in reflective practices when interacting with ChatGPT?
Definitions
Reflective practices: Learning from past actions to generate new insights and reframe design spaces. More experienced professionals tend to reflect more often in their work. Promoting reflective practices is one way to help move novice students into advanced practitioners (Schön, 1983).
Design practices: Designers typically engage in the following actions when designing.
Name: Mention aspects of a task, like clarifying design requirements.
Frame: Focus in on subproblems.
Move: Experiment with ideas and actions. We identified instances of brainstorming as moving.
Reflect: See reflective practices above.
(Liao, et al., 2023).
Procedures
RQ01: How do designers prompt ChatGPT in a design task?
To answer this, we inductively coded participants' prompts to ChatGPT using the following codebook.
Table 1. ChatGPT Prompt Codebook
RQ02: How do designers engage in reflective practices when interacting with ChatGPT?
To answer this, we focused on two aspects:
Iteration between design practices
Design is a multiphased process, so we visualized participants' iteration between these design phases.
Reflection focus
We inductively coded reflection utterances.
Results
RQ01: How do designers prompt ChatGPT in a design task?
Figure 1. Prompt Coding Results
Our analysis of participants' prompts to ChatGPT revealed a primary focus on information gathering and ideating. So, during the tasks, participants tended to use ChatGPT to gather more information about the design space and ideate new ideas to solve the design problem.
Our analysis of participants' prompts to ChatGPT revealed a primary focus on information gathering and ideating. So, during the tasks, participants tended to use ChatGPT to gather more information about the design space and ideate new ideas to solve the design problem.
RQ02: How do designers engage in reflective practices when interacting with ChatGPT?
First, we wanted to understand how our participants iterated between design practices during the task. Remember that design practices are typical actions taken when designing (Liao, et al., 2023).
Figure 2. Iteration in Design Practices
Take a look at the second round of designing labeled "withChatGPT" above. During the ChatGPT round of designing, participants had a greater quantity and diversity of reflection. This indicates collaboration with generative AI might be valuable for design professionals and students.
​
Next, we analyzed participant's reflective practices. It was here that we noticed something fascinating
Figure 3. Aspects of Reflection
As visible above, our participants' reflection tended to focus on evaluating the AI, noting ChatGPT's helpfulness (or lack thereof) at about 38% of total reflection. And, reflecting on the quality of design ideas at about 17% of total reflection. We also found that some participants, like P11 for example, reflected in a variety of ways. This led us to an interesting insight. ​
Reflective Participants
Several participants had a variety of reflection aspects like P11–we’ll call them reflective participants. Reflective participants interacted with the AI differently. While most participants used GPT primarily for information gathering, reflective participants balanced their usage between information gathering, ideation, and critiquing. This might indicate that diverse usages of the AI tool encourage more reflection. To demonstrate, consider this line from a reflective participant.
P11:
“Originally, I thought of the sidebar menu … But then ChatGPT made me realize … One of the things that made it so annoying for my special education students is that they have to scroll so much, so to make it easier you can see the whole week layout.”
P6 started the interaction by gathering information from ChatGPT about the issues children had to frame their thinking around special education students. GPT’s responses led them to reflect on new features followed by an ideating session with GPT moving towards accessibility layouts, like the whole week layout mentioned.
As visible from this quote, P6 was a reflective participant, iterating on ideas. They also used GPT with more balance between information gathering and ideating, which we hypothesize may have contributed to their iteration in reflection.
Uses of AI
Information gathering, ideating
Reflection
Frame, reflect, move
Implications
Research
For research, there is a potential for AI tools in design. As we saw reflection increased during conversations with GPT. Design researchers may find valuable insights from collaboration with AI. Especially when framing tools for iterative ideation and reflection.
Research: AI tools–Potential for designers when framed for iterative use.
Education
For education, applying AI tools for various applications, such as information gathering, ideation, critique, and template creation, may result in more diversity of reflection. As we discussed, participants who had multifaceted uses of the tool tended to reflect in more aspects. Educators may look into the potential of scaffolding AI tools to promote more variety in uses.
Education: Applying AI tools for various applications may increase reflective thinking.
Wrap-up
Future Work
Future work could look for links between reflective practices and design quality. We also used a readily available version of ChatGPT, our next steps are to develop a fine-tuned generative AI model for design based on this research. With that developed model we could discover a method to scaffold AI tools for multiple uses beyond information gathering.
Division of Labor
As this project is a snapshot of research with Dr. Nguyen of USU, a division of labor is provided.
Jake
-
Presentation design
-
Design of visuals
Dr. Nguyen
-
Human study design
-
Research direction (grant application, owner of idea, etc.)
Collaborative
-
Conduction of interviews
-
Coding (qualitative analysis)
Works Cited
Schön, D. A. (1983). The reflective practitioner: How professionals think in action. Routledge
​
Liao, Q. V., Subramonyam, H., Wang, J., & Wortman Vaughan, J. (2023, April). Designerly understanding: Information needs for model transparency to support design ideation for AI-powered user experience. In Proceedings of the 2023 CHI conference on human factors in computing systems (pp. 1-21).