Instead of typing furiously and constantly hitting backspace, what if you could code by just drawing out your ideas?
This vision is becoming a reality thanks to Ryan Yen (MMath ’24), a recent master’s graduate of the Cheriton School of Computer Science, and Professors Jian Zhao and Daniel Vogel. While at Waterloo, Yen co-developed Code Shaping, an AI-powered software that allows programmers to edit their code through free-form sketches.
Inspired by recent advancements in large-language models, Ryan Yen created Code Shaping alongside Professors Jian Zhao (supervisor) and Daniel Vogel. Now as a PhD student at MIT, Yen wants to leverage AI to support and enhance programming.
Code Shaping typically takes place on a tablet. Instead of typing, a user can use their stylus to annotate on top and around their code. When they press the generate button, an AI model will interpret and convert their sketches into code.
For example, to plot a visualization, users can circle a line of code they want as the data points. Then, they can draw an arrow pointing to a sketch of a bar graph with another arrow to indicate where the code implementation for plotting the graph should appear. Code Shaping can support any freeform sketches users make, like diagrams, charts, or mathematical symbols.
Suppose a user has coded a web page. Instead of editing the code line by line, they can sketch a new button or form field and draw an arrow to show where the corresponding code should be inserted. Then, the AI model will turn their design into actual code.
This innovative method can change how users code— beyond the keyboard.
A demo of Code Shaping, which converts sketches into visuals like diagrams, graphs and charts—bringing a new way of coding
Besides, Code Shaping could help streamline a user’s workload. For example, they could list their tasks in steps like writing “① Copy Code” with an arrow pointing to a particular section, followed by “② Arrange Dates in mm/dd/yyyy format.” By breaking their code into steps in a declarative way, programmers can organize and articulate their thoughts better.
Instead of manually replacing all the missing values, a programmer can simply write out the instruction for the AI model to act on. This innovative approach could make coding less strenuous.
Yen was inspired by the recent advancements of large-language models (LLM) and whether they can “allow users to express themselves more freely.” Now as a PhD student at Massachusetts Institute of Technology (MIT), he wants to explore if LLMs can change “how we interact with code.”
“There are so many nuances that humans think about. We have so many beautiful and rich things inside our brains,” explains Yen.
“But when we think about programming now, it’s mainly a rigid code syntax. We can only use our keyboard to type things out. I feel like this method is a waste. So, my research explores different ways to loosen these types of constraints, or how we can leverage AI to bridge these natural human expressions back into code.”
Programmers often turn to their whiteboards or notebooks to map out their work. It helps them untangle complex and abstract ideas. However, there hasn’t been a platform that combines sketching and coding, causing programmers to lose valuable insights from their brainstorming sessions. The closest one was from the 1960s, which converted sketches to functional programs, but it was created for non-programmers and didn’t allow code editing. As a result, programmers didn’t have control over their synthesized code.
Through Code Shaping, users can use “sketches to control the code itself,” says Yen. While conducting various user studies, several participants noted that Code Shaping allowed them to plan and to think deeply about their work. Overall, this method can revolutionize the software development space.
Despite some skepticism from other researchers, including doubts about coding on an iPad, Yen persisted.
“I was debating whether I should give up. But Professor Vogel reminded me of the beauty of being an academic researcher: having this rare freedom of exploring ideas without worrying about commercialization.”
That intellectual freedom paid off.
Code Shaping won a Best Paper Award at the 2025 ACM Conference on Human Factors in Computing Systems (CHI) — one of the top-ranked conferences in computer science and the leading international HCI conference. Yen does not plan to commercialize Code Shaping. Instead, he has open-sourced it, allowing the community of developers worldwide to use and build on his innovations. During his PhD, he will focus on leveraging AI tools to support and enhance coding.
The research, Code Shaping: Iterative Code Editing with Free-form AI-Interpreted Sketching, was published in the proceedings of CHI 2025.