Interview with AI Creator Meng Shengyu: Not Getting Lost in the AIGC Wave

As a sci-fi fan, I was lucky to be interviewed by the China Science Fiction Research Center about "one day for humans, one year for AI", the creator's subjecthood in AIGC, and how to swim in the sea of AI tools without losing yourself.

Translated by AI

As a sci-fi fan, I was lucky to be interviewed by the China Science Fiction Research Center. We talked about some really interesting topics: “one day for humans, one year for AI”, the creator’s subjecthood in AIGC, and how to swim in the sea of AI tools without losing yourself.

Introduction

A wave of AIGC (AI-generated content) is spreading around the world, giving creators new momentum to realise their science-fiction imaginings. To explore AIGC’s impact on science-fiction creation, the China Science Fiction Research Center’s research group on AIGC and the sci-fi film and television industry has conducted a series of interviews. This issue’s guest is Meng Shengyu (Simon Meng), a cross-disciplinary AI researcher and artist.

Meng Shengyu had been researching and creating with AI for years before the “first year of AIGC”. As an artist, he began making AI art as early as 2017 and has worked commercially with brands such as Xiaomi and Chevrolet using AIGC. As a researcher, he has studied the underlying logic of AIGC in depth and done a great deal of AI science communication in the media and at forums. And as a crossover practitioner who can code, he built the open-source text-to-3D model Dreamfields-3D on top of Dream Fields (AI text-to-3D generation), drawing wide attention from the field, including the original authors of the Dream Fields paper. In this interview, through his distinctive perspective and rich experience, we look at the future possibilities where AIGC and art meet, and at how creativity and technology can merge.

A history of creating with AIGC

Q: When did you start creating with AI? How have the ways you use AI and your experience of it changed? Which tools do you use most?

Meng Shengyu: I started using AI to create around the end of 2016 and the start of 2017, when generative adversarial networks were first causing a stir, specifically CycleGAN (cycle-consistent GANs) and pix2pix (conditional adversarial networks that map images to images). Earlier AI models could generate some content, but for images, designers cared most about generating images from text, and those earlier models couldn’t produce clear, legible images. Once GANs developed into CycleGAN and pix2pix, though, under certain conditions they began to produce images you could actually read.

The trend now is that the constraints on AI image generation keep shrinking. Getting a clear image used to come with many limits. There might be limits on computation time, or you had to restrict generation to a very narrow category; StyleGAN, for instance, which was very popular, could only generate faces, or each model could only generate one kind of thing. And if you wanted to use an original image for style transfer, you were also very constrained, and it was awkward to use. Usually nobody built graphical interfaces for AI image software, so it demanded more programming ability. Even with lots of constraints, the results were still very different from human work. So people sometimes chose to set aside the difference from human style for the moment and first check whether the image was clear or the composition rich. At that time there was still a big gap between AIGC and human creation.

Now these three weaknesses are being fixed rapidly: the barrier to use is extremely low, generation is very fast, there are very few limits, and the results can basically pass for real. If you don’t look closely, it’s sometimes hard to tell whether something was made by a human or an AI. There’s even a trend where the test people use to decide whether an image is AI-generated is whether it has enough flaws, because many think AI results may be better than what an ordinary person would produce. So the boundary between humans and AI has blurred. These three trends are plain to see.

Of the tools I use, the one I use most is ControlNet (a neural network architecture that controls diffusion models by adding extra conditions, making AI output more controllable) in Stable Diffusion (a text-to-image model based on latent diffusion). Designers need to generate lots of images. Before, the first results often weren’t what we needed, and as we got closer to the goal it was very hard to control precisely, so we couldn’t align the output with industry standards. ControlNet solved these problems. Now we can precisely control whether an image is sharp, whether particular elements appear, and what composition it follows.

Q: Last November you used AI to make a promotional film for a Xiaomi phone, and you shared your process on your WeChat account. Beyond what you wrote, what difficulties did you run into? How would you do it better with today’s technology?

Meng Shengyu: The difficulty was still controllability. What I found most interesting at the time was wanting to give the work a narrative involving time, approaching it from the past, the present and the future. Xiaomi was very supportive and liked the concepts. The problem was that the changes in the work weren’t fully controllable. For instance, the last frame turns into a meteorite and then gradually becomes a phone. Some of the changes along the way were satisfying, but irrelevant elements might suddenly appear in the middle, or a frame might suddenly turn into something else, and then you had to fix those in-between frames by hand. Overall it could get close to the effect I wanted, but not in the details. That was the hard part.

Promotional film for the international edition of a Xiaomi phone

If I did it now, I’d probably change approach and overhaul the whole workflow rather than use that pipeline (using Stable Diffusion for mood images, fine-tuning on the phone model with DreamBooth, generating keyframes with the trained model, interpolating frames into animation and smoothing it by hand). Runway Gen-2 (AI text-to-video), for example, is very good now. I might shoot a real video and then transform it. That work was done last November, and now it’s “one day for AI, one year for humans”; everything is completely different. I’d probably try some multimodal1 methods for the project.

Q: Your AI co-created work Barcelona Reconstructed in Coloured Sand uses photographs of sand from Barcelona. How exactly did the collaboration with AI work? How did you use multimodal models?

Meng Shengyu: It actually has to do with my own exploration. Take that latest work: judging by likes, there’s nothing special about it, so why did I make it? After powerful image generation models like Stable Diffusion and Midjourney (AI painting tools) came out, I stopped creating for a long time, because I couldn’t find a reason to create, or a reason to use these tools. When I used VQ-GAN (a GAN-based generative model that turns images or text into high-quality images) or Disco Diffusion (an AI painting tool), if you weren’t careful about method, the results were ugly and unattractive. Later I found that by intervening by hand, putting constraints on the generation process based on an understanding of a model’s characteristics, such as choosing certain subjects and particular parameter combinations, you could pull an overly blurry AI image to the edge between legible and illegible, creating works between abstraction and figuration, between human control and machine randomness. I found that really interesting, and I was keen to publish those works.

When Stable Diffusion and Midjourney appeared, they showed powerful generative abilities: the images were very sharp and, in surface aesthetics, perhaps very beautiful. At that point, publishing the work I’d made with the earlier tools was unlikely to interest anyone. But if I created directly with Midjourney or Stable Diffusion, I felt I lacked a reason to create. They’re tools, but they’re so powerful that a few simple operations produce something similar to what would take a lot of effort in the real world. If I were going to call it art, I felt I’d need, as with real art, a serious or distinctive understanding of the world and an angle of self-expression before making it, and I hadn’t found that angle or reason.

Then I looked at it differently and asked: what can only be done now that AI exists, and couldn’t be done before? I found that question fascinating. Everyone is generating images with Midjourney and Stable Diffusion, but I could achieve similar effects with a camera and Photoshop, or with 3D modelling; it would just take more time.

So is there really anything that only became possible with AI? When I went to Barcelona, I found the sand there beautiful, and I wondered what would happen if I mixed it with images, or with the city. Sand is itself a material used to build cities, and I’m very interested in the connection between the micro and the macro. Traditional ways of blending images are usually either simple collage in Photoshop, or subjectively extracting certain features of the sand grains to make a painting. But now, with multimodal models, you can project an image of the city of Barcelona and a microscopic image of sand into “latent space”.

“Latent space” corresponds to a high-dimensional function space in mathematics. You can think of it as the neural network encoding the meaning of each image as a coordinate, a feature vector. I can mix the feature vectors behind the two images in different proportions and generate a new image, which amounts to using the computer’s understanding of the images to blend them and regenerate. An image blended this way is certainly different from one blended with other tools, and I found that really interesting.

Human–machine co-created work Barcelona Reconstructed in Coloured Sand

Q: Many of your works have a science-fiction flavour, such as the different apocalypses of humanity in parallel universes. Have particular sci-fi writers or art movements influenced your work?

Meng Shengyu: I actually quite like science fiction, though I haven’t read that much. I once read all of Liu Cixin’s fiction, from novels to short stories. People jokingly call Liu Cixin “a serial destroyer of universes”, since his stories often end with the world being destroyed. After reading the Remembrance of Earth’s Past trilogy, I realised for the first time that AI could generate what I imagined from text, so I tried “destroying the Earth” to see what it would look like. Recently I’ve been reading Arthur C. Clarke, and Childhood’s End really shook me.

Another reason I like science fiction is that I’m quite sensitive, and many things in life make me anxious and worried. When I read science fiction, it’s as if I step out of a reality full of trivial troubles. So I thought: since AI is so powerful and can set my abilities free, I shouldn’t make work about everyday subjects any more.

As AI tools change, my creative leanings change too. At first, getting AIGC tools to be realistic was hard. As I said, back then they were better suited to things between the real and the unreal. Now that the tools have changed, my leanings shift as well. Barcelona Reconstructed in Coloured Sand, for instance, isn’t very sci-fi; it’s more like a fairy tale or a kind of art.

The limits of AIGC and breaking through them

Q: How do you think AI might affect science-fiction creation?

Meng Shengyu: At the micro level, science fiction can take the form of text, images, video and so on. With these tools, efficiency is bound to rise, or rather the number of options we have per unit of time increases. So for equally talented people, with the help of tools, the quality of their work may also improve.

At the macro level, what I find interesting about science fiction is that it marks a boundary of humanity’s constrained, rational imagination. Outside science fiction, in genres such as xianxia or fantasy, I can let my imagination run wild; there may be some constraints in terms of human sense, or intelligibility, but there are no constraints from physical laws or science. Science fiction may be the boundary of imagination under the constraint of these objective rules.

Since AI arrived, some of the objective rules we understood have changed, such as “seeing is believing”. Recently an AI appeared that can derive new physical formulas directly from phenomena, whereas traditional formulas might have to be built from lower-order formulas or derived by other mathematical methods. This AI derives them directly from probabilities instead. That may change how we understand the world.

It’s even more true in ethics: AI has given science fiction many new subjects. Some areas used to be off-limits, or we didn’t know how to write about them, and now suddenly they feel open to exploration. So at the macro level, AI has expanded the boundaries of sci-fi writing and changed some of the background assumptions of earlier works.

Q: Do you think today’s automated decisions and creativity from technology are reliable?

Meng Shengyu: It depends on how you define reliability, but overall it isn’t that reliable; people still need to act as gatekeepers. Objectively, it often hallucinates. For example, sometimes when I’m writing a paper, I have a point and need references for it but can’t be bothered to look them up. I know someone must already have written about it and verified it, so I ask ChatGPT to find them, and it gives me a pile of things that look plausible and relevant. But when I look up those papers, about 50% don’t exist.

For now, objectively, automated decisions and creativity from technology certainly aren’t reliable. Subjectively, say I want AI to make something according to my intentions; because of problems in its training data, it’s still affected by many biases. People used to use AI to screen CVs. Two or three years ago, CVs still included a photo of the candidate, and someone found that if you wore glasses in your photo, you were more likely to be shortlisted. If your background was a bookshelf or an upmarket room, the chances went up again. In reality, there’s no necessary link between glasses and how competitive a CV is; there’s only some statistical association, but the model absorbed that bias. So, concretely, it’s still not reliable enough.

Q: Beyond biases in the original data, human-made content is declining while AI-generated content is exploding. Could it be that, past a certain point, AI or algorithms can no longer generate anything new? How might that be overcome?

Meng Shengyu: Someone once asked me what would happen if you trained an AI on AI-generated images. I said it definitely wouldn’t work; it would overfit, fitting ever more closely to its own range. But I now think I need to revise that. In practice, I’ve noticed a few interesting things.

First, GPT-4 is currently the best model. What’s the advantage of GPT models over earlier language models? They use a question-and-answer format, aligning their existing knowledge with high-quality Q&A datasets. But the pre-training code isn’t open source, and neither is the pre-training dataset.

High-quality Q&A datasets are hard to find, so many new open-source language models use an interesting design. They build or collect a set of questions, ask GPT-4, record its answers, turn those into a dataset and train the open-source model on it. The resulting model can reach 80–90% of ChatGPT’s quality with far fewer parameters.

Similarly, many people think Midjourney’s quality is good because it was trained on lots of high-quality commercial images. But Midjourney is paid, has few plug-ins and is less controllable, whereas Stable Diffusion is open source, with many plug-ins and more control. So many people extract images generated by Midjourney and train Stable Diffusion on them, using Midjourney’s output to raise the quality of Stable Diffusion’s images. In other words, they skip collecting a real high-quality image dataset and use Midjourney’s output directly to tune Stable Diffusion’s aesthetic tendencies.

The situation now is that there are many different models, each with different data sources, and those sources are very rich, with high-quality generated images and text. So in future we may really train one AI on another AI’s output, adding some human elements and adjustments. I think this paradigm can create an endless stream of new things. It feels as though the limit has, to some extent, been broken; it’s completely different from before.

There are some fun phenomena too. From the end of the year before last until last June, Disco Diffusion was the best generative model. Its output was fairly blurry and abstract, but it often surprised you with a kind of leap. A couple of weeks ago, browsing Reddit, I saw someone ask: “Has anyone made a retro Disco Diffusion-style model for Stable Diffusion? I miss that style.” The times are changing so fast: a year ago it was the SOTA model (state of the art, the best and most advanced model), and a year later it has become a nostalgic retro style. That really struck me.

Silicon-based lifeforms created with AI

Surviving in the age of AIGC

Q: As artistic creation becomes increasingly automated, where do you think the advantage of human creation lies?

Meng Shengyu: I don’t think art can be fully automated; only parts of the creative process can. I think the two most important things in art are the subjecthood of the creator and the subjecthood of the reader. Unless AI acquires a fully independent personality, it can’t replace the creator in either. Even if we can generate huge numbers of images with AI, a person still has to choose among them, and that selection is very important. Say we pick up stones from the ground and finally find one. We didn’t make that stone, but once it’s cleaned and placed in the right spot, it becomes a work of art. So I’d say humans themselves can’t be replaced; what gets replaced is some of the work in the creative process. The human advantage will remain until a true silicon-based civilisation appears.

Q: For latecomers to AIGC, how can they build a full understanding of how the technology has developed, as you have?

Meng Shengyu: I’d recommend Andrew Ng’s deep learning course (he’s an adjunct professor of computer science and electrical engineering at Stanford). It was the first course I took. It took me more than a year; halfway through I found it very hard. Before that I’d only studied calculus, not advanced maths, linear algebra or probability. The head of Bilibili recently said the video with the most watch time on Bilibili is advanced maths, and I can tell you from experience that it’s true. I first re-studied four advanced maths textbooks, about 200 lessons, then came back to deep learning. Then I found I wasn’t familiar enough with Python, so I took a Python course. You ask whether there’s a book between beginner and advanced level that explains the principles without the calculations and uses them to connect the technologies. I haven’t found one. I’ve wondered whether to write such a book, but it feels like a huge amount of work.

You can also start by finding the representative model in each field, such as Stable Diffusion and GPT-4, read their papers directly to understand their core frameworks, and then trace the related technologies backwards. Take Stable Diffusion: one level up is Latent Diffusion, which adds latent variables to diffusion. First work out what “latent” and “diffusion” mean, then go back before diffusion: diffusion developed from DDPM (Denoising Diffusion Probabilistic Models). Then work out what DDPM is, and go further back. You have to read a lot of papers, so it isn’t easy. For GPT, tracing back leads to the Transformer (a deep-learning model that uses self-attention to speed up training), then perhaps the attention mechanism (which lets a model focus on and learn from important information), then perhaps the RNN (recurrent neural network). What came before RNNs I don’t know. That’s roughly how to trace backwards, looking up the relevant knowledge depending on how far back you go. You don’t need to understand every detail; start by reading the relevant papers.

Work generated by Dreamfields-3D, developed by Meng Shengyu

Q: As a non-technical person, new AI technologies keep appearing and give me fresh experiences every day. A key question: how do you avoid getting lost in all these technical experiences? Faced with two new technologies, how do you choose?

Meng Shengyu: Not getting lost involves two things. First, don’t get lost choosing between all the tools; second, don’t lose sight of why you’re using them.

I started following these tools very early and studied neural networks and deep learning systematically. No technology appears out of thin air; each develops step by step from what came before. So whenever I see a new tool, I roughly know what technology it uses, how it improves on earlier tools, what its drawbacks are, what new things it can do and where its limits lie. That way I can fit it into my own body of knowledge, compare it, and judge whether it’s genuinely new and useful or just repeating earlier work without much meaning.

There are so many tools now that we can’t learn them all; learning two or three is about right. The principle I’d hold to is “learn what you’ll use”. Start from the tasks you face now, and learn selectively according to your main tasks and goals. When you see relevant tools, pay closer attention, focus on a few and learn them in depth.

About the guest

Meng Shengyu (Simon Meng) is a cross-disciplinary researcher in architectural design, art, AI and ecology; a PhD candidate in architecture at the University of Innsbruck (UIBK); holds an MArch from University College London (UCL); and is a part-time tutor on urban design master’s programmes at UIBK and UCL. He writes the multi-platform channel “Simon的白日梦” (Simon’s Daydream). His work has been exhibited in London, Edinburgh, Venice and elsewhere, and he has created AIGC work with Xiaomi, Chevrolet, National Geographic China and Chow Tai Fook.

Notes

Contributors: Zhang Zirui, Qihang Scholar at the China Science Fiction Research Center

Zhao Yahui, Young Talent Reserve

This article is an output of the China Science Fiction Research Center project “AIGC Empowering Intelligent Production in the Sci-Fi Film and Television Industry”, which studies AIGC’s role in making science-fiction film and television. Stay tuned for more!

Layout: Shen Dan

Editor: Yan Jinfang

Reviewed by: Yao Lifen

Footnotes

  1. “Modality” is a biological concept proposed by the German scientist Helmholtz: the channels through which organisms receive information via sensory organs and experience, such as the human modalities of sight, hearing, touch, taste and smell. Multimodal AI is a new paradigm that combines various data types (images, text, speech, numerical data) with multiple intelligent processing algorithms to achieve better performance. ↩

Search