The Center for Data Innovation spoke with Eric Risser, chief technology officer and founder of Artomatix, a company based in Dublin that uses AI to create virtual 3D worlds for the video game, movie, and industrial design industries. Risser discussed how his technology alleviates some of these sectors’ challenges and how creative AI will evolve.
Eline Chivot: How did you start Artomatix? What are the challenges your technology addresses?
Eric Risser: I started working in machine creativity during my bachelor’s degree at the University of Central Florida (UCF) back in 2004. I was offered a job as an undergraduate research assistant at UCF’s computer graphics lab, and my assignment was to come up with a research question which had to be important, difficult, and unsolved. I had always been interested in computer graphics and AI, and had a bit of knowledge in those fields. So I thought I could combine these two in some way, by making an AI that can create computer graphics. It was just a general idea at first, to justify my position, but then I really got into the research project. I continued with a master’s degree at Columbia University and then was offered a doctoral position at Trinity College Dublin. At the time (2012), there was no industry for the system I had created. AI was still going through one of its “winters”—regular AI was not much discussed, and creative AI even less so. I found myself at a crossroads: I could either get a job in industry or continue in academia, with a post-doctorate or a professorship. I really loved the field I was in, and I knew that at a high level, content creation was a major problem in the entertainment sector as well as for other verticals. As I had a working solution for a massive issue, I thought I could leverage it to do something about that. So this is what led to Artomatix.
Before being a research assistant, I was into video game development in high school and college, working online, with development teams of programmers. I was therefore more inclined to “give it a shot,” with this background, because I had ended up learning how to make game art in my free time. This is why and how I knew the art creation process was tedious, laborious, and repetitive. I knew the kind of process artists would go through to create something, how slow it is, how everything represents a lot of work. On top of that, many of the tools we still use today, such as Adobe Photoshop or 3D Studio Max, were established in the 1980s. They got better over time, with a few extra buttons, but they haven’t fundamentally changed.
In parallel, in academia, I had been joining conferences organized by industry, including the game industry, so I knew about some its trends: Team sizes and budgets for games were ballooning and increasing in magnitude every five years, making that sector more inaccessible and financially challenged than before. At these conferences, I heard the concerns were that there was a lot of content to be made, but that there weren’t enough people or funds to create it. Previously, it was possible to make a decent game within three years as a developer, perhaps not on par with a console game, but you could punch above your weight class. These days, timelines are beyond five years, budgets and teams have increased, with hundreds of people working in studios, and thousands outsourced as mercenaries.
I thought my prototype could solve the problem that art costs too much and takes too long for the creative and industrial design spaces. Artomatix offers better technology to solve this.
Chivot: Your technology is based on computer graphics, deep learning, and computer vision. How does it work, and why is it particularly suited for digital media creation and to assist human artistic creativity?
Risser: In terms of the actual technologies or technological bricks, we had to code up a lot of Artomatix from scratch. Artomatix is based on various algorithms and technologies, ranging from standard computer graphics algorithms to standard machine learning algorithms, all the way to more experimental technologies, across a wide range of methods: Various dimensionality reduction approaches, expectation-maximization (EM) algorithms along with deep learning and neural networks based solutions, are built into Artomatix. We took a hybrid approach, meaning the best of different worlds, and combined them together, which was needed to practically work on the problem we had set out to solve. We had to go for quality, flexibility, and speed.
The human-computer interaction aspect was not easy to develop. At first, the industry could not necessarily use our algorithm, although it was working—but it would be artists steering this tool, using it to curate their world, and artists need a rich, intuitive, responsive, beautiful, and flawless workflow.
There are many examples out there that a computer can learn from, and developing Artomatix was about programming a computer by example to build solutions. Traditional programming is about entering rules, such as “add number X,” “place number Z in location D,” etc. This is how and what our human architecture was geared up for. Think of such software as advanced “paint brushes,” whereas we are building “assisted painters,” based on an example-based workflow. With example-based programming, our computers learn differently, by using experiences, connecting them to millions of other fragments of experiences, and creating new ones. Let’s say you put in an image, whose pixels are arranged based on the content itself. The AI learns those arrangements in pixels, not in exact rules, but rather in flexible, “plastic” rules. Creative AI is about using statistical models of—for example—3D models, animations, sounds, and images, and building new things that follow similar probabilities and share characteristics, to then distribute them in a feature space.
Example-based programming as a workflow essentially involves three ingredients. The first one is seed data, as you never start with a blank canvas like you would in traditional painting. It could be something you’ve painted in Photoshop for instance. The second ingredient is human intent—a human needs to have some objective for that data, for instance: Do we need more of it, turn it into something else, or enhance it in some way? And the third element is about automating that intent. Our solution is different than others in that it is the first in this space to truly offer this “automation story.”
Chivot: Can you give an example of your process for working with a client? At which stage of the art production do you intervene?
Risser: We’ve worked with a startup that is building virtual reality experiences. The team goes to places which their users would want to travel through virtually, such as Machu Picchu, and scans them. They can scan about 90 percent of a space, but there is a remaining 10 percent which typically comes out skewed or distorted, for instance there may be a dark area with a shadow. Without our solutions, this team would end up spending as much time on these 10 percent as they would by building them from scratch, which is repetitious, tedious. The 90 percent they’ve scanned constitutes that seed data and those examples which our software can learn from and expand upon.
Making an average texture or material scanned from the real world seamless requires between one and five hours of manual work, depending on the complexity of the texture, and tasks need to be repeated four thousand times. So using our software to do this automatically represents a huge return on investment.
Chivot: How good is machine creativity at expanding or assisting the creativity and imagination of producers and digital graphic artists, and at replicating the variety of the real world into the digital world?
Risser: It is interesting to think about where Artomatix fits into the pipeline and workflows, and what value does it really bring. We’ve built an AI that can mimic human-like artistic creativity. We are not replacing artists, and I think of them as those creative nuclear reactors. But if everything in the creative industry is inefficient and suboptimal, and if costs are ballooning, it is because 99 percent of what human artists are doing is low level creativity work and coverage work. Removing scenes manually can take hours. Human artists are those elite special forces in that industry, working with big ideas which leads to content creation. They have sophisticated, high-level creativity. But they end up having less time to hone these skills and be creative at this level, because a lot of what’s needed to accomplish their work is about scanning and cleaning up, which involves focusing on low level tasks such as removing details on texture or blemishes from the scanning process, cleaning stains on sidewalks, or adding scratches on walls so it looks real. A lot of what artists end up doing is beneath them.
This is where our solutions come into play. Our software is capable of ideating, giving suggestions to artists, and inspiring them. But where it adds the most value and helps in improving cost efficiency, is really by doing the repetitive groundwork.
Of course, the machine doesn’t have a sense of emotions, and some of those ingredients I mentioned—i.e., seed data and intent—still come from a human, who then remains in the pilot’s seat.
Artomatix is a front runner in that space. At the moment, we’re focused on materials, with images mapped over 3D geometry. Very soon, we’ll be moving into 3D geometry itself, which means “world building”—building entire virtual worlds.
Chivot: To what extent is creative AI the future of content creation, and where can it be used next?
Risser: The world is currently generating art based on a manual labor model, with Adobe and Autodesk as two leading players. It’s one copy, one human, going hand in hand. But as computers advance exponentially in terms of memory and power, 3D worlds will expand as well. With augmented reality and virtual reality, the demand for 3D content is going to increase even more. In other words, computers are creating way more demand for 3D content than the labor force has supply to keep up with this demand. The bottlenecks of projects are timeline, a lack of human resources, and costs.
This is why I believe creative AI is the inevitable future of content creation. Ten, 20 or perhaps 30 years from now, I hope to hear artists joking about how they used to remove textures and scenes by hand, and how everything was completely manual and so tedious. This industry is bound to transform itself, the way we went from using sailing ships to travel across oceans to using airplanes.
Ultimately, the whole point of computers is that they can automate tedious, repetitive tasks. We’re moving towards automating more and more of these tasks. Computers will do the low-level artwork, while artists will be able to focus more on the content and the strategy, and this is what the future of creation looks like.
A similar evolution happened in the accounting business, with spreadsheets. In the eighties, accounting was about getting really good at multiplying numbers, and you needed a lot of people to do this manually—they would make mistakes, because tasks would be boring and repetitive. For every accountant doing the interesting strategy side, such as optimizing for tax efficiency, you needed a team of say 10 people doing basic maths all day. With this groundwork being done automatically, the accountant has become a more high-profile, more interesting, more strategic job. This transformation elevated the profession. That’s the future I see for artists in all spaces—music, 3D modelling, texturing, and animation, the whole design world—computers and artists will work through a co-creation process, giving each other suggestions, feedback, and iterations.