AI Video Game Upscaling: The New FMV?

I have nothing against AI. Is it art? I’ve never been pedantic on the term. Some people will tell you video games aren’t art in the way movies are, yet video games as storytelling has created a new form of storytelling. If storytelling is art, then some games are indeed art, or at least make good use of it when it’s just a game for fun like puzzle games. It’s like Tetris or Bejeweled have been given some deep backstory. Yet. AI has its uses like any other tool. It’s when you try to replace better tools and their users for a given task that we start having problems.

Above you see NVIDIA’s newest GeForce chip, using AI to upscale images to make the people and environments look more realistic, like a physical place. It’s continuing the belief that the pretty pictures are the most important part of the game. When games use this poorly it takes the place of working on the gameplay. On the other hand, more realistic visuals could help a game like LA Noire, which made reading people’s expressions and body movements a central play mechanic. So don’t think I’m trashing or pushing AI here. Artificial intelligence is a tool like anything else and can’t replace the emotion of a human artist or the skill of a human creator, doctor, or anyone else who uses AI in their field.

I do have this issue with “realism”, however. I have a different form of weird paranoia about all this based on what I’ve seen from the entertainment industry and the Hollywood mindset that goes beyond Hollywood into Silicon Valley. It’s a mindset that seems to continue an anti animation trend, the fear of making things look “cartoony” and pushing for “photorealism”. We have seen this before in games, the idea of making it look like you’re operating an actual person in an actual world. Anyone remember FMV?

The above video should give you an idea of Full Motion Video, games that were done with video clips chosen based on a button press. I’m not here to hate on FMV games, either. I have some and they’re pretty fun, but they’re also limiting. Clue basically uses them as cutscenes, which is fine. Crime Patrol and Thunder In Paradise are essentially quick time events, interrupting the current clip where your POV character dies versus one where the bad guy does. Dragon’s Lair and Space Ace on the animation side is the same thing only with cartoons and the QuickTime events keep the current clip going. I’m not going to trash a mechanic when done right. It’s the mindset of the creators I’m worried about, and they said it right there in the clip: the idea that they’re trying to be Hollywood.

In my Art Of Storytelling series I went over how video games tell a story in a way none of the other media formats can. I worry that the video game industry, in subservience to the media pecking order, thinks itself below movies in the same way comics do. They don’t acknowledge that there isn’t one form of media that rises above all others. Movies aren’t even the oldest form of storytelling. After handing down spoken word stories, the play on stage would be earlier, with movies just emulating that with a world that you can usually spin the four walls around. Video games don’t need to be a movie, they need to be video games. If you’re using the story side of games (in times past there was a suggestion to call it interactive entertainment, but that ignores the fact that it’s still a game) you need to do it as a game first and story second.

Undertale isn’t exactly a graphics power house, with a design that goes back to the Commodore line of computers at best. And yet fans get behind the story and the differences between a fully violent run, a fully pacifistic run, or a combination based on the enemy and what you believe is necessary as the player. Shovel Knight is a throwback to the NES and Genesis visually but is still a fun game. I haven’t heard a lot about the story side, but if Ninja Gaiden can provide an interesting story that kept the IP through the next generation of games and storytelling through video games, it’s not impossible. Final Fantasy started out as 2D sprites and told some darn good stories. The graphics alone do not make a strong video game. Mortal Kombat (the early years) and Cuphead use pictures and hand drawings as sprites, but they don’t skimp on the gameplay according to fans, though I haven’t watched a full playthrough or been able to play it myself to see how well it works as a story.

Anybody remember Nerfuls? Look them up and you still won’t see what “realistic” Pac-Man would look like because he can’t exist in the real world.

There’s a reason FMV games became a joke: the gameplay. When the creators focused on the visuals the stories turned out bad. Even modern games suffer when the focus is on the visual. How much development time is spent on trying to make 3D characters look real at the cost of the time needed for solid play mechanics and level design? Thrown in shoving in the latest feature du jour whether it fits this game or not, and all the dirty ways to monetize past the already increasingly excessive list price like DLC and loot crates, and the game is lost to buggy mechanics. Look at the more recent Star Wars games like Outlaws, one of the most infamously broken games released in recent memory.

I looked through a bunch of responses and the division seems to be more along the pro/anti AI side in general than actual perspectives on the change itself. In this case it does get into one of the AI worries in that it comes off lazy. Using the AI means you don’t have to spend more time on the graphics and getting each character and asset correct. That doesn’t sound like a bad thing for cutscenes and would allow more time to get the bugs out and mechanics/level design improved, but as others pointed out the computer would have to be able to run this as in-world game graphics. That would be more intensive and need yet stronger computers.

Anybody remember the original Crysis, a game with such high level graphics that it was a joke for years about how powerful your computer ran. “My computer can solve Pi to 5000 numbers and still launch the space shuttle and write Shakespeare at the same time.” “But can it run Crysis?” AI still isn’t ready to make the end results, and many artists find it insulting that they try. Apparently you aren’t allowed to make anything if you can’t afford it. That’s a whole other debate, and worth having. I’m still sorting my opinion out to anything definite enough to write about as everybody has a point, but as one commentator I follow notes, AI is as bad as it’s going to get. It’s just that right now it’s still pretty bad. You can still tell they aren’t real by the way they move even if there aren’t bugs of their own like clipping, extra parts, eyes that wander like Mass Effect: Andromeda characters, and those other tells that VFX artists warn about when judging Hollywood computer animation effects.

Which brings me to my point: animation. That supposed The Lion King demake wasn’t live-action. It’s was CG characters against a real world backdrop. It was still animation. They recently announced canceling a Robin Hood remake. DISNEY’S Robin Hood. You know, this:

I don’t care if you used AI or CG with a dedicated animation team. There is no way you’re pulling this off in “realistic” visuals. You’d be better off doing regular Robin Hood, which at that point you shouldn’t bother looking to the cartoon. Ah, that word: “cartoon”. Hollywood hates cartoons, and there are game developers who seem to hate making something that looks like an animated work, which video games are. That’s how we got FMV in the first place. The problem is still the same as choosing between live and animated actors. You still lose that sense of “unreality”, limiting how fantastic your world can be.

Since the Hollywood mindset seems transfixed on being “grounded in reality” even in a fantasy world (and usually a narrow view of “real”), any game creators with the same mindset, political or otherwise, falls into that same tone. Would Mario look better real? No, it would look wrong. Pac-Man is either a circle with a wedge mouth or a sphere with a big nose. They made Sonic closer to the game designs, which they got away with in the movie because they’re from another world, because the humanoid hedgehog they wanted to make would frighten small children AND their parents.

Using the unreality of animation allows you more visual freedom to create a believable world, and the more realistic you try to make monsters, fantasy creatures, and superheroes the harder it is to believe in their more fantastic aspects. It’s why I enjoy animated superhero stories more than live-action ones, while doing the same with sci-fi or fantasy depends on the series. There is a place for what NVIDIA is trying to do, but ultimately I see nothing wrong with making a video game look like a video game. With a good story and good game design it’s plenty to get the player invested in the world. Making a character ugly on purpose is dumb, but convince me a character is beautiful in that world and I’ll go along with it. Look at Lara Croft’s history. Graphics and graphical abilities are always going to get better. It’s the nature of the beast, not the enemy. It’s also not the most important part of your game. Embrace the unreality in the right ways, genres, and formats and you’ll find it much easier to dive into the fake world.