Google wants computers to teach composing and witting

Douglas Eck of Google’s Magenta project has spent about 15 years studying AI and music, and these days he’s a research scientist on the Google Brain team, leading Magenta—Google’s open-source research project that’s aimed at making art and music with machine learning.

“But you have to start somewhere. And I think if you make serious progress on something as complicated as music, and as important to us as music, then my hope is that some of that will map over into other domains as well”, says Eck in an interview for the MIT Technology Review.

The researcher points to a musical improvisation of the AI and shows that it produces not only notes, but also decides how fast and how loud the individual notes are played. As a basis for learning, the AI is reproducing what it was trained on, namely, a lot of piano performances done as part of a piano competition.

The next challenge for the team is that the AI not only improvises, but creates longer compositions with an overlapping structure. The researchers of the AI intentionally don’t want to make regulations. “Our goal is to come up with this end-to-end model that figures out all of these levels of structure on its own.” says Eck.

Better, but not good enough

In addition to composing music, the researchers tried to teach the AI the drawing and fed them with a bunch of drawings done by people playing Pictionary against a machine-learning algorithm, another Google AI drawing experiment made by Google Creative Lab.

“Because it can’t memorize all the strokes it’s seeing, its job is just to reproduce lots of cats or whatever, it’s forced to learn what’s important about cats… And so when you play with this model you can ask it to generate new cats out of thin air. It generates really interesting looking cats that look, I think, uncannily like how people would draw cats”, says Eck.

But not all the efforts of the team were successful. Thus the researchers did not succeed in teaching the AI the wit. Eck wanted to invent jokes based on simple pointers and word games. For this purpose, the AI would first have understood the meaning of a sentence and then had to manipulate it in order to achieve a surprising effect. However, this was not successful.

The latest project of the team is about the question of whether AI can learn from feedback. This is a central feature of outstanding artists: “The Beatles had 12 albums and every one of them was different. And they were all showing that these musicians are learning from feedback they’re getting from peers and from crowds, but also other things that are happening with other artists. They’re really tied in with culture. Artists are not static… I think we’re just so, so, so far from this AI having a sense of what the world is really like”, says Eck.