Music

In our laboratory, we’re researching the theme related to “Music”. The technology for enjoying music has evolved over the last few decades, and for further music recognition/understanding, the researches for music with machine learning (deep learning) techniques is being conducted.

With the advent of phonograph records, we can enjoy the music whenever you want. Also, with the proliferation of digital audio technology, we can enjoy the music more convenient with mobile devices.
With this evolution of technology, music has become easier to “listen to”. We think the next step of this research is to “extend the way of enjoying the music”. From the next paragraph, we introduce the research for the technology to proceed to the next step.

●● New discoveries, encounters with music

In this decade, with the spread of video/music streaming services, everyone has been able to release and share music content. The number of contents is exploding, making it difficult to find the music that the user prefers.
To solve this issue, we are thinking the music recommends a system that can find the music contents preferred by the users prefer easily. To realize this system, the researches of “music analysis” and “music expression” are indispensable, From next, we introduce examples of this researches.

● Music Analysis

We are researching “Music analysis”, which estimates the feature of the music such as musical instruments, melody, rhythm and so on.

・Musical instruments estimation

　From the music, we estimate the musical instruments which the music consists of. The composition of the instruments in the music is an important factor that characterizes the music. This research can be applied for researches such as “musical genre estimation”, “automatic transcription”, “music research” and so on.

・Scale estimation / Key estimation
　From the music, we estimate /extract the musical scale, tempo, musical key and so on.

● Music Expression

We are researching “Music expression”, which express music not just a sound, but an interpreted form, like the musical genre.

・Genre estimation
　We are researching “Musical genre estimation” that estimates the musical genre (pops, rock, classic, etc) from the feature of music. The musical genre is considered one of the indicators for classifying/searching for a large amount of music content.

・Music recommendation system

We are researching “music recommendation” that searching and presenting the music the user prefers. We usually search the music based on the artist’s name or musical genre. However, this searching condition is very restrictive and we can miss the encounter of whole new music contents that the user will prefer. As a type of encounter these contents, we are researching the system that introduces a whole new music content from the model which learns the user’s preference in advance. We are working to support users’ new encounters with music by combining comprehensive information on music like lyrics or mood and personal preferences.

●● Make the piano practice convenient

Music analysis technology is now starting to be used in music education. In order for the person having the music education to improve their performance skills more enjoyably and efficiently, it is necessary to increase the time for individual practice without the guidance of teachers with accurate and motivating educational content. We have been researching on the development of the support systems for piano performance, especially for beginners, who can easily and efficiently learn at home and promote their performance.

i) Individual Practice Support System

We are researching the system that presents the performed melody and rhythm visually by plotting the performed sound in relation to the musical score. Using this system, the performer can know where to practice more and can practice the melody and rhythm efficiently by repeatedly practicing the weak points on the performance. This system is expected that the performers can achieve faster progress.

For acquiring the performed musical information such as the melody, rhythm, tempo and so on, we are using the deep learning algorithm, like Deep Neural Network(DNN), Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), Temporal Convolutional Network (TCN) and so on. We expect these algorithms to make more accurate and robust musical information acquisition than the conventional method.

We are focus on not only the music analysis but also the higher visibility display system. The system with easy to understand will make the individual practice enjoyable and progress faster. We are also researching how to present the result of music analysis.

ii)Piano Practice Support Contents “Video Text”

We made the video contents “Video Text” which is consists of videos of performance from three different viewpoints and musical scores displayed in conjunction with the videos. Using this video, the performer can check the points that are important for performing the piano, such as points to be aware of when performing and how to use fingers, even if in the situation of individual practice.

Conclusion

We introduced a few techniques for “expanding the way of enjoying music”. With machine learning / Deep learning, the technology for enjoying music continues to evolve to the next step.