AI is striking a chord in the music game industry. Music AI carries more intricate & precise musical synchronization and the more immersive the experience, the more users we retain to enjoy the game. In the last AMA BEATS - Issue #3, we talked/discussed the Music AI, “The technology behind music & gameplay synchronization” - NOTE BOT. This issue will discuss further the implementation of Music AI technology into gameplay. It will describe the process of machine learning and what’s the solutions for challenges regarding the issue following up to make sure the process runs smoothly.
The Power AI/ML
To transform a song into playable action, we must go through a process to analyze and detect Music Variants, such as drum, vocal, melody, chord, and instrument of the chosen song.
And then there’re numerous technical variants that we need to analyze such as
Onset: The time a note is played. Ex. the time a pianist starts pressing a key, the time a drummer hit a drum.
Offset the time a note is stopped
Power: the energy of the sound, we usually use dB (decibel) to measure it
Detecting Music Variants and Technical Variants has to be precise to ensure correct synchronization between gameplay & background music.
Before the Machine Learning system, we must be done all of these tasks in a manual way, which means we need a music expert who listens and write down all of this information. Applying those manual works to thousands of songs is a challenge; applying them to millions of songs is impossible.
We started to build our ML system 2 years ago, and after a thousand improvements, now it delivers the result with great accuracy in detection with a huge performance improvement. Now the process is like a Conveyor belt. All of the machine learning tasks we have optimized to be done in just 90 seconds for each audio file, 1000 times faster than our old process.
The implementation
For us, AI Music is a suite of solutions to store, produce, distribute and analyze a large amount of musical content for multiple apps. Including:
A content production and delivery pipeline to our end-users
A music database with millions of songs
A toolkit that turns each song into a beat map based on musical elements within the song (pitch, highlights, tempo…)
A toolkit that adapts the beat maps to fit different gameplays and the difficulty of the game levels.
On the other hand, we have an automated pipeline which is continuous receive and process new/update from the Publishers
Automatic adding of the newest songs with the highest audio quality that have the right to use by the publishers, also grant the permissions to Amanotes
Automatic update of the newest version of the licenses and this happens as fast as possible.
End users will be able to see the newest songs that Amanotes have the license for the region of the user and this happens automatically in nearly real-time.
Before the automated pipelines, all the licensed songs have been set up manually and it usually took a couple of days. It was also a challenge when we took down a song that had an expired license or changed the region that the license allowed.
Top 3 the Challenges & Solutions
Large configuration files to store the song catalog
It is hard to update the song catalog, also the operation team needs to create many duplicate files when we wanted to serve different song catalogs for each region. Also, the file config cannot store the huge songs catalog, it will be a very heavy file and affect the application's performance.
By serving a microservices backend system that is deployed in multiple continentals, this catalog is dynamic, personalized for each country, and routine updates with the latest popular chart in the world, all Amanotes end users can reach the huge song catalog.
We’re confident to bring the fastest experience to the end users, 99,9% of API requests will be processed in just 100 mili seconds.
Scale up globally
Running a system & make sure it has stability was a challenge, scaling it is more challenging. All 100k songs and then delivering them to the end users is a huge challenge for us.
We have to deploy over 400 servers and run it 24/7 to deliver the catalog as fast as possible to the end users.
Ensure Security.
The cost involved in securing the application is less than a fraction of the potential revenue loss because of a hack. Optimal protection means securing your app against user-side attacks, network attacks, and server-side attacks.
To illustrate, we have secured all our public APIs and encrypted the audio files so Amanotes applications can only decrypt them. On the server, we have tamper-resistant logging and anti-abuse.