Good on you for documenting and giving concrete examples of the limitations of the software on the homepage. Looks like a valuable tool for some use-cases.
Agree. Too many companies sell you on pretty GUI and some functionality yet skip over cases where their software just doesn't perform as well or as well as the sales make it seem like it would.
I've been wanting something similar to this for a long time.
Imagine a world in which even composition can be available to anyone, simply based on sound rather than theory. Decentralization of composition is at hand. Well done!
Man, this is absolutely amazing! I guess this is how (approx) Shazam and co works.
So, if each of these "notes" could be treated as entry in what used to be a .MOD or MIDI files, but I believe you would need to create a library of sounds for the latter. Hello old-new compression tech ?
I had been looking into music transcription a while back when toying with MATLAB back in the day, but simply didn't have the appropriate programming skills to use another language that would compile to something more user friendly.
What did you use to make this programs, what sort of libraries/extensions? Also, as far as algorithms go, what's the general flow of it? Read in the input and break it into the frequency domain and use visual tools to map them to the spectrograph? Feel free to not answer (or keep things simplified without detail) to protect your IP.
Finally, what's next for your project? Music transcription directly to music writing software like Sibelius or whatever composition programs people are using nowadays to make musical arrangements? Or are you looking to enter that space for yourself?
I developed the algorithms in python, which is great for fast, iterative development. Then I re-wrote it all in C++, using Qt to decode audio, but I didn't use any 3rd party libraries on the algorithm side. The spectrogram is a variation on the Constant-Q transform (Brown, 1991), so instead of the Fourier transform, which has linearly spaced frequency bins, the frequencies are spaced exponentially, like the human ear. Finding the notes from the spectrogram is surprisingly challenging, given how easy it is to pick them out using your eyes and brain, and obviously there's still a lot of room for improvement on that side.
I tried to integrate it with MuseScore notation software. If the user has that installed on their machine, it will open the result in MuseScore. In the future I'd like to improve the accuracy of the note detection part. That's the area that could use the biggest improvement.
Interesting, I have to say I'd never heard of the Constant-Q, I'll have to read up on it. I ran into similar issues trying to translate frequency to notes. It's definitely ambitious as a project and I'll be keeping tabs on it. Good luck.
This seems like it'd be particularly useful for songwriters, since you could just play the music and have the program write it down for you, to clean up and otherwise adjust later.
This is awesome. What's the accuracy like? I'm gonna try it out even though I can't read sheet music. It would be awesome to see someone use this program with some kind of addon that could transcribe songs into playable tracks on games like RockSmith or Guitar Hero.
Thanks, you can see a couple examples of accuracy on the home page. It generally does pretty well for slow-medium tempo instrumental songs, but not as well for fast songs with a lot of percussion/distortion. It creates a MusicXML file, which can be directly converted to midi, so it should be possible to play in Guitar Hero, assuming they use MIDI files.
If it has trouble with fast tempo, is it not possible for it to slow down the track for analysis (making a record of how much it slows it down by) then as a second-pass speed up the result?
I'm quite a noob when it comes to musical theory, so that might be a silly question.
That's a good idea, but the challenge is separating notes from other stuff like noise and drums. Really short notes are harder to separate and in order to slow down the tempo, you either need to stretch out the drums and noise as well (which means it's still just as hard to separate them) or figure out a better way to separate them before you slow it down, in which case you've already got the better separation before you slow the tempo.
Wow! The program looks amazing! I tried using it with the opening piano lick to "Feelin' Alright" by Joe Cocker, and the result was, well, nothing like the song. :( There are bongo drums and another percussion instruments at the beginning that may be throwing it off... Here is a link to a small OGG file I used to test it with. I'd love to get your feedback.
That's interesting. Thanks for the feedback. I looked at the clip and it's a pretty simple song, which I would expect the program to do a decent job on. In the visualization window, I can see the main notes (C, G, Bb) at the beginning. But I can see a lot of partials from the bongo drums that are probably being mistaken for note harmonics. Bass drum is easier for it to ignore, but the bongos are semi-melodic, so they kind of look and sound like notes and the program gets confused.
Thanks. I started it as a side project a couple years ago. It was fun to work on and I learned a lot about frequency analysis, Fourier transforms, etc.
26 comments
13 u/onegin 12 Jan 2016 18:28
Good on you for documenting and giving concrete examples of the limitations of the software on the homepage. Looks like a valuable tool for some use-cases.
1 u/AreWeHuman 13 Jan 2016 11:01
I feel like a lot more developers/companies that sell software should be this honest. It's definitely a plus in my book.
1 u/WhiteRonin 13 Jan 2016 17:27
Agree. Too many companies sell you on pretty GUI and some functionality yet skip over cases where their software just doesn't perform as well or as well as the sales make it seem like it would.
4 u/antiplebbitor 12 Jan 2016 18:30
Awesome concept.
I've been wanting something similar to this for a long time.
Imagine a world in which even composition can be available to anyone, simply based on sound rather than theory. Decentralization of composition is at hand. Well done!
3 u/atat 13 Jan 2016 02:40
... what kind of magic is this ...
Man, this is absolutely amazing! I guess this is how (approx) Shazam and co works.
So, if each of these "notes" could be treated as entry in what used to be a .MOD or MIDI files, but I believe you would need to create a library of sounds for the latter. Hello old-new compression tech ?
2 u/jimmyrussel 12 Jan 2016 19:06
I had been looking into music transcription a while back when toying with MATLAB back in the day, but simply didn't have the appropriate programming skills to use another language that would compile to something more user friendly.
What did you use to make this programs, what sort of libraries/extensions? Also, as far as algorithms go, what's the general flow of it? Read in the input and break it into the frequency domain and use visual tools to map them to the spectrograph? Feel free to not answer (or keep things simplified without detail) to protect your IP.
Finally, what's next for your project? Music transcription directly to music writing software like Sibelius or whatever composition programs people are using nowadays to make musical arrangements? Or are you looking to enter that space for yourself?
3 u/AnthemScore [OP] 12 Jan 2016 20:14
I developed the algorithms in python, which is great for fast, iterative development. Then I re-wrote it all in C++, using Qt to decode audio, but I didn't use any 3rd party libraries on the algorithm side. The spectrogram is a variation on the Constant-Q transform (Brown, 1991), so instead of the Fourier transform, which has linearly spaced frequency bins, the frequencies are spaced exponentially, like the human ear. Finding the notes from the spectrogram is surprisingly challenging, given how easy it is to pick them out using your eyes and brain, and obviously there's still a lot of room for improvement on that side.
I tried to integrate it with MuseScore notation software. If the user has that installed on their machine, it will open the result in MuseScore. In the future I'd like to improve the accuracy of the note detection part. That's the area that could use the biggest improvement.
1 u/iamrage 13 Jan 2016 04:48
Just curious, would this work with SoundCloud? https://developers.soundcloud.com/docs/api/guide#playing
0 u/jimmyrussel 13 Jan 2016 05:21
Interesting, I have to say I'd never heard of the Constant-Q, I'll have to read up on it. I ran into similar issues trying to translate frequency to notes. It's definitely ambitious as a project and I'll be keeping tabs on it. Good luck.
2 u/BB-3 16 Jan 2016 19:02
This seems like it'd be particularly useful for songwriters, since you could just play the music and have the program write it down for you, to clean up and otherwise adjust later.
1 u/meatspun 12 Jan 2016 22:51
Can't wait to see the next version of this thing after you tweak it more. Cool knowing something like this is being made.
1 u/Wedge38 13 Jan 2016 20:07
This is pretty sweet. I appreciate the work you put into this.
0 u/Codewow 12 Jan 2016 17:35
This is awesome. What's the accuracy like? I'm gonna try it out even though I can't read sheet music. It would be awesome to see someone use this program with some kind of addon that could transcribe songs into playable tracks on games like RockSmith or Guitar Hero.
3 u/AnthemScore [OP] 12 Jan 2016 17:40
Thanks, you can see a couple examples of accuracy on the home page. It generally does pretty well for slow-medium tempo instrumental songs, but not as well for fast songs with a lot of percussion/distortion. It creates a MusicXML file, which can be directly converted to midi, so it should be possible to play in Guitar Hero, assuming they use MIDI files.
1 u/Aradiel 12 Jan 2016 18:47
If it has trouble with fast tempo, is it not possible for it to slow down the track for analysis (making a record of how much it slows it down by) then as a second-pass speed up the result?
I'm quite a noob when it comes to musical theory, so that might be a silly question.
2 u/AnthemScore [OP] 12 Jan 2016 20:02
That's a good idea, but the challenge is separating notes from other stuff like noise and drums. Really short notes are harder to separate and in order to slow down the tempo, you either need to stretch out the drums and noise as well (which means it's still just as hard to separate them) or figure out a better way to separate them before you slow it down, in which case you've already got the better separation before you slow the tempo.
-1 u/0x5f3759df 12 Jan 2016 21:00
All the examples only give piano parts. How good is the discrimination of different instruments? That's the real hard part.
0 u/AnthemScore [OP] 12 Jan 2016 21:10
It doesn't try to recognize different instruments, just notes. That would be cool, but just note recognition by itself is very challenging.
0 u/0x5f3759df 12 Jan 2016 22:02
Well, you do give some explanation of how it could in theory be done, which I think is correct. This PhD thesis might be interesting.
0 u/xaric 12 Jan 2016 18:23
Wow! The program looks amazing! I tried using it with the opening piano lick to "Feelin' Alright" by Joe Cocker, and the result was, well, nothing like the song. :( There are bongo drums and another percussion instruments at the beginning that may be throwing it off... Here is a link to a small OGG file I used to test it with. I'd love to get your feedback.
0 u/AnthemScore [OP] 12 Jan 2016 18:38
That's interesting. Thanks for the feedback. I looked at the clip and it's a pretty simple song, which I would expect the program to do a decent job on. In the visualization window, I can see the main notes (C, G, Bb) at the beginning. But I can see a lot of partials from the bongo drums that are probably being mistaken for note harmonics. Bass drum is easier for it to ignore, but the bongos are semi-melodic, so they kind of look and sound like notes and the program gets confused.
1 u/xaric 12 Jan 2016 18:52
That makes sense. This one example doesn't in any way detract from the awesome work you've done. How long was this in the making?
0 u/AnthemScore [OP] 12 Jan 2016 19:54
Thanks. I started it as a side project a couple years ago. It was fun to work on and I learned a lot about frequency analysis, Fourier transforms, etc.
0 u/elgindelta 13 Jan 2016 10:59
Dude, lemme buy a copy, i sell music that i compose and y hy wiuld be n amazing tool to plug into directly
0 u/vinay02 15 Jan 2016 09:17
you are genius