It would be amazing to enhance the “MIDI Input” functionality in the Sample Editor so that it can be applied in real time.
- Open an audio file in the sample editor and create the segments for VariAudio.
- Select the real-time playing mode. (see explanation below!)
- Hit play in the Sample Editor.
- Play MIDI notes. The audio will follow the playing. Depending on the selected mode, the pitch will be changed for existing segments or new segments will be created, following the MIDI input.
Regarding bullet point No.2: There is -admittedly- one difficult question for the programmers that must be answered beforehand:
What happens if the user plays more than one note during a running segment?
I suggest that there should be two modes: “Pitch only” or “Create Segments”.
“Pitch only”: If the Sample Editor is in playback mode and MIDI input is selected, the pitch of the playing segment will be adjusted to the MIDI note that has been played at last during that segment. I think that referring to the last note is the best way, since usually the player will have a delay in his live playing. Thus, if he is too late or even misses the desired target pitch, he can still adjust his playing as long as the segment is audible. If no MIDI notes are played, the pitch will not be changed.
“Create Segments”: Same as before, but the pitch of the VariAudio is “glued” to the MIDI notes. That means that new segments will be created on each MIDI note, and the pitch changes are auditioned in real time. I see that this might get quite complicated if VariAudio detects a pitch during breathes and other side effects. But on the other hand, this is a very creative tool, so the user will accept some additional effort to make his playing match the audio. If no MIDI notes are played… well… maybe silence? That would come close to a TalkBox effect, but you have to see once the first tests are made. It might also be that we need another checkbox in the UI for that reason, though -admittedly- the function is already complex enough.
I think that this is no more than a one-week sprint (ToDo: 0.5 days, Doing: 2 days, Done = testing and back to previous step: 2.5 days) with two developers in your great team.