Abandoning Automatic Alignment (for the moment)

I’m well past my set deadline for getting automatic phoneme alignment working. The current approach is simply too brittle.

So I’ll just set the default alignment based on the beat duration, and manually adjust the phonemes. It’s a lot more work, but it’ll get me back on track again.

Working on Phoneme Alignment

I’ve been spending a lot more time than I’d like working on phoneme alignment.

The main problem is that vocals aren’t always clean. For example, there are often states that should be ignored as a phoneme moves from one state to another, like with a release of vowel into silence. As the energy drops, the identification may move from voiced to fricative, with a couple of frames dipping to silence, before it comes to actual silence.

Similarly, frication may appear between silence and voiced frames at the start of phonamation.

Voiced phonemes like /b/ may look like an unvoiced /p/, and the “stop” on stop consonants may not appear.

All this makes it a complex problem to solve. So the backtracking needs to be aware of variants for phonemes. Here, you can see multiple combinations for stop consonants:

  ["B"]   = { class = TYPE_VOICED, min = 1, options = {
    {"STOP", "B"}, 
    {"B"}, {"STOP", "P"}, 
    {"P"}, 
    {"STOP"}
  } },
  ["D"]   = { class = TYPE_VOICED, min = 1, options = {
    {"STOP", "D"}, 
    {"D"}, 
    {"STOP", "T"}, 
    {"T"}, 
    {"STOP"}} },
  ["G"]   = { class = TYPE_VOICED, min = 1, options = {
    {"STOP", "G"}, 
    {"G"}, 
    {"STOP", "K"}, 
    {"K"}, 
    {"STOP"}
  } },

Obviously, within a run of a particular class, there may be multiple phonemes. Here’s an example where /t/, /s/ and /h/ appear:

/T/, /S/ and /H/ appearing in a row.

The code can handle complex sequences fairly well. For example, /l ae f s – g r ae s – y ea m/

The reclist string /l ae f s g r ae s y ae m/

Still, there are many cases where the code fails because of some unexpected pattern. So I’ll keep working on this, and try to figure out the best fallback strategy on failure.

Back to Zero Crossings

I’d been using the “color” band ratios to determine the type of voicing, but with changes in the code, it ceased to work properly.

After far too much time trying to get it to work, I’ve decided to fall back to using zero-crossing data to determine the general category (silence/fricative/voiced) and the “color” bands to determine the which voiced phoneme I was looking at.

The main thing that turned me away from that approach was that voiced fricatives would occasionally be above the zero line, making zero crossing fail.

The simple – and obvious solution, in retrospect – was to calculate a DC offset to move the samples back down to where they can be properly detected.

The values look sensible, so the next step is to see if it can differentiate voiced phonemes from each other.

Bootstrapping Phoneme Recognition

Since I don’t have the energy – at this time – to properly debug the Viterbi code and find out why it didn’t “learn” the phonemes, it’s time to consider my next options.

The first – and simplest – option would be to create a small reclist of phonemes, manually flag them, and have the analyzer load that data.

That’s the route I was going to take, but I’ve decided to take one more plunge and see if I can get this program to bootstrap. The work I put into coding to avoid working sometimes amazes me.

The first thing to do was add back in the “missing” colorband, so there was enough information to recognize phonemes. Gemini suggested including the frequency of the peak vowel band, which seemed sensible.

The next problem was coming up with a “bootstrap” reclist. The code could already recognize stop consonants (via their silence, of course), and frication. I worried for a bit that I might need to distinguish between two adjacent fricatives, like “of five”, but in real life, those just blend into a single phoneme.

And when they are distinct, that’s because the movement of the tongue puts in a stop. So I’ll deal with it later, but I don’t think it’ll be a real issue.

My zero-crossing code can already distinguish between silence and… well, not silence. So the simplest thing to do is create phoneme separated by spaces.

So I split the phonemes into vowels and voiced consonants, then further split the vowels into monothongs and diphthongs. My original list had the following diphthongs:

AY sky
AW cow
EY hey
IY tea
OY boy
OW grow
UW blue

So then I recorded them to see how easy it would be to automatically split the diphthong. And I noticed something weird: A lot of these didn’t look like diphthongs.

Sure, they had a light release at the end. But for the most part, the following looked and sounded a lot like monothongs. In fact, the only true diphthongs seemed to be:

AY sky
AW cow
OY boy

For the of the “fake” diphthongs, I can just look at the leading sound for matching:

EY hey
OW grow
UW blue

That leaves the following monothongs:

AA pa
AE cat
AH up
EH fed
ER bird
IH big
IY tea
UH book

Then I started looking at the liquid consonants: /l/, /r/ and /w/ and /y/.

Since I planned on splitting consonants at a stable point, I had selected the end of the glide as the split point. That was easy on the /l/ and /r/, especially if I used a trailing consonant, like car or fall.

Problematically, there didn’t seem to be a good way to find the target portion I wanted with the W or Y. They were weak, and I couldn’t just pick the final sound of the syllable, because it included a release.

Then I noticed the diphones had the same targets:

• /ay/ ends on the /y/,
• /aw/ ends on the /w/; and
• /er/ ends with /r/

Problem solved. I’d just use the same targets. Even less “bootstrap” phonemes to record.

For the consonants, I’m skipping the stop consonants, because the stops are more reliable indicators than anything else. That leaves a very short list:

THEM - DH eh M
SHAWN - SH aa N
VING - V ih NG
FALL - F aa L
SAYZ - S eh Z
LEIGE - L ey ZH

So that’s the current plan: record the minimal reclist, have the code extract the phonemes, and on to the next step… hopefully.

Identifying Sounds Quality Revisited Yet Again

After dragging my feet too long, I ran the Virterbi code. The result was as I expected: terrible. I have a pretty good idea what the issues are, and I really don’t want to spend the many hours getting it to work.

At least, not at this moment.

It’s starting to look like manual alignment would have been much, much faster.

But I’m not quite ready to give up on some form of auto-alignment.

So I took a big step back and asked: What are the things that are easy to measure?

One simple metric is zero crossings: the number of time a wave passes through the x-axis. On a vowel, it looks something like this:

A vowel crosses the x-axis on a regular basis.

But compare that with a fricative, like an /s/:

An /s/ crosses much more frequently. Product is larger than appears in package.

Now, if you find the min/max value between zero crossings, you can get the average peak. A vowel has less crossings, and larger peaks. A fricative has more crossings, but smaller peaks. Silence looks like a fricative, but has much smaller peaks.

So we can easily determine between the following:

• Silence
• Fricative
• Voiced sounds

Using these, it’s easy to find dependable “signposts” in a vocal signal to identify where particular kinds of sounds are, like fricatives or stop-consonants, and even vowels.

There’s one more super-easy to implement tool: auto-correlation. It’ll tell you how much a signal looks like a nearby signal (and what that distance is). It’s useful for a lot of things, such as:

• Voiced sounds have high auto-correlation values; unvoiced have low values;
• You can use the peak correlation value to determine the pitch.
• You can look at auto-correlation changes to determine when a sound is changing, or stable

In theory, using a combination of these (and watching the auto-correlation values for peaks), I should be able to determine where sounds change, and therefore where some types of sounds are.

Unfortunately, it’s not enough to for a lot of sounds, so I started looking elsewhere.

So it’s back to the “color” bands. Note that the top of the voicebar/bottom of the vowel band is now set to F0:

-- perform the FFT
AudioTools.forward(fftData, false)
-- multi-band analysis
local binFreqStep = sampleRate / fftData.N
local powerScale = 2.0 / (fftData.N * fftData.N * 0.375)
local bandPeaks = {1e-12, 1e-12, 1e-12}
-- Band Mapping:
-- [1]: Murmur/Voicebar (< F0 Hz) 
-- [2]: Vowel Band (F0 - 3000 Hz)
-- [3]: High/Fricative (> 3000 Hz)
for i = 0, fftData.HALF_N do
local r, im = fftData.work_r[i], fftData.work_i[i]
local rawPower = (r * r + im * im) * powerScale
local freq = i * binFreqStep
if freq < 200 then
  if rawPower > bandPeaks[1] then bandPeaks[1] = rawPower end
elseif freq < 3000 then 
  if rawPower > bandPeaks[2] then bandPeaks[2] = rawPower end
else
  if rawPower > bandPeaks[3] then bandPeaks[3] = rawPower end
end
end
-- Convert to Log Scale
for bandIdx = 1, 3 do
  bandPeaks[bandIdx] = math.log10(bandPeaks[bandIdx])
end

Since the frequency of zero-crossings directly correlate with the frequency of the band, this is basically doing the same sort of thing that I was doing with zero-crossings, only in the frequency domain.

In order to make good decisions about whether a sound is most like a nasal, vowel or fricative, I analyze the sung word “shun”, which contains and example of all three:

• /n/ supplies a nasal murmur example for band 1;
• /aa/ supplies an open vowel example for band 2;
• /sh/ supplies an example of frication in band 3

The /n/ is searched for the frame with a maximum bandPeak[1]. Similarly the /aa/ is searched the frame with the maximum bandPeak[2]. However, the /sh/ is the average of the frames’ bandPeak[3], since it’s more noisy and diffuse over the bands.

The tilt is calculated using the bandPeak values from the same frame (except for the frication, which is an average). These are stored in a lookup table, although there are fallback values for the routine to bootstrap itself:

calib.vowelTilt   = -0.6    -- The maximum (bandPeak[1]-bandPeak[2]) during a clear 'AA'
calib.nasalTilt   = 0.5     -- The maximum (bandPeak[1]-bandPeak[2]) during a clear 'N'
calib.fricDiff    = 0.2     -- The average (bandPeak[3]-bandPeak[2]) during a clear 'SH'
calib.definition  = 0.5     -- The maximum (bandPeak[2]+bandPeak[3])/bandPeak[1] during a vowel
calib.murmurFloor = -4.0    -- The maximum value of bandPeak[1] during clear 'N'   
calib.vowelFloor  = -4.0    -- The maximum value of bandPeak[2] during clear 'AA'
calib.fricFloor   = -4.0    -- The average value of bandPeak[3] during clear 'SH'

Note that these values are stored as logarithmic values, so what looks like addition and subtraction is actually multiplication and division:

0.1 ~1dB (1.25x power)
0.3 ~3dB (2.0x power)
0.5 ~5dB (3.0x power)
1.0 10dB (10.0x power)

With that, we can make decisions about voicing:

local function interpret_colorbands(bandPeaks)
  -- 0. DECISION BOUNDARIES
  local nasalBoundary = (calib.vowelTilt + calib.nasalTilt) / 2 -- halfway between vowel and nasal
  local vowelCeiling  = calib.vowelTilt + 0.4
  local fricThreshold = calib.fricDiff - 0.1 
  -- 1. SIGNAL DETECTION
  local hasMurmur = bandPeaks[1] > calib.murmurFloor
  local hasVowel  = bandPeaks[2] > calib.vowelFloor
  local hasFric   = bandPeaks[3] > calib.fricFloor
  -- 2. SPECTRAL SHAPE
  local tilt       = bandPeaks[1] - bandPeaks[2]
  local fricDiff   = bandPeaks[3] - bandPeaks[2]
  local currentDef = (bandPeaks[2] + bandPeaks[3]) - bandPeaks[1]
  local relClass = "liquid" 
  -- 3. CLASSIFICATION HIERARCHY
  if not hasVowel and not hasMurmur and not hasFric then
    relClass = "silence"
  elseif hasHigh and fricDiff > fricThreshold then
    relClass = "fricative"
  elseif hasVowel or hasMurmur then
    -- Only allow closure if there is a MASSIVE drop in upper energy
    -- and a significant shift in tilt toward the bass.
    local isVeryMuffled = tilt > (calib.vowelTilt + 0.5) -- 5dB shift toward bass
    local hasLostDefinition = currentDef < (calib.definition - 1.0) -- 10dB drop in color
    if isVeryMuffled and hasLostDefinition then
      relClass = "voiced_closure"
    elseif tilt < vowelCeiling then
      relClass = "vowel"
    elseif tilt > nasalBoundary then
      relClass = "nasal"
    else
      relClass = "liquid"
    end
  end
  return hasMurmur, hasVowel, hasHigh, relClass
end

Getting this to work well required some additional work. The first issue was that occasionally single frames would randomly be wrong, like a voiced_closure appearing in the middle of a vowel.

The solution was to weight the values by including the prior and next frame:

value = (prior_band * .25) + (current_frame * .50) * (next_frame * .25)

The removed the jitter.

The next issue was a bit more complex: frames were being identified as nasal that should have been vowel.

The root cause was the resolution of the FFT. The frames are only 10ms in length. At a sample rate of 44100, that meant only 441 samples/frame. To prevent the next frame from “leaking” into the current frame, I used an 512 point FFT.

That was correct, but it gave a bin resolution of only 86Hz.

The result was the code was unable to between the F0/nasal murmur which belonged in band 1, and a low vowel’s first formant, which belonged in band 2.

The solution was to use an FFT of size 2048, and zero pad the data. I’d already been windowing the signal to avoid “ringing” from the abrupt start and stop.

This gave sufficient resolution to distinguish the energy between the first two bands, while I kept the 512 point FFT to calculate the average of the third band.

Once that was done, the code was properly able to distinguish between the voicebar and vowels with low energy.

So now I can pass in a waveform like this (image from Praat, with lettering from MS Paint):

“jang – fact – thanks”, in case you’re wondering. Yes, it’s a nonsense word from a reclist.
And yes, that last /n/ should be /ng/.

And have it identify the key points:

Seconds     Class       Annotation by me
-------     ---------   -------------------
0           silence     leading silence
0.5         fricative   burst of the /jh/
0.55        vowel       start of /ae/
0.97        fricative   start of the /f/
1.1         vowel       start of /aa/
1.29        silence     stop preceding /k/
1.43        fricative   burst of /k/
1.6         vowel       start of /ae/
1.85        silence     stop of /k/
1.94        fricative   start of /k/
2.17        silence     end of /s/

But while this is mostly accurate, it’s incomplete. For example, it fails to catch the nasals from 0.87-0.95 and 1.80-1.89, which are clearly visible in the spectrogram as having a low nasal murmur (B1) with very little in vowel band (B2). So that’s just wrong.

It’s also unable to distinguish between the burst of the /k/ and the /s/, but that’s because it isn’t there.

So this code is performing fairly well, but there are hard problems that it won’t be able to solve – at least, not without adding autocorrelation. Because “vowel” by itself isn’t enough to distinguish an /l/ from and /aa/.

Even with only this, it should be enough to put the phoneme markers in the general ballpark, which is what I really need it to do.

Refining Voicing Classification

While the voicing classification was sort of working, it was much too fiddly.

To recap, the code needs to know the difference between “full” voicing (with all the harmonics) and “voicebar” or “murmur” voicing, where only the first harmonic is audible. This happens with voiced fricatives, like /v/ and /zh/.

After spending way too much time manually adjusting values, I realized that this code would immediately fail when presented with a different singer. What to do?

The obvious answer was to add a pre-processing pass. Since vowels make up more than 90% of the audio, I should be able to sample the audio, and come up with reasonable estimates, right?

Well, not so much. The results were not good.

The next idea was to try to detect runs of “murmur” voicing frames, which should normally only last a short time. These would likely belong to vowels, and the code could get average values from them.

After multiple attempts with that, it became clear that that this approach was flawed – likely because the detection code simply was looking at the wrong things.

So it was time for a different approach at creating a classifier. My key question was: if I can hear the harmonic frequencies, why can’t my classifier? And not only could I hear them, but they were stunningly obvious on the spectrogram, too.

I already had the code for determining “color” bands, which I altered slightly:

• Murmur/Voicebar (< 200 Hz)
• Low-Mid/Color 1 (200 – 1000 Hz)
• High-Mid/Color 2 (1000 – 3000 Hz)
• High/Fricative (> 3000 Hz)

So instead of only summing the energy into the bands, I tracked the highest value. This prevented the peak from being diffused by low-energy bins in the same band:

  -- loop through the FFT
  for i = 0, fftData.HALF_N do
    -- get the real and imaginary components, and covert to magnitude
    local r, im = fftData.work_r[i], fftData.work_i[i]
    local rawPower = (r * r + im * im) * powerScale

    -- calculate the frequency of the bin
    local freq = i * binFreqStep

    -- calculate power
    totalPower = totalPower + rawPower
    if rawPower > globalMaxPower then globalMaxPower = rawPower end

    -- scale the power by the frequency. we use this to calculate the
    -- centroid frequency of the power
    sumFreqPower = sumFreqPower + (freq * rawPower)

    -- determine which bin to place this in
    -- note we store the _peak_, as well as the sum
    if freq < 200 then
      bandPowers[1] = bandPowers[1] + rawPower
      bandCounts[1] = bandCounts[1] + 1
      if rawPower > bandPeaks[1] then bandPeaks[1] = rawPower end
    elseif freq >= 200 and freq < 1000 then 
      bandPowers[2] = bandPowers[2] + rawPower
      bandCounts[2] = bandCounts[2] + 1
      if rawPower > bandPeaks[2] then bandPeaks[2] = rawPower end
    elseif freq >= 1000 and freq < 3000 then 
      bandPowers[3] = bandPowers[3] + rawPower
      bandCounts[3] = bandCounts[3] + 1
      if rawPower > bandPeaks[3] then bandPeaks[3] = rawPower end
    elseif freq >= 3000 then 
      bandPowers[4] = bandPowers[4] + rawPower 
      bandCounts[4] = bandCounts[4] + 1
      if rawPower > bandPeaks[4] then bandPeaks[4] = rawPower end
    end
  end

  -- dividing by total power brings back an approximate frequency
  local centroid = sumFreqPower / totalPower

The other major change – and the reason for having the bandCounts – was to scale the value the same as a spectrogram (and human hearing) would, using log10 values. This should make it clear when there were vowels (color bands 2 and 3), which should then be treated not as murmurs.

I also added a special case for aspiration. The prior code was treating it as “murmur”, so the vocoder was adding a voiced pulse which made it softly go “ziiiiiip“.

This got much better results. It’s not perfect, and there are still some “magic” numbers in the code.

I can hear an occasional “thump” as the move from voiced to unvoiced fails to transition gracefully. That’s probably an issue with the “inertia” calculation. When moving from a “voiced” frame to an “unvoiced” but noisy frame, if the voicing cuts out too quickly, the noise fills in the shape, and there’s an audible artifact.

But for 95% of the audio, it sounds quite good, so I think it’s a good place to stop and go back and go back to the automatic phoneme segmentation code.

Voicing Decisions

One of the problems I’ve been encountering with the new vocoder has been one of voicing. The vocal cords aren’t always running at full throttle. That means that at the start of voicing, the first few pulses only produce the first harmonic, and then the remaining harmonics are added.

That’s because it takes time and effort to get the vocal cords going.

The logic for this isn’t terribly difficult. Just set an amplitude threshold, and when the amplitude of the pulse falls below that point, set the counter to zero. Once the amplitude rises above the threshold, start incrementing the counter. The logic looks something like (there’s probably a bug in the calculation of t):

-- increment the counter?
if amplitude < MINIMUM_AMPLITUDE then
  -- clear the pulse count, we're only generating one harmonic
  pulseCount = 0
else
  -- past the threshold
  pulseCount = pulseCount + 1
end
--  how many harmonics?
local harmonicsCount
if pulseCount <= 6 then
  -- still only one harmonic
  harmonicCount = 1
elseif pulseCount > 6 and pulseCount < 12 then
  -- start adding harmonics
  -- ratio for the lerp
  local t = math.floor(pulseCount / 12)
  -- lerp the number of harmonics, use non-linear lerp
  harmonicCount = lerp(1, MAX_HARMONICS, t*t)
else
  -- full harmonics
  harmonicCount =  MAX_HARMONICS
end

Now, the actual pulse count should likely be a function of time, but I’ll look into that later.

The above code prevents the harmonics of the pulse from overwhelming the initial attack. Otherwise, you get a “buzzy” sound at the start of the voicing, and that sounds bad.

The next problem is a bit more difficult: in addition to the “ramp up” that happens when voicing passes an energy threshold, there are also “murmurs”, where there is only a voice bar present and no higher harmonics. This often happens under voiced fricatives like /v/ or /zh/.

So in addition to producing voicing, it needs to be the right kind of voicing: full, or nasal/murmur.

Of course, this can only happen when the analysis code has enough information.

After multiple iterations, I’ve got the following states:

• Silence (0 Hz)
  totalPower < SILENCE_THRESHOLD (1e-9)
  
• Fricative/Unvoiced (1200 Hz)
  highPowerSum > (lowPowerSum * 100.0) OR High Spectral Flatness
  
• Nasal / Murmur (1500 Hz)
  lowPowerSum > (highPowerSum * 5.0)
  
  
• Voiced/Vowel (Nyquist)
  High Harmonic-to-Noise ratio; Low Spectral Flatness.
  

These all trigger slightly different behaviors. For example Fricative/Unvoiced differs from Silence in that Silence will generate nothing, while Fricative/Unvoiced will prevent harmonics from being generated,

At this point, things are still going hilariously wrong, with the detector constantly confusing Murmur with Vowel, yielding things like:

Those bands are the vocoder toggling randomly between murmur and full voicing.

Back to debugging!

Working On A New Vocoder

Yeah, you read that right. In the middle of working on the phoneme alignment code, I’ve decided to take a step back and make some major changes to the vocoder.

The prior method was to create the harmonics by summing the waves, one sample at a time. That’s a slow way of doing things, but there are a lot of advantages to doing things that way.

Still, I was had a nagging feeling: what if I just generated a glottal pulse, like everyone else?

So I went down that rabbit hole, and it seemed like it could be done pretty easily with my current framework. The core process for each pulse is:

• Generate an LF pulse of the correct length for the desired frequency.
• Put it into an FFT.
• Shift it so the MGC (minimum glottal closure) is in the center of the FFT frame.
• Perform an FFT to convert it to frequency space.
• Multiply the amplitudes by the spectral envelope amplitudes, but leave the phases derived from the LF pulse alone.
• Peform an IFFT.
• Take the entire “grain” and add it to your output buffer.
• Lather, rinse, repeat.

That’s the core idea, anyway. And it works pretty good.

But it turns out to be one of these 90% solutions. It mostly works, but there are problems. And so the endless game of Wack-A-Mole begins.

There are problems like this:

See that burst of high frequency harmonics following the /S/?
It’s not supposed to be there.
And the /M/ that follows isn’t supposed to be messed up, either.

The above code is caused by the periodicity code thinking that the silence is a vowel, because…

Well, it’s been a long couple of days, and I’m sure the grind will go on for some more days.

Working on Phoneme Alignment

I’ve started working on putting all the parts of the phoneme alignment code together. It feels like it’s going to be a slog, because there are so many parts to it.

There are two main parts to it: the analysis, and the Viterbi search.

The analysis code reads through the .wav file in 10ms chunks, and analyzes it for these 11 metrics:

• Energy – normalized sum of the FFT
• Flux – change of the FFT bins from the prior frame
• Periodicity (via autoregression)
• “Color” bands (200-1000Hz, 1000-3000Hz, 6000-2250Hz)
• Aperiodic bands (same as above)

Aperiodicity is the measure of the difference between each bin in the band, roughly:

    -- estimate what the value of the bin is by averaging the surrounding bins
    local expected = (bins[i-1] + bins[i+1]) * 0.5

    -- get the difference between the predicted value (average) and the actual
    b_noise = b_noise + math_abs(m - expected)

This idea originated from Masanori Morise, and is used in the WORLD D4C (Decomposition of 4-band/component) algorithm.

The “color” bands reduce what is typically a large Mel-band filterbank into a much smaller set of values, while still capable of identifying specific vowel sounds.

This is based on an idea by Kanru Hua. It calculate the values for the same bands that D4C uses, but for the purpose of identifying vowels.

This works by calculating the RMS of each band, finding the band that is the loudest, and normalize the bands with that value, making it (more or less) insensitive to volume.

In addition to tracking all these values, it tracks the deltas as well. The delta is actually taken over 5 frames, as a K=2 Regression with the formula:

delta_t = (2*(c[t+2]-c[t-2]) + (c[t+1]-c[t-1])) / 10

All this has been coded. The main change was to create a function to read and write the frame data, so it could be efficiently re-used.

There is a table of default, rough values for each ARPAbet phoneme. Each phoneme can be expanded into one or two sub-phonemes.

For example, AA expands to a single sub-phoneme AA, while AW expands out to the sub-phonemes AW and WX (using Software Automatic Mouth nomenclature).

The stop consonant B becomes two sub-phonemes, B (closure) and BX (burst), and so on. Here’s the (current) table of phonemes with more than one sub-phoneme. I’ve adopted the nomenclature of using an X for the second phoneme (similar to the dipthongs), although 2 might make more sense:

-- DIPHTHONGS
AW -> AW, WX
AY -> AY, YX
EY -> EY, YX
OW -> OW, WX
OY -> OY, YX
UW -> UW, YX
-- STOPS: Silence/Closure -> Burst
P -> P, PX
T -> T, TX
B -> B, BX
D -> D, DX
G -> G, GX
-- AFFRICATES: Stop Closure -> Fricative Release
CH -> CH, CX
JH -> JH, JX

All of these sub-phonemes have very rough initial default values.

After expanding these “prime” phonemes into sub-phonemes, transition phonemes are created between every phoneme in the reclist, with values created by interpolating the surrounding phonemes. So the transition B_W is (essentially) the averages of the means and variances of B and W,

For example, the line in the Moresampler reclist:

TH R OW N - G OW S T - HH OW L D

is stripped down to only phonemes, and silence tokens are placed in the front and back:

SIL TH R OW N G OW S T HH OW L D SIL

Each phoneme is expanded into sub-phonemes, and those sub-phonemes are connected by transition phonemes:

SIL
SIL_TH 
TH 
TH_R
R 
R_OW 
OW 
OW_WX 
WX 
WX_N 
N 
N_G 
G 
G_GX 
GX
GX_OW 
OW 
OW_WX 
WX 
WX_S 
S 
S_T
T 
T_TX 
TX
TX_HH 
HH 
HH_OW 
OW 
OW_WX 
WX 
WX_L 
L 
L_D 
D 
D_DX 
DX 
DX_SIL 
SIL

This list of phonemes serves as the “states” that the Viterbi algorithm will step through. The measure of error is the difference between the observed value (the frame) and the predicted value (the phoneme values):

-- Standard Mahalanobis distance / Log-Likelihood
local mll_cost = 0.0
for i = 1, 22 do
  -- difference between the observed value of the frame,
  -- and the mean value stored in the phoneme
  local diff = observed[i] - phoneme.mean[i]
  -- calculte the log likelihood, and scale by the weight
  mll_cost = mll_cost + (diff * diff * phoneme.inv_var[i]) * weight[j]
end

As noted, the mean and inv_var are initially just rough estimates. After each pass, new values for the phonemes are recalculated based on the results of the Viterbi algorithm. As a result, the code should slowly “learn” how the singer in the reclist sounds, and the phonemes will be more and more accurately aligned.

There are three main passes, with each pass focusing on a different set of attributes:

• Pass 1: Energy and Periodicity to find the phonemes;
• Pass 2: “Color” and Periodicity to distinguish vowels;
• Pass 3: Deltas to focus on transitions

Each pass will have 7 iterations. It should take about 3 iterations to settle on an average value for a phoneme, and 7 for that value to stabilize. Obviously, I may have to tweak the number of iterations. I don’t (yet) track any measure of how well the training is going.

In the first pass, the weights are adjusted to focus on the energy and periodicity, with a variance of 50. (Each pass sets the remaining values to 1.0, unless otherwise noted):

printConsole("Energy/Periodicity Pass")
varFloor = 50
-- Raise energy and Energy Delta
weight[IDX_ENERGY] = 2.0
weight[DELTA_OFFSET + IDX_ENERGY] = 2.0
-- Raise periodicity
weight[IDX_PERI] = 1.5
-- Lower color bands and color band deltas
weight[IDX_COLOR_1]  = 0.2
weight[IDX_COLOR_2]  = 0.2
weight[IDX_COLOR_3]  = 0.2
weight[IDX_COLOR_4]  = 0.2
weight[DELTA_OFFSET + IDX_COLOR_1]  = 0.2
weight[DELTA_OFFSET + IDX_COLOR_2]  = 0.2
weight[DELTA_OFFSET + IDX_COLOR_3]  = 0.2
weight[DELTA_OFFSET + IDX_COLOR_4]  = 0.2

The next pass focuses on the Spectral elements – the “spectral” bands and periodicity, while the weight on the deltas is fairly low. The variance is tightened down to 200:

printConsole("Spectral Analysis Pass")
varFloor = 200
-- Raise color bands and Periodicity
weight[IDX_COLOR_1]  = 2.0
weight[IDX_COLOR_2]  = 2.0
weight[IDX_COLOR_3]  = 2.0
weight[IDX_COLOR_4]  = 2.0
weight[IDX_PERI]  = 2.0
-- lower all the deltas
for i = 0, 11 do
  weight[DELTA_OFFSET + i] = 0.5
end

Finally, on the last pass, the focus is on finding the transitions, so the deltas are boosted and everything else toned down, and the variance again tightened down:

printConsole("Transition Analysis Pass")
varFloor = 500
-- all static values drop to .5, all deltas raised to 3.0
for i = 0, 11 do
  weight[i] = 0.5
  weight[DELTA_OFFSET + i] = 3.0
end

At each pass, a simplified Viterbi algorithm attempts to align the tokens with the given sequence. After all the lines have been matched, new mean and inv_var values are calculated for all the phonemes and transitions, and the pass is run again using those new values. Note the use of the varFloor when calculating the inv_vars:

-- For each phoneme in the dictionary
-- Loop through all the features
for j = 1, 22 do
    -- the mean is the sum of the observed values, divided by the count
    local mean = phoneme.sum[j] / phoneme.count
    -- the variance
    local var = (phoneme.sumSq[j] / phoneme.count) - (mean * mean)
    
    -- Enforce a floor so precision doesn't explode to infinity
    if var < 1e-6 then var = 1e-6 end
    -- store the mean
    phoneme.mean[j] = mean
    -- inverse variance
    phoneme.inv_var[j] = 1.0 / math.max(var, varFloor)
end

Well, that’s the theory. In practice, I’m still putting the code into place and debugging it, so I can’t say how well this will work.

But it should, because… math.

Recorded a Reclist

I put all the parts together and built a simple reclist recorder:

Recorded the Moresampler reclist

Of course, it involved the usual tracking down of bugs.

Now that I’ve got data, I can start working on the automatic phoneme alignment code.