next up previous
Up: HMM Project Previous: Build an HMM Trainer

Testing

In order to evaluate how well our HMM trainer does, we need a test to determine how well our trained model mimics the original model. Please refer to the HMM document for a discussion. In it, I define a quantity called $ \nu$($ \lambda$ $ \lambda{^\prime}$) that measures the accuracy to which the trained model $ \lambda{^\prime}$ mimics the original model $ \lambda$. We wish to know how various parameters of the training process affect $ \nu$($ \lambda$ $ \lambda{^\prime}$).

Our HMM trainer has a number of parameters. Two parameters refer to the Markov model that we use as the ``actual'' process, i.e., the one being simulated.

N:
The number of states in the Markov model being modeled.
M:
The number of observations in the Markov model being modeled.
There are at least three parameters controlling the training process.

T:
The length of each observation sequence.
L:
The number of observation sequences.
C:
The maximum number of iterations allowed for the trained model to converge. This may be irrelevant if convergence is fast.
For your evaluation of HMM training, I would like to see a 4-dimensional (maybe 5) table of results showing statistics on $ \nu$($ \lambda$ $ \lambda{^\prime}$) for various values of NMT and L (and maybe C). For each table entry, you must do a number of trainings against a number of $ \lambda$'s that are generated randomly. In this way, we will have some measure of the HMM training process for those parameters. For example, set N = 3, M = 4, T = 5 and L = 20 and run, say, a dozen experiments. In each experiment, generate a new $ \lambda$ rendomly, train a $ \lambda{^\prime}$ and calculate $ \nu$($ \lambda$ $ \lambda{^\prime}$). For these, say, dozen values of $ \nu$($ \lambda$ $ \lambda{^\prime}$), report the mean and standard deviation. Now do this for a large variety of values of the variables explained above. With luck, we will see some trends in the data.
next up previous
Up: HMM Project Previous: Build an HMM Trainer
Webmaster: Jim Schmolze, schmolze@eecs.tufts.edu.  [Get postscript version]