Up: HMM Project
Previous: Build an HMM Trainer
In order to evaluate how well our HMM trainer does, we need a test to determine
how well our trained model mimics the original model. Please refer to the HMM
document
for a discussion. In it, I define a
quantity called
( ) that measures the accuracy
to which the trained model mimics the original model .
We wish to know how various parameters of the training process affect
( ).
Our HMM trainer has a number of parameters. Two parameters refer to the Markov
model that we use as the ``actual'' process, i.e., the one being simulated.
- N:
- The number of states in the Markov model being modeled.
- M:
- The number of observations in the Markov model being modeled.
There are at least three parameters controlling the training process.
- T:
- The length of each observation sequence.
- L:
- The number of observation sequences.
- C:
- The maximum number of iterations allowed for the trained model to
converge. This may be irrelevant if convergence is fast.
For your evaluation of HMM training, I would like to see a 4-dimensional (maybe
5) table of results showing statistics on
( )
for various values of
N, M, T and L (and maybe C).
For each table entry, you must do a number of trainings against a number of
's that are generated randomly. In this way, we will have some
measure of the HMM training process for those parameters. For example, set
N = 3, M = 4, T = 5
and L = 20 and run, say, a dozen experiments. In each experiment, generate
a new rendomly, train a and calculate
( ).
For these, say, dozen values of
( ), report the
mean and standard deviation. Now do this for a large variety of values of the
variables explained above. With luck, we will see some trends in the data.
Up: HMM Project
Previous: Build an HMM Trainer
Webmaster: Jim Schmolze, schmolze@eecs.tufts.edu. [Get postscript version]