Model Setup & Training¶
Train and Test¶
Assuming your data is in tnt
format you can encode the data ane train a indictrans.trunk.StructuredPerceptron
classifier.
>>> from indictrans import trunk
>>> #load trianing data
... X, y = trunk.load_data('indictrans/trunk/tests/hin2rom.tnt')
>>> #build ngram-context
... X = trunk.build_context(X, ngram=4)
>>> #fit encoder
... enc, X = trunk.fit_encoder(X)
>>> #train structured-perceptron model
... clf = trunk.train_sp(X, y, n_iter=5, verbose=2)
Iteration 1 ...
Train-set error = 1.5490
Iteration 2 ...
Train-set error = 1.0040
Iteration 3 ...
Train-set error = 0.8030
Iteration 4 ...
Train-set error = 0.6900
Iteration 5 ...
This will train the perceptron for 5 epochs (specified via the n_iter
parameter).
Then you can use the trained classifier as follows:
>>> #load testing data
... X_test, y = trunk.load_data('indictrans/trunk/tests/hin2rom.tnt')
>>> #build ngram-context for testing data
... X_test = trunk.build_context(X_test, ngram=4) # ngram value should be same as for train-set
>>> #encode test-set
... X_test = [enc.transform(x) for x in X_test]
>>> #predict output sequences
... y_ = clf.predict(X_test)
>>> y[10] # True
[u'c', u'l', u'a', u'ne', u'_']
>>> y_[10] # Predicted
[u'c', u'l', u'a', u'n', u'_']
>>> y_[100] # True
[u'p', u'a', u'r', u'aa', u'n', u'd', u'e']
>>> y_[100] # Predicted
[u'p', u'a', u'r', u'aa', u'n', u'd', u'e']
Note that you need to build-context
using the same ngram
value as used for trainig data. Also you need to encode
test data using the encoder enc
developed on training data.
Train directly from Console¶
indictrans-trunk provides a much easier way to train, test and save models directly from console.
user@indic-trans$ indictrans-trunk --help
-d , --data-file training data-file: set of sequences
-o , --output-dir output directory to dump trained models
-n , --ngrams ngram context for feature extraction: default 4
-e , --lr-exp The Exponent used for inverse scaling oflearning rate:
default 0.1
-m , --max-iter Maximum number of iterations for training: default 15
-r , --random-state Random seed for shuffling sequences within each
iteration.
-l , --verbosity Verbosity level: default 0 (quiet moe)
-t , --test-file testing data-file: optional: stores output sequences
in `test_file.out`
user@indic-trans$ indictrans-trunk -d hin2rom.tnt -o /tmp/rom-ind/ -n 4 -e 0.1 -m 5 -l 3 -t hin2rom.tnt
Iteration 1 ...
First sequence comparision: 0-27 0-95 0-30 0-10 ... loss: 4
Train-set error = 1.8090
Iteration 2 ...
First sequence comparision: 120-46 86-86 63-63 120-120 95-95 123-123 10-10 ... loss: 1
Train-set error = 0.6560
Iteration 3 ...
First sequence comparision: 123-123 110-110 40-40 46-46 ... loss: 0
Train-set error = 0.3820
Iteration 4 ...
First sequence comparision: 2-2 95-95 86-86 77-77 64-64 31-31 120-120 80-80 10-10 ... loss: 0
Train-set error = 0.2240
Iteration 5 ...
First sequence comparision: 40-40 120-120 31-31 120-120 125-125 120-120 123-123 117-117 31-31 120-120 ... loss: 0
Train-set error = 0.1540
Testing ...
Assuming hin2rom.tnt
was given as test-file
, the output file will be generated with the name hin2rom.tnt.out
.