Something More for Research

Explorer of Research #HEMBAD

AdaBoost in OpenCV

Posted by Hemprasad Y. Badgujar on January 15, 2015


AdaBoost – machine learning algorithms. Boosting algorithms are designed for the type of training T «weak” classifiers. Individually, these classifiers are usually simple. Each classifier Kt (t = {1, 2, … T}) associated with a weight at, which is used when combining the results from all classifiers. The input feature vector
X = (x1, x2, … xi, …, XM)
(where i – the number of feature) labels marked binary yi = {-1, +1} (other Boosting algorithms may be a range of values).
Distribution Dt ( i), which must be initialized in advance, according to how much “worth” wrong assessment of each characteristic. A key feature of Boostinga is that when the algorithm progresses, the “cost” will be developed so that the training of “weak” classifiers will focus on those characteristics that were previously in the “weak” classifiers gave poor results.
The algorithm:
1. D1 (i) = 1 / m, for i = 1, …, m.
2. For t = 1, …, T:
2.1. Find classifier Kt minimizing Dt (i) the weight error:


where

for yj ≠ K_j (X), while ε_j <0.5, otherwise the output.
2.2. Set for K_t weight where ε_t – minimum error in the previous step.

2.3. Update the weights for features where Zt normalizes all signs.

At the end of the described algorithm takes a new input vector X and classified on the basis of the weighted sum of the theory in more detail you can read in other sources, and we’ll see a practical example. The library OpenCV – a file letter_recog.cpp.

For training base used letter-recognition.data site http://archive.ics.uci.edu/ml/ . The sample consists of 20,000 elements. Each element 17 is composed of the attributes: 1 – the section member, i.e. character of text A, B, C, D … – only 26 options; 16 attributes – signs of recognition. Criteria for recognition can be found in the description of the sample on this site. Used for training 10,000 items to check are also 10,000 elements.letter_recog.cpp example has been simplified as follows:

#include "opencv2/core/core_c.h"
#include "opencv2/ml/ml.hpp"
#include <cstdio>
#include <vector> 

int
read_num_class_data( const char* filename, int var_count,
                     CvMat** data, CvMat** responses )
{
    const int M = 1024;
    FILE* f = fopen( filename, "rt" );
    CvMemStorage* storage;
    CvSeq* seq;
    char buf[M+2];
    float* el_ptr;
    CvSeqReader reader;
    int i, j;

    if( !f )
        return 0;

    el_ptr = new float[var_count+1];
    storage = cvCreateMemStorage();
    seq = cvCreateSeq( 0, sizeof(*seq), (var_count+1)*sizeof(float), storage );

    for(;;)
    {
        char* ptr;
        if( !fgets( buf, M, f ) || !strchr( buf, ',' ) )
            break;
        el_ptr[0] = buf[0];
        ptr = buf+2;
        for( i = 1; i <= var_count; i++ )
        {
            int n = 0;
            sscanf( ptr, "%f%n", el_ptr + i, &n );
            ptr += n + 1;
        }
        if( i <= var_count )
            break;
        cvSeqPush( seq, el_ptr );
    }
    fclose(f);

    *data = cvCreateMat( seq->total, var_count, CV_32F );
    *responses = cvCreateMat( seq->total, 1, CV_32F );

    cvStartReadSeq( seq, &reader );

    for( i = 0; i < seq->total; i++ )
    {
        const float* sdata = (float*)reader.ptr + 1;
        float* ddata = data[0]->data.fl + var_count*i;
        float* dr = responses[0]->data.fl + i;

        for( j = 0; j < var_count; j++ )
            ddata[j] = sdata[j];
        *dr = sdata[-1];
        CV_NEXT_SEQ_ELEM( seq->elem_size, reader );
    }

    cvReleaseMemStorage( &storage );
    delete[] el_ptr;
    return 1;
} 

int build_boost_classifier( char* data_filename )
{
    const int class_count = 26;
    CvMat* data = 0;
    CvMat* responses = 0;
    CvMat* var_type = 0;
    CvMat* temp_sample = 0;
    CvMat* weak_responses = 0;

    int ok = read_num_class_data( data_filename, 16, &data, &responses );
    int nsamples_all = 0, ntrain_samples = 0;
    int var_count;
    int i, j, k;
    double train_hr = 0, test_hr = 0;
    CvBoost boost;

    if( !ok )
    {
        printf( "Could not read the database %s\n", data_filename );
        return -1;
    }

    printf( "The database %s is loaded.\n", data_filename );
    nsamples_all = data->rows;
    ntrain_samples = (int)(nsamples_all*0.5);
    var_count = data->cols;

    CvMat* new_data = cvCreateMat( ntrain_samples*class_count, var_count + 1, CV_32F );
    CvMat* new_responses = cvCreateMat( ntrain_samples*class_count, 1, CV_32S );

    // 1. unroll the database type mask
    printf( "Unrolling the database...\n");
    for( i = 0; i < ntrain_samples; i++ )
    {
        float* data_row = (float*)(data->data.ptr + data->step*i);
        for( j = 0; j < class_count; j++ )
        {
            float* new_data_row = (float*)(new_data->data.ptr +
                            new_data->step*(i*class_count+j));
            for( k = 0; k < var_count; k++ )
                new_data_row[k] = data_row[k];
            new_data_row[var_count] = (float)j;
            new_responses->data.i[i*class_count + j] = responses->data.fl[i] == j+'A';
        }
    }

    // 2. create type mask
    var_type = cvCreateMat( var_count + 2, 1, CV_8U );
    cvSet( var_type, cvScalarAll(CV_VAR_ORDERED) );
    // the last indicator variable, as well
    // as the new (binary) response are categorical
    cvSetReal1D( var_type, var_count, CV_VAR_CATEGORICAL );
    cvSetReal1D( var_type, var_count+1, CV_VAR_CATEGORICAL );

    // 3. train classifier
    printf( "Training the classifier (may take a few minutes)...\n");
    boost.train( new_data, CV_ROW_SAMPLE, new_responses, 0, 0, var_type, 0,
        CvBoostParams(CvBoost::REAL, 100, 0.95, 5, false, 0 ));
    cvReleaseMat( &new_data );
    cvReleaseMat( &new_responses );
    printf("\n");
   
    temp_sample = cvCreateMat( 1, var_count + 1, CV_32F );
    weak_responses = cvCreateMat( 1, boost.get_weak_predictors()->total, CV_32F );

    // compute prediction error on train and test data
    for( i = 0; i < nsamples_all; i++ )
    {
        int best_class = 0;
        double max_sum = -DBL_MAX;
        double r;
        CvMat sample;
        cvGetRow( data, &sample, i );
        for( k = 0; k < var_count; k++ )
            temp_sample->data.fl[k] = sample.data.fl[k];

        for( j = 0; j < class_count; j++ )
        {
            temp_sample->data.fl[var_count] = (float)j;
            boost.predict( temp_sample, 0, weak_responses );
            double sum = cvSum( weak_responses ).val[0];
            if( max_sum < sum )
            {
                max_sum = sum;
                best_class = j + 'A';
            }
        }

        r = fabs(best_class - responses->data.fl[i]) < FLT_EPSILON ? 1 : 0;

        if( i < ntrain_samples )
            train_hr += r;
        else
            test_hr += r;
    }

    test_hr /= (double)(nsamples_all-ntrain_samples);
    train_hr /= (double)ntrain_samples;
    printf( "Recognition rate: train = %.1f%%, test = %.1f%%\n",
            train_hr*100., test_hr*100. );

    printf( "Number of trees: %d\n", boost.get_weak_predictors()->total );
    

    cvReleaseMat( &temp_sample );
    cvReleaseMat( &weak_responses );
    cvReleaseMat( &var_type );
    cvReleaseMat( &data );
    cvReleaseMat( &responses );

    return 0;
}
 

int main()
{
        // Чтение данных из файла
        build_boost_classifier( "letter-recognition.data");
        return 0;
}
 

All the action takes place in the function build_boost_classifier, which we consider. Originally described by a function read_num_class_data done downloading a file, ie creating a matrix with signs (16 signs per cell) read_num_class_data, as well as the matrix responses, which states – what class image. Next is initialized, including the creation of templates for learning. Note that the number of elements in the matrix for training new_data multiplied by the number of classes (26 different letters), and the dimension is 16 elements, and 17. Matrix new_responses also increases in size in class_count times (26). Below (after // 1. unroll the database type mask) is filled matrices and it becomes clear what changes the size of the matrix.
Each element of the training sample corresponds to only one class – the letters A, S, T, … .and etc. But it is also necessary to consider that this element tells us that this combination of features can not be any other class of images. Therefore, all the signs are duplicated on the number of classes. Increase to 17 elements is because we have with each of these signs of an attribute class number (0, 1, … 25).
With regard to the results matrix – there indicated TRUE or FALSE for each class, and that speaks to what class they belong data. Ie 26 classes, only one is TRUE for one element of training sample.
After filling matrices created another matrix mask that splits the data into images and signs on the category of images:
CV_VAR_ORDERED – numeric value attribute of the image;
CV_VAR_CATEGORICAL – image category.
After that move learning – a function of the class train CvBoost. Without going into details of the class definition, consider the function train.

bool  CvBoost :: train ( 
        const  Mat & trainData ,  
        int tflag ,  
        const  Mat & responses ,  
        const  Mat & varIdx = Mat (),  
        const  Mat & sampleIdx = Mat (),  
        const  Mat & varType = Mat (),  
        const  Mat & missingDataMask = Mat (),  
        CvBoostParams  params = CvBoostParams (),  
        bool update = false  
);

In the description of the function indicates that it coincides with CvStatModel :: train (). Responses must be categorical, meaning that boosted trees can not be constructed for regression and there should be two classes.
Parameters:

trainData data for training. By default, the vector data is in rows.tflag flag that indicates in what format the data:
CV_ROW_SAMPLE – means that the vector data is in the row (row);
CV_COL_SAMPLE – columns.

responses answers.

varIdx
documentation written about it a little, but I understand that this option is to not use all the sample for learning, and a part. Ie select a subset of the input set, if not necessary – set 0. sampleIdx seems that has the same meaning as varIdx, although the documentation is not lit again. Perhaps one parameter for the data for the other responses. But it is usually not used and also write 0. vartype type of input variables in the form of a matrix, which describes where the data is, and where these data categories.

missingDataMask Because some training algorithms handle the absence of data (for example, was not measured one of the signs), it is possible to fill this matrix specifying which attributes to what elements to miss. If not used, specify 0.

params Settings Boosted-classifier. Transmission structure CvBoostParams.

Update Indicates whether the classifier updated by default – no.

CvBoostParams :: CvBoostParams

        int boost_type ,  
        int weak_count ,  
        double weight_trim_rate ,  
        int max_depth ,  
        bool use_surrogates ,  
        const  float * priors 
);

Parameters: boost_type type algorithm:
CvBoost :: DISCRETE Discrete AdaBoost.
CvBoost :: REAL Real AdaBoost.
CvBoost :: LOGIT LogitBoost.
CvBoost :: GENTLE Gentle AdaBoost.

weak_count number of “weak” classifiers.

weight_trim_rate threshold – the level of significance. Put 0.95 to reduce the duration of the experiment, but keep good quality.

max_depth The maximum depth of the tree.

use_surrogates priors array priori probabilities of classes sorted by class value. In this example, the following is written:

boost . train ( new_data , CV_ROW_SAMPLE , new_responses , 0 , 0 , var_type , 0

        CvBoostParams ( CvBoost :: REAL ,  100 ,  0.95 ,  5 ,  false ,  0  ));

At first glance it is not clear why the size of the matrix is in this case of 18 elements. However, if you look at the dimension new_data + dimension new_responses, there will be 17 + 1 = 18. And further: real AdaBoost, 100 “weak” classifiers 0.95 level of significance, the maximum depth of 5. Then there is a check on the learning outcomes of training and screening samples. Selected data element by element, and the function is called:

float  CvBoost :: predict ( 
        const  CvMat * sample ,  
        const  CvMat * missing = 0 ,  
        CvMat * weak_responses = 0 ,  
        CvSlice slice = CV_WHOLE_SEQ ,  
        bool raw_mode = false ,  
        bool return_sum = false  
);

Parameters:
sample input sample signs of one element.
missing Additional mask, which may indicate which attributes should be skipped. If you do not have to miss anything, then 0.
weak_responses Optional output parameter vector floating-point responses of each weak classifier. The number of elements in the vector must be equal to the length of the cutoff.
Slice continuous sequence of weak classifiers subset to be used for prediction. By default, all the weak classifiers are used.
raw_mode Specify false.
return_sum If true, return the amount of votes.

In the example, this function is called for each class, computing, for which class the sum of weights of “weak” classifiers weak_responses more. Well, actually, if the result is the same as in the responses to the signs, it increases the value of correctly recognized the responses to the learner (train_hr) and verification (test_hr) samples. At the end of the screen displays the result of the reliability of recognition for teaching and screening samples, as well as the number of “weak” classifiers. Yes, 70% is a small percentage of correct recognition. However, there should not be sin on himself learning algorithm. First, we must understand that in the signs used in the sample and the fact that for the characters – quality. Secondly, for teaching methods, this sample with the complex and ever-changing environment, will not be sufficient. But in general, there has been considered one of the learning algorithms that you can use for their own purposes.

Advertisements

One Response to “AdaBoost in OpenCV”

  1. Kadirou31 said

    hi,
    many thanks for this !! it was very helpfull for me 😉
    could you please show me how to make a regression instead of -1 or 1 ??

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Extracts from a Personal Diary

dedicated to the life of a silent girl who eventually learnt to open up

Num3ri v 2.0

I miei numeri - seconda versione

ThuyDX

Just another WordPress.com site

Algunos Intereses de Abraham Zamudio Chauca

Matematica, Linux , Programacion Serial , Programacion Paralela (CPU - GPU) , Cluster de Computadores , Software Cientifico

josephdung

thoughts...

Tech_Raj

A great WordPress.com site

Travel tips

Travel tips

Experience the real life.....!!!

Shurwaat achi honi chahiye ...

Ronzii's Blog

Just your average geek's blog

Karan Jitendra Thakkar

Everything I think. Everything I do. Right here.

VentureBeat

News About Tech, Money and Innovation

Chetan Solanki

Helpful to u, if u need it.....

ScreenCrush

Explorer of Research #HEMBAD

managedCUDA

Explorer of Research #HEMBAD

siddheshsathe

A great WordPress.com site

Ari's

This is My Space so Dont Mess With IT !!

%d bloggers like this: