International Journal of Computer Science Trends and Technology (IJCS T) Volume 4 Issue 4, Jul - Aug 2016

RESEARCH ARTICLE

OPEN ACCESS

Adaptive Process For Bagging Based Ensembles

Rajni David Bagul

[1],

Prof. Dr. B. D. Phulpagar [2]

PES Modern College of Engineering

Savitribai Phule Pune University
Shivajinagar, Pune
India

ABSTRACT
Ensemb le of classifiers is the approaches of multip le classifiers are learned fro m same dataset and this multip le trained
classifiers are used to predict the unlabelled data. The Performance of using Ensemble of classifiers is best than using single
classifier. The Bagging stands for Bootstrap aggregation are the approaches of ensemble. In bagging number of bags(n) are
formed where each bag contains number of instances (m) fro m training examples. Format ion of bags is referred as bootstrap
sampling in which m instances are selected randomly fro m training data and instanced can b e repeated. Once n bags are formed ,
no of classifiers(c) are trained using n bags. c classifiers are used for further prediction. It has some drawbacks such as all
classifiers are considered as of equal importance; there is no method optimal bag creation. Same prob lem was faced by Random
subspace classifier based ensemble (RSCE) and solved by applying Hybrid Adaptive ensemble learning framework (HA EL).In
proposed work, to solve issues in the Bagging HA EL framework is applied.Fro m experimental results, Bagging with HA EL
improves classification accuracy than simple bagging.
Keywords:- Ensemble learning ,Bagging, Random Subspace.

I.

INTRODUCTION

There are many ensemble approaches described as follows.

1. Boosting
In boosting weight is assign to the training instances that
determines how the instances was classified in the previous
iteration. The subspace is badly classified are inserted in the
training dataset for the next iteration. The instances which are
hard to classify on which boosting pays more attention.
Boosting does not provide any mechanis m to imp rove the
learning of base classifiers.
2. Bagging
It is introduced by Breiman and it is sampling based
ensemble classifier approach. Fro m the entire training set
bagging generate the mu ltiple base classifiers by train ing them
on the subsets randomly dawn. The pred ictions of the base
classifiers are co mbined into a final pred iction by majority
voting. The bagging only suitable for small dataset. The
sampling procedure of bagging creates many subsets by
bootstrap sampling which results variations in base classifiers.
3. Random Subspace
Rando m subspace is an ensemble creat ion method in which
use of feature sub sets to create the different data subsets to
train the base classifiers. One of the important ensemble
approaches is Rando m subspace classifier ensemb le. In RSCE
feature/attribute set is sub spaced randomly and for each sub
space, classifier is constructed using any learning algorithms.

ISSN: 2347-8578

These constructed classifiers are used to classify the test

instance with voting majority approach. RSCE method has
two major limitations.
Classifiers are d istributed equally without depending
on which classifier is constructed of which subspace.
For examp le s1, s2 are two subspace and c1, c2 are
classifiers constructed using s1, s2 respectively. s1
contains important attributes and s2 does not contains
any important attributes then also s1,s2 are treated
equally.

Which subspace should be selected so that it will

increase the accuracy i.e. Sub space selection is
completely random. So met imes due to some
irrelevant subspace selection, irrelevant classifiers
are constructed and hence results are irrelevant.

To overcome above limitation the drawbacks and proposed

Hybrid adaptive ensemble learn ing (HA EL) framework and
applied to RSCE. HA EL includes the two adaptive process 1)
Base classifier co mpetit ion adaptive process (BCCAP) 2)
classifier ensemble interaction adaptive process (CEIAP).
First adaptive process gives weightage to the classifiers
according there importance. To select the optimized subspace
is describe in second adaptive process.

www.ijcstjournal.org

Page 68

International Journal of Computer Science Trends and Technology (IJCS T) Volume 4 Issue 4, Jul - Aug 2016

II. LITERATURE SURVEY

In literature ensemble of classifiers have many approaches
such as Boosting[1],Bagging[2],Random forest[3],Random
subspace[4].To improve the classification accuracy ens emble
of classifiers integrates multiple classifiers to classify the
instance.
A) Boosting
To improve the classifiers accuracy in which subsequent
classifier models are trained on misclassified instances of
previous classifier model. The set of developed classifiers is
used for predicting the labels of the instances using voting
majority.
B) Bagging
In bagging n bags are formed where each bag contains m
instances from t rain ing examples. Formation of bags is
referred as bootstrap sampling in wh ich m instances are
selected randomly fro m training data and instanced can be
repeated. Once n bags are formed, n classifiers are trained
using n bags. C classifiers are used for further prediction.
C) Random subspace
Features in the training dataset are sub-spaced randomly. For
example there are n features in the train ing data then to form
subspace any m features are s elected randomly where m << n.
Nu mber o f subspace (S) are formed then data is sub-spaced
according to feature subspace. This sub-spaced datasets are
used to form S classifiers. Work related to ensemble of
classifiers can be divided into three categories. First category
is about design of the ensemble approaches; second category
is about how to improve existing ensemble solutions.
Design of the Ensemble approaches:
The problem of space complexity of ensemble focuses on
instance selection to reduce the space complexity. Aim o f the
instance selection is that selected instances and whole dataset
should yield classifiers which will give same results. Instance
selection method is combined with boosting and proposed
generic ensemble approach with instance selection. Fro m the
experimental results it is clear that proposed approach in this
paper can form simple and better ensembles than Random
subspace based knn ensemble, other traditional ensemble
approaches for c4.5 and SVM are describe in N.GarciaPedrajas [6].
The drawbacks of using undirected bi-relat ion graph for
prediction describe in G.Yu,C. Do menoconi[7].To overcome
this problem paper proposed a system wh ich used directed birelation graph with transductive multi-label classifier.
Ensemb le of TM C is used then to improve the accuracy of
prediction. Different fro m tradit ional methods that make use
of mu ltip le data sources by data integration, TMEC takes
advantage of multiple data sources by classifier integration.

ISSN: 2347-8578

In this paper proposed cluster based ensemble of classifiers.

Base classifiers give boundaries of clusters then cluster
confidences are mapped to class decision is described in B.
Verma and A.Rah man[8].Training dataset is categorized into
clusters using labels, these categorized data used to train the
classifiers. Base classifiers evaluate the cluster boundaries and
generate cluster confidence vectors. Proposed system makes
learning efficient by modifying domain for classifiers.
Work to improve the ensemble approaches
In this paper method which evaluates how many classifiers in
ensemble are needed to be queried for prediction of co mplete
ensemble. In general, unlabeled instance is classified by all
classifiers in the ensemble describe in D. Hernandez-Lobato
[9]. There is no need to classify instance with all classifiers in
the ensemble, only subset of classifier can predict the
prediction of all classifiers. Vot ing process is stopped when
probability of p rediction change is below threshold. Nu mber
of queries to be done which will reduce the probability of
change below threshold depends on the instance. Instances
which are on classification boundary need more queries. This
method is called as instance based pruning and can be
combined with any ensemble approach.
Distinct pruning strategies which are designed to improve
ensemble of classifiers are analysed is describe in paper [10].
In pruning selects the subset of function in the ensemble
which can perform better than whole ensemble. The approach
aggregation order is random therefore generalization error
decreases as number of functions is increased describe in
simp le bagging. Many pruning methods are based on
modifying order of aggregation of classifiers ensemble.
Performance of ensemble of classifiers can be improved by
ordering aggregation to minimize the generalization error.
Performance of the ensembles with pruning strategies are is
better than normal ensembles.
In this paper investigated the effect of the accuracy and the
diversity in the design of a mult ilayer perceptron (MLP) -based
classifier ensemb le describe in paper [11]. They also
considered how to reduce added classification errors in a
classifier ensemble based on the Walsh coefficients.
Kuncheva et al. [12] studied how to use a kappa-error d iagram
to analyze the performance of classifier ensemble approaches.
The drawbacks of RSCE (discussed in introduction) and
proposed Hybrid adaptive ensemble learning (HAEL)
framework and applied to RSCE to overcome above limitation
describe in paper [5]. HA EL includes the two adaptive process
1) Base classifier co mpetit ion adaptive process (BCCAP) 2)
classifier ensemble interaction adaptive process (CEIAP).
First adaptive process gives weightage to the classifiers
according there importance. To select the optimized subspace
is describe in second adaptive process.

www.ijcstjournal.org

Page 69

International Journal of Computer Science Trends and Technology (IJCS T) Volume 4 Issue 4, Jul - Aug 2016
Applications of the Ensemble
In this paper non-parallel p lane pro ximal classifier ensemble
which gives more accuracy than single non-parallel p lane
proximal classifier. This ensemble is applied to classify
unknown tissue samples using known gene expressions as
training data are describe in paper [13]. Genetic algorith m
based scheme is used to train non-parallel plane pro ximal
classifier. To predict the unlabelled data, classifiers with
positive performance are selected.
To aggregate the
prediction results min imu m average pro ximity-based decision
combiner is used. System is co mpared with SVM proves that
it gives co mparable accuracy with les s average training time.
Takemura et al. [14] designed combined heterogeneous
classifier ensemb les using a kappa statistic diversity measure,
and applied it to the electro myography signal datasets.
Takemura et al. [14] used the classifier ensemble approach to
identify breast tumours in the u ltrasonic images. In the area of
data mining, Windeatt et al. [15] applied the M LP ensembles
to perform feature ranking. Rasheed et al.

Base classifier
with weights

III. SYSTEM ARCHITECTURE

A) Bag of instance creationIn this section N bags of ins tances created from train ing
dataset Dtr. Training dataset Dtr contain m records. Size m
of the bag is taken as input fro m user. m instances are
selected randomly to create the bag of the instances. In such
way n bags are created.
B) Classifier learning
N classifiers are trained on n bags created in previous step. In
our work, system used J48 i.e. weka imp lementation of C4.5
decision tree algorithm. Specifically,

Fig.1.Overview of hybrid adaptive ensemble learning (HAEL) for the random

subspace-based classifier ensemble approach.

ISSN: 2347-8578

To represent an attribute the decision tree adopts an internal

node, uses an outgoing branch of the node to represent an
attribute value, and associates each leaf node with a
classification. Most of the decision trees apply the informat ion
gain, which is calculated fro m the entropies in the informat ion
theory, for performing the gain, wh ich is calculated fro m the
entropies in the information theory, fo r performing the
Learn ing process c1, c2, c3, cn classifiers are generated in this
step. When instance i is to be classified it is classified by all n
classifiers and output of classification of instance I by c1 is
denoted by y1.So classification step gives y1, y2, yn. Majority
voting is done to get final classification result by ensemble .n
classifiers are learned.
C) Hybrid adapti ve ensemble learning process
1) Base classifier competition adaptive process
Weight is evaluated for each classifier in the ensemble
describe in this step. From previous step we get set of n
classifiers.

Competition
among
different local
environment

Local
environment

Competition
within local
environment

Terminati
End.
on
Random
Fig.2.Adaptive process of base classifier competition
in SAEL.
Condition
competition
?
WI = 1/n is the initial weight for each classifier. First step of
this process is to assign the initial weight to the each classifier.
In the next step T training instances are selected randomly
fro m main train ing data(Dtr). Each selected instance is
classified by each classifier. Each classifier maintains
predicted values of each instance. Indicator vector IV is
generated for each classifier is generated whose length is
equal to T. ith Value in the indicator vector denotes the
whether ith p redicted value was correct or not. 0 values
indicate correct predict ion and 1 indicates error. Instances S
which are pred icted correctly by some classifiers and
predicted wrongly by some classifiers are considered for the
further procedure.
Weight of the classifier is increased if it predicts the S
correctly otherwise reduced. Predicted error of the classifiers
for the i-th training samples is calculated as follows:
Ei =
/
(1)

www.ijcstjournal.org

Page 70

International Journal of Computer Science Trends and Technology (IJCS T) Volume 4 Issue 4, Jul - Aug 2016
Where IVin indicates value in the indicator vector for ith
instance and nth classifier
Cumulative error of samples is calculated as
E=
(2)
Where T are instances whose prediction is varied across the
classifiers. T < T. Local environments are created as per
described in the [5]. Each classifier is assigned to the local
environment. Three operators are used for weight calculat ion
of classifier. 1) Co mpetition among classifiers in the same
local environment (CSLE)
, the co mpetition among the
classifiers In different local environments (CDLE), and the
random co mpetit ion (RC), and executes three operators one
by one.
SAELs

Cooperation

Competition

Wj = Wj a1 Wj
if (AccJ < AccB)
Where Wj and Wb is updated weight

Above process is repeated until termination condition is

reached for each local environ ment. Terminat ion condition
may be any of the following the number of iterat ions is larger
than the maximu m number of iterations pre-specified by the
user and 2) the weight vector does not change in several
iterations after termination of the iterat ive process average
weight of the classifiers is calcu lated. Classifiers whose
weight is above average are reset to 1 otherwise it is set to 0.
2) Classifier ensemble interaction adaptive process
Set of Ensembles is given as input in this step. Set of
ensembles are updated with aim to optimize the bags. All
ensembles are with updated weight of classifiers using
BCCAP. Ensemble of classifiers is considered as classifier in
this step. S = {S1, S2, S3Sn} is set of ensembles. Selected
instances for train ing of ensemb le of classifier (n) are
indicated by indicator vector Vpn, 0 denotes that instances is
not used by ensemble else 1. Aggregated indicator vector Vp
for pth ensemble defined as follows.
Vp =
Vpb
(3)
Where Wb is weight values of bth classifier

End

Termination
Condition?

Random
Exchange
.

Fig.3System Architecture: Adaptive process for bagging based ensemble

In CDLE classifiers two distinct local environments are

compared and weight is updated as shown below. In CSLE
classifiers in same local environ ments are compared and
weight is updated as shown below. In RC, any two random
classifiers are co mpared and weight is updated as shown
below.

If b and j are two classifiers and Wb and Wj is prior

weight respectively.

AccB and AccJ is predicted accuracy of b and j

classifier then

Wb = Wb + a1 Wb If (AccB > AccJ)

Wb = Wb
if (AccB= AccJ)
Wb = Wb a1 Wb if (AccB < AccJ)
Wj = Wj + a1 Wj
if (AccJ > AccB)
Wj = Wj
if (AccB= AccJ)

ISSN: 2347-8578

For calculating the similarity between two ensembles each

ensemble is having one aggregated indicator. Knn of
ensemble is calcu lated using aggregated indicator and
considered as Local environment of the ensemble.
Three operators are used for interaction in ensemble of
classifiers.
Cooperation operator Input to this operator will be
two ensembles fro m same local environment. First
all the classifiers in both ensembles are sorted
according to their weight. Half nu mber of classifiers
from both sites is selected to form new ensembles.

Co mpetition Operator Input to this operator will be

three ensembles which are selected randomly
irrespective of local environment. Ensemble which
has lowest accuracy on training dataset is removed
from set of ensembles in this process.

Random operator - Input to this operator will be

single ensemble. Any bag in this ensemble is
replaced by any random bag and its respective
classifier. If performance of newly generated
ensemble is less than old one, newly generated
ensemble is used for further process otherwise
change is discarded.

www.ijcstjournal.org

Page 71

International Journal of Computer Science Trends and Technology (IJCS T) Volume 4 Issue 4, Jul - Aug 2016
Above three operators are executed until termination
condition is reached wh ich is given by user. Output of this
step will be S with an updated ensembles. Ensemble of
classifier with minimu m cu mulative error is selected as
optimal ensemble.
4) Testing
In this step instances are tested using ensemble o f classifier
obtained in above process.

IV. EXPERIMENTAL RESULT

Da ta set Name

HBBE

BBE

Aus tra lian

85.50%

84.04%

Bupa

66.81%

60.68%

T ABLE 1: Comparison Of Accuracy (in %) Of HBBE And BBE

HAEL (HRSBE) and normal rando m subspace based

ensemble (RSBE) is also implemented. Co mparison of results
of HRSBE and RSBE is done in [5]. HRSBE gives more
accuracy than RSBE but it takes more t ime and memory than
RSBE. Fro m those results [5] it was expected that HBBE will
give more accuracy than BBE. Fo r experiment, number of
SAEL was set 5, nu mber of classifiers for each SA EL was set
5, terminat ion condition for CEIAP was 5 and termination
condition for BCCAP was 5. In all cases J48 will be used as
base classifier. Bagging size was set equal to 50%. Table 1
and Graph 1 show the obtained results. Australian and Bupa
datasets from KEEL [17] repository were used for
experiments.66% data was used for train ing purpose and 34 %
data was used for testing purpose.
B) Results
BBE gives 60.68% accuracy on bupa dataset and HBBE gives
66.81% accuracy on bupa dataset. For Australian dataset
BBE g ives 84.04C% accuracy and HBBE g ives 85.50%
accuracy. Fro m result table 1 and Graph 1 it is clear that
HBBE gives more accuracy than simp le BBE. Time
requirement and memo ry requirement were observed for both
systems. Obliv iously HBBE takes much more time and
memory than simple BBE.

V. CONCLUSIONS
This paper describes the effectiveness of ensemble learning,
various approaches for ensemble learning, Problems in
Random subspace ensemble learning and method to overcome
this method. We observed bagging based ensembles also
faces same problems as RSCE and proposed method HBBE
to solve the problem in bagging based ensembles. Bagging
Based ensemble by using HAEL approach gives results into
increase in prediction accuracy.

ACKNOWLEDGMENT
Rajani Bagul received the B.E in

T ABLE 2: Comparison Of Accuracy Of HBBE And BBE

A) Experimental Setup
Goal of the experimental evaluation is to check the
effectiveness of the proposed approach. In proposed work
HAEL is applied on Bagging based ensemble. Accuracy, t ime
and memory requirements of the Bagging based ensemble
with HA EL (HBBE) and existing bagging based ensemble
(BBE) are co mpared. Random subspace based ensemble with

ISSN: 2347-8578

Information Technology from

Savitribai
Phule Pune University, Pune. From
2007
2013. and M.E.(Appear) Degree in
Computer Engineering from PES
aph
Modern College of Engineering. Pune University 2014.

www.ijcstjournal.org

Page 72

International Journal of Computer Science Trends and Technology (IJCS T) Volume 4 Issue 4, Jul - Aug 2016

REFERENCES
[1]

[2]
[3]
[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

Y. Freund and R. E. Schapire, A decisiontheoretic generalization of on-line learning and

an application to boosting, J. Co mput Syst
Sci.,vol. 55, no. 1, pp.119139, 1997
L. Breiman, Bagging predictors, Mach.
Learn., vol.24, no. 2, pp. 123140, 1996.
L. Breiman, Random forests, Mach. Learn.,
vol. 45,no. 1, pp. 532, 2001.
T. K. Ho, The rando m subspace method
forconstructing decision forests, IEEE Trans.
Pattern Anal. Mach. Intell., vol. 20, no. 8, pp.
832844, Aug.1998.
Zhiwen
Yu,Hybrid
Adaptive
ClassifierEnsemb leSenior Member, IEEE, Le
Li, Jiming Liu,Fellow, IEEE, and Guoqiang
Han,vol 45,NO.2,February 2015.
N. Garca-Pedrajas, Constructing ensembles
of classifiers by means of weighted instance
selection, IEEE Trans. Neural Netw., vol. 20,
no. 2, pp. 258277, Feb. 2009.
G. Yu , C. Do meniconi, H. Rangwala, G.
Zhang, and Z. Yu, Transductive mult i-label
ensemble classificat ion for protein function
prediction, in Proc.18th ACM SIGKDD
KDD, New York, NY, USA,2012, pp. 1077
1085.archive/macros/latex/contrib/supported/IEEEt r
anc.
B. Verma and A. Rah man, Clusterorientedensemble classifier: Impact of
mu lticlustercharacterizat ion
on
ensemble
classifier learning,IEEE Trans. Knowl. Data
Eng., vol. 24, no. 4, pp.605618, Apr. 2012
PDCA12-70 data sheet, Opto Speed SA,
Mezzovico, Switzerland.
D. Hernandez-Lobato, G. Martinez-Muo z, and
A.Suarez, Statistical instance-based pruning
inensembles of independent classifiers, IEEE
Trans.Pattern Anal. Mach. Intell., vol. 31, no.
2, pp. 364369, Feb. 2009.
G. Martinez-Muo z, D. Hernandez-Lobato, and
A.Suarez, An analysis of ensemble pruning
techniques based on ordered aggregation,
IEEE Trans. Pattern Anal. Mach. Intell., vol.
31, no. 2, pp. 245259, Feb.2009.
T. Windeatt and C. Zor, Min imising added
classification error using Walsh coefficients,

ISSN: 2347-8578

[12]

[13]

[14]

[15]

[16]

[17]

www.ijcstjournal.org

IEEE Trans. Neural Netw., vol. 22, no. 8, pp.

13341339, Aug. 2011.
L. I. Kuncheva, A bound on kappa-error
diagramsfor analysis of classifier ensemb les,
IEEE Trans.Knowl. Data Eng., vol. 25, no. 3,
pp. 494501, Mar.2013.
S. Ghorai, A. Mukherjee, S. Sengupta, and P.
K.Dutta, Cancer classificat ion fro m gene
expression data by NPPC ensemb le,
IEEE/ACM Trans.Comput. Biol. Bioinf., vol. 8,
no. 3, pp. 659671,MayJun. 2011.
A. Takemu ra, A. Sh imizu, and K.
Hamamoto,Discrimination of breast tumors
in ult rasonic images using an ensemble
classifier based on the adaboost algorith m
with feature selection, IEEE Trans.Medical
Imaging, vol. 29, no. 3, pp. 598609,
Mar.2010.
T. W indeatt, R. Duangsoithong, and R.
Smith,Embedded
feature
ran king
for
ensemble M LP classifiers, IEEE Trans.
Neural Netw., vol. 22, no. 6,pp. 988994, Jun.
2011.
S. Rasheed, D. W. Stashuk, and M. S.
Kamel,Integrating heterogeneous classifier
ensembles forEM G signal deco mpos ition
based on classifieragreement, IEEE Trans.
Inf. Tech. Biomed., vol. 14,no. 3, pp. 866
882,May 2010.
http://sci2s.ugr.es/keel/datasets.php

Page 73