Model Prediction of Defects in Sheet

Metal Forming Processes

Mario Dib1(B) , Bernardete Ribeiro1 , and Pedro Prates2

1
Center of Informatics and Systems of the University of Coimbra (CISUC),
University of Coimbra, Coimbra, Portugal
mario.dib@student.uc.pt, bribeiro@dei.uc.pt
2
Center for Mechanical Engineering, Materials and Processes (CEMMPRE),
University of Coimbra, Coimbra, Portugal
pedro.prates@dem.uc.pt

Abstract. Predicting defects is a challenge in many processing steps

during manufacturing because there is a great number of variables
involved in the process. In this paper, we take a machine learning per-
spective to choose the best model for defects prediction of sheet metal
forming processes. An empirical study is presented with the objective to
choose the best machine learning algorithm that will be able to perform
accurately this task. For building the model, three distinct datasets were
created using numerical simulation for three mild steel materials: mild
steel, DH600, HSLA340. The numerical simulation was performed on the
basis of sixteen input features representing characteristics of the materi-
als. Moreover, two kinds of defects, springback and maximum thinning,
each one is binary with 1 (defects) and 0 (non-defects) were considered in
the simulator. The experimental setup consists of running MLP, CART,
NB, RF and SVM algorithms using cross-validation for correctly choos-
ing model parameters. The results were averaged in 30 runs and the
standard deviations recorded. The initial conclusion is that the learning
algorithm scores differently depending on the type of defect and con-
ditions of the experiment. Although the preliminary results show good
performance of the algorithms in simulated environment, a further study
with real data will be addressed in future work.

Keywords: Machine learning · Manufacturing process

Predictive model · Defect prediction · Algorithm comparison

1 Introduction

Sheet metal forming is a manufacturing process widely used in the production

of metal components for the most diverse industries, namely the automotive,
naval, aeronautical and machinery industries. However, sheet metal forming pro-
cesses defects occur very often, making the overall procedure very costly; and
the numerous variables involved in forming processes, related with the material
c Springer Nature Switzerland AG 2018
E. Pimenidis and C. Jayne (Eds.): EANN 2018, CCIS 893, pp. 169–180, 2018.
https://doi.org/10.1007/978-3-319-98204-5_14
170 M. Dib et al.

properties, tooling geometry and process parameters, makes it difficult for engi-
neers to accurately predict the occurrence of forming problems, such as crack-
ing, localized necking, springback, among others. Also, the apparently random
occurrence of forming defects due to sources of scatter (e.g. material proper-
ties, tooling geometry and process parameters) adds further complexity to the
problem of defects prediction.
In this context, machine learning techniques may aid to solve this problem. In
particular, they can be trained with available data for building defects prediction
models. The rationale is that the models can generalize in unseen data and
successfully identify and check such defect patterns.
Although it is certain that the machine learning technique is a viable method
to be applied in the manufacturing industry to discover the origin of the defects,
as described above, it is valid to point it out that there is, first, a necessity
to discover how to build a model capable of learn the favorable conditions and
actions for the defect to appear in a metal component.
Once this model is developed and with the knowledge of how to avoid the
most common problems of the manufacturing of sheet metal, the process could
be improved since the industry would be able to invest on the best materials for
the production of certain pieces and with this simple measure it would also save
time and money by not having to discard the defective pieces.
The focus on this paper is to create a model capable of learn about the
conditions that will cause the defects to appear in a metal component, evaluate
the results generated by the created model and perform a thorough comparison
of machine learning models to verify which algorithms are capable to provide
good results to solve the prediction problem. In summary, it aims to make a
comparison of the results achieved by applying machine learning, through its
algorithms, to predict defects in the manufacturing process of sheet metals.
In Sect. 2 the experimental sheet metal forming background will be described.
The proposed approach used in this experiment will be detailed in Section 3.
The results will be showed and discussed in Sect. 4. Finally, the conclusions are
presented in Sect. 5.

2 Background

2.1 Sheet Metal Forming

Sheet metal forming is a manufacturing process that is widely used in the pro-
duction of metal components for the most diverse industries, namely for the auto-
motive, naval, aeronautical, machinery and household appliances industries [1].
In this process, metal sheets are plastically deformed into a desired shape by the
action of forming tools, which typically consist of a punch, a die and a blank
holder.
Firstly, a blank (i.e. non-deformed metal sheet) is placed over the die, and
then it is pressed into the die by the movement of the punch to obtain the
desired shape; the flow of the sheet material into the die is typically controlled
 Model Prediction of Defects in Sheet Metal Forming Processes 171

with a blank holder. In general, sheet metal forming processes allow obtaining
high quality components with high cadence and low cost; however, the variability
inherent to mechanical properties, tool geometry and process parameters makes
formed components often prone to defects such as wrinkling, tearing, excessive
thinning and springback.
These defects can appear in any step of a forming process, and it can be
very difficult to predict the location and the moment that they occur due to
the large amount of variables involved in the process. The ever-increasing com-
petitiveness in the automotive and aeronautical industries has demanded very
high quality and robustness requirements, particularly in components that have
a direct impact on occupant safety.
In this regard, the Finite Element Method (FEM) is a well-established com-
putational tool that plays a key role in predicting defect-prone regions in com-
ponents. Recently, some authors integrated statistical descriptions of variability
within FEM, for assessing the sensitivity of defect predictions to scatter [1–3].
However, the integration of finite element analysis with statistical tools involves
a high computational effort, incompatible with the response time required by
the industry.
The design of the forming process of a metal component requires a high
level of knowledge and is still very dependent on industrial experience. The use
of computational tools to support design (CAD), planning (CAPP), engineer-
ing (CAE) and manufacturing (CAM) has contributed to facilitate analysis and
reduce design time. However, it is always necessary for specialists to make deci-
sions at different stages.

2.2 Machine Learning

In a changing environment, a system should be able to learn and adapt to such
changes. In other words, the system needs to be intelligent. With the ability to
learn, the system designer does not have to foresee the solutions for all possible
situations [4]. In this context machine learning is a powerful tool to handle such
problems.
To perform a thorough data analysis that automates the development of
models, algorithms that learn interactively from example data or past expe-
rience are used, allowing computers to find patterns without being explicitly
programmed [4]. These models may be predictive or descriptive, or both [4], and
they should be able to learn how to predict defects based on previously data and
independently adapt when exposed to a new data.
Machine learning techniques can be classified in two categories: supervised
and unsupervised learning. In the first category, the algorithms use labeled data
to train and learn, while in the unsupervised category they don’t use data with
a classification of observation [5]. In this paper we have only used supervised
techniques.
There are a few studies in the literature on the application of machine learn-
ing for the identification and/or prediction of defects in metal manufacturing
172 M. Dib et al.

processes such as casting [6], rolling [7,8], extrusion [9] and sheet metal form-
ing [10,11]. However, to our knowledge, no comparative study was performed so
far on different machine learning algorithms for the prediction of defects in sheet
metal forming processes.

Fig. 1. Detection method scheme.

3 Model Defects Prediction Approach

3.1 Experimental Setup
The proposed scheme in Fig. 1 is composed by two sections, one for training the
model and the other for model’s evaluation. Both sections use the same dataset,
split in training and evaluation data, and the same experiment configuration.
Notice that 10-fold cross validation is used for setup of the model parameters.
For this experiment, the following configuration was used:
– Dataset training size: 70%;
– Dataset evaluation size: 30%;
– Random seed: 7;
– Number of splits: 10.
The data normalization was performed after the configuration of the exper-
iment because some algorithms would provide a better result with all the data
at the same scale. The next step was the algorithms selection to be evaluated
in regard of the best performance to identify the defect’s occurrence patterns.
Each selected algorithm was trained with the training data and later the models
were evaluated with the new data provided by the validation data. The learned
model uses the sample data and predicts whether a given sheet metal material
belongs to the defect or non-defect class.
 Model Prediction of Defects in Sheet Metal Forming Processes 173

3.2 Experimental Datasets

Three distinct datasets were created based on numerical simulation results of

the U-channel forming process, as schematized in Fig. 2, each dataset referring
to one type of sheet material. The U-channel tooling comprises three main ele-
ments: the blank-holder, the die and the punch, where the blank-holder applies
a constant force on the blank and the punch presses the metal sheet up to a
total displacement of 30 mm, so that a plastic deformation occur in the blank to
form a U-shaped part. The blank-holder force (BHF), which is the force that the
blank-holder will apply on the blank, is assumed constant during the forming
process and for this experiment the default values of 4.9 kN and 19.6 kN were
used, leading to different responses.
To simulate this forming process, the BHF was applied on a combination of
sixteen parameters of the forming process for each dataset, that are related with
mechanical properties, in order to model variability in the mechanical properties
of each material, which include: Young’s modulus (E), anisotropy coefficients (r0,
r45 and r90), initial tensile stress-strain data generated from parameters Y0, C
and n (see Table 1) and initial sheet thickness (t0). The variables values were
created using random numbers assuming that the variability in each input feature
(excluding the blank holder force) is described by a normal distribution within
a confidence interval of 95%, with well-defined mean and standard deviation
values. The mean values for each input feature are those corresponding to the
reference materials from Table 1; the standard deviation values were chosen in
agreement with the literature cited in Prates et al. [3]. The effects’ combination
of the variables lead to springback and maximum thinning defects or to a result
without any kind of defect.
Accordingly, three types of fictitious materials were studied, with mechan-
ical behavior typical of mild, DP600 and HSLA340 steel sheets, as shown in
Table 1 [12].

Table 1. Materials mechanical properties [12].

Variables Mild steel DP600 HSLA340

E[GPa] 206 210 210
v 0.3 0.3 0.3
r0 1.790 1.010 0.820
r45 1.510 0.760 1.070
r90 2.270 0.980 1.040
Y0 [MPa] 157.12 330.3 365.30
C [MPa] 565.32 1093.00 673
n 0.259 0.187 0.131
t0 [mm] 0.78 0.78 0.78
174 M. Dib et al.

Fig. 2. U-channel scheme [3].

The two identified defects, springback and maximum thinning, were sepa-
rated to create a binary classification with 1 (defects) and 0 (non-defects) to be
considered in the simulator.
Accordingly, a defect exists if the final score is higher than the reference val-
ues shown in Table 2, which were obtained from the numerical simulations of the
reference materials in Table 1. Both DP600 and HSLA340 had 170 FEM simula-
tion runs performed, while Mild Steel had 174 FEM simulation runs performed.
All the numerical simulations were carried out with the DD3IMP in-house FEM
code, developed and optimized for simulating sheet metal forming processes [13].
The dataset for this experiment was created with the values of each input vari-
able with it’s respective final score, for each defect type in each material type.

Table 2. Non-defect reference values.

Material Springback sb [mm] Maximum thinning th [%]

BHF = 4.9 kN BHF = 19.6 kN BHF = 4.9 kN BHF = 19.6 kN
Mild steel 6.165 2.601 2.82 9.62
DP600 11.151 8.522 2.07 5.83
HSLA340 8.747 2.70 5.115 7.66
 Model Prediction of Defects in Sheet Metal Forming Processes 175

3.3 Defects Classifier Selection

There is no easy way to know which algorithm will have the best performance to
solve a proposed problem. Usually, it is difficult to understand the factors that
affect the performance of a specific algorithm on a problem well enough to make
the decisions the algorithm selection problem requires with confidence [14].
Five algorithms were randomly selected through machine learning disciplines.
Three of them were classification algorithms, one is a regression algorithm and
the last one is a statistical learning algorithm. The selected algorithms for the
research design is presented in the following list:

– Multilayer Perceptron (MLP) [15];

– Random Forest (RF) [16];
– Decision Tree (CART) [17];
– Naive Bayes (NB) [18];
– Support Vector Machine (SVM) [19].

The models were built from the scratch using python v3.6.2 and related
libraries, such as SciPy Ecosystem and Scikit-learn, based on this methodol-
ogy [20]. Six machine learning models were created for two defects types in each
metal type. The data was normalized for all the algorithms performances because
the models results when it used normalized data were better, specially for the
Multilayer Perceptron algorithm.
In addition, for all the models the same configuration with random weights
was used in order to be possible to perform the model’s results comparison. A
brief description of each algorithm can be seen below.

Multilayer Perceptron (MLP): A Multilayer Perceptron or MLP is a feed-

forward artificial model for neural networks. Its goal is to map sets of input data
onto a set of appropriate outputs, using multiple layers of nodes in a directed
graph, with each layer fully connected to the next one. Each node is a neuron,
also called as processing element, except for the input nodes, with a nonlinear
activation function [15]. The technique used by this model to train the network
is called backpropagation, which consists in calculates the gradient of a loss func-
tion with respect to all the weights in the network, so that the gradient is fed
to the optimization method which in turn uses it to update the weights, in an
attempt to minimize the loss function.

Random Forest (RF): Random Forests are a combination of tree predictors

such that each tree depends on the values of a random vector sampled indepen-
dently and with the same distribution for all trees in the forest. They can be
used for classification, regression and other tasks, that operate by constructing a
multitude of decision trees at training time and outputting the class that is the
mode of the classes or mean prediction, classification and regression respectively,
of the individual trees [16].
176 M. Dib et al.

Decision Tree (CART): Decision Tree algorithms were originally intended

for classification which constructs a flowchart-like structure where each internal
node denotes a test on an attribute, each branch corresponds to an outcome
of the test, and each external node denotes a class prediction. At each node,
the algorithm chooses the best attribute to partition the data into individual
classes. When decision tree induction is used for attribute subset selection, a
tree is constructed from the given data. All attributes that do not appear in the
tree are assumed to be irrelevant. The set of attributes appearing in the tree
form the reduced subset of attributes [17].

Naive Bayes (NB): The Naive Bayes classifiers are probabilistic classifiers
based on applying Bayes’ theorem, greatly simplify learning by assuming that
features are independent and important given class [18].

Support Vector Machine (SVM): SVMs are supervised learning meth-

ods used for classification, regression and outliers detection, which constructs
a hyperplane or set of hyperplanes in a high or infinite-dimensional space. In
other words, given labeled training data (supervised learning), the algorithm
outputs an optimal hyperplane which categorizes new examples [19].

4 Results and Discussion

In order to assess the effectiveness of the proposed algorithm comparison, several
experiments were carried out in classification under covariate shift. In particular,
the performance for predicting defects was evaluated, by averaging the results
over 30 runs and the standard deviations recorded, with the metrics accuracy,
precision, recall, f1-score and AUC (area under the curve).
All the information of the small dataset samples were stored in lists and later
the mean and the standard deviation were calculated. To analyze the model’s
results in the sense of the defect prediction, different types of material were con-
sidered as mentioned above. It is important when making decisions to evaluate
how good the generalization in unseen data is. As the classification is binary, the
ROC Curve is an adequate metric since it gives all the possible operation points
to choose from [21].
Were used 10-fold cross validation on the data to train the model and the
models were trained in a Unix environment with 2.7 GHz processor and 16 Gb
Ram. In the Table 3, the algorithm with the best performance is marked as red.
The experiments have revealed overall good results, although for some clas-
sifiers the performance has shown to be below average. However, in the most of
the runs, the classification algorithms provided the best results, either for the
MLP, the CART or the RF, for all types of metal. With exception of the max-
imum thinning defect in DP600 metal, the score was very close to the second
best model, which was built with the MLP algorithm. The best algorithms for
all the models are illustrated in the Table 3.
 Model Prediction of Defects in Sheet Metal Forming Processes 177

Table 3. Experiment results

Accuracy Precision Recall F1-Score AUC

Algorithms
Mean std Mean std Mean std Mean std Mean std

Mild Steel - Springback Results

MLP 81% 0.012 81% 0.011 81% 0.013 81% 0.013 80.67% 0.012
CART 88% 0.026 89% 0.012 88% 0.026 88% 0.026 87.69% 0.026
NB 70% - 72% - 70% - 69% - 70.09% -
RF 85% 0.039 85% 0.037 85% 0.04 85% 0.04 84.71% 0.038
SVM 85% - 86% - 85% - 85% - 85.04% -
Mild Steel - Maximum Thinning Results
MLP 92% 0.0089 92% 0.0094 92% 0.005 92% 0.0054 91.01% 0.008
CART 86% 0.023 86% 0.024 86% 0.024 86% 0.024 84.91% 0.024
NB 89% - 89% - 89% - 89% - 87.68% -
RF 89% 0.013 90% 0.013 90% 0.012 89% 0.009 88.31% 0.011
SVM 91% - 91% - 91% - 90% - 89.30% -
DP600 - Springback Results
MLP 91% 0.010 92% 0.010 91% 0.010 91% 0.010 91.90% 0.010
CART 87% 0.017 87% 0.018 87% 0.018 87% 0.018 86.54% 0.015
NB 84% - 85% - 84% - 84% - 85,11% -
RF 86% 0.024 86% 0.028 85% 0.024 85% 0.024 84.76% 0.022
SVM 92% - 92% - 92% - 92% - 92.01% -
DP600 - Maximum Thinning Results
MLP 94% 0.010 95% 0.006 93% 0.010 93% 0.010 95.20% 0.008
CART 91% 0.009 92% 0.010 91% 0.010 91% 0.010 92.11% 0.010
NB 94% - 94% - 94% - 94% - 94.12% -
RF 97% 0.015 97% 0.015 97% 0.015 97% 0.015 96.42% 0.015
SVM 92% - 92% - 92% - 92% - 90% -
HLSA340 - Springback Results
MLP 85% 0.014 86% 0.013 85% 0.014 85% 0.014 82.60% 0.016
CART 88% 0.015 89% 0.019 88% 0.015 88% 0.015 86.49% 0.016
NB 84% - 85% - 84% - 84% - 81.77% -
RF 93% 0.017 93% 0.018 93% 0.017 93% 0.017 91.69% 0.018
SVM 80% - 81% - 80% - 80% - 76.77% -
HSLA340 - Maximum Thinning Results
MLP 91% 0.019 92% 0.016 91% 0.020 91% 0.020 91% 0.021
CART 94% - 94% - 94% - 94% - 93,98% -
NB 86% - 87% - 86% - 86% - 85,88% -
RF 94% - 94% - 94% - 94% - 93.98% -
SVM 80% - 82% - 80% - 80% - 79.63% -
178 M. Dib et al.

The results were obtained by each model and were analyzed individually.
The final results showed that the classification algorithms successfully predicted
the defects in the considered materials. However, we noticed that there was not
unanimity and the SVM algorithm was the best predicting the springback defect
in the metal DP600, but even so, the MLP score was only 0.12% worse than
the SVM score, meaning both models could provide comparable The Table 4
demonstrate the greatest outcomes for each defect-material combination.
Although, in this first moment, it was not possible to evaluate the accuracy of
machine learning models with real data due to the unavailability of the data, the
final outcomes showed there are different models that performed better for each
type of material and its associate defects. Therefore, the next steps will demand
models adjustments to take the best of each model. In this way, a more confident
conclusion of which machine learning algorithm is apt to predict defects more
accurately in real scenarios.

Table 4. Model results

Material types Defect class Model Score

Mild steel Springback CART 87.69%
Max. thinning MLP 91.01%
DP600 Springback SVM 92.01%
Max. thinning RF 96.42%
HSLA340 Springback RF 91.69%
Max. thinning RF and CART 93.98%

5 Conclusion
Based on this experiment results, it is possible to have more than one option to
build a machine learning model that is able to produce satisfactory outputs in
regard of the defect prediction in a manufacturing environment. Whilst most of
the scores had similar results independently of the material type or the defect
class, an argument can be made in favor of the classification algorithms, because
they had the best performance results for the accuracy and AUC parameters.
In this sense, it would be a safe selection to use them as a standard choice to
execute this type of prediction.
Although some algorithms did not perform well in some environments, as in
case of a specific material-defect combination, it could have happened because
of the small size of the training dataset, since the machine learning algorithms
overall could learn better with larger samples of data, in accordance with each
specific situation. That is why the usage of larger samples of data is one aspect
that could be improved in the future to achieve better results.
 Model Prediction of Defects in Sheet Metal Forming Processes 179

Another aspect to be improved in the future is the personal configuration

for each model. For this experiment the models were not refined in order to
obtain the best possible result. In fact, the standard configuration, with some
adjustments, to make the models outputs to be comparable was used.
Besides of the improvements suggested above, this experiment was helpful
because it provides useful insights of which are the best algorithms to per-
form predictions. In this project’s context this is a good start point for further
investigations.

Acknowledgements. This work was funded by the Portuguese National Innova-

tion Agency (ANI), for the support under the project SAFEFORMING - Sistema
Inteligente de Preveno de Defeitos em Componentes Estampados a Frio, co-funded
by FEDER, through the program Portugal-2020 (PT2020) and by POCI, with ref-
erence POCI-01-0247-FEDER-017762. Pedro Prates was supported by a grant for
scientific research from the Portuguese Foundation for Science and Technology (ref.
SFRH/BPD/101465/2014). All supports are gratefully acknowledged.

References
1. Huang, C., Radi, B., Hami, A.: Uncertainty analysis of deep drawing using sur-
rogate model based probabilistic method. Int. J. Adv. Manuf. Technol. 86, 9–12
(2016). https://doi.org/10.1007/s00170-016-8436-4
2. Wiebenga, J.H., Atzema, E.H., An, Y.G., Vegter, H., Boogaard, A.H.: Effect of
material scatter on the plastic behavior and stretchability in sheet metal form-
ing. J. Mater. Process. Technol. 214(2), 238–252 (2014). https://doi.org/10.1016/
j.jmatprotec.2013.08.008
3. Prates, P.A., Adaixo, A.S., Oliveira, M.C., et al.: Numerical study on the effect
of mechanical properties variability in sheet metal forming processes. Int. J. Adv.
Manuf. Technol., pp. 1–20 (2018). https://doi.org/10.1007/s00170-016-8436-4
4. Alpaydin, E.: Introduction to Machine Learning, 3rd edn. The MIT Press,
Cambridge, Massachusetts, London (2016)
5. Aleem, S., Capretz, L.F., Ahmed, F.: Benchmarking machine learning techniques
for software defect detection. Proc. Int. J. Softw. Eng. Appl. (IJSEA) 6. Western
University, London, Ontario, Canada (2015)
6. Santos, I., Nieves, J., Penya, Y.K., Bringas, P.G.: Optimising machine-learning-
based fault prediction in foundry production. In: Proceedings of IWANN 2009:
Distributed Computing, Artificial Intelligence, Bioinformatics, Soft Computing,
and Ambient Assisted Living. Caligny, France and Bamberg, Germany (2009)
7. Lieber, D., Stolpe, M., Konrada, B., Deuse, J., Morik, K.: Quality prediction in
interlinked manufacturing processes based on supervised & unsupervised machine
learning. Procedia CIRP 7, 193–198 (2013). https://doi.org/10.1016/j.procir.2013.
05.033
8. Siyang, T., Xu, K.: An algorithm for surface defect identification of steel plates
based on genetic algorithm and extreme learning machine. Metals - Open Access
Metall. J. 7, 311 (2017). https://doi.org/10.3390/met7080311
9. Barcellona, A.: Neural network techniques for metal forming design. In: Pro-
ceedings of the Thirtieth International MATADOR Conference. Palgrave, London
(1993)
180 M. Dib et al.

10. Wang, J., Wu, X., Thomson, P.F., Flitman, A.: A neural networks approach
to investigating the geometrical influence on wrinkling in sheet metal forming.
J. Mater. Process. Technol. 105, 215–220 (2000). https://doi.org/10.1016/S0924-
0136(00)00534-3
11. Wenjuan, L., Qiang, L., Feng, R., Zhiyong, L., Hongyang, Q.: Springback prediction
for sheet metal forming based on GA-ANN technology. J. Mater. Process. Technol.
187–188, 227–231 (2007). https://doi.org/10.1016/j.jmatprotec.2006.11.087
12. Bouvier, S., Teodosiu, C., Maier, C., Banu, M., Tabacaru, V.: Selection and identifi-
cation of elastoplastic models for the materials used in the benchmarks. 18-Months
Progress Report of the Digital Die Design Systems (3DS) (2001)
13. Menezes, L.F., Teodosiu, C.: Three-dimensional numerical simulation of the deep-
drawing process using solid finite elements. J. Mater. Process. Technol. 97, 100–106
(2000). https://doi.org/10.1016/S0924-0136(99)00345-3
14. Kotthoff, L., Gent, I., Miguel, I.: A preliminary evaluation of machine learning in
algorithm selection for search problems. In: Proceedings of The Fourth Interna-
tional Symposium on Combinatorial Search. University of St. Andrews, Scotland,
UK (2011)
15. Collobert, R., Bengio, S.: Links between perceptrons, MLPs and SVMs. In: Pro-
ceedings of the 21st International Conference on Machine Learning. Banff, Canada
(2004)
16. Breiman, L.: Random Forests. Statistics Department, University of California
Berkeley, CA, USA (2001)
17. Han, J., Kamber, M., Jian, P.: Data Mining Concepts and Techniques, 3rd edn.
Morgan Kaufmann, San Francisco (2011)
18. Rish, I.: An Empirical Study of the Naive Bayes Classifier. IBM Research Division,
Thomas J. Watson Research Center, Yorktown Heights, NY, USA (2001)
19. Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20, 273–297 (1995).
https://doi.org/10.1007/BF00994018
20. Brownlee, J.: Machine Learning Mastery With Python: Understand Your Data,
Create Accurate Models and Work Projects End-To-End, 1.4th edn. Jason
Brownlee, Melbourne, Australia (2016)
21. Fawcett, T.: An introduction to ROC analysis. Pattern Recogn. Lett. 27, 861–874
(2006)