Available online at www.sciencedirect.

com

ScienceDirect
Procedia Engineering 80 (2014) 66 75

3rd International Symposium on Aircraft Airworthiness, ISAA 2013

Surrogate-based sensitivity analysis for safety assessment of

general aviation heavy-fueled engines
Jiaokun CAO *, Shuiting DING, Farong Du
Airworthiness Technologies Research Center, Beihang University, Beijing, 100191, P.R.China

Abstract
This paper has developed a framework for surrogate-based sensitivity analysis in SSA considering the complexity of
Heavy-Fueled Engines (HFE) models. Multi-dimensional HFE whole engine models are extreme time consuming and
computationally expensive and not practical for engineering applications. To reduce the computational costs,
Artificial Neural Networks (ANN) is selected as surrogate model to establish high-fidelity HFE system model. Based
on the developed ANN model, global sensitivity analysis is conducted to provide information on parameters
importance, which are significant in complex system safety analysis. The results reveal that, by using global
sensitivity analysis, the parameters could be ranked with respect to their importance, including first order indices and
total sensitivity indices. For the particular study, the importance indices indicate that compression ratio and start
angle of injection are more important with respect to the influence on the maximum pressure for HFE. The results
also show that ANN-based surrogate model is an efficient way for sensitivity analysis in HFE safety assessment.
2014 Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license

2013 Published by Elsevier Ltd. Selection and peer-review under responsibility of ENAC
(http://creativecommons.org/licenses/by-nc-nd/3.0/).
Selection and peer-review under responsibility of Airworthiness Technologies Research Center, Beihang University/NLAA.
Keywords: Artificial Neural Networks; Sensitivity analysis; Safety assessment; Heavy-Fueled Engines; Airworthiness; General
aviation.

1. Introduction
General aviation piston engines have relied nearly solely on gasoline since the World War II. However,
as gasoline costs, taxation, safety, and availability apply increasing pressure to the market, the HeavyFueled Engines (HFE), which run on diesel, kerosene or other heavy fuels, are gaining popularity.
Meanwhile, HFE are imposing new challenges on safety design due to the new features not covered by

* Corresponding author. Tel.: +86-10-8233-9079.

E-mail address: jiaokun_cao@126.com.

1877-7058 2014 Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license

(http://creativecommons.org/licenses/by-nc-nd/3.0/).
Selection and peer-review under responsibility of Airworthiness Technologies Research Center, Beihang University/NLAA.
doi:10.1016/j.proeng.2014.09.061

Jiaokun Cao et al. / Procedia Engineering 80 (2014) 66 75

existing regulations[1]. In order to secure the aircraft safety and satisfy the airworthiness requirements,
safety assessment progress and system development process should interact with each other, as suggested
by System Development Process Model in Aerospace Recommended Practice (ARP) 4754[2]. System
Safety Assessment (SSA) verifies whether the safety requirements are met in the implemented design
according to ARP 4761[3].
In the process of SSA, sensitivity analysis is an essential tool. First, at design stage, it is desired to
identify the most important factors in order to get supplementary information by questioning the
manufacturer on a limited number of elements or by enhancing some aspects of analysis model[4]. And
secondly, if the safety requirements were not met, design should be changed until it satisfies the
requirements. Hence, it is typically useful to figure out which factors are the most important. Finally,
uncertainties may arise from manufacture, operation environment, maintenance, etc. After the safety
assessment, some special measures should be taken to ensure that the actual safety level of the most
important factors is consistent with those used in SSA, if necessary.
The methodology of integrating sensitivity analysis into SSA has been proposed by literature [5] based
on global sensitivity analysis methods. Literature [5] adopted a quasi-dimensional thermodynamics model
to represent the engine, while accurate, high-fidelity models are desired in practice. Typically highfidelity models are time consuming and computationally expensive and poses a serious impediment to the
sensitivity analysis. As a result, compact and cheap surrogate models are used to reduce the computation
costs. It is worth noting that high-fidelity models still needed during the processes of constructing
surrogate models. Indeed, several examples of surrogate-based sensitivity analysis have been provided in
the literatures [6, 7]. For these examples, multi-dimensional computationally expensive models are
executed due to their small computing scale. However, considering the multi-scale, multi-physics
coupling physical properties associated with HFE, even for constructing processes of surrogate models,
multi-dimensional HFE whole engine models are still extreme time consuming and computationally
expensive and not practical for engineering applications. Hence, reducing the computational costs of
high-fidelity models are necessary for HFE sensitivity analysis. In this paper, the whole system is divided
into several modules and these modules are modeled in different dimensions to reduce the computational
costs. Then the modules are coupled together by using module interfaces which transfer data between
different dimensional models. Thus, a high-fidelity HFE system model is established with relatively
affordable computational costs.
As for the surrogate model, Artificial Neural Networks (ANN) is selected for its wide use in engine
modeling. ANN consists of many interconnected processing elements typically referred to as artificial
neurons or nodes. These neurons are able to perform massive parallel computations to establish
correlations between input and output data sets[8]. ANN is capable of solving any non-linear problems by
acquiring information and recognizing the relationship between input parameters and output responses.
In this study, firstly, a framework for surrogate-based sensitivity analysis in SSA is provided
considering the complexity of HFE models. Then, as a specific example, a HFE model constituted by
quasi-dimensional thermodynamics modules and multi-dimensional Computational Fluid Dynamics
(CFD) modules is established. Finally, a surrogate model is constructed by using ANN and sensitivity
analysis is conducted based on the ANN model.
2. Methods
The framework developed in this paper is shown in Fig. 1. First, a deterministic engine model is
needed which should include parameters of interest and design outputs to be compared with the defined
safety criteria. Second, samples are generated for the factors used in engine model according to factors
distribution definition. Last but not the least, safety assessment and sensitivity analysis are conducted

67

68

Jiaokun Cao et al. / Procedia Engineering 80 (2014) 66 75

using the data provided by the engine model. If the SSA results meet the safety requirements, the design
steps into next stage. Otherwise, design should be improved until the requirements are satisfied. As
discussed above, the sensitivity analysis provides helpful information for the design improvements.
A 50 kW two-stroke lightweight heavy-fuel reciprocating engine is considered in this paper. As one of
the safety criteria, cylinder pressure should be confirmed to satisfy the power requirements but not to
exceed the safety boundary. Then, an engine processes model is necessary. In this whole engine model,
in-cylinder process is modeled with three-dimensional CFD methods, including gas flow patterns, fuel
spray behavior and combustion performance, while intake and exhaust flows are represented by onedimensional gas dynamic models to reduce computational costs. During the calculations, the data
exchanges between quasi-dimensional and multi-dimensional modules are accomplished by module
interfaces. In this case, flows at the connections are assumed to be fully developed. In other words, the
patterns of field variables at the interfaces are predefined reasonably. Hence, data from quasi-dimensional
modules could be extended to be multi-dimensional, while multi-dimensional data could be compressed
to be quasi-dimensional.
Next
stage

Engine model
High-fidelity Model

Design
Improvement

Safety
Assessment
Sensitivity
Analysis

Factors &
Safety Criteria

Quasidimensional

Interface

Multidimensional

Surrogate
Model
Samples
Generation

Data Statistics

Data
Interface

Fig. 1 A framework for surrogate-based sensitivity analysis in SSA

In this study, the variables of interest include rotation speed (Rs), compression ratio (Cr), start angle of
fuel injection (Ainj), intake temperature (Tin) and intake pressure (pin) and the maximum in-cylinder
pressure was chose as the output, considering that the maximum in-cylinder pressure in HFE is much
larger than that in traditional gasoline engines. A Back Propagation (BP) ANN model connecting these
input variables with the maximum in-cylinder pressure is constructed through the above high-fidelity
engine system model. Back-propagation neural networks with LevenbergMarquardt (LM) as the
learning algorithm have successfully been employed to model engine responses[9-13]. For these
applications, ANN generally includes three types of layers which are input layer, hidden layer and output
layer. The ANN configuration adopted in this study is illustrated in Fig. 2. Inputs are transferred to the
hidden layer, where activation functions model the complex engine behavior.

Fig. 2. ANN configuration in this study

69

Jiaokun Cao et al. / Procedia Engineering 80 (2014) 66 75

Although there is no one network configuration that could be applied universily, in engine modeling
applications, typically, the suitable number of neurons for the hidden layer ranges from 10 to 25, and the
transfer functions for hidden layer are log-sigmoid and output layer are linear. 340 data points were
collected for the ANN surrogate modeling. The ranges of data points were summarized in Table 1. The
input and output points were first normalized according to the ranges of the transfer functions. The
normalization could be expressed as

Z  Z min,i
a1,i   a2,i  a1,i  i
Z
max,i  Z min,i

Xi

(1)

where Zi is the original input or output, and (a1,i, a2,i) is selected based on the transfer function. The
normalized data points were divided into three parts with the ratio of 70:15:15, including training data,
validation data and testing data. These data was used to design ANN through supervised learning
approach. A forward pass of ANN generates errors between desired and actual outputs for training data,
and weight and bias changes were conducted according to the error and learning rules. The learning
process runs iteratively in order to reduce the errors of the training data. However, over-fitting could be
encountered during the learning process, where ANN could only produce accurate prediction for the
known data but could not give good prediction for new data. This problem is partially addressed by using
early-stopping when the errors for the validation data could not decrease.
Table 1 Ranges of engine model parameters
Model parameters

Symbols

Ranges

Rotation speed (rev/min)

Rs

2300~2600

Compression ratio

Cr

16~18

Start angle of injection ()

Ainj

-40~-20

Intake temperature (K)

Tin

273~313

Intake pressure (105Pa)

pin

1.3~1.8

Sensitivity measures based on variance decomposition give useful information for parameter
priority[14]. These model parameters are modeled by random variables, whose probability distributions are
chosen according to engineering experience from former engine models and expert judgments. For this
study, Table 2 summarizes the random variables and their distributions.. These variables are supposed to
be independent.
Assuming that Y=f(X) is the output of the model and X is the random vector of model input parameters
X=(x1, x2,, xk). Then, the first order sensitivity indices are defined as

Si

Vi ( E-i (Y | xi ))
.
VY

(2)

For a system with k factors there may be interaction terms up to the order k[15]:

S  S  S
i

ij

j !i

ijm

   S12k

1.

(3)

j !i m ! j

By substituting xi with x-i in the equation (2), we obtain V-i(Ei(Y| x-i))/VY . By definition this is the first
order effect of x-i, which can be demonstrated to equal the sum of all terms in equation (3) that do not

70

Jiaokun Cao et al. / Procedia Engineering 80 (2014) 66 75

include xi. Hence STi=1-V-i(Ei (Y| x-i))/VY equals the sum of all terms that do include xi. The total
sensitivity indices are expressed as

STi

1

V i ( Ei (Y | x i ))
VY

E i (Vi (Y | x i ))
VY

(4)

where V(g) and E(g) denote the variance and the expectation operators, respectively, and x-i means all
variables but xi.
The parameters associated to the highest values of first order indices are supposed to be the most
important factors in terms of their effects on the system output. The total indices give information on the
presence of interaction in the model. A significant difference between first order and total indices
indicates that interactions with other parameters are very important[16].
Sobols sequence was employed to determine the sensitivity indices by considering the factors
separately. Specifically, in this study, formulas proposed by [17] were used.
Table 2 Distributions of engine model parameters for global sensitivity analysis
Model parameters

Symbols

Distributions

Rotation speed (rev/min)

Rs

Uniform, 2300, 2600

Compression ratio

Cr

Normal, 17, 1.83

Start angle of injection ()

Ainj

Uniform, -40, 20

Intake temperature (K)

Tin

Uniform, 273, 313

Intake pressure (105Pa)

pin

Uniform, 1.3, 1.8

3. Results and discussions

3.1. ANN modeling
Tolerance value and R-value were adopted in this study to quantitatively evaluate the prediction
capability of the ANN. Here, tolerance value is the relative error between predicted and actual data and Rvalue is the correlation coefficient obtained by performing linear regression between predicted and actual
data. Smaller value of tolerance indicates higher accuracy of the predicted data while R-value closer to
unity signifies higher accuracy of the predicted data.
Specifically, 10 data points were randomly selected from the test data set to illustrate the prediction
capability of the ANN, and these data points are summarized in Table 3. For the selected 10 random data
points, their tolerance values and R-values are shown in Fig. 3 and Fig. 4, respectively. The tolerance
values are below 0.005 and the R-value is 0.99999. Hence, the tolerance values and the R-value indicate
that the developed ANN was able to successfully predict the engine model output, i.e., maximum incylinder pressure. Then the developed ANN could be utilized in the following sensitivity analysis.
Table 3 10 random data points from test data set

71

Jiaokun Cao et al. / Procedia Engineering 80 (2014) 66 75

x 10

Tin(K)

Rs(rev/min)

Cr

Ainj()

2405

17.5

-25.7

288.05

1.42

2549

17.3

-21.9

280.64

1.76

2476

16.9

-22.2

290.13

1.43

2465

17.1

-33.3

292.28

1.68

2575

16.6

-26.0

277.82

1.39

2386

17.5

-36.0

296.58

1.44

2527

16.4

-39.4

282.05

1.35

2526

17.4

-25.1

288.38

1.59

2414

16.4

-30.0

296.32

1.64

10

2470

16.7

-30.4

283.07

1.57

-3

Tolerance value

4
3
2
1
0

4
6
Test points No.

10

Fig. 3. Tolerance values for 10 test data points

Regression: R=0.99999
0.6
0.55

P=A
Fit
Data

Predicted values (P)

0.5
0.45
0.4
0.35
0.3
P=1.0*A+0.00026

0.25
0.2
0.2

pin(105Pa)

No.

0.3

0.4

0.5

0.6

Actual values (A)

Fig. 4. Linear regression and R-value between predicted and actual data for 10 test data points

72

Jiaokun Cao et al. / Procedia Engineering 80 (2014) 66 75

3.2. Sensitivity analysis

Mean of normalized maximum pressure

For this study, 5104 independent simulations have been performed for each of 5 factors for a total of
3.5105 model runs.
Fig. 5 and Fig. 6 show the convergence of the mean and variance of normalized maximum pressure,
respectively. The maximum pressure is normalized by the mean of maximum pressure. As shown in Fig.
5 and Fig. 6, both of the curves become steady at the end of the simulations.
1.1
1.05
1
0.95
0.9
0.85

Sample size

5
x 10

Variance of normalized maximum pressure

Fig. 5. Convergence of the mean for normalized maximum pressure.

0.04

0.035

0.03

0.025

0.02

Sample size

5
x 10

Fig. 6. Convergence of the variance for normalized maximum pressure.

Convergence of the first order indices and total indices is displayed in Fig. 7 and Fig. 8 respectively.
Table 4 shows the final results of the first order indices and the total indices.
By comparing the convergence histories of the first order indices and the total indices, it is found that
the convergence of the total indices is more stable than that of the first order indices. And for the
convergence of the first order indices, parameters with higher value have more stable convergence
histories than those with lower value.
As the first order indices show, the most important parameters are start angle of injection and
compression ratio, followed by intake pressure. The results indicate that t start angle of injection and
compression ratio should be dedicatedly designed and strictly controlled. It could also be concluded that
rotation speed has the least effects on the maximum in-cylinder pressure compared to the other
parameters considered here.

Jiaokun Cao et al. / Procedia Engineering 80 (2014) 66 75

The total indices give information on the presence of interaction in the model. The most important
factor is compression ratio instead of start angle of injection, which is different from first order indices
ranking. This means that compression ratio depends on other parameters more, compared with start angle
of injection. However, the ranking is changed due to slight difference. In general, similar to the first order
indices, start angle of injection and compression ratio have the largest total indices value.
The results provide information about the importance of the variables of interest. For the particular
study, the importance indices indicate that compression ratio and start angle of injection are more
important with respect to the influence on the maximum pressure. More efforts should be put on these
two variables in the following design and manufacturing procedures or design improvements, if necessary.
This conclusion agrees with the general engineering judgment.
0

10

Ainj

Si (First Order Indices)

Cr
-1

10

Pin

-2

10

in

-3

10

-4

10

Sample size

5
x 10

x 10

Fig. 7. Convergence of the first order indices

0

STi (total sensitivity indices)

10

Ainj
-1

10

Pin

-2

10

Tin
-3

10

Rs
-4

10

Sample size

Fig. 8. Convergence of the total indices

Table 4 Importance ranking according to first order indices and total indices
Symbols

Si

STi

Rs

0.0009

0.0007

Cr

0.4304

0.4556

Ainj

0.4407

0.4492

Tin

0.0068

0.0076

pin

0.0911

0.0976

73

74

Jiaokun Cao et al. / Procedia Engineering 80 (2014) 66 75

4. Conclusions
This paper has developed a framework for surrogate-based sensitivity analysis in SSA considering the
complexity of HFE models. Under the framework, a particular study was applied on general aviation HFE.
The results provide information about the importance of the variables of interest. For the particular
study, the importance indices indicate that compression ratio and start angle of injection are more
important with respect to the influence on the maximum pressure for HFE.
The results reveal that the framework and corresponding methods are efficient. The results show that
ANN-based surrogate model is an efficient way for sensitivity analysis in HFE safety assessment, and the
modeling methods provided in this paper could offer high-fidelity data in the process of constructing the
surrogate model with relatively low computational costs.
References
[1] Schneider J., Wilson T., Griffis C., Pierpont P. Heavy-Fueled Intermittent Ignition Engines: Technical
Issues. DOT/FAA/AR-08/42, 2009.
[2] Shih T.-M., Thamire C., Sung C.-H., Ren A.-L. Literature Survey of Numerical Heat Transfer (2000
2009): Part I. Numer Heat Tranf A-Appl, 2010. 57(3-4): 159-296.
[3] ARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne
Systems and Equipment. SAE International, 1996.
[4] Zentner I., Tarantola S., de Rocquigny E. Sensitivity analysis for reliable design verification of
nuclear turbosets. Reliab Eng Syst Saf, 2011. 96(3): 391-7.
[5] Cao J.K., Ding S.T., Sensitivity Analysis for Safety Design Verification of General Aviation
Reciprocating Aircraft Engine, in 2nd International Symposium on Aircraft Airworthiness, Z.
Shuguang, et al., Editors. 2011, Elsevier Science Bv: Amsterdam.
[6] Queipo N.V., Haftka R.T., Shyy W., Goel T., Vaidyanathan R., Kevin Tucker P. Surrogate-based
analysis and optimization. Prog Aerosp Sci, 2005. 41(1): 1-28.
[7] Stephens D.W., Gorissen D., Crombecq K., Dhaene T. Surrogate based sensitivity analysis of process
equipment. Appl Math Model, 2011. 35(4): 1676-87.
[8] Basheer I.A., Hajmeer M. Artificial neural networks: fundamentals, computing, design, and
application. J Microbiol Methods, 2000. 43(1): 3-31.
[9] Parlak A., Islamoglu Y., Yasar H., Egrisogut A. Application of artificial neural network to predict
specific fuel consumption and exhaust temperature for a Diesel engine. Appl Therm Eng, 2006. 26(8
9): 824-8.
[10] Mohamed Ismail H., Ng H.K., Queck C.W., Gan S. Artificial neural networks modelling of engineout responses for a light-duty diesel engine fuelled with biodiesel blends. Appl Energy, 2012. 92(0):
769-77.
[11] Ghobadian B., Rahimi H., Nikbakht A.M., Najafi G., Yusaf T.F. Diesel engine performance and
exhaust emission analysis using waste cooking biodiesel fuel with an artificial neural network. Renew
Energy, 2009. 34(4): 976-82.
>@ 2X] + 6DUWDV , %D\GDQ +( 3UHGLFWLRQ RI GLHVHO HQJLQH SHUIRUPDQFH XVLQJ ELRIXHOV ZLWK
artificial neural network. Expert Syst Appl, 2010. 37(9): 6579-86.
[13] Kara Togun N., Baysec S. Prediction of torque and specific fuel consumption of a gasoline engine by
using artificial neural networks. Appl Energy, 2010. 87(1): 349-55.
[14] Saltelli A., Tarantola S., Campolongo F., Ratto M. Sensitivity Analysis in Practice: A Guide to
Assessing Scientific Models. Hoboken, N.J.: Wiley, 2004.

Jiaokun Cao et al. / Procedia Engineering 80 (2014) 66 75

[15] Sobol' I.M. On sensitivity estimation for nonlinear mathematical models. Matematicheskoe
Modelirovanie, 1990. 2(1): 112-8.
[16] Saltelli A., Ratto M., Tarantola S., Campolongo F. Sensitivity analysis practices: Strategies for
model-based inference. Reliab Eng Syst Saf, 2006. 91(1011): 1109-25.
[17] Saltelli A., Annoni P., Azzini I., Campolongo F., Ratto M., Tarantola S. Variance based sensitivity
analysis of model output. Design and estimator for the total sensitivity index. Comput Phys Commun,
2010. 181(2): 259-70.

75