Real-Time Ship Motion Forecasting Using Deep Learning

Mohammad Hasanur Rashid Jing Zhang Zhao Minghao

School of Information Science and School of Information Science and School of Information Science and
Engineering, Shandong Provincial Engineering, Shandong Provincial Engineering, Shandong Provincial
Key Laboratory of Network-Based Key Laboratory of Network-Based Key Laboratory of Network-Based
Intelligent Computing, University of Intelligent Computing, University of Intelligent Computing, University of
Jinan, Jinan, Shandong 250022, China, Jinan, Jinan, Shandong 250022, China, Jinan, Jinan, Shandong 250022, China
m.hasanur25@gmail.com ise_zhangjing@ujn.edu.cn, 18428339012@163.com
Corresponding Author

ABSTRACT 1 INTRODUCTION
It is still challenging to continuously observe the marine ship mo- Literature studies show that it is a precarious proposition to conduct
tion in a harsh environmental condition. Combined science and operations and works at the sea because of high seas, internal
technology assistance makes human maritime activities undergo waters, harsh environment, contiguous zone, etc. In the future, we
a revolution with increasing artificial intelligence aboard ships. seek to improve our safety, reduces life risk and workability, or
For instance, it is now possible to embed sensor processing to au- operational availability by collecting detailed information about
tomatize tasks that humans could perform at a time. As a result, the sea surface. The incident wavefield and predicting precise time
piloting ships among sea waves is a particular interest. In this paper, and defined thresholds exceed operating limit when conditions
we address the prediction based ship’s motion using deep learn- remain within limits [1]. It can also use for research and studies
ing. To create a 3D image, we use a computer graphic software for coherent (phase and amplitude) knowledge where requires the
named Blender. It is possible to simulate ships floating at sea level, ocean surface over a wide area of interest.
recording sea surface images and the vessels’ motion parameters, Nowadays, traditional water transportation methods are un-
considering their pitch and roll. We employ different kind of neural dergoing tremendous changes. People are demanding higher re-
network models for training include convolutional neural network quirements, such as being more efficient, comfortable, and safe, etc.
(CNN), the combination of convolutional neural network with long Among them, safety is one of the critical development directions.
short term memory (LSTM), and the combination of convolutional Automated navigation, real-time ship motion forecasting, ocean
neural network with a gated recurrent unit (GRU) network. After waves reconstruction, navigational state prediction, and other tech-
training, we analyze different models’ performance and discuss nologies emerge in this context [2]. These directions can handle
how different time gap affects models’ performances. some more common scenarios such as weather prediction, side-
by-side cargo transfer, etc. In the meantime, they introduced some
CCS CONCEPTS unexpected problems, which are related to stability, energy con-
servation, artistic control, and spatial resolution [3]. Besides, the
• CCS –> Computing methodologies –> Modeling and sim-
waves are also approximated by some methods as wave particles
ulation –> Model development and analysis –> Modeling
and wave packets.
methodologies;
In 2008, Zamani et al. presented some data-driven models for the
Caspian Sea wind-wave process. They used buoy measurements to
KEYWORDS determine the problems associated with their models to forecast
Ship Motion Forecasting, Deep Learning, CNN, LSTM, GRU significant wave heights for several hours ahead. They designed
all their models based on artificial neural networks (ANN) and
ACM Reference Format: instance-based learning (IBL) [4]. Their experiments showed that
Mohammad Hasanur Rashid, Jing Zhang, and Zhao Minghao. 2021. Real- the ANN’s yielded slightly better than the measurements with IBL.
Time Ship Motion Forecasting Using Deep Learning. In The 2nd International
In 2016, Kusters et al. designed a standalone system to predict ves-
Conference on Computing and Data Science (CONFCDS 2021), January 28–
30, 2021, Stanford, CA, USA. ACM, New York, NY, USA, 5 pages. https:
sel motion. Moreover, that operation depends on time-series based
//doi.org/10.1145/3448734.3450923 upon wide-area ocean-surface radar operational-observations [5].
The system can provide detailed trends of the measured resultant
vessel motions, sea heights, and real-time wave characterization
Permission to make digital or hard copies of all or part of this work for personal or giving operational vessel planning guidance for hours and days
classroom use is granted without fee provided that copies are not made or distributed ahead of the statistical ship-motion forecaster.
for profit or commercial advantage and that copies bear this notice and the full citation Later in 2018, Oehmcke et al. presented a deep learning archi-
on the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, tecture used for a marine virtual sensor application that is utilized
to post on servers or to redistribute to lists, requires prior specific permission and/or a input quality information [6]. They proposed a novel input quality
fee. Request permissions from permissions@acm.org.
based on the dropout layer to take advantage of information about
CONFCDS 2021, January 28–30, 2021, Stanford, CA, USA
© 2021 Association for Computing Machinery. the input sensors’ quality- That virtual sensor model has been built
ACM ISBN 978-1-4503-8957-0/21/01. . . $15.00
https://doi.org/10.1145/3448734.3450923
CONFCDS 2021, January 28–30, 2021, Stanford, CA, USA Mohammad Hasanur Rashid et al.

In this way, we can collect the sea surface conditions in front of the
ships and the ships’ corresponding motion parameters. Eventually,
we managed 5,000 pictures of the sea surface images of the two
boats, respectively. After generating these two data sets, we mix
them for further usage.

2.2 Reference data Processing for Supervised

Training
Figure 1: Blend image dataset example for input. Before using the data for training, there needs to process of the
data according to the model requirement. The primary purpose is
to reduce computational workloads and make the movement more
upon a combination of convolutional and recurrent long short-term accurate. For images, the data process mainly consists of three
memory layers. parts, picture cropping, image resizing, and picture normalization
However, using deep learning methods to predict waves or the [11]. The original image has a resolution of 270×486. We select the
motion of a ship is rarely studied. The waves play an essential role entire image for a 3x linear reduction (resolution of 90×162) and
in the navigation of the vessel. Many accidents can avoid if we then normalize the pixel values- between -1 and 1. These data are
can know the ship’s navigation state in advance and make timely collected into a JSON file in the form of a dictionary, the name-
adjustments. With technology development, such as machine learn- value pairs of an unordered collection where the names ( It is also
ing, advanced ship motion status prediction is gradually becoming called keys, each key is the name of a sea surface image collected)
possible [7]. In this research, we use a blender image to feed them represents the frame number [2, 7]. The corresponding value is a
into a neural network for training. An example of blender 3D im- list of the ship’s motion parameters. As we want to evaluate how
ages is shown in Figure 1, considering the surface and the ship’s long in advance, the ship motion prediction has performed the
state. Finally, we analyze the neural network models to predict the best. We create datasets with different time shifts by associating
ship’s state from sea surface photos. The input is stored in its net- each image with the image’s valueN 1 step after, keeping the ship’s
works instead of a database; hence the loss of data does not affect corresponding parameters unchanged.
its working to perform multiple tasks in parallel without affecting In this way, we change the correspondence of name-value pairs.
the system performance. The neural network will then learn from this relationship to predict
the ship’s movement at different times ahead. It should be men-
2 DATA tioned that when using data for training, the data is loaded in the
2.1 Data Collection order of the sequence of frames [12]. The CNN-LSTM and CNN-
GRU model requires data to be time-ordered. Also, the CNN model
A ship can experience six degrees of freedom, includes pitch, heave, requires two pictures that are close in time as input; otherwise,
roll, surge, yaw, sway, which are defined as the ship motion. We the model will not learn correctly if the two images are arbitrarily
mainly consider the parameters pitch and roll because yaw is pri- chosen. The frame interval between the two images can be adjusted
marily controlled by man (fixed in our case), while heave, sway, and (after a few tries, we set it to 3). The output is the corresponding roll
surge are relatively less important to measure the motion. So we and pitch of the ship [13]. Since the two images can provide more
choose pitch and roll as prediction targets. We take more factors into information, the model should compare the difference between
account to study further, such as SATA processing using Blender. the two images to determine the ship’s motion parameters more
Blender is a professional open-source 3D computer graphics soft- accurately.
ware toolset. It can be used to create 3D images, animated films,
video games, visual effects, art, and so on [8]. Blender’s features 3 METHODS
include 3D modeling, rendering, texturing, raster graphics editing,
smoke simulation, fluid, sculpting, particle simulation, animating, 3.1 Deep Learning
camera tracking, match moving, motion graphics, compositing, and Convolutional neural networks (CNN) is a feed-forward artificial
video editing. neural network, a deep neural network class, and applied to analyze
To process the data, we use a simple simulation technique repre- visual imagery [8, 10]. CNN has several advantages over other tech-
senting that the boat will respond to sea motion, without looking niques, like fewer weights as some parameters are shared, smaller
for physical accuracy or very realistic wave images. Therefore, we computational workload, and better performance. They are used
simulate a moving sea with two ships approximated by simple in various areas, including speech recognition, natural language
cubes floating on it with a simple physical model [9]. The boat processing, video classification, image and pattern recognition, etc.
moves with the waves in two different positions; one is horizontal, We suppose that all inputs (and outputs) will be independent when
and another moves vertically with the sea wave. A camera is set using a convolutional network.
to record the movement of the sea surface in front of each ship. Our first model uses only one image as input, and the output is
The cameras are stabilized, which means they rarely sway with the the corresponding ship’s roll and pitch. It contains two convolu-
ship to reduce the pictures’ noise, eliminating the need for angle tional layers. Each of them is followed by a relu layer, a normal-
correction and other redundant operations [10]. At the same time, ization layer, a max-pooling layer, and two fully-connected layers.
we use a python script to record the ships’ roll and pitch at all times. Afterimage preprocessing, the input will hold three color channels
Real-Time Ship Motion Forecasting Using Deep Learning CONFCDS 2021, January 28–30, 2021, Stanford, CA, USA

Figure 2: Block diagram of a CNN-LSTM.

R.G.B, the raw pixel values of the image; in this case, a picture
of width 90, height 162. The Convolutional layer is connected
to local regions as the input, which will compete with the output
of neurons, and each computing is a dot product between their
weights and a small area they will be connected to the input vol-
ume [14]. Such as [45×81×16] if we decide to use 16 filters whose
size is [5×5], a stride of 2, padding of 2, and dilation of 1.
Figure 3: Block diagram of CNN-GRU.
A recurrent neural network (RNN) is a class of artificial neural
networks where connections between nodes form a directed graph
along a sequence. This RNN allows it to exhibit temporal dynamic of candidate value Ct
e to be added to the state. With the information
behavior. Derived from feed-forward neural networks, RNN’s can above, we can now update the cell state Ct at time step t.
use their internal state (memory) to process variable-length se-
quences of inputs. RNN also makes them applicable to tasks such Ct = ft ∗ Ct −1 + i t ∗ C̃t (5)
as unsegmented, connected handwriting recognition, or speech Finally, we calculate to the output. The output is determined by
recognition. So RNN is widely applied to various problems, like cell stateCt −1 , hidden state ht −1 , and input X t .
speech and image recognition, language modeling, translation, etc.
ot = σ (W0 [ht −1 , x t ] + b0 ) (6)
3.2 Combination of Convolutional and ht = ot ∗ tanh(Ct ) (7)
Recurrent Neural Network ot is what parts of the cell state we are going to output. yt is the
Long Short-Term Memory (LSTM) is a type of artificial recurrent final output at time step t, which is also the hidden state at time
neural network (RNN) used in deep learning to learn long-term step t.
sequence dependency among a dataset sequence. LSTM’s have a We should combine convolutional and LSTM networks to
significant advantage in making predictions in time-series predic- use their respective advantages fully. As the image is a three-
tion DN problems of understanding the context required, corrected dimensional array, CNN needs to be processed first to extract the
context, and rather than having this predefined prediction. LSTMs image features. Then, we apply the function to convert the elements
have a chain-like structure, a repeating module which can help to a vector. In this way, we can use a vector to represent an image
them memorize information for long periods [10, 14]. Indeed, they [15]. Next, we pass a vector sequence generated in the previous
allow choosing what information to keep in memory or forget, thus steps to an LSTM network to continue learning. Finally, the data is
solving long-term dependencies. The architecture of a CNN-LSTM given to a fully-connected layer to create the output.
is shown in Figure 2 Although the LSTM model has shown remarkable performance
The equations corresponding to the structure of the module are: in learning long-term sequential data, the LSTM is a lot of weights
that need to be trained in its intricate architecture. A new simplified
ft = σ (Wf ∗ [Ct −1 , ht −1 , x t ] + bf ) (1) model is introduced, which is called the GRU (Gated Recurrent Unit),
i t = σ (Wi ∗ [Ct −1 , ht −1 , x t ] + bi ) (2) to reduce the training time of the LSTM. The GRU model generally
takes a shorter time for training than similar performance to the
C̃t = tanh(Wc ∗ [ht −1 , x t ] + bC ) (3) LSTM. The architecture of a CNN-GRU is shown in Figure 3
ot = σ (W0 ∗ [Ct , ht −1 , x t ] + b0 ) (4)
4 NUMERICAL EXPERIMENTS
X t is the input at time step t,ht −1 is the hidden state at time step
t−1, andyt is the output at time step t.Ct −1 is the cell state at time t 4.1 Experimental Setup
−1. The yellow unit is called the "forget Gate layer", decides what In this section, we choose the prediction frame gap of 3 as an
to remember and what to forget. The middle is the "update layer," example to analyze. To measure the quality of a model’s predictions,
consisting of a sigmoid layer and a tanh layer. The sigmoid layer we use entropy. We can see that the CNN network’s performances
decides which value will update, and the tanh layer creates a vector with one image as input the same as the CNN-LSTM with one-shot
CONFCDS 2021, January 28–30, 2021, Stanford, CA, USA Mohammad Hasanur Rashid et al.

Figure 4: Relation between prediction frame gap and test loss of (a) CNN model, (b) CNN-LSTM model, and (c) CNN-GRU model
with one input image.

[9, 15]. To some extent, the performances of the former are even We performed training with a batch size of 8 for 100 epochs, a
better, which is unexpected. However, the amount of information weight decay rate of 1 × 10−3, and using a studying rate of 1 ×
that does not contribute to the motion knowledge discovery was 10−5 for the optimizer. As shown in Figure 4, test loss fluctuates
discarded. There is an upper threshold or a particular point of with different prediction frames because the initialization of model
diminishing returns for the number of data points included in the parameters, data load, sequence, and other subtle factors is also
sea surface’s learned system, beyond which the additional data do the training result. However the overall minimum test loss can be
not provide further useful information to the historical sea surface achieved when the prediction frame gap is about 3, regardless of
system. the model type. It indicates that the prediction of a ship’s motion
Therefore, the discovery process of sea surface pattern knowl- parameters based on sea surface images is feasible.
edge can link to entropy, which measures the degree of disorder
in a system. Information theory entropy is widely employed to 5 CONCLUSION
predict ship motion. Entropy provides a measure of the extent to We proposed a neural network approach to forecast ship motion
which the movement can expect based on the area’s historical pat- from images of the incoming waves in this work. We generated a
terns. Within this framework, entropy can quantify the information synthetic dataset for this problem and compared the CNN model’s
gain that the derived motion patterns will predict. In geographical performance, CNN-LSTM model, and CNN-GRU model. Among
clustering studies, the notion of entropy has been suggested and the three models, the performance of CNN-GRU is the best. Both
recently applied to detect abnormal activities in video surveillance two CNN and CNN-LSTM models not comparable to the CNN-
[13]. Consequently, the detection of potential anomalies can link GRU model. In general, the prediction result is consistent with the
to the motion entropy: the capability to recognize low-likelihood 3D simulation, but it is rough in terms of detail. Also, due to the
behaviors successfully enhance areas where the motion patterns hull’s characteristics, pitch prediction is more complicated than
are highly regular. Therefore, the associated level of disorder is low. roll, which leads to low pitch prediction.

4.2 Result ACKNOWLEDGMENTS

With the dataset described in the previous section, we trained our This research is supported by funding of the National Natural Sci-
three neural models for different prediction horizon (frame gaps). ence Foundation of China (NSFC) (2017–2020, No. 51679058). And
Real-Time Ship Motion Forecasting Using Deep Learning CONFCDS 2021, January 28–30, 2021, Stanford, CA, USA

financing of Shandong Natural Science Foundation in China (2020- [7] Zhi ZHOU, "Predicting Ship’s Motion Based on Machine Learning," unpublished.
2022 No.ZR2019LZH005) [8] M. R. Ahmed, M. S. Ahammed, S. Niu and Y. Zhang, "Deep Learning Approached
Features for ASD Classification using SVM," 2020 IEEE International Conference
on Artificial Intelligence and Information Systems (ICAIIS), Dalian, China, 2020,
REFERENCES pp. 287-290.
[1] Daesoo Lee, Seung Jae Lee, "Motion predictive control for D.P.S. using predicted [9] Redouane Lguensat, Ronan Fablet, Julien Le Sommer, Sammy Metref, Emmanuel
drifted ship position based on deep learning and replay buffer," International Cosme, Kaouther Ouenniche, Lucas Drumetz, Jonathan Gula," Filtering Internal
Journal of Naval Architecture and Ocean Engineering, Volume 12, 2020, Pages Tides From Wide-Swath Altimeter Data Using Convolutional Neural Networks"
768-783. unpublished.
[2] Thomas Bolton, Laure Zanna," Applications of Deep Learning to Ocean Data [10] Kaiming He; Xiangyu Zhang; Shaoqing Ren; Jian Sun," Deep Residual Learning
Inference and Subgrid Parameterization," Journal of Advances in Modeling Earth for Image Recognition," 2016 IEEE Conference on Computer Vision and Pattern
Systems, Volume11, Issue1, January 2019, Pages 376-399. Recognition (CVPR), Electronic ISSN: 1063-6919.
[3] Filippo Bergamasco, Andrea Torsello, Mauro Sclavo, Francesco Barbariol, Alvise [11] M. R. Ahmed, Y. Zhang, Y. Liu and H. Liao, "Single Volume Image Generator and
Benetazzo,” WASS: an Open-Source Pipeline for 3D Stereo Reconstruction of Deep Learning-Based ASD Classification," in IEEE Journal of Biomedical and
Ocean Waves,” Computers and Geosciences, Volume 107, p. 28-36. Health Informatics, vol. 24, no. 11, pp. 3044-3054, Nov. 2020.
[4] Ahmadreza Zamani; Dimitri Solomatine; Ahmadreza Azimian; Arnold Heemink," [12] Weiqiang Zhu, Yixiao Sheng, Yi Sun," Wave-dynamics simulation using deep
Learning from data for wind–wave forecasting," Ocean Engineering, ISSN: 0029- neural networks," June 2017
8018, Vol: 35, Issue: 10, Page: 953-962. [13] J. Li, Lianmin Zheng," DEEPWAVE: Deep Learning based Real-time Water Wave
[5] J.G. Kusters; K.L. Cockrell; B.S.H. Connell; J.P. Rudzinsky; V.J. Vinciullo," Future- Simulation," 2018.
Waves™: a Real-Time Ship Motion Forecasting System employing Advanced [14] Philipp Fischer, Thomas Brox," U-Net: Convolutional Networks for Biomedical
Wave-Sensing Radar," OCEANS 2016 MTS/IEEE Monterey, Electronic ISBN:978- Image Segmentation," Part of the Lecture Notes in Computer Science book series
1-5090-1537-5, Print on Demand(PoD) ISBN:978-1-5090-1527-6 (LNCS, volume 9351).
[6] Stefan Oehmcke; Oliver Zielinski; Oliver Kramer," Input quality aware convo- [15] Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton," ImageNet Classification
lutional LSTM networks for virtual marine sensors," Neurocomputing, ISSN: with Deep Convolutional Neural Networks," Communication of the A.C.M., May
0925-2312, Vol: 275, Page: 2603-2615. 2017, Vol. 60, No. 6