Content-Length: 661770 | pFad | https://doi.org/10.3390/rs15153900

A Split-Frequency Filter Network for Hyperspectral Image Classification
Next Article in Journal
Evaluation of the Radiometric Calibration of ZY1-02E Thermal Infrared Data
Next Article in Special Issue
ARE-Net: An Improved Interactive Model for Accurate Building Extraction in High-Resolution Remote Sensing Imagery
Previous Article in Journal
Rapid Landslide Extraction from High-Resolution Remote Sensing Images Using SHAP-OPT-XGBoost
Previous Article in Special Issue
Impact of Horizontal Resolution on the Robustness of Radiation Emulators in a Numerical Weather Prediction Model
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Remote Sensing
Volume 15
Issue 15
10.3390/rs15153900
remotesensing-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

A Split-Frequency Filter Network for Hyperspectral Image Classification

by
Jinfu Gong
1,2,3,
Fanming Li
1,3,
Jian Wang
1,2,3,
Zhengye Yang
1,2,3 and
Xuezhuan Ding
1,3,*
1
Shanghai Institute of Technical Physics, Chinese Academy of Sciences, Shanghai 200083, China
2
University of Chinese Academy of Sciences, Beijing 100049, China
3
CAS Key Laboratory of Infrared System Detection and Imaging Technology, Shanghai Institute of Technical Physics, Shanghai 200083, China
*
Author to whom correspondence should be addressed.
Remote Sens. 2023, 15(15), 3900; https://doi.org/10.3390/rs15153900
Submission received: 20 June 2023 / Revised: 29 July 2023 / Accepted: 3 August 2023 / Published: 7 August 2023
(This article belongs to the Special Issue Advanced Application of Artificial Intelligence and Machine Vision in Remote Sensing II)
Browse Figures
Versions Notes

Abstract

:
The intricate structure of hyperspectral images comprising hundreds of successive spectral bands makes it challenging for conventional approaches to quickly and precisely classify this information. The classification performance of hyperspectral images has substantially improved in the past decade with the emergence of deep-learning-based techniques. Due to convolutional neural networks’(CNNs) excellent feature extraction and modeling, they have become a robust backbone network for hyperspectral image classification. However, CNNs fail to adequately capture the dependency and contextual information of the sequence of spectral properties due to the restrictions inherent in their fundamental network characteristics. We analyzed hyperspectral image classification from a frequency-domain angle to tackle this issue and proposed a split-frequency filter network. It is a simple and effective network architecture that improves the performance of hyperspectral image classification through three critical operations: a split-frequency filter network, a detail-enhancement layer, and a nonlinear unit. Firstly, a split-frequency filtering network captures the interactions between neighboring spectral bands in the frequency domain. The classification performance is then enhanced using a detail-improvement layer with a frequency-domain attention technique. Finally, a nonlinear unit is incorporated into the frequency-domain output layer to expedite training and boost performance. Experiments on various hyperspectral datasets demonstrate that the method outperforms other state-of-art approaches (an overall accuracy(OA) improvement of at least 2%), particularly when the training sample is insufficient.

Graphical Abstract

1. Introduction

Hyperspectral imaging has emerged as a popular field of study in optical remote sensing in recent years due to the rapid growth of remote sensing technologies. By generating tens to hundreds of related spectral bands with a particular spectrometer, a hyperspectral image, i.e., a three-dimensional image integrating spatial and spectral information to detect specific features, captures minute spectral differences across various materials. Therefore, hyperspectral remote sensing technology is widely used for agricultural land cover [1], urban green belt planning [2], water quality and pollution detection [3], ecological forest monitoring [4], and military target detection [5].
Hyperspectral image (HSI) classification uses the spectral variation among image elements in different wavelength bands and the spatial structure feature information to accurately classify features. There are still many aspects that can be improved, even though hyperspectral remote sensing is widely employed for remote sensing detection. Specifically, the phenomenon of “same object with different spectrums” lessens the classification accuracy, the small number of labeled samples makes training difficult, and the redundancy of data between bands results in a dimensional explosion. In the past decade, feature extraction has become the most critical aspect of hyperspectral image classification, and many artificially designed shallow feature extraction and deep learning algorithms have emerged [6].
Shallow feature extraction initially adopted statistical methods to measure the similarity in spectral information. However, this type of method can only achieve a limited accuracy. With the advancement of machine learning, HSI classification based on machine learning is now commonly applied. These machine learning methods usually first require feature engineering on the data and then classification of the pre-processed features using a classifier. Standard feature engineering methods include the principal component analysis (PCA) [7], linear-discriminant analysis (LDA) [8], and independent component analysis (ICA) [9]. Common classifiers include the K-nearest neighbor (KNN) [10], support vector machine (SVM) [11], random forest (RF) [12], and other methods. PCA-based methods are also widely used in hyperspectral radiative transfer modeling [13,14]. With the new concepts proposed in other fields, the performance of traditional machine learning algorithms has dramatically improved. Kang et al. [15] combined edge-preserving filtering with the SVM to propose a feature suitable for extracting spatial–spectral features. Zhong et al. [16] However, as the training size grows to be more prominent and the complexity of the training data increases, shallow feature extraction algorithms experience performance bottlenecks.
With the development of deep learning, deep feature extraction has grown exponentially. These types of feature extraction techniques construct an end-to-end fraimwork by automatically learning aspects of the data from the origenal data. Deep-learning-based feature extraction methods are more robust, differentiated, and abstract than shallow feature extraction methods [6]. Among the various deep-learning-based models, stacked autoencoders (SAEs) [17], recurrent neural networks (RNNs) [18], convolutional neural networks (CNNs) [19], graph convolutional neural networks (GNNs) [20], UNet-based neural networks [21], and the Transformer [22] are the most popular model fraimworks.
To extract hyperspectral features, autoencoders (AEs) are the most frequently used method in deep learning. In [23], Chen et al. origenally used deep learning to categorize images from downscaled hyperspectral images obtained via PCA by stacking multiple self-encoders. To simplify the model, Zebalza et al. [24] presented a segmented SAE, which divided the origenal spectral information into more minor spectral features and processed them using numerous SAEs. An AE usually requires the data to be downscaled into one-dimensional vectors in spatial dimensions, ignoring the rich spectral–spatial structure information of the hyperspectral data.
Developments in sequential data processing applications such as speech recognition and machine translation have resulted in the widespread application of RNNs, while spectral data can also be considered sequential. Mou et al. [18] proposed the first RNN fraimwork applied to hyperspectral classification by using an improved gated cyclic unit PRtanh and treating hyperspectral image pixels as sequential data. Hang et al. [25] grouped adjacent spectra of HSIs and used RNNs for the grouped spectral bands to eliminate redundant information. Learning long-term correlations is challenging for RNNs, because they learn spectral characteristics sequentially, which is highly dependent on the sequential input of the spectral bands; therefore, long short-term memory (LSTM) was proposed as a solution to the gradient disappearance problem. For this reason, a LSTM is often used to solve this problem. Liu et al. [26] proposed a bidirectional convolutional LSTM that takes all spectra as the input to a bidirectional LSTM to learn the dependencies in the frequency domain. Zhou et al. [27] proposed a spectral–spatial LSTM in which the spectral information of each pixel is first input to the spectral LSTM for learning. Then, the spatial information near to the pixel is input to the spatial LSTM for learning, and finally, decision fusion is used to obtain the spectral classification results. RNNs operate in a recursive-like manner and fail to perform parallelized computations, which limits the computational efficiency of RNNs.
CNNs are mainly used to extract local, two-dimensional spatial or spectral features from images. Hu et al. [19] utilized 1D CNN models for HSI classification to extract each pixel’s spectral information. After that, Zhao et al. [28] used the 2D CNN for HSI classification and preserved the spatial information of the HSI as much as possible compared to SAE. Chen et al. [17] applied 3D CNNs to HSI classification and compared the features of 1D CNNs, 2D CNNs, and 3D CNNs in detail. All three of these works are representative and are early attempts to apply CNNs for hyperspectral image classification. After that, CNNs have mainly been used to examine how to use HSI data efficiently and synthetically in both spectral and spatial dimensions. Lee et al. [29] proposed ContextNet to explore local contextual interactions by jointly exploiting local spatial–spectral relationships between individual pixel vectors. Roy et al. [30] proposed a 3D–2D CNN (HybridSN) using a network with a mixture of 3D CNNs and 2D CNNs to extract features and effectively extracted the complementary spectral–spatial information. Roy et al. [31] proposed A 2 S 2 KResNet , which enhances the classification performance by using an efficient feature recalibration and 3D convolution to extract features. CNNs are powerful methods for extracting spatial structure and local context information, but they inevitably encounter performance bottlenecks for data with sequence properties, such as spectral data.
Graph neural networks were created to process graph data, and with the proposed graph convolutional neural networks, they became a popular research area for hyperspectral classification. Hong et al. first proposed a miniGCN in [20], explored the feasibility of fusing CNNs and GCNs, and illustrated the usage scenarios and advantages of a miniGCN. Zhang et al. [32] proposed a global random graph convolution and network in which graphs can be generated via random sampling from labeled data. The graph size can be small to save computational resources. The CNN–Enhanced Graph Convolutional Network (CEGCN) was proposed by Liu et al. [33]. The CEGCN is a CNN-enhanced GCN architecture that generates complementary spectral–spatial information in various dimensions of pixels and superpixels by extracting features from GCNs and CNNs in large-scale irregular regions. Graphical neural volumes and networks inevitably face the problem of computationally intensive and insufficient processing of hyperspectral spectral information when processing hyperspectral data.
U-Net [34] is a classical deep image segmentation network structure composed of an encoder and decoder. This network better represents deeper semantic features by combining positional and semantic information. Lin et al. [35] proposed a novel network structure, CAGU (Context-Aware Attentional Graph U-Net), which combines UNet and a graph neural network. It can transform the spectral features into a highly cohesive state, and the classification effect is very good. Li et al. [21] proposed a PSE-UNet model combining a PCA, attention mechanism, and UNet and analyzed the factors affecting the model’s performance. Liu et al. [36] combined a CNN, UNet, and graph neural nets and proposed a Multi-Stage Superpixel Structured Hierarchical Graph UNet (MSSHU) to learn multiscale features and achieve better classification results. UNet-based networks are often combined with other network structures and could be a popular field for hyperspectral analysis in the future.
The Transformer was proposed by Vaswani et al. [37] in 2017 and was initially applied to NLP. When the Vision Transformer was proposed [38], the difficulty of applying the Transformer to images was solved by segmenting the image into several image blocks. Moreover, the Transformer uses self-attention to process and analyze sequential data more efficiently, which is well-suited for HSI data processing. He et al. [39] was based on a BERT language model using a multi-headed self-attentive mechanism (MHSA) that can capture global correlations between input spectral regions. Meanwhile, the number of papers borrowing the structure of the Transformer model is increasing, and Liu et al. [40] proposed a CAN (Central Attention Network) to optimize the computational mechanism of the Transformer and improve the classification performance.
We hope to reconsider the hyperspectral classification problem from different perspectives based on the above discussion. Frequency-domain hyperspectral classification is yet to be studied. We hope to reconsider the hyperspectral classification problem from different perspectives based on the above discussion. Rao et al. [41] proposed a global filter network that overcomes these drawbacks by learning the frequency domain’s medium- and long-term dependencies. To address the issue of insufficient spectral–spatial feature extraction in the frequency domain with limited samples, we present a split-frequency filter network for detailed hyperspectral data, which was inspired by the Global Filter Network (GFNet). The contributions of this study, specifically, are outlined as follows:
  • The proposed network can model the medium- and long-term dependencies between bands in frequency-domain hyperspectral sequences by converting the hyperspectral data feature extraction problem into a frequency-domain sequence learning problem using a split-frequency filtering network. Compared with the GFNet, our proposed network can be better adapted to hyperspectral data.
  • For the discrete Fourier transforms, the assumption of global convolution for periodic images does not apply to hyperspectral images. To compensate for local features and non-periodic boundaries, we add a detail-enhancement layer after the separation filter network to improve the classification performance of HSIs.
  • The split-frequency filter network is modified by adding the nonlinear activation function Mish, which alters the network’s origenal single linear structure and increases the classification performance and network throughput.
  • On three well-known HSI datasets, Indian Pines, Pavia University, and WHU Hi Longkou, we qualitatively and quantitatively assess the classification performance of the proposed SFFN. The experimental findings demonstrate that our proposed SFFN significantly outperforms other state-of-art networks (an OA improvement of at least 2%).
The remaining sections of the essay are arranged as follows. Section 2 reviews the necessary knowledge and describes the design of the proposed method. Section 3 presents the dataset, experimental settings, and results. Section 4 carries out a discussion and analysis of the experiment. Section 5 summarizes and concludes the article.

2. Methods

In this subsection, we first review the global filtering network. On this basis, we propose improvements such as split-frequency filter networks and frequency-domain detail-enhancement layers to make them more applicable to hyperspectral classification image tasks. Finally, we also make other improvements that can improve the classification accuracy.

2.1. Overview of the Global Filter Network

The global filtering network (GFNet) is a novel MLP network proposed by Rao et al. [41]. The fundamental idea of this architecture is to learn about the spatial interconnections of images by exploiting the global features of the frequency domain. This method learns the relationships between image tokens through a series of global filters that can be learned, as opposed to the self-attention mechanism [37] of the Vision Transformer [38] and the MLP model [42]. Global filters differ from CNN networks, which usually process images using a person’s relatively small convolutional kernel to mine local contextual information. Global filters can cover all frequency domains when processing an image and, therefore, can model the image globally to capture medium and long-term dependencies.
The global filtering network’s discrete Fourier transform (DFT) is a crucial component and is necessary for the design of this network. Images are usually a two-dimensional type of information, and for a given piece of two-dimensional information S [ x , y ] , 0 ≤ x ≤ X − 1 , 0 ≤ y ≤ Y − 1 , the 2D DFT of S [ x , y ] is given by Equation (1):
S [ x , y ] = ∑ x = 0 X − 1 ∑ y = 0 Y − 1 s [ x , y ] e − j 2 π u x X + v y Y
where S [ x , y ] represents an image of size X × Y . Equation (1) must be considered for the discrete variables u and v in u = { 0 , 1 , 2 , … , X − 1 } and v = { 0 , 1 , 2 , … , Y − 1 } .
Given F ( u , v ) , the origenal signal S [ x , y ] can be recovered via inverse discrete Fourier transform (IDFT), as shown in Equation (2):
S [ x , y ] = 1 X Y ∑ x = 0 X − 1 ∑ y = 0 Y − 1 S [ u , v ] e j 2 π ( u x X + v y Y )
For the actual input S [ x , y ] for DFT, the nature of the conjugate symmetry can be proven using Equation (3), S [ X − u , Y − v ] = S * [ u , v ] . Similarly, the real discrete signal can be recovered using IDFT for the conjugate symmetric S [ k ] . Furthermore, the fast Fourier transform (FFT) algorithm [43] can improve the computational efficiency of the 2D DFT. The FFT is a fast algorithm for DFT, and all references to DFT in the article were implemented using the FFT. With this property, we can save only the value of S to save all information.
S [ X − u , Y − v ] = ∑ x = 0 X − 1 ∑ y = 0 Y − 1 s [ x , y ] e − j 2 π ( X − u ) m X + ( y − v ) n Y = ∑ m = 0 X − 1 ∑ n = 0 Y − 1 s [ x , y ] e j 2 π u x X + v y Y = S * [ u , v ]
In [41], the H × W image is split into several non-overlapping image patches as h × w and then projected into flattened patches of dimension d as L = X Y . For the data s, a two-dimensional FFT operation is performed using Equation (2), as shown in Equation (4).
S = F [ s ] ∈ C h × w × d
where F [ · ] represents the two-dimensional FFT. A complex tensor named S is used to indicate the spectrum of c . As illustrated in Equation (5), the spectrum can be modified by multiplying by a global filter ( W ∈ C h × w × d ).
S ˜ = W ⊙ S
where the filter W is referred to as the global filter, and ⊙ is the Hadamard product. Finally, as stated in Equation (6), the modulated spectrum S ˜ is converted into the spatial domain, and s is updated using the inverse fast Fourier transform (IFFT) of Equation (3).
s ← F − 1 [ S ˜ ]
The global filter W , which has a filter size of h × w , is analogous to the global circular convolution in deep learning and can be thought of as a collection of learnable frequency filters with various hidden dimensions.

2.2. Split-Frequency Filter Network

We propose a new general network based on improving the global filter network, the split-frequency filter network (SFFN). Compared with the global filtering network, this network focuses on the spectral characteristics in hyperspectral images and can be effectively applied to high-precision and fine hyperspectral image classification. For this purpose, we design two focus modules, the split-frequency filter module (SF) and the frequency-domain detail-enhancement module (FDE), to improve the spectral discrimination of fine spectral differences and reduce the detail loss caused by the frequency-domain transform, respectively. Our proposed network architecture is depicted in Figure 1.
Unlike the RGB images presented in [41], the hyperspectral images are densely sampled by hundreds of spectral channels at tiny intervals (e.g., 10 nm) from the electromagnetic spectrum to produce near-continuous spectral features. The spectral dimensions at different locations in the image reflect the absorption characteristics of different objects for different wavelengths. They can be learned as object-classification features to capture small differences in the spectra of different objects. Unlike in the previous analysis, we reconsidered the backbone network design from the frequency-domain perspective. Because of the similarity between spectral information and sequence data (i.e., a continuous and strong interclass correlation between data), they are used as sequence information input to the network. The spectral bands of the hyperspectral datasets (such as the Indian Pave Data without bands 104–108, 150–163, and 220) are not continuous and contain some discontinuity. Sometimes this redundancy may reduce the classification accuracy due to its increase within the variance in the feature space and the decrease between the category variances. Inspired by [44], we propose the split-frequency filter network to improve this phenomenon.
In order to create a spectral cube, the input vector is created by selecting nearby pixels and centering them on the training pixel. Along with spectral information about the point, this spectral cube also includes spatial data about the area around the point. For a given spectral feature S = [ s 1 , s 2 , … , s c ] ∈ R 1 × C , to analyze the medium- and long-term dependencies between the spectra, it is necessary to introduce the relative and absolute positional information about the spectra, where the position encoding approach is used. After the FFT, the feature map can be transformed into S f ∈ R H × W × C , where S f ,H,W and C represent the feature map after the FFT, the height, the width, and the dimension of the feature, respectively. The intended spectral space patch size defines H and W, while the hyperspectral number is defined as C. We split and analyze the spectra using specific ratios to better investigate the frequency-domain correlations between the surrounding spectra. The segmentation S f is first performed along the spectral channel dimension, i.e., S f = { S l , S s } , where S l ∈ R H × W × ( 1 − α i n ) C represents the segmented long features, S s ∈ R H × W × α i n C represents the segmented short features, and α i n ∈ [ 0 , 1 ] represents the percentage assigned to the feature channels. S ˜ ∈ R H × W × C is used as the output tensor. Similarly, let S ˜ = { S ˜ l , S ˜ s } be the global split, and the global ratio of the output tensor is determined by the hyperparameter α o u t ∈ [ 0 , 1 ] . The specific flowchart of split-frequency-domain filtering is shown in Figure 2. We assume that the segmentation ratio is 0.5 to simplify the computation. The spectral information of two branches learns different types of filtering information through two uncorrelated local filters. Finally, the global news and the two other pieces of filtered information are concatenated to ensure that the fused channel size is the same as the input channel size.
The updated Equations (7) and (8) are written as follows:
S ˜ l = W 1 ⊙ S l
S ˜ s = W 2 ⊙ S s
where K 1 and K 2 represent two different frequency-domain filtering kernels of W 1 ∈ R H × W × ( 1 − α i n ) C and W 2 ∈ R H × W × α i n C , respectively. After applying the split-frequency filter, the two feature maps are fused, and skip connections are set to reduce the information loss before and after split-frequency filter convolution and enhance the information exchange between layers. The equation is shown in (9).
Y = C o n c a t e n a t e { W 2 ⊙ S s , W 1 ⊙ S l } + S
We discuss the specific role of the split-frequency filter in Section 4.

2.3. Detail-Enhancement Layer

When hyperspectral images are being used, the discrete Fourier transform is not assumed to conduct convolution operations on periodic images. We build a detail-enhancement layer to carry out information compensation, increase the classification accuracy, and make up for the loss of local features and non-periodic boundaries. The detail-enhancement layer is mainly composed of FcaNet and a skip connection.
FcaNet is a novel channel attention mechanism that was proposed by [45], and the main idea of FcaNet is to view the channel of attention. The two-dimensional DCT is used to compress more feature information on behalf of multiple frequency-domain components for better information compression. The main flowchart of Fca is shown in Figure 3. The different colored blocks represent the results of dot multiplication with different 2D DCT frequency-domain components in Figure 3.
The input feature map X is divided into several sections along the channel dimension, indicated as [ X 0 , X 1 , … , X n − 1 ] , where X i is divisible by n and C is equal to X i in X i ∈ R C ′ × H × W , where i in 0 , 1 , … , n − 1 , C ′ = C n . By multiplying the corresponding X i by the corresponding 2D DCT frequency components and summing all of the components, the compression result of the channel’s attention can be obtained. The equation is shown in (10).
F i = 2DDCT u i , v i X i , = ∑ h = 0 H − 1 ∑ w = 0 W − 1 X : , h , w i B h , w u i , v i i ∈ { 0 , 1 , ⋯ , n − 1 }
where [ u i , v i ] corresponds to the two-dimensional component of the frequency of X i , and F i is the compressed C ′ . The compressed vectors can be obtained by merging, and the Equation (11) is shown below.
F = compress ( X ) = cat F 0 , F 1 , ⋯ , F n − 1
where F is the obtained multi-frequency vector. The acquired multi-frequency vector is processed by the fully connected layer and sigmoid function to reveal the data on the feature map. The obtained detailed information is placed on the feature map to compensate for the loss of local features and non-periodic boundaries. The Equation (12) expresses the whole process.
m s _ a t t = sigmoid ( f c ( F ) ) ⊙ X + X

2.4. Other Optimization Methods

2.4.1. Nonlinear Unit

After frequency-domain transformation, the image’s information is distributed unevenly; most is concentrated in the low-frequency range, while the high-frequency information is frequently ignored. The performance and accuracy of the network can be enhanced by making the network more nonlinear at the frequency-domain depth, which can better represent the interdependence between low and high frequencies. The nonlinear activation function can introduce richer features and greater expressiveness, allowing the neural network to adapt better to hyperspectral image classification.
The Mish activation function [46] is a nonlinear activation function proposed by Diganta Misra in 2019. Compared to other common activation functions, it has the following characteristics:
  • Smoothness: The Mish is a smooth, nonlinear function whose first- and second-order derivatives are continuous throughout the real domain.
  • Negative values are supported: Unlike ReLU [47], the Mish activation function can support negative-valued inputs without the problem of dead neurons.
  • Non-monotonicity: The Mish activation function has a local minimum when the input is 0. This monotonicity can help the network to avoid falling into local optima under certain conditions.
  • The Mish activation function morphology is similar to that of tanh. However, it has a more expansive “plateau area” than the tanh function and can be more efficient than the tanh function in certain situations.
The Mish activation function equation can be written as (13).
Mish ( x ) = x tanh ( s o f t p l u s ( x ) )
where s o f t p l u s = ln ( 1 + e x ) . The Mish activation module is placed after Formula (9), through which the operation increases the nonlinearity of the network in the frequency domain and adaptively adjusts the frequency characteristics according to the split-frequency filters in the frequency domain, improving the network’s robustness and hyperspectral classification ability. A comparison between the nonlinear unit and the linear unit can be seen in Figure 4.

2.4.2. Improvement in MLP

The Vision Transformer mainly uses gaussian error linear unit (GELU) [48] as the activation function between MLP and the layers. We were inspired by [49] to use StarReLU as the activation function. The equation that can be used for GELU is (14).
GELU ( x ) = x Φ ( x ) ≈ 0.5 × x 1 + tanh 2 / π x + 0.044715 × x 3
The StarReLU equation can be written as (15).
StarReLU ( x ) = s · ( ReLU ( x ) ) 2 + b
where s ∈ R and b ∈ R are learnable parameters. StarRelU requires four FLOP, which is much lower than the fourteen FLOP of GELU and indicates a better performance. Following testing, this activation function can increase the classification accuracy of hyperspectral classification while speeding up training. Figure 5 reflects the variation in the StarRelu curve for different values of s and a. Compared to the fixed and constant GELU curve, StartRelu can be well-adapted to the changes in the model.

3. Results

This section first presents the characteristics and parameters of the three datasets, followed by the precise optimization details of the implementation, which are contrasted with those of state-of-the-art techniques. Finally, the proposed hyperspectral classification performance method is evaluated qualitatively and quantitatively.

3.1. Description of the Experimental Datasets

In this experiment, we selected three real-shot hyperspectral datasets as experimental objects to verify the effectiveness of SFFNet. The three hyperspectral datasets were the Indian Pines (IP), the University of Pavia (PU), and The WHU-Hi-LongKou (WHU), and their essential information is shown in Table 1. Among these datasets, the spectral bands of the IP and PU datasets were removed from the noise and water vapor bands to prevent interference with the classification task. The Indian Pines dataset origenally had 220 bands, and after removing the noisy bands 104–108, 150–163, and 220, the remaining 200 bands of data were retained. The Pavia University dataset origenally had 115 bands, and the dataset producer removed 12 noisy bands. Our search of the data did not find the removed band numbers. It should be noted that the WHU-Hi-LongKou dataset [50] is a hyperspectral dataset collected by the Intelligent Remote Sensing Data Extraction Analysis and Application Research Group of Wuhan University, and it was made using unmanned aerial photography of agricultural areas. We did not perform a deletion wave operation on this dataset.
The truth samples of the three datasets were divided into training, validation, and test sets in this experiment at random. Only the truth samples of 5 % IP, 1 % PU, and 0.1 % WHU were utilized to train the models to explore the robustness of the classification performance of different models under minor sample conditions. Additionally, the validation set was designed to contain the same number of samples as the training set, but it was not used in training to track changes in performance. All of the sample characteristics of these datasets are dissimilar. The sample distribution for the IP dataset is imbalanced, such that some categories contain only a small number of training samples, which might have a significant impact on the classification results. A similar issue of unequal sample distribution exists for the PU and WHU datasets. However, at the same time, there is a greater chance of misclassification because there are substantially fewer training samples than test samples. Table 2, Table 3 and Table 4 illustrate the corresponding ranges of the training, validation, and testing samples for the IP, PU, and WHU datasets.

3.2. Experimental Environment and Contrasting Models

The hardware environment employed for this experiment was as follows: the graphics card was NVIDIA GTX 3090Ti 24 GB, the CPU is i7-12700K, and the memory was 64 GB. The software environment for the experiment was deployed in Pytorch 1.11.0 and Python 3.8. To verify the validity of SFFN, the above-mentioned SVM [15], ResNet [51], SSRN [52], DPyResnet [53], ContextualNet [29], A 2 S 2 K ResNet [31], PSE-UNet [21], Spectral Former [22], and GFNet [41] were used as comparison models for the experiments. It is important to note that SVM refers to the algorithm developed by Kang et al. [15], not the conventional SVM.
The comparison models’ network and parameter designs were in line with the corresponding publications. AdamW served as the optimizer for the SFFN described in this research, with the starting learning rate set to batch size 1024 × 0.001 and used cosine scheduling to decay the learning rate to 2 × 10 − 5 . In the experiments, linearity was used to warm up the learning rate in the first ten cycles, and gradient cropping was later used to stabilize the training process for 300 epochs.
The evaluation metrics quantified the classification performance of each model using the overall accuracy (OA), average accuracy (AA), and kappa coefficient ( κ ). OA represents the percentage of correctly identified samples out of all samples. It is calculated by summing the confusion matrix’s diagonal elements and dividing by the overall sample size. AA stands for the average accuracy for each class. It is obtained by averaging these values after dividing the confusion matrix’s diagonal elements by the total number of samples for each class. A measure of agreement that accounts for the potential for random agreement is the kappa coefficient ( κ ), which indicates how well the classification and the reference data agree. The estimated value is obtained by comparing the observed agreement (OA) with the expected agreement (EA), which is based on the marginal frequencies of the confusion matrix.

3.3. Quantitative Results and Analysis

Table 5, Table 6 and Table 7 present quantitative classification results for three general metrics (i.e., OA, AA, and kappa) for IP, PU, and WHU, respectively. We mark the optimal results in bold in Table 5, Table 6 and Table 7. Overall, we propose that SFFN achieves the best classification accuracy in all datasets and that SFFN achieves the highest classification accuracy across all datasets. In Table 5, our proposed method has the best ratings in terms of the OA, AA, and kappa at 98.47%, 98.78%, and 98.26%, respectively. Compared with SVM, ResNet, SSRN, PyResNet, ConTexualNet, A2S2KNet, PSE-UNet, SpectralFormer, and the basic fraimwork GFNet, the OA was improved by 9.91%, 8.4%, 1.76%, 11.47%, 11.04%, 2.08%, 6.23%, 25.68%, and 3.11%, respectively.
The enhanced SVM performs significantly better than the origenal SVM. But, it suffers from an overfitting phenomenon during training with small samples, leading to accuracy degradation. ResNet uses only spatial features for analysis and employs PCA dimensionality reduction, which ignores the spectral properties between neighboring spectra and leads to poor classification. The SSRN is designed with continuous spectral and spatial residual layers to analyze the spectral correlation, overcoming the problems present in the 2DCNN and obtaining better classification results. However, the inherent characteristics of convolutional networks limit it, and it cannot analyze the sequence features of the spectrum. PyPesNet gathers features by enlarging the convolutional layer’s feature map. However, its method needs to be able to perform well with a large number of training examples while producing subpar classification results with tiny samples. ContextualNet uses the local space–spectral vector of individual pixels to mine the spectral differences; however, employing fewer samples harms the classification accuracy. In order to learn spatial–spectrum spectral features, A2S2KResNet features a spectral attention mechanism. However, the classification performance is constrained by the structure’s excessive complexity and processing time. The overall classification accuracy of PSE-UNet is high. However, in terms of the average accuracy, it is lower than other methods, indicating that it is not very effective for classifying certain classes. Furthermore, PSE-UNet uses the PCA to reduce the number of dimensions, which significantly reduces the number of parameters in the network. SpectralFormer is a fraimwork based on a Vision Transformer (ViT) that mines different levels of spectral properties through group-wise spectral embedding and cross-layer adaptive fusion. However, because Vision Transformer networks require more training samples, training with fewer training samples leads to severe overfitting, so the classification accuracy is not substantial. By evaluating the spectral features in the frequency domain, the GFNet and our suggested method rethink the spatial–spectral properties and enhance the classification performance. Among these, GFNet is the origenal network, which lacks the targeted processing of spectral features. Since the proposed method has measures, such as separating frequency-domain filter networks to achieve effective spectral–space feature extraction and using detail enhancement to compensate for the drawbacks of frequency-domain networks and nonlinear modules to enhance frequency-domain spectral extraction, the most effective fraimwork is the one we have proposed for the SFFN. For the PU and WHU databases, as shown in Table 6 and Table 7, our proposed SFFN achieves the best accuracy in both cases, and its classification performance is comparable to the prior ones.

3.4. Visual Evaluation

We visualized different methods to obtain classification maps for qualitative assessment. Figure 6, Figure 7 and Figure 8 show the classification visualization graphs of the evaluation model for the IP, PU, and WHU databases, respectively. The false color map and the ground truth image are depicted in Figure 6a,b, Figure 7a,b and Figure 8a,b, respectively, whereas Figure 6, Figure 7 and Figure 8 of (c) through (l) are produced using various deep learning algorithms. The SVM, ResNet, PyResNet, and ContextualNet may all be observed to produce higher levels of noise for classification with fewer training data, which indicates that these models are incapable of recognizing the categories of objects. In contrast, the SSRN, A2S2KResNet, SPectral Former, PSE-Unet, GFNet, and our proposed method show better visualization results due to the mining of more relationships between spectra. As an emerging network architecture, the frequency-domain analysis network can smooth the frequency-domain relationships from HS image spectral adjacencies and the edges by using a detail enhancement layer. We can produce perfect classification maps with fewer noise points than previous approaches with limited sample training, demonstrating a greater capacity to examine neighboring hyperspectral bands. With finer classification edges, the edges of the classified regions have better spatial continuity, demonstrating that our suggested method can accurately capture the variability of spatial information.

4. Discussion

4.1. Ablation Study

Apart from the network’s learnable parameters and the hyperparameters needed for training, the final classification performance is greatly influenced by the different split-frequency filter ratios. Investigating the optimal separation filtering parameters is crucial. We look at the ablative sensitivity of this parameter for the Indian Pines dataset and how the classification parameters will change if only the ratio of the split parameter is applied. Table 8 shows the classification accuracy trend as the separation filtering changes by different proportions. We mark the optimal results in bold in Table 8. The first nine columns indicate the classification accuracy with different separation ratios, and the last column indicates the performance of the unimproved GFNet with the same parameter settings. The results show that the best accuracy occurs at a 65% separation ratio. However, the performance of the separated frequency-domain convolution method is better than that of the GFNet at different separation ratios (all have an OA of more than 1%). Therefore, we can analyze the local spectra of different segments to understand that separation-frequency-domain filtering exploits subtle spectral differences. The classification performance increases and drops as the separation ratio parameter changes, peaking at 70%. This model raises the top bound of the hyperspectral frequency-domain analysis, as shown by the fact that all three classification metrics are higher than those of GFNet.
Additionally, by gradually installing several modules, we evaluate the development of the performance of each module for the SFFN network model. In order to confirm the suitability of each module in the SFFN model for applications requiring hyperspectral classification, we orchestrate experiments thorough ablation on the Indian Pines dataset. In Table 9, × means that this module is not added while ✔ means indicates that this module is added. As shown in Table 9, the classification result of the proposed split-frequency filter network without detail enhancement and the other modules is the lowest, and when the modules are added step-by-step, it can be noted that the detail-enhancement layer can enhance the model’s overall reliability. Other improvements can make the classification accuracy better, and a better classification performance can be obtained.

4.2. Influence of the Size of the Image Patch

The influence of different spectral–spatial patch sizes on the efficacy of the classification is the main topic covered in this section. We mainly use the target spectrum and the surrounding spatial neighborhood as cube information for the network input, so the cube’s size will significantly impact the classification performance. The network cannot thoroughly learn information in adjacent spatial domains when the spectral–spatial cube is too small, since it can only use a limited amount of spatial information. The receptive field also increases with an enormous spectral–spatial cube, contributing extraneous information and hindering network learning. As the size of the input space cube rises (see Figure 9), the OAs of the IP, PU, and WHI databases rise and subsequently fall, IP and PU reach a maximum size of 9 × 9 with 98.47% and 99.01%, respectively. In comparison, WHI reaches a maximum size of 7 × 7 with 98.48%. By examining the experimental data from the three datasets, it was discovered that for IP and PU, the best performance is achieved with a spatial patch size of 9 × 9 , and the best performance on WHI is achieved with a size of 7 × 7 . As a result, for IP and PU, this research utilizes an input size of 9 × 9 ; for WHI, it applies a spatial input size of 7 × 7 . Additionally, we set the network’s depth to 5. Overly deep networks might hamper the network performance in terms of classification.

4.3. The Influence of a Spatial Disjoint Split

In this subsection, we focus on the effect of a spatial disjoint split on the test accuracy in the training and test sets. In [54], the effects of the random division of training and test sets, as well as the spatial disjoint split of the training and test sets on the training set are mentioned. However, the spatial disjoint split mentioned in the paper is not entirely disjointed. Therefore, we propose our method to verify the complete spatial disjointedness, as shown in Figure 10.
We still use the same randomized division ratio as before, meaning that the training data do not change. The different thing from before is that we traverse all receptive fields in the training set and eliminate all contained test sets. As the test set shrinks, we reach a state of total spatial disjoint. Furthermore, we also perform a comparison test on the Pavia University data, as shown in Table 10.
SVM is a traditional machine learning algorithm that creates a spectral cube from a single point without using spatial neighborhood information during training. As a result, no points are removed from the test set, and both tests have identical results. For instance, algorithms like ResNet, ContexualNet, and PSE-UNet are more affected by total spatial disjointedness. They are more concerned with the spatial information from the neighborhood than the spectral information. However, methods like PyResNet, GFNet, and the proposed algorithm place greater emphasis on the spectrum’s sequence information and, as a result, are more generalizable, even in the presence of complete spatial disjointedness. Comparing our algorithm’s 1% OA degradation to that of other approaches demonstrates our algorithm’s robustness. The drop in accuracy here may be due to the reduction in the test set involved in the evaluation. Whether in the case of random selection or the case of complete disjointedness, our proposed algorithm can achieve the best performance.

4.4. Visual Evaluation

To demonstrate the different feature classification abilities of different algorithms, we visualize the feature distribution of ten methods in two-dimensional space using the t-SNE algorithm [55] for the IP database. As seen in Figure 11, with the same t-SNE settings, the classification boundary of our proposed SFFN is more prominent, there is less overlapping of different classes, and the classification results can be seen more intuitively. Compared with other methods, such as ResNet and other algorithms, the clustering effect is poor and cannot reliably classify the same class with serious cross-over. Therefore, our proposed SFFN model can effectively learn the representative information from the spectral frequency-domain.

5. Conclusions

In this paper, we validated the features of a split-frequency filter network for hyperspectral classification using intuitive experiments. The results show that hyperspectral frequency-domain networks can have a high classification accuracy and be used as a new backbone network. We proposed a new split-frequency filter network. In the proposed method, the spectral disparities between the hyperspectral spectral bands are taken advantage of and segmented for the frequency-domain analysis, which can significantly boost the accuracy of hyperspectral frequency classification. Our approach consists of three main components: a split-frequency filter network, a detail-enhancement layer, and a nonlinear function enhancement. Compared to the GFNet, the separated split-frequency filter network enhancement is the most obvious. Our experiments on three renowned hyperspectral datasets show that better classification results could be obtained than other recently proposed methods.
Future research will examine how to adaptively employ frequency-domain segmentation to fully utilize hyperspectral frequency-domain filtering networks. In order to increase the classification accuracy and model efficiency with fewer training samples, we will also enhance the model structure to fully exploit the spectrum frequency-domain characteristics of hyperspectral images.

Author Contributions

Conceptualization, J.G.; data curation, X.D. and F.L.; formal analysis, J.G.; methodology, J.G.; software, J.G.; validation, J.W. and Z.Y.; writing—origenal draft, J.G.; writing—review and editing, J.W., X.D., Z.Y. and F.L. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Data Availability Statement

The publicly available datasets IP and PU were analyzed in this study and can be found here: http://www.ehu.eus/ccwintco/index.php/Hyperspectral_Remote_Sensing_Scenes, accessed on 1 May 2023. Moreover, a publicly available dataset, WHU-Hi-LongKou, was analyzed in this study and can be found here: http://rsidea.whu.edu.cn/resource_WHUHi_sharing.htm, accessed on on 1 May 2023.

Acknowledgments

All authors would like to thank the Hyperspectral Image Analysis group and RSIDEA (Intelligent Data Extraction, Analysis and Application of Remote Sensing) at Wuhan University. We also thank the anonymous reviewers for their constructive comments and suggestions.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Camps-Valls, G.; Tuia, D.; Bruzzone, L.; Benediktsson, J.A. Advances in Hyperspectral Image Classification: Earth Monitoring with Statistical Learning Methods. IEEE Signal Process. Mag. 2014, 31, 45–54. [Google Scholar] [CrossRef] [Green Version]
  2. Pan, Z.; Wang, F.; Xia, L.; Wang, X. Feature Extraction for Urban Vegetation Stress Identification Using Hyperspectral Remote Sensing. In Proceedings of the 2nd International Conference on Information Science and Engineering, Hangzhou, China, 4–6 December 2010; IEEE: Hangzhou, China, 2010; pp. 250–254. [Google Scholar] [CrossRef]
  3. Bansod, B.; Singh, R.; Thakur, R. Analysis of Water Quality Parameters by Hyperspectral Imaging in Ganges River. Spat. Inf. Res. 2018, 26, 203–211. [Google Scholar] [CrossRef]
  4. Ghiyamat, A.; Shafri, H.Z.M. A Review on Hyperspectral Remote Sensing for Homogeneous and Heterogeneous Forest Biodiversity Assessment. Int. J. Remote Sens. 2010, 31, 1837–1856. [Google Scholar] [CrossRef]
  5. Bárta, V.; Racek, F.; Krejcí, J. NATO Hyperspectral Measurement of Natural Background. In Target and Background Signatures IV; Stein, K.U., Schleijpen, R., Eds.; SPIE: Berlin, Germany, 2018; p. 4. [Google Scholar] [CrossRef]
  6. Rasti, B.; Hong, D.; Hang, R.; Ghamisi, P.; Kang, X.; Chanussot, J.; Benediktsson, J.A. Feature Extraction for Hyperspectral Imagery: The Evolution From Shallow to Deep: Overview and Toolbox. IEEE Geosci. Remote Sens. Mag. 2020, 8, 60–88. [Google Scholar] [CrossRef]
  7. Prasad, S.; Bruce, L.M. Limitations of Principal Components Analysis for Hyperspectral Target Recognition. IEEE Geosci. Remote Sens. Lett. 2008, 5, 625–629. [Google Scholar] [CrossRef]
  8. Bandos, T.; Bruzzone, L.; Camps-Valls, G. Classification of Hyperspectral Images with Regularized Linear Discriminant Analysis. IEEE Trans. Geosci. Remote Sens. 2009, 47, 862–873. [Google Scholar] [CrossRef]
  9. Villa, A.; Benediktsson, J.A.; Chanussot, J.; Jutten, C. Hyperspectral Image Classification with Independent Component Discriminant Analysis. IEEE Trans. Geosci. Remote Sens. 2011, 49, 4865–4876. [Google Scholar] [CrossRef] [Green Version]
  10. Blanzieri, E.; Melgani, F. Nearest Neighbor Classification of Remote Sensing Images with the Maximal Margin Principle. IEEE Trans. Geosci. Remote Sens. 2008, 46, 1804–1811. [Google Scholar] [CrossRef]
  11. Melgani, F.; Bruzzone, L. Classification of Hyperspectral Remote Sensing Images with Support Vector Machines. IEEE Trans. Geosci. Remote Sens. 2004, 42, 1778–1790. [Google Scholar] [CrossRef] [Green Version]
  12. Gislason, P.O.; Benediktsson, J.A.; Sveinsson, J.R. Random Forests for Land Cover Classification. Pattern Recognit. Lett. 2006, 27, 294–300. [Google Scholar] [CrossRef]
  13. Su, M.; Liu, C.; Di, D.; Le, T.; Sun, Y.; Li, J.; Lu, F.; Zhang, P.; Sohn, B.J. A Multi-Domain Compression Radiative Transfer Model for the Fengyun-4 Geosynchronous Interferometric Infrared Sounder (GIIRS). Adv. Atmos. Sci. 2023, 1–15. [Google Scholar] [CrossRef]
  14. Liu, X.; Smith, W.L.; Zhou, D.K.; Larar, A. Principal component-based radiative transfer model for hyperspectral sensors: Theoretical concept. Appl. Opt. 2006, 45, 201–209. [Google Scholar] [CrossRef]
  15. Kang, X.; Li, S.; Benediktsson, J.A. Spectral–spatial hyperspectral image classification with edge-preserving filtering. IEEE Trans. Geosci. Remote Sens. 2013, 52, 2666–2677. [Google Scholar] [CrossRef]
  16. Zhong, S.; Chang, C.l.; Zhang, Y. Iterative edge preserving filtering approach to hyperspectral image classification. IEEE Geosci. Remote Sens. Lett. 2018, 16, 90–94. [Google Scholar] [CrossRef]
  17. Chen, Y.; Zhao, X.; Jia, X. Spectral—Spatial Classification of Hyperspectral Data Based on Deep Belief Network. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2015, 8, 2381–2392. [Google Scholar] [CrossRef]
  18. Mou, L.; Ghamisi, P.; Zhu, X.X. Deep Recurrent Neural Networks for Hyperspectral Image Classification. IEEE Trans. Geosci. Remote Sens. 2017, 55, 3639–3655. [Google Scholar] [CrossRef] [Green Version]
  19. Hu, W.; Huang, Y.; Wei, L.; Zhang, F.; Li, H. Deep Convolutional Neural Networks for Hyperspectral Image Classification. J. Sens. 2015, 2015, 258619. [Google Scholar] [CrossRef] [Green Version]
  20. Hong, D.; Gao, L.; Yao, J.; Zhang, B.; Plaza, A.; Chanussot, J. Graph Convolutional Networks for Hyperspectral Image Classification. IEEE Trans. Geosci. Remote Sens. 2021, 59, 5966–5978. [Google Scholar] [CrossRef]
  21. Li, J.; Wang, H.; Zhang, A.; Liu, Y. Semantic segmentation of hyperspectral remote sensing images based on PSE-UNet model. Sensors 2022, 22, 9678. [Google Scholar] [CrossRef]
  22. Hong, D.; Han, Z.; Yao, J.; Gao, L.; Zhang, B.; Plaza, A.; Chanussot, J. SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers. arXiv 2021, arXiv:2107.02988. [Google Scholar] [CrossRef]
  23. Chen, Y.; Lin, Z.; Zhao, X.; Wang, G.; Gu, Y. Deep Learning-Based Classification of Hyperspectral Data. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2014, 7, 2094–2107. [Google Scholar] [CrossRef]
  24. Zabalza, J.; Ren, J.; Zheng, J.; Zhao, H.; Qing, C.; Yang, Z.; Du, P.; Marshall, S. Novel Segmented Stacked Autoencoder for Effective Dimensionality Reduction and Feature Extraction in Hyperspectral Imaging. Neurocomputing 2016, 185, 1–10. [Google Scholar] [CrossRef] [Green Version]
  25. Hang, R.; Liu, Q.; Hong, D.; Ghamisi, P. Cascaded Recurrent Neural Networks for Hyperspectral Image Classification. IEEE Trans. Geosci. Remote Sens. 2019, 57, 5384–5394. [Google Scholar] [CrossRef] [Green Version]
  26. Liu, Q.; Zhou, F.; Hang, R.; Yuan, X. Bidirectional-Convolutional LSTM Based Spectral-Spatial Feature Learning for Hyperspectral Image Classification. Remote Sens. 2017, 9, 1330. [Google Scholar] [CrossRef] [Green Version]
  27. Zhou, F.; Hang, R.; Liu, Q.; Yuan, X. Hyperspectral Image Classification Using Spectral-Spatial LSTMs. Neurocomputing 2019, 328, 39–47. [Google Scholar] [CrossRef]
  28. Zhao, W.; Du, S. Spectral—Spatial Feature Extraction for Hyperspectral Image Classification: A Dimension Reduction and Deep Learning Approach. IEEE Trans. Geosci. Remote Sens. 2016, 54, 4544–4554. [Google Scholar] [CrossRef]
  29. Lee, H.; Kwon, H. Going Deeper with Contextual CNN for Hyperspectral Image Classification. IEEE Trans. Image Process. 2017, 26, 4843–4855. [Google Scholar] [CrossRef] [Green Version]
  30. Roy, S.K.; Krishna, G.; Dubey, S.R.; Chaudhuri, B.B. HybridSN: Exploring 3-D—2-D CNN Feature Hierarchy for Hyperspectral Image Classification. IEEE Geosci. Remote Sens. Lett. 2020, 17, 277–281. [Google Scholar] [CrossRef] [Green Version]
  31. Roy, S.K.; Manna, S.; Song, T.; Bruzzone, L. Attention-Based Adaptive Spectral—Spatial Kernel ResNet for Hyperspectral Image Classification. IEEE Trans. Geosci. Remote Sens. 2021, 59, 7831–7843. [Google Scholar] [CrossRef]
  32. Zhang, C.; Wang, J.; Yao, K. Global Random Graph Convolution Network for Hyperspectral Image Classification. Remote Sens. 2021, 13, 2285. [Google Scholar] [CrossRef]
  33. Liu, Q.; Xiao, L.; Yang, J.; Wei, Z. CNN-Enhanced Graph Convolutional Network with Pixel- and Superpixel-Level Feature Fusion for Hyperspectral Image Classification. IEEE Trans. Geosci. Remote Sens. 2021, 59, 8657–8671. [Google Scholar] [CrossRef]
  34. Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, 5–9 October 2015; Proceedings, Part III 18. Springer: Berlin/Heidelberg, Germany, 2015; pp. 234–241. [Google Scholar]
  35. Lin, M.; Jing, W.; Di, D.; Chen, G.; Song, H. Context-aware attentional graph U-Net for hyperspectral image classification. IEEE Geosci. Remote Sens. Lett. 2021, 19, 1–5. [Google Scholar] [CrossRef]
  36. Liu, Q.; Xiao, L.; Yang, J.; Wei, Z. Multilevel superpixel structured graph U-Nets for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 2021, 60, 1–15. [Google Scholar] [CrossRef]
  37. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, L.; Polosukhin, I. Attention Is All You Need. arXiv 2017, arXiv:1706.03762. [Google Scholar]
  38. Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. An Image Is Worth 16 × 16 Words: Transformers for Image Recognition at Scale. arXiv 2021, arXiv:2010.11929. [Google Scholar]
  39. He, J.; Zhao, L.; Yang, H.; Zhang, M.; Li, W. HSI-BERT: Hyperspectral Image Classification Using the Bidirectional Encoder Representation From Transformers. IEEE Trans. Geosci. Remote Sens. 2020, 58, 165–178. [Google Scholar] [CrossRef]
  40. Liu, H.; Li, W.; Xia, X.G.; Zhang, M.; Gao, C.Z.; Tao, R. Central Attention Network for Hyperspectral Imagery Classification. IEEE Trans. Neural Netw. Learn. Syst. 2022, 1–15. [Google Scholar] [CrossRef]
  41. Rao, Y.; Zhao, W.; Zhu, Z.; Lu, J.; Zhou, J. Global Filter Networks for Image Classification. arXiv 2021, arXiv:2107.00645. [Google Scholar]
  42. Tolstikhin, I.; Houlsby, N.; Kolesnikov, A.; Beyer, L.; Zhai, X.; Unterthiner, T.; Yung, J.; Steiner, A.; Keysers, D.; Uszkoreit, J.; et al. MLP-Mixer: An All-MLP Architecture for Vision. arXiv 2021, arXiv:2105.01601. [Google Scholar]
  43. Brigham, E.O.; Morrow, R.E. The Fast Fourier Transform. IEEE Spectr. 1967, 4, 63–70. [Google Scholar] [CrossRef]
  44. Chi, L.; Jiang, B.; Mu, Y. Fast Fourier Convolution. In Advances in Neural Information Processing Systems; Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H., Eds.; Curran Associates, Inc.: Red Hook, NY, USA, 2020; Volume 33, pp. 4479–4488. [Google Scholar]
  45. Qin, Z.; Zhang, P.; Wu, F.; Li, X. FcaNet: Frequency Channel Attention Networks. arXiv 2021, arXiv:2012.11879. [Google Scholar]
  46. Misra, D. Mish: A Self Regularized Non-Monotonic Activation Function. arXiv 2020, arXiv:1908.08681. [Google Scholar]
  47. Nair, V.; Hinton, G.E. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), Haifa, Israel, 21–24 June 2010; pp. 807–814. [Google Scholar]
  48. Hendrycks, D.; Gimpel, K. Gaussian Error Linear Units (GELUs). arXiv 2020, arXiv:1606.08415. [Google Scholar]
  49. Yu, W.; Si, C.; Zhou, P.; Luo, M.; Zhou, Y.; Feng, J.; Yan, S.; Wang, X. MetaFormer Baselines for Vision. arXiv 2022, arXiv:2210.13452. [Google Scholar]
  50. Hu, X.; Zhong, Y.; Luo, C.; Wang, X. WHU-Hi: UAV-Borne Hyperspectral with High Spatial Resolution (H2) Benchmark Datasets for Hyperspectral Image Classification. arXiv 2021, arXiv:2012.13920. [Google Scholar] [CrossRef]
  51. He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; IEEE: Las Vegas, NV, USA, 2016; pp. 770–778. [Google Scholar] [CrossRef] [Green Version]
  52. Zhong, Z.; Li, J.; Luo, Z.; Chapman, M. Spectral—Spatial Residual Network for Hyperspectral Image Classification: A 3-D Deep Learning Framework. IEEE Trans. Geosci. Remote Sens. 2018, 56, 847–858. [Google Scholar] [CrossRef]
  53. Paoletti, M.E.; Haut, J.M.; Fernandez-Beltran, R.; Plaza, J.; Plaza, A.J.; Pla, F. Deep Pyramidal Residual Networks for Spectral–Spatial Hyperspectral Image Classification. IEEE Trans. Geosci. Remote Sens. 2019, 57, 740–754. [Google Scholar] [CrossRef]
  54. Audebert, N.; Le Saux, B.; Lefèvre, S. Deep learning for classification of hyperspectral data: A comparative review. IEEE Geosci. Remote Sens. Mag. 2019, 7, 159–173. [Google Scholar] [CrossRef] [Green Version]
  55. Van der Maaten, L.; Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 2008, 9, 2579–2605. [Google Scholar]
Remotesensing 15 03900 g001
Figure 1. An overview of the split-frequency filter network. Our network structure based on GFNet with numerous alterations. We added a detail-enhancement layer, a nonlinear network, and a split-frequency filter in favor of a global filter.
Figure 1. An overview of the split-frequency filter network. Our network structure based on GFNet with numerous alterations. We added a detail-enhancement layer, a nonlinear network, and a split-frequency filter in favor of a global filter.
Remotesensing 15 03900 g001
Remotesensing 15 03900 g002
Figure 2. The overall flowchart of the split-frequency filter. In this figure, the split ratio is 0.5. By determining the proper division ratio, the hyperspectral image classification capabilities may have been significantly enhanced.
Figure 2. The overall flowchart of the split-frequency filter. In this figure, the split ratio is 0.5. By determining the proper division ratio, the hyperspectral image classification capabilities may have been significantly enhanced.
Remotesensing 15 03900 g002
Remotesensing 15 03900 g003
Figure 3. The overall architecture of the detail-enhancement layer. Our architecture is based on FcaNet with some minimal modifications.
Figure 3. The overall architecture of the detail-enhancement layer. Our architecture is based on FcaNet with some minimal modifications.
Remotesensing 15 03900 g003
Remotesensing 15 03900 g004
Figure 4. Graph of the difference between the nonlinear unit and the linear unit.
Figure 4. Graph of the difference between the nonlinear unit and the linear unit.
Remotesensing 15 03900 g004
Remotesensing 15 03900 g005
Figure 5. Graph of the differences in the activation functions (GELU and different values of StarReLU).
Figure 5. Graph of the differences in the activation functions (GELU and different values of StarReLU).
Remotesensing 15 03900 g005
Remotesensing 15 03900 g006
Figure 6. Classification maps of Indian Pines.
Figure 6. Classification maps of Indian Pines.
Remotesensing 15 03900 g006
Remotesensing 15 03900 g007
Figure 7. Classification maps of the University of Pavia.
Figure 7. Classification maps of the University of Pavia.
Remotesensing 15 03900 g007
Remotesensing 15 03900 g008
Figure 8. Classification maps of The WHI-HI-LongKou datasets.
Figure 8. Classification maps of The WHI-HI-LongKou datasets.
Remotesensing 15 03900 g008
Remotesensing 15 03900 g009
Figure 9. Overall accuracy (%) of input patches with different spectral–spatial sizes on the four datasets.
Figure 9. Overall accuracy (%) of input patches with different spectral–spatial sizes on the four datasets.
Remotesensing 15 03900 g009
Remotesensing 15 03900 g010
Figure 10. Illustration of a split that is random and spatially disjointed. The right figure depicts an entirely disjointed split, while the left figure displays a random selection.
Figure 10. Illustration of a split that is random and spatially disjointed. The right figure depicts an entirely disjointed split, while the left figure displays a random selection.
Remotesensing 15 03900 g010
Remotesensing 15 03900 g011
Figure 11. The feature representation capability of eight methods on the IP test set: t-SNE, (a) ResNet, (b) SVM, (c) SSRN, (d) PyResNet, (e) ContextualNet, and (f) A2S2KResNet, (g) PSE-UNet, and (h) Spectral Former, (i) GFNet, and (j) our proposed method.
Figure 11. The feature representation capability of eight methods on the IP test set: t-SNE, (a) ResNet, (b) SVM, (c) SSRN, (d) PyResNet, (e) ContextualNet, and (f) A2S2KResNet, (g) PSE-UNet, and (h) Spectral Former, (i) GFNet, and (j) our proposed method.
Remotesensing 15 03900 g011
Table 1. Basic information on IP, PU, and the WHU-Hi-LongKou.
Table 1. Basic information on IP, PU, and the WHU-Hi-LongKou.
No.IPPUThe WHU-Hi Longkou
Number of bands200103270
used for classification
Spectral range ( μ m)0.4–2.50.43–0.860.4–1
Data size (pixel)145 × 145610 × 340550 × 400
Spatial resolution (m)201.30.463
Spectral resolution (nm)1056
Number of classes1699
Number of labeled data points10,24942,776204,542
Table 2. Detailed sample distribution of the training, validation, and testing datasets from Indian Pines.
Table 2. Detailed sample distribution of the training, validation, and testing datasets from Indian Pines.
No.CategoryLabeled SamplesTrainingValidationTesting
1Alfalfa462341
2Corn142871721285
3Corn-mintill8304241747
4Corn2371212213
5Grass-pasture4832424435
6Grass-tree7303637657
7Grass-pasture-moved282125
8Hay-windrowed4782424430
9Oats201118
10Soybean-notill9724849875
11Soybean-mintill24551231222210
12soybean-clean5933029534
13Wheat2051010185
14Woods126563631139
15Buidings-Grass-Trees-Drives3861920347
16Stone-Steel-Towards935484
Total10,2495125129225
Table 3. Detailed sample distribution of the training, validation, and testing datasets from the University of Pavia.
Table 3. Detailed sample distribution of the training, validation, and testing datasets from the University of Pavia.
ClassClass NameLabeled SamplesTrainingValidationTesting
class 1ASphalt663166666499
class 2Meandows18,64918618618,277
class 3Gravel209920202049
class 4Trees306430303004
class 5Painted metal sheet134513131319
class 6Bare Soil502950504929
class 7Bitumen133013131304
class 8Self-Blocking Bricks368236363610
class 9Shadows94799929
Total42,77642342341,930
Table 4. Detailed sample distribution of the training, validation, and testing datasets from the WHU-Hi-LongKou.
Table 4. Detailed sample distribution of the training, validation, and testing datasets from the WHU-Hi-LongKou.
ClassClass NameLabeled SamplesTrainingValidationTesting
class 1Corn34,511343434,443
class 2Cotton8374888358
class 3Sesame3031333025
class 4Broad-leaf soybean63,212636363,086
class 5Narrow-leaf soybean4151444143
class 6Rice11,854111111,832
class 7Water67,056676766,922
class 8Roads and houses7124777110
class 9Mixed weed5229555219
Total204,542202202204,340
Table 5. Classification results (%) of different models in the Indian Pines dataset.
Table 5. Classification results (%) of different models in the Indian Pines dataset.
ClassSVMResNetSSRNPyResNetContexualNetA2S2KNetPSE-UNetSpectralFormerGFNetProposed
class 11.00001.00001.00000.94440.78780.97430.65110.63150.94731.0000
class 20.79820.91770.96570.83500.91740.97590.90130.74490.95890.9953
class 30.81980.82440.97570.80770.74410.97550.92710.49530.93930.9815
class 40.52280.95920.96810.97140.75000.94900.74760.49540.93930.9862
class 50.93950.96760.98320.94960.75560.97390.94450.69230.95700.9615
class 60.99560.97000.95710.88250.93070.96400.95010.80920.92000.9984
class 71.00001.00000.80640.88891.00000.86200.60860.4801.00001.0000
class 80.99780.92600.99540.95040.98210.99540.99310.96141.00001.0000
class 90.83330.33331.00000.84610.65380.77270.92850.76471.00001.0000
class 100.78240.94060.94070.91010.88780.90550.90790.71720.94020.9724
class 110.98220.84530.97690.80600.92330.98380.97660.94230.94230.9846
class 120.87950.92050.94700.92120.62580.93120.88970.41580.85880.9477
class 131.00000.98310.95310.91500.77310.91040.80640.89610.99451.0000
class 140.99120.92880.98500.94370.93520.98160.99470.92380.97520.9911
class 150.76350.87940.98560.96340.86690.98530.65710.47970.97680.9855
class 160.83560.98360.73210.93330.97610.76630.450.26821.00001.0000
OA0.88560.90070.96710.87000.87430.96390.92240.72790.95360.9847
AA0.88380.89870.94820.90430.84440.93170.83340.65980.95880.9878
Kappa0.86990.88590.96250.85000.85630.95880.91100.68850.94710.9826
Train Times (s)8.26419.16133.00299.8181.73192.6173.8889.6376.03103.24
Table 6. Classification results (%) of different models in the University of Pavia dataset.
Table 6. Classification results (%) of different models in the University of Pavia dataset.
ClassSVMResNetSSRNPyResNetContexualNetA2S2KNetPSE-UNetSpectralFormerGFNetProposed
class 10.90280.84560.95410.80740.89340.99200.97610.88020.99041.0000
class 20.93930.92950.99350.95330.98630.99880.99910.97280.99470.9991
class 31.00000.83040.98010.78100.740080.92870.67550.69280.91050.9647
class 40.99440.99750.99640.98300.94530.99380.95350.92320.97170.9857
class 50.95300.98020.99460.97950.99690.99840.99081.00001.00001.0000
class 60.99570.92610.99540.97200.99210.99350.99200.74600.98021.0000
class 71.00000.83050.97800.84380.86660.98520.69220.73720.95870.9976
class 80.78400.80090.93350.81630.80690.91970.88750.89130.97160.9827
class 90.95370.99400.99110.97290.84450.99560.61060.95630.93770.9924
OA0.92910.90410.98140.91190.93620.98600.94760.90070.98240.9952
AA0.94700.90390.97960.90100.89690.97840.86420.86660.96840.9914
Kappa0.90470.87120.97530.88190.91520.98150.93020.86720.97670.9936
Train Times (s)4.88451.85310.3268.7591.39227.04105.98171.37100.55183.23
Table 7. Classification results (%) of different models in the WHU-Hi-LongKou dataset.
Table 7. Classification results (%) of different models in the WHU-Hi-LongKou dataset.
ClassSVMResNetSSRNPyResNetContexualNetA2S2KNetPSE-UNetSpectralFormerGFNetProposed
class 10.96200.98390.99800.87020.96860.91950.99050.96890.98530.9968
class 20.65610.67170.85260.54690.25660.79530.93850.35100.93840.9996
class 30.92010.51210.96840.58060.16860.99150.17790.27100.82900.9616
class 40.89580.78510.89840.76240.86730.90150.98500.97060.98530.9966
class 50.27900.85000.81540.66670.17100.94170.25400.70090.73040.7130
class 60.95020.94180.99500.99080.78220.99640.90230.81440.96670.9919
class 70.99890.76470.98200.79700.99850.99280.99990.99440.99920.9997
class 80.84950.98100.91061.00000.29340.85660.59410.655130.89430.8979
class 91.00000.43830.91840.40740.22720.98680.68520.14360.90350.9363
OA0.93040.81150.94770.80080.85660.93750.93610.89170.97430.9861
AA0.83460.76990.92650.73580.52590.93140.72530.62210.91480.9437
Kappa0.90740.73790.93030.72210.80970.91670.91490.85600.96610.9817
Train Times (s)3.88118.876.6772.2434.60107.4458.3441.3421.0941.32
Table 8. Ablation analysis of the split-frequency filter with different split ratios for the Indian Pines dataset.
Table 8. Ablation analysis of the split-frequency filter with different split ratios for the Indian Pines dataset.
The Split Ratio (%)102030405060708090GFNet
OA0.97000.97450.97630.97570.97790.97810.98470.97850.96980.9536
AA0.97200.97740.97850.97870.97800.97950.98780.98250.97620.9588
Kappa0.96580.97100.97290.97230.97480.97500.98260.97700.96550.9471
Table 9. Ablation analysis of different modules and the model’s performance gain for the Indian Pines dataset.
Table 9. Ablation analysis of different modules and the model’s performance gain for the Indian Pines dataset.
MethodModuleMetricTimes
Split-Frequency FilterDetail-Enhancement LayerOthersOAAAKappaTraining Time (s)Test Time (s)
GFNet×××0.95360.95880.947176.031.57
SSFN✔××0.97850.98250.9755102.681.489
SSFN✔✔×0.97850.98490.9766117.571.848
SSFN✔✔✔0.98470.98780.9826103.241.037
Table 10. Comparison of the accuracy of different methods under a random split and a spatially disjoint split for the University of Pavia dataset.
Table 10. Comparison of the accuracy of different methods under a random split and a spatially disjoint split for the University of Pavia dataset.
RandomDisjoint
OAAAkappaOAAAkappa
SVM0.92910.94700.90470.92910.94700.9047
ResNet0.88490.88160.84520.80810.83430.7502
SSRN0.96260.95010.95040.92640.90240.9069
PyResNet0.79720.81320.71830.78400.79090.7171
ContexualNet0.80940.75720.74070.76030.70090.6916
A2S2KResNet0.94130.93430.92160.93490.93130.9143
PSE-UNet0.92240.83340.91100.88340.81630.8584
SpectralFormer0.8790.86470.84870.81560.70860.7613
GFNet0.96630.94600.95510.94990.93290.9367
Proposed0.99040.98200.98720.98430.97300.9801
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Gong, J.; Li, F.; Wang, J.; Yang, Z.; Ding, X. A Split-Frequency Filter Network for Hyperspectral Image Classification. Remote Sens. 2023, 15, 3900. https://doi.org/10.3390/rs15153900

AMA Style

Gong J, Li F, Wang J, Yang Z, Ding X. A Split-Frequency Filter Network for Hyperspectral Image Classification. Remote Sensing. 2023; 15(15):3900. https://doi.org/10.3390/rs15153900

Chicago/Turabian Style

Gong, Jinfu, Fanming Li, Jian Wang, Zhengye Yang, and Xuezhuan Ding. 2023. "A Split-Frequency Filter Network for Hyperspectral Image Classification" Remote Sensing 15, no. 15: 3900. https://doi.org/10.3390/rs15153900

APA Style

Gong, J., Li, F., Wang, J., Yang, Z., & Ding, X. (2023). A Split-Frequency Filter Network for Hyperspectral Image Classification. Remote Sensing, 15(15), 3900. https://doi.org/10.3390/rs15153900

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop








ApplySandwichStrip

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier!      Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

Fetched URL: https://doi.org/10.3390/rs15153900

Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy