文献类型：期刊文献

英文题名：A deep learning multimodal fusion framework for wood species identification using near-infrared spectroscopy GADF and RGB image

作者：Pan, Xi[1,2,3] Yu, Zhiming[4] Yang, Zhong[1,2]

第一作者：Pan, Xi

通信作者：Yang, Z[1];Yang, Z[2]

机构：[1]Chinese Acad Forestry, Res Inst Wood Ind, Beijing 100091, Peoples R China;[2]Natl Forestry & Grassland Adm, Key Lab Wood Sci & Technol, Beijing 100091, Peoples R China;[3]Beijing Forestry Univ, Coll Mat Sci & Technol, Beijing 100083, Peoples R China;[4]Beijing Forestry Univ, Coll Mat Sci & Technol, Beijing 100083, Peoples R China

年份：2023

外文期刊名：HOLZFORSCHUNG

收录：;EI(收录号：20235015192701);Scopus(收录号：2-s2.0-85179000405);WOS:【SCI-EXPANDED(收录号:WOS:001101427400001)】；

基金：The authors gratefully acknowledge the xylarium of Southwest Forestry University for supporting the wood specimens used in this study. Additionally, the authors thank Beijing Great Technology Co., Ltd. for supporting the handheld NIR spectrometer.

语种：英文

外文关键词：convolutional neural networks (CNN); data fusion; near-infrared (NIR) spectroscopy; RGB image; wood species identification

摘要：Accurate and rapid wood species identification is vital for wood utilization and trade. This goal is achievable with the fast development of deep learning (DL). Several studies have been published related to this topic; however, they were limited by their generalization performance in practical applications. Therefore, this study proposed a DL multimodal fusion framework to bridge this gap. The study utilized a state-of-the-art convolutional neural network (CNN) to simultaneously extract both short-wavelength near-infrared (NIR) spectra and RGB image feature, fully leveraging the advantages of both data types. Using portable devices for collecting spectra and image data enhances the feasibility of onsite rapid identification. In particular, a two-branch CNN framework was developed to extract spectra and image features. For NIR spectra feature extraction, 1 dimensional NIR (1D NIR) spectra were innovatively encoded as 2 dimensional (2D) images using the Gramian angular difference field (GADF) method. This representation enhances better data alignment with CNN operations, facilitating more robust discriminative feature extraction. Moreover, wood's spectral and image features were fused at the full connection layer for species identification. In the experimental phase conducted on 16 difficult-to-distinguish wood samples from the Lauraceae family, all achieved identification metrics results exceed 99 %. The findings illustrate that the proposed multimodal fusion framework effectively extracts and fully integrates the wood's features, thereby, improving wood species identification.