Cross-modality learning privileged information for multimodal data

Joint multimodal data in remote sensing is highly valuable because different sensors capture complementary information, allowing them to compensate for each other’s limitations. However, in practice, multimodal data collection is resource-intensive and multimodal data alignment with different resolutions is complex and challenging. These challenges make it difficult to achieve real-time observation, especially in urgent scenarios, such as responding to oil spills in marine environments. To address these issues, this paper introduces a novel cross-modality learning framework for oil spill detection using SAR and hyperspectral imagery. The proposed approach combines contrastive learning (CL) to extract and align features across multiple modalities with supervised learning to incorporate privileged information and refine predictions. The framework exploits complementary insights from diverse data sources by integrating these learning strategies, significantly improving unimodal prediction accuracy. Experimental results demonstrate that the proposed method outperforms state-of-the-art techniques in both hyperspectral and SAR image detection tasks.

N/A

International Geoscience and Remote Sensing Symposium IGARSS

N/A

2153-7003

N/A

N/A

4