Skip to content

rafiibnsultan/GeoSAM

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

27 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

GeoSAM: Fine-tuning SAM with Sparse and Dense Visual Prompting for Automated Segmentation of Mobility Infrastructure

(Hello, this is currently the old approach we tried. The "updated" approach that can take "text" as prompts in addition to sparse or click prompts can be found in the GeoSAM_with_text branch)

This repository is dedicated to the work of GeoSAM. Please find the paper here: Link

Also, please find the link for the weights.

See the demo here.

This work has been submitted. Waiting for the decision.

Abstract:

The Segment Anything Model (SAM) has shown impressive performance when applied to natural image segmentation. However, it struggles with geographical images like aerial and satellite imagery, especially when segmenting mobility infrastructure including roads, sidewalks, and crosswalks. This inferior performance stems from the narrow features of these objects, their textures blending into the surroundings, and interference from objects like trees, buildings, vehicles, and pedestrians - all of which can disorient the model to produce inaccurate segmentation maps. To address these challenges, we propose Geographical SAM (GeoSAM), a novel SAM-based framework that implements a fine-tuning strategy using the dense visual prompt from zero-shot learning, and the sparse visual prompt from a pre-trained CNN segmentation model. The proposed GeoSAM outperforms existing approaches for geographical image segmentation, specifically by 26%, 7%, and 17% for road infrastructure, pedestrian infrastructure, and on average, respectively, representing a momentous leap in leveraging foundation models to segment mobility infrastructure including both road and pedestrian infrastructure in geographical images.

GeoSAM

## Acknowledgement We want to thank these two works for their open-source code and contributions to the respective fields!

Segment Anything Model (SAM)

MAPPING THE WALK: A SCALABLE COMPUTER VISION APPROACH FOR GENERATING SIDEWALK NETWORK DATASETS FROM AERIAL IMAGERY.

Citations

If these codes are helpful for your study, please cite:

@article{sultan2023geosam,
  title={GeoSAM: Fine-tuning SAM with sparse and dense visual prompting for automated segmentation of mobility infrastructure},
  author={Sultan, Rafi Ibn and Li, Chengyin and Zhu, Hui and Khanduri, Prashant and Brocanelli, Marco and Zhu, Dongxiao},
  journal={arXiv preprint arXiv:2311.11319},
  year={2023}
}

About

Fine-tuning SAM with Multi-Modal Prompts for Mobility Infrastructure Segmentation

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published