A Large-Scale Dataset and Synthetic Data Generation Pipeline for Blind Navigation

NeurIPS 2026 ED Track

† Equal contribution

Explore NavAble's accessibility-focused dataset, synthetic data generation pipeline, and evaluation in our NeurIPS 2026 project video.

Abstract

Reliable recognition of accessibility-critical objects, such as accessible pedestrian signals, door-activation buttons, and handrails, is essential for technologies that support independent mobility for blind and low-vision (BLV) people. Yet these objects are often missing or underrepresented in existing vision datasets, and available data offer limited diversity, inconsistent annotations, and viewpoints that do not match assistive devices.

We introduce NavAble, a large-scale dataset and synthetic data generation pipeline for BLV navigation. NavAble combines 36,466 curated images with 5,681 collected image records, including 100 supplementary revolving-door frames, across 11 accessibility-critical categories. Its generator transforms web images into simulation-ready 3D assets and renders them in NVIDIA Isaac Sim. The current synthetic release contains 450,860 frames across 37 environments, with semantic segmentation, depth, surface normals, and 2D/3D bounding boxes.

Experiments across seven segmentation architectures reach 71.70% foreground mIoU on the paper's real-world benchmark. Adding synthetic data to real-only training improves five of the seven architectures by up to 7.19 mIoU points. Evaluation on a quadruped guide dog robot and a cane-shaped mobility-assistive device shows relative transfer gains on their egocentric viewpoints, while absolute accuracy remains insufficient for autonomous deployment.

Perceiving the objects that make navigation accessible.
NavAble brings together real-world data and controllable synthetic generation for blind and low-vision navigation.

11
accessibility object classes
450.9K
synthetic frames
37
simulation environments
500
released 3D assets

Key Contributions

The NavAble Dataset

Three complementary data sources expand coverage of the objects that support everyday mobility decisions.

ComponentCountRole
Curated real-world data36,466 imagesExisting sources remapped to a shared class taxonomy.
Newly collected real-world data5,681 records5,581 released records plus 100 supplementary revolving-door images.
Synthetic data450,860 framesControllable appearance, lighting, environments, and viewpoints.

Counts follow the current Hugging Face release. The 5,581 released collected records are split into 3,703 train, 396 validation, and 1,482 test records. The 100 supplementary revolving-door images are separate from these splits and await split assignment in the release.

The released-mask audit identifies 5,479 distinct photos within those 5,581 records. Per-class counts reflect images containing a class; classes can overlap, so their counts should not be summed to obtain the dataset size.

APS buttonBus stopBus stop signCrosswalkDoor buttonElevatorElevator buttonEscalatorHandrailPedestrian signalRevolving door

APS: accessible pedestrian signal. The main benchmark evaluates 10 foreground classes; revolving door is evaluated separately on the 100 supplementary frames (paper, Appendix H).

Manually Collected, Densely Annotated Data

We captured egocentric views across 50 indoor and outdoor locations with a chest-mounted RealSense D435. Every visible target instance receives a dense, pixel-level mask under shared annotation rules, followed by three full passes of cross-checking and correction.

SAM 3: 19.06% → 70.49% foreground mIoUA 51.43-point gain after fine-tuning on our collected real data alone.

Generate Synthetic Data for Your Own Task

Use NavAble’s Isaac Sim extension and companion command-line interface to build a dataset for your own object classes and viewpoints. The reusable workflow extends beyond BLV navigation to other perception tasks.

NavAble generator: user-defined object queries retrieve images, foundation models reconstruct 3D assets, and an object placement plus a camera trajectory produce labelled renders across assets, scenes, and lighting.
From object queries to textured 3D assets, reusable camera paths, and labelled renders in NVIDIA Isaac Sim.
  1. Choose your objects. Turn class-specific image queries into simulation-ready 3D assets with automated filtering and reconstruction, followed by mesh review.
  2. Set up once. Place an asset and record a camera trajectory with simple teleoperation. Replay the path across object variants, environments, and lighting.
  3. Export the supervision you need. Get paired RGB, semantic segmentation, depth, surface normals, and 2D/3D boxes directly from the simulator.
A pedestrian signal placed in a simulated street scene, surrounded by blue camera frustums marking a recorded trajectory.
Just two scene inputs: an object placement and a camera trajectory. Record once, replay with new conditions.

5 trajectories → up to 60,000 labelled frames

The paper’s example replays 5 trajectories across 4 environments and 10 asset variants to produce 20,000 annotated frames. Repeating the renders under 3 lighting conditions expands this to 60,000 frames without recording new paths.

Synthetic Data Demo

Lighting variation, annotation modalities, and a nine-object montage from the synthetic data generator.

The video first shows a pedestrian-signal button under changing illumination, then paired renderings and annotations, and finally nine examples of accessibility-related infrastructure.

Segmentation Evaluation

We benchmark seven segmentation architectures: SegMAN, Mask2Former, SAM 3, SAN, SegFormer, EncNet, and DeepLabv3+. Four training configurations separate the effects of collected data, curated data, and synthetic supervision, with 28 trained checkpoints released. The paper's main benchmark uses 1,482 real-world test images and reports foreground mIoU over 10 classes, excluding background and revolving door.

RealReal + CuratedReal + SyntheticReal + Curated + Synthetic

SegMAN with real, curated, and synthetic data reaches 71.70% foreground mIoU, compared with 26.07% for the strongest evaluated zero-shot baseline, CAT-Seg. Adding synthetic data to real-only training improves five of seven architectures, with a maximum gain of 7.19 mIoU points for SegMAN. EncNet also improves (+2.34 points); DeepLabv3+ and SAM 3 perform best on real-only training. Combining curated and synthetic data yields SegMAN's largest gain over real-only training: +9.50 points.

Results follow Tables 2 and 4 of the paper, using its training data and benchmark masks. Foreground mIoU excludes background; background-inclusive averages in the checkpoint documentation use a different denominator. Dataset counts above describe the current release.

Transfer to Assistive Robots

Offline evaluation on egocentric footage from Glide (157 images, 5 labelled classes) and a quadruped guide dog robot (155 images, 9 labelled classes).

Cane-shaped and quadruped assistive platforms, input camera frames, and SegFormer predictions comparing real-only training with real plus synthetic data.
Figure 4 from the updated paper: the platform, input frame, real-only prediction, and real-plus-synthetic prediction. The top two rows show Glide; the bottom two show the quadruped.

Adding synthetic supervision to SegFormer improves foreground mIoU from 11.94% to 16.17% on Glide (+4.23 points) and from 21.56% to 23.08% on the quadruped (+1.52 points), as reported in Table 5 of the paper. Mean precision also improves on both platforms.

These are relative transfer gains. Absolute accuracy of 16.17%–23.08% foreground mIoU does not establish readiness for autonomous assistive navigation; broader class coverage, geographic diversity, and reliable closed-loop navigation remain open directions.

BibTeX

If you use NavAble in your research, please cite our paper.

@inproceedings{hwang2026navable,
  title     = {{NavAble}: A Large-Scale Dataset and Synthetic Data Generation Pipeline for Blind Navigation},
  author    = {Hwang, Hochul and Monon, Jahir Sadik and Yang, Soowan and
               Patel, Shiven Umeshbhai and Nguyen, Anh Nhat Hong and Nguyen, Kien and
               Garg, Keshav and Gage, Dylan and Hordofaa, Duretti and
               Anjna, Anshu and Ohn-Bar, Eshed and Kim, Donghyun},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  url       = {https://neurips.cc/virtual/2026/loc/sydney/poster/138983}
}