Peterande/D-FINE

We conduct object detection using D-FINE and YOLO11 on a complex street scene video from YouTube. Despite challenging conditions such as backlighting, motion blur, and dense crowds, D-FINE-X successfully detects nearly all targets, including subtle small objects like backpacks, bicycles, and traffic lights. Its confidence scores and the localization precision for blurred edges are significantly higher than those of YOLO11.

• E25: Re-trained and extended the pretraining to 25 epochs.
• AP^val is evaluated on Objects365 full validation set.
• AP⁵⁰⁰⁰ is evaluated on the first 5000 samples of the Objects365 validation set.

1. Download COCO2017 from OpenDataLab or COCO.

2. Modify paths in coco_detection.yml

   train_dataloader:
       img_folder: /data/COCO2017/train2017/
       ann_file: /data/COCO2017/annotations/instances_train2017.json
   val_dataloader:
       img_folder: /data/COCO2017/val2017/
       ann_file: /data/COCO2017/annotations/instances_val2017.json

1. Download Objects365 from OpenDataLab.

2. Set the Base Directory:

export BASE_DIR=/data/Objects365/data

1. Extract and organize the downloaded files, resulting directory structure:

${BASE_DIR}/train
├── images
│   ├── v1
│   │   ├── patch0
│   │   │   ├── 000000000.jpg
│   │   │   ├── 000000001.jpg
│   │   │   └── ... (more images)
│   ├── v2
│   │   ├── patchx
│   │   │   ├── 000000000.jpg
│   │   │   ├── 000000001.jpg
│   │   │   └── ... (more images)
├── zhiyuan_objv2_train.json

${BASE_DIR}/val
├── images
│   ├── v1
│   │   ├── patch0
│   │   │   ├── 000000000.jpg
│   │   │   └── ... (more images)
│   ├── v2
│   │   ├── patchx
│   │   │   ├── 000000000.jpg
│   │   │   └── ... (more images)
├── zhiyuan_objv2_val.json

1. Create a New Directory to Store Images from the Validation Set:

mkdir -p ${BASE_DIR}/train/images_from_val

1. Copy the v1 and v2 folders from the val directory into the train/images_from_val directory

cp -r ${BASE_DIR}/val/images/v1 ${BASE_DIR}/train/images_from_val/
cp -r ${BASE_DIR}/val/images/v2 ${BASE_DIR}/train/images_from_val/

1. Run remap_obj365.py to merge a subset of the validation set into the training set. Specifically, this script moves samples with indices between 5000 and 800000 from the validation set to the training set.

python tools/remap_obj365.py --base_dir ${BASE_DIR}

1. Run the resize_obj365.py script to resize any images in the dataset where the maximum edge length exceeds 640 pixels. Use the updated JSON file generated in Step 5 to process the sample data. Ensure that you resize images in both the train and val datasets to maintain consistency.

python tools/resize_obj365.py --base_dir ${BASE_DIR}

1. Modify paths in obj365_detection.yml

   train_dataloader:
       img_folder: /data/Objects365/data/train
       ann_file: /data/Objects365/data/train/new_zhiyuan_objv2_train_resized.json
   val_dataloader:
       img_folder: /data/Objects365/data/val/
       ann_file: /data/Objects365/data/val/new_zhiyuan_objv2_val_resized.json

Download COCO format dataset here: url

To train on your custom dataset, you need to organize it in the COCO format. Follow the steps below to prepare your dataset:

1. Set remap_mscoco_category to False:

   This prevents the automatic remapping of category IDs to match the MSCOCO categories.

   remap_mscoco_category: False

2. Organize Images:

   Structure your dataset directories as follows:

   dataset/
   ├── images/
   │   ├── train/
   │   │   ├── image1.jpg
   │   │   ├── image2.jpg
   │   │   └── ...
   │   ├── val/
   │   │   ├── image1.jpg
   │   │   ├── image2.jpg
   │   │   └── ...
   └── annotations/
       ├── instances_train.json
       ├── instances_val.json
       └── ...

   • images/train/: Contains all training images.
   • images/val/: Contains all validation images.
   • annotations/: Contains COCO-formatted annotation files.

3. Convert Annotations to COCO Format:

   If your annotations are not already in COCO format, you'll need to convert them. You can use the following Python script as a reference or utilize existing tools:

   import json
   
   def convert_to_coco(input_annotations, output_annotations):
       # Implement conversion logic here
       pass
   
   if __name__ == "__main__":
       convert_to_coco('path/to/your_annotations.json', 'dataset/annotations/instances_train.json')

4. Update Configuration Files:

   Modify your custom_detection.yml.

   task: detection
   
   evaluator:
     type: CocoEvaluator
     iou_types: ['bbox', ]
   
   num_classes: 777 # your dataset classes
   remap_mscoco_category: False
   
   train_dataloader:
     type: DataLoader
     dataset:
       type: CocoDetection
       img_folder: /data/yourdataset/train
       ann_file: /data/yourdataset/train/train.json
       return_masks: False
       transforms:
         type: Compose
         ops: ~
     shuffle: True
     num_workers: 4
     drop_last: True
     collate_fn:
       type: BatchImageCollateFunction
   
   val_dataloader:
     type: DataLoader
     dataset:
       type: CocoDetection
       img_folder: /data/yourdataset/val
       ann_file: /data/yourdataset/val/ann.json
       return_masks: False
       transforms:
         type: Compose
         ops: ~
     shuffle: False
     num_workers: 4
     drop_last: False
     collate_fn:
       type: BatchImageCollateFunction

1. Set Model

export model=l  # n s m l x

1. Training

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/dfine/dfine_hgnetv2_${model}_coco.yml --use-amp --seed=0

1. Testing

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/dfine/dfine_hgnetv2_${model}_coco.yml --test-only -r model.pth

1. Tuning

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/dfine/dfine_hgnetv2_${model}_coco.yml --use-amp --seed=0 -t model.pth

1. Set Model

export model=l  # n s m l x

1. Training on Objects365

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/dfine/objects365/dfine_hgnetv2_${model}_obj365.yml --use-amp --seed=0

1. Tuning on COCO2017

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/dfine/objects365/dfine_hgnetv2_${model}_obj2coco.yml --use-amp --seed=0 -t model.pth

1. Testing

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/dfine/dfine_hgnetv2_${model}_coco.yml --test-only -r model.pth

1. Set Model

export model=l  # n s m l x

1. Training on Custom Dataset

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/dfine/custom/dfine_hgnetv2_${model}_custom.yml --use-amp --seed=0

1. Testing

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/dfine/custom/dfine_hgnetv2_${model}_custom.yml --test-only -r model.pth

1. Tuning on Custom Dataset

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/dfine/custom/objects365/dfine_hgnetv2_${model}_obj2custom.yml --use-amp --seed=0 -t model.pth

1. [Optional] Modify Class Mappings:

When using the Objects365 pre-trained weights to train on your custom dataset, the example assumes that your dataset only contains the classes 'Person' and 'Car'. For faster convergence, you can modify self.obj365_ids in src/solver/_solver.py as follows:

self.obj365_ids = [0, 5]  # Person, Cars

You can replace these with any corresponding classes from your dataset. The list of Objects365 classes with their corresponding IDs:

New training command:

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/dfine/custom/dfine_hgnetv2_${model}_custom.yml --use-amp --seed=0 -t model.pth

However, if you don't wish to modify the class mappings, the pre-trained Objects365 weights will still work without any changes. Modifying the class mappings is optional and can potentially accelerate convergence for specific tasks.

For example, if you want to double the total batch size when training D-FINE-L on COCO2017, here are the steps you should follow:

1. Modify your dataloader.yml to increase the total_batch_size:

   train_dataloader:
       total_batch_size: 64  # Previously it was 32, now doubled

2. Modify your dfine_hgnetv2_l_coco.yml. Here’s how the key parameters should be adjusted:

   optimizer:
   type: AdamW
   params:
       -
       params: '^(?=.*backbone)(?!.*norm|bn).*$'
       lr: 0.000025  # doubled, linear scaling law
       -
       params: '^(?=.*(?:encoder|decoder))(?=.*(?:norm|bn)).*$'
       weight_decay: 0.
   
   lr: 0.0005  # doubled, linear scaling law
   betas: [0.9, 0.999]
   weight_decay: 0.0001  # need a grid search
   
   ema:  # added EMA settings
       decay: 0.9998  # adjusted by 1 - (1 - decay) * 2
       warmups: 500  # halved
   
   lr_warmup_scheduler:
       warmup_duration: 250  # halved

If you'd like to train D-FINE-L on COCO2017 with an input size of 320x320, follow these steps:

1. Modify your dataloader.yml:

   train_dataloader:
   dataset:
       transforms:
           ops:
               - {type: Resize, size: [320, 320], }
   collate_fn:
       base_size: 320
   dataset:
       transforms:
           ops:
               - {type: Resize, size: [320, 320], }

2. Modify your dfine_hgnetv2.yml:

   eval_spatial_size: [320, 320]

1. Setup

pip install onnx onnxsim
export model=l  # n s m l x

1. Export onnx

python tools/deployment/export_onnx.py --check -c configs/dfine/dfine_hgnetv2_${model}_coco.yml -r model.pth

1. Export tensorrt

trtexec --onnx="model.onnx" --saveEngine="model.engine" --fp16

1. Setup

pip install -r tools/inference/requirements.txt
export model=l  # n s m l x

1. Inference (onnxruntime / tensorrt / torch)

Inference on images and videos is now supported.

python tools/inference/onnx_inf.py --onnx model.onnx --input image.jpg  # video.mp4
python tools/inference/trt_inf.py --trt model.engine --input image.jpg
python tools/inference/torch_inf.py -c configs/dfine/dfine_hgnetv2_${model}_coco.yml -r model.pth --input image.jpg --device cuda:0

1. Setup

pip install -r tools/benchmark/requirements.txt
export model=l  # n s m l x

1. Model FLOPs, MACs, and Params

python tools/benchmark/get_info.py -c configs/dfine/dfine_hgnetv2_${model}_coco.yml

1. TensorRT Latency

python tools/benchmark/trt_benchmark.py --COCO_dir path/to/COCO2017 --engine_dir model.engine

1. Setup

pip install fiftyone
export model=l  # n s m l x

1. Voxel51 Fiftyone Visualization (fiftyone)

python tools/visualization/fiftyone_vis.py -c configs/dfine/dfine_hgnetv2_${model}_coco.yml -r model.pth

1. Auto Resume Training

bash reference/safe_training.sh

1. Converting Model Weights

python reference/convert_weight.py model.pth

1. Overview of D-FINE with FDR. The probability distributions that act as a more fine- grained intermediate representation are iteratively refined by the decoder layers in a residual manner. Non-uniform weighting functions are applied to allow for finer localization.

1. Overview of GO-LSD process. Localization knowledge from the final layer’s refined distributions is distilled into earlier layers through DDF loss with decoupled weighting strategies.

Visualizations of FDR across detection scenarios with initial and refined bounding boxes, along with unweighted and weighted distributions.

The following visualization demonstrates D-FINE's predictions in various complex detection scenarios. These include cases with occlusion, low-light conditions, motion blur, depth of field effects, and densely populated scenes. Despite these challenges, D-FINE consistently produces accurate localization results.