INTRODUCTION

The PKU-Vidar-DVS dataset is a large-scale multimodal neuromorphic object detection dataset with temporally continuous labels. It is constructed by the National Engineering Research Center for Visual Technology, Peking University.

Collection Steps and Calibration. This dataset is recorded using our hybrid camera system, which includes a Vidar (resolution 400*250) and a DAVIS346. As shown in Fig.1, the input light is equally divided into Vidar and DAVIS346 via a beam splitter (i.e., Thorlabs CCM1-BS013). On this basis, we design the spatio-temporal calibration procedures to synchronize two cameras within the shared view at the same time.

Data recordings and Annotation. Our PKU-Vidar-DVS dataset contains 9 indoor and outdoor challenging scenarios (see Fig. 2) by considering velocity distribution, illumination change, category diversity, and object scale, etc. We use the hybrid camera system to record 490 sequences including Vidar spikes and DVS events. In each sequence, we collect approximately 5 seconds as the raw data pool. To provide bounding boxes from asynchronous visual streams, frames are reconstructed from Vidar spikes at 50 FPS. After spatio-temporal calibration, all labels are provided by a well-trained professional annotation team.

Data Statistics. Manual annotations in the recordings are provided at a frequency of 50 Hz. As a result, this dataset has 103.3k labeled timestamps and 229.5k labels in total. Afterward, we split them into three subsets for training, validation, and testing. Notably, this is the first work to build a neuromorphic multimodal object detection dataset involving high-speed and low-light scenarios. Besides, more details can be found in Table 1.

Type	Sequence number	Classes	Timestamps	Labels
Training set	263	9	55.0k	133.2k
Validation set	111	9	23.7k	47.3k
Testing set	116	9	24.6k	48.9k
All	490	9	103.3k	229.5k

Table 1 The details of the PKU-Vidar-DVS dataset.

LICENCE

1. Vidar spikes, DVS events, and the corresponding annotation results can only be used for ACADEMIC PURPOSES. No COMERCIAL USE is allowed.
2. Copyright @ National Engineering Research Center for Visual Technology and Institute of Digital Media, Peking University (PKU-IDM). All rights reserved.

All publications using the PKU-Vidar-DVS dataset should cite the paper below:

Jianing Li, Xiao Wang, Lin Zhu, Jia Li, Tiejun Huang, Yonghong Tian. Retinomorphic object detection in asynchronous visual streams. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2022.

DOWNLOAD

You can download directly from here .

Email: pkuml at pku.edu.cn

Address: Room 2604, Science Building No.2, Peking University, No.5 Yiheyuan Road, Haidian District, Beijing, P.R.China.