Poster: Mind the Structural Gap - Evaluating Automotive CAN Datasets as Interchangeable Benchmarks for Intrusion Detection

Network intrusion detection systems (NIDS) proposed for the automotive CAN are routinely evaluated on public datasets under the implicit assumption that these datasets represent comparable notions of benign traffic. We challenge this assumption through a measurement study of six public CAN datasets: CH, OTIDS, CAN-MIRGU, ROAD, SynCAN, and X-CANIDS. Characterizing each along four structural dimensions ID space size, transition density, payload entropy, and inter-ID correlation, we find variation of up to 10× in ID space and 2× in transition density, and different structurally isolated clusters in terms of payload entropy and transition density. These findings show that public CAN datasets are not interchangeable benchmarks, and that evaluation results are strongly conditioned on dataset structure.