(C) PLOS One
This story was originally published by PLOS One and is unaltered.
. . . . . . . . . .



A comprehensive guide to CAN IDS data and introduction of the ROAD dataset [1]

['Miki E. Verma', 'Rewiring America', 'United States Of America', 'Robert A. Bridges', 'Cyber Resilience', 'Intelligence Division', 'Oak Ridge National Laboratory', 'Oak Ridge', 'Tn', 'Michael D. Iannacone']

Date: 2024-02

Although ubiquitous in modern vehicles, Controller Area Networks (CANs) lack basic security properties and are easily exploitable. A rapidly growing field of CAN security research has emerged that seeks to detect intrusions or anomalies on CANs. Producing vehicular CAN data with a variety of intrusions is a difficult task for most researchers as it requires expensive assets and deep expertise. To illuminate this task, we introduce the first comprehensive guide to the existing open CAN intrusion detection system (IDS) datasets. We categorize attacks on CANs including fabrication (adding frames, e.g., flooding or targeting and ID), suspension (removing an ID’s frames), and masquerade attacks (spoofed frames sent in lieu of suspended ones). We provide a quality analysis of each dataset; an enumeration of each datasets’ attacks, benefits, and drawbacks; categorization as real vs. simulated CAN data and real vs. simulated attacks; whether the data is raw CAN data or signal-translated; number of vehicles/CANs; quantity in terms of time; and finally a suggested use case of each dataset. State-of-the-art public CAN IDS datasets are limited to real fabrication (simple message injection) attacks and simulated attacks often in synthetic data, lacking fidelity. In general, the physical effects of attacks on the vehicle are not verified in the available datasets. Only one dataset provides signal-translated data but is missing a corresponding “raw” binary version. This issue pigeon-holes CAN IDS research into testing on limited and often inappropriate data (usually with attacks that are too easily detectable to truly test the method). The scarcity of appropriate data has stymied comparability and reproducibility of results for researchers. As our primary contribution, we present the Real ORNL Automotive Dynamometer (ROAD) CAN IDS dataset, consisting of over 3.5 hours of one vehicle’s CAN data. ROAD contains ambient data recorded during a diverse set of activities, and attacks of increasing stealth with multiple variants and instances of real (i.e. non-simulated) fuzzing, fabrication, unique advanced attacks, and simulated masquerade attacks. To facilitate a benchmark for CAN IDS methods that require signal-translated inputs, we also provide the signal time series format for many of the CAN captures. Our contributions aim to facilitate appropriate benchmarking and needed comparability in the CAN IDS research field.

Funding: This manuscript has been authored by UT-Battelle, LLC under ContractNo. DE-AC05- 00OR22725 with the U.S. Department of Energy. The publisher, by accepting the article for publication, acknowledges that the U.S. Government retains a non-exclusive, paid up, irrevocable, world-wide license to publish or reproduce the published form of the manuscript, or allow others to do so, for U.S. Government purposes. The DOE will provide public access to these results in accordance with the DOE Public Access Plan ( http://energy.gov/downloads/doe-public-access-plan ). This research was sponsored in part by Oak Ridge National Laboratory’s (ORNL’s)Laboratory Directed Research and Development program and by the DOE. There was no additional external funding received for this study. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2024 Verma et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

1 Introduction

Modern vehicles are increasingly drive-by-wire, relying on continual communication of small computers called electronic control units (ECUs). Nearly ubiquitous in modern vehicles, Controller Area Networks (CANs) facilitate the data exchange among ECUs by providing a common network with a standard protocol. While lightweight and reliable, the CAN standard has well-known security flaws, lacking authentication, encryption, and other important security features. Furthermore, attack vectors to intra-vehicle CANs are growing in scope as vehicles are increasingly offering channels of connectivity. While exploitation of the CAN bus in previous works is often implemented directly, e.g., by mandatory on-board diagnostics II (OBD-II) ports [7, 8], successful attacks to vehicle CANs also can occur indirectly/remotely through a variety of vehicle interfaces, such as wireless communication channels [9, 10].

Consequently, CAN security and vulnerability research has accelerated, with most literature focused on proving how “hackable” vehicles are [7–14], or proposing novel CAN intrusion detection systems (IDSs) [5, 15–17]. CAN IDS research has grown rapidly, suffering from an inability to reproduce or replicate and compare methods. As a result, proposed detection techniques are often not tested on appropriate data due to lack of availability. E.g., Hossain et al. [18] simulate attacks in real CAN data by adding frames in order to validate an LSTM-based CAN IDS. In theory their IDS may detect much more subtle attacks, e.g., masquerade attacks (see Sec. 2.2), but without available data, this cannot and was not tested. Further, using simulated data or attacks limits fidelity compared to real vehicular CAN data with real, and physically verified, attacks.

To address this problem, we introduce the Real ORNL Automotive Dynamometer (ROAD) dataset, a novel CAN IDS dataset comprised of real automotive CAN data. ROAD contains CAN data from the vehicle during ambient driving including a wide variety of driver activities. The dataset has labeled attacks ranging from easy to difficult to detect. More specifically, ROAD includes multiple variantions of real (i.e. non-simulated) fuzzing, fabrication, unique advanced attacks, and simulated masquerade attacks. The goal is to allow appropriate testing and comparable benchmarking of CAN IDS.

To this end, we also provide a thorough guide to all publicly available CAN IDS datasets to aid researchers in selecting and the most appropriate dataset for testing their method. Our survey of previous datasets surveys the publicly-available datasets suitable for CAN IDS testing. We provide for each dataset their references details to help researchers, in particular, data characteristics (real/synthetic, raw CAN and/or signal-translated data, number of vehicles, total time) and attack characteristics (what types of fabrication, suspension, masquerade, or other are present, real/simulated, and whether the attacks are identifiable with simple timing-based methods). See Tables 1–3. We also include a discussion of the uses of the datasets, and findings from our in-depth data analytics on each dataset.

The remainder of this paper is organized into the following sections. The introduction provides the reader with an overview of the state of CAN IDS research. Here, we illustrate two major roadblocks prohibiting this research from advancing and focuses mainly on highlighting the dearth of quality data. We point out the consequences that scarcity of data is having on the community and map them directly to this paper’s contributions. Section 2 provides necessary background on CAN protocol and vehicle attack terminology. In particular, we partition attacks into categories fabrication attacks (which add frames to the bus, e.g., DOS and fuzzing), suspension attacks (which prevent frames from being sent thereby removing them from the bus), masquerade attacks (replacing legitimate frames with malicious frames), and other. Specific definitions of types of these attacks are discussed. Section 3 comprises the survey, analysis and discussion of all previous CAN attack datasets. Section 4 introduces our new CAN attack dataset, and Section 5 concludes this work.

1.2 Problem addressed By real automotive CAN data we mean a CAN capture from a vehicle. By synthetic automotive CAN data we mean data generated by a process designed to emulate a real automobile’s CAN. Further, we use the term real attack to refer to actual tampering with messages on a CAN, e.g., by adding a node that sends frames, augmenting a node to send malicious frames, or by removing a node that would, in normal conditions, send frames. A simulated attack refers to a method to augment a CAN log post collection. Such data is unavailable for four reasons. First, it is difficult and time-consuming to produce, with the exception of a few basic attacks. Due to available software both open-source, e.g., SocketCAN (https://python-can.readthedocs.io/en/master/interfaces/socketcan.html), CANutils https://github.com/linux-can/can-utils and proprietary, e.g., CANalzyer (https://www.vector.com/int/en/products/products-a-z/software/canalyzer) and VehicleSpy (https://intrepidcs.com/products/software/vehicle-spy), and OBD-II access to many vehicle CANs, reading and sending arbitrary messages on the bus is easy. However, removing/overwriting legitimate messages or sending meaningful messages with a targeted effect is very difficult and is often a per-vehicle endeavor as the former requires advanced hacking skills and the latter involves reverse engineering signals (issue (1)). Thus, CAN data with ambient traffic or crude fabrication attacks are readily available, but real CAN data with subtle attacks or attacks with a targeted physical effect is scarce. Second, producing realistic CAN attack data carries inherent risks to the passengers, bystanders, and to the vehicle itself. Dynamometers or dedicated tracks, which allow driving in a safe and controlled laboratory environment, can be used, but such facilities are large investments are unavailable to most researchers. Furthermore, risks of permanent damage loom (e.g., “bricking” an ECU). Third, disclosure of sensitive information is an inhibitor. OEMs consider their CAN encodings intellectual property. Additionally, responsible vulnerability disclosure may be necessary if new attacks are discovered, which at a minimum delays release of data. Further, releasing data with targeted attacks may be viewed unfavorably by OEMs, resulting in lawsuits if not handled responsibly. In short, developing subtle attacks can be prohibitively expensive as researchers must have a dedicated modern vehicle for study, appropriate facilities for safety, access to deep offensive security expertise, and potentially legal support. To our knowledge, there are currently six publicly available vehicle CAN datasets with labeled attacks (see Table 1). Likely due to the inherent difficulties in producing real CAN attack data described above, in these datasets, the only real attacks in present real data are fabrication attacks (by message injection)—all other attack captures are either real data with simulated attacks, or are entirely composed of synthetic data. All have significant limitations when supporting CAN IDS development. Fabrication attacks are generally simple to detect with timing-based methods and are thus limited in scope. Due to the complex dynamics of the broadcast CAN protocol, the simulated CAN attacks ignore aberrations in message timing, content, and presence that naturally occur, and therefore change data quality in unknown ways. Further, physical verification of the effect of the simulated attack on the vehicle is not possible. Succinctly, there is no publicly available, real CAN data with labeled attacks that is of sufficient quality to permit assessment of many CAN IDS methods.

1.3 Consequences A survey by Loukas et al. [49] classifies 17 surveyed automotive CAN IDS papers by utilizing the following evaluation methods: “analytical” (theoretical only, no evaluation on data), “simulation” (evaluated on synthetic CAN data or real CAN data with simulated attacks), and “experimental” (evaluated on real CAN data with real attacks). The distribution of the surveyed papers evaluated throughout the article is: analytical (3), simulated (8), experimental (6). We believe this percentage and number of IDS evaluation with real CAN data is too small. A second consequence is that CAN IDS works are not comparable, or at least not compared. Rajbahadur et al. [50] surveys an even larger set of papers, e.g. with a much wider scope of “Anomaly Detection for Connected Vehicle Cybersecurity”). The investigators found that: Much of the research is performed on simulated data (37 out of the 65 surveyed papers) … much of the research does not evaluate the newly proposed techniques against a baseline (only 4 out of the 65 surveyed papers do so), which may lead to results that are difficult to quantify. This also reinforces the findings of Loukas et al. regarding synthetic data/simulated attacks. It appears CAN IDS contributions come from researchers with a wide variety of backgrounds. While this milieu provides a diverse set of approaches, the area suffers by lacking a uniform body of knowledge, and the lack of depth seems to inhibit the steady development of ideas and systematic, quantifiable progress. To again quote Rajbahadur et al., The varied use and scattered publication of anomaly detection [for connected vehicle cybersecurity] research has given rise to a sprawling literature with many gaps and concerns … we urge researchers to address these identified shortcomings. To summarize, quantifiable comparison across competing and complementary IDS methods is currently not possible. Standardized datasets are necessary for head-to-head comparisons and for replicability, or better, reproducibility. To continue to progress with rigor, the CAN IDS research community needs to produce and adopt a publicly shareable collection of CAN datasets with labeled attacks. This sentiment was reiterated and acted on by Hanselmann et al. [5] in their recent CAN IDS work: To the best our knowledge, there is no standard data set for comparing methods. We try to close this gap by evaluating our model on both real and synthetic data, and we make the synthetic data publicly available. We hope that this simplifies the work of future researchers to compare their work with a baseline. Finally, we find that IDSs are often evaluated against inappropriate test data. For example, IDSs promising detection of advanced, subtle attacks are tested only on CAN data with exceptionally noisy attacks, or works use attacks that disrupt timing, then ignore timing in evaluation to test payload-based detection. In order to not disparage other IDS works, we cite our own insufficient evaluation of our proposed CAN IDSs as examples [63, 64]. The consequence is that many promising IDS methods, which are excessive for the easily detectable attacks in currently available data, are never truly evaluated on the more advanced attacks they target. We add to these cries for a more systematic and rigorous progression of CAN IDS research.

[END]
---
[1] Url: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0296879

Published and (C) by PLOS One
Content appears here under this condition or license: Creative Commons - Attribution BY 4.0.

via Magical.Fish Gopher News Feeds:
gopher://magical.fish/1/feeds/news/plosone/

You are viewing proxied material from magical.fish. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.