BRYAN/COLLEGE STATION, Tx – Humans can easily navigate a vehicle in both on- and off-road environments to successfully avoid obstacles. Algorithms also exist to teach vehicles to drive efficiently on roadways – but many missions for the military take place in off-road and diverse terrain.
The Texas A&M Engineering Experiment Station (TEES) and the U.S. Army Combat Capabilities Command (DEVCOM) – Army Research Laboratory (ARL) have partnered to release the first synchronized off-road annotated imagery and Light Detection and Ranging (LIDAR) dataset for semantic segmentation – RELLIS-3D.
Dr. Srikanth Saripalli, director of the TEES Center for Autonomous Vehicles and Sensor Systems, researcher for RELLIS-3D and professor in the J. Mike Walker ’66 Department of Mechanical Engineering at Texas A&M University, said this joint work with ARL researchers, Phil Osteen and Dr. Maggie Wigness, will be essential in evaluating machine learning algorithms in off-road environments.
“There is a lot of data available specifically for self-driving cars that are on roadways,” he said. “What we wanted to do was to develop a similar kind of dataset for off-road military vehicles.”
In an operational environment, military vehicles are most likely going to be running missions and training off road, meaning datasets such as this will become more crucial to the Department of Defense.
RELLIS-3D uses LIDAR data synchronized with imagery data. Saripalli and his team synchronize the LIDAR and imagery data as results come in. Then they take the data and, together with ARL, go through each frame and identify different obstacles like trees, rocks, dirt roads and more. This provides the necessary annotations for evaluating state-of-the-art machine learning and deep learning algorithms that are currently used for detection and tracking of objects, vehicles and people.
The data was all collected on the Texas A&M University System’s RELLIS Campus, influencing the name of the new dataset. The 3D piece of the name comes from the data having 3D LIDAR data, as well as imagery data.
The Army and TEES’ research relationship is long-standing, and the RELLIS-3D collaboration is just one of many different projects the two organizations are working on together. TEES, through a cooperative agreement with the Army Futures Command (AFC), in Austin, TX, established the George H.W. Bush Combat Development Complex (BCDC) on the RELLIS Campus to help the Army modernize and accelerate the delivery of critical capabilities for our nation’s defense.
“We started working on this even before the BCDC’s project with the AFC,” Saripalli said. “My group started working with Dr. Phil Osteen at ARL on calibration of sensors and that led to this project,” he added. “This has been funded fully by the AFC ”
Saripalli said with datasets like RELLIS-3D, having as much variation as possible is important. For example, this data was collected on off-road terrain in Texas, but it doesn’t represent snowy conditions.
“These sudden and constant changes in terrain require an off-road navigation system to constantly re-evaluate its planning and control parameters,” Wigness said. “This is essential to ensure safe, reliable and consistent navigation.”
Moving forward, Saripalli’s hope is to collect more data from more places such that these datasets can be used to train better algorithms and evaluate them after the training.
With the testing space available at the RELLIS Campus and the BCDC’s Innovation Proving Ground (IPG), a 350-acre outdoor test and experimentation area, tests and research like RELLIS-3D will continue to expand. A Mobility Challenge Course, Off-Road Test Area, Subterranean Test Area and the multi-faceted, private 5G network make the IPG capable of rigorously assessing innovative technologies and prototypes in operationally relevant, real-world settings.
“I hope this becomes the benchmark dataset for machine learning algorithms in off-road scenarios. I hope any researcher who comes up with an algorithm for off-road terrain tests it on the RELLIS-3D dataset,” Saripalli said. “I hope this leads other researchers to come up with their own datasets for autonomous vehicles in off-road terrain.”
Read more about the RELLIS-3D project on the Unmanned Systems Lab website.