DARPA project creates new language for interfacing with multiple sensor data streams.

Dr. Hong ManResearchers at Stevens Institute of Technology have recently been awarded a DARPA grant to undertake one of the most ambitious challenges ever presented by the defense research agency. Dr. Hong Man and Dr. Yu-Dong Yao, both of the Department of Electrical and Computer Engineering, are helping the US military tackle the incredible task of intelligently managing all sensor and surveillance data collected in their global quest to secure American lives and interests.

"In today's digital landscape we all struggle with information management, but for the military, effective data use impacts decisions that can save lives," says Dr. Michael Bruno, Dean of the Schaefer School of Engineering and Science. "Any DARPA grant is an achievement, but this is particularly prestigious for Professors Man and Yao, given the priority and size of this pervasive problem."

Dr. Yu-Dong YaoTo streamline military data, Dr. Man and Dr. Yao have proposed using cognitive linguistics to bridge the semantic gap between the deluge of raw sensor data and the situation awareness that currently only human analysts can provide. By adding this linguistic framework to the daily flood of information from military radar systems, video surveillance, and audio recordings, the Stevens researchers enable sensors to understand, communicate, and even respond to threats. This unique approach turns a sensor's stream of 1s and 0s into actionable information that can accelerate the decision-making process in critical situations.

According to principles of cognitive linguistics, language is shorthand for our experience of the world, breaking down the complex environment around us into concepts that we can use to reason and communicate. By teaching sensors to linguistically identify primitive concepts—like thing, places, and actions—in their data streams, the systems can then produce a language that functions across sensing platforms and can be made coherent to humans.

"The scope of the problem naturally draws one's attention to the scientific specialty of cognitive linguistics for inspiring insights into the nature of semantic representation as the intersection between human languages (i.e. queries) and technical languages (i.e. sensor data processing outputs)," reports Dr. Man, the Principal Investigator on the Stevens grant.

This grant was made in response to DARPA's Mathematics of Sensing, Exploitation and Execution (MSEE) program announced earlier this year. With the goal of centralizing all Department of Defense sensor data and automating much of the data interpretation, it is a program so ambitious that Wired Magazine termed it a search for "a new language to explain everything."

Dr. Hong Man is an Associate Professor at Stevens and directs the Visual Information Environment Laboratory, which researches the integration of visual information acquisition, processing, presentation, understanding, communication, archiving, and user interaction. Dr. Yu-Dong Yao brings extensive industry experience to Stevens as Professor and Department Director for Electrical and Computer Engineering. Dr. Yao also oversees the Wireless Information Systems Engineering Laboratory, providing resources for exploration of advances in wireless communication systems, including performance analysis, algorithm development, and experimentation.

Visit the Department of Electrical and Computer Engineering to learn more about research and education at Stevens, or Undergraduate Admissions or Graduate Admissions to apply.