Understanding the environment and offering advice in near real time is not a simple problem to solve. When we looked at the problem, we thought about previous problems that we solved, what we had found that worked, what did not work and how those previous problems relate to the current problem. We tried to balance uniqueness and commonality and we tried to bound the problem. What is the environment, what are the resources, what are the constraints — what are the set of requirements that we have had to meet — all play in generating a list of possible solutions.

In this one case, the one liner that best describes what we tried to accomplish was to enhance the probability that someone in the military or law enforcement will survive clearing a room, that bad guys are neutralized, and that good folks are able to leave the room unharmed.

The center piece for our scenario is the door kicker or officer entering the room. We need to know the state of the individual that is making decisions. The thought is to understand what the individual is seeing and hearing, and understand how stable the person is — heart rate, breathing pattern, anxiety. We need to understand what weapon is pointed where. We need to know where threats are located and where the good folks are located. We need to know the internal structure of a room. And we need to share information across team members in real time.

Our eyes, ears, nose, tongue, and skin are sensors. They are used to send signals to our brain for processing. Under most circumstances, we see, hear, touch and smell everything around us. Our environment is understood. But when the body is put into a stressful situation, the brain starts managing the flow of sensory data.

Where once we saw a panorama – blue sky and green trees spreading across a horizon, our brain kicks into survival mode. Tunnel vision — we start to focus very tightly on perceived threats — shrinks the stream of visual information. The brain starts shutting down our ability to hear sounds. Where once we could hear someone whisper, someone could be standing next to us and we would not hear a thing.

There are multiple ways to minimize the loss of information to the brain during times of stress. One way is called sensor augmentation. We are all familiar with sun glasses. Sun glasses are used to augment our vision by blocking sun light. We can take this a step further. We can use credit card size computers to record audio and video in the surrounding environment. Sensors attached to computers are not impacted by stress.

We collect data describing physiological states. We collect what humans hear and see. We collect the state of weapons. We assess threats. Based upon a shared location, we collect data streams from multiple officers. We analyze the data. We fuse it. We classify and calculate probabilities. And we ship the results of our analysis back to the end user.

We reviewed multiple cloud platforms. Because most of the software is open source, we can run across any of the major cloud vendors. The advantages of the cloud are obvious. We can automatically scale out to tens of thousands of streams. We can easily provide micro services. Because of the large volume of streaming data, we can store and retrieve using Cassandra, and Accumulo. By using Apache Beam and Flink as a runner, we can support stream, mini batch and batch processing.

Any questions?
6035056500