Live-streaming a marshland for fun - and science

Image 1 of 2

In this Thursday Nov. 1, 2018 photo, Brian Mayton, a research assistant at the MIT Media Lab Responsive Environment group, holds his mobile phone receiving live stream video and audio data at a marshland in Plymouth, Mass., which is equipped with wireless sensors, cameras and microphones to create a virtual reality world inspired by nature's rhythms. Researchers at the Massachusetts Institute of Technology hope that by live-streaming data, sights and sounds at the Tidmarsh Wildlife Sanctuary, they can help scientists understand wildlife restoration techniques and let other virtual visitors experience nature remotely. (AP Photo/Charles Krupa)

If a tree falls in the Tidmarsh Wildlife Sanctuary, it doesn't matter if there's no one around. You can hear it anyway.

That's because researchers have hidden dozens of wireless sensor nodes, microphones and cameras among the cattails and cedars of this Plymouth, Massachusetts nature preserve. Sounds picked up from the marsh and nearby woodland feed into an artificial intelligence system that can identify frogs or crickets, ducks or a passing airplane.

One goal is to help scientists better understand changing climates and improve wildlife restoration techniques. Beyond that, though, researchers want to use the collected data to help power an online virtual reality world — a kind of alternate universe modeled on live conditions in the marsh, but populated with fanciful creatures invented in a computer science lab.

Could this be the future of the nature walk?

As wireless sensors get cheaper, longer-lasting and more sophisticated, they're increasingly turning up everywhere. We're already seeing them in "smart" homes and cities, pulling in data that can be analyzed in real time to smooth traffic flows, save energy, monitor pollution or respond to crime. But what happens when you apply such an internet-connected network to nature?

A research team at the Massachusetts Institute of Technology has been trying it out at Tidmarsh, a former cranberry bog converting back to natural wetlands just a few miles from where the Pilgrims landed in 1620.

Remotely spying on nature isn't new, but the project goes far beyond simple webcams fixed on a hawk's nest or sea lions' favorite pier — or even the more sophisticated acoustic sensors designed to detect animal poachers.

The team's goals for what they call the Living Observatory include supporting wildlife restoration efforts. The sensors measure temperature, moisture and other environmental conditions.

But a broader mission is to offer people — including children — a deeper understanding of nature using their laptops, phones or headsets. They can do so remotely or in person while walking a nature trail, said the project's visionary, Glorianna Davenport, a retired professor and co-founder of the MIT Media Lab.

"It's gorgeous to walk in the woods and not be fiddling with a cell phone," Davenport said. On the other hand, she added, what if you can learn more about the microbial environment, or the return of an endangered species, from a well-crafted smartphone app or a virtual reality game?

If it works here, Davenport said, researchers are already envisioning more ambitious projects deep in the Amazon rainforest — or on the moon.

The idea has skeptics who are worried about the intrusion of technology and constant surveillance into the world's last places without it . The Massachusetts Audubon Society manages the 480-acre sanctuary and took some time before it agreed to outfit it with live-streaming cameras and microphones. It was assured that human voices would be scrambled.

Educators have also asked Davenport why she would want to encourage kids to carry around their smartphones instead of just appreciating nature without them.

"And I went, 'Why not?' That's how they learn. That is their mechanism of interacting," she said.

The sanctuary is beginning to flourish as it changes from a heavily-fertilized industrial cranberry farm into a wetland full of insects, birds and native plants. The base camp for MIT scholars is also Davenport's home, which she built in 1999 after first visiting the property in the early 1980s. As landowners, she and her husband partnered with Audubon to transform the land into what it might have looked like before it was carved into manmade bogs in the 19th century.

The research projects reflect the diverse interests of multimedia scholars. Davenport is a documentary filmmaker. Others are computer scientists or musicians.

One project creates soundtracks driven by sensor readings — such as higher pitches that indicate warmer temperatures.

Another resembles a more ethereal version of the Pokemon Go augmented reality game, but with elk-like phantoms galloping around a virtual world modeled on the marsh. If the sensors pick up a rainstorm at the real-life Tidmarsh, the animated creatures appear to get wet. If there's a loud, sudden sound, they become startled.

Yet another experiment involves strapping on a special headset while walking through the sanctuary. On a hot afternoon in late summer, MIT researcher Gershon Dublon and his colleagues tromped around the preserve in chest-high waders to show how the sensors can amplify a human's observation of nature.

"The forest is a lot more active than you would think, because wildlife is quieter when you're nearby," Dublon said.

The headset endows its users with a kind of supersensory power. Tap one ear and you can zoom your hearing toward a nearby pond where ducks are swimming. Gaze in another direction, tap again and listen in on a secluded spot beneath a canopy of trees.

It's a chance to hear elusive animals that scurry away long before humans get anywhere near them.

You can also travel in time, as MIT Media Lab professor Joseph Paradiso did last week — transporting himself out of the sleepy November landscape by tapping into sounds picked up from the same place six months earlier.

"We played the spring, and to me, that was a revelation," he said. "Hearing a dead landscape come alive as if you're there."