Wildflow AI

I came across Wildflow randomly one day and found the results compelling and the technology interesting. Essentially, this demo shows how, with some pictures taken at the right angles, the team at Wildflow can reconstruct a full 3D digital model of a reef that you can view and move around in.

At each location, it identifies and correlates the photos that were used in the construction of that region or object. You can then move around the area, exploring all the differing coral that exists in the region. The panel on the right shows the source photographs that captured the spot you are looking at, along with the depth of the selected location.

Select a different photo and the viewer shows you which part of the reconstruction it contributed to:

You can even trace an individual photograph back to the exact camera position it was taken from, drawn here as a line running from the camera out to the point on the reef:

Because the whole scene renders in real time in the browser, you can fly through it freely rather than watching a pre-rendered video:

We’ll brush over the deep technical details, but from a high-level, this system uses two main components to take raw images and turn them into a 3D space you can move through in real time on your browser.

The first step is to turn the photos into a set of known camera positions and a rough 3D scaffold of the scene, a process called photogrammetry, or more precisely Structure-from-Motion, done here with Agisoft Metashape. It matches the same visual features across overlapping photos and then runs bundle adjustment, which jointly solves for every camera \(C_i\) and every 3D point \(X_j\) by minimizing the total reprojection error \(\sum_{i,j} \lVert x_{ij} - \pi(C_i, X_j) \rVert^2\), the gap between where each point actually lands in a photo and where the current estimate would project it. The fully open source COLMAP does the same job, and its paper Structure-from-Motion Revisited is a good technical entry point.

The second step is to take that scaffold and turn it into a 3D space you can move through in real time. This requires recent advances in radiance fields, and specifically 3D Gaussian Splatting (Kerbl and collaborators, SIGGRAPH 2023). The basic concept is to represent the scene as millions of tiny translucent blobs, each a 3D Gaussian \(G(x) = \exp\!\left(-\tfrac{1}{2}(x-\mu)^\top \Sigma^{-1} (x-\mu)\right)\) with its own center \(\mu\), covariance \(\Sigma\), color, and opacity, which a differentiable renderer projects to the screen and blends front to back. An optimizer then nudges those Gaussians until the rendered views match the original photos, and because drawing a frame is just sorting and splatting rather than the per-pixel ray marching of the earlier NeRF work, it runs live in the browser through the GaussianSplats3D viewer.

Putting all of that together, the end to end pipeline, from dive photos to the scene in your browser, looks roughly like this:

You can look at the specific technical implementation in the GitHub repo.

Of course, this same technology can be integrated into VR systems for detailed exploration, or into existing systems to allow scientists to better monitor coral conditions. As is the case with most early systems, the potential benefits here are just not fully known yet, but it is a promising methodology for greater understanding of the physical world.

Our coral reefs are dying. Threatened by climate change, overfishing, and pollution, corals are increasingly becoming bleached, fish populations are struggling, and the general ecosystems are disappearing.

The first step to solving any problem is finding a measurement tool to help us understand the problem and act as a signal to guide us when attempting to solve it. Scientists from around the world are hard at work collecting measurements and data to help us in the effort. This can be one more effective tool in the toolkit, allowing scientists and the general public to understand, contribute to, and share a detailed reconstruction of the state of our coral reefs, moving us one step closer to properly conserving what we can.

You can find additional details about the mission on their main website.

Along this journey, I also found out about other exciting resources for others to explore related to this mission, such as the Coral Restoration Database by the International Coral Reef Initiative. The Allen Coral Atlas stitches together millions of high-resolution satellite images into a single global map of the world’s shallow tropical reefs, using machine learning to sort the seafloor into habitats like sand, rubble, rock, and living coral so that managers anywhere can see what their reefs are actually made of. This could be a cool website post of its own but I figured it was closely related enough to be mushed into this one. I highly recommend you check it out as well. NOAA’s Coral Reef Watch turns satellite sea surface temperatures into near real time bleaching alerts, accumulating the heat stress a reef has endured into a measure called Degree Heating Weeks and even forecasting where bleaching is likely in the weeks ahead. And CoralNet tackles the analysis bottleneck, letting researchers upload survey photos of the reef floor and having its models automatically label what sits under thousands of sample points, whether coral, algae, or bare substrate, work that used to be done by hand one dot at a time.

Our reefs are disappearing, and it is hard to want to protect something you have never seen. As someone who has been Scuba Diving several times, the reefs are a beautiful sight teeming with life, unlike the abyss of the depths of the ocean (though they certainly have life down there too, it’s not nearly as plentiful). The coral reefs are an essential part of the ocean ecosystem. Tools like this help make the invisible visible. Dive in, wander around a reef from your desk, and share it with someone who has never been in such an environment.