Workshop and Challenge: Automated Analysis of Marine Video for Environmental Monitoring

Invited Speaker Bios and Abstracts


Underwater Video Collection and Exploitation for Stock Assessment by the NOAA Fisheries Science Centers

Randy Cutter, PhD

NOAA Fisheries provides science-based conservation and management for sustainable fisheries and aquaculture, marine mammals, endangered species, and their habitats. More and more, NOAA’s science utilizes remote sensing tools such as underwater and aerial camera systems to conduct more efficient and non-lethal surveys of invertebrates, fish, birds, and marine mammals. These systems prove efficient for data collection and provide methods to survey animals and assess populations and stocks.  However, their data streams are large and are rapidly exceeding the processing capabilities of human analysts.  Because of this, NOAA has supported development of open-source image annotation, labelling, and analysis tools to automate the detection, classification, enumeration, and measurement of animals and their habitats. This talk will provide examples of image data streams acquired by NOAA Science Centers, introduce the imaging tools and frameworks under development to analyze them, and discuss the potential of embedded image analysis to support future surveys using autonomous platforms.

Randy Cutter is a Research Oceanographer for NOAA Fisheries in San Diego, California and studies marine ecosystems of Antarctica. He has been exploring and mapping oceans, estuaries, and the seafloor for many years using underwater camera systems and sonars deployed on ships and remotely-operated and autonomous vehicles. He is a representative of NOAA’s Strategic Initiative for Automated Image Analysis.

Dynamic Data Driven Observatories for Identification

Sai Ravela, PhD
The ocean and atmosphere are vast. Observing them to quantify environmental information of interest including abundance, diversity and movement is a challenging and important problem. Much of the application of computational vision remains uncoupled to the observation process. In this talk, I will discuss a way of closing the loop using theory, models, data and information.
I begin by discussing the state of the art system for individual identification along with some of its applications across species scales. Then, I will pivot into embedding this recognition within a dynamic adaptive autonomous observatory. In particular, I start with large physics-based models (of oceans and atmosphere) and reduce them using the patterns they produce into simple models that predict the evolution of the local environmental state in realtime. We use these reduced model to quantify uncertainty,  and then plan observational strategies to maximize information gain. The data that is obtained is assimilated into into the models and the cycle repeats. The closed loop behavior is studied through a tractable information theoretic approach in high-dimensional, non-linear and non-Gaussian models. Through experiments and methodological arguments, we show that identification process can in this way become much more efficacious. We also argue that such a  framework may be essential for making progress in observing natural environments, in contrast to learning with the assumption of no known environmental or agent dynamics at all.
Bio coming soon.

Statistical Tomography of Microscopic Life

Yoav Schecter, PhD

The exponential growth of underwater visual data and marine-life images makes it practically impossible to rely solely on manual annotation for analysis. We propose a way in which image sequences automatically yield unsupervised analysis of plankton populations. The approach yields the plankton size-distribution and 3D structure recovery of the plankton’s inner structure. It is useful both for in-situ submerged microscopy and in-lab microscopy of collected samples. This is a new type of optical computed tomography: a statistical 3D tomography based on random frames in an ensemble of live plankton. Massive capture of plankton images means that each live specimen gets imaged only once, at an unknown random orientation, scale, and shift. Despite this uncontrolled randomness, we achieve both volumetric recovery of the plankton genus and the size of each specimen. We define and solve a self-calibration problem of estimating rotation, shift, and scale. Contrary to common self-calibration in computer vision, which had dealt with fixed opaque objects, we derive a solution for semi-transparent volumetric organisms, each specimen of which is viewed once. We show for the first time experimental results of real plankton, recovered based on our novel approach.

After physics degrees and research specializing in wave optics and being a former weather forecaster, Yoav Schechner earned PhD in Electrical Engineering in 2000. He specializes in computational photography and diverse physics-based inverse problems. He had been a research scientist at Nayar’s lab (CS, Columbia U.). Since joining the Technion as faculty in 2002, he has been involved in underwater computer vision research, dealing with visibility recovery, range estimation, refractive distortions, virtual periscopes, exploitation of shallow-water flicker, underwater cooperative path planning and in-situ plankton tomography. Residing at the coastal town Haifa, Yoav further experiences the sea regularly and year-around in various water sports. He was a visiting scientist in MIT-CSAIL, Caltech and NASA’s Jet Propulsion Laboratory. The JPL experience was an introduction to big spaceborne projects, prompting him to a take a sustained atmospheric research effort and climate-research collaborations. Recent works include “3D in Natural Random Refractive Distortions” (The Fumio Okano Best 3D Paper Award); “Computational imaging on the electric grid” (Best Student Paper, CVPR 2017) and “Dynamic heterodyne interferometry” (Best Paper Award, ICCP 2018).

A Revised Underwater Image Formation Model

Derya Akkaynak, PhD

Researchers aiming to color correct underwater images are frequently faced with unstable results: available methods are either not robust, are too sensitive, or only work for short object ranges. This is usually explained by the challenge in the correction, e.g., images having low SNR, severe distortions and loss of color, etc.  Here we suggest that there is a more fundamental reason to these instabilities than merely `low quality images’ and show that they stem from using an inaccurate image formation model that descends from atmospheric imaging. Unlike in the atmosphere, attenuation in the ocean is a strong function of wavelength. The current underwater image formation model is based on the dehazing model and overlooks this wavelength dependency, implicitly introducing errors in the estimation of both the direct signal and the backscattered signal. We propose a revised model, which we validated through simulations using oceanographic data and in situ experiments underwater, that remedies this shortcoming.

Our revised model employs two different wideband coefficients for attenuation and backscatter, which the current model assumes to be the same. Furthermore, we show that each of these coefficients have dependencies – all of which were ignored by the current model – on camera sensor response, optical properties of the water body, ambient light spectrum, depth, distance between the sensor and objects in the scene, and their reflectance. The revised model may offer insights as to why many algorithms fail to achieve robust reconstruction of attenuated colors in underwater imagery. For example, it reveals that the attenuation and backscatter coefficients are unique to a scene and can vary even in the same scene depending on the imaging angle. Similarly, it shows why coefficients derived from one location will not work on imagery obtained from another, or across different sensors. Most importantly, it highlights the need for estimating object ranges accurately in a given scene, and calls for the development of new methods for color reconstruction

Derya Akkaynak is a mechanical engineer and oceanographer (PhD MIT & Woods Hole Oceanographic Institution ‘14) whose research focuses on problems in underwater imaging and computer vision. In addition to using off-the-shelf RGB cameras for scientific data acquisition underwater, she also uses hyperspectral sensors to investigate how the world appears to non-human animals. Derya has professional, technical, and scientific diving certifications and has conducted fieldwork in the Bering Sea, Red Sea, Antarctica, Caribbean, Northern and Southern Pacific and Atlantic, and her native Aegean.

Call for Papers

This workshop will convene experts and researchers interested in learning about the challenges, current work, and opportunities in marine video analytics, including video captured both under and above water. As in other domains, maritime video collection is expanding exponentially, particularly for monitoring of fish, shellfish, marine mammals, coral reefs, bottom habitats, and other wildlife. Biologists, oceanographers, stock assessment officials and others will benefit greatly from increased automated video processing, which is currently manual, labor intensive and very tedious. Indeed, manual annotation of video is the largest bottleneck to studying many marine biological problems such as estimating populations of fish, whales, shellfish and seals. The vision community has a compelling opportunity to significantly impact this important area of environmental science.

The workshop is currently accepting papers involving computer vision applied to maritime environmental monitoring, including:

  • Types of data (e.g. number, size, species classification) resource managers need to extract from video data
  • Animal detection and tracking in video, including stereo video
  • Animal species classification
  • Segmentation of similar individuals in high-density populations (e.g. schooling fish, seals hauled out on beaches)
  • Animal behavior analysis and classification
  • 3D reconstruction and volumetric assessment of animals
  • Normalcy modeling and anomaly detection
  • Aquatic plant and coral classification
  • Habitat and substrate classification
  • Underwater SLAM
  • Other imaging sensing modalities (sonar, lidar, hyperspectral)
  • Physics-based vision as it pertains to marine imaging, e.g. refraction at the surface and fluorescence
  • Methods that operate in far-field, low-contrast conditions
  • Image and video enhancement for underwater conditions including high turbidity
  • Interactive methods including on-the-fly model training

Please see our CMT page for paper submissions. The page limit is 8 pages, not including references, following the standard CVPR format. Each paper will be reviewed by at least two members of the program committee.

Important Dates

  • April 2 (11:59 PM EDT): Paper submissions due for standard track (non data-challenge) papers
  • February 23: Data challenge initial training data preview (phase 0)
  • April 20: Data challenge updated training data released. Download.
  • May 7: Final challenge data released (no annotations). Download.
  • May 11: Data challenge submission server opens.
  • May 23: Data challenge final submissions due
  • May 30: Competition results announced

Data Challenge

This workshop also features a data challenge in this unique domain. Large amounts of underwater image and video data have been collected and annotated by the National Oceanic and Atmospheric Administration (NOAA) from a variety of image and video sensors deployed underwater in a variety of oceanic environments and climates. The data shows many species of fish, shellfish, underwater plants, and other biota. Data and annotations will be available before the workshop, with well-defined problems to address fish detection, species classification, and scallop detection.

Submissions will be evaluated against a test sequence withheld until the end of the competition, and the winners invited to the conference. Please note that you do not have to submit a paper to the workshop to participate in the data challege.

Related and Previous Workshops

A few prior workshops have involved maritime environmental monitoring. At ICCV 2013, the Underwater Vision Workshop was the first (that we are aware of) to focus on the proposed domain. Three workshops on Automated Analysis of Video Data for Wildlife Surveillance (AVDWS), chaired by this workshop’s organizers, were held in association with IEEE WACV 2015, 2016, and 2017. The 2015 AVDWS included ten oral presentations (3 invited, 7 submitted) covering topics from counting and sizing fish to monitoring giraffe behavior in thermal video. The 2015 AVDWS also included a discussion of the DARPA Visual Media Reasoning Program and the role of citizen science in optical data processing. The 2016 AVDWS included a discussion of a Kaggle challenge for the automated detection of right whales from aerial imagery, bird recognition in mobile phones, and a web-based tool for the analysis of seafloor images from coral reefs. More recently, “The First Workshop on Visual Wildlife Monitoring” was held in conjunction with ICCV 2017. The ICCV workshop covered a wide range of wildlife topics, as did our three WACV workshops, including both terrestrial and marine problem domains.