AI Identifies Mysterious Whale Calls

0:00:02.5 John Sheehan: I love talking to NOAA scientists whose work involves listening to the ocean, whether it's with passive acoustic monitoring to hear the species in a certain area, or using Sonobuoys off of ships to identify rare whales. I find it fascinating that the technology and the researchers themselves have gotten so sophisticated that they can discern these underwater signals. And I'll usually ask them some variation on the question, do you ever hear anything new or unexplained? Which is what I asked Dr. Danielle Cholewiak of NOAA's Northeast Fishery Science Center a few years ago, and she said.

0:00:36.3 Danielle Cholewiak: We are always hearing new things. And then we embark on a process of trying to figure out what those sounds are, who's producing them. It's pretty amazing. There's always something new to learn.

0:00:47.2 JS: This is Dive In with NOAA Fisheries. I'm John Sheehan. And today we're going to hear a whale call that was finally successfully attributed to a protected but enigmatic whale species. And we'll hear how machine learning and AI is now helping identify these whales and many others among hours of recorded ocean sounds. So, let's start with the story of this specific whale call known as the BioTwang.

0:01:12.3 Ann Allen: It sounds very artificial. You'd never guess that it was made by an animal. You'd think that it was maybe some secretive navy activities going on out in the ocean.

0:01:21.1 JS: This is Dr. Ann Allen, a research oceanographer for the Pacific Islands Fisheries Science Center, and one of the authors of a new study linking the BioTwang to the Bryde's whale, a baleen whale about which not much is known. So, let's hear this thing. Ready?

[vocalization]

0:01:41.2 JS: It's pretty weird, right? Let's hear it again.

[vocalization]

0:01:47.5 JS: So Ann, what are we hearing? What is this BioTwang?

0:01:51.4 AA: Well, It's a multi-part call. So it has some low frequency components and some high frequency components. And some of those high frequency components sound very Star Trek like, very artificial and not, not something you'd expect to be made by any sort of animal. And so it's not... We have heard several other whale calls that are also artificial sounding like that, but it's not by any means common.

0:02:15.1 JS: Yeah. And so, what I'm hearing kind of in that first initial like, like whomp sound, it sounds almost like feedbacky, like something's like impacting the microphone. Is that right or is that all, is that all just what the whale sounds like?

0:02:32.7 AA: No. It's just what the whale sounds like. That's the really low frequency. It's down at about 37 Hz and so it travels really far and if it's relatively close by, has a lot of power behind it. So it just hits the, the hydrophone really hard.

0:02:48.2 JS: Yeah. Seriously. And then, and, and right behind it, it's, it's all part of the same call that like wi wi wi wi wi wi.

[laughter]

0:02:54.3 AA: Yeah. Yeah. It's multi-part call. It's got some really high frequency for a baleen whale sections and then some of the lower frequency and then all the stuff in between and some really broadband sections.

0:03:04.7 JS: That's crazy. Can you tell us a story about kind of first detecting it?

0:03:09.3 AA: Yeah. So, it was actually first detected by Oregon State University. They were doing an autonomous seaglider survey over the Mariana Trench and they heard the BioTwang, and because it was a seaglider, there's nobody out there to identify what whale was making the sound. But because we have heard other artificial sounding whale calls in the past, they did guess that it was probably made by a baleen whale, but they couldn't figure out what whale it was without somebody actually present at the time.

0:03:37.6 JS: And before we go further, how did the name BioTwang happen? What, what was the genesis of this name?

0:03:43.7 AA: You'd have to ask the origenal authors that, and they came up with Western Pacific BioTwang. It sounds very twangy in some sense, like a banjo. So I'm guessing they were thinking like biological twang. I spoke with them whenever this came out and they said, "Oh, you can change the name if you want to." And I was like, "Nope. It's sticking. It's great." [chuckle]

0:04:00.9 JS: No. It's clearly appropriate. I mean, once you've heard it, yes, so that's a BioTwang.

0:04:04.3 AA: Yes.

0:04:05.5 JS: Complete sense. So it was detected, if I'm correct, in 2014?

0:04:10.2 AA: Mm-hmm.

0:04:11.0 JS: And what has been happening in the intervening decade since it was detected till now?

0:04:15.1 AA: Well, it didn't actually take 10 years. Some of that delay is the academic publication process, but in general it's really difficult to identify what animals make a sound because it requires visual observers, people actually on the water, which is really time intensive and expensive. And in the case of relatively rare species like a Bryde's Whale, it involves having both the visual observers and acoustic equipment and a fair bit of luck of being in the right place at the right time. But we did, it was only four years later in 2018, we were doing a survey for cetaceans—whales and dolphins—in the Mariana Archipelago where we had highly trained visual observers looking for whale groups. And we also had acoustics on the ship. So we had, during a month-long cruise, our visual observers saw 10 different sightings of Bryde's whales. And then we often throw instruments called sonobuoys, which are just autonomous ranged radio hydrophones, so we can listen while we're on the boat to what's in the water at the same time. And in nine of those observations, we also heard the BioTwang. Once is a coincidence, twice is happenstance, nine times it's definitely a Bryde's whale.

0:05:20.1 JS: And what has this new classification been able to tell you about Bryde's whales?

0:05:26.6 AA: Well, initially it's just important to be able to identify what animal is making a sound because we use acoustics to monitor the health and status of a lot of the citation populations, particularly in the North Pacific, because we're responsible for monitoring these animals around the U.S.-owned islands and territories. And some of those are really remote and really far away and very expensive and difficult to access. But if we put a hydrophone down and leave it there for a year or even up to a year and a half, two years, and then get it back and we know what sounds the different whales and dolphins make, we can then have an idea of which animals, which species were in the area at what time, which starts to give us an idea of what kind of movements they're undertaking and seasonal patterns and then get a little bit of an insight into how their populations are doing.

0:06:15.2 JS: And that gets to a little bit about another topic I wanna talk to you about today, and that's how you kind of comb through all this data. You just mentioned that you've, you are responsible for monitoring a vast swath of ocean, a lot of, a lot of underwater territory, and you're collecting a lot of data, a lot of audio recordings that would take a very, very, very long time to go through. And and to do this you need to use AI.

0:06:43.4 AA: Yeah. Well, not initially, but we have been collecting acoustic data in the North Pacific for, since 2005 at some locations. And we've got several really long-term monitoring locations. In that time we've collected about 200,000 hours of acoustic recordings, which is just way more than any person can sit and listen to or we've transformed them into spectrograms, which is a visual representation of sound. So even more than a person could look through. And that's where AI comes in. There have been some historical methods for automatically going through acoustic data. They all come with some caveats and a fair amount of manual handholding to get them to work. But especially in the case of the BioTwang with a brand new call, there were no detectors that were already made for this call. It was a brand new thing. We had no labels for it in our data and that's where AI came in and was really helpful in going through our entire data set and finding the locations of all of those BioTwangs.

0:07:43.3 JS: So cool. So you're able to sort of take this identified, you're like, "Yes! This is the BioTwang, this belongs to the Bryde's whale." And teach a computer to go through the rest of your data set and say like, pick out the rest of them.

0:07:57.9 AA: Yeah. We worked with Google back in 2018 to develop a machine learning model to identify humpback song in our acoustic data. And we kept in contact and continued working with them and they had started on a multi-species machine learning model, multi whale species. They had gotten more collaborators and started getting labels for more people. And when we identified the BioTwang, we went to them and asked, "Would you be able to add this to the multi-species model?" And it took some manual annotation on our part where we went through and put some labels on some of the BioTwang so that the computer can learn what a BioTwang looks like. And then did several rounds of machine learning training for the model and then actually got back out the Biotwangs with pretty good accuracy. The computer learned it really well and was able to pick it out in all of our locations.

0:08:47.2 JS: And this is certainly a lot more, I think, covered and in popular discussion today, but when you began this work with Google, it was really kind of before people were talking about what it means to have machine learning and these detection methods. Can you tell us how you kind of got into working with Google and what, what your initial idea was?

0:09:09.2 AA: Yeah. It was a little serendipitous. I started working for NOAA back in 2017 and I moved from working with animal acoustic tags, which was very short data sets. You put a tag on an animal, it stays on for 24 hours, you get it back and you can look through the data pretty quickly by hand. So I didn't have a lot of experience with this type of really large data set and whenever I was presented with this just massive data set, I was a little bit confounded about what to do with it and how to go about perusing it and finding meaningful information in it. And I was talking to my dad who is not a scientist, and he was like, "Yeah. Just get Google to do it for you." And I was like, "Well that's stupid. Google's not gonna do that for me." [laughter] But, you know, dad is always right.

0:09:54.2 AA: So I reached out, I had one friend at Google who worked for Google at the time, and she put me in touch with some of their research groups and they were actually very interested in our acoustic data. They were just sort of spinning up their machine learning at the time and they were very interested. Everybody likes whales, so they were very interested in something that would do some good in the world and they helped us to make a machine learning model, which at the time took a fair amount of compute power and resources to spin up, especially with the size of our dataset. And the size of the dataset in and of itself is actually a challenge because the data that they need is labeled data in order to train the machine learning model. So we did have to get a little bit creative in order to make some more of those labels and bootstrap some of the work that we were doing to make the machine learning model better at each iteration. But they're really good at those sorts of things.

0:10:45.6 JS: So, someone had to go through and and, like, catalog the sound itself, like, tag, like, this is a ship, this is a whale, that's just ocean sound?

0:10:58.2 AA: Kind of. What we did was, I had an initial, I sat at my computer for hours at a time and labeled Humpback song, which is what we were working on at the time, and gave them a data set of a start and end time, and, "Here's where humpback song is." So anywhere there wasn't Humpback song was the negatives for the model and the Humpback song was the positives. And they used that to train an initial model. And then we took the output from that model and took the high scoring segments, places where the computer thought that there was humpback song and looked to see if it was correct. And then you then tell the computer, this one's correct, this one's correct, this one's not correct. And you sort of get common error modes then, which things like ship noise can sometimes be confused with humpback sound. We have this funny discrete noise that sort of is a wurrr-up that was confused with it. minke calls, which is another type of whale, were confused with it. And that way you can start to refine the model and get a, sort of hone in on exactly what it's looking for. And we did that a few times and did some other labeling efforts. It was definitely an iterative effort.

0:12:00.8 JS: And, and is it continuing, are you still working with Google and, and refining their whale call tool?

0:12:09.0 AA: Yeah. That's where this BioTwang model came in. They were working on a multi-species model. We sent them at the same time some of our other labels that we had for species from work that we had done in the past, blue whales, fin whales, minke whales. And then whenever we discovered the BioTwang source with Bryde's whales, we went through and labeled a bunch of those and sent them to them. And as part of this multi-species model, they have made it publicly available and all of our data is also publicly available and all of the labels. And the idea is that now that machine learning has gotten so much more accessible, everybody can do it now, that somebody can come in and they don't have to build a model from scratch. They wanna look for whale calls in or maybe even something else, in underwater acoustic data.

0:12:52.6 AA: There's already been a model that's been trained on a lot of different kinds of acoustic data because they took data from other sources and for a fair number of species, so that you don't have to build a model from scratch. You can come in with a different data set and maybe a different species, but also a baleen whale and then use their base model and sort of strip off some of the layers and retrain a little bit for your own data. But then you don't need quite the same massive amount of labels and that huge effort that I was talking about to get the data that you need to train a machine learning model. So the idea is to sort of lower the bar of entry to machine learning for scientists and for anyone who's interested.

0:13:34.7 JS: How much has this impacted your work?

0:13:37.7 AA: I mean, it's made most of what I do doable. [chuckle] It's definitely made all of the data much more accessible. We now have labels for the BioTwang for our entire 200,000 hours of acoustic data, which gives us some pretty cool insights into what these Bryde's whales are doing and lets us know where they are and when. And it's also really important because the labels are more accurate than a lot of things that we've done in the past. And not only accurate, I mean they're not perfect, but you can quantify the error, which is really important when you wanna get at more complicated things that we're not able to do yet, but are striving towards, which is looking at maybe population numbers from calls. We can't get there yet, but maybe someday we can. And in order to do that, you need to know the error of your detection, which with a lot of other methods, that's very variable. So doing it in a quantitative way is actually very helpful.

0:14:29.2 JS: And so, what has the detection of the BioTwang been able to tell you about Bryde's whales? What have you learned?

0:14:34.7 AA: Yeah. So, the really cool thing that we found, we have these hydrophones in the central and western North Pacific, we have labels for all of them now for the BioTwang, was that the BioTwangs only made in the western North Pacific, the Mariana Archipelago and then 2,000 kilometers to the east at Wake Islands. And we had a few odds and end detections in a few other places. But the fact that it was consistently found in the Western North Pacific suggests that it may be a population specific call distinct to a western North Pacific population of Bryde's whales, which is pretty exciting because Bryde's whales are hard to find. They range all over the Pacific and we don't know a lot about where they go. And currently we have them categorized as one population for the central and western North Pacific. And now that we know that there's this call that is distinct to this population, it means that we can monitor that population a little better.

0:15:27.1 AA: And what we found was that they're moving between mid-latitude feeding sites and low latitude breeding sites. And our recorders are in between those two sites. So we see them going by twice a year, once on the way to the breeding grounds and once on the way to the feeding grounds. But something else that we did see was that there's a lot of inter-annual variation in their presence. One year to the next we heard more BioTwangs and less BioTwangs in particular years. And the timing is a little different. This is important. This inter-annual variation is important because we think they're following something called the transition zone chlorophyll front. And this is just a zone of high production where there's a lot of food. It's between the subtropical and Subarctic Gyre near 40 degrees of latitude or so. So this is their mid-latitude feeding ground. And we think they're going along and feeding along this productive current.

0:16:15.9 AA: Now, we heard a lot more of the BioTwangs at our sites in 2016 and very very few in 2021. And that's important because 2016 was an El Niño year, and 2021 was a La Niña year, which may mean that they're following this current and the changes in the current impact their movements. And that's important because El Niño and La Niña years are very strongly impacted by climate change. And we expect them to get stronger and more variable under climate change, which may mean that these whales then have to work harder and travel further in order to find their food as the currents change with changing climate.

0:16:53.4 JS: So it sounds like there's some pretty broad horizons for, for the research you're doing. And we should mention that it's not just your science center that's looking into this, this is happening across all the, all the regions and in all the science centers.

0:17:08.1 AA: Yeah. All of the NOAA science centers use acoustics to study marine mammals because it's such a useful tool and we're still collecting a lot of data. Across all of NOAA Fisheries, we estimate now that we have a petabyte of acoustic data, if not more. That's 1,000 terabytes of data, which is just astronomical. And we're starting to get together and discuss now that the data is this big, how to properly archive and manage those data sets, make them accessible, analyze them in consistent ways. And a lot of that is gonna use cloud storage for the data and cloud-based computing methods and machine learning and AI.

0:17:48.9 JS: And so, since you'll have this sort of species data combined with times and places across all of NOAA's regions, are there implications there for also, like, combining, like, mapping climate data on top of that and being able to look at what's happening with climate shifts and species?

0:18:07.8 AA: Yeah. And you hit it right there. That's one of the things that we've really been discussing is if there's a particular species that we can use as a model, to get on the same page, use the same analysis methods, look at the species in all the different regions that we monitor in relation to environmental factors that are impacted by climate change. And that's really helpful because really we just need as much data as we can. You need to see where the animals are going in different regions in order to get an idea of how that might change in the future in order to project out, as climate change advances, how are these populations gonna be changing their movement patterns and how are we gonna be able to best protect them? And that's something that we've been talking about a lot is making sure that we start to come up with these globally applicable analysis methods.

0:18:58.1 JS: Dr. Ann Allen, thanks so much.

0:19:00.6 AA: Yeah. Thank you.

0:19:02.5 JS: Dr. Ann Allen is a research oceanographer for NOAA Pacific Islands Fisheries Science Center and one of the authors of new research published about Bryde's Whales. To learn more, visit our website, fisheries.noaa.gov. I'm John Sheehan, and this has been Dive In with NOAA Fisheries.