Birds overlaid on spectrograms
Illustrations: Hvass & Hannibal

The Sounds of Science


Marrying acoustic insights from ornithology with artificial intelligence, doctors and researchers at Weill Cornell Medicine hope to use the human voice to screen for disease.

By Andrew Myers

Sound as Biomarker

Bird, spectrograms and people

Dr. Anaïs Rameau is the granddaughter of a poet who lost his voice to cancer. Though he died when she was young, that spiritual connection steered Dr. Rameau into otolaryngology, where she now helps patients with difficulty speaking, swallowing or breathing.

Recently, Dr. Rameau has drawn attention for her pioneering research using patient voice analysis to detect aspiration — the inhalation of food or liquid into the lungs. Aspiration is made worse by difficulty swallowing and can lead to pneumonia, the top cause of death for people with certain age-related conditions, such as dementia and Parkinson’s disease.

“My grandfather didn’t die from his cancer. He died of pneumonia possibly brought on by changes in his swallowing,” Dr. Rameau recalls. “The changes in swallowing are audible. We can hear it. And now, we are using acoustic analysis to detect difficulties swallowing and coughing that might not be otherwise apparent to screen for pneumonia risk.”

Dr. Rameau recently earned a Beeson Emerging Leaders Career Development Award from the National Institute on Aging to develop a bedside app that tests for aspiration using sound. She now investigates whether vocal analysis, aided by artificial intelligence, could provide biomarkers for a broad array of physical, neurological and mental illnesses, ranging from cancer and heart disease to Alzheimer’s and autism. All can have associated changes to the voice, and ultimately, Dr. Rameau says, their biomarkers could lead to earlier, more accurate diagnoses, allow doctors to monitor patients’ health from afar, and even gauge how well treatments are working.

“The voice is connected to so many physical and mental processes,” Dr. Rameau says. “We think it is a window into illness and dysfunction.”

A Little Birdy Told Me

Intrigued by the potential of this under-explored area of medical science, but lacking acoustics expertise, Dr. Rameau turned to the highly regarded Cornell Lab of Ornithology, where scientists have spent decades studying birdsongs. There she met Dr. Holger Klinck, director of the K. Lisa Yang Center for Conservation Bioacoustics and one of the world’s foremost experts in the sounds of the animal kingdom.

Dr. Klinck’s team developed Raven Pro, a powerful software used to analyze animal vocalizations — birdsongs especially, but also sounds made by elephants, whales and other “critters,” as Dr. Klinck calls his subjects.

Raven Pro turns birdsongs, whale songs, elephant calls and primate chatter into beautiful visual representations. To the uninitiated, a chart of the trills of a melodic birdsong might resemble the elegant calligraphy of a Chinese brushstroke master. The multi-hued spectrograms go deeper still, displaying sound in three dimensions: time, pitch and intensity (volume) in colorful heat maps.

Dr. Klinck and team draw out understanding of animal communication and behavior from these representations and that is how they came to help Dr. Rameau screen for swallowing dysfunction and pneumonia risk from a simple cough. The two labs used Raven Pro to compare and analyze spectrograms of patient coughs to identify cough weakness and to show that people with weaker coughs were at greater risk 
of pneumonia.

Dr. Rameau was then able to use these very same tools to evaluate the effectiveness of corrective injection laryngoplasty — the injection of biological and biomimetic materials into the larynx to help patients with trouble swallowing and weakened voice.

“Just as it can tell us what’s wrong, sound can also tell us what’s right,” Dr. Rameau says.

Feed the Algorithm

More recently, Dr. Rameau has been testing her hypothesis through advances in artificial intelligence. She has teamed with Dr. Olivier Elemento, director of the Englander Institute for Precision Medicine and co-principal investigator of the NIH-funded BRIDGE2AI Voice project that is exploring the use of machine learning to analyze voice recordings for biomarkers. AI has already been used to analyze digitized X-rays, CT scans, MRIs, biopsy slides and even written medical records to search for hidden patterns of disease and dysfunction. And as telemedicine boomed during the COVID pandemic, so did interest in digital biomarkers of sound — among other reasons, to potentially distinguish between cough types in patients being treated remotely.

Dr. Elemento says the greatest challenge is not training the models, but finding the data. Unlike the large language model GPT for words, or ImageNet for digital images, there is no ready collection of standardized voice recordings to analyze to draw out understanding of how the human voice is affected by dementia, dysphagia or any number of medical conditions. “We have to create the data,” he says.

Dr. Elemento estimates BRIDGE2AI Voice will need 40,000 samples from recruited and consenting participants to amass a suitable dataset from which to train an effective and universal model that can diagnose any number of conditions from human sounds. By starting from scratch, Dr. Elemento says, BRIDGE2AI Voice can avoid the sort of racial, gender, age, geographic and other biases that have compromised other medical AI projects. Numerous promising medical AI studies have been challenged, if not negated, for underrepresenting minority populations in their data collection.

“We are thinking ahead of time, instead of after the fact, about bias, about privacy, about protecting patients’ medical data,” he says. “It’s an important advantage.”

Lend a Voice

Helping BRIDGE2AI reach those lofty standards is Alexandros Sigaras, a computer scientist who is both creating the app used to collect all those samples and, eventually, training the AI models based on them. The app not only collects the requisite audio samples but includes extensive medical questionnaires and cognitive challenges that will be used later to match diagnoses to voice samples.

“Years from now, you wouldn’t want to be studying the link between voice and dementia only to find you hadn’t collected the right data,” Sigaras says. “That takes extreme care in the collection phase. We want to get it right the first time.“

On the threshold of a promising data collection effort, Dr. Rameau returns to her grandfather whose cancer was evident to his wife in a worsening hoarseness that preceded his diagnosis.

“If my grandmother could understand the connection between my grandfather’s voice and his health,” Dr. Rameau says, “why not an algorithm?” 

Summer 2024 Front to Back

  • Features

    Science Over Stigma

    By probing the physical cause of obesity, researchers have repudiated harmful misconceptions, leading to new, highly effective medications.
  • Features

    The Sounds of Science

    How insights from ornithology, coupled with advances in AI, could enable doctors to screen for disease using the human voice.
  • Features

    Bones’ Secret Cells

    Research led by Dr. Matthew Greenblatt and his lab is revealing connections between bone stem cells and a surprising array of conditions — including cancer.
  • Notable

    Expansion in Midtown

    A 216,000 square-foot expansion of clinical and research programs at 575 Lexington Ave. will provide state-of-the-art clinical care at the Midtown Manhattan location.
  • Notable

    A Dramatic Growth in Research

    In the decade since the Belfer Research Building’s opening, Weill Cornell Medicine’s sponsored research funding has more than doubled.
  • Notable


    Heart disease presents differently in resource-poor countries like Haiti. Dr. Molly McNairy and colleagues are working to identify underlying causes and prevention.
  • Notable


    Weill Cornell Medicine faculty members are leading the conversation about important health issues across the country and around the world.
  • Notable

    News Briefs

    Notable faculty appointments, honors, awards and more — from around campus and beyond.
  • Grand Rounds

    Living With Endometriosis: A 12-Year Journey

    How the right treatment reduced the pain of endometriosis
  • Grand Rounds

    Taking Action Against Lung Cancer

    Monitoring by Weill Cornell Medicine’s Incidental Lung Nodule Surveillance Program can lead to early cancer detection.
  • Grand Rounds

    News Briefs

    The latest on teaching, learning and patient-centered care.
  • Discovery

    Gut Check

    New evidence shows that a bacterium found in the gut of livestock could be a trigger of multiple sclerosis in humans.
  • Discovery

    Researchers Chart the Contents of Human Bone Marrow

    A new method for mapping the location and spatial features of blood-forming cells within human bone marrow provide a powerful new means to study diseases that affect it.
  • Discovery


    The latest advances in faculty research, published in the world’s leading journals.
  • Alumni


    Forging critical connections to move research from the bench to the bedside, our alumni are making an impact.
  • Alumni


    What’s new with you? Keep your classmates up to date on all your latest achievements with an Alumni Note.
  • Alumni

    In Memoriam

    Marking the passing of our faculty and alumni.
  • Alumni


    Marking celebratory events in the lives of our students, including the White Coat Ceremony and receptions for new students.
  • Second Opinion

    Equal Risk

    Does race have a role in calculations of health risks?
  • Exchange

    Health Equity

    Two faculty members discuss the importance of community-engaged research in their work to help combat cancer disparities fueled by persistent poverty.
  • Muse

    Finding Strength in Art

    Surin Lee is a Weill Cornell Medicine-Qatar medical student, Class of 2026, and a visual artist.
  • Spotlight

    Partners in Solving Surgical Challenges

    Dr. Darren Orbach (M.D. ’98, Ph.D.) and Dr. Peter Weinstock (M.D. ’98, Ph.D.) are pioneering the use of practice simulations to ensure successful complex surgeries.