Updating audio – from archive through UI

Overview

Here’s how we handle all the audio that rolls in from the recorders every few weeks…  Click on the images to embiggen them.

 

Collect and store audio data from recorders

I still move files from the recorders by hand.  This is the only step that can make really big trouble down the line and I just don’t trust my scripting skill that much.

Audio data winds up in several places:  a local workstation, redundant backups of yearly volumes, and the RAID-0 archive.

  • Visit the bioacoustic recorders and rotate SD cards (I do it biweekly, weather permitting)
  • Copy/paste recorder files from SD cards to yearly directories on my local workstation.
  • Back those files up continuously for 2 years, store older years on at least two external backup drives, store at least one backup set offsite
  • Also copy/paste recorder files from SD cards to the corresponding yearly archive volume in the 80 Tbyte RAID array running on the BirdOMatic computer.

 

 

The Pipeline Script (One script to rule them all)

All the automated steps of the  process are encapsulated in this BASH (Pipeline) script.  I run it after the archive has been updated with new files from the recorders.  The Pipeline script is a single place to collect and pass parameters into scripts, remind me of things to do before starting and allows me to tailor the run by turning on and off steps.  It’s a great example of how to pass parameters into Python, BASH and R all from the same script.

Note: these scripts have a lot of comments to remind me of what’s going on — you’ll find them out on our GitHub site if you’re looking for details.

 

 

Run BirdNET–Analyzer to identify calls

The first step of the automated process is to rerun the analysis of the current-year directory of the bioacoustic audio archive to create a selection.table file of ID/Results for each audio file that hasn’t already been analyzed

  • TIP: set field recorder sampling rate to 48k – avoids the need to resample audio, much shorter processing time
  • TIP: use the –skip_existing_results option to narrow the analysis to new audio only
  • TIP: use the –lat and –lon options to narrow the possible-species list to the location of the recorders
  • TIP: use the –threads option to manage the load on the computer
  • TIP: use the file and folder naming conventions listed below

Steps:

  • The Pipeline script passes input and output directory parameters to the birdnet_analyzer.analyze script to do this, but if here’s a reminder of how to do this by hand.
    • CD into BirdNET-Analyzer script directory
cd /Users/mikeoconnor/Documents/BirdNET-Analyzer\ github/BirdNET-Analyzer
    • Launch the Python virtual environment defined during installation
source venv/bin/activate
    • Run the script – hardcode parameters (paths, GPS coordinates, etc.) into this command line
python3 -m birdnet_analyzer.analyze --threads 8 --lat 44.415947 --lon -91.700659 --skip_existing_results -o $HOME/Documents/BirdWorkbench/birdnet_analyzer_RUN_FILES/run_files_2025 /Volumes/Bio/Bio\ 2025/From\ Recorders
    • Shut down the virtual environment when BirdNET Analyzer is finished
deactivate

Note: these scripts have a lot of comments to remind me of what’s going on — you’ll find them out on our GitHub site if you’re looking for details.

 

 

Combine results into a single text file – one row per ID, one combined file per year

Rerun the whole yearly directory of BirdNET results files to generate a new version of the run_files_COMBINED_20nn.txt file with all the results for the year

  • TIP: use the file and folder naming conventions listed below
  • The Pipeline script passes input and output directory parameters to the BirdNET_run-file_combiner script to do this, but if you want to do it by hand:
    • Edit the Directory statements at the top of the script to hardcode input and output paths (see Naming Conventions below)
    • Run the script from the command line
sh $HOME/Documents/BirdWorkbench/BirdWorkbench_BASH_scripts/BirdNET_run-file_combinerV3.sh
    • Note: the latest version of this script skips the “params” file generated by newer versions of BirdNET-Analyzer

Note: these scripts have a lot of comments to remind me of what’s going on — you’ll find them out on our GitHub site if you’re looking for details.

 

 

Convert the merged selection.tables files into an .FST file for the visualizer program.

This step refreshes the data table that the R UI loads each time a new session request comes in.   A number of columns are calculated from the selection-table data and local weather data is posted for each ID.

This script (regenerate_FST_files.R):

  • Reads a single-year text file of BirdNET IDs,
  • Cleans up the data,
  • Uses fuzzy logic to join each row with averages calculated from SQL-based local weather observations,
  • Creates a single-year FST file, and
  • Combines it with other such files to create an all-years FST that’s used as input by the UI script.
  • The Pipeline script passes input and output directory parameters to the regenerate_FST_files.R script to do this, but if you want to do it by hand:
    • TIP: use the file and folder naming conventions listed below
    • Edit the top of the script to hardcode/replace all the variables and paths that are being passed in from the Pipeline script
    • Call the script with:
rscript $HOME/Documents/BirdWorkbench/BirdWorkbench_R_Scripts/regenerate_FST_files.R

Note: these scripts have a lot of comments to remind me of what’s going on — you’ll find them out on our GitHub site if you’re looking for details.

 

 

Refresh the weather database

Weather data comes from a SQLlite database that is completely separate from this project.  It’s documented here as a reminder for me and as example of how additional data can be merged with BirdNET-Analyzer data.

  • Log into the weather archive machine, navigate to desktop/Valley.wssite file and “show package contents” to locate the weather.db file
  • Copy the weather.db file to the weather_snoop_valley_weather folder on the BirdOMatic computer
  • Rename it valley.weather.db

 

Wrapping up

There are several options available at the end of the Pipeline script that provide the ability to restart the production and/or test versions of the UI script — which will pull in the new version of the .FST.