Suggestions and feedback from a biologist & how to connect via cell phone in the field

[DD4WH]

@mcguirepr89
Thanks for being so active with this BirdNET implementation!

My profession is (among other things) teaching ornithological methods for nature conservationists, so this list of ideas/suggestions is surely biased and emphasizes the use of your system, not necessarily the technological perspective :-).

I collected some ideas for improving performance, flexibility and promoting professional use of your BirdNET-system and grouped them into three categories:

1) Installation/network
• The use of the installation script is super easy and makes it possible for an unexperienced RPi-owner to install this system
• In my home network, after installation, the system does not work without a LAN-connection to the RPi. The Pi is connected to my home WLAN, but when I pull the LAN-cable, it is no longer possible to connect to the RPi from another computer in the network [however, I can connect to it via a remote desktop application like NoMachine]
• It would be good to fully integrate the possibility of remote desktop access into the installation process: I initially tried Anydesk, but that does not work, because Anydesk does not (yet) offer an installer for a 64-bit RPi-system (and using the 32-bit-RPi-installer installs something, but it is not possible to start Anydesk --> many sources on the web report this without any solution to it). I helped myself with the alternative NoMachine, which does have an installer (use the one for ARMv8 architecture) and allows remote access even from my smartphone and with functional audio streaming!

2) Settings for more flexibility
• BirdNET-system uses the eBird distribution data to filter the species list for the internal BirdNET classification process. There should be an option to switch this ON/OFF during runtime or at least in the config.py-file. I plan to use the system with mobile phone connectivity making it impractical to regularly download such an amount of data. Additionally, for conservation work, it is important to have classifications for rarer species even if they have not been included in the eBird data. For example, I am monitoring pygmy owl (which is quite frequent here), but there are no records on eBird for this region, so it is never recognized by the system as it is at the moment. I helped myself by setting the eBird threshold to 0.0, but then ALL species in the world find entry into my classification results. And still, the eBird data is being downloaded!
• A must for me is the possibility to use custom species lists, so that you can use the/a global BirdNET model, but filter this for my region. By the way, I have a list for European species (about 700 species) derived from the species list of the BirdNET-Lite classifier species list with 5000+ species. If anybody is interested, I can post it here or elsewhere
• There should be the possibility to make settings in real time on the web interface: Settings should include: eBird threshold, spectrogram overlap, spectrogram minimum length, minimum confidence, prediction sensitivity

3) More features
• The points mentioned above have higher priorities from my perspective, but here come some ideas, which could make the system even more attractive, but will probably be a little more complex to implement:
• A simple bargraph with the number of extracted recordings per species (as a first overview graph) would be very nice: this could help immensely in separating wrong classifications from securely identified species. This could also be implemented very fast on the birdstats.local page as a simple figure after the species
• Stefan Kahl has already programmed a beautiful streaming interface which shows a running realtime spectrogram of the incoming audio, simultaneously broadcasts the audio stream and shows the subsequent classification results underneath as well as a list of the classifications in the last hour/day. This is already available as RPi-code in a github repo (although I have never managed to get this installed). That would be very nice to have as an add-on feature of BirdNET-system which can be switched off, if CPU resources or RAM do not allow
• The notification system for mobile phone does not work for me, simply because the App „pushed“ is not functional on Android systems. There are many complaints in the app ratings stating that the app does crash on startup, and this is also the case with my android phone. So this does not seem to be a solution for Android users in general
• Phone notification: would also be nice to have an option to be notified when a specific species has been heard: would make sense in my example of pygmy owl monitoring!
• In the longterm and for more professional use cases it would be good to be able to swap classification models. One could then use different models for different use cases: NFC (night flight calls), owls, global models, regionally tuned models. However, of course this would need specifically trained models (either trained by the researcher himself or by Stefan Kahl :-))

Please understand these points as an input to make your wonderful system even more wonderful!

Regards, Frank DD4WH

[mcguirepr89]

Frank, you're awesome! Thank you, first of all, for dedicating so much of yourself to conservation efforts and for educating others in the subject -- we need more folks like you!
I'm going to respond to what I can as best as I can:

1. Installation/network
I have been considering a few alternative networking arrangements for scenarios like yours and for setups that might one day be truly remote. Though a few things to note: even when the BirdNET-system is offline, it does still collect audio, run it through BirdNET, and perform the extractions. So, the system does continue running, though you won't be able to view any of this data without getting connected to the machine in some way.

Two possible solutions to this which I will explore at some point:

• The RPi is able to create its own Access Point/(Local WiFi Network) which you could join in order to connect to the device and view its web interface. The basic idea is that the BirdNET-system would broadcast a WiFi network, say BirdNETwork, and then any device (your Android) could connect to that network and access the http interfaces. The down side to this option is that it would not have any WAN connectivity, so devices on that BirdNETwork would not have any internet access. This is actually not an issue, though, from the system's perspective. The step you see in the birdnet_analysis.service log (http://birdlog.local) during which the system states it is "LOADING eBIRD GRID DATA... DONE! 13800 GRID CELLS" does not use any internet connectivity for this pull, it is loading this all locally. This is good news for your use case with mobile phone connectivity. There may be also be a way to connect the Raspberry Pi to your phone's hotspot (before going into the field) and achieve the same network visibility as the Access Point solution. This solution does not, however, work if you are using the network card to connect to WiFi in the first place.
  • (To be clear, on your home network, if devices use the same router but have no visibility of one anther, there is very likely a subnetting issue. It may also be possible that the router blocks multicasting, which would make all of the mDNS avahi-aliases invisible).
• The other solution is similar, though instead of broadcasting an access point from the network card, it is possible to use the RPi's bluetooth to broadcast a connection point. I have not personally tinkered with this, but am convinced that it is possible.

Thank you for the feedback on the AnyDesk bit-version issue -- I will change my documentation to recommending the NoMachine setup you've had success with and will keep an eye out for other solutions.

• Lastly, though this option still requires connectivity to the same LAN, I can enable the VNC server on the RPi which could then be accessible from any VNC client, for which there are options available across all operating systems and machine architectures.

2. Settings for more flexibility

• I will need to work with Stefan Kahl to be able to make adjustments to how BirdNET works, but I'm sure this input is valuable to the project and I will keep it in the conversation. After I have worked out as many kinks with this repo as I can reasonably find, I plan to build a companion fork to the BirdNET-LITE repo which can accept a custom species list. I will keep you updated on any progress I have with that so that your research areas can be more directly addressed.
• I LOVE the idea of being able to adjust settings in the web interface. I'm a crappy web-designer, but understand backend server stuff well, and imagine this could be put together relatively easily with some PHP scripting. (Anyone interested in helping out, let me know or give it a go!!!!!) I'll see what I can do!

3. More features

• I love the bargraph idea. I have created a variety of reports for the http://birdstats.local page, one of which had some frequency data and another of which listed the species by most recent. I will revisit those with this suggestion in mind.
  • Stefan Kahl has already programmed a beautiful streaming interface which shows a running realtime spectrogram of the incoming audio, simultaneously broadcasts the audio stream and shows the subsequent classification results underneath as well as a list of the classifications in the last hour/day. This is already available as RPi-code in a github repo (although I have never managed to get this installed). That would be very nice to have as an add-on feature of BirdNET-system which can be switched off, if CPU resources or RAM do not allow. PLEASE SHOW ME THIS!!! I didn't know about this -- it sounds so amazing!!
• The phone notifications are an awesome little feature and can be tweaked (though I haven't included that script) to send notifications for specific species. I am disappointed that the App is a dud for Android users. I definitely added this features because there was very little barrier to entry for me to get it working with iPhone, but I have no experience developing iOS applications, so again, if anyone wants to give it a go, let me know!!!!!!!! I will add the specific species notifier within the next few pulls.
• And finally, I love the idea of incorporating different classification models. I'm hoping someone with experience using some of those other models might be able to chime in about how to get started making those additions.

Thank you so much for your feedback and providing some insight from the biologists' perspective ~~ very cool.

My best,
Patrick

[DD4WH]

Patrick, thank you so much for your detailed answer and the suggestions. I have to find some time to try some of your suggestions and I will report my success here.

Just a quick answer on your question regarding this:

Stefan Kahl has already programmed a beautiful streaming interface which shows a running realtime spectrogram of the incoming audio, simultaneously broadcasts the audio stream and shows the subsequent classification results underneath as well as a list of the classifications in the last hour/day. This is already available as RPi-code in a github repo (although I have never managed to get this installed). That would be very nice to have as an add-on feature of BirdNET-system which can be switched off, if CPU resources or RAM do not allow. PLEASE SHOW ME THIS!!! I didn't know about this -- it sounds so amazing!!

RPi code is here:
https://github.com/kahst/BirdNET-Demo

and in action it pretty much looks like this:

https://birdnet.cornell.edu/live/

I have seen it running live at a conference where Stefan Kahl presented BirdNET with a Raspberry Pi runnning that code, absolutely amazing!

Have a nice weekend! Frank DD4WH

[mcguirepr89]

Thanks, Frank -- I'll definitely be checking out that repo this weekend ~~ I look forward to any updates and hope you have a great weekend, too 😁

[mcguirepr89]

It turns out that the live demo can use the same pre-built tflite binaries that BirdNET-Lite uses, so I will be incorporating the realtime/live analysis (that unfortunately does not yet show the live spectrogram, but does show live analysis with prediction scores in TRULY realtime) pretty soon.
I'll let you know when the new feature is test-worthy.

[DD4WH]

Excellent news!
Really looking forward to that!
Sorry for my silence the last days, work is eating up much time...

[mcguirepr89]

@DD4WH I have worked a bit with your suggestions and have the rpitesting branch working with a Caddyfile that allows the Raspberry Pi to connect to my cell phone's hotspot, then I can reach the BirdNET-system at http://raspberrrypi.local -- I was hopeful that the setup would work for you out of the box, but the installation that you used sets the first Caddyfile block to only listen to http://localhost and then the rest of the http services are broadcast via mDNS. While the mDNS features aren't available using this hotspot connection, the main Extractions page is available with this method. I will be pulling the rpitesting changes into the main BirdNET-system-for-raspi4 branch in the next few days after more testing. Lots of great success today, but I seem to always overlook something, so I'm waiting a bit longer before pushing these changes. Once I have pushed the changes, I hope you'll consider reinstalling to add this new feature.

[DD4WH]

sounds good! As soon as you have a version ready on github and want me to test, just drop me a note and I will follow the installation procedure on another SD card and test the new functionalities for you!

[mcguirepr89]

Hello, friend! I wanted to update you that the newest changes to the installer (including the Caddyfile tweaks I mentioned that should allow you to set the BirdNET-system up to use your phone's hotspot when it is available) have been fully merged into the main "BirdNET-system-for-raspi4" branch.
I wanted to invite you to test out the new installation (though it may be imperceptibly different from the Birder's Guide Installer).

The only steps that may be different will be the additional step of adding the hotspot connection to the Raspberry Pi's WPA-supplicant file, a file that stores the WiFi info it needs to connect to nearby networks.
To add the credentials to the WPA-supplicant.conf file, you can simply do the following:

1. Run the BirdNET-system installation normally (i.e., by just following the Wiki guide)
2. After the installation has completed, turn on your phone's hotspot.
3. With your phone's hotspot "visible," click the small network icon at the top right of the desktop to display the available WiFi networks, selecting your hotspot and entering its corresponding password.

Now, when you power on the Raspberry Pi and it is out of range of your home WiFi, it will hunt for your phone's hotspot and will automatically connect to it so long as it is available. This will allow you to access the BirdNET-system Extractions page using the URL http://raspberrypi.local.

I have tested this several times, but only have access to my one cell phone and can only hope that the process works as easily for you as it has for me.

I will keep you updated and want to thank you again for your involvement and valuable feedback. The next few projects will involve some of what we discussed -- especially getting started on an analogous BirdNET-system for the BirdNET-Lite model.

Best regards, Patrick

PS I haven't had the chance yet to build the demo/live repo you sent, but haven't forgotten about it~~

[DD4WH]

Patrick, thank you so much !

I will be testing this out as soon as I find some more time this weekend and report back to you here!

Thanks again for all your hard work!

Frank DD4WH

[DD4WH]

Hi Patrick,

today I installed the new installer, here come my results and comments:

1.) installed the recommended RPi OS with balenaEtcher to a 16GB microSD card
2.) plugged LAN cable from my local router into the RPi and followed the setup protocol (NEW: update all OS files before runnig the BirdNET system instal batch file)
3.) ran the installation oneline command from your WIKI ("curl etc. bash")
4.) entered my passwords and lat lon into the config file
5.) INSTALLATION ended SUCCESSFULLY! Perfect!
6.) installed NoMachine on the Pi to allow remote desktop access
7.) rebooted the RPi and unplugged LAN cable (WiFi not connected!), so from now on RPi is without internet and local network
8.) opened web browser on the RPi --> no internet connection and no connection possible to birdlog.local
9.) Android phone: is connected to local WiFi; switched on HOTSPOT
10.) connected RPi to HOTSPOT and typed in hotspot password on the RPi
11.) RPi connects to HOTSPOT
12.) Android smartphone: impossible to connect to raspberrypi.local --> possible reason for this is the unavailibility of mDNS service on Android browsers!???
13.) open browser on the RPi and opening birdlog.local --> works !
14.) Assessing the RPi from the Android phone via NoMachine --> works! and I can see the log pages via the RPi browser
15.) steps 9 to 14 even work, if the Android phone is NOT connected to the internet via local WiFi, but only uses a mobile LTE connection to the internet (like in real field conditions)

So, in summary, it works!!!

• I can bring out the BirdNET-system RPi into the field without any internet connection.
• when I go to the field, I can connect to the RPi via Android phone hotspot and view the RPi browser via NoMachine (provided I have internet connection with my smartphone!)

A little bit complicated, but the basic functionality is there!

Thanks, Patrick for making this available! My impression is that the major obstacle is the Android smartphone not connecting to mDNS services . . but I do not know anything about these networking questions, so this is -at most- sophisticated guessing . . .

Keep up the good work!

Best regards,

Frank

[mcguirepr89]

Frank~~ Thank you so much for your testing and for the detailed feedback -- it is invaluable to BirdNET-system!

I had a feeling there would be some weirdness with the Android vs iPhone results, but I am very glad that you can now take your BirdNET-system into the field and get connected.

On that subject, I reached out to NoMachine to inquire about their licensing for a project like this one. They are interested in learning more before requiring a .edu license, (but since I'm unemployed and have no degree, maybe they'll understand I am not affiliated with any .edu!) I'm scheduling a web meeting with their VP of Sales to discuss the project further in hopes that I will be allowed to integrate the NoMachine installation into the main BirdNET-system installation script. Wish me luck in the meeting and keep your fingers crossed that I present an appealing enough use-case for the NoMachine folks to feel generous and intrigued :)

My best,
Patrick

[mcguirepr89]

Hello, again
I wanted to share a small victory with you -- I spoke with the North American VP of Sales for NoMachine and explained the project in detail. While NoMachine does not and cannot officially endorse NoMachine's use in the BirdNET-system, I was assured that it would be permissible to include NoMachine into the base installation so long as I make clear to all users that its use is subject to the NoMachine free license. This will allow entities that wish to use the BirdNET-system outside of the bounds of personal use to reach out to NoMachine on their own to discuss .edu licensing. This same relationship exists between BirdNET-system and Anaconda, so the onus will be on users to stay within the bounds of personal use -- likewise, the onus will be on the user to pursue other license types for their use-case should it no longer qualify as personal use.
Soooooooooooooooo, one of the next features will be having NoMachine baked into the installation :)

Thanks again for the great suggestions --
My best,

[mcguirepr89]

Also, I found an issue with the new installer that created an erroneous variable which the birdnet_recording.service and livestream.service use. I believe this means you would want to re-install BirdNET-system to fix this. Sorry for the inconvenience, but thank you so much for your testing and feedback as always. I will be sure to let you know when there is a new installer to test which will incorporate NoMachine.

[mcguirepr89]

Hello again --
Update -- the rpitesting branch has been pulled into the main BirdNET-system-for-raspi4 branch, so NoMachine is officially included in the installer.
Please feel free to test it out if you have the will and time to do so.
My best