Support efficient large Wildlife Recorder imports

Issue #462 resolved
Adam Winer created an issue

One issue with very large Wildlife Recorder (and other format) imports is that they need geocoding en masse.

Simply scrolling through 1000 locations is unbearable; and even worse, it’s expensive for me - using the Google geocoding API for 1000 locations costs $5, and that’s a lot of cost to pay for a single user (who might repeatedly import, which is getting worse!)

Idea: for large imports (say, anything with > 10 locations, and where those locations all have place names - true for Wildlife Recorder, but not for Birdlasser), use eBird geocoding to:

  • Find a state
  • Download checklists from the state from eBird
  • Use those checklists to compute some sort of polygon shape (convex hull?!? though that might go very poorly for some states)
  • Find all points that line within that polygon, and map them all to the state automatically
  • Proceed to the first point that wasn’t mapped, and repeat.

Some implementation thoughts:

  • This doesn’t get US counties, but not sure I particularly care for very large sets of locations. Alternatively, I could do counties for 10 locations, and states for 100 locations, or something like that
  • Also, I imagine there’s some things which are states but have very very few hotspots in eBird; at some point, falling back to the country is probably a good thing.

Comments (5)

  1. Adam Winer reporter

    Or, second idea, find an open-source geographic boundaries dataset, and use that to do a first pass for geocoding, then look at eBird?

  2. Adam Winer reporter

    OK, looks like https://geotools.org/ has code that can handle this; here’s a stackexchange code sample:

    https://gis.stackexchange.com/questions/182447/reverse-geocode-latitude-longitude-with-geotools-using-natural-earth-shapefiles

    The ArcGIS stuff requires a subscription ($100 a year the lowest) so that’s out. This one is public domain and at least gets countries (and US states, if desired):

    https://www.naturalearthdata.com/downloads/50m-cultural-vectors/

  3. Adam Winer reporter

    Implemented! Doesn’t help if you have a zillion locations without lat/long, but a dramatic improvement for those with.

  4. Log in to comment