Vision Model Updates

iNaturalist currently uses vision models in two main places: 1) a private web-based API used by the website and the iNaturalist iOS and Android apps, and 2) within the recently updated Seek app. When Seek 2.0 was released in April, it included a different vision model than we were using on the web. At that time the web-based model was a third-generation model we started using in early 2018. That web-based model was trained with the idea it would be run on servers, and servers can be configured to have far more computing power than a mobile device. As a result that model was far too large to be run on mobile devices.

Early this year, with an updated Seek in mind, we started another training run with two main goals: shrinking the file size of the model, and allowing it to recommend taxonomic ranks other than species (e.g. families, genera, etc.).

Smaller

The mobile version of the model needs to be small in terms of file size to minimize the amount of data app users would need to download. Smaller models can also be used by more devices as they need fewer resources to run (e.g. memory, battery), and can generate results faster, which is important for Seek's real-time camera vision results. These models take a lot of time and money to train, so we also wanted a model that could be simultaneously trained to produce a large web-based version and a smaller version for use in mobile devices.

Unfortunately, shrinking the file size like this slightly decreased model accuracy compared to the larger web-based version (kind of similar to image compression), and we found that was an unavoidable tradeoff. We take this into account when processing the model results, and on average for a similar error rate, the mobile version might recommend a taxon at a higher taxonomic rank than the web-based version. The taxon results we show to users shouldn't be less accurate, but they may be less specific.

More Species Represented

We wanted the model to include more species data, even when some species don't have enough photos to be recognized as species level. There are some species with a small amount of photos that, if we trained on that small set of photos, likely wouldn't have enough information for the model to reliable recognize those species.

Our 2018 model only included taxa at rank species. We set a threshold for number of photos, and species below the threshold were not included. We could still recommend higher taxa by doing some post-processing of results, but the model itself would only assign scores to species. In our latest training run we allowed the photos from species under the threshold to be rolled up into their ancestor taxa until the threshold was reached, and we allowed the model to assign scores to these non-species nodes. This allows more species to be represented in this newer model, sometimes at the genus level mixed up with photos of other species in the genus under our threshold. Now instead of not knowing anything about these species, the model can at least identify the genus or family, etc.

Replaced Web-Based Model

On May 24 we replaced the existing web-based model from 2018 with the web-based version of the 2019 model used by Seek. You may have noticed taxa higher than species starting to appear in vision suggestions, and this is why. That means the iNaturalist iOS and Android apps, and web vision suggestions have all been using this new model for the last few weeks. As of right now we are using the same model everywhere. The model has a web-based version (used by the website and iNaturalist apps) and a mobile version (used by Seek). The two versions are from the same training run, but can produce slightly different results because of compression in the smaller mobile version, and because the web-based version can weight results based on nearby species occurrences.

Evaluating

By adding photos of more species, we hoped to see an increase in accuracy for observations for species less represented in iNaturalist observations. One way we thought to get insight into that is to look at identifications made through vision suggestions that are current or improving (good) vs withdrawn or maverick (maybe less good). Here's a chart of such identifications for flies (Diptera) and spiders (Arachnida). The dotted red line represent with the web-based model changed. The difference isn't glaring, but it does look like for these groups vision-based IDs are sticking more.

The model was also trained more recently, so we hoped to see an increase in accuracy for observations in countries that have added a lot of observations since 2018. Here's a chart comparing the accuracy of the older model to the new model for a sample of random observations in California, Europe, Oceania, and South Africa (we could/should have included more countries with different observation and species densities, but these still show the trend). In this chart, "Right" means the model's best suggestion was consistent with the actual observation taxon (it suggested the right taxon or its ancestor), "Wrong" means the best suggestion was inconsistent with the actual taxon, and "Unable to determine" means the model was not confident enough to make a suggestion.

For observations from California, the models get nearly the exact same number right, with the older model making more specific suggestions. For observation from Europe, again the two models get nearly the same amount right, but this time the new model makes the more specific suggestions. In Oceania, the new model is getting more right and is more specific. And looking in South Africa, the new model gets a lot more right, with nearly double the amount of correct species recommendations (and fewer wrong).

Finally, the new model can now suggest taxa higher than species beyond the suggested common ancestor we've always had. This means the vision suggestions will be a mix of species and higher taxa. Here's a chart of the rank of all vision-based identifications over the last month or so. Nothing really stands out in this view of the data - there are maybe more identifications recently, and you can maybe see more at rank genus (the larger yellow bar).

By removing identifications at rank species, it’s clearer there have been more vision-based identifications of higher taxa since the new model went into effect.

By going even further and remove ranks below order, it's clear that there's been a vast uptick in vision-based identifications at these ranks. It should be noted that on May 31 the Seek app started to allow observations to be submitted to iNaturalist, and as explained above Seek will tend to recommend taxa at higher ranks than the web-based vision of the model, and we can see these higher taxon identifications really pick up after that. Better to have accurate but broad identifications, than inaccurate species identifications.

So that's an update on iNaturalist's vision models, and why some things may have appeared to change recently, like the addition of non-species recommendations. We think these changes are overall for the better, and hopefully you agree. We're interested to hear you thoughts. Did anyone notice the change? Now that you know the model should perform better for less observed species, I'd be interested to hear people's experiences with different taxa, or in different places.

Posted on 14 de junho de 2019, 07:07 PM by pleary