TensorFlow Logo

Creating a Scavenger Hunt on Android With TensorFlow

While going through my files sorting them to find anything that may be of benefit to future Toads, I came across a blog post I had started writing early in my tenure at Metal Toad.  Many of the thoughts have stayed the same since, and some things have gotten easier over the years. For example, Google has been trying to make using the camera easier on Android with their Jetpack libraries.

Here is the unedited version of the post:

Prior to Google IO 2017, I would have expected creating an image classifier that functions on an Android device to either not work, be terribly slow, or be completely inaccurate.  Fortunately, with the current Machine Learning offerings from Google, including TensorFlow, none of these are true assuming you take some care into putting everything together.

For the winter hackathon, my group created a scavenger hunt prototype using TensorFlow for the image classification.  The main objective of the prototype was to be able to have a preloaded model, a list of clues, and a list of answers and be able to play through the game taking pictures of each item and having the device verify if the picture satisfies the clue.

We learned quickly that utilizing TensorFlow to create the classifier was very straightforward once we had all of the images to train with because the tools presented in the TensorFlow for Poets tutorials were able to abstract out a lot of the details for this case.

TensorFlow Lessons Learned

  • Image resources weren’t good sources, half of them were coming up as html with .jpeg extensions

  • Talk about creating the model and how in this case it isn’t different from TensorFlow for Poets

  • Talk about the camera

    • Differences between Camera and Camera2, differences between devices and the decision to go with CameraKit for the sake of time

  • Talk about making sure the image was the same dimensions as the model was expecting

Pitfalls of Working with TensorFlow

  • Data needs to be clean and have enough of it in the right categories

  • Camera usage on Android is tedious without a library like CameraKit

  • Incoming data must be in the same form as our model was trained to recognize from

  • More data for “random” category to reduce the likelihood of false positives in other categories in addition to generally needing more data for each category

Date posted: March 10, 2020

Add new comment

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • You can enable syntax highlighting of source code with the following tags: <code>, <blockcode>, <cpp>, <java>, <php>. The supported tag styles are: <foo>, [foo].
  • Web page addresses and email addresses turn into links automatically.
  • Lines and paragraphs break automatically.

Metal Toad is an Advanced AWS Consulting Partner. Learn more about our AWS Managed Services

Have questions?