Location Recognition Using A Mobile Phone Camera

Project Description

While publishing a joint article with Prof. Kelvin Sung regarding Mobile Technology and Museum Visitation, I investigated how museums nowadays are utilizing mobile technologies to attract visitors and enhance their museum experiences. Many museums are adopting mobile technologies to help visitors "wayfind" within the museum, however, current indoor location-aware solutions, such as WiFi-triangulation and Bluetooth, are unable to pinpoint the location of the visitor with acceptable accuracy, or require large infrastructure investments which makes such solutions unpractical for museums. Towards this end, I was motivated to research and design a practical system which is able to recognize the location of museum visitors using photos captured with a mobile phone. The outcome of this project provides a framework which can be extended for further study and eventually be utilized in the Pacific Science Center museum game project led by Prof. Kelvin Sung.

During this project, I implemented an experimental framework for pinpointing indoor location by recognizing objects in photos. The front end UI was built on the Microsoft Windows Phone 7.1 platform, and the back end utilizes Windows Azure cloud services for data storage and processing. The image recognition code was implemented with the EmguCV library which is a .Net wrapper for the OpenCV library.

To understand how to build this framework, I read several academic papers which outlined the use of computer vision techniques for indoor localization and were primarily tested in the museum environment. As a result, I chose to apply the Speeded Up Robust Features (SURF) algorithm in my project after comparing the pros and cons of the SURF algorithm with others such as Scale Invariant Feature Transform (SIFT) and the Viola-Jones object detection framework.

Since I had no prior experience with mobile and cloud application development, this project gave me an in depth understanding of the mobile and cloud development process, and allowed me to gain hands on experience designing an entire system from end-to-end (including research, project planning, analysis, requirements, design, implementation, testing, and deployment). I also learned about novel techniques for performing image recognition. In addition, I made several contributions to computer vision forums by fixing tools and answering questions on how to migrate EmguCV based applications to the cloud.

Although the framework of the solution is completed and can be used to perform location recognition, several improvements can be considered for future work:

  • Extend the solution so it can select the appropriate detection parameters based on the type of object being detected. While the solution is currently optimized for detecting objects such as paintings, more experimentation needs to be performed to determine the parameters for detecting other types of objects.
  • Adding a web-based UI for managing model images. Currently the system allows the addition and deletion of model images through the mobile phone client UI, however, this is still cumbersome for managing a large number of model images.
  • Supplementing the solution with other location-aware technologies to reduce the set of candidate images to compare against. Currently the solution performs comparison against all model images which linearly increases the detection time as more model images are added.

Project Overview Diagram

Model upload process

Image detection process

Project Screenshots

Project screenshot array

References

  1. Tsai H and Sung K. Mobile Applications and Museum Visitation. IEEE Computer 2012;45(4):95-8.
  2. Aizawa K, Yamasaki T, Kawaji H and Kawamura S. Location identification for visitor behavior log in museum. In Proceedings of the 9th ACM SIGGRAPH Conference on Virtual-Reality Continuum and its Applications in Industry. ACM, New York, NY, USA, 2010.
  3. Bay H, Ess A, Tuytelaars T, Van Gool L. Speeded-Up Robust Features (SURF). Computer Vision Image Understanding 2008 JUN;110(3):346-59.
  4. Fasel B and Van Gool L. Interactive museum guide: accurate retrieval of object descriptions. In Proceedings of the 4th international conference on Adaptive multimedia retrieval: user, context, and feedback. Springer-Verlag, Berlin, Heidelberg, 2007.
  5. Feryanto A and Supriana I. Location recognition using detected objects in an image. In Electrical Engineering and Informatics (ICEEI), 2011 International Conference on. 2011.
  6. Lowe DG. Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision 2004;60(2):91-110.
  7. Ruf B, Kokiopoulou E, Detyniecki M. Mobile Museum Guide Based on Fast SIFT Recognition. 2010;5811:170-83.
  8. Viola P and Jones M. Rapid object detection using a boosted cascade of simple features. In Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on. 2001.
  9. Werner M, Kessel M and Marouane C. Indoor positioning using smartphone camera. In Indoor Positioning and Indoor Navigation (IPIN), 2011 International Conference on. 2011.

Back to top

Project Info

Henry Tsai

Student
Min-Han Henry Tsai

Faculty Advisor
Kelvin Sung