{"id":540,"date":"2023-03-14T07:08:16","date_gmt":"2023-03-14T07:08:16","guid":{"rendered":"http:\/\/kiphaynes.com\/?p=540"},"modified":"2025-08-28T15:33:32","modified_gmt":"2025-08-28T15:33:32","slug":"gesture-controlled-smart-device-control-using-ar-glassware-to-control-smart-devices-in-the-home","status":"publish","type":"post","link":"https:\/\/kiphaynes.com\/?p=540","title":{"rendered":"Gesture Controlled Smart Device Control: Using AR Glassware to Control Smart Devices in the Home"},"content":{"rendered":"\n

I created this little demo with a Hololens 2, MR Toolkit and Unity when I first bought a house here in Washington. I had been doing some POC work in AR for Lowe\u2019s Innovation Labs<\/a>, and had an interest in combining AR and Smart Home\/IOT Devices. I made something like this back in Sweden with the IKEA Tradfri system<\/a>, but wanted to use the latest MR Toolkit at that time.<\/p>\n

\"\"
Imagine a world where data is embedded in every object around you, and it can be easily viewed in your AR Glassware. In the current paradigm, that data has to be requested from a central cloud or edge based repository. Localization is, in a sense, part of that monolithic paradigm as well. A space must be pre-mapped or filled with known anchors in order to overlay spatial data at scale. It\u2019s a problem that we have been working on at the Labs for some time. This is especially challenging in a GPS denied environment. Now imagine that physical objects are aware of their location (even if they are moved around) and serve as dynamic spatial anchors as well as sources of local information. Things like product state, pricing, history, health, battery status, etc. Imagine going into a retail space where the products push or offer their localized data to the AR viewer.
This is a simple working prototype of what that world could look like. In this POC, all of the data sources are local and the anchors located at the light locations had to be set up manually.
I\u2019ve always liked the idea of controlling things with voice or gesture, and gathering data just by looking over at the object.<\/p>\n

\nhttps:\/\/kiphaynes.com\/wp-content\/uploads\/HoloLens-Lights.mp4<\/a><\/video><\/div>\n


Now we have the magic leap 2, with even better anchors and spatial mapping, and the Apple AR glassware is just around the corner. With new IOT standards like Matter<\/a> and the growing popularity of open source home control like Home Assistant<\/a>, it\u2019s an exciting time for the Smart Home.<\/p>\n

\"\"
Stable Diffusion<\/figcaption><\/figure>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

I created this little demo with a Hololens 2, MR Toolkit and Unity when I first bought a house here in Washington. I had been doing some POC work in AR for Lowe\u2019s Innovation Labs, and had an interest in combining AR and Smart Home\/IOT Devices. I made something like this back in Sweden with<\/p>\n

…<\/span>Read More<\/a><\/p>\n","protected":false},"author":2,"featured_media":564,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"class_list":["post-540","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology","xf__entry"],"_links":{"self":[{"href":"https:\/\/kiphaynes.com\/index.php?rest_route=\/wp\/v2\/posts\/540","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kiphaynes.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kiphaynes.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kiphaynes.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/kiphaynes.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=540"}],"version-history":[{"count":5,"href":"https:\/\/kiphaynes.com\/index.php?rest_route=\/wp\/v2\/posts\/540\/revisions"}],"predecessor-version":[{"id":616,"href":"https:\/\/kiphaynes.com\/index.php?rest_route=\/wp\/v2\/posts\/540\/revisions\/616"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kiphaynes.com\/index.php?rest_route=\/wp\/v2\/media\/564"}],"wp:attachment":[{"href":"https:\/\/kiphaynes.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=540"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kiphaynes.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=540"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kiphaynes.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=540"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}