Up next in 10
How we can enable Mixed Reality in Power Apps || Power Platform Virtual Conference
Nov 6, 2023
Introduction to Mixed Reality in Power Apps. We are going to build an application on how to measure our real-world space, view 3D, and 2D objects. View our object using Mixed Reality in Power Apps. About Speaker: Collaboration Application Analyst focus on SharePoint, Power Platform (Power Apps & Power Automate), and Microsoft Teams based in Singapore. Microsoft MVP in Business Application and #HumansofIT Ambassador. I am curious about how things work, especially when it comes to technology. Learning and connecting were my great companions. I want to help others by leveraging the value of the services offered by Microsoft 365. In any way possible, I try as much as I can to help the tech community as I love doing it. Conference Website: https://www.2020twenty.net/power-platform-virtual-conference/ C# Corner - Community of Software and Data Developers https://www.c-sharpcorner.com #conference #powerplatform #mixedreality #csharpcorner
View Video Transcript
0:00
Thanks everyone for joining here. Hi, thank you so much for coming in today's session
0:08
and I hope you and your family is healthy and well during these unprecedented times
0:13
My name is Mariangela Serpolis, and an application yst here in Singapore
0:18
and also a Microsoft MVP. And also, I'm part of Microsoft Nemours of IT Ambassador
0:25
So I love to read books during my free time and watch movies and anime
0:29
And also you can find me on Twitter, LinkedIn, and my blog by scanning the QR code below on this
0:36
So that's me. So today we are going to explore Mixed Reality and its component in Power Apps
0:44
And then we are going to explore the requirements needed to get started
0:48
And also we are going to create an app and explore some business case scenarios
0:55
that we could apply the Mixed Reality in our application. So mixed reality has become a crucial part of our digital transformation
1:06
So it helps business visualize and interact with digital objects in the real world
1:10
We can easily view the products we order online, take measurements, and easily visualize them from our home or business area instead
1:20
So most of the current mobiles are augmented reality capability. So without the use of HoloLens or other AR wearable capability devices, we can experience
1:31
augmented reality through our cell phones and how we can build an application with mixed
1:37
reality and use our phones to visualize the products and takes measurement. So to do that, Power Platform is a low-code platform so we can easily build application
1:48
workflow automation, visualize data, and creating a bot with the use of Power Apps
1:53
Power Automate, Power Virtual Agents, and Power BI. So we could also easily infuse AI as well in our applications
2:02
As Power Platform has been rapidly growing, so one of the features that has been added
2:07
and still in public preview is mixed reality. So it's still in public preview
2:14
so we recommend not to use it in your production application yet until it reaches to general availability
2:23
So there are four components that are available. View in 3D objects and 2D images. View in MR, measure in MR, and view shapes in MR
2:36
So these four three components, the view in MR to the entry the objects, so you can view 3D content and images in mixed reality
2:46
The view shape, it helps you view shapes by using its width, height and depth properties in the mixed reality
2:54
Measure in MR, so it will take a measurement in the real world without the use of the measure and tape
3:00
And also view in 3D, it is a viewing a 3D content
3:05
So one of the business case scenarios is when I click this Mixed Reality button
3:12
This is using the Measure NMR component. So it will open the live feed camera to take the measurement of these two glass cases as
3:20
I want to change it to a bookshelf. So I just point a device from point A to point B, take the measurement in centimeter and filter
3:27
the available bookshelf from the measurements that I get. So I could view it in 3D objects
3:33
That's another component. And then also you can zoom in and zoom out and then view in MR components to visualize
3:40
the object in the real world. And from there, I can take a photo from it
3:45
So this photo, you can send it to your customer or your supplier for your approval
3:57
So to build this mixed reality and power-ups, so there are some requirements that we need
4:03
to check because it's still in public preview. So you might need to enable the mixed reality feature in each of your Canvas app
4:12
And then you need to have a mixed reality capable device, AR core capable device phones
4:19
And then the supported file type for the 3D objects is .glb file
4:24
And of course, your data source should be course compliant URLs. So you need to check the course compliance that you have, the settings that you have in your business area
4:38
So let's do some demo. So let me share my screen. Okay
4:44
So this one is our simple app. So this is just a simple app
4:49
So please ignore the UI and stuff. So from here in your Power Apps, if you go here in the settings, unfortunately, I can't find the
5:03
mixed reality in the preview features, but I need to check in what's the reason behind
5:12
Probably it's in the JPEG, I'm not sure. But yeah, so usually it will go here in the mixed reality
5:18
it will see the mixed reality mixed reality settings here. And then from there, you will see the mixed reality
5:30
under here in the insert category. So you will have here in the
5:36
so you will have the view in MR, view shape in MR, measure in MR
5:40
And the view in 3D component is available under the media. So you will have here the view in 3D
5:47
So it's still in preview. Also, you can view it here in the left navigation
5:53
So you can see it here as well, insert. You will see the mixed reality here, view in MR, V-shape, measure
6:01
as well as the view in 3D. Now, to do that simple app, so what I have here is this one
6:12
is a measure in MR because I want to take a measurement using the length and then get that measurement to filter the furniture list
6:27
that I have, which my data source is this one, SharePoint. So I have here the list of all the
6:32
furniture that is available in my store. Let's say I am a furniture store. And then this data here
6:45
I attach the .glb file. So I have here the 3D objects here
6:51
So each of the list items that I have, I have an attachment, which is .glb
6:57
And then, so what I need to do here, so I just insert this measure in MR
7:04
And then the properties, as you will see here. So if you go here in advance, let me select that one
7:13
So the property is, so if you insert this one, you can change the display type to icon, icon plus text or text
7:27
So if you use the icon plus text, you can change the text here
7:31
Let's say take a measurement. Let's say, yeah, I already have used a measurement, take a measurement
7:40
Just a simple one. And then for this demo, I'll just use the icon and then make it smaller
7:51
And then the unit of measurement that I use is centimeters. So you could use feet, inches
7:58
and meters. And then the measurement type, you could use the area, area, volume
8:03
distance, and volume. So for this demo, I'm going to use the distance and because I'm going
8:08
and then I'm going to use the length properties later. And then I set the advanced and under the
8:16
advanced settings I set the on change of this component. So I set a variable. Let me check
8:25
Let me make it a bigger one. Okay, so I set a variable width in centimeter to a value of this
8:34
one. Oh, wait, sorry. Okay. So I set the variable weight in centimeter to the value of first
8:44
measurement measure and measure MR in MR component to the length. So I'm going to get
8:52
that. So when I'm taking a measurement, so I will get the value of the measurement that I get from
9:00
using this component. And then I'm going to save it to a variable set width centimeter
9:07
And then this width in centimeter, I put it here in the label so that it will display to me
9:15
it will display what is the centimeter that I get from using this component. And then so I use the
9:24
the text. So I set it to width in centimeter. And then I have here the browse catalog. The only
9:31
thing that it does is it will navigate it to the second screen. So the second screen that I have
9:39
here is a gallery. And then the gallery, the items here comes from a furniture list, which is the
9:49
SharePoint list that I have here. And then the fields that I have here is I use the price
10:02
description, title, and then the length, width, and centimeter, length, width. I only use length and width because in my furniture list, I have this width and then height
10:19
but I'm going to use it here to filter that out later
10:25
And then, so if I navigate this one, it will open another screen
10:33
So if I select that gallery, it will open a third screen
10:39
So what I have here, this one is a view in 3D component
10:44
So I insert it from here. So I use this one. And then the source for this one, because this is View in 3D
10:52
View in 3D is only a 3D. I need the 3D components for this one
10:57
So the 3D source for this one comes from the first gallery that selected that attachment value
11:06
So this one comes from this one, from this gallery. So if I select this one, the shelf to plane, so if I select this one, there is an attachment value, which is the 3D object of this item
11:31
So this is the source. And then I insert another gallery here so that it will display all the details of the second screen, the gallery one
11:46
So what I have here is the image description and the title
11:51
And then I have here the view in MR. This one is view in MR
12:00
So the properties for this one is, if I go here, okay, so the text is view and MR, and then they advance
12:13
So the source is the first gallery that selected the attachment value
12:18
So the value for the source for this one is from the attached file of the items
12:26
And because I want to take photos or screenshots of the view and MR that I have in Mixed Reality So under onChange there is a properties
12:45
in view and MR photos. So if it's blank or error, view and MR photos
12:52
So if it's empty, if it's blank, if it's not blank, I'm going to collect all the photos that I took
13:03
using this viewNMR component and then collect it during the, and collect it and put it in the photos collection
13:12
And so I'm go, and then I, and then I insert another gallery
13:18
which is this one. And then the items is the photos that I collect from this viewNMR
13:25
So that's how you, how you built this one. So let's say you don't have any 3D
13:32
objects and you want to use the 2D images. So instead of using this, the source one
13:42
under the source, let's say you don't have the attached 3D objects. So you could use the image
13:53
of the item so you could use we are going to use
14:07
I'm going to use first oh sorry I'm going to use gallery one that selected that image
14:17
So instead of the 3D objects, it will display an image into the real world
14:30
So it will look like this. If I switch to this one. So this one is using the
14:40
if I'm going to use the 3D objects, this is the one that I showed earlier. So it will take a
14:45
measurement and then submit it, browse the catalog, so the available measurement that I get
14:54
and then it will view the 3D objects and then instead of and then in the view in MR it will
15:04
display the 3D objects that I shown a while ago in the view in 3D. So this one is a 3D objects
15:11
And then let's say you don't have the 3D objects. So here you could use an image
15:21
So this is the one that I mentioned a while ago. So this one is using an image instead of a 3D
15:27
So even though you don't have a 3D objects, you can still use an images
15:35
Let's say you don't want to have any images. You could use a shape to view
15:41
You could also use the shapes. So let's say if we go here, we are going to insert
15:50
Let's say you don't have. Let me zoom in. So we could place a shape here
16:05
here. So we are going to insert the V-shape place a cube. This property is the same as the
16:19
the measure NMR. So you could choose a text here. So let's say V-shape. And then display type
16:30
you could have the icon or icon plus text, right? Text, so it depends on your preference
16:36
So let's say in my end, I'm going to use the icon plus text because they have the same icon
16:41
So it's kind of a bit confusing. And then, let me try to fix that one
16:52
OK, let me move it a bit. Okay, so the view shapes here
17:00
So under the shape width, height, and depth. So we are going to use the value from our SharePoint list
17:09
This is the disk that, so we are going to use the, under the width
17:17
Wait, width. Oops, sorry. Width, width. Okay, shape depth. So we are going to use this item that we have
17:41
and then under shape height instead of one we are going to use this item
17:53
that height and then width we're going to say use it as well
18:02
So using this properties, the shape, width, height, and width
18:12
So it will display the exact width, height, and depth of this furniture in the real world
18:19
instead of using your images or your 2D object. So if you play this one, this is what it looks like
18:29
If we select this modern shelf, so if we place a cube, it will look like this one
18:41
So instead of 3D objects or images, so you will have a cube instead or a tango
18:52
Now I have another case that I play around or you could also explore more about it This one I used the object detection So I used AI builder object detection
19:06
And then what I have here is if I play this one, if I have a toy, let's say I'm a toy star
19:14
I have a toy star. And then I want to see the exact
19:18
the 3D objects of this image. I mean, this image that I have, I have this toy that I want to see it online because I can't visualize it in a 2D image
19:35
So I want to see if there is an available 3D objects for this one. So what I have here, I use that AI builder
19:43
So let me play this one. So if I select that one
19:48
So let's say I am a customer and I have this image
19:52
like I want to see it in a 3D view. So I want to check if there is an available 3D objects
20:00
for this one. So let's say I have this one and now they have a 3D object
20:06
So I can appreciate this object. Okay. Yeah, the model takes longer
20:11
So just ignore that. Oops. So now I have the 3D object
20:18
Now I can appreciate this object. So let's say, I think I love this one
20:23
because it's so detailed that other than the images that I have
20:29
So now I could use, with this, I will buy this object
20:35
So it will easy for the users or the customers or the business users to visualize the things that you want
20:43
So is it pretty cool? So what I have here actually is
20:47
I have here the, I use this one, the SharePoint list. So I have the SharePoint list that I have here
20:58
And then I attached the 3D object of this toy. And then I use the model, the object detection model
21:12
And then I trained this model until it will reach to the score of 100
21:18
I'm not sure why it's 100, but it's pretty cool that it reads 100% confidence
21:24
And then in my Power Apps, so I just insert the object detection
21:31
which is here in straight AI Builder. So I use the object detector
21:40
Please note that this is a premium connector. So yeah, you need to check your license
21:45
And so the properties for this one, I use the AI model, which is Katana collection
21:55
And then the collection, this is the one, the AI builder that I just trained
22:02
So I'm going to use this model. And then because the properties of the object model
22:10
has a result and then tag ID, so I insert a gallery
22:15
And then in the gallery, the items here is the object detector.result
22:20
So this one is that tag name and then the tag ID
22:25
So I did a tweak here. So I saved the name and the tag ID in my SharePoint list, this one
22:35
And then I insert another gallery here, which this gallery, what it does is it filter the product details
22:45
which is this one, the SharePoint list. And then I filter it out to a gallery that selected the tag ID
22:54
which is this one. And then the tag ID that I have in my SharePoint list
23:01
So you could use any data source that you have, but in my end, I use SharePoint for this one
23:07
And then because the view in 3D component, it is media so attachment it could be a media content or what's the proper term for that one
23:24
as long as it has a media content and the source should if let's say it's a media content or
23:38
attach content, attachment, you always should have the a press, this one, a press, and then the data source. Oh, this one. So you
23:51
should have this one. So that if you have this one, it means the
23:56
view 3d component will run. So that's, that's, so that's why I
24:03
I use the attached value from the SharePoint list. And then, okay, so the source for this view 3D component
24:12
so I use this gallery five, and then first gallery five that selected the attachment dot value
24:20
And then that's how I come up with this simple scenario. So my conclusion is
24:31
So it is easy to build or create an application with mixed reality and power-ups without the use of code
24:44
So it's just only easy breezy. And if you want to learn more
24:50
about building mixed reality and power-ups, So you could try and check this reference
25:01
And also you could connect with me on Twitter and LinkedIn. Thank you so much
#Augmented & Virtual Reality
#Consumer Electronics
#Engineering & Technology
#Virtual Reality Devices
#Virtual Worlds