Nov 06, 2018 Development Reading Time 10 minutes

Augmented Reality with Sitecore Commerce

How It Works


After pointing iPhone’s camera to the real product, application detects the object by comparing it with 3D scan, stored in Sitecore and downloaded to the device. Then the app displays product’s information, loaded from Sitecore in augmented reality, in a bubble above the recognized object.

3D scans can be created with ARKit Scanner app installed on iOS device. The app allows to export the scan to a file (in .arobject format), which later can be uploaded to Sitecore Media Library and assigned to a product from Sitecore Commerce.

Implementation Details

The Solution consists of two parts:

Product API

Product API exposes two endpoints:
1. Get all products in the category, for given category item ID and return paths to object scan files assigned to the products:

Sample result:

To assign the scan file to the product, it should be placed under the /sitecore/media library/AR Maps item, in a folder named same as product ID in Sitecore Commerce.
Value of the root item can be configured in ARCommerce.Feature.Catalog.config file.

2. Get product details for given product id:

Sample response:

Both endpoints use base classes from SXA Storefront libraries and return the results as json, most of the logic can be found in ARCatalogRepository.cs.

Mobile App

Client App is written in Objective-C and uses few 3rd party libraries for the network communication (AFNetworking) and visual components (SVProgressHUD and SCLAlertView).

3D Object detection uses ARWorldTrackingConfiguration and ARReferenceObject classes. But reference objects are not created with files from Assets Catalog, but with the ones downloaded from Sitecore Commerce Product API. Additionally the names of ARReferenceObject objects are updated with Product IDs, which later, helps to find the product for recognized 3D objects.

Products Synchronization

When launched, app will try to download product scans from /api/cxa/ARCatalog/GetProductScans API method and save them to temporary storage in the app sandbox (check details in ProductRepository.m). If the user didn’t configure necessary parameters, it displays the settings pop-up instead. In this pop-up you should enter your website URL and Item Id of the product category, for which you uploaded .arobject files. It’s possible to change the settings, or trigger synchronization with server afterwards, by tapping “Sync Products” or “i” button.

Object Recognition

When you point to the object with device’s camera, app will call delegate method in ViewController.m where it get product ID assigned to the 3D object:

Now inside the method, we are able to call /api/cxa/ARCatalog/ProductInformation from Product API, to download the details and product’s photo. Next we display all the information inside SCNNode class in augmented reality above recognized object:

When user taps on this information bubble, app opens the product in Sitecore Commerce website.

Limitations and Possible Improvements

  • 3D object detection works in 60 frames per second, therefore you need scan files to be placed on the device storage. In implemented application, scan files are stored in Sitecore Media Library and later downloaded to application during sync process.
  • Application can only recognize 3D shapes, not colours, or texts. One improvement, which can be added to the app is detecting the 2D images with ARKit.

Additional notes from Apple article:

  • ARKit looks for areas of clear, stable visual detail when scanning and detecting objects. Detailed, textured objects work better for detection than plain or reflective objects.
  • Object scanning and detection is optimized for objects small enough to fit on a tabletop.
  • An object to be detected must have the same shape as the scanned reference object. Rigid objects work better for detection than soft bodies or items that bend, twist, fold, or otherwise change shape.
  • Detection works best when the lighting conditions for the real-world object to be detected are similar to those in which the original object was scanned. Consistent indoor lighting works best.
References