Smart retailing – the brick and mortar response to today’s online challenges

Smart retailing has become firmly established as a reality rather than a concept. It reflects how retailers are working to deal with the disruptive forces the $15 trillion sector now faces; an upheaval that a joint Intel Labs/The Store WPP report refers to as ‘The second era of digital retail ’.

Below, we look at these present challenges for retailers, and how they can use IoT technology – smart devices, dense arrays of sensors, massive yet affordable computing resources and sophisticated analytical capabilities – as part of their response. We take a closer look at the technologies underlying grocery operations like Amazon Go, then contrast these with the strategies now appearing in two other important retail sectors – clothing and furnishing.

A perfect storm in retail

The ‘Second era’ report describes how retail is facing a perfect storm of change fuelled by a wide set of powerful technological, social, demographic, ecosystem, business, and economic forces. Customers now have a new set of expectations related to omni-channel (seamless virtual + bricks-and-mortar ) shopping, customisation and personal experience, efficiency, transparency, and the quality of the experience itself. They also enjoy increased choice as retailers battle for their custom in increasingly competitive environments.

Manufacturers equally expect more from their retail outlets; increased visibility and new services including shopper analytics, targeted advertising, and other analytics and insights. Additionally, the shift towards online sales has changed the retail landscape forever and requires totally new thinking; this change will continue to disrupt retail as increased delivery speeds undermine immediacy as a channel advantage for traditional retail.

Against this background there is also the combined challenge and opportunity created by Moore’s Law, which is delivering ever-more computer capability as costs, sizes and energy barriers diminish sufficiently to disrupt retail strategy. The implication is that ultimately any object can be made both smart and connected as these barriers disappear. These properties translate into commercial advantages if the data made available from the growing hordes of smart devices can be successfully captured by powerful computers and converted to information that provides actionable insights.

According to the Intel report, while the first era of retail digitisation was about supply chain management, inventory and payment systems, the second era will be shaped by sensors, data analytics, robotics, natural interfaces, and computing ubiquity. The drive is to improve the shopping experience by making it more personal, efficient and fun for consumers, while continuously improving retail efficiency, enabling new business models, maximising revenue, and speeding fulfilment and delivery.

Inside the smart store

If smartness is about gathering data and communication with customers and suppliers, then the store’s shelving and its potential for intelligence plays a core role. Make a shelf smart, and it will revolutionize the level of service retailers are able to offer to manufacturers and shoppers alike. It will interact with the shopper in a way that is natural, comfortable, and fully respectful of shopper privacy. Shelves will understand natural human language, context, and even sense emotional states. They will serve shoppers intelligently by assessing whether they are stressed, relaxed, in a hurry, confused, in discovery, or close to making a purchase decision.

Shelves can serve particular promotions as shoppers is close to making a purchase descision

Fig1: Shelves can serve particular promotions as shoppers are close to making a purchase descision.

Like any good sales person, the shelf will have a personality that combines deep product knowledge, trustworthiness, great shopper insight and strong selling skills. It will navigate a wide range of conversations, make choosing easier for the shopper and move them towards purchase. Smart shelves will also handle loss prevention, and manage samples, inventory and assets. To unlock maximum value, smart shelves will need to be supported by a sophisticated back-end server infrastructure able to gather, store, and analyse data, and deliver media and other services to the shelf.

According to the Intel report, shelving could rationalise into three types; ‘Good’, ‘Better’ and ‘Best’. Good shelves will have basic sensing and limited display capabilities, and no communications. Better shelves will have more sophisticated sensors, more local intelligence, and better cloud interaction. Proximity sensors will be replaced by capabilities to see, smell, feel, understand and intuit the world around them. They will detect the contents they hold using cameras, RFID readers, weight sensors or other technologies. At the same time, they will interact with the shopper in front of them using 3D cameras, microphones, proximity and touch detectors, together with local computing resources that will obviate the need for cloud processing with its privacy concerns.

A practical example of a sensor-rich smart shelf design appears below, under ‘Highly-integrated smart shelves’.

The best shelves will add further cloud-based resources to these capabilities, to deliver product information, social media reviews, discounts, and personalized shopping guidance for each shopper. Dynamic or personalised pricing, and promotional offers, will also be possible. Some shelves may use the shopper’s smartphone – the display, touch screen, microphone and even processor – as a part of the interaction. Others will rely on their own hardware. This could include displays, from OLED or LED to high quality video or ultimately holographic devices. However, display deployment will have to be carefully considered, as too many bright screens in a limited area may become overwhelming.

Conversely, easy-to-use, fun interfaces – possibly including touch, gesture, augmented or virtual reality – could make two-way conversations and exercises in customisation highly attractive to shoppers.

Fig2: Easy to use interfaces could be highly customised and attractive to shoppers.

Smart shelves with sufficient computing power will also be able to engage in natural conversations, responding, for example, to a shopper’s question about where to find a product. The shelf could then make further suggestions of possible interest to the customer, based on its knowledge of the customer’s purchasing history.

Manufacturers will benefit from insights into traffic, linger times, customer demographics, and success of offers or advertising. Shelf sensors, and cameras using machine vision, could constantly report on inventory levels, and arrange for replenishments to be sent when products are running low. The sensors could also spot shoplifting attempts. Customer behaviour gathered by shelf sensors and store movement trackers, and sent back to a cloud analytics resource, will need to be integrated with data gathered relating to online behaviour – clicks, hovers, shopping carts and wish lists.

An innovative technology called NeWave Smart Shelf allows retailers to continuously monitor shelf stock levels without needing RFID tags on individual items. The tags are located instead on the shelf’s product pusher, and become visible when the item is removed. The system can also raise an alarm if too many items are removed simultaneously, signalling a theft in progress. This can be backed up by video capture of the activity.

Perishable foods can be protected by passive UHF temperature-sensing RFID inlays available as low-cost alternatives to active RFID tags or data loggers. These SMARTRAC devices’ on-chip temperature-sensing circuit can digitise a product’s temperature reading into a 12-bit number which can be read by a UHF reader, along with the tag’s unique identifier. Basic moisture-sensing capabilities are also provided, based on measurement of impedance changes.

Highly-integrated smart shelves

Above, we have highlighted different benefits that can be gained from smart shelf technologies. Yet many retail operators will be interested in sourcing integrated solutions comprising components that co-operate to provide a better shopper experience while gathering and analysing retail data for management and improvement.

Packaged in-store analytics solutions are available from companies like Hybris Labs. Their ‘Funky Retail’ solution can identify customer presence, count the product lift-ups, measure the lift-up time, and relate individual product lift-ups to a promotional product video.

However, smart shelf solutions that make more intensive use of multiple-type sensor arrays as envisioned by Intel are in the pipeline; these will offer more detailed, real-time information to both shoppers and the retailer. One example is a United States Patent Application, titled ‘Smart shelves for retail industry’ filed by inventors from IBM in February 2016 and published in August 2017.

Each shelf in this system has a mesh arrangement of sensors that includes strain sensors, photo-detectors, microphones and spillage detectors, together with a data processing system for handling the sensor signals. The sensor mesh layer is fitted into the bottom of each shelf. The system also includes a set of video displays for showing characteristics of the products being sold from the shelf; these characteristics are delivered by a set of wireless transmitters.

The storage devices can be of any type. The user input devices can be any mix of keyboard, mouse, keypad, and/or devices for image capture, motion sensing, smell detection, light detection, microphone, or fused devices containing more than one of these functions. Other devices can also be used. Multiple shelves containing different products can be integrated into one system; each shelf’s data processing unit can interface with a central store server which controls various store systems such as personal scheduling, personal information, lighting, security item monitoring and stock control.

The central server can then provide access information such as pager, SMS or email addresses for employees so that they can be contacted and updated about low stock or other situations.

The video displays can show product characteristics such as price, weight, chemical freshness determined by colour or methane emission, nutritional values, calories, recipes, expiration dates and other information as required. They can also show promotions of related products of possible interest.

Strain sensors within the mesh can provide a voltage signal proportional to the weight, and therefore the number of products placed on to the shelf. A ‘restock alert’ signal can be generated if the weight drops below a preset critical value. Photo-detectors can have filters to indicate that a certain item with a specific colour is on the shelf. If the item shows a slow change in colour – for example a banana turning from yellow to brownish, or milk coagulating in a bottle and changing colour – the photodetector’s changing voltage will alert the retailer accordingly.

Microphones can monitor for sounds that indicate when a container expands due to its contents being compromised or handled improperly on the shelves. These microphones can be implemented using strips of piezoelectric material that generate a signal on detection of small vibrations. Overall, the various sensors can be integrated into one sheet comprising one layer for a strain detector, the next for a photo detector, another for an acoustic sensor, and others as required.

The strain sensors can include a patterned foil laminated to the bottom of the smart shelf. Circuits can be printed onto the foil using silicon, germanium and/or other materials that make it sensitive to different phenomena. For example, chemical sensing can be achieved by printing tin oxide on top of a transistor, because the current flowing through the circuit will increase as methane levels rise. Similarly, a light detector can be created at the junction of two dissimilar materials such as silicon and germanium. This can detect light in a spectral band associated with merchandise packaging colour.

The sensor layers can be perpendicular strips forming a cross bar structure with arbitrary orientation. Multiple discrete elements can be included to allow items to be placed haphazardly on the shelf, if this arrangement is more desirable than a regular ‘row and column’ arrangement.

Data can be sent from the radio transmitters using Bluetooth, spread spectrum radio, mesh radio, ZigBee, Global Systems for Mobile Communications (GSM), Code Division Multiple Access (CDMA), General Packet Radio Service (GPRS), Wideband Code Division Multiple Access (WCDMA), Enhanced Data Rates for GSM Evolution (EDGE) (also known as Enhanced GPRS or EGPRS), CDMA 2000 or other wired, wireless or hybrid standards.

This is a possible solution from one vendor, but retailers may find themselves using multiple vendors across their operation. Accordingly, standards for data analytics must emerge; the industry will need to define standard interfaces and a set of open APIs that enable developers to collaborate with each other across standardised platforms. For example, content delivery will benefit from standard screen sizes, formats and resolutions for on-shelf advertising.

Customer location detection

While smart shelves, and the processing power they may be fronting, are key to the emerging ‘second era of retailing’, they are complemented by another important data source; location-based services, with sufficient precision to be applied within the confines of a store. Such data can be analysed to understand shopper footstreams to gain increased insight into shopper behaviour, store layout and user experience.

Location tracking is either passive, when the customer is merely carrying their smart device but not using it, or active, when they are using it to obtain information or service based on their location. Different techniques, of various levels of accuracy, refinement and development state, can be summarised as below:

Wi-Fi triangulation: Already in deployment, but of low accuracy – about 30m.
Wi-Fi fingerprint: A more sophisticated version of Wi-Fi triangulation that uses learning algorithms to map a store’s wi-fi profile. Early trials indicate a precision of 2 – 5m.
Bluetooth LE-based beacons: Beacons such as Apple’s iBeacon technology can trigger offers to a shopper’s device when they are within range of an iBeacon transmitter. The range of the transmitter can be adjusted to cover a small area (5m radius) or the entire store.
Accelerometer and inertia: A smartphone’s accelerometer can be used with limited accuracy and success; currently it is only viable when used to augment other location techniques.
Semantic location: This uses signal processing on Wi-Fi signals over time to help refine location when it is unclear which side of a wall a shopper is located.
Ambient audio: Different stores have different ambient noise signatures. This can be exploited to help with other, inconclusive location information to make a final determination of position.
Active audio: Some stores are experimenting with adding audio signatures to their in-store piped music to help devices understand where they are.
Other approaches include visual triangulation, visual fingerprint (similar to Wi-Fi fingerprint), and magnetic field, which uses a smartphone’s digital compass to detect magnetic fields present within stores.
Custom designs, such as the approach used in the Amazon Go store – see below.

These techniques are expected to evolve over time. The best accuracy is achievable by combining several of these approaches together. Traffic flow analysis needs roughly 2m accuracy, enough to assess which aisle a shopper is standing in. And a customer-facing store guide may require 1m accuracy to be truly valuable to the shopper and guide them right to the product they are looking for.

Amazon Go store

Amazon has recently opened their first Amazon Go store, where shoppers can select the items they want and then leave without having to go through a checkout. Instead, the store uses a mix of computer vision, deep learning algorithms and sensor fusion (where data from several different sensors are "fused" to compute something more than could be determined by any one sensor alone ) to identify a person and their purchases.

Customers must scan an app to enter the store, after which everything they take is logged via camera and shelf sensors and placed in a virtual cart. The system then bills the customer's Amazon account when they leave. Techniques used in fulfilling this concept include:

Customers check in by scanning their smartphone’s Amazon Go app’s QR code
The store tracks them with dozens of ceiling-suspended sensors
A combination of video feeds with image analysis and laser arrays is used to identify people and items in the store. The technology is like that used in self-driving cars.
Data from these sensors and video feeds is aggregated and combined with machine learning, resulting in a package that Amazon calls ‘Just walk out technology’.
If a shopper picks up a carton of milk, the technology adds it to their virtual cart. It also removes it if it is returned to the shelf.
Amazon charges their account when they leave the store.
One shopper tested the technology by turning off his phone, taking items and returning them to the wrong place. The app still tallied his items correctly.

Clothing stores and magic mirrors

Since e-commerce began threatening stores last decade, retailers have been trying to make their locations operate more like the web. One place where improvements can be made is the fitting room, as shoppers who use this are seven times more likely to make a purchase than those who simply browse the sales floor, according to research by Alert Tech .

As one response, Oak Labs, a startup founded in 2015 by former eBay executives, has developed a fitting room mirror that offers an interactive experience. A woman enters with jeans and a blouse. Sensors read the radio-frequency ID tags on the clothes and display the items on a touchscreen embedded behind the glass. A recommendation engine—like those ubiquitous online ones—suggests complementary pieces such as shoes and a belt. The customer can choose a language other than English and adjust the lighting (options might include “dusk” and “club”). If an item doesn’t fit or the colour isn’t right, she taps the mirror, which triggers a request on store clerks’ mobile devices.

The furniture industry

While furniture can be ordered online just as easily as food or clothes, it’s not nearly as easy to return if it doesn’t look right in its chosen position. Yet, according to IBISWorld, 15% of the $70 billion US furniture market has moved online. An article in Forbes describes how the industry is achieving this through augmented reality, 3D rendering and computer vision tools that let customers see how a piece looks in a room.

Augmented reality allows customers to virtually ‘try out’ furniture. For example, Pottery Barn’s 3D Room View app for iOS allows customers to instantly stage new furniture items at home using their iPhone or iPad. Customers can see an augmented reality view of their room and drop in full sets of furniture for consideration. Companies like Wayfair, IKEA and Houzz have also implemented augmented reality solutions for furniture “try-on,” but Williams-Sonoma Inc (WSI) – which owns Pottery Barn and other stores – has even bigger plans for the technology with the acquisition of Outward, an augmented reality startup . One plan involves adding more assets for consumers to virtually stage and enabling customers to “try out” multiple furniture brands together.

WSI has employed Outward to generate photorealistic renderings of their products, which they used to replace some of the photography across the WSI brands. Outward also provided 3D renderings that enabled 360-degree views of WSI products, so customers can see the furnishings from all sides.

3D furniture ‘try-ons’ are also available from Modsy , another visualisation company. Their app allows users to take a few smartphone pictures of their room, including clutter, and receive a 3D model of the room, rendered as empty. Styling tools then allow viewing of multiple layout options and different products. Human Style Advisors can provide help if needed.

Conclusions

Retailers are facing over-capacity in their sector and fierce competition from online shopping channels. Smart retailing is a set of hardware and software technologies that allow retailers to fight back, by offering shoppers better experiences and by revealing deeper insights into their daily operations and how they can be improved.

In this article we have seen in general how smart retailing is being implemented in terms of smart shelf and shopper location. We have then reviewed more specific examples of how smart retail is currently being used – in the Amazon Go shop, a clothing store and a furniture retail application.