New technologies are poised to revolutionize several economic sectors by changing the way we communicate. Among them are virtual reality and augmented reality, which are vectors of change in several sectors, including construction and health care.
Both have profoundly altered traditional training methods, and both are destined to play an increasingly important role in skills development and improvement. These two technologies are being used for rapid learning of procedures and technical skills by placing people in situations that mimic real life. Thanks to virtual reality and augmented reality, for example, future health professionals can safely simulate real-life situations by dealing with virtual patients. A learner’s knowledge retention can thus be validated prior to working with real patients. What’s more, these technologies have opened up new possibilities for sharing information on the life cycle of a project, such as the completion of a construction site. Because the workforce is highly mobile in this sector, integrating a virtual environment into day-to-day activities, including worksite visits, helps improve information-sharing between managers and on-site workers.
As we saw in Part 1 of this article, virtual reality and augmented reality account for a rapidly evolving industry and a significant growth market with numerous advantages for companies, and Montréal has the assets to successfully stake out a leadership position in this burgeoning market.
Montréal holds multiple advantages in this emerging niche market
Given that high-tech sectors account for significant proportion of Montréal’s economic base, not least in information and communications technologies (ICTs), the city is well positioned to leverage these multiple assets to further drive the ICT sector forward. Montréal:
- Is a video gaming and artificial intelligence hub. Both have the potential to play a role as drivers of economic development in this niche by leveraging Montréal’s existing reputation;
- Has a rich pool of creative talent thanks to the presence of higher learning establishments that offer training programs;
- Offers highly competitive financial incentives, including the Canada Media Fund (CMF), and multiple government initiatives meant to promote the digital shift undertaken by companies;
- Has a venture capital network (e.g., Teralys Capital, Real Ventures, Tandem Expansion Funds, etc.) and well-established incubators (e.g., TandemLaunch, District 3 Innovation Center, Centech, CEIM, etc.);
- Is home to several university and institutional research centres, which are using virtual reality and augmented reality to advance research;
- Benefits from the presence of industrial clusters, including TechnoMontréal, Aéro Montréal, Montréal InVivo, and the Québec Film and Television Council (QFTC).
Definition
- Virtual reality is a totally immersive experience that transports the user into an imaginary world in three dimensions (3D) created by a software program inspired, as much as possible, by reality. It generally requires the use of a headset into which virtual objects (e.g., images, videos) are projected and with which the user can interact in real time.
- Augmented reality consists of enriching the real word with virtual information created by a software program with the objective of intensifying the user’s understanding of and interaction with his or her environment. It usually involves the use of special glasses, a computer, tablet or smart phone to which virtual information has been added to provide further information to the user. In this regard, the sense of vision, in particular, is brought into play.
- Mixed reality, also known as hybrid reality, is an environment in which objects from real and virtual worlds can co-exist and interact. A mixed reality device is capable of detecting the user environment in order to project interactive virtual information into it. Mixed reality requires the use of a device, such as glasses with transparent lenses, to visualize virtual objects projected in the form of holograms.
This article – the logical follow-up to Part 1 on the subject – presents the other innovative high-potential technologies that promise to further promote Montréal on the national and international stages. While the focus, in Part 1, was on the power of virtual reality and augmented reality to attract investment, Part 2 outlines recent technological advances in research and industry, thus concluding this two-part article on virtual reality and augmented reality.
VIRTUAL REALITY AND AUGMENTED REALITY: ATTRACTIVE INDUSTRIES FOR INVESTORS
Virtual reality and augmented reality have become increasingly attractive for private investment, particularly in the form of venture capital and corporate investment. In 2016, global investments in this market totalled $1.8 billion, a three-fold increase over 2015, when a total of $594 million was invested in this sector. This represents an annual growth rate of 204% between 2015 and 2016. From 2011 to 2016, global investments in virtual reality and augmented reality grew by an average of 71.5% per year.
This investment trend should be maintained in years to come, with a global market value that’s forecast to double by 2017 (to reach $14 billion) and further accelerate starting in 2020 (to reach $162 billion), according to figures cited in the previous article. According to recent projections by Digi-Capital, global investments in virtual reality and augmented reality will reach a record $2.5 billion dollars in 2017. These statistics reflect the growing interest in these new technologies, at a time when the market is still is its infancy.
MONTRÉAL-BASED INNOVATIONS BRINGING PROFOUND CHANGES TO SEVERAL SECTORS
The rapid growth of virtual reality and augmented reality is fueling creativity and innovation in several connected areas of Montréal’s economy, from the development of services to specialized equipment. The solutions thus put forward are helping solve a variety of concrete problems, including in continuing professional development and the digitization of factories. Montréal-based companies and researchers are earning distinction in such varied sectors as health, manufacturing, leading-edge scientific research, and engineering.
Amin Hammad | Construction 2.0: Improving collaboration and safety using mixed reality and augmented reality
Amin Hammad, a professor at Concordia University and researcher at the Concordia Institute for Information Systems Engineering (CIISE), is developing new concepts aimed at integrating digital technology into business practices employed by construction companies. Professor Hammad and his team are using mixed reality and augmented reality to create integrated virtual environments that facilitate interactions between specialists and on-site workers and ensure workplace safety during construction projects.
Mixed reality used for remote interactive collaboration in facilities management
In the construction-related related fields of public works and infrastructure management, for example, an intensive exhange of information is required to complete projects. Building Information Modeling (BIM) is among the automated maintenance management solutions favoured to date. However, this database presents major challenges for on-site users, as it requires powerful computers and the presence of seasoned specialists. Retrieving information manually can be difficult and time-consuming for on-site workers.
In order to facilitate real-time interaction between on-site inspectors and more experienced office managers, Professor Hammad has developed a collaborative mixed reality (MR) approach meant to improve collaboration and visual communication. In this specific instance, MR display technology improves visual perception of the facility by superimposing 3D virtual objects with textual information above the real building objects viewed.
Professor Hammad’s technology integrates information on facilities inspected with complementary sensory data. Two modes of interaction are used to ensure visual communication between the two employees:
- The on-site inspector uses a tablet or smartphone with augmented reality to receive instructions from the manager;
- The manager uses an augmented reality application installed on an office computer and a virtual reality headset to communicate with the on-site worker.
Depending on the time (for example, based on an inpection or maintenance calendar) and the location of on-site personnel, the manager can remotely view inspection and maintenance operations in real time. This helps to ensure informed decision-making on the part of inspectors as it pertains to existing or potential problems. The manager is thus able to see exactly what the on-site inspectors see.
To study the applicability of the method proposed, a system prototype was developed and tested as part of a case study on building maintenance. To date, the results obtained demonstrate the efffectiveness of this virtual decision-making collaboration. What this means is that this technology, initially developed for building maintenance purposes, can be adapted to other fields, such as health and defence.
A multi-agent virtual system for improving coordination and safety on construction sites
According to the U.S. Bureau of Labor Statistics, in 2012 the main earthmoving machines accounted directly or indirectly for almost 10% of fatal accidents on construction sites. These alarming statistics, coupled with the fact that construction work environments are not as structured or predictable as those of other industries, indicate the need for improved coordination and equipment management on construction sites.
Drawing on open-source data from the city, Professor Hammad and his team designed a 3D multi-agent system (MAS) environment for training workers in the practical skills required to safely and effectively operate on-site machinery. Professor Hammad’s platform combines GPS data for controlling the movement of earthmoving machines, with advanced safety management methods and virtual agents that assist each on-site worker. The MAS learning mechanism is designed to produce proactive and reactive responses with the objective of preventing collisions involving machinery.
- The proactive mode uses augmented reality to issue real-time proximity alerts to equipment operators based on sensory data.
- The reactive mode is based on an overall analysis of operator safety performance and draws on data recorded during near-miss incidents.
A case study was carried out to validate the proposed method. The MAS structure was deemed capable of effectively coordinating a fleet of earthmoving machinery and potentially improving safety during earthmoving projects.
Jeremy Cooperstock | From the real-time display of medical procedures to augmented reality used for first responder training
Professor Jeremy Cooperstock, director of McGill University’s Shared Reality Lab, is using virtual reality and augmented reality to develop new technological solutions meant to facilitate and enrich human-human or human-machine interaction. In particular, he has created the “Autour” augmented reality mobile application for people with visual impairments.
Virtual reality used to create interactive surfaces controlled by human walking
Professor Cooperstock is putting virtual reality to use in the development of interactive ground surfaces, that’s to say surfaces that react to human walking. His long-term objective is to transfer skills previously acquired in everyday tasks associated with walking to new interactive settings, where multisensory feedback can be used to create unitary multimodal percepts of virtual ground surfaces. These real augmented environments can be projected onto various ground surfaces, such as a sandy beach, a snow-covered surface, or a frozen pond. The research involves the integrated and interchangeable use of graphics and haptic-auditory modalities in floor interfaces to validate fundamental aspects of human multisensory ground surface perception via contact interactions in walking, and determine the usability of the interaction methods thus developed.
Virtual ground surfaces will ultimately lead to the development of a range of applications in smart cities, including smart signage on commercial streets, virtual tourism, and immersive games.
- Virtual tourism: Users will be able to control their position on a map and carry out a virtual visit of a city, with the ability to see the surrounding cityscape.
- Commerce: This technology could be used to create evocative multisensory displays on commercial streets, for example to advertise an in-store sale on ski products when someone approaches a storefront.
- Mobile immersive games: Integrating the haptic sense into a surface will create a more immersive experience than that of video and audio alone.
- Information delivery would be improved. It would thus be possible, for example, to inform a person waiting at a bus stop that his or her bus will be late.
Cooperstock and his team are currently working on bringing these capacities into a mobile context, by integrating haptic feedback directly into footwear, thereby making it possible to send haptic signals to users, regardless of where they are. This could lead to important applications for mobile information communication, health rehabilitation, sports and games.
Improving augmented reality training and real first responders interventions
In the event of an emergency fire response, poor visibility caused by thick smoke and darkness can prevent firefighters from safely taking action and responding in as timely a fashion as they’d like. In order to offer a solution, Professor Cooperstock is developing a new augmented reality tool aimed at improving the effectiveness of emergency first responders. The emergency responder tool is equipped with a mobile device that records video streams and positional information, and the unit connects to a wireless head-mounted display (eyeglasses) that provides a view in augmented reality. The device will also integrate sensors to measure ambient temperature and the first responders’ physiological parameters, in particular heart rate and breathing. This technological equipment will eventually make it possible to superimpose visual information over the eyeglasses and thus allow instantaneous communication between future firefighters and a trainer in a training setting. This communication will consist of a video offering various live views, together with data broadcast by first responders wearing augmented reality eyeglasses. The data would then be retransmitted to all first responders and accessible to a trainer via a Web application.
This technological solution, designed primarily to improve firefighter training efficiency, will ultimately be deployed under real emergency firefighting situations. It offers a variety of advantages:
- Allows a trainer to know more about a given situation, based on an aerial view and real-time information on specific actions undertaken by first responders, or from the perspective of an emergency first responder. This lets the trainer offer advice is real time, where required, for example by asking a first responders to follow the recommended search model. The trainer will also be able to modify the training scenario at any time in order to include challenges that future firefighters are likely to face in real-life situations.
- Gives first responders access to information on training scenarios thanks to a hands-free head-mounted display that lets them visualize the path they’ve taken along with virtual markers indicating positions or important objects in the environment.
- Makes it possible to evaluate the first responder’s performance potential in order to better identify and correct any weakness prior to placing that responder in a real situation. First responders will also be able to review and improve their performance.
Benoît Ozell | Virtual reality and augmented reality used to develop next-generation video games or train highly skilled health care personnel
Professor Benoît Ozell runs the Laboratoire d’infographie et de réalité virtuelle, which is affiliated with Polytechnique Montréal. Its research activities are concentrated in the high-tech fields of 3D graphics, virtual reality and augmented reality, and are carried out in collaboration with local and international companies with the objective of resolving specific problems. The lab’s activities concern various high-tech sectors of Montréal’s economy, including health care and video gaming. Its recent technological developments include collaborative environments used to train medical personnel in complex surgical procedures or clinical psychology testing.
Virtual reality for next-generation video gaming and training personnel in neurosurgical procedures
Montréal is recognized worldwide as a video gaming hub. However, the industry, in a continued effort to speed up the production cycle for future games, is using virtual reality to optimize gaming development stages. In collaboration with industry, Professor Ozell and his team have modelled the movements of human faces with the goal of making video game figures more realistic.
The laboratory is also putting virtual reality to use in the development of new interactive simulation tools for complex surgical procedures. With the addition of force-feedback, virtual operating rooms are more realistic, which makes it easier to teach resident physicians how to plan a surgical procedure and tailor treatment to a given patient. Among the interactive tools that have drawn interest is one designed for use in spinal cord surgery on patients with scoliosis, while another tool displays the various emotions of psychiatry patients in a virtual environment. In the first instance, students presented with a situation involving the malformation of an artery learn how to safely close it. In the second instance, they learn to analyze patient behaviour in order to identify the proper treatment.
Augmented reality in the service of health care training
In collaboration with CAE Healthcare, which offers a range of surgical simulators that let the user see inside the human body, Professor Ozell and his team are developing new visual cues that will make learning scenarios mimic reality much more closely. Specifically, these efforts are meant to add a sense of presence to current simulators by designing an avatar to represent a virtual instructor overseeing the performance of an operation.
OSSimTech | Virtual reality simulators used to broaden the scope of learning in open surgery
The medical community has high expectations for simulation learning. The interest lies in ensuring that health professionals’ first medical acts, increasingly, are performed and repeated on virtual patients before they begin to work with real patients. Until recently, however, none of the simulators available allowed a surgeon, using 3D eyeglasses and haptic instruments, to work on a patient in virtual reality and feel a range of gestures, including cutting, drilling, and sawing a bone. OSSimTech has thus staked out a leadership position in this highly promising field.
The Montréal-based company has developed the Sim-Ortho, an innovative immersive training platform that uses virtual reality and haptic technologies to promote the acquisition and consolidation of knowledge and know-how (psychomotor capacity paired with practical and decision-making aptitudes) as part of a continuing initial training program for orthopedic surgeons.
The Sim-Ortho simulator integrates haptic and tactile feedback to present a highly accurate representation of various surgical procedures, including tool handling and manipulation, trauma procedures, as well as knee and spinal surgeries. Since open surgery requires considerable force and is highly manual, haptic feedback allow residents to reproduce real touch sensations and feel in their hands what they see on the screen. What’s more, the simulator gives future surgeons a chance to deal with the kind of real-life emergencies they will eventually face in the performance of their duties, in a risk-free environment. A virtual simulation would see residents deal with a cardiac episode, for example, that increases their level of stress. This learning simulation would then evaluate their performance and decision-making abilities. The simulator also lets future surgeons practice gestures hundreds of times. Such gestures might include drilling a hole or sawing a bone. And unlike an operating theatre, where an error could cause irreversible damage to a patient, the simulator lets residents make mistakes and learn from them at no risk.
The OSSimTech simulator offers orthopedic residents a personalized open surgery training tool with numerous advantages, both in terms of its learning mode and the socioeconomic gains it generates.
Advantages of the training system:
- Accelerated learning curve and development of technical (manual, gestural) and decision-making skills in the performance of complex surgical procedures.
- Practical training covering a broad spectrum of real-life clinical cases, from the most common to the rarely seen, which resident physicians are not necessarily exposed to during their residency years.
- Technology that permits an objective and measurable assessment of resident physicians in order to determine if they are ready to work safely with a real patient.
- Unique system that allows the user to repeat the training program as often as necessary: preparation, action, evaluation, understanding, improvement.
- Standardization of current training methods and more objective assessment of technical skills.
Socioeconomic benefits:
- Increase in the skilled labour force through access to virtual patients and a personalized learning path adapted to each resident.
- Continuing professional development helps reduce potentially costly surgical errors.
With its virtual reality simulator and established collaborations with research centres (institutional and university-based) worldwide, OSSimTech has earned a well-deserved reputation for creativity and developed a unique Montréal brand image. The company has ambitions to become a global leader in open surgery training and build towards a future where all surgeons and residents will be required to use simulators to validate and revalidate their technical aptitudes and operating skills before they operate on real patients.
CAE Santé | Augmented reality shaping the future of medical training and improving patient safety
Simulation training is an accessible, low-risk solution for training health professionals in life-saving procedures, rapid response, interprofessional team training, disaster response, and the administration of anesthesia.
CAE Santé is drawing on its expertise in aviation to improve patient safety during medical procedures. The company is developing a range of leading-edge learning tools for current and future health professionals, notably physicians, nurses and ambulance attendants. Specifically, the company offers a range of highly realistic simulators for various surgical procedures and childbirth, so that professionals and students can practice risk-free before they work with real patients.
In a recent technological advance, the company patented an ultrasound simulator called VimedixAR. This tool connects to Microsoft’s HoloLens augmented reality headset and is enriched with holograms representing various parts of the human anatomy (e.g., heart, lungs). Students can enlarge, move or pivot the human organ to suit their practice purposes. For example, they can isolate the heart hologram, enlarge it, and even move around it to gain a different view of the organ. In addition, future professionals in training will be able to interact and move freely in a 3D clinical environment, to better understand how the anatomy and its circulatory, respiratory and skeletal structures are integrated. The real-time display feature lets them determine the position, in 3D, of a medical instrument for purposes of organ manipulation. As students examine the different parts of the anatomy, they instantaneously observe how the ultrasound beams slice them to generate an ultrasound image, thus providing the proper teaching required for sound decision-making.
While CAE Santé has already earned a reputation for other medical simulation products, this new augmented reality technology offers several advantages for the health care sector, particularly for medical and nursing schools.
- Accelerates learning and increases the number of ultrasound experts.
- Offers shared training experiences in an immersive clinical environment.
- Exposes medical students to rare pathologies not often encountered in clinical training settings.
- Lets physicians practice positioning devices or cardiac implants more rapidly and with greater precision, before they practice on real patients.
Virtual reality and augmented reality technologies hold strong potential, particularly for attracting investment. Relevant new applications in the city’s high-tech economic sectors showcase Montréal-based creativity and innovation and play a vital role in meeting specific economic challenges, including those presented by training and skilled labour, where they are being used to develop technical skills by placing people in real-life learning situations. Their technical training potential is also very high, and they will be critical to modernizing traditional sectors through digitization and developing real-time communications tools designed to strengthen collaboration between professionals.
As a North American leader recognized for its high-tech know-how, notably in video gaming and visual effects, Montréal is well positioned to fully leverage the advantages offered by virtual reality and augmented reality. The city is home to a talented and creative workforce and a critical concentration of world-class research centres, and it offers the financial incentives needed to support its entrepreneurial ecosystem. In a context where innovation plays an increasingly important role in competitiveness, it is essential to draw to a variety of optimization levers. Virtual reality and augmented reality represent one such lever. These technologies are already mature enough to ensure their democratization in business practices. It will be important to closely track the advances and developments in the years ahead, particularly with the advent of artificial intelligence, which could spark the development of new solutions in virtual reality and augmented reality.
