Navigating the COVID pandemic using Augmented Reality to aid in Pharmaceutical Manufacturing.

Introduction

Augmented Reality (AR) is an interactive digital service through which information can be overlaid on objects and live data collected from the real world. It provides the user with additional information about the system through an interface that enhances their experience. This information can be accessed through smartphones, tablets & computers or more interactive systems such as a heads-up display or eyeglasses. Concepts for an AR experience can be traced back to the 1990s where applications were developed for NASA and the US Air Force Research Lab. Google released an optical head mounted display called Google Glass in 2014. It was heavily covered in the media for its widespread applications as well as ease of use in people’s day-to-day lives. However, the product failed to get as much traction as anticipated before, primarily due to concerns around safety and privacy. Additionally, the public failed to find everyday uses for the Google Glass.
There are common misconceptions on the differences between Virtual Reality (VR) and Augmented Reality (AR). While, AR builds on real-life environments by adding elements that enhance the user experience, VR creates a completely simulated environment that closes out any response from the physical word. AR has seen a variety of applications in healthcare & the pharmaceutical industry. From assisting doctors with medical procedures, to researching the effects of drugs on the human body, AR is helping improve patient outcomes. The IMA group has efforts centered on smart services by deploying an infrastructure of AR glasses and a connected network for better maintenance and inspection. Fig.1 shows a schematic of Augmented Reality. Various AR glasses are available from worldwide manufactures such as Realwear and Vuzix today. These are targeted for use in specific applications such as maintenance & service, remote support and troubleshooting in production environments. Such sectors are finding tremendous applications of AR and AR glasses. A number of systems are interconnected in manufacturing industries such as automotive & pharmaceuticals around the world, and this interconnectivity is valuable for maintaining production efficiency. This has also increased the level of complexity involved in operating a particular production line. In addition, as projects evolve over time, a number of resources and subject matter expertise needs to be coordinated. The location of experts that may assist on-site users may be different and their availability may be limited. Although Emails & Smartphones have aided in improving connectivity, they are not the solution to all problems. This is where Augmented Reality comes in.
IMA Life has identified use of AR for the pharmaceutical manufacturing environment including freeze dryers. A freeze dryer is a complex interconnected system, where multiple equipment such as compressors, vacuum pumps and heat transfer assemblies are working together. The availability of specialists for servicing such complex production

Figure 1. Schematic of an Augmented Reality application

equipment is always limited. In particular, during the pandemic, when travel schedules were changing on a daily basis, availability of skilled technicians is few and far between. Additionally, there may be several subject matter experts that are required to troubleshoot a certain problem including engineers, electricians and service managers. Having a dedicated platform to coordinate and remotely assist with all such requests is of great interest.
A major challenge that everyone around the world is facing is the COVID-19 pandemic. Due to numerous travel and safety precautions, many companies including major pharmaceutical customers of IMA Life were faced with adapting to a new way of engaging customers. An accelerated path was adopted to integrate the use of Augmented Reality at IMA Life that we describe in this paper. Applications: From remote service assistance to obtaining guidance for executing test procedures, the applications for augmented reality are varied. Fig. 2 illustrates the vision that IMA plans to follow for integrating the use of AR. IMA pivoted from this opportunity to roll out remote assistance services and field service activities from their manufacturing facility in Buffalo, NY using AR glasses. Sample services offered by IMA are described in the case studies section. For this, IMA Life evaluated multiple technologies and software platforms offering the AR experience. In this paper, we look at case studies with using Augmentir and Realwear. The AR glasses HMT-1, shown in Fig. 3 from Realwear were integrated with Augmentir’s software platform.

Figure 2. Deployment of Augmented Reality at IMA Life

Case Study 1: Tech-Transfer Support

This case study discusses the technology transfer support of a freeze drying process by execution of vial heat transfer coefficient (Kv) as requested by a Contract Manufacturing Organization (CMO). This was the first instance of IMA employing AR glasses during the ongoing COVID-19 outbreak in USA. Under normal circumstances this activity would have been executed via a 1 week on-site support at the client’s facility.
On its part, the client wanted to take advantage of the equipment and lyophilization process expertise available at IMA Life for an upcoming scale up involving an expensive protein based drug. Once the client expressed interest in the AR glasses technology offered by IMA, the AR hardware was initialized by IMA by installation of the recommended Android Package (APK). A SOP was created after performing functional testing of the AR system at Tonawanda, NY and it was shipped to client along with the AR hardware.

Figure 3. HMT-1 Augmented Reality Glasses from Realwear

Upon receipt of the hardware and SOP at the client site, the AR glasses were setup for the local network with support from an IMA development Scientist. The setup of the hardware took less than an hour and the process went smoothly once the client became familiar with the AR glasses. During the actual Kv testing, a real-time support via live video streaming was provided when the client activated the call expert functionality of the AR glasses interface via voice command. The technical support activity from IMA included a careful inspection of shelf loading, temperature probe positions in the lyo shelf, lyo recipe check and system engineering parameters check in the SCADA along with a real-time response to the client questions.
Overall, the augmented reality glasses played a critical role in identifying errors and implementation of updates for the Kv testing steps. The setup of this technology required little effort. During the live video streaming, the temperature probes were repositioned at the required locations in the lyo shelves.

Figure 4. Images captured from the AR glasses in use during remote tech transfer support

Case Study 2: Installation Feasibility Study

In early 2020, a major pharmaceutical company inquired about the feasibility of a VERISEQ® controlled nucleation retrofit on a production lyophilizer at a European facility. The VERISEQ® Nucleation system is designed to induce nucleation in sub-cooled liquid formulations by introducing ice fog into the lyophilizer chamber. The ice crystals contact (seed) the surface of the product and nucleation proceeds rapidly from the top downwards in the vials.
Typically, a VERISEQ® feasibility assessment requires an on-site visit, where an IMA Life service engineer would examine the mechanical area around the lyophilizer to assess the required hardware integration and utilities requirements. An on-site visit from a service engineer was planned but due to COVID-19 travel restrictions, this became impossible.
With travel restrictions in mind, IMA Life proposed the use of AR glasses to perform a virtual site visit. Using this approach, the facilities engineer would wear the AR glasses, streaming a live video and audio feed of the lyophilizer mechanical room back to IMA Life in Buffalo, NY.
A set of AR glasses was shipped to Europe, crossing the borders and customs without issue. An initial set-up/communications test call was scheduled with the client for the AR glasses in advance of the feasibility assessment. The client was able to set up and connect the AR glasses online independently prior to the initial set-up, commenting they “got on well navigating the set-up of the unit.”

Figure 5. Sample screenshot of video stream with the customer taking a measurement

The communications test call confirmed familiarity with the operation of the AR glasses prior to the start of the assessment.

A week later, on the day of the feasibility assessment, the AR glasses proved to be beneficial in the assessment. The AR glasses, worn on the users head, allowed for continuous video stream and communication between the client and IMA Life while the client navigated safely through the multi-floor mechanical area with multiple ladders and platforms, requiring the full use of both arms. Also, as the client had their hands unoccupied (as opposed to a cell phone or tablet inhand), they were able to actively take measurements with a measuring tape (Fig. 5) and to utilize their hands to point out components and specifics brought up in audio discussions back to IMA Life. Additionally, another advantage of the AR glasses is they could be clipped to a hard hat, which this area and similar other mechanical areas often require. On the receiving end of the video stream through web browser in Buffalo, NY, the video stream was uninterrupted and the field of view effectively captured a wide area including the customer’s arms and hands as needed. The audio was clear, with background noise typically present in a mechanical room not captured in the audio stream due active noise cancellation. In this VERISEQ® retrofit feasibility assessment, the AR glasses proved to be a useful substitute to an onsite visit in this VERISEQ® retrofit feasibility assessment, ensuring benefits that a standard mobile device video or audio call would have provided.

What does the future hold?

The uses of Augmented Reality are not restricted to providing service support. Every freeze dryer that IMA builds, goes through an elaborate procedure called the Factory Acceptance Test (FAT) before being shipped out to a customer site. A major challenge in conducting an FAT is coordinating test time and customer schedules. Through the use of AR, the FAT experience can be streamlined. Using AR glasses, IMA technicians will be able to provide a walkthrough of the entire machine, while the customer views
and monitors the system remotely. Furthermore, converting all documents currently written for an FAT to an online platform will enable both IMA and the customer to monitor the progress on the FAT in real time without the need to be physically present. Another challenging aspect of large production sites is handling the vast amounts of incoming shipments. At IMA, we envision using the AR glasses and platforms to assist the warehouse in adopting a hands-free approach. The AR glasses will help scan barcodes and product information and the AR platform will categorize each shipment according to the destination department. A strong database connected to SAP/CRM will minimize any errors, while benefiting the users. Although these implementations require a significant time investment, this project has been accelerated due to the COVID-19 situation. Following the successful use of the remote assist feature at 2 different customer sites, IMA has started to train the service group on uses of the AR platform along with the AR glasses. With increasing customer requests for support and numerous such applications, IMA Life plans to integrate AR into its standard product platform in the near future.