Graphical User Interfaces & User-centric Interaction
Deep learning models have proven their recently unmatched performance in various use cases (see Object Detection, Semantic Segmentation & Pose Estimation for examples). However, it is known that their hunger for data is enormous. In a normal case, datasets of several hundred to millions of images including manually generated desired outputs (labels) are required to train a model. This problem poses a significant barrier in productively using Deep Learning. Data efficiency is especially a problem in robotics as collecting new data often requires actuators to be employed.
INTRODUCTION
Flexibility, Modularity and Intuition
With technological advances and ever increasing computational power behind the scenes, GUIs rely on established systems like "What you see is what you get" (WYSIWYG) and "Windows, Icons, Menus and Pointer " (WIMP). The mimicry of real world operations is probably the most well-known concept amongst consumers finding virtual representations on their devices’ screens for existing experience of interaction: swiping like flicking pages of a book and two-finger-rotate to turn something, but also introducing new paradigms like the infamous pinch-to-zoom or even more finicky gestures using multiple fingers at once.
In robotics, best practice user interfaces serve to ease the interaction, add intuitiveness and comprehensibility of complex operations. Touch-displays even found their way into manufacturing environments and one can find tablet-PCs and smartphones with different IP-protection classes. We at GESTALT are experts in designing and developing graphical user interfaces for robotics. We target user-centric design, carry out user studies and provide interfaces for connecting external services as well as industrial robots from arbitrary manufacturers.
Quality-Inspection Dashboard: geometry measurements, damage-detection and surface inspection (with polarization)
Proprietary user interface for autonomous transport vehicle interaction: on the left main part is the map, that is procedurally generated with SLAM from laser sensors and filled with semantic understanding from RGB imagery; on the right is the dashboard sidebar containing three distinct segments: the AGV control for manual intervention and scanning, the workflow manager and the asset-library for route planning and factory-oversight.
TECHNOLOGICAL OUTLINE
UI or UX? GUI or NUI? Unimodal or multimodal? User-centric!
The role model for the design of modern human-machine-communication is the natural human-human-communication, which for the most part is multimodal via speech (auditory) and via gestures (visual or haptic). The objective of using natural communications is to increase the acceptance of the technical system by the user as well as a to reduce training durations and operating cycles. Based on the paradigm of direct interaction, principles of action and cognitive processes are addressed, which transfer the interaction with real objects as closely as possible to the interaction with virtual objects within the user interfaces. In general, one seeks to get rid of technical aids, e.g. a special glove or markers for gestural interaction, in order to avoid negative effects on the guiding criteria of effectiveness, efficiency and user satisfaction. In recent years, the term Natural User Interface (NUI) was established in contrast to the classic term GUI.
Whereas classic GUIs rely on input devices such as keyboard and mouse, NUIs also consider novel types and methods of interaction such as voice input, touch and multi-touch operations. On a closer look at voice control, it is quite useful for service robotics in domestic environments but suffers from a lack of robustness within noisy industrial environments. Due to this issue, in the field of industrial robotics, voice control is currently only used for teleoperation purposes and for the close collaboration of humans and robots in laboratory environments.
Multimodality is the usage of multiple communication channels simultaneously or consecutively. This principle plays an important role in the design of modern control systems. Numerous scientific publications have shown that the multimodal design of industrial control and programming systems increases user productivity and acceptance. The prerequisite is an application and user-specific design of the systems. Multimodal control systems combine, for example, gestures (finger, hand and touch gestures), speech and various forms of output, such as augmented reality.
What to choose unimodal or multimodal communication, GUIs or NUIs?
Actually, there is no general response. Instead, it is more about the user-centric design referring to user-oriented interfaces achieving high usability and user experience. Rather, the central feature of the user-centric design process is a goal-oriented, iterative interchange between analysis, design and evaluation processes and the further iterative incorporation of the respective results.
Dashboard for a multi-robot control scenario, Design: Telekom Design Gallery, Implementation and Integration: Gestalt Robotics
APPLICATIONS & USE CASES
Human-Machine-Interfaces for Industrial Control
Optimized man-machine interfaces in (partially) automated technical systems support cognitive activities of the user such as planning, programming, diagnostics, monitoring, maintenance and service. Due to their abilities of abstraction, interpolation and adaptation, humans will still be required in most industrial processes within the next decades.
Modern industrial control systems are usually characterized by a high range of functions and complex operating, monitoring and coordination tasks. In contrast, the consumer products are primarily tailored to the needs and capabilities of users and consequently provide only a limited, easy-to-use scope of control functions. However, due to the complexity of industrial control, the user-centric adaptation of the complex operation functions is the main challenge.
The central objective of interaction design for industrial human-machine-systems is to realize an effective and efficient process management of the information exchange between human and technical system. Optimized interaction concepts increase efficiency but also increase the acceptance of the technical system and provide motivating and appealing work activities.
CONCLUSION
It's all about the context and the user
The possibilities in combining modalities and interaction methods within GUIs seem endless. What makes the difference and should be leading factors are the individual user (group) and the context. A user interface for a fair exhibit or a company's show room will definitely be different from a user interface for robots within the steel industry.
We develop tailored graphical user-interfaces, and evaluation through user studies for all kinds of robotic applications. Get in touch with us to discuss your use cases. You can contact us under info@gestalt-robotics.com or give us a call at +49 30 616 515 60 – we would love to hear from you.