Remote Control System for Heavy Machine by 3D VR

1. Prior Simulation- Examination and confirmation can be done without actually going to the site!

Desk checking by 3D VR simulation is especially effective for verification of work plans in the initial stage. There is no need to go to the site, prepare equipment, or prepare personnel. It can also be utilized for pre-training and safety education as well. In the event of a disaster, the situation in the disaster area can be immediately reproduced by generating a base topography using the Geographical Survey Institute tiles of the Geographical Survey Institute and using aerial photographs and 3D point cloud data measured using UAVs. Instead of immediately getting on to the real thing, it is possible to formulate a work plan and pre-verify it before entering the site. In addition, 4D simulation enables work planning, progress management, and deeper understanding of the process for workers. Central ministries, local branch offices, and local governments can share a cloud database and implement disaster prevention measures in cooperation with people at the site.

Quickly reproducing the disaster area with 3D VR

▲Reproduction by aerial photography

▲Reproduction by point cloud data

Examination of local restoration plan by 3D VR

Utilization of 3D VR for work planning, verification, information sharing, and progress management

▲Cooperation between each site

▲Temporary road plan

▲Heavy equipment deployment work plan, temporary equipment installation plan

2. Training/ Safety education- Experience the same operation as the real thing in advance!　Usable for training and safety education as well

By operating a control device that reproduces the cockpit of a heavy machine or a real remote control device, you can operate the heavy machine model in the 3D VR space at the same time as operating the actual machine. If you have a motion device, you can experience tilt, acceleration, and vibration.　By using this technology, you can try out the work contents in the 3D VR environment that reproduces the site before operating the actual machine in the field. It can be used not only for pre-training of workers, but also for training and safety training of workers during normal times. In addition, VR goggles give you a stereoscopic view of the entire space, giving you the feeling of being at the scene. AR and MR are also supported.

▲Example of VR motion sheet
and cockpit of VR goggles (NICHIJO)

▲Example of cockpit
with motion platform
(NEXCO Snow Removal Simulator)

▲Remote controller of
compact type backhoe
(Kyushu Technical Office, Kyushu Regional Development Bureau)

3. Work monitoring, work guide, work instruction See what you want, as many as you need, from anywhere.

The coordinates and state of heavy machinery during work (posture, movement of moving parts, speed, acceleration, etc.) can be acquired wirelessly (5G) in real time and reproduced in 3D VR space. If you display this with a virtual camera, you can monitor your work in real time. In the 3DVR space, you can set any viewpoint, angle, angle of view, and number, so you can monitor according to your purpose without blind spots. More reliable monitoring is possible by using it together with the actual camera image. In addition, it is also possible to display explanations, diagrams, arrows, etc. of work contents, superimpose and display CAD data of work areas (example: excavation area / depth) as work guides, and directly display work instructions from the management center. (Referred from Public Works research Institute)

▲Work Guidance

▲Work Instructions

▲Work Monitoring

4. Control of multiple machines - Coordinated control and monitoring of multiple heavy machines

A centralized management system that performs coordinated control of multiple heavy machines. It connects individual remote-control systems to the network, and commands are issued to each remote control system from the centralized management PC. While monitoring the status of multiple heavy equipment in real time, you can control the remote control of multiple heavy equipment such as simultaneous start / stop / emergency stop and individual start / stop / emergency stop from a bird's-eye view of the entire project. In monitoring, camera images from any viewpoint such as the viewpoint from the driver's seat, external viewpoint, and overall view of each heavy machine can be used for monitoring, so you can quickly grasp the overall progress and problems areas.

▲3D VR monitor for overall view

▲State of each heavy machine, real camera, virtual camera

5. Support for autonomous driving - Integrated management of automatic driving system.

Similar to remote-controlled heavy equipment, it is also possible to centrally manage multiple heavy equipment that operates automatically.
It can be used as a management center for autonomous driving, such as monitoring during autonomous driving and remote controlling in emergencies. Autonomous driving is at the stage where it has finally begun to be put into practical use, and is being developed by general contractors and research institutes. 3D VR is especially effective for such applications. You can use the 3DVR system as a development platform for autonomous driving by performing simulations for verifying autonomous driving algorithms in the 3DVR space.

Autonomous driving supporting system

The basic configuration is the same as the integrated system at the time of remote control, and all or part of the remote control part is performed by automatic operation.

There is a high possibility that the loading, transportation, and removal of soil by dump truck can be automated. The approach to the backhoe and the approach to the soil removal site are performed manually by remote control, and the transportation on the way (running on temporary roads, etc.) can be automated. The main functions of autonomous driving are driving to the destination, avoiding obstacles, and stopping when they are unavoidable. 3DVR can also be used to verify the algorithm.

▲Research on the automatic operation of other heavy machinery such as backhoes and vibrating rollers is also in progress

The 3D VR simulation software UC-win/Road, which is the foundation of this technology, is the latest 3D VR software developed in-house by FORUM8 and first released in 2000, and it has been evolving for more than 20 years.
It supports BIM/CIM data and also supports J-Land XML and IFC data formats. In addition to simulation, it is also used for real-time cooperation between 3D VR and the real world, remote control, monitoring, and automatic control by coordinating with external devices. The following introduces basic functions related to this technology.

●Terrain generation

Quickly create terrain in disaster areas - 3D VR platform in the DX era

You can automatically generate any terrain around the world and realistically reproduce 3D terrain with aerial and satellite photographic images. As it supports open data such as GSI tiles, NASA’s SRTM data, Open Street Map (OSM), and J-Land XML, you can easily create a 3D VR space of the target area from the topography, aerial photographs, road data, and building information (position, shape, height) of any area specified on the map in a short time. You can directly display the point cloud data acquired by UAV etc. or convert it to terrain. You can also use a free database to install features in the 3DVR space including BIM/CIM models as it supports the IFC format. With these features, you can quickly create a 3D VR space in the disaster area and ensure a prompt initial response.

▲3D VR terrain generation of the target area by GSI tile
(Example of mountainous area)

▲Example of aerial survey of point clouds on DEM

●Coordination with UAV

Utilize for photogrammetry of the disaster area, point cloud data acquisition, and monitoring of heavy machinery during work!

You can freely set the flight route of the UAV in the 3D VR space and control the flight automatically. Since it is possible to take videos and photographs, the latest situation in the disaster area can be measured and reflected in the 3D VR space as aerial photographs and point cloud data. It is also possible to transmit surveillance images from the sky by making it fly automatically according to the work area and process when using heavy machines.

●Scenario

Experience work training scenarios and safety education scenarios!

It is a function to register various events that occur in the 3DVR space in advance and control the operation of the simulation according to the occurrence status of events. By setting various situations associated with the work and preparing scenes according to the situation, you can have various experiences in the 3D VR space, which can be utilized for pre-training. It can also be used for training operators and safety education at all times as well.

●4D simulation

Verification of work progress and progress management by 3D VR animation

By defining the project schedule in association with the 3D model on the Gantt chart and performing animation operation while installing, moving, and removing the 3D model as it progresses, it is possible to use it for verification and understanding of the work contents at the preliminary stage. When actually working, it can be used for checking the work contents and managing the progress.

●Driving Simulators

Control heavy equipment with real-time communication

It is a function to register various events that occur in the 3DVR space in advance and control the operation of the simulation according to the occurrence status of events. By setting various situations associated with the work and preparing scenes according to the situation, you can have various experiences in the 3D VR space, which can be utilized for pre-training. It can also be used for training operators and safety education at all times as well.