Online meetings and lectures have become a part of life due to the COVID-19 pandemic. I myself regularly use audio and visual technology to communicate with people in other places. However, compared to in-person conversations, such experiences lack a sense of realism. When online, I am unable to fully convey the information I put out, and at the same time, I cannot fully receive the information being sent out by the other party.

This is the same when using a head mounted display (HMD) to enjoy virtual reality (VR) on a big screen. I am sure you have experienced poor image resolutions and laggy movements. One reason is that end-user terminals—such as HMDs—do not have sufficient power to process large amounts of information. How then can we improve the sense of realism?

Distributed processing to create visual experiences that feel real

My laboratory seeks to address this issue. We use various computing resources linked via networks, including edge servers installed near communication base stations for sophisticated computing, and cloud servers that carry out far more high-level and large-scale computing than edge servers.

If processing is allocated to such computing resources, we can reduce the processing load on end-user terminals. We use a method known as network computing to work on giving shape—in the form of an application—to the kind of processing that should take place at a certain time and place, after understanding the state of networks and taking into consideration the type of contents.

There are times when we intentionally reduce the amount of data for 360-degree images. Instead of reducing data across the board, we predict where the user will look and reduce the amount of data for places with low probabilities of being seen. This is all for the sake of controlling image quality to create visual experiences that feel real.

Using see-through HMDs, it is possible to overlay virtual objects on top of real spaces. This technology is called mixed reality, and it can be used to detect objects from images captured by the user’s camera. In addition, a technology has emerged where a three-dimensional object is expressed as a collection of points known as a point cloud.

Network computing will become more important in the future

To achieve such convenient functions, it is necessary to execute processing with an extremely high computing load, such as the artificial intelligence technology known as deep learning. At the same time, terminals have to be smaller and lighter to allow use by more people.

If we prioritize the size and weight aspects, it will be difficult to raise the computing capabilities of computers being installed in end-user terminals. Therefore, the support of network computing—which distributes processing for execution at the edge or on the cloud—is expected to become increasingly important in the future.

Research usually starts from reading relevant academic papers and discussing them with students as well as exchanging opinions with other researchers at conferences. We identify issues as well as the shortcomings of existing research, grow ideas, and eventually formulate algorithms, which are the procedures for solving problems.

For example, while an architect may require an enormous amount of effort to design a building and actually construct it, we can develop a prototype in half a year. It is interesting to be able to quickly give shape to ideas.

The book I recommend

“Computer Networks”
by Andrew S. Tanenbaum, Pearson

The internet is an ingenious design, and this book allows you to properly understand the reason for its widespread adoption. I was deeply impressed when I read it during the doctoral program at graduate school. It is a standard textbook with several revisions over many years, and even today, I continue to use it at my laboratory’s roundtables.

Interviewed: November 2022