A new technology platform uses algorithms to create 3D city models from image and video data.
The platform, known as “VarCity”, is so versatile that it can use, evaluate, and automatically combine all possible kinds of image sources: aerial photographs, 360-degree panoramic images taken with special vehicles, and even standard photos such as those published by tourists on social networks and online. The technology also uses video material from sources like YouTube and public webcams.
“The more images and videos the platform can evaluate, the more precise the model becomes…”
Thanks to machine learning algorithms, the technology can recognize image content such as buildings, windows and doors, streets, bodies of water, people, and cars. Without human assistance, the 3D model “knows,” for example, what sidewalks are and—by evaluating webcam data—which streets are one-way only.
Adding life to city models
“The whole thing goes so far that we can perform a highly detailed analysis of traffic flows on the street and also measure pedestrian traffic, even in real time depending on the application,” explains Hayko Riemenschneider, a postdoctoral researcher at ETH Zurich and VarCity project manager. This makes counting traffic extremely easy: it can be done exclusively with cameras and no longer requires sensors to be installed on the road.
“Our technology is dynamic. It brings life, people, and vehicles to 3D city models, which is one of its major advantages over well-known applications such as Google Street View,” says Riemenschneider.
The technology was also devised in line with the principle of “privacy by design”, with the algorithms generating anonymized information based on image data. Pedestrian flows and vehicle traffic can be shown by avatars, making VarCity compatible with data privacy.
Cameras work together to track a single person
Lots and lots of data
The researchers developed the computer algorithms for the model over the past five years as part of a European Research Council (ERC) project. To showcase their work, they have created a 3D model of the city of Zurich, which they are now presenting in a video as they wrap up the project.
The VarCity technology is based on the evaluation of huge volumes of data. “The more images and videos the platform can evaluate, the more precise the model becomes,” says Kenneth Vanhoey, a postdoc researcher. “The aim of our project was to develop the algorithms for such 3D city models, assuming that the volume of available images and videos will also increase dramatically in the years ahead.”
The scientists use triangulation to create a three-dimensional model from image data. This requires several pictures of an object from various angles; an analysis of the distances and angles in the various images then allows the corner points of the object to be precisely located in the relevant space.
Parking and transportation possibilities
The researchers are not offering the models themselves as an application, but the technology behind them has many possible applications, including in urban and transport planning.
The project participants have founded several startups to market these applications: Spectando offers virtual building inspections for the real estate market, while Casalva performs virtual damage analyses on buildings for insurance companies, for example after extreme weather events such as hailstorms. No special devices are required for these applications—standard smartphone photos are sufficient.
Software makes 3D maps of buildings in real time
ETH spin-off Parquery, that has emerged from the VarCity project, also uses this technology. The company developed a parking management system that works exclusively with cameras installed near parking spaces and does not require parking space sensors.
The computer evaluates the video data in real-time and recognizes cars, so it knows which parking spaces are free and also whether a car is parked across two spaces. This system was first used in the Swiss city of Locarno, with further projects in over 30 cities in Switzerland and abroad.
The ETH spin-off UniqFEED uses the methods to identify perimeter advertising and individual players during television broadcasts of sports events, allowing it to calculate how long they are visible during the broadcast and to personalize the perimeter ads for various user groups.
Luc Van Gool, a professor at ETH Zurich’s Computer Vision Lab, lead the VarCity project. An ERC Advanced Grant to Van Gool funded the work.
Source: ETH Zurich
