Every day, users upload millions of photos on platforms, such as Flickr, Instagram or Facebook. A study of researchers of Karlsruhe Institute of Technology (KIT) now shows that these photos can be used to assess the social importance of certain landscapes. For this purpose, they developed a new image analysis method based on artificial intelligence. The results might be of particular importance for landscape management and presentation. The results are published in the journal Ecological Indicators.

Functioning ecosystems are of crucial importance to humans: they supply major resources, such as food, water or fuel. But they also play a cultural role as resorts, holiday destinations or esthetic photo motifs. “The many photos published by people on online platforms reveal that landscapes have an enormous cultural importance,” says Dr. Heera Lee of KIT’s Institute of Meteorology and Climate Research in Garmisch-Partenkirchen. In her opinion, this has to be considered in sustainable development and landscape planning. Consequently, Lee and her team have developed a method for landscape planners to quickly and reliably use photos in social media to draw conclusions with respect to the socio-cultural importance of landscapes.

Her study “Mapping cultural ecosystem services 2.0 – Potential and shortcomings from unlabeled crowd-sourced images” covers the region of the Mulde basin in Saxony as an example. “Here, the Ore Mountains attract people, who want to go hiking, mountain biking or do winter sports,” Lee says. The amount of photos posted by visitors on different platforms is accordingly high. For her study, Lee and her team collected photos that were posted on Flickr from January 01, 2005 to December 31, 2016. During this period, 725 different persons uploaded about 13,000 photos of the region on the online platform.

The evaluation of the photos reflects in high detail how tourists and the local population use the area of the Mulde basin for leisure activities or for enjoying nature. Now, landscape planners can identify the “hotspots” and better understand the socio-cultural importance of certain areas. On this basis, they can limit access to endangered areas or create offers and facilities to support their activities.

Image Evaluation with the Help of Machine Learning and Artificial Intelligence

To obtain reliable and relevant statements with respect to the socio-cultural use of landscapes, Lee and her team developed a new method that also considers the contents of the photo: “We did not only want to know where and when the photos were taken but mainly what was photographed. This was the only way to identify cultural hotspots in the region studied.”

The researchers obtained information on the photos from the tags given by the users at the time of publication. They show where a photo was taken and what can be seen on it. However, the researchers had to cope with two challenges: the quality of the tags given by the users and the duration of evaluation. Although Flickr suggests tags for photos and videos, they are not used by all users. As 2500 of these photos studied had no tags at all and 590 had only one single tag, the team of researchers decided to use automatic tags from an image recognition algorithm based on deep convolutional neural networks through a cloud computing platform, Clarifai. This algorithm allocated 20 tags to each photo. In total, 2317 different tags were applied to about 13,000 photos.

“We then assumed that similar photos contained similar tags. If the similar tags co-occurred on multiple photos, we considered this as an indicator of the photos belonging together,” Lee says. With the help of social network analyses, the researchers divided the tags into nine thematic clusters and assigned the photos to one or several of these clusters. Close-up views of flowers or butterflies were assigned to the cluster “Existence,” as they showed objects existing in the areas studied. Lee and her team then checked the accuracy of the method by means of a manual evaluation: about 20% of tags had been assigned wrongly. The researchers, however, expect that reliability of assignment will be improved significantly by the further development of artificial intelligence. In addition, Lee and her team compared the results with empirical studies of the cultural use of agricultural areas with similar results.

The method developed also accelerates evaluation. Allocation of the tags to the nearly 13,000 photos took about three hours. For comparison: manually, a person is able to tag about 140 photos per hour. “With this method, we were able to reliably and quickly classify and allocate photos from social media,” Lee says. The method can also be transferred to other regions of which photos have been published on the Internet. The European Union and the Federal Ministry of Education and Research of Germany funded Heera Lee’s research with two projects, namely OPERAs, and CONNECT of BiodivERsA 2011, respectively.