KENNESAW, Ga. | Jan 29, 2020

When it comes to gauging public opinion, there is perhaps no better outlet than a social media network.

For the U.S. Embassy in Beijing, however, retrieving pertinent information from thousands of social media posts made in a foreign language could take months. With the help of Kennesaw State University faculty and students, it can now be done instantly.

From April to October, four undergraduate and two graduate students joined computer science professor Dan Lo in developing a computer program that uses natural language processing to identify trends in social media posts on Weibo and WeChat, two platforms used extensively in China with about 445 million and one billion monthly active users, respectively. The program is designed to “scrape” social media posts and turn them into datasets to be used for data analytics and mining, resulting in a platform that allows officials to better gauge what captures the attention of Chinese netizens in a given timeframe.

The project was one of five awarded to Kennesaw State during the fall semester by the U.S. Department of State’s Diplomacy Lab, a program that harnesses the scholarly efforts of college students and faculty to solve complex global challenges. While some projects are “course-sourced” in the form of a special topics classes, Lo said he chose to pursue his project as a directed study with computer science students. The team spoke regularly with the U.S. Embassy in Beijing to better understand the diplomatic and technological challenges they faced, such as the ability to retrieve enough data to sense a trend in social media posts.

“Though it’s similar to Twitter, the data it holds isn’t as accessible to the public,” Lo said. “Throughout the project, we learned that there are mechanisms in place to prevent crawling programs. For example, if you tried to retrieve 10 posts in five seconds, they would block your IP address.”

The team developed a workaround that allowed them to collect information anonymously. What would have taken a person months to collect can now be done in just seconds. Though they could collect data faster, they still faced the challenge of decoding Chinese characters, which could hold a multitude of meanings.

“Context could completely change the meaning of a character,” said Charles Gardner, a graduate computer science student who assisted in the project. “In the end, we created a process to determine the contextual relationship between words in order to better track the sentiment being conveyed.”

In its current form, users can plug in keywords such as “America” or “Hong Kong,” prompting the program to retrieve all relevant posts. From there, posts are categorized to identify central themes, allowing the users to analyze public opinions on the given topic.

“What I enjoyed most about this project is how applicable this technique is to the real world,” Gardner said. “Often we are able to demonstrate our skills in class projects, but this one is something that can directly benefit our client, and that made this experience incredible.”

Altogether, KSU students spent more than 1,300 research hours on the project, resulting in two papers accepted by the 2nd International Workshop on Big Data for Marketing Intelligence and Operation and the 2019 IEEE Conference on Big Data. This semester, Lo will begin working on another Diplomacy Lab project through the U.S. Embassy in Poland identifying fake news and disinformation campaigns. Though in its infancy, Lo said he aims to create a program that can rate the accuracy of news postings by percentages.

KSU is one of 35 universities involved in the Diplomacy Lab public-private partnership that enables the State Department to “course-source” research and innovation related to foreign policy challenges. Partner universities, including Ivy League institutions such as Yale University and Columbia University, conduct research on the State Department’s most critical matters, including climate change, democracy and human rights, counterterrorism, global health, energy security, gender equality, economic policy trafficking of individuals, food security, and conflict and stabilization.

Twice a year, the State Department compiles a menu of projects that have been collected or solicited from embassies and state offices across their profile, said Sherrill Hayes, program director for the Ph.D. in Analytics and Data Science and professor of conflict management, who serves as KSU’s Diplomacy Lab coordinator. Generally, the projects are items officials have an interest in but don’t necessarily have the capacity to execute.

“The purpose of this, since its inception, was to give students real-world experience on a variety of problems that affect American diplomacy,” Hayes said. “This is a university-government partnership that is very focused on giving students these opportunities to work on projects that are both broad in topic and interdisciplinary in nature, and our University is uniquely positioned to answer the call.”

---

Written by Travis Highfield
Photography by David Caselli