The hitchhiker's guide to web-mediated text: Method handbook for quantification of online linguistic data in a country-specific context

Official research report, Linguistic Explorations of Societies (Work Package 1). Working paper series 2022:1, The Quality of Government Institute, Department of Political Science, University of Gothenburg. ISSN 1653-8919. Author: Jonas Andersson Schwarz (Södertörn University, Sweden). Published March 2022. Full text preserved as `paper.pdf` (188 pages).

About the report

A method handbook, written 2019–2021 as the main research report for Work Package 1 of the cross-disciplinary research project Linguistic Explorations of Societies (LES), funded by the Swedish Research Council (Vetenskapsrådet). The report surveys how online linguistic data (blogs, forums, news, social media) can be quantified in country-specific contexts, and includes a comparative review of commercial data providers of web-mediated text.

Relevance: Twingly

The report contains a dedicated vendor profile of Twingly, the blog search and social data company founded by Martin Källström, in its survey of commercial social-data providers, and Twingly's data was used empirically in the LES project's analysis of language distributions.

Key points of the Twingly coverage:

• Twingly is profiled alongside providers such as Sysomos, Webhose (Webz.io), Socialgist, and Talkwalker. The report notes: "Twingly was founded in Linköping, Sweden, in 2006. Like Sysomos, their data mining initially focused on blog indexing." It highlights that Twingly "saw a leading edge in providing coverage of European languages," making its offering comparatively more regionally focused than many US-American competitors, with categorization and analysis through automated data enrichment processes employing several categorizations in 35 languages, including entity and sentiment recognition for some languages.

• Like Webhose, Twingly provides API access, indexing blogs, forums, news, dark web pages, and Russian social media posts on VKontakte.

• The report relays Twingly's stated volumes: over a million blog posts added per day; 8,000 new active blogs added every day; 10 million forum posts per day from 9,000 different online forums; 3 million news stories per day from 150,000 sources across over 100 countries. Twingly "used to scrape public Facebook pages, up towards 17 million posts per day, but make no mention of it anymore."

• Twingly was of interest to the LES project thanks to its comprehensive regional coverage and its striving to represent a broad range of languages and countries.

• In the empirical section, the report notes that "for the Swedish data provider Twingly, a small number of lists were sent to us by one of the company's representatives, tallying the language distributions of the various categories (forum posts, blog posts, etc.)" — since the company specializes in blog monitoring and scraping, its most comprehensive linguistic diversity was represented in its compilation of blog posts. Figure 2 of the report presents Twingly's stated average number of blog posts per day for the last three months (December 2018) in relation to estimated language size (kurtosis 9.18, skewness 2.96).