Multilingual and MultimodalHate Speech Detection in Social Media

Deteccion Multilingue y Multimodalde Mensajes de Odio en Redes Sociales

Gretel Liz De la Pena SarracenUniveristat Politecnica de Valencia

gredela@posgrado.upv.es

Resumen: In this doctoral thesis we propose the design and development oftechnologies for the automatic hate speech detection. The hypothesis on whichthe project is based is that hate detection can be improved by incorporating othersources of information such as images into text processing. In this way, we intendto develop strategies for automatic hate detection from a multimodal approach.Furthermore, the project will take into account the multilingual analysis of hatespeech, using transfer learning strategies for the treatment in languages with littleinformation. For the development of the research, we plan to build a dataset thatallows multilingual and multimodal processing. In general, the work will be focusedon deep learning techniques for the proposal of approaches for hate speech detection.Palabras clave: Hate Speech Detection, Multilingual System, Multimodal System,Transfer Learning, Deep Learning,

Abstract: En esta tesis doctoral proponemos el diseno y desarrollo de tecnologıaspara el tratamiento automatico de mensajes de odio. La hipotesis en la que sesustenta el proyecto es que la deteccion de odio puede mejorar al incorporar, enel procesamiento de textos, otras fuentes de informacion como las imagenes, queen varias ocasiones son compartidas junto a dichos mensajes. De esta forma, pre-tendemos desarrollar estrategias para la deteccion automatica de odio desde unenfoque multimodal. Por otra parte, en el marco del proyecto tendremos en cuentael analisis multilingue de mensajes de odio, haciendo uso de estrategias de trans-ferencia de aprendizaje para el tratamiento en idiomas con poca informacion. Parael desarrollo de la investigacion, nos planteamos construir un conjuno de datos quepermita el procesamiento multilingue y multimodal. En general, el trabajo estaraenfocado en tecnicas de aprendizaje profundo en la propuesta de aproximacionespara la deteccion de odio.Keywords: Deteccion de Mensajes de Odio, Sistema Multilingue, Sistema Multi-modal, Aprendizaje por Transferencia, Aprendizaje Profundo

1 Motivation and Background

Nowadays, the web has become one of themain means of communication. Althoughthis is an advantage in many ways, it is alsoa problem due to the spread of negative mes-sages such as those with hateful content. Ingeneral, many sources define a message withhate speech (HS) for any expression whosecontent is abusive, insulting, intimidating,that incites violence, hatred or discrimina-tion. This type of message is directed againstpeople based on their race, ethnicity, reli-gion, sex, age, physical condition, disabil-ity, sexual orientation, political conviction,

etc. (Erjavec y Kovacic, 2012; Nobata et al.,2016). Furthermore, these types of messagescan contribute to a general climate of intoler-ance that makes attacks more likely againstaffected groups1.

Messages with hate speech are usually vi-ral, so they are potentially dangerous. There-fore, its detection becomes a task of inter-est in different fields of research. Many ofthe mechanisms used to identify HS rely onreports from users and content moderators.This is a way that makes it possible to detect

1https://www.ilga-europe.org/what-we-do/our-advocacy-work/hate-crime-hate-speech

23

Proceedings of the Doctoral Symposium on Natural Language Processing from the PLN.net network (PLNnet-DS-2020) Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)

CEUR Workshop Proceedings (CEUR-WS.org)

some cases of HS, however the large volumeof daily publications makes it difficult to de-tect hate manually. Therefore, building sys-tems capable of detecting HS automaticallyis a very relevant task.

In fact, automatic hate speech detectionhas become a trending topic within NaturalLanguage Processing (NLP) in recent years.This has been reflected with the developmentof different shared tasks related to HS detec-tion. As a result, different research groupshave been involved in the treatment of theproblem. Thus, approaches based on dif-ferent strategies for text analysis have beenproposed (Schmidt y Wiegand, 2017a; For-tuna y Nunes, 2018; Robinson, Zhang, yTepper, 2018; Al-Hassan y Al-Dossari, 2019;Zhang y Luo, 2019; Arango, Perez, y Poblete,2019; Basile et al., 2019; Schmidt y Wiegand,2017b). Such approaches range from the useof traditional machine learning techniques tomodels based on some deep learning architec-tures.

Some works are based on the use of lex-ical resources. Others use template-basedmethods, where words frequently associatedwith hate expressions and their contexts areidentified and subsequently used in detect-ing hate phrases. On the other hand, tra-ditional statistical methods such as LogisticRegression, Decision Trees and Support Vec-tor Machines have also been widely used inthe task. Furthermore, different types of fea-tures have been used, such as those obtainedwith the grammatical labeling (Dinakar, Re-ichart, y Lieberman, 2011) and the n-grams(Liu y Forss, 2014). Generally, these featuresare extracted at the lexical level, construct-ing a vector in which the presence or absenceof the words of the defined vocabulary is indi-cated. Many other works have used methodsbased on deep learning. In this case, the fo-cus has been more on the architecture of themodels than on the linguistic analysis. Themost used topologies are convolutional andrecurrent networks, and more recently theTransformers models such as BERT. Thesemodels have been used independently, butalso for the construction of more complexmodels from the combination of different ar-chitectures, as our proposals in some pastshared tasks (De la Pena, 2019; De la Pena yRosso, 2019; De la Pena Sarracen et al., 2018;Muniz Cuza, De la Pena Sarracen, y Rosso,2018; De la Pena Sarracen y Rosso, 2019).

Despite the fact that different approacheshave been proposed and evaluated for auto-matic hate detection, the current situationstill demands an in-depth analysis in whichsome factors are taken into account. The mo-tivation of our research arises from the vari-ation of HS given the diversity of languagesand the lack of information in the analysiswhen considering only text. On the one hand,the language can vary due to the diversityof languages. This is that some expressionsused in one language to convey hatred do notmake sense in other languages when translat-ing it. This is seen even more clearly in vari-ants within the same language. For exam-ple, a text in Mexican Spanish, which is notconsidered hateful in Mexico, may be consid-ered hateful in Spanish from Spain due to thedifference in the use of expressions in thesecountries. Therefore, it is necessary to de-sign strategies that take into account differ-ences between languages and analyze varia-tions within the same language according todifferent regions when detecting hate. On theother hand, another point to keep in mind isthat in hateful texts specific linguistic struc-tures and expressions do not have to appear.In fact, hatred can be expressed implicitly,hence detection can get false results by usingonly the text as a source. This suggests theuse of other sources additional to the text foranalysis.

2 Research Hypothesis

The main research hypothesis we want to in-vestigate is that the inclusion of other sourcesof information besides text to implementmultimodal analysis can improve the perfor-mance of automatic hate speech detection. Inaddition, we would like to investigate thesetopics, including multilingual models thattake into account the variability of languages.Therefore, the main objectives of the researchcan be listed as follows:

i) Design and evaluate multilingual strate-gies for automatic hate speech detection. Weaim to design mechanisms with which differ-ent languages can be processed taking intoaccount the variability between them. At thesame time, we aim to take advantage of lan-guage with a large amount of data to enrichthe treatment of other languages with littleinformation by using transfer learning.

ii) Propose multimodal strategies takinginto account visual information together with

24

the textual one. The idea is to incorporate vi-sual content into hate speech detection. Weplan to use models based on deep learning de-signed to process images and text, and com-bine them for a better understanding andidentification of hate.

iii) Create a dataset containing differentlanguages and information sources. As partof this objective, we aim to design a mecha-nism to obtain phrases that can express ha-tred to build linguistic resources for the con-struction of the corpus. We would like topropose a semi-supervised strategy to tag thecorpus that will be multimodal by containingtext, images, and geolocation information.

iv) Experiments on the new dataset withvarious approaches on the system and fea-tures. With the new dataset we pretend tocarry out experiments to evaluate the pre-viously used approaches. In addition, weplan to design new approaches consideringthe new geolocation information that will al-low us to study the publication of hate speechin geographic areas.

3 Methodology and Experiments

According to the objectives, we started byparticipating in two shared tasks. We pro-posed and evaluated multimodal models inone of the tasks and a multilingual systemin the other. Relevant details are describedbelow:

3.1 Memotion Analysis(SemEval-2020 Task 8)

Memotion Analysis shared task has been or-ganized in order to bring the attention to-wards Internet memes processing (Sharma etal., 2020). Three subtasks were defined inthe task, which are Sentiment Classification(subtask A), Humour Classification (subtaskB) and Scales of Semantic Classes (subtaskC). The main goal of the subtask A is to clas-sify a meme as positive, negative or neutralaccording to sentiment content. The subtaskB aims to identify the type of humour ex-pressed among the categories: sarcasm, hu-mour, offense and motivation. These con-cepts are closely related to hateful content.In particular, sarcasm is a phenomenon thatmust be considered in hate speech detec-tion, since hate is often transmitted implic-itly through sarcasm. Finally, the subtask Cfocuses on quantifying the level to which aparticular category is expressed.

We proposed a multilingual model thatcombines the analysis of textual and visualinformation (De la Pena y Rosso, 2020c).The pretrained BERT base model is used forthe text processing, and a pretrained VGGmodel for images. Features obtained fromboth text and image analysis are combined tofeed a simple classifier that obtains the finalcategories, as Figure 1 shows. We performan ablation study in course of the proposaldesign to analyze the importance of the mul-timodal model. Furthermore, we analyze dif-ferent models rather than BERT and VGG,including traditional machine learning mod-els such as Support Vector Machines.

Figure 1: Multimodal model

The image features vector is concatenatedwith the textual features vector to obtain ageneral features vector. This vector is a highlevel representation of a meme and is used inthe final softmax layer to obtain the output.

In the experiments, we used macro F1for each of the subtasks, and then the av-erage. Table 1 shows a comparison of theaverage among the results obtained with theproposed model and variants in which someof its components are removed. Basically, ineach evaluated model, one of the models thatprocesses text or images was eliminated. Itshould be noted that when a simple modelis used instead of the multimodal model, theresults are even worse.

Model AverageMultimodal Model 0.4300Text Model 0.3042Image Model 0.3306

Table 1: Macro F1 results

Table 2 shows the results obtained in thetest set for subtasks B and C, which are re-lated to hate speech detection. The numberof participants was 31 and 28 for the subtasksrespectively, where we reached the positions 4

25

and 6. When analyzing the general ranking,it can be seen that even the best results werearound to the value 0.5, which suggests onceagain that the models do not learn correctly.We believe that the values are due to the la-belled of the memes, since for each categoryis not very clear the annotation that can bevery subjective. The quality of the datasetprovided in this task increases the need forthe creation of the multimodal corpus thatwe consider as part of the doctoral thesis.

Model Subtask B Subtask CPos F1 Pos F1

Best system 1 0.5183 1 0.3224Our system 4 0.5093 6 0.3143Last system 32 0.4002 29 0.1267Baseline 10 0.5002 19 0.3008

Table 2: Results in the test set

3.2 Multilingual OffensiveLanguage Identification inSocial Media (SemEval-2020Task 12)

Multilingual Offensive Language Identifica-tion in Social Media is a shared task whichaims to identify offensive language in texts.The languages included in the task are En-glish, Arabic, Danish, Greek and Turkish(Zampieri et al., 2020).

We propose a system based on BERT(De la Pena y Rosso, 2020b). The first stepin the strategy is a preprocessing. In thisstep the English tweets are cleaned. Firstly,misspelled words are corrected with the sup-port of the TextBlob2 tool. We think it is animportant step since many users tend to mis-spell words and this can lead to a large num-ber of elements outside of the vocabulary.Also, we replaced each emoji with a phrasethat describes its meaning with emoji3 tool.

3.2.1 Features

We included a feature analysis to use the in-formation for discrimination between classes.The first group of features is based on sometexts basic properties: (i) the length of thetweets, (ii) the number of misspelled wordsand iii) the use of punctuation marks, whichis the number of times that one of the signsin the set {?! ’[...]} is used in the text, whichindicate exclamation, question or omission of

2https://textblob.readthedocs.io/en/dev/3https://github.com/carpedm20/emoji/

phrases. The element [...] corresponds toa sequence of more than one dot. Anothergroup of features is based on semantic prop-erties: (i) the use of emoticons, as well as(ii) the noun phrases. In both cases, a vec-tor is constructed with the emoticons or nounphrases present in a text. The representationin this vector space is based on TF-IDF andthe dimensionality of the vectors is reducedby using the Principal Component Analysis(PCA) technique. Then, it is added to thisvector a last component indicating the num-ber of emoticons or noun phrases in the orig-inal text.

3.3 Method

The general architecture of the proposed sys-tem is showed in Figure 2.

Figure 2: General system architecture

The system consists of a BERT basedmodel at the text level. This model is usedas an embedding generator from the text.Hence, a vector representation (BERT out)is obtained given a text. Basically, the vec-tor is the output of the special token [CLS]included in the processing in BERT. After-ward, this vector is concatenated with thefeatures vector obtained before, and a nor-malization layer is applied to the result. Fi-nally, the vector is fed to a softmax layer topredict the output.

In the analysis of the Arabic, Danish,Greek and Turkish languages, the same ar-chitecture was used. The main difference liesin the model used to obtain the vector oftext embeddings. In this case the MBERTmodel is used, which has been trained with104 languages in the same way as BERT forEnglish. In the model the tokens from differ-ent languages share an embedding space anda single encoder. There are no cross-lingualobjectives specifically designed nor any cross-lingual data, like parallel corpora. How-ever, MBERT produces a representation thatseems to generalize well a cross languages for

26

a variety of tasks.The experiments are carried out with the

10-fold cross validation stratified technique.The measure is macro F1-score, according tothe one used for the ranking of the systemsin the competition.

Table 3 shows a summarization of theexperimental results for offensive languageidentification, obtained for English. In theproposed model (Proposal) all the featuresare taken into account and BERT is used.We can check the superiority of the proposalcompared to the baselines.

Proposal 0.9496

BaselinesSVM 0.8825CNN 0.8910BiLSTM 0.9204

Table 3: Macro F1 for English

Table 4 shows the results for languagesother than English. On the one hand, we cansee that the features are not very relevant,since the difference in the results is not sig-nificant with respect to those obtained withthe model where the features are not used.For Arabic, Greek and Turkish, the proposedmodel achieves better results with respect tothe baselines as well as in English.

Model LanguagesArabic Danish Greek Turkish

Proposal 0.8064 0.7048 0.7350 0.7218

BaselinesSVM 0.6912 0.7258 0.7295 0.7033CNN 0.7343 0.6503 0.6782 0.6675BiLSTM 0.7556 0.6620 0.7049 0.6932

Table 4: Macro F1 for Arabic, Danish,Greek and Turkish

3.4 Keyword Extraction andBERT-based TransferLearning

The current work is focused on proposing astrategies to extract words and phrases thatcan express hatred. We study different ap-proaches that have been proposed for key-word extraction in a general context. Then,we design a strategy that uses tagged cor-pus to extract phrases taking into accountnot only the frequency of appearance in hate-ful texts, but also analyzing the non-hatefultexts of the corpus. An interesting issue is

that some users often use words that can in-dicate hate, in contexts where hate is not ac-tually expressed.

According to this issue, in (De la Pena yRosso, 2020a) we propose a strategy whichuses a new measure to rank offensive wordsfrom a tagged corpus on the basis of the cu-mulative distribution function and the har-monic mean of relative frequencies of theterms of offensive and non-offensive texts.

We use two scores based on relative fre-quencies and the Cumulative DistributionFunction. So that, we obtain two measuresfor each words that describe the terms dis-tribution in a cumulative way. Finally, theharmonic mean is used to combine both mea-sures. It gives the greatest weight to thesmallest item of a series, so that it givesa curve straighter than the arithmetic one.Thus, we obtain the final measure.

Figures 3(a) and 3(b) show the 50 mostrelevant terms according to the frequencyand the new measures for the offensive classrespectively. In the analysis we use the Offen-sEval 2019 data set (Zampieri et al., 2019).The darker portion of each bar indicates themeasure value for the term in the offensiveclass and the lighter part for the non-offensiveclass. We can check that the high frequencyof a term in offensive texts is not a good dis-criminator. The first terms in the rankingalso have a high frequency in non-offensivetexts. The frequency is even greater for thenon-offensive class than for OFF in mostcases. In contrast, when using the new mea-sure the first terms like ’f*ck’, ’sh*t’ and ’a*s’are more discriminative.

Furthermore, we proposed an approachbased on transfer learning for the multilin-gual offensive language detection. The modeluses MBERT to represent mapping betweenlanguages, which is concatenated with otherBERT-based model, trained for offensive lan-guage identification in English. Then, weinclude in the model a residual connectionfrom the MBERT to the classifier output, fol-lowed by a layer normalization. The idea isto directly add the corresponding informa-tion from the generated representation of thetext to the information obtained from thepre-trained model for classification. Finally,the complete model is trained for each newlanguage.

We evaluated this proposal for the fourlanguages in the dataset of Offenseval 2020,

27

(a) Frequency

(b) New Measure

Figure 3: Most relevant offensive terms

taking English as the original language. Inall languages the results are better when thetransfer learning strategy (among languages)is applied, taking as reference the results ob-tained when we used only MBERT. Also,we analyzed the residual connection of themodel, and the results get worse when it iseliminated.

4 Future Work

Once the shared task have finished and re-sults are obtained to extract offensive words,we plan to begin the construction of the mul-tilingual and multimodal dataset. Therefore,we will follow the steps listed below:

1. Filter messages on the Twitter socialnetwork using phrases taken from the

previous process and words taken fromthe Hatebase4 tool. We will use theTwitter API to download tweets in a cer-tain period of time.

2. Find potential haters by analyzingthe users who publish the downloadedtweets and already identified accounts onTwitter.

3. Download the tweets of the identifiedusers in real time to obtain the publica-tions that have not yet passed the Twit-ter filter.

In this step tweets will be filtered for theidentified users, downloading those thatcontain, in addition to the text, visualand geolocation information.

4. Develop experiments with different mod-els on the new built dataset.

The idea is to evaluate models based ondeep learning with architectures such asrecurrent and convolutional networks, aswell as Transformer models like BERTand ALBERT. On the other hand, wepropose to design and evaluate strategiesbased on deep representation learningtechniques on knowledge graphs analy-sis. Our purpose is to incorporate mul-timodal information and to address theexplainability in the hate speech detec-tion with the analysis on graphs.

Bibliografıa

[Al-Hassan y Al-Dossari2019] Al-Hassan, A.y H. Al-Dossari. 2019. Detection of HateSpeech in Social Networks: A Survey onMultilingual Corpus. En 6th InternationalConference on Computer Science and In-formation Technology.

[Arango, Perez, y Poblete2019] Arango, A.,J. Perez, y B. Poblete. 2019. Hate SpeechDetection is Not as Easy as You MayThink: A Closer Look at Model Valida-tion. En Proceedings of the 42nd Inter-national ACM SIGIR Conference on Re-search and Development in InformationRetrieval, paginas 45–54.

[Basile et al.2019] Basile, V., C. Bosco,E. Fersini, D. Nozza, V. Patti, F. M. R.Pardo, P. Rosso, y M. Sanguinetti. 2019.Semeval-2019 task 5: Multilingual Detec-tion of Hate Speech against Immigrants

4https://hatebase.org/

28

and Women in Twitter. En Proceedingsof the 13th International Workshop onSemantic Evaluation, paginas 54–63.

[De la Pena2019] De la Pena, G. L. 2019.GL at SemEval-2019 Task 5: IdentifyingHateful Tweets with a Deep Learning Ap-proach. En Proceedings of the 13th Inter-national Workshop on Semantic Evalua-tion, paginas 416–419.

[De la Pena y Rosso2019] De la Pena, G. L.y P. Rosso. 2019. DeepAnalyzer atSemEval-2019 Task 6: A Deep Learning-based Ensemble Method for IdentifyingOffensive Tweets. En Proceedings of the13th International Workshop on SemanticEvaluation, paginas 582–586.

[De la Pena y Rosso2020a] De la Pena, G. L.y P. Rosso. 2020a. Harmonic Mean ofRelative Frequencies of Terms and BERT-based Transfer Learning for MultilingualOffensive Language Detection. Submitted.

[De la Pena y Rosso2020b] De la Pena, G. L.y P. Rosso. 2020b. PRHLT-UPV atSemEval-2020 Task 12: BERT for Multi-lingual Offensive Language Detection. Ac-cepted at SemEval 2020.

[De la Pena y Rosso2020c] De la Pena, G. L.y P. Rosso. 2020c. PRHLT-UPV atSemEval-2020 Task 8: Study of Multi-modal Techniques for Memes Analysis.Accepted at SemEval 2020.

[De la Pena Sarracen et al.2018] De laPena Sarracen, G. L., R. G. Pons, C. E.Muniz-Cuza, y P. Rosso. 2018. HateSpeech Detection Using Attention-basedLSTM. En EVALITA@ CLiC-it.

[De la Pena Sarracen y Rosso2019] De laPena Sarracen, G. L. y P. Rosso. 2019.Aggressive Analysis in Twitter using aCombination of Models. En In Proceed-ings of the First Workshop for IberianLanguages Evaluation Forum (IberLEF2019), CEUR WS Proceedings.

[Dinakar, Reichart, y Lieberman2011]Dinakar, K., R. Reichart, y H. Lieberman.2011. Modeling the Detection of TextualCyberbullying. En Fifth InternationalAAAI Conference on Weblogs and SocialMedia.

[Erjavec y Kovacic2012] Erjavec, K. y M. P.Kovacic. 2012. “You Don’t Under-stand, This is a New War!” Analysis of

Hate Speech in News Web Sites’ Com-ments. Mass Communication and Society,15(6):899–920.

[Fortuna y Nunes2018] Fortuna, P. yS. Nunes. 2018. A Survey on Au-tomatic Detection of Hate Speech inText. ACM Computing Surveys (CSUR),51(4):1–30.

[Liu y Forss2014] Liu, S. y T. Forss. 2014.Combining N-gram based Similarity Anal-ysis with Sentiment Analysis in Web Con-tent Classification. En KDIR, paginas530–537.

[Muniz Cuza, De la Pena Sarracen, y Rosso2018]Muniz Cuza, C. E., G. L. De la Pena Sar-racen, y P. Rosso. 2018. AttentionMechanism for Aggressive Detection.En Proceedings of the Iberian LanguagesEvaluation Forum co-located with 35thConference of the Spanish Society forNatural Language Processing, (IberEval2018), CEUR Workshop Proceedings.Vol. 2150. Pages 114-118.

[Nobata et al.2016] Nobata, C., J. Tetreault,A. Thomas, Y. Mehdad, y Y. Chang.2016. Abusive Language Detection in On-line User Content. En Proceedings of the25th International Conference on WorldWide Web, paginas 145–153. InternationalWorld Wide Web Conferences SteeringCommittee.

[Robinson, Zhang, y Tepper2018] Robinson,D., Z. Zhang, y J. Tepper. 2018. HateSpeech Detection on Twitter: FeatureEngineering vs Feature Selection. EnEuropean Semantic Web Conference,paginas 46–49. Springer.

[Schmidt y Wiegand2017a] Schmidt, A. yM. Wiegand. 2017a. A Survey on HateSpeech Detection using Natural LanguageProcessing. paginas 1–10, 01.

[Schmidt y Wiegand2017b] Schmidt, A. yM. Wiegand. 2017b. A Survey on HateSpeech Detection using Natural Lan-guage Processing. En Proceedings of theFifth International Workshop on NaturalLanguage Processing for Social Media,paginas 1–10.

[Sharma et al.2020] Sharma, C., W. Paka,Scott, D. Bhageria, A. Das, S. Po-ria, T. Chakraborty, y B. Gamback.2020. Task Report: Memotion Analysis

29

1.0 @SemEval 2020: The Visuo-LingualMetaphor! En Proceedings of the 14th In-ternational Workshop on Semantic Evalu-ation (SemEval-2020), Barcelona, Spain,Sep. Association for Computational Lin-guistics.

[Zampieri et al.2019] Zampieri, M., S. Mal-masi, P. Nakov, S. Rosenthal, N. Farra, yR. Kumar. 2019. Predicting the Type andTarget of Offensive Posts in Social Me-dia. En Proceedings of the 2019 Confer-ence of the North American Chapter of theAssociation for Computational Linguistics(NAACL), paginas 1415–1420.

[Zampieri et al.2020] Zampieri, M.,P. Nakov, S. Rosenthal, P. Atanasova,G. Karadzhov, H. Mubarak, L. Der-czynski, Z. Pitenis, y c. Coltekin. 2020.SemEval-2020 Task 12: Multilingual Of-fensive Language Identification in SocialMedia (OffensEval 2020). En Proceedingsof SemEval.

[Zhang y Luo2019] Zhang, Z. y L. Luo. 2019.Hate Speech Detection: A Solved Prob-lem? The Challenging Case of Long Tailon Twitter. Semantic Web, 10(5):925–945.

30