12th International Conference on Information & Communication Technology and System (lCTS) 2019

Sentiment Analysis of Restaurant CustomerReviews on TripAdvisor using Naive Bayes

Rachmawan Adi LaksonoDepartment ofInformation Technology

ManagementInstitut Teknologi Sepuluh Nopember

Surabaya, Indonesiarachmawanadi@gmail.com

Kelly Rossa SungkonoDepartment ofInformatics

Institut Teknologi Sepuluh NopemberSurabaya, Indonesia

kelly@its.ac.id

Cahyaningtyas Sekar WahyuniDepartment ofInformatics

Institut Teknologi Sepuluh Nopemb erSurabaya, Indonesia

cahyaningtyas.18051@rnhs.its.ac.id

Riyanarto SarnoDepartment ofInformatics

Institut Teknologi Sepuluh NopemberSurabaya, Indonesiariyanarto@if.its.ac.id

Abstract- Sentiment analysis is one method for classifyingdocuments to identify positive or negative opinions. Customersatisfaction has an essential point for customer service.Customer behaviour is currently doing a lot of reviews in onlinemedia such as on trip advisor. A restaurant is a business thatrequires more attention in the service to consumers byimproving service to customers continuously. This study tries toclassify Surabaya restaurant customer satisfaction using NaiveBayes. Data sampling is crawling by using WebHarvy Tools.The result from this research shows that these two methods getthe customer response accurately and Naive Bayes method ismore accurate than TextBlob sentiment analysis with a differentaccuracy of 2.9%.

Keywords- customer satisfaction, Naive Bayes,sentiment analysis, textBlob

I. INTRODUCTION

Customer satisfaction is an opinion or feeling betweenexpectation and reality obtained by consumers [1]. Today,many customers write opinions in the form of reviews abouttheir obtained satisfaction on online media, such asTripAdvisor. Customer reviews on the online media becomeimportant as it might increase the popularity of the product orservice sold by the seller.

A restaurant is a business [2], [3] that prepares and servesfood for customers and exchanges for a certain amount ofmoney. Although improving quality through this method isconsidered useful, only some restaurants use customersatisfaction analysis to improve their services. Also, manyalgorithms might be used for doing the study [4], [5].

Restaurant customer satisfaction research through reviewson online media such as TripAdvisor is still rare. Generally,restaurant customer satisfaction analyses through productdata, nutrition data and food preparation [6]. One of theprevious studies is an analysis of hotel customer satisfactionin Ponorogo district. The study used hotel customer reviewdata on TripAdvisor [7].

Restaurant reviews on TripAdvisor are still in the form oftext, customer reviews are included in the text miningcategory, the results of these data will be classified into twovalues, positive or negative [8]. Retrieving data onTripAdvisor using WebHarvy software, for preprocessingreview data such as remove stopword, remove punctuation is

978-1-7281-2133-8/19/$31.00 ©2019 IEEE 49

done with the help ofPython, while for classifying data usingWaikato Environment for Knowledge Analysis (WEKA)software with the Naive Bayes method and also usingTextBlob which is a python-based sentiment analyzer tocompare. Naive Bayes is chosen because this method has beenwidely implemented in sentiment analysis [9]-[11] .

This aim of this study is to analyze restaurant customerreviews from online TripAdvisor in the best ten Surabayarestaurants. Another aim is to find the best method foranalyzing restaurant customer review data by comparingNaive Bayes method and TextBlob [12], [13] sentimentanalysis since the two methods have fundamental differencesin terms of calculations.

II . LITERATURE REVIEW

Customer satisfaction is an essential concern in the fieldof marketing and research in terms of consumer behaviour.As in the habits of hotel consumers when they get excellentservice, they will transmit to others mouth to mouth [14] .

Text mining or retrieval of data from a collection ofdocuments stores frequently with the help ofanalysis tools ormanuals [13] . Through the analysis process of several textmining perspectives, information can be produced that can beused to increase profits and services.

Sentiment analysis is used to find opinions from theauthor about a specified entity [15]. Sentiment analysis of areview is an opinion investigation ofa product [16] . The basisof sentiment analysis is using Natural Language Processing(NLP), text analysis and some computational portions toextract or omit unnecessary parts to see the pattern of thesentence negative or positive [17].

In the l Sth century, Reverend Thomas Bayes developeda method known as Naive Bayes that used probability andopportunity approaches. The workings of the Naive Bayesalgorithm can be seen in Equation (1). Naive Bayes calculatesfuture probability predictions from data or experiences thathave been given, based on the opportunity point ofview [18].One characteristic of the Naive Bayes Classification is theexistence of independent input variables which assume thepresence of an articular feature from a class that is mutuallyindependent ofother features [19] .

Real dataPositive Review Nezatlve Review

Positive TP FPReviewMethodsNegativeReview FN TN

DataReview

Data Mining :Crawling

Data

WebTripAdvisorRestaurantReview

Fig. 2. Data mining process

As a comparison, the results of this study are alsocompared to the TextBlob sentiment analysis, a sentimentanalyzer that has a Natural Language Toolkit (NLTK) andPattern processing basis [21]. TextBlob can also be used fortext mining, text processing modules for python winners, andeven text analysis. TextBlob also provides simple APIs forgeneral Natural Language Processing (NLP) processing suchas part-of-speech tagging, tokenizing sentences, noun phraseextraction, sentiment analysis, classification, translation [22] .

WEKA, is one ofthe workbenches that has many choicesof machine learning methods for research. Initiated since1992, the use of WEKA allows users to try and compareseveral different machine learning methods on the new dataset quickly, and there are several Algorithms available inseveral languages [23]. WEKA has been very accepted invarious academics, the business environment, to be used as adata mining research tool. The analysis is working onenvironment using WEKA 3.8.2.

One ofits abilities is the classification method either withsupervision or no supervision. Some popular classificationsare the Naive Bayes method, Decision tree J48, RandomForest, k-Nearest Neighbors (k-NN), Sequential MinimalOptimization (SMO) or also called the Support VectorMachine (SVM) method..

III. RESEARCH METHOD

This paper takes restaurant review data on TripAdvisor,especially restaurants in Surabaya by web crawling method,and will be analyzed using the Naive Bayes method on WEKAand a comparison with the TextBlob sentiment analysis. Thereare several steps of the research method.

A. Data collectionRetrieval of customer review data is done by crawling by

using the WebHarvy tool as in the methods Fig. 2. Data istaken from web TripAdvisor, as in Fig. 3. TripAdvisor is thelargest travel community review site on the web . This webwas introduced in the year of2000, and now covers 212.000hotels, more than 30.000 destinations, and 74.000 attractionsaround the world. Fig . 3 shows an overview ofreviews fromTripAdvisor.

In Fig . 3, there are 2 data shown, in the red box is the titleofthe review, while the green box is a review ofthe customer.This paper will use customer review data as a data source.

B. Data processing and analysisFig. 4 shows the work steps on processing and analysis

data . On the processing data and analysis, there are severalsteps that the procedure must be followed. The Trimlowercase process is done to change all the letters in smallbricks not mixed large and small so that the uniforms are likethis: "Good", "NoiSy" becomes "noisy" good.

(1)

(2)

(3)

(4)

Precision = TP(TP+FP)

Recall =~(TP+ FN )

F Score = (2xPrecision x Recall)(Precision+Recall)

Accurac = (TP+TN) (5)Y (TP+TN+FP+FN)

where: TP = the positive review of the real data is classifiedas the positive review obtained by the method

TN = the negative review ofthe real data is classifiedas the negative result obtained by the method

FP = the negative review ofthe real data is classifiedas the positive review obtained by the method

FN = the positive review of the real data is classifiedas the negative review obtained by the method

Methods of precision, recall, and accuracy is used tocheck the accuracy of the results of the process. A confusionmatrix is created to provide performance classification data.Elements of confusion matrix in Fig . 1 are True positive (TP)when both human and method predict are positive and TrueNegative (TN) when both human and method predict arenegative. False negative (FN) is used when the humanprediction is positive while method prediction is negative andFalse Positive (FP) is used when the human prediction isnegative while method prediction is positive [20] .

The level of accuracy between what the user wants andthe results of the system process is called Precision, can beseen in Equation (2), whereas Recall is the average successof the system in the process of finding information, listed inEquation (3), Precision and Recall calculations are used toavoid measurement errors for deviation values as shown inEquation (4) . Accuracy is the degree of truth between thepredictive value, and the actual value is shown in Equation(5) . Precision value is obtained by dividing TP with thepositive results obtained by the method. Because this paperanalyzes customer review, so the positive results are thereviews classifying as positive reviews. Recall value isobtained by dividing TP with positive results based on thereal data . F Score value uses the Recall value and Precisionvalue. The accuracy is the division of the amount of TP andTN to the amount of data.

where:

P(HJ Ix) = states the probability arises Hjif known x.P(xIHj) =The likelihood function of H, to xP(Hj) = Prior probabilityP(x) = evidence

Fig. I . Confusion matrix

50

Overview Rev ieVJs Location O&A Details

IW ll k lVlI 1Surabaya,Indonesia

f'i 4 3 •• 1 2

Yulic HV'1S urabaya,Indon esia

[3 3

Fig. 3. TripAdvisor Web

~ R e vie we d 1 week ago 0 v ia mobiler----------~ !?~J is;i~ L:! ~ __:~ i'ts- gura-rri'..-..-ba karl's-tiighly- rs co-ni"rri'..-nded!Ayam i)akai"isanotfier- - -:I recomendat ion. E a t with sornbct p e n c it a n d kangkung h o tplat e IL J

Date of v isit: December 2018

.~ T h a n k iw ilk iw il

~'~Hviewe(j 1 week ~g(}

NiceF o o d was delicious. Service was good. Not so expensive. You must t ry

th ..ir "Aya m Bakar" Serving t im.. quite fast. Overall ok >

Date o f v isit: November 20 18

.~ T h 'lnk YulleH'N

Preprocessing:-Trim lowercase

- -Removepunctuation-Stopword removal

~ ~Clasification Clas ification Pro cessProcess: :-Naive Bayes -TextBlob Sentimentclasification Analyzer

t tData Data

Clasificat ion : Clasification :Naive Bayes TextBlob

tCompare Result :-Naive Bayes &TextBlob

Data Mining:Web TripAdvisorRestaurant rev iew

Data review

Stopword removal is a process of removing words thatoften appear but do not have meaning in languages, such as"the", "a", "an", "in". Like the example below: "let me prefacethis review," said gardeners" became" let preface review,saying, mad keen gardeners ".

The remove punctuation process is the process ofremoving punctuation that often appears and usually does nothave much meaning like "-, /,:,? " After the preprocessing textis complete, the next process will be text calcification usingthe Naive Bayes method and the comparison using Blob Text.

IV. RESULT

Crawling techniques take the process of receivingcustomer review data on Tripadvisor, especially restaurantsin Surabaya. The data taken is data on restaurant names andreviews. The results of the data collection obtain 337 data,269 data are used for training data and 68 data for testing data .Some data resu lts from the data collection process as shownin TABLE I. There are three kinds of information in TABLEI, i.e. several reviews (No.), the name of the restaurant, andthe reviews. There are only ten reviews that are shown inTABLE I.

The next step is to prepare the review data for trainingand testing data. Trim lowercase or make the letters uniforminto lowercase letters. TABLE II are examples of trimlowercase processing resu lts. As seen in the first review ofTAB LE II, the sentence "About the steak, well, the taste wasnice" has been processed into "about the steak, well, the tastewas nice".

The next preprocessing is shown in Fig. 5. Fig. 5 explainsthe process of removing punctuation and words that oftenappear in the text but have no meaning. As in Fig. 5, anexamp le of the results of eliminating punctuation andremoving stop words.

Fig. 4. Data analysis process

51

Fig. 5. Remove stopwords proc ess

"nice and clean restaurant, good service. good food,the best is garlic steam fish.. so yummy... price also

ok"

"nice clean restaurantg~ service good food bestgarlic steam fish yummy price ok"

No. Restaurant Name Reviews

I Steak Hut Manyar Kertoarjo About the steak, well, the tastewas nice

2 Steak Hut Manyar Kertoarjo The price is standard

3 Steak Hut Manyar Kertoarjo I think, they need moreinnovation for their menu

4 Steak Hut Manyar Kertoarjo About the place, felt comfortable

5 Steak Hut Manyar Kertoarjo overall, worth to try thisrestaurant

6 Steak Hut Manyar Kertoarjo best luck

7 Steak Hut Manyar Kertoarjo Affordable price, good ambience,recommended for family event

8 Steak Hut Manyar Kertoarjo So many promotion from anycredit card

9 Steak Hut Manyar Kertoarjo Great service & affordable pricefor steak

10 Steak Hut Manyar Kertoarjo Usually I visited this restaurantonce in a month

No . Before Trim Lowercase After Trim Lowercase

IAbout the steak, well, the about the steak, well, the tastetaste was nice was nice

2 The price is standart the price is standart

3 I think, they need more i think , they need moreinnovation for their menu innovation for their menu

4 About the place, felt about the place , felt comfortablecomfortable

5 overall, worth to try this overall, worth to try thisrestaurant restaurant

6 best luck best luck

Affordable price , good affordable price, good ambience,7 ambience, recommended for recommended for family eventfamily event

8 So many promotion from any so many promotion from anycredit card credit card

9 Great service & affordable great service & affordable priceprice for steak for steak

10 Usually I visited this usually i visited this restaurantrestaurant once in a month once in a month

THE CLASSIFICATION OF TESTING DATA BY USING NAIvEBAYES AND TEXT BLOB

=== Confusion Matrix ===a b <-- class ified as63 141 a=-1181741 b = 1

Time taken to Build mod el: 0.27 secondsCorrectly Classified 237 88.1041 %Incorectly Classified 32 11.8959 %

No. Review NB TB EX

1. steak well taste nice I I I

2. place felt comfortable I I I

3. overall worth try restaurant I I I

4. best luck -I * I I

5. affordable price good ambience I I Irecommended family event

6. many promotion cred it card -I * I I

7. great service affordable price steak I I I

8. usually visi ted restaurant month I -I * I

9. recommended steak restaurantI I Isurabaya

10. crowded I I I

I!. cozy place decen t food I I I

12. nice environment atmosphere I I I

13. nice mea l larger portion perfect I I I

14. cozy place good service I I I

15. good food perfect salad I I I

16. chicken cordon blue delicious I I I

17. love I I I

18. thumbs up -I * I I

19. lunch fami ly surabaya I I I

20. good taste good service steak hut- I * I Imanyar ketoarjo

21. recommended I I I

22. great lunch family I I I

23. nice steak hut salad nz sirloi n steak - I * I I

24. come next -I * I I

25. convenience environment great tasteI I Isteak

26. also love burger new mozarella I I Ichicken schni tzel

TABLE III.

Fig. 6. Training data and resu lt

TRlM LoWERCAST RESULT

PART OF TH E RESULT OF D ATA COLLECTION

TABLE II.

TABLE!.

52

Naive Bayes TextBlob

TP FP 25 10 34 19

FN TN 9 24 2 13

56. you starving dont go there -1 1* -1

57. quite take time especially restaurant -1 1* -1crowded

58. lobster expensive -1 -1 -1

59. large restaurant crowded local foreign 1* 1* -1

60. prepare serve late menu -1 -1 -1

61. always crowded -1 1* -1

62. thing parking fee flat rate longer -1 1* -1expensive ticket is

63. place always crowd dinner time 1* 1* -1

64. variety foods many nothing special 1* 1* -1

65. enjoyed ice cream solace -I 1* -I

66. The food terribly overrated 1* -1 -I

67. The chicken dry tough marinad e taste -1 -1 -Ilike kecap manis

68. The price expensive quali ty poor -I -I -Irating

After going through preprocessing the data, a model wasmade with training data using WEKA 3.8.2 with Naive Bayesclassifiers. The model data is stored and will be used forevaluation with other data or testing data, Fig. 6 is the resultof the data model. As seen in Fig. 6, this research obtains 63TN and 174 TP by using Naive Bayes . The total of reviewsthat are correctly classified is 237 reviews .

After training data with the Naive Bayes method, then re­evaluate the model with testing data. The testing data was alsotested in the TextBlob Sentiment analyzer for comparison ofaccuracy with the Naive Bayes method. The following is theresult of testing data with the Naive Bayes and TextBlobmodels that have been implemented written as in TABLE III.TABLE III shows 68 testing data. Each review will beclassified by using Naive Bayes as the method of thisresearch, TextBlob as the compari son method, and by anexpert. 1 is denoted the positive review and -1 is denoted asthe negati ve review. The sign (*) means the result is differentfrom the expert judgment.

All the results obtained by Naive Bayes and Textblob willbe analyzed by using confusion matrix. TABLE IV shows theconfusion matrix of those two sentiment analysis method.Based on TABLE IV, Naive Bayes obtains 25 TP and 24 TN.Then, TextBlob collects 34 TP and 13 TN. Naive Bayes hasa higher FN number than TextBlob.

27. thanks good service delicious beef -1* 1 1steak i will back here

28. thanks lot -I * 1 I

29. steak hut restaurant specialties steak -1* 1 1menu

30. usually promotion price cooperation 1 -1* 1credit card certain bank

good delicious food clean spacious31. place great large groups good friendly 1 1 1

service

32. good food good price comfort place 1 1 1

33. must visit indonesi an food surabaya 1 1 1

34. grilled fish shrimp etc. I I I

35. think need innovation menu -1* 1 1

36. crowded visit weekend 1* 1* -I

37. area nonsmoking smoking small 1* -I -Ispace

38 come thought small restaurant totally -1 -1 -1wrong

long time tried dinner fish bit small

39. tempe also small port ion nasi goreng 1* -I -Imerah amazed us super tiny bitportion small bowl its

40. food price expensive quality average -1 -1 -1portion small

41. sorry coming back again -1 -1 -1

42. disappointed female staff -1 -1 -1

43. want ask spoon serve food without -1 1* -1spoon take food and one coming

44. and response -1 1* -1

45. look back and yell louder voice -1 1* -1tunggu sebentar

46. end she is coming table all -1 1* -1

47. and sometime ask staff take food -1 -1 -1without spoon food

48. know staff makes bad experience -1 -1 -1coming here

49. ac working well here -1 1* -1

50. not cold -1 1* -1

little bit expensive crabs prices beside 1*51. park ing area difficult imagine raining 1* -1

season

52. noisy lots loud people here -1 1* -1

53. privacy dont come romantic dinner -1 1* -1

54. good seafood want go prepared noisy 1* 1* -1evening

55. restaurant quite big crowded 1* 1* -1

53

where: NB

TB

EX

TABLE IV.

: Naive Bayes

: TextBlob

: Expert

C ONFUSION M ATRIX OF N AiVE B AYES AND T EXTBLOB

ACKNOWLEDGMENT

The authors would like to thank to Institut TeknologiSepuluh Nopember, The Ministry of Research, Technologyand Higher Education of Indonesia, Direktorat Riset danPengabdian Masyarakat, and Direktorat Jenderal PenguatanRiset dan Pengembangan Kementerian Riset, Teknologi, danPendidikan Tinggi Republik Indonesia for supporting theresearch

Conversely, Naive Bayes has a lower FP number thanTextBlob. The accuracy of the confusion matrix listed inTABLE IV is processed by using Equation (5). The accuracyofNaive Bayes is shown in Equation (6), and the accuracy ofTextBlob is shown in Equation (7) . The results of thecalculation of the above formula for the Naive Bayes methodhave data accuracy of 72.06%, whereas by using theTextBlob analyzer obtained an accuracy of 69.12%. Fromthese results, the Naive Bayes method using WEKA tools hasa higher percentage of accuracy than the TextBlob sentimentanalysis.

V. CONCLUSION

Based on the result of this paper that done on restaurantcustomer reviews in Surabaya, customer satisfaction analysismight be used by the Naive Bayes classification method andTextBlob sentiment analysis that might be able to learnsentiment from customers, since customer satisfaction isessential in terms of the restaurant business. The results alsoindicate that the Naive Bayes method has a value of 72.06%accuracy and is slightly better (2.94%) than TextBlobsentiment analysis. Further research can be done byincreasing the number and variety ofreview data, or by othermethods, to increase the value of accuracy.

A. Hasan, S. Moin , A. Karim , and S. Shamshirband, "Machinelearning-based sentiment analysis for twitter accounts,"Mathematical and Computational Applications, vol. 23, no . I, p.11,2018.

K Berezina, A. Bilgihan, C. Cobanoglu, and F. Okurnu s,"Understanding satisfied and dissatisfi ed hotel customers: textmining of online hotel review s," Journal ofHospitality Marketing& Management, vol. 25, no. I , pp . 1-24,2016.

R. Feldman, "Techniques and applications for sentiment analysis ,"Communications ofthe ACM, vol. 56, no. 4, pp . 82-89,2013.

E. Haddi, X. Liu, and Y. Shi, "The role of text pre-processing insentiment analysis," Procedia Computer Science, vol. 17, pp. 26­32,2013.

R.-E. Fan, K-W. Chang, C.-J. Hsieh , X.-R . Wang, and C.-J. Lin,"LIBLINEAR: A library for large linear classific ation," Journal ofmachine learning research, vol. 9, no. Aug, pp. 1871-1874,2008.

M. Jupri and R. Sarno, "Taxpayer compliance classification usingC4. 5, SYM, KNN, Naive Bayes and MLP," in 2018 InternationalConference on Information and Communications Technology(ICOIACT), 2018, pp. 297-303 .

T. Pranckevicius and V. Marcinkevicius, "Comparison of naivebayes, random forest, decision tree, support vector machines, andlogistic regression classifiers for text reviews classification,"Baltic Journal ofModern Computing, vol. 5, no . 2, pp. 221-232,2017 .

D. L. Gupta , A. K Malv iya, and S. Singh, "Performance analys isofclassification tree learning algorithms," International Journal ofComputer Applications, vol. 55, no. 6,2012.

B. Agarwal, N. Mittal , P. Bansal, and S. Garg, "Sentiment analysisusing common-sense and context information," Computationalintelligence and neuroscience, vol. 2015, p. 30, 2015 .

S. Vijayarani, R. Janani , and others, "Text mining: open sourcetokenization tools-an analysis, " Advanced ComputationalIntelligence: An International Journal (ACI/) , vol. 3, no. I , pp . 37­47,2016.

Comment Titles in Booking. corn Using Probabilistic LatentSemantic Analys is," in 2018 6th International Conference onInformation and Communication Technology (lCoICT), 2018, pp.514-519.

B. Y. Pratama and R. Sarno, "Personality classification based onTwitter text using Naive Bayes, KNN and SYM," in Data andSoftware Engineering (ICoDSE), 2015 International Conferenceon,2015,pp.170-174.

F. S. Fitri , M. N. S. Si, and C. Setianingsih, "Sentiment Analysison the Level of Customer Satisfaction to Data Cellular ServicesUsing the Naive Bayes Class ifier Algorithm," in 2018 IEEEInternational Conference on Internet of Things and IntelligenceSystem (IOTAIS), 2018, pp . 201--206.

U. W. Wijayanto and R. Sarno, "An Experimental Study ofSupervised Sentiment Analysis Using Gaussian Naive Bayes," in2018 International Seminar on Application for Technology ofInformation and Communication, 2018, pp. 476--481.

H. Gauba, P. Kumar, P. P. Roy, P. Singh, D. P. Dogra, and B.Raman, "Prediction of advert isement preference by fusing EEGresponse and sentiment analysis," Neural Networks, vol. 92, pp .77--88,201 7.

M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and 1.H. Witten, "The WEKA data mining software : an update," ACMSIGKDD explorations newsletter, vol. II , no. I, pp. 10-18,2009.

D. Grabner, M. Zanker, G. Fliedl, M. Fuchs, and others,Classification of customer reviews based on sentiment analysis.Citcsccr, 2012 .

[22]

[20]

[21]

[23]

[17]

[13]

[19)

[14]

[16]

[15]

[12]

[10)

[9]

[18]

[8]

[11)

(6)

(7)

K Ilieska, "Customer satisfaction index--as a base for strategicmarketing management," TEM Journal, vol. 2, no . 4, p . 327, 2013 .

Y. Caesarita and K R. Sungkono, "Identifying Bottlenecks andFraud of Business Process using Alpha ++ and Heuristic MinerAlgorithms (Case study : CV . Wicaksana Artha )," pp . 143-148,2017 .

REFERENCES

R. Sarno and K R. Sungkono, "Coupled Hidden Markov Modelfor Process Disco very of Non-Free Choice and Invisible PrimeTasks," Procedia Computer Science, vol. 124, pp. 134-141,2018.http ://doi.org/IO.lOI6/j.procs.2017.12.139.

Accuracy Naive Bayes = 72.06%

Accuracy TextBlob = 69.12%

[2]

[I)

M. Fikri and R. Sarno, "A Comparative Study of SentimentAnalysis using SVM and SentiWordNet," Indonesian Journal ofElectrical Engineering and Computer Science (lJEECS), vol. 13,no. 3,2019.

B. Rintyarna, R. Sarno, and C. Fatichah, "Enhancing theperformance of sentiment analysis task on product reviews byhandling both local and global context," International Journal ofInformation and Decision Sciences, vol. 11,2018.

J. E. Mills and L. Thomas, "Assessing customer expectations ofinformation provided on restaurant menus: A confirmatory factoranalysis approach," Journal ofHospitality & Tourism Research ,vol. 32, no . I, pp . 62-88,2008.

[7] D. A. K Khotimah and R. Sarno, "Sentiment Detection of

[4]

[6]

[5]

[3)

54