Opinion Mining, Sentiment Analysis, and Opinion Spam Detection

Feature-Based Opinion Mining and Summarization
(or Aspect-Based Sentiment Analysis and Summarization)
Detecting Fake Reviews
(Media coverage: The New York Times, The Economist, BusinessWeek and more ... )

Opinion Lexicon --------- Datasets --------- Talks --------- Publications

New Book: Sentiment Analysis: mining opinions, sentiments, and emotions. Cambridge University Press, 2015.

Book: Sentiment Analysis and Opinion Mining (Introduction and Survey), Morgan & Claypool, May 2012.

See "Feature-Based Opinion Mining and Summmarization" in Microsoft Live/Bing Search and Google Product Search (paper).

• Note: I don't know the techniques used by Microsoft Live/Bing (9/28/2007), but Google has a paper. To see the model, please check out (Hu and Liu, KDD-2004) and (Liu et al, WWW-2005) below, or the books above (better). Try search for a camera and click on reviews. You will see summarized user opinions on product features/aspects in a bar chart.

NLP Handbook Chapter: Sentiment Analysis and Subjectivity, 2nd Edition, Eds: N. Indurkhya and F.J. Damerau, 2010.

Opinion Parser: my sentiment analysis system has been licensed to two companies.

• The system analyzes sentiments, opinions and emotions, extracts sentiment targets: entities, topics and their aspects/features, and handles comparative sentences.
• I cannot make the system open-source due to its commercial use. If you want to know how it works, please read my new sentiment analysis book, which gives a lot of details.

Tutorial: Sentiment Analysis Tutorial - (references), given at AAAI-2011, August 8, 2011 - (Check out the new book)

Interesting Piece from New Republic: If you want to be a successful novelist, should you be sentimental in your writing or not?

Recent Keynote and Invited Talks (not updated) (Older Talks)

1. Invited Talk. “Sentiment Analysis with Lifelong Learning.” ETS, December 7, 2015.
2. Invited Talk. “Sentiment Analysis with Lifelong Learning.” Brigham Young University, December. 3, 2015.
3. Keynote speech. “Sentiment Analysis, Lifelong Learning and Intelligent Personal Assistants.” The 2015 Conf. on Technologies and Applications of Artificial Intelligence (TAAI-2015). Taiwan, Nov. 20-22, 2015.
4. Invited talk. “Sentiment analysis and lifelong machine learning.” Frontiers in Computational Mathematics: AMS Central Fall Sectional Meeting, October 2-4, 2015.
5. Keynote speech. “The State of Sentiment.” Sentiment Analysis Symposium, New York City, July 15-16, 2015.
6. Invited tutorial. "Sentiment analysis: mining opinions, sentiments, and emotions." Sentiment Analysis Symposium, New York City, July 15-16, 2015.
7. Keynote speech. “Deception Detection via Pattern Mining of Web Usage Behavior” Workshop on Data mining For Big Data: Applications, Challenges & Perspectives, Morocco, March 25, 2015
8. Keynote speech. “Social Media Analysis via Continuous Learning.” Adobe Text Analytics Summit, Feb 26, 2015.
9. Older Talks

1. Introduction

This work is in the area of sentiment analysis and opinion mining from social media, e.g., reviews, forum discussions, and blogs. In our KDD-2004 paper, we proposed the Feature-Based Opinion Mining model, which is now also called Aspect-Based Opinion Mining (as the term feature here can confuse with the term feature used in machine learning). The output of such opinion mining is a feature-based opinion summary or aspect-based opinion summary. The commonly known sentiment classification is a sub-task. Our current work is in two main areas, which reflect two kinds of opinions (or evaluations)

• Mining regular (or direct) opinions. Ex: (1). This camera is great. (2). After taking the drug, I got stomach pain.
• Mining comparative opinions. Ex: Coke tastes better than Pepsi.

Since 2006, we have also worked on

• Fake review and opinion spam detection. Fake reviews are also called bogus reviews or fraudulent reviews. See the papers [WWW-2007, WSDM-2008, CIKM-2010a, CIKM-2010b, WWW-2012]

2. Sentiment Analysis or Mining of Regular Opinions

In this research, we aim to mine and to summarize online opinions in reviews, tweets, blogs, forum discussions, etc. Specifically, we mine features or aspects of entities (e.g., products) or topics on which people have expressed their opinions and determine whether the opinions are positive or negative. For opinion summarization, we advocate the quantitative aspect and the target of opinions because 50% of the people say something is bad is not the same as 5% say it is bad.

Abstraction of the problem: Feature-based opinion mining and summarization (aspect-based opinion mining and summarization) of multiple reviews (KDD-04 and WWW-05)
Formal definitions can be found in my book "Sentiment Analysis and Opinion Mining". They are based on several of our papers in 2004 and 2005. The abstraction provides a model of online opinions, describes what should be extracted from opinion sources (e.g., reviews, forums, and blogs) and how the results may be organized and presented to the user. The main mining tasks are:

• mining entities and their features (or aspects) that have been commented on or evaluated by people,
• determining whether the comment/opinion on each entity feature (or aspect) is positive, negative or neutral (aspect-based sentiment classification), and
• summarizing the results.

Opinion Lexicon (or Sentiment Lexicon)

• Opinion Lexicon: A list of English positive and negative opinion words or sentiment words (around 6800 words). This list was compiled over many years starting from our first paper (Hu and Liu, KDD-2004).
• Comparative words: A list of non-standard English compariative words and phrases for sentiment analysis. This list was compiled over many years starting from our first paper (Jindal and Liu, SIGIR-2006)
• Although necessary, having an opinion lexicon is far from sufficient for accurate sentiment analysis. See this paper: Sentiment Analysis and Subjectivity or the Sentiment Analysis book.
• Try Search for the Best Restaurant based on specific aspects, e.g., "best burger," "friendliest service." The system is a demo, which uses the lexicon (also phrases) and grammatical analysis for opinion mining.

Data Sets

• Annotated: Customer Review Datasets (5 products) associated with the paper (Hu and Liu, KDD-2004).
• Annotated: Additional Customer Review Datasets (9 products) some used in (Ding, Liu and Yu, WSDM-2008), which improves the lexicon-based method proposed in (Hu and Liu, KDD-2004)
• Annotated: More Customer Review Datasets (3 products) used in (Liu et al., IJCAI-2015)
• Amazon Product Review Data (more than 5.8 million reviews) used in (Jindal and Liu, WWW-2007, WSDM-2008; Lim et al, CIKM-2010; Jindal, Liu and Lim, CIKM-2010; Mukherjee et al. WWW-2011; Mukherjee, Liu and Glance, WWW-2012) for opinion spam (fake review) detection. You can also use it for sentiment analysis. It has information about reviewers, review texts, ratings, product info, etc. Due to the large file size, you may need to use Download Accelerator Plus (DAP) to download. If you use this data, please cite (Jindal and Liu, WSDM-2008).
• Pros and cons dataset used in (Ganapathibhotla and Liu, Coling-2008) for determining context (aspect) dependent sentiment words, which are then applied to sentiment analysis of comparative sentiences (comparative sentence dataset). The same form of Pros and Cons data was also used in (Liu, Hu and Cheng, WWW-2005).
• Comparative sentence dataset used in (Jindal and Liu, SIGIR-06) and (Jindal and Liu, AAAI-2006). Comparative sentence dataset used in (Ganapathibhotla and Liu, Coling-2008).
• Blog author gender classification data set associated with the paper (Mukherjee and Liu, EMNLP-2010)
• Debate data set used in (Mukherjee and Liu, ACL-2013; Mukherjee et al. ACL-2013 ).
• Yelp Filtered Reviews for Opinion spam or fake detection associated with the paper (Mukherjee et al.ICWSM-2013).

Publications - (sentiment analysis)                Publications - (opinion spam or fake review detection)

1. Konstantin Bauman, Bing Liu, and Alexander Tuzhlin. Aspect Based Recommendations: Recommending Items with the Most Valuable Aspects Based on User Reviews. To appear in Proceedings of SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2017). Halifax, Nova Scotia - Canada, August 13 - 17, 2017

2. Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, Bing Liu. Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory. arXiv:1704.01074 [cs.CL], 2017. This paper has been reported extensively in Tech News Media recently (April and May 2017).

3. Lei Shu, Hu Xu, and Bing Liu. Lifelong Learning CRF for Supervised Aspect Extraction. To appear in Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL-2017, short paper), July 30-August 4, 2017, Vancouver, Canada.

4. Lei Shu, Bing Liu, Hu Xu, and Annice Kim. Lifelong-RL: Lifelong Relaxation Labeling for Separating Entities and Aspects in Opinion Targets. Proceedings of 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP-2016), November 1–5, 2016, Austin, Texas, USA.

5. Shuai Wang, Zhiyuan Chen, and Bing Liu. Mining Aspect-Specific Opinion using a Holistic Lifelong Topic Model. Proceedings of the International World Wide Web Conference (WWW-2016), April 11-15, 2016, Montreal, Canada.

6. Qian Liu, Bing Liu, Yuanlin Zhang, Doo Soon Kim and Zhiqiang Gao. Improving Opinion Aspect Extraction using Semantic Similarity and Aspect Associations. Proceedings of Thirtieth AAAI Conference on Artificial Intelligence (AAAI-2016), February 12–17, 2016, Phoenix, Arizona, USA.

7. Zhiyuan Chen, Nianzu Ma and Bing Liu. Lifelong Learning for Sentiment Classification. Proceedings of the 53st Annual Meeting of the Association for Computational Linguistics (ACL-2015, short paper), 26-31, July 2015, Beijing, China.

8. Qian Liu, Zhiqiang Gao, Bing Liu and Yuanlin Zhang. Automated Rule Selection for Aspect Extraction in Opinion Mining. Proceedings of International Joint Conference on Artificial Intelligence (IJCAI-2015), July 25-31, 2015.

9. Huayi Li, Arjun Mukherjee, Jianfeng Si and Bing Liu. Extracting Verb Expressions Implying Negative Opinions. Proceedings of Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI-15). 2015

10. Zhiyuan Chen, Arjun Mukherjee, and Bing Liu. Aspect Extraction with Automated Prior Knowledge Learning. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL 2014), June 22-27, Baltimore, USA.

11. Qian Liu, Zhiqiang Gao, Bing Liu and Yuanlin Zhang. A Logic Programming Approach to Aspect Extraction in Opinion Mining. Proceedings of IEEE/WIC/ACM International Confernece on Web Intelligence (WI-2013), 2013.

12. Zhiyuan Chen, Arjun Mukherjee, Bing Liu, Meichun Hsu, Malu Castellanos, and Riddhiman Ghosh. Exploiting Domain Knowledge in Aspect Extraction. Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP-2013), October 18-21, 2013, Seattle, USA.

13. Arjun Mukherjee, Vivek Venkataraman, Bing Liu, and Sharon Meraz. Public Dialogue: Analysis of Tolerance in Online Discussions. Proceedings of The 51st Annual Meeting of the Association for Computational Linguistics (ACL-2013), August 4-9, 2013, Sofia, Bulgaria.

14. Arjun Mukherjee, Bing Liu. Discovering User Interactions in Ideological Discussions. Proceedings of The 51st Annual Meeting of the Association for Computational Linguistics (ACL-2013), August 4-9, 2013, Sofia, Bulgaria.

15. Jianfeng Si, Arjun Mukherjee, Bing Liu, Qing Li, Huayi Li, and Xiaotie Deng. Exploiting Topic based Twitter Sentiment for Stock Prediction. Proceedings of The 51st Annual Meeting of the Association for Computational Linguistics (ACL-2013, short paper), August 4-9, 2013, Sofia, Bulgaria.

16. Zhiyuan Chen, Bing Liu, Meichun Hsu, Malu Castellanos, and Riddhiman Ghosh. Identifying Intention Posts in Discussion Forums. Proceedings of The 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT-2013), June 9-15, 2013, Atlanta, USA.

17. Bing Liu. Sentiment Analysis and Opinion Mining. Morgan & Claypool Publishers, May 2012.

18. Arjun Mukherjee and Bing Liu. Modeling Review Comments. Proceedings of 50th Annual Meeting of Association for Computational Linguistics (ACL-2012), July 8-14, 2012, Jeju, Republic of Korea.

19. Arjun Mukherjee and Bing Liu. Aspect Extraction through Semi-Supervised Modeling. Proceedings of 50th Annual Meeting of Association for Computational Linguistics (ACL-2012), July 8-14, 2012, Jeju, Republic of Korea.

20. Arjun Mukherjee and Bing Liu. Mining Contentions from Discussions and Debates. Proceedings of SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2012), Aug. 12-16, 2012, Beijing, China.

21. Lei Zhang, Riddhiman Ghosh, Mohamed Dekhil, Meichun Hsu, Bing Liu. Combining Lexicon-based and Learning-based Methods for Twitter Sentiment Analysis. HP Labs Technical Report, 2011.

22. Lei Zhang and Bing Liu. "Extracting Resource Terms for Sentiment Analysis," Proceedings of the 5th International Joint Conference on Natural Language Processing (IJCNLP-2011), November 8-13, 2011, Chiang Mai, Thailand.

23. Zhongwu Zhai, Bing Liu, Lei Zhang, Hua Xu, Peifa Jia. Identifying Evaluative Opinions in Online Discussions. Proceedings of AAAI-2011, San Francisco, USA, August 7-11, 2011.

24. Lei Zhang and Bing Liu. "Identifying Noun Product Features that Imply Opinions." ACL-2011 (short paper), Portland, Oregon, USA, June 19-24, 2011.

25. Guang Qiu, Bing Liu, Jiajun Bu and Chun Chen. "Opinion Word Expansion and Target Extraction through Double Propagation." Computational Linguistics, March 2011, Vol. 37, No. 1: 9.27.

26. Zhongwu Zhai, Bing Liu, Hua Xu, Peifa Jia. "Constrained LDA for Grouping Product Features in Opinion Mining." Proceedings of PAKDD-2011, Shenzhen, China, 2011. (Best Paper Award)

27. Lei Zhang and Bing Liu. "Entity Set Expansion in Opinion Documents." Proceedings of the ACM Conference on Hypertext and Hypermedia (HT-2011), Eindhoven, Netherlands, June 6-9, 2011.

28. Zhongwu Zhai, Bing Liu, Hua Xu and Peifa Jia. "Clustering Product Features for Opinion Mining." Proceedings of Fourth ACM International Conference on Web Search and Data Mining (WSDM-2011), Feb. 9-12, 2011, Hong Kong, China.

29. Arjun Mukherjee and Bing Liu. "Improving Gender Classification of Blog Authors." Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP-10). Oct. 9-11, 2010, MIT, Massachusetts, USA.

30. Xiaowen Ding and Bing Liu. "Resolving Object and Attribute Coreference in Opinion Mining." Proceedings of the 23rd International Conference on Computational Linguistics (COLING-2010), August 23-27, Beijing, China.

31. Zhongwu Zhai, Bing Liu, Hua Xu and Peifa Jia. "Grouping Product Features Using Semi-Supervised Learning with Soft-Constraints" Proceedings of the 23rd International Conference on Computational Linguistics (COLING-2010), August 23-27, Beijing, China.

32. Lei Zhang and Bing Liu. "Extracting and Ranking Product Features in Opinion Documents." Proceedings of the 23rd International Conference on Computational Linguistics (COLING-2010), August 23-27, Beijing, China.

33. Bing Liu. "Sentiment Analysis: A Multifaceted Problem." Invited paper, IEEE Intelligent Systems, 25(3), 2010, pp. 76-80.

34. Bing Liu. "Sentiment Analysis and Subjectivity." Invited Chapter for the Handbook of Natural Language Processing, Second Edition. March, 2010.

35. Ramanathan Narayanan, Bing Liu and Alok Choudhary. "Sentiment Analysis of Conditional Sentences." Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP-09). August 6-7, 2009. Singapore.

36. Guang Qiu, Bing Liu, Jiajun Bu and Chun Chen. "Expanding Domain Sentiment Lexicon through Double Propagation." Proceedings of the 21st International Joint Conference on Artificial Intelligence (IJCAI-09), Pasadena, California, USA, July 11-17, 2009.

37. Xiaowen Ding, Bing Liu and Lei Zhang. "Entity Discovery and Assignment for Opinion Mining Applications," Proceedings of ACM SIGKDD Interntaional Conference on Knowledge Disocvery and Data Mining (KDD-09, industrial track), June 28-July 1, 2009, Paris.

38. Bing Liu. "Opinion Mining." Invited contribution to Encyclopedia of Database Systems, 2008.

39. Murthy Ganapathibhotla and Bing Liu. "Mining Opinions in Comparative Sentences." Proceedings of the 22nd International Conference on Computational Linguistics (Coling-2008), Manchester, 18-22 August, 2008.

40. Xiaowen Ding, Bing Liu and Philip S. Yu. "A Holistic Lexicon-Based Appraoch to Opinion Mining." Proceedings of First ACM International Conference on Web Search and Data Mining (WSDM-2008), Feb 11-12, 2008, Stanford University, Stanford, California, USA.

41. Xiaowen Ding and Bing Liu. "The Utility of Linguistic Rules in Opinion Mining." SIGIR-2007 (poster paper), 23-27 July 2007, Amsterdam.

42. Nitin Jindal and Bing Liu. "Identifying Comparative Sentences in Text Documents" Proceedings of the 29th Annual International ACM SIGIR Conference on Research & Development on Information Retrieval (SIGIR-06), Seattle 2006.

43. Nitin Jindal and Bing Liu. "Mining Comprative Sentences and Relations." Proceedings of 21st National Conference on Artificial Intellgience (AAAI-2006), July 16.20, 2006, Boston, Massachusetts, USA.

44. Bing Liu, Minqing Hu and Junsheng Cheng. "Opinion Observer: Analyzing and Comparing Opinions on the Web"Proceedings of the 14th international World Wide Web conference (WWW-2005), May 10-14, 2005, in Chiba, Japan.

45. Minqing Hu and Bing Liu. "Mining Opinion Features in Customer Reviews." Proceedings of Nineteeth National Conference on Artificial Intellgience (AAAI-2004), San Jose, USA, July 2004.

46. Minqing Hu and Bing Liu. "Mining and summarizing customer reviews." Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD-2004, full paper), Seattle, Washington, USA, Aug 22-25, 2004.

Publications - (Opinion spam or fake review detection)           (Check out my Opinion Spam Detection project homepage)

1. Huayi Li, Geli Fei, Shuai Wang, Bing Liu, Weixiang Shao, Arjun Mukherjee and Jidong Shao. Bimodal Distribution and Co-Bursting in Review Spam Detection. To appear in Proceedings of International World Wide Web Conference (WWW-2017), April 3-7, 2017, Perth, Australia.

2. Jing Wang, Clement. T. Yu, Philip S. Yu, Bing Liu, Weiyi Meng. “Diversionary comments under blog posts." Accepted. ACM Transactions on the Web (TWEB), 2015.

3. Huayi Li, Zhiyuan Chen, Arjun Mukherjee, Bing Liu and Jidong Shao. "Analyzing and Detecting Opinion Spam on a Large-scale Dataset via Temporal and Spatial Patterns." Short paper at ICWSM-2015, 2015.

4. Huayi Li, Arjun Mukherjee, Bing Liu, Rachel Kornfieldz and Sherry Emery. Detecting Campaign Promoters on Twitter using Markov Random Fields. to appear in Proceedings of IEEE International Conference on Data Mining (ICDM-2014), December 14-17, 2014.

5. Huayi Li, Zhiyuan Chen, Bing Liu, Xiaokai Wei and Jidong Shao. Spotting Fake Reviews via Collective Positive-Unlabeled Learning. to appear in Proceedings of IEEE International Conference on Data Mining (ICDM-2014, short paper), December 14-17, 2014.

6. Tieyun Qian, Bing Liu. Identifying Multiple Userids of the Same Author. Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP-2013), October 18-21, 2013, Seattle, USA.

7. Arjun Mukherjee, Abhinav Kumar, Bing Liu, Junhui Wang, Meichun Hsu, Malu Castellanos, and Riddhiman Ghosh. Spotting Opinion Spammers using Behavioral Footprints. Proceedings of SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2013), August 11-14 2013 in Chicago, USA.

8. Geli Fei, Arjun Mukherjee, Bing Liu, Meichun Hsu, Malu Castellanos, and Riddhiman Ghosh. Exploiting Burstiness in Reviews for Review Spammer Detection. Proceedings of The International AAAI Conference on Weblogs and Social Media (ICWSM-2013), July 8-10, 2013, Boston, USA.

9. Arjun Mukherjee, Vivek Venkataraman, Bing Liu, and Natalie Glance. What Yelp Fake Review Filter Might Be Doing. Proceedings of The International AAAI Conference on Weblogs and Social Media (ICWSM-2013), July 8-10, 2013, Boston, USA.

10. Arjun Mukherjee, Bing Liu, and Natalie Glance. Spotting Fake Reviewer Groups in Consumer Reviews. International World Wide Web Conference (WWW-2012), Lyon, France, April 16-20, 2012.

11. Guan Wang, Sihong Xie, Bing Liu, Philip S. Yu. Identify Online Store Review Spammers via Social Review Graph. ACM Transactions on Intelligent Systems and Technology, accepted for publication, 2011.

12. Guan Wang, Sihong Xie, Bing Liu, Philip S. Yu. Review Graph based Online Store Review Spammer Detection. ICDM-2011, 2011.

13. Arjun Mukherjee, Bing Liu, Junhui Wang, Natalie Glance, Nitin Jindal. Detecting Group Review Spam. WWW-2011 poster paper, 2011.

14. Ee-Peng Lim, Viet-An Nguyen, Nitin Jindal, Bing Liu and Hady Lauw. "Detecting Product Review Spammers using Rating Behaviors." The 19th ACM International Conference on Information and Knowledge Management (CIKM-2010, full paper), Toronto, Canada, Oct 26 - 30, 2010.

15. Nitin Jindal, Bing Liu and Ee-Peng Lim. "Finding Unusual Review Patterns Using Unexpected Rules." The 19th ACM International Conference on Information and Knowledge Management (CIKM-2010, short paper), Toronto, Canada, Oct 26 - 30, 2010.

16. Nitin Jindal and Bing Liu. "Opinion Spam and Analysis." Proceedings of First ACM International Conference on Web Search and Data Mining (WSDM-2008), Feb 11-12, 2008, Stanford University, Stanford, California, USA.

17. Nitin Jindal and Bing Liu. "Review Spam Detection." Proceedings of WWW-2007 (poster paper), May 8-12, Banff, Canada.

Created on May 15, 2004 by Bing Liu; and Minqing Hu.