Wikigrep distribuido: búsquedas avanzadas en la wikipedia

Abstract

In this project we created a regular expressions search engine that uses the Wikipedia database of articles. The system allows the use of to enter a regular expression and makes an asynchronous request to initialize an EC2 cluster; it searches for the pattern inside all the Wikipedia and then returns the result, displaying a list of all the occurrences of the pattern and a link to the Wikipedia Article. We used the Amazon Web Services, Java libraries to manipulate Wikipedia Articles, the Hadoop framework and a dataset of the Wikipedia Articles. We tested some regular expressions that couldn’t be searched for using neither traditional search engines nor the Wikipedia Search Engine. Our tests show that an advanced search engine could be cheap to implement providing high scalability through the use of cloud computing and data-intensive computing techniques.