InformationRetrieval

Instructor: Dr. A. Nikabadi

Course content: CS276 Standford University

Semester: Fall 2022

This project is for Information Retrieval course which aims to implement a search engine for both phrase queries and Free text queries on Fars News Dataset.

First Phase

1. Preprocessing on data (Noramlization, Tokenization, Stemming, Removing Stopwords)

2. Working with both most used NLP persian toolkits : hazm, parsivar

3. Created a positional inverted index

4. Used Zipf's law

5. Used Heaps law

6. Searching by Normal quries, Phrase Queries (used permuterm index), Boolean queries

7. Ranking results

Second phase

1. Show words in vector representation

2. Compute tf-idf

3. Compute cosine similarity between query terms and documents

4. Used Index elimination techniques such as creating champion list

5. Rank results based on most relevent results

---

Contributors : Rojina kashefi & Leili Barekatein