web_scrapping

This is an industry level web scrapper made in python written in jupyter notebook. Purpose of this program is to collect data of 86K products for ML based recommender system. It is made using BeautifulSoup, requests, pandas, and other libraries, and data structres of python. For collecting data at enormous rate by utilizing full capacity of machine, concepts of multiprocessing and code optimizations are used. For better readability of code, concepts modules, classes, and objects of python are used. For storing data directly in SQL database, pipelines are created to streamline process of scrapping, collecting, cleaning, and storing data into database.