Data Crawling Projects

This repository showcases data crawling projects for previous research.

Image Copyright ©: TarikVision

The following tools were used to accomplish different crawling tasks

• Requests
• BeautifulSoup
• Scrapy
• Scrapy-Splash
• Aquarium
• Selenium
• Zyte - Smart Proxy Manager
• Tor-Privoxy Docker
• Spotify API (Spotipy)

Python scripts with interesting functionalities

OOP Crawling Workflow with Selenium

• GrammyCrawl: artistCrawl.py, artistCrawl_deployment.py
• SpotifyCrawl: aboutCrawl.py, aboutCrawl_deployment.py
• ImdbProfessionalCrawl: imdb_role_crawler.py, imdb_role_deployment.py

OOP Crawling Workflow with Spotipy API

• SpotifyCrawl: spotifyCrawl.py, spotifyCrawl_deployment.py, acousticAttr.py

Scrapy-Splash Infinite Scrolling (Lua Language)

• IMDbCrawl: imdb_user_review.py

Scrapy Login with Customised Horizontal/Vertical Crawling

• BeerAdvocateCrawl: beerReview.py

Scrapy on Complex Pattern Site

• IMDbProfessionalCrawl: imdb_award.py

Vertical Crawling with Scrapy

• RateYourMusicCrawl: 00_genre-level.py

Horizontal Crawling with Scrapy

• MetacriticCrawl: mt_movie_url.py

Vertical/Horizontal Crawling with Scrapy

• BeerAdvocateCrawl: forum.py

Scrape Data from GoogleSearch

• GoogleSearch: artistSearchFromMusicBrainz.py, grammyGoogleSearch.py

Data Parsing with BeautifulSoup

• RateYourMusicParsing: 01_parse_data.py