Scrapy Samples

This project contains sample scraping projects/tutorials as detailed in my blog. If you need help with downloading Scrapy and installing the dependencies, please watch this video - http://www.youtube.com/watch?v=eEK2kmmvIdw&feature=plcp

BaseSpider

• This is a simple tutorial on how to write a crawler using Scrapy (BaseSpider) to scrape and parse Craigslist Nonprofit jobs in San Francisco and store the data to a CSV file.
• Tutorial: http://mherman.org/blog/2012/11/05/scraping-web-pages-with-scrapy/
• Video: http://www.youtube.com/watch?v=1EFnX1UkXVU

CrawlSpider

• This tutorial adds to the BaseSpider code, in order to build a recursive crawler to scrape multiple pages. Make sure you read the first tutorial first.
• Tutorial: http://mherman.org/blog/2012/11/08/recursively-scraping-web-pages-with-scrapy/
• Video: http://www.youtube.com/watch?v=P-_TpZ54Vcw

To-do

1. Add more use cases