A web scraper that scrapes data related to blockchain technology such as blogs, videos, tweets, hackathons, courses, and projects, in a regular time interval. A node web server with an express HTTP framework is used to serve data. The scraped data is stored in the mongo database. Mongoose ODM is used for this purpose.
GET /api/blog
GET /api/project
GET /api/hackathon
GET /api/videos
GET /api/courses
GET /api/udemy
GET /api/posts
To run this project, you will need to add the following environment variables to your .env file
PORT
DATABASE_URL
TWITTER_API_KEY
TWITTER_API_SECRET
TWITTER_ACCESS_TOKEN
TWITTER_ACCESS_TOKEN_SECRET
Clone the project
git clone https://github.com/theblockchainchief/blockchain_scraper.git
Go to the project directory
cd blockchain_scraper
Install dependencies
npm install
{"blogs":{"medium":["blockchain","crypto","web3"],"hashnode":["blockchain","web3","crypto"],"devTo":["blockchain","web3","crypto"]},"projects":{"github":["blockchain","crypto","web3"]},"hackathon":{"devpost":["blockchain"]},"videos":{"youtube":["web3","blockchain","crypto"]},"courses":{"edX":["blockchain"],"coursera":["blockchain"]},"posts":{"twitter":["blockchain","nft","crypto"]}}
Start the server
npm run start