{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":170908840,"defaultBranch":"master","name":"RentCrawler","ownerLogin":"Morelatto","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2019-02-15T18:16:36.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/13797079?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1701216591.0","currentOid":""},"activityList":{"items":[{"before":null,"after":"079d3a931c866263923e97a5c9b2ceda01f07cae","ref":"refs/heads/dependabot/pip/cryptography-41.0.6","pushedAt":"2023-11-29T00:09:51.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"dependabot[bot]","name":null,"path":"/apps/dependabot","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/29110?s=80&v=4"},"commit":{"message":"chore(deps): bump cryptography from 41.0.4 to 41.0.6\n\nBumps [cryptography](https://github.com/pyca/cryptography) from 41.0.4 to 41.0.6.\n- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)\n- [Commits](https://github.com/pyca/cryptography/compare/41.0.4...41.0.6)\n\n---\nupdated-dependencies:\n- dependency-name: cryptography\n  dependency-type: indirect\n...\n\nSigned-off-by: dependabot[bot] <support@github.com>","shortMessageHtmlLink":"chore(deps): bump cryptography from 41.0.4 to 41.0.6"}},{"before":null,"after":"14b1c96a0cc0127ac2540d88bffd108bb7f3ac31","ref":"refs/heads/dependabot/pip/urllib3-2.0.7","pushedAt":"2023-10-17T21:03:21.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"dependabot[bot]","name":null,"path":"/apps/dependabot","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/29110?s=80&v=4"},"commit":{"message":"chore(deps): bump urllib3 from 2.0.6 to 2.0.7\n\nBumps [urllib3](https://github.com/urllib3/urllib3) from 2.0.6 to 2.0.7.\n- [Release notes](https://github.com/urllib3/urllib3/releases)\n- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)\n- [Commits](https://github.com/urllib3/urllib3/compare/2.0.6...2.0.7)\n\n---\nupdated-dependencies:\n- dependency-name: urllib3\n  dependency-type: indirect\n...\n\nSigned-off-by: dependabot[bot] <support@github.com>","shortMessageHtmlLink":"chore(deps): bump urllib3 from 2.0.6 to 2.0.7"}},{"before":"c3e403150d1e0773135443d094ed82d617bcfddb","after":"cd53b6571a3e800b850cbd65487dbb236978f026","ref":"refs/heads/master","pushedAt":"2023-10-03T00:23:12.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"Morelatto","name":null,"path":"/Morelatto","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13797079?s=80&v=4"},"commit":{"message":"chore(README.md): update execution instructions and add explanation for data configuration\n\nThe execution instructions in the README.md file were updated to provide more clarity on how to run the project. Additionally, an explanation was added for the data configuration in the `data` folder.\n\nThe following changes were made:\n\n- Added a new section titled \"Execução\" to provide instructions for setting up the Redis database before running the crawlers.\n- Added a description of the JSON mapping file in the `data` folder, explaining how it is used in the data extraction process.\n- Added a description of the pagination parameters in the JSON mapping file.\n- Updated the description of the URLs used by the crawlers to extract data, mentioning that they are currently pointing to the city of São Paulo and explaining how to update them for a different location.\n- Updated the Docker section to mention that the necessary parameters should be configured via a `.env` file.\n- Updated the Redis URL in the Docker section to remove the password from the URL.\n- Updated the \"Rodar\" section to provide the command for running the `vr` spider.\n- Removed the \"Salvar em JSON\" section as it is no longer relevant.\n\nThese changes improve the clarity and completeness of the README.md file, making it easier for users to understand and execute the project.","shortMessageHtmlLink":"chore(README.md): update execution instructions and add explanation f…"}},{"before":"b765218d47b221fdd0e50e582828563a3fe3eb71","after":"c3e403150d1e0773135443d094ed82d617bcfddb","ref":"refs/heads/master","pushedAt":"2023-10-02T23:48:30.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Morelatto","name":null,"path":"/Morelatto","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13797079?s=80&v=4"},"commit":{"message":"feat(data): add configuration files for Quinto Andar and Viva Real APIs\n\nAdd configuration files for Quinto Andar and Viva Real APIs to the data directory. These files contain the necessary information such as the API endpoints, headers, and request bodies for making requests to these APIs. These configuration files will be used in the application to fetch data from Quinto Andar and Viva Real APIs for further processing.\n\nfeat(zap.json): add zap.json file to store API data for ZapImóveis integration\n\nThe zap.json file is added to the data directory. This file will be used to store the API data for the ZapImóveis integration. The data object in the file contains the URL and headers for making the API request, and the params object contains the pagination information. This file will be used by the application to fetch and process the data from the ZapImóveis API.","shortMessageHtmlLink":"feat(data): add configuration files for Quinto Andar and Viva Real APIs"}},{"before":"f4751f6e300df56e645f752164e4ab805b11da7c","after":"b765218d47b221fdd0e50e582828563a3fe3eb71","ref":"refs/heads/master","pushedAt":"2023-10-02T23:46:15.000Z","pushType":"push","commitsCount":11,"pusher":{"login":"Morelatto","name":null,"path":"/Morelatto","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13797079?s=80&v=4"},"commit":{"message":"Merge branch 'master' of github.com:Morelatto/RentCrawler","shortMessageHtmlLink":"Merge branch 'master' of github.com:Morelatto/RentCrawler"}},{"before":"c42c570a63f578787f9dc5814df132923db59aae","after":null,"ref":"refs/heads/dependabot/pip/cryptography-41.0.4","pushedAt":"2023-09-22T14:04:00.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"Morelatto","name":null,"path":"/Morelatto","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13797079?s=80&v=4"}},{"before":"19f12786aabd7f240c9b81f4648e31de45b6a2d3","after":"f4751f6e300df56e645f752164e4ab805b11da7c","ref":"refs/heads/master","pushedAt":"2023-09-22T14:03:56.000Z","pushType":"pr_merge","commitsCount":2,"pusher":{"login":"Morelatto","name":null,"path":"/Morelatto","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13797079?s=80&v=4"},"commit":{"message":"Merge pull request #7 from Morelatto/dependabot/pip/cryptography-41.0.4\n\nchore(deps): bump cryptography from 41.0.3 to 41.0.4","shortMessageHtmlLink":"Merge pull request <a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1907764786\" data-permission-text=\"Title is private\" data-url=\"https://github.com/Morelatto/RentCrawler/issues/7\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/Morelatto/RentCrawler/pull/7/hovercard\" href=\"https://github.com/Morelatto/RentCrawler/pull/7\">#7</a> from Morelatto/dependabot/pip/cryptography-41.0.4"}},{"before":null,"after":"c42c570a63f578787f9dc5814df132923db59aae","ref":"refs/heads/dependabot/pip/cryptography-41.0.4","pushedAt":"2023-09-21T21:07:41.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"dependabot[bot]","name":null,"path":"/apps/dependabot","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/29110?s=80&v=4"},"commit":{"message":"chore(deps): bump cryptography from 41.0.3 to 41.0.4\n\nBumps [cryptography](https://github.com/pyca/cryptography) from 41.0.3 to 41.0.4.\n- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)\n- [Commits](https://github.com/pyca/cryptography/compare/41.0.3...41.0.4)\n\n---\nupdated-dependencies:\n- dependency-name: cryptography\n  dependency-type: indirect\n...\n\nSigned-off-by: dependabot[bot] <support@github.com>","shortMessageHtmlLink":"chore(deps): bump cryptography from 41.0.3 to 41.0.4"}},{"before":"443bdf175d6ce557e3d186022611b952fc6ce366","after":"19f12786aabd7f240c9b81f4648e31de45b6a2d3","ref":"refs/heads/master","pushedAt":"2023-09-21T02:26:13.000Z","pushType":"push","commitsCount":3,"pusher":{"login":"Morelatto","name":null,"path":"/Morelatto","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13797079?s=80&v=4"},"commit":{"message":"feat(quintoandar.py): add page_number to cb_kwargs in the follow method to keep track of the current page number being scraped\n\nfix(quintoandar.py): add dont_filter=True to the follow method to prevent filtering of duplicate requests\nfix(quintoandar.py): change business_context from \"RENT\" to \"SALE\" to scrape properties for sale instead of for rent\nfix(quintoandar.py): update the bounds and center coordinates in the search query filters to target the Morumbi neighborhood in São Paulo\nfix(quintoandar.py): change sorting criteria from \"relevance_rent\" to \"relevance_sale\" to sort properties by relevance for sale\nfix(quintoandar.py): update the keyword_match to target the Morumbi neighborhood in São Paulo","shortMessageHtmlLink":"feat(quintoandar.py): add page_number to cb_kwargs in the follow meth…"}},{"before":"2a4198a60f32487447ef945d90c882aa55ce4aeb","after":"443bdf175d6ce557e3d186022611b952fc6ce366","ref":"refs/heads/master","pushedAt":"2023-09-20T01:19:10.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"Morelatto","name":null,"path":"/Morelatto","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13797079?s=80&v=4"},"commit":{"message":"chore(README): updates README","shortMessageHtmlLink":"chore(README): updates README"}},{"before":"02eb1854d493adeff639edbcaa25a58e4207036e","after":"2a4198a60f32487447ef945d90c882aa55ce4aeb","ref":"refs/heads/master","pushedAt":"2023-09-20T00:59:54.000Z","pushType":"push","commitsCount":7,"pusher":{"login":"Morelatto","name":null,"path":"/Morelatto","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13797079?s=80&v=4"},"commit":{"message":"refactor(spider files): remove unnecessary 'dont_filter' parameter from scrapy.Request calls\n\nThe 'dont_filter' parameter was removed from the scrapy.Request calls in the 'quintoandar.py', 'vivareal.py', and 'zap.py' spider files. This parameter is not needed and was causing confusion in the code.","shortMessageHtmlLink":"refactor(spider files): remove unnecessary 'dont_filter' parameter fr…"}},{"before":"2ca3ff99bb3f945237ab3f278fdb05647e9dffda","after":"02eb1854d493adeff639edbcaa25a58e4207036e","ref":"refs/heads/master","pushedAt":"2023-09-15T04:18:58.000Z","pushType":"push","commitsCount":8,"pusher":{"login":"Morelatto","name":null,"path":"/Morelatto","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13797079?s=80&v=4"},"commit":{"message":"fix(settings.py): decrease CONCURRENT_REQUESTS_PER_DOMAIN to 1 and add DOWNLOAD_DELAY of 2 seconds to reduce load on target website\n\nfeat(settings.py): enable AUTOTHROTTLE_ENABLED and set AUTOTHROTTLE_START_DELAY to 2.5 seconds to automatically adjust crawling speed","shortMessageHtmlLink":"fix(settings.py): decrease CONCURRENT_REQUESTS_PER_DOMAIN to 1 and ad…"}},{"before":"f95cc41339331f37dd371e2d3391843786c6c28f","after":null,"ref":"refs/heads/dependabot/pip/scrapy-2.6.2","pushedAt":"2023-08-07T19:28:45.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"Morelatto","name":null,"path":"/Morelatto","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13797079?s=80&v=4"}},{"before":"b6c149a3637a1aa65d5b630862d3f98ce661b168","after":null,"ref":"refs/heads/dependabot/pip/redis-4.4.4","pushedAt":"2023-08-07T19:28:39.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"dependabot[bot]","name":null,"path":"/apps/dependabot","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/29110?s=80&v=4"}},{"before":"fa39b5bf12de3076c45779a2086265e57f68e6b8","after":"2ca3ff99bb3f945237ab3f278fdb05647e9dffda","ref":"refs/heads/master","pushedAt":"2023-08-07T00:08:23.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Morelatto","name":null,"path":"/Morelatto","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13797079?s=80&v=4"},"commit":{"message":"chore(README.md): fix typo in dependencies section heading\n\nchore(README.md): update dependencies section to include Poetry as a requirement for the project","shortMessageHtmlLink":"chore(README.md): fix typo in dependencies section heading"}},{"before":"4888fb7d825be83017bef288d6d0cc7e2cadee57","after":"fa39b5bf12de3076c45779a2086265e57f68e6b8","ref":"refs/heads/master","pushedAt":"2023-08-07T00:04:50.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"Morelatto","name":null,"path":"/Morelatto","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13797079?s=80&v=4"},"commit":{"message":"docs(README.md): fix typo in section title, change \"Salvar para arquivo JSON\" to \"Salvar em JSON\"\nrefactor(quintoandar.py): remove unused imports and variables, simplify loop iteration in parse() method","shortMessageHtmlLink":"docs(README.md): fix typo in section title, change \"Salvar para arqui…"}},{"before":"b68f245b8428f84ea6c772945d7061718c78f7e0","after":"4888fb7d825be83017bef288d6d0cc7e2cadee57","ref":"refs/heads/master","pushedAt":"2023-08-06T23:48:27.000Z","pushType":"push","commitsCount":17,"pusher":{"login":"Morelatto","name":null,"path":"/Morelatto","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13797079?s=80&v=4"},"commit":{"message":"docs(README.md): fix typo in section title, change \"Salvar para arquivo JSON\" to \"Salvar em JSON\"\nrefactor(quintoandar.py): remove unused imports and variables, simplify loop iteration in parse() method","shortMessageHtmlLink":"docs(README.md): fix typo in section title, change \"Salvar para arqui…"}},{"before":"16a9a2e4911642f44b82568fc6984eb8974945a3","after":null,"ref":"refs/heads/dependabot/pip/redis-4.5.3","pushedAt":"2023-03-31T15:25:29.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"dependabot[bot]","name":null,"path":"/apps/dependabot","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/29110?s=80&v=4"}},{"before":null,"after":"b6c149a3637a1aa65d5b630862d3f98ce661b168","ref":"refs/heads/dependabot/pip/redis-4.4.4","pushedAt":"2023-03-31T15:25:25.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"dependabot[bot]","name":null,"path":"/apps/dependabot","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/29110?s=80&v=4"},"commit":{"message":"Bump redis from 4.3.3 to 4.4.4\n\nBumps [redis](https://github.com/redis/redis-py) from 4.3.3 to 4.4.4.\n- [Release notes](https://github.com/redis/redis-py/releases)\n- [Changelog](https://github.com/redis/redis-py/blob/master/CHANGES)\n- [Commits](https://github.com/redis/redis-py/compare/v4.3.3...v4.4.4)\n\n---\nupdated-dependencies:\n- dependency-name: redis\n  dependency-type: direct:production\n...\n\nSigned-off-by: dependabot[bot] <support@github.com>","shortMessageHtmlLink":"Bump redis from 4.3.3 to 4.4.4"}},{"before":null,"after":"16a9a2e4911642f44b82568fc6984eb8974945a3","ref":"refs/heads/dependabot/pip/redis-4.5.3","pushedAt":"2023-03-27T22:32:59.591Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"dependabot[bot]","name":null,"path":"/apps/dependabot","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/29110?s=80&v=4"},"commit":{"message":"Bump redis from 4.3.3 to 4.5.3\n\nBumps [redis](https://github.com/redis/redis-py) from 4.3.3 to 4.5.3.\n- [Release notes](https://github.com/redis/redis-py/releases)\n- [Changelog](https://github.com/redis/redis-py/blob/master/CHANGES)\n- [Commits](https://github.com/redis/redis-py/compare/v4.3.3...v4.5.3)\n\n---\nupdated-dependencies:\n- dependency-name: redis\n  dependency-type: direct:production\n...\n\nSigned-off-by: dependabot[bot] <support@github.com>","shortMessageHtmlLink":"Bump redis from 4.3.3 to 4.5.3"}}],"hasNextPage":false,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"startCursor":"Y3Vyc29yOnYyOpK7MjAyMy0xMS0yOVQwMDowOTo1MS4wMDAwMDBazwAAAAO6XdGK","endCursor":"Y3Vyc29yOnYyOpK7MjAyMy0wMy0yN1QyMjozMjo1OS41OTE2NDZazwAAAAMMcIxj"}},"title":"Activity · Morelatto/RentCrawler"}