Bridgy connects your web site to social media. Likes, retweets, mentions, cross-posting, and more. See the user docs for more details, or the developer docs if you want to contribute.
Bridgy is part of the IndieWeb ecosystem. In IndieWeb terminology, Bridgy offers backfeed, POSSE, and webmention support as a service.
License: This project is placed in the public domain.
You'll need the Google Cloud SDK (aka gcloud
) with the gcloud-appengine-python
, gcloud-appengine-python-extras
and google-cloud-sdk-datastore-emulator
components. Then, create a Python 3 virtualenv and install the dependencies with:
python3 -m venv local
source local/bin/activate
pip install -r requirements.txt
# needed to serve static file assets in dev_appserver
ln -s local/lib/python3*/site-packages/oauth_dropins/static oauth_dropins_static
gcloud config set project brid-gy
Now, you can fire up the gcloud emulator and run the tests:
gcloud beta emulators datastore start --no-store-on-disk --consistency=1.0 --host-port=localhost:8089 < /dev/null >& /dev/null
python3 -m unittest discover -s tests -t .
kill %1
If you send a pull request, please include or update a test for your new code!
To test a poll or propagate task, find the relevant Would add task line in the logs, eg:
INFO:root:Would add task: projects//locations/us-central1/queues/poll {'app_engine_http_request': {'http_method': 'POST', 'relative_uri': '/_ah/queue/poll', 'app_engine_routing': {'service': 'background'}, 'body': b'source_key=agNhcHByFgsSB1R3aXR0ZXIiCXNjaG5hcmZlZAw&last_polled=1970-01-01-00-00-00', 'headers': {'Content-Type': 'application/x-www-form-urlencoded'}}, 'schedule_time': seconds: 1591176072
...pull out the relative_uri
and body
, and then put them together in a curl
command against the background
service, which usually runs on http://localhost:8081/, eg:
curl -d 'source_key=agNhcHByFgsSB1R3aXR0ZXIiCXNjaG5hcmZlZAw&last_polled=1970-01-01-00-00-00' \
http://localhost:8081/_ah/queue/poll
To run the entire app locally, run this in the repo root directory:
dev_appserver.py --log_level debug --enable_host_checking false \
--support_datastore_emulator --datastore_emulator_port=8089 \
--application=brid-gy ./app.yaml ./background.yaml
(Note: dev_appserver.py is incompatible with python3. if python3 is your default python, you can run python2 /location/of/dev_appserver.py ...
instead.)
Open localhost:8080 and you should see the Bridgy home page!
If you hit an error during setup, check out the oauth-dropins Troubleshooting/FAQ section. For searchability, here are a handful of error messages that have solutions there:
bash: ./bin/easy_install: ...bad interpreter: No such file or directory
ImportError: cannot import name certs
ImportError: No module named dev_appserver
ImportError: cannot import name tweepy
File ".../site-packages/tweepy/auth.py", line 68, in _get_request_token
raise TweepError(e)
TweepError: must be _socket.socket, not socket
error: option --home not recognized
There's a good chance you'll need to make changes to granary or oauth-dropins at the same time as bridgy. To do that, clone their repos elsewhere, then install them in "source" mode with:
pip uninstall -y oauth-dropins
pip install -e <path-to-oauth-dropins-repo>
ln -sf <path-to-oauth-dropins-repo>/oauth_dropins/static oauth_dropins_static
pip uninstall -y granary
pip install -e <path to granary>
To use dev_appserver with local granary and oauth-dropins, you'll need to either replace their GitHub lines in requirements.txt
with -e <path-to-local-repo>
, or apply this patch to dev_appserver.py to make it use your virtualenv in place.
To deploy to App Engine, run scripts/deploy.sh
.
remote_api_shell
is a useful interactive Python shell that can interact with the production app's datastore, memcache, etc. To use it, create a service account and download its JSON credentials, put it somewhere safe, and put its path in your GOOGLE_APPLICATION_CREDENTIALS
environment variable.
Deploying to your own app-engine project can be useful for testing, but is not recommended for production. To deploy to your own app-engine project, create a project on gcloud console and activate the Tasks API. Initialize the project on the command line using gcloud config set project <project-name>
followed by gcloud app create
. You will need to update TASKS_LOCATION
in util.py to match your project's location. Finally, you will need to add your "background" domain (eg background.YOUR-APP-NAME.appspot.com
) to OTHER_DOMAINS in util.py and set host_url
in tasks.py
to your base app url (eg app-dot-YOUR-APP-NAME.wn.r.appspot.com
). Finally, deploy (after testing) with gcloud -q beta app deploy --no-cache --project YOUR-APP-NAME *.yaml
To work on the browser extension:
cd browser-extension
npm install
npm run test
You need to be logged into Instagram in your browser. The extension doesn't have a UI, but you can see what it's doing on your Bridgy user page, eg brid.gy/instagram/[username]
. Note that it doesn't work with Firefox's Facebook Container tabs add-on. If you have that enabled, you'll need to disable it to use Bridgy's browser extension.
If you're working on the browser extension, or you're sending in a bug report for it,, its JavaScript console logs are invaluable for debugging. Here's how to get them in Firefox:
Thanks for trying! And for offering to send logs, those would definitely be helpful. Here's how to get them:
- Open
about:debugging
- Click This Firefox on the left
- Scroll down to Bridgy
- Click Inspect
- Click on the Console tab
Here's how to send them in with a bug report:
- Right click, Export Visible Messages To, File, save the file.
- Email the file to bridgy @ ryanb.org. Do not post or attach it to a GitHub issue, or anywhere else public, because it contains sensitive tokens and cookies.
So you want to add a new silo? Maybe MySpace, or Friendster, or even Tinder? Great! Here are the steps to do it. It looks like a lot, but it's not that bad, honest.
- Find the silo's API docs and check that it can do what Bridgy needs. At minimum, it should be able to get a user's posts and their comments, likes, and reposts, depending on which of those the silo supports. If you want publish support, it should also be able to create posts, comments, likes, reposts, and/or RSVPs.
- Fork and clone this repo.
- Create an app (aka client) in the silo's developer console, grab your app's id (aka key) and secret, put them into new local files in the repo root dir, following this pattern. You'll eventually want to send them to @snarfed too, but no hurry.
- Add the silo to oauth-dropins if it's not already there:
- Add a new
.py
file for your silo with an auth model and handler classes. Follow the existing examples. - Add a 100 pixel tall button image named
[NAME]_2x.png
, where[NAME]
is your start handler class'sNAME
constant, eg'twitter'
. - Add it to the app front page and the README.
- Add a new
- Add the silo to granary:
- Add a new
.py
file for your silo. Follow the existing examples. At minimum, you'll need to implementget_activities_response
and convert your silo's API data to ActivityStreams. - Add a new unit test file and write some tests!
- Add it to
api.py
(specificallyHandler.get
),app.py
,index.html
, and the README.
- Add a new
- Add the silo to Bridgy:
- Add a new
.py
file for your silo with a model class. Follow the existing examples. - Add it to
app.py
andhandlers.py
(just import the module). - Add a 48x48 PNG icon to
static/
. - Add a new
[SILO]_user.html
file intemplates/
and add the silo toindex.html
. Follow the existing examples. - Add the silo to
about.html
and this README. - If users' profile picture URLs can change, add a cron job that updates them to
cron.py
.
- Add a new
- Optionally add publish support:
- Implement
create
andpreview_create
for the silo in granary. - Add the silo to
publish.py
: import its module, add it toSOURCES
, and update this error message.
- Implement
Good luck, and happy hacking!
App Engine's built in dashboard and log browser are pretty good for interactive monitoring and debugging.
For alerting, we've set up Google Cloud Monitoring (née Stackdriver). Background in issue 377. It sends alerts by email and SMS when HTTP 4xx responses average >.1qps or 5xx >.05qps, latency averages >15s, or instance count averages >5 over the last 15m window.
I occasionally generate stats and graphs of usage and growth from the BigQuery dataset (#715). Here's how.
-
Export the full datastore to Google Cloud Storage. Include all entities except
*Auth
and other internal details. Check to see if any new kinds have been added since the last time this command was run.gcloud datastore export --async gs://brid-gy.appspot.com/stats/ --kinds Activity, Blogger,BlogPost,BlogWebmention,Facebook,FacebookPage,Flickr,GitHub,GooglePlusPage,Instagram,Mastodon,Medium,Meetup,Publish,PublishedPage,Reddit,Response,SyndicatedPost,Tumblr,Twitter,WordPress
Note that
--kinds
is required. From the export docs, Data exported without specifying an entity filter cannot be loaded into BigQuery. -
Wait for it to be done with
gcloud datastore operations list | grep done
. -
for kind in Activity BlogPost BlogWebmention Publish Response SyndicatedPost; do bq load --replace --nosync --source_format=DATASTORE_BACKUP datastore.$kind gs://brid-gy.appspot.com/stats/all_namespaces/kind_$kind/all_namespaces_kind_$kind.export_metadata done for kind in Blogger Facebook FacebookPage Flickr GitHub GooglePlusPage Instagram Mastodon Medium Meetup Reddit Tumblr Twitter WordPress; do bq load --replace --nosync --source_format=DATASTORE_BACKUP sources.$kind gs://brid-gy.appspot.com/stats/all_namespaces/kind_$kind/all_namespaces_kind_$kind.export_metadata done
-
Check the jobs with
bq ls -j
, then wait for them withbq wait
. -
Run the full stats BigQuery query. Download the results as CSV.
-
Open the stats spreadsheet. Import the CSV, replacing the data sheet.
-
Check out the graphs! Save full size images with OS or browser screenshots, thumbnails with the Download Chart button. Then post them!
The datastore is automatically backed up by an App Engine cron job that runs Datastore managed export (details) and stores the results in Cloud Storage, in the brid-gy.appspot.com bucket. It backs up weekly and includes all entities except Response
and SyndicatedPost
, since they make up 92% of all entities by size and they aren't as critical to keep.
(We used to use Datastore Admin Backup, but it shut down in Feb 2019.)
We use this command to set a Cloud Storage lifecycle policy on that bucket that prunes older backups:
gsutil lifecycle set cloud_storage_lifecycle.json gs://brid-gy.appspot.com
Run this to see how much space we're currently using:
gsutil du -hsc gs://brid-gy.appspot.com/\*
Run this to download a single complete backup:
gsutil -m cp -r gs://brid-gy.appspot.com/weekly/datastore_backup_full_YYYY_MM_DD_\* .
Also see the BigQuery dataset (#715).