-
Notifications
You must be signed in to change notification settings - Fork 2
Twitter as a Message Bus.
License
clickyotomy/twitter-message-bus
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
twitter-message-bus ------------------- Twitter as a message bus. INTRODUCTION This is a proof of concept to show how Twitter can be used as a global message bus. Twitter has a streaming service where a program can listen to tweets posted by a user; tweets based on a topic, location, etc. The same (more specifically a user-stream) is used here. A node posts a tweet on Twitter, through a registered account and all the nodes that have subscribed to that account's user-stream can listen to the messages. Since Twitter has a 140 character limit, we store the data on a GitHub Gist and tweet the ID of the gist. All the components in this program are loosely coupled, any customization can be plugged-in easily. COMPONENTS [-] Authentication: Verifying the sender as well as the integrity of the transported messages is handled by Keybase. [-] Message Bus: Twitter handles transporting messages back-and-forth. [-] Data: Data is stored in an encrypted format (using Keybase) on GitHub. REQUIREMENTS You will need a Linux machine which has redis, disque and python-2.7+ installed. requirements.txt will install all the required Python libraries. ARCHITECTURE [-] The disque job queue has two queues setup: 'in' and 'out'. The 'in' queue receives all the incoming tweets from Twitter; the 'out' queue contains tweet, gist ID's which will have a set expiry time. These tweets/gists will be tracked from here and deleted accordingly, once their TTL is expired. [-] A daemon will listen to the Twitter Streaming API and dumps tweets to the 'in' queue. [-] A daemon will listen to the 'out' queue and perform deletion of expired tweets and gists. [-] A daemon will listen to the 'in' queue (like a worker); reads and decrypts messages (using Keybase) by fetching data from gist(the tweet contains the ID of the gist). [-] A script will encrypt and sign the contents of the raw data. MODULES This is a brief description of the Python modules in the source. [-] auth: Logic for encryption, description, signing and verifying data. [-] gist: CRUD operations on GitHub Gists. [-] stream: Listen to the Twitter Stream; dump tweet to the 'in' queue. [-] push: Create a gist on GitHub (with the signed, encrypted) message, post the gist ID as a tweet. [-] pull: Listen to the 'in' queue, fetch the contents of the gist (from ID); decrypt and verify the content. [-] expiry: Listen to the 'out' queue, keep a track of tweets/gists to be deleted, delete them when the expiration time has reached. USAGE push.py [-h] [-s HOST:PORT [HOST:PORT ...]] [-d] -r KEYBASE-ID [-t N] (-i FILE | -m MESSAGE) Push data to the message bus. optional arguments: -h, --help show this help message and exit -s HOST:PORT [HOST:PORT ...], --sockets HOST:PORT [HOST:PORT ...] a list containing the host, port numbers to listen to; defaults to localhost:7711 (for disque) -d, --debug enable debugging -r KEYBASE-ID, --recipient KEYBASE-ID keybase-id to send -t N, --ttl N a TTL (in seconds) for the data on Twitter and GitHub; if not specified, the data will remain as a gist, tweet -i FILE, --in-file FILE -m MESSAGE, --message MESSAGE -------------------------------------------------------------------------------- pull.py [-h] [-s HOST:PORT [HOST:PORT ...]] [-d] [-r DELAY] Read messages from the message bus. optional arguments: -h, --help show this help message and exit -s HOST:PORT [HOST:PORT ...], --sockets HOST:PORT [HOST:PORT ...] a list containing the host, port numbers to listen to; defaults to localhost:7711 (for disque) -d, --debug enable debugging -r DELAY, --retry DELAY queue check frequncy (in seconds); defaults to 8 -------------------------------------------------------------------------------- stream.py [-h] [-s HOST:PORT [HOST:PORT ...]] -c CHANNEL [CHANNEL ...] [-d] Listen to tweets; dump them to the queue. optional arguments: -h, --help show this help message and exit -s HOST:PORT [HOST:PORT ...], --sockets HOST:PORT [HOST:PORT ...] a list containing the host, port numbers to listen to; defaults to localhost:7711 (for disque) -c CHANNEL [CHANNEL ...], --channels CHANNEL [CHANNEL ...] Twitter accounts to follow -d, --debug enable debugging -------------------------------------------------------------------------------- expire.py [-h] [-s HOST:PORT [HOST:PORT ...]] [-d] [-r DELAY] Delete gists, tweets if a TTL is set. optional arguments: -h, --help show this help message and exit -s HOST:PORT [HOST:PORT ...], --sockets HOST:PORT [HOST:PORT ...] a list containing the host, port numbers to listen to; defaults to localhost:7711 (for disque) -d, --debug enable debugging -r DELAY, --retry DELAY queue check frequncy (in seconds); defaults to 8 -------------------------------------------------------------------------------- SETUP: redis: http://redis.io/topics/quickstart disque: https://github.com/antirez/disque#setup Keybase: https://keybase.io/docs/command_line Make sure you sign into Keybase and the client is up. Credentials: Setup a Twitter account, create an application and generate client-id, client-secret, access-token and access-token-secret. https://dev.twitter.com/oauth/overview/application-owner-access-tokens Setup a GitHub account, generate a personal access token; make sure 'gist' is in the list of selected scopes. https://git.io/vmNUX Store the API credentials in the source directory (after the clone) under vault/keys.json, in this format: { "github": "github-personal-access-token", "twitter": { "consumer-key": "twitter-app-consumer-key", "consumer-secret": "twitter-app-consumer-secret", "access-token": "twitter-app-access-token", "access-token-secret": "twitter-app-access-token-secret" } } twitter-message-bus (basic setup; check options for more functionality): $ git clone https://github.com/clickyotomy/twitter-message-bus \ GIT_CLONE_DIR $ DISQUE_INSTALL_DIR/src/disque-server [OPTIONS] $ cd GIT_CLONE_DIR $ pip install -r requirements.txt $ ./stream.py -c 'twitter-handle' $ ./expiry.py $ ./pull.py $ ./push.py -m 'Do. Or do not. There is no try.' -r 'twitter-handle' # DISQUE_INSTALL_DIR: Path where disque was cloned and built. # GIT_CLONE_DIR: Path where you cloned this repository. NOTES: [-] stream, expiry and pull run as daemons, you can pipe the output to a log-file to monitor them. [-] Keybase is still in alpha, so feel free to change the auth module. [-] As of now, there is support only for text/* mimetypes. [-] Since this is a proof of concept, the pull module does not do anything other than display the received message. [-] The source code is documented to the point. May the Force be with you.
About
Twitter as a Message Bus.
Topics
Resources
License
Stars
Watchers
Forks
Packages 0
No packages published