Piping economics throughout the Twitterverse
I am on a crusade to make Twitter a place for useful economic information and informative economics discussion. That goal is partially accomplished by following economists doing interesting and relevant research and subscribing to economics Twitter lists. The problem is that so many tweets—even those by economists—have nothing to do with economics.
I recently rediscovered Yahoo Pipes and found that it can interact with Twitter in a way that does not require any coding on my part and does not require that the application be served on my own website.
The result is TweetEcon, an experimental retweeting economics Twitterbot. Here is a sample of the tweets TweetEcon has retweeted:
How to build your own retweeting Twitterbot
1. Get a Yahoo Pipes account and create a new pipe.
2. Get the tweets to be retweeted. Under Sources, grab Fetch Feed. Go to Twitter Search and enter a search term, such as economics. On the results page, click the link that says Feed for this query. This will take you to the RSS feed. Notice that there are a lot of worthless tweets that use the word economics. That is why the next step is necessary.
3. Filter the tweets retrieved. Under Operators, grab Filter.
First, filter out any retweets. This serves two purposes: (a) it prevents retweet looping in which the pipe retweets its own tweets, and (b) most retweets are editorial rather than informative. The swear words are blocked for obvious reasons. Also, a huge portion of economics tweets say something like “My boyfriend’s cousin just lost his job at Best Buy. This economy sucks!” Also another huge portion of tweets are from economics students griping about having to wake up at 9:00 a.m. so they can make it to their 10:15 a.m. class, which is why school related terms are blocked.
Unfortunately, tweets with #hashtags have a massive noise-to-signal ratio. Here is one example:
See, no economics in the tweet, but this twit decided to use the #economy hashtag.
The most important filter is the filter that keeps only those tweets that have a link in it. Tweets without links tend to provide observations or opinions, rather than useful information. I decided that I will only retweet posts that have a link.
The filter is probably where most of the future tweaking will be.
4. Form the retweet. Under Operators, grab Loop. The first field puts the author’s Twitter user name at the beginning of what will be my retweet. The second field is a single space. The third field is the tweet that is being retweeted.
5. Form the retweet, part 2. Under Operators, grab Regex. This step replaces the Twitter username URL (e.g., http://twitter.com/ericfruits) with the username used by Twitter (e.g. @ericfruits) and adds “RT” in front. Note, that the “RT @username” must go at the beginning of the tweet. Otherwise, the tweet may be too long and cut off the RT. If that happens, you run the risk of getting into a retweet loop.
6. (Optional) Sort the retweets from most recent to oldest. Under Operators, grab Sort.
7. Test the pipe. You may get some error messages (especially those that end in “420″). Do not worry. That means that Twitter and Yahoo Pipes are having a temporary spat. Wait a few minutes and try again. If it still does not work, save the pipe, go back to the My Pipes page, then click on the pipe and/or click on the RSS feed for the pipe. This step will drive you crazy because you will not know if you have done something wrong.
8. Tweet the pipe’s results. Go to My Pipes, click on the pipe, then click on Get as RSS. Now that you have the RSS, you can use Twitterfeed and/or Feedburner to convert the RSS output to tweets. Twitterfeed is fairly self-explanatory. To have Feedburner tweet the RSS, click on Publicize, then click on Socialize and follow the steps.
I use both Twitterfeed and Feedburner. Because of the time and tweeting limits of both services, there is no real overlap of tweets. For example, my pipe retrieves approximately 50 tweets an hour. Twitterfeed will send out a maximum of 10 tweets an hour and Feedburner sends out a maximum of 16 tweets an hour (maybe). So far, Twitterfeed is a much more reliable tweeter than Feedburner.
Happy pipe-tweeting!