RSS reversed: From Feed to Blog
We all use RSS mostly to read updates from different sites. What if we could reverse this, and use RSS to actually populate a site?
Here is a story of RSS, in reverse. Maybe we should call it SSR instead of RSS…
Both for work, and in my spare time, I scan the news (and the Internet as a whole) for humanitarian, aid and development articles.
1. Using RSS the conventional way
Up till recently I exclusively used PageFlakes, a simple tool allowing you to display a multitude of RSS feeds from different websites on one page.
This gave me an instantaneous overview of what is being published in the world. But, I needed to visually scan the different feed portlets. And I had no history: once a news source is refreshed, new items come up and old ones are popped off the list. You missed it, too bad…
And it displayed “any news”, not just humanitarian news. I could not really filter out contents of interest to me, nor could I search the items.
2. Using RSS to populate a site, the script way
Thus I made The Other World News, which aggregates news from different sources, and displays it in a straight text format.
How to do this? Pretty simple:
I identified about 100 300 online news sources, covering the topics I was interested in. I took the RSS feeds of all these sites and combined them into several “streams of data”, one stream per type of news source, using a free tool called NewsGator .
For each stream of data, Newsgator lets you do some basic formatting, identifying where you want the data, time, title, text contents and lets you trim the length of the output. The actual output is a script, which looks like this:
I used that line of code in a post, or on some of my blogs I put it in a side column. Each time the page is loaded, Newsgator runs the script, and displays the latest news items.
That works well and displays only the latest humanitarian news. But, once again, it did not store the data, it was “only” a way to display the latest news items.
3. Using RSS to populate a site, the geeky way
So I needed something more: a way to keep all news posts stored, so they were search-able, tag-able, re-use-able… And here is where I am very pleased with myself. Ha! A nerd’s hand is so easily filled…
The fruits of my labour:
Aaahh. Now we are talking! Those of you following “The Road” regularly, know I have been working on these three sites – I call them my meta-sites – for a while… It took me sweat, blood and tears before I had the three sites as I wanted:
I have a site AidNews gives the humanitarian news (only). Each feed item is automatically stored as a summary, with a link to the original press article. Occasional pictures go in it too.
For Those Who Want to Know does the same, but the 250+ sources are blogs, humanitarian sites and press releases from development organisations, etc….
And while I was at it, AidBlogs aggregates summaries of the latest blogs from my fellow aidworkers.
Because I am a humanitarian nerd, I have to tell you how it works. I just gotta…
Just like in chapter 3, I used about my hundreds of online news sources (the quality of your input defines the quality of your output). I take the RSS feeds of all these sites and run them through NewsGator to combine all of them. This time, the output is a new RSS feeds rather than a script.
I take several Newsgator feeds which I combine into one feed, filter out formatting problems, delete duplicate news items, and take out only the most recent 30 posts, using Yahoo Pipes. Talking of nerd-tools, “Yahoo Pipes” is a good example! But with a great user interface:
If you ever feel geeky, and you want to play with RSS feeds, give “Pipes” a go.. You will be amazed about its powers.
Once more, Yahoo Pipes generates a new RSS feed, which I import into Tumblr, a very simple but powerful blogging platform. What is unique about Tumblr, is the built-in ability to import RSS feeds, which they convert into the posts.
Taraa…! That gives me what I want: every hour, I get a dozen new news articles in one blog. Automatically…! The big advantage is that all these posts are searchable. Here is an search on “Sudan” from my AidNews site.
Through time, my blog network grew more and more complex, check out my blog diagram for an overview of what more magical things I did with RSS feeds.
One word of caution though! Whatever you republish, taken from RSS streams from other sites, ensure you always link back to the original site and article. Otherwise, you are leeching: benefiting from the labour of others.
Follow me
Peter. Flemish, European, aid worker, blogger, expeditioner, sailor, traveller, husband, father, friend, nutcase. Not necessarily in that order. (
Thanks for the tip. Seeing the small numbers in Google analytics, i was thinking about removing the retweet button.
BTW, considering the enormous amount of you follow i follow crowd in the twitter, would people care to click those links
I don’t know if your retweet button shortens the URL with bit.ly, but -as described in this post- that could give you a black and white figure on the traffic the tweets give.
I do, on my links and have been astonished by the amount of traffic each tweet gives.
It is true there is a “scratch mine, I’ll scratch yours” mentality in the following in Twitter, but not so much in retweeting. At least not in the social community I have created around each of my Twitter accounts.
And maybe that is key in all of it: ensure the quality of your followers. Maybe one tip: I *never* autofollow. This means that “my” social community would never follow me, simply because I would follow back.
I’d like to say they follow my tweets, because they like the content
Hope that helps a bit,
Peter
I have just discovered your site via ICT-KM, and it will keep me busy for hours…. I have been blogging for a few months now, we are one of the NGOs who want to use social media more and are still finding out how best to do that (ILEIA, see http://www.leisa.info). So your site is full of useful stuff (so far I’d only found social media for marketing your business).. thanks and keep up the good work!
Karen
While these sites may be free for us, the end user, google and yahoo take the numbers we provide for them, and they scan the content we enter on their services, and use it as fodder to sell their profitable products, such as adwords.
It is in their interest to keep these sites working well, as the critical mass we provide is the very product they can leverage to advertisers.
That being said, Michael Keizer is correct- if the service is mission critical, buy the pro account, and get on the line with their support if needed.
Do you know of an alternative to Pipes?
@Hank:
Don’t think there is any pro-account formula for Yahoo Pipes unfortunately. The only alternative I found is http://pipes.deri.org/ but looks even more of a hackers tool, and I don’t know how well it performs.
I think I will make my own…
Peter.
i just love to Twitter everyday with my friends. Twitter is much better than blogging in my opinion and it is very addictive too.
. ..
thank you for this good information
nice post . Very helpful information. Thank you.
Hi, this blog post is very well-written and appears extremely useful. But I was just wondering if you could clear something up? You mention Newsgator as a product, but the link to their website shows Newsgator is a company with a lot of products – it isn’t clear which tool or product you used to amalgamate feeds and produce script. Could you explain this step in some more detail?
If you post here please email me to let me know
Many thanks
Jez
Hi Jez,
You are totally correct. Newsgator changed their services end August. The online aggregator functions they used to have is no longer available.
If you are looking for a feed aggregator or RSS-to-script function, have a look at this post:
http://www.blogtips.org/free-rss-tools/
best,
Peter
i think the problem is solved now..
Yahoo Pipes went back up after almost 3 weeks of intermittent problems. There are still problems saving new or modified Pipes, though. That problem exists since at least 2 months….
Peter, thank you for including us in your review. I am happy you like the Dlvr.it service. Stay tuned. LOTS of good things coming. Including many more outputs – Facebook coming very soon.
Also, thanks for the feedback on the stats. In order to “provide interesting statistics,” mash the data and provide some real intelligence we use the Dlvr.it short URL. It makes the data consistent and allows us to do some interesting analysis – lots of things going on in the lab now.
Stay tuned…
Best,
Bill Flitter
Founder, dlvr.it
comprehensive post! Another tools for RSS to twitter is hootsuite, I havent tried it out.. but it gives the feature.
I just discovered Hootsuite’s RSS-to-Twitter function. Will try it out, and update the post.
Thanks for the reminder.
Peter
I agree I think social media is really more important for better communicating with your supporters and building a strong foundation and network of people interested in your cause.
These tips are great. Thanks for sharing.
[...] Which blog platform is the most customizable? http://www.blogtips.org/selecting-a-blog-platform-customizability – view page – cached [...]
[...] How successful is your Blog? [...]
[...] This post was mentioned on Twitter by Blog Tips [...]
[...] Blogging for Nonprofit | How to use Posterous as an alternative for Twitpic? [...]
[...] This post was mentioned on Twitter by Blog Tips, Old Mac Donald. Old Mac Donald said: New RSS feed to Twitter tools: Google and Feedburner entered the market of RSS-to-Twitter and URL shortening tools. Will http://url4.eu/wj97 [...]
[...] here to see the original: New RSS feed to Twitter tools Tweet This Or Share Through Other Social [...]
[...] http://www.blogtips.org/browser-size-revisited/In a previous post, we looked at the optimum “blog real estate”: the surface of your blog which is visible on a visitor’s browser window. I mentioned that, according to the statistics from 350000 visitors my main blog, 91% was using a monitor width of 1024 pixels …. You mention Newsgator as a product, but the link to their website shows Newsgator is a company with a lot of products – it isn’t clear which tool or product you used to amalgamate feeds and produce script. … [...]
[...] This post was mentioned on Twitter by Tweeters Tools, R Dilip Kumar. R Dilip Kumar said: What is the best website width? http://bit.ly/8TKfo1 [...]
[...] Visit link: What is the best website width? [...]
[...] Ernest Hemingway’s writing tips for bloggers Ernest Hemingway’s Top 5 Tips For Writing Well Hemingway on writing [...]
[...] How to lay out the home page of your blog? [...]