How to combine RSS feeds with Yahoo Pipes
Many bloggers use RSS feeds to check on the latest posts from different websites and blogs. Bloggers often use widgets to display RSS feeds from related blogs, or their Twitter and Delicious updates. Or they might simply pull RSS feeds into an RSS reader like Google Reader.
Comes a time where it might be useful to combine different RSS feeds into one feed. Imagine you have a blog about fundraising, and you have bookmarked 10 different fundraising sites of interest to you. You would like to display in an RSS widget on your blog with the latest updates on those blogs. You would not want to have 10 widgets, each for one RSS feed, would you?
The easiest is to combine different feeds into one. And Yahoo Pipes is still the best tool to do so.
Yahoo Pipes is a versatile tool you can use to combine different content, mash it up to create new content, but here is a step by step tutorial for the most simple thing you can do with Yahoo Pipes: combine different feeds.
Step 1: Getting started with Yahoo Pipes
Step 2: Create a Pipe
A “Pipe” is a workflow you create to take RSS (or other) input, mix it up, and create a new output.
On the home page of Yahoo Pipes, click on “Create a Pipe”. This will give you a worksheet to construct a pipe, using a nice graphical user interface.
In the left column, you will see a collapsible menu with the main working modules, and the main part of the screen is your worksheet:
Step 3. Create a workflow for your pipe
- The first thing we have to do, is to tell Yahoo Pipes where to get the input RSS feeds from, the feeds we want to combine. From the left module list, drag the module “Fetch Feed” (from the “Sources” list) into your work sheet on the right:
- Note that once you dragged the “Fetch Feed” module, Yahoo Pipes automatically also puts a second module “Pipe Output” in your worksheet. Leave it there for the time being.
- The “Fetch Feed” module will open up, and you can fill in the different feeds you want to combine.
As an example, let’s put two feeds in here, one from BlogTips and one from Blogging Today. In this case, both are Feedburner feeds, but it does not matter. You can take Atom feeds, RSS 1.0 or RSS 2.0. Yahoo Pipes will process them properly.
- After filling in the first feed URL, click the “+” icon and fill in the second feed.
- At any time during the process of creating a pipe, you can click on a module, and as it gets highlighted in orange, the debugging window at the bottom of the sheet will change when you hit “refresh”.
The debugging module basically shows the output of that module. You can click the triangle icons to see the different internal components.
This is very useful if you create complex pipes, but for now, we will leave it aside.
- The next step is to connect the “Fetch Feed” module to the “Pipe Output” module: Click on the blue dot (the connection dot) at the bottom of the “Fetch Feed” module, and drag it to the blue dot at the top of the “Pipe Output” module:
- And in principle you are done now.
Step 4. How to combine many many many RSS feeds
While each “Fetch Feed” module will let you enter about 20 feeds, you might need to combine more feeds than that. Easy!
- Drag another “Fetch Feed” module from the left “Sources” list into your worksheet
- Fill in your feeds.
- If you still have the first “Fetch Feed” module already connected to your “Pipe Output” module, disconnect it: Just hover over the connecting blue line, and scissors will appear. Click on the scissors, and the connection line will disappear.
- Now you need to combine both “Fetch Feed” modules into one. Drag the “Union” module from the “Operators” module list on the left, into your worksheet.
- Now connect the two “Fetch Feed” modules to the input connectors of the “Union” module (at the top) and connect the output connector of the “Union” module to the “Pipe Output” module.
Remember, you only need to do this, if you have a lot of input feeds. Each “Fetch Feed” module will easily cater for 20 input feeds.
- If need be, you can add more “Fetch Feed” modules and connect them all to the “Union” module. I suggest to limit to 60-80 input feeds per pipe, otherwise the pipe becomes too slow.
Step 5. Save your pipe
Congratulations, you have just constructed a pipe which combines your RSS feeds. Now save the Pipe.
- Click on the tab “Untitled” at the top left of your screen, and give the pipe a name, and click “OK”
- Now click on the “Save” button in the right corner
- If you have many input feeds in your pipe, saving might take a while.
There have been times where Yahoo Pipes had performance problems and saving just hung, or gave obscure error messages, but it seems those times are now past. (touch wood)
Step 6. Create an RSS output from your pipe
While you have now successfully created a Pipes workflow, you still don’t have the combined RSS output of your pipe yet, do you? Two simple steps to get there:
- After you saved your pipe, click on “Run Pipe”
- A new window will open, and a screen will appear showing “Running Pipe”
- Dependent on how many feeds you combine, this might take 2 or take 60 seconds, before the items in your feed will appear.
Do check the bottom of your feed to see if there were any errors. You might have entered a wrong feed, or entered a home page URL instead of a feed, a common mistake!
- Your pipe, combining your different RSS feeds, can be used with different tools (add it to My Yahoo!, to your Google Home page, to your Google Reader, get a badge to put the output on your blog, etc…). If you just want the link to the RSS output, click on “Get as RSS”
- And voila, there is the RSS output. Copy the URL of the RSS output and you can reuse it anywhere you want.
Step 7. Refining your Yahoo Pipes output
Now we’re done. Well… except for one thing. Well, except for two things. Well, except for three things:
- As it stands right now, the RSS output does not have the feed items sorted by date, which is no good. We will need to sort the RSS feed, and publish it with the most recent RSS items first.
- If you combine many different feeds, the RSS output might contain many items, and that could clog up wherever you want to reuse the feed. We will need to limit the amount of items in the Yahoo Pipes’ RSS feed.
- And again, if you use many different RSS feeds as input, one input blog might have re-used input from the other, so we will want to take out duplicate entries.
So roll up your sleeves once more for the “final act”:
- Disconnect the “Union” and the “Pipe Output” module
- From the “Operators” menu on the left, drag the modules “Sort”, “Unique”, “Truncate” onto your worksheet.
- Connect them in this order, and fill in the different fields:
- In the “Sort” module, select “item.pubDate” and “descending” order
- In the “Unique” module, select “item.title”
- In the “Truncate” module, fill in the maximum number of items in your output field. We used 40 in our example.
- Connect the different modules (From “Union” or “Fetch Feed” if you don’t use the “Union” module, to the “Sort”, the “Unique”, the “Truncate” and the “Output Pipe”.
- Save the pipe, and you are done. Yahoo Pipes will automatically change your output RSS according to the modifications you saved from your worksheet.
So now you are really done. Unless if you want to…
Step 8. Do more sophisticated stuff with Yahoo Pipes. (optional)
Yahoo Pipes allows you to do many many many different things. You can take input and mash it up to create virtually new content. You can check out the pipes others have created clicking “Browse” on Pipes’ home page. People come up with the weirdest and most sophisticated ideas, using translators, automatically integrating FLICKR pictures in a pipe, etc…
However, the most commonly needed option, is to change or filter out some of the content from the input pipes. For that, the “Regex” module from the “Operators” menu is the most used workhorse.
There are few “programming syntaxes” which are as geeky as “Regex”, so beware, this is not for the “faint of heart”. But just one example: Let’s say you want to delete all images from the input feeds, and generate a “text only” output feed. And as we started this post saying you have different blogs and sites about “fundraising”, let’s change the output marking any occurrence of the word “fundraising” in bold:
- Disconnect the “Truncate” module from the “Pipe Output” module.
- Drag the “Regex” module from the “Operators” menu on the left, into your worksheet
- In the “In” field, select which part of the RSS feed you want to work with. In most feeds, the actual content of every RSS item is stored in the “item.description” field, so select that field
Some RSS feeds store their content in other fields, so you might want to use the debugger view of Pipes to find out where the content is stored.
- To replace all images, we need to take out the “img” tag from the content. So we want to replace anything from the beginning of the img tag until the end, with “nothing”. The Regex formula for that is:
…which means: take any string, beginning with “<img,” following by anything (the “.*?” part) and ending with “>”. We need to replace that with “nothing” so leave the “with” field empty.
And we want to replace all the occurrences of this in the input, so we click the “g” option in the regex module.
- We will now look for all occurences of the word “fundraising” (including those starting with a capital “F”), and put the “bold” tags around it. Pfft, easy!
- Our regex module will then look like this:
- Don’t forget to connect the “Truncate”, “Regex” and “Pipes Output” module..
- Save your pipe, and once again, you are done.
As said, you can go as complex as you want. If you get into trouble, call on your peers for support in the Yahoo Pipes Discussion Forum. People will be ready to help you.
9. What I use Yahoo Pipes for?
I use Yahoo Pipes for many different things.
I have different pipes where I take feeds from dozens of news sites and combine them into a single RSS output, which I read using Google Reader, both on my laptop, iPad and iPhone.
I also use Pipes to channel different feeds onto several Twitter and Facebook accounts, using a series different techniques.
But I mostly use Yahoo Pipes to combine, process and mashup feeds from over 1,000 sites and blogs about aid, development, humanitarian issues and the environment. I republish summaries, as I describe in an earlier post.
Check out the end result in Humanitarian News, AidNews, AidResources, News On Green, AidBlogs and The NonProfit Blogs.
Have fun with Pipes!
Picture courtesy Rajtilak