Follow BlogTips via RSS Get BlogTips updates via Email Follow @SM4NP - Social Media for NonProfit

How to combine RSS feeds with Yahoo Pipes

Posted on Jan 16th, 2011 by

Pipes

Many bloggers use RSS feeds to check on the latest posts from different websites and blogs. Bloggers often use widgets to display RSS feeds from related blogs, or their Twitter and Delicious updates. Or they might simply pull RSS feeds into an RSS reader like Google Reader.

Comes a time where it might be useful to combine different RSS feeds into one feed. Imagine you have a blog about fundraising, and you have bookmarked 10 different fundraising sites of interest to you. You would like to display in an RSS widget on your blog with the latest updates on those blogs. You would not want to have 10 widgets, each for one RSS feed, would you?

The easiest is to combine different feeds into one. And Yahoo Pipes is still the best tool to do so.

Yahoo Pipes is a versatile tool you can use to combine different content, mash it up to create new content, but here is a step by step tutorial for the most simple thing you can do with Yahoo Pipes: combine different feeds.

Step 1: Getting started with Yahoo Pipes

  1. Register for a Yahoo account
  2. Go to Yahoo Pipes and log in with your Yahoo account

Step 2: Create a Pipe

A “Pipe” is a workflow you create to take RSS (or other) input, mix it up, and create a new output.

On the home page of Yahoo Pipes, click on “Create a Pipe”. This will give you a worksheet to construct a pipe, using a nice graphical user interface.
In the left column, you will see a collapsible menu with the main working modules, and the main part of the screen is your worksheet:

Yahoo Pipes main screen

Step 3. Create a workflow for your pipe

  1. The first thing we have to do, is to tell Yahoo Pipes where to get the input RSS feeds from, the feeds we want to combine. From the left module list, drag the module “Fetch Feed” (from the “Sources” list) into your work sheet on the right:

    Dragging a Yahoo Pipes Fetch Feed

  2. Note that once you dragged the “Fetch Feed” module, Yahoo Pipes automatically also puts a second module “Pipe Output” in your worksheet. Leave it there for the time being.
  3. The “Fetch Feed” module will open up, and you can fill in the different feeds you want to combine.
    As an example, let’s put two feeds in here, one from BlogTips and one from Blogging Today. In this case, both are Feedburner feeds, but it does not matter. You can take Atom feeds, RSS 1.0 or RSS 2.0. Yahoo Pipes will process them properly.
  4. After filling in the first feed URL, click the “+” icon and fill in the second feed.

    Fetching feeds with Yahoo Pipes

  5. At any time during the process of creating a pipe, you can click on a module, and as it gets highlighted in orange, the debugging window at the bottom of the sheet will change when you hit “refresh”.

    Yahoo Pipes debugging screenThe debugging module basically shows the output of that module. You can click the triangle icons to see the different internal components.
    This is very useful if you create complex pipes, but for now, we will leave it aside.

  6. The next step is to connect the “Fetch Feed” module to the “Pipe Output” module: Click on the blue dot (the connection dot) at the bottom of the “Fetch Feed” module, and drag it to the blue dot at the top of the “Pipe Output” module:

    connecting two modules in Yahoo Pipes

  7. And in principle you are done now.

Step 4. How to combine many many many RSS feeds

While each “Fetch Feed” module will let you enter about 20 feeds, you might need to combine more feeds than that. Easy!

  1. Drag another “Fetch Feed” module from the left “Sources” list into your worksheet
  2. Fill in your feeds.
  3. If you still have the first “Fetch Feed” module already connected to your “Pipe Output” module, disconnect it: Just hover over the connecting blue line, and scissors will appear. Click on the scissors, and the connection line will disappear.

    disconnecting modules in Yahoo Pipes

  4. Now you need to combine both “Fetch Feed” modules into one. Drag the “Union” module from the “Operators” module list on the left, into your worksheet.

    union module in Yahoo Pipes

  5. Now connect the two “Fetch Feed” modules to the input connectors of the “Union” module (at the top) and connect the output connector of the “Union” module to the “Pipe Output” module.

    connecting the union module in Yahoo PipesRemember, you only need to do this, if you have a lot of input feeds. Each “Fetch Feed” module will easily cater for 20 input feeds.

  6. If need be, you can add more “Fetch Feed” modules and connect them all to the “Union” module. I suggest to limit to 60-80 input feeds per pipe, otherwise the pipe becomes too slow.

Step 5. Save your pipe

Congratulations, you have just constructed a pipe which combines your RSS feeds. Now save the Pipe.

  1. Click on the tab “Untitled” at the top left of your screen, and give the pipe a name, and click “OK”

    give a name to your Yahoo Pipe

  2. Now click on the “Save” button in the right corner
  3. If you have many input feeds in your pipe, saving might take a while.
    There have been times where Yahoo Pipes had performance problems and saving just hung, or gave obscure error messages, but it seems those times are now past. (touch wood)

Step 6. Create an RSS output from your pipe

While you have now successfully created a Pipes workflow, you still don’t have the combined RSS output of your pipe yet, do you? Two simple steps to get there:

  1. After you saved your pipe, click on “Run Pipe”
  2. A new window will open, and a screen will appear showing “Running Pipe”

    running yahoo pipes

  3. Dependent on how many feeds you combine, this might take 2 or take 60 seconds, before the items in your feed will appear.
    Do check the bottom of your feed to see if there were any errors. You might have entered a wrong feed, or entered a home page URL instead of a feed, a common mistake!
  4. Your pipe, combining your different RSS feeds, can be used with different tools (add it to My Yahoo!, to your Google Home page, to your Google Reader, get a badge to put the output on your blog, etc…). If you just want the link to the RSS output, click on “Get as RSS”

    Get output from Yahoo Pipes

  5. And voila, there is the RSS output. Copy the URL of the RSS output and you can reuse it anywhere you want.

    RSS output from Yahoo Pipes

Step 7. Refining your Yahoo Pipes output

Now we’re done. Well… except for one thing. Well, except for two things. Well, except for three things:

  1. As it stands right now, the RSS output does not have the feed items sorted by date, which is no good. We will need to sort the RSS feed, and publish it with the most recent RSS items first.
  2. If you combine many different feeds, the RSS output might contain many items, and that could clog up wherever you want to reuse the feed. We will need to limit the amount of items in the Yahoo Pipes’ RSS feed.
  3. And again, if you use many different RSS feeds as input, one input blog might have re-used input from the other, so we will want to take out duplicate entries.

So roll up your sleeves once more for the “final act”:

  1. Disconnect the “Union” and the “Pipe Output” module
  2. From the “Operators” menu on the left, drag the modules “Sort”, “Unique”, “Truncate” onto your worksheet.
  3. Connect them in this order, and fill in the different fields:

    refining a yahoo pipe

  4. In the “Sort” module, select “item.pubDate” and “descending” order
  5. In the “Unique” module, select “item.title”
  6. In the “Truncate” module, fill in the maximum number of items in your output field. We used 40 in our example.
  7. Connect the different modules (From “Union” or “Fetch Feed” if you don’t use the “Union” module, to the “Sort”, the “Unique”, the “Truncate” and the “Output Pipe”.
  8. Save the pipe, and you are done. Yahoo Pipes will automatically change your output RSS according to the modifications you saved from your worksheet.

So now you are really done. Unless if you want to…

Step 8. Do more sophisticated stuff with Yahoo Pipes. (optional)

Yahoo Pipes allows you to do many many many different things. You can take input and mash it up to create virtually new content. You can check out the pipes others have created clicking “Browse” on Pipes’ home page. People come up with the weirdest and most sophisticated ideas, using translators, automatically integrating FLICKR pictures in a pipe, etc…

However, the most commonly needed option, is to change or filter out some of the content from the input pipes. For that, the “Regex” module from the “Operators” menu is the most used workhorse.

There are few “programming syntaxes” which are as geeky as “Regex”, so beware, this is not for the “faint of heart”. But just one example: Let’s say you want to delete all images from the input feeds, and generate a “text only” output feed. And as we started this post saying you have different blogs and sites about “fundraising”, let’s change the output marking any occurrence of the word “fundraising” in bold:

  1. Disconnect the “Truncate” module from the “Pipe Output” module.
  2. Drag the “Regex” module from the “Operators” menu on the left, into your worksheet
  3. In the “In” field, select which part of the RSS feed you want to work with. In most feeds, the actual content of every RSS item is stored in the “item.description” field, so select that field
    Some RSS feeds store their content in other fields, so you might want to use the debugger view of Pipes to find out where the content is stored.
  4. To replace all images, we need to take out the “img” tag from the content. So we want to replace anything from the beginning of the img tag until the end, with “nothing”. The Regex formula for that is:(<img.*?>)
    …which means: take any string, beginning with “<img,” following by anything (the “.*?” part) and ending with “>”. We need to replace that with “nothing” so leave the “with” field empty.
    And we want to replace all the occurrences of this in the input, so we click the “g” option in the regex module.
  5. We will now look for all occurences of the word “fundraising” (including those starting with a capital “F”), and put the “bold” tags around it. Pfft, easy!
  6. Our regex module will then look like this:

    Yahoo Pipes Regex module

  7. Don’t forget to connect the “Truncate”, “Regex” and “Pipes Output” module..
  8. Save your pipe, and once again, you are done.

As said, you can go as complex as you want. If you get into trouble, call on your peers for support in the Yahoo Pipes Discussion Forum. People will be ready to help you.

9. What I use Yahoo Pipes for?

I use Yahoo Pipes for many different things.

I have different pipes where I take feeds from dozens of news sites and combine them into a single RSS output, which I read using Google Reader, both on my laptop, iPad and iPhone.

I also use Pipes to channel different feeds onto several Twitter and Facebook accounts, using a series different techniques.

But I mostly use Yahoo Pipes to combine, process and mashup feeds from over 1,000 sites and blogs about aid, development, humanitarian issues and the environment. I republish summaries, as I describe in an earlier post.
Check out the end result in Humanitarian News, AidNews, AidResources, News On Green, AidBlogs and The NonProfit Blogs.
In Blogging Today I also aggregate the latest updates from “Blogs about Blogging”, and The Weird News keeps a smile on my face…

Have fun with Pipes!

Picture courtesy Rajtilak




40 Comments to “How to combine RSS feeds with Yahoo Pipes”

  1. Pipes Tutorial says:

    Very nice post on Yahoo Pipes, and well explained. I think a lot of people underestimate what Yahoo Pipes is and what it can do. Definitely it’s most common use is combining multiple RSS feeds into one, but it can also mashup content from around the web by using different modules. It’s a great tool for everyday Internet users and affiliate marketers.

    Thanks for sharing :)

  2. amanda says:

    hello…
    I have 2 RSS feeds that I want to combine. One contains pictures, and the other text.

    I want to combine the 2 RSS feeds into one so that each article that comes out the other end contains both 1 picture and text.

    I’ve tried the ‘Union’ operator, but that just combines the two RSS feeds into one. End result is that I get one article which has the picture, and the next article contains the text… I want both elements to be in the single article…

    Any ideas would be hugely appreciated.

    Regards

    • Peter says:

      Hi Amanda,
      What you describe is indeed how “Union” works: It combines feeds, it does not combine feed items, which is more complex. I would suggest you ask this question in the discussion forum of Pipes.

      P.

  3. Ben says:

    I have 10 feeds where I want to take the most recent item in each feed and combine those into one feed. How would I do that?

    • Peter says:

      Ben,
      while the Pipes discussion forum is the best place for support questions, I am always happy to help.

      I would, for each feed: Fetch feed -> connected to a “truncate” module where you truncate to “1″ (the most recent item) and then combine all 10 of these with the “union” module.

      As the union module only combines max 5 feeds, you will have to combine 5 at a time, and run from those two union modules to another one, to combine all.

      Hope this helps,

      Peter

  4. Zach says:

    Hey, love your article–I’m very excited to explore and shocked that Yahoo/other readers don’t make this easier. One question…

    “You might have entered a wrong feed, or entered a home page URL instead of a feed, a common mistake!”

    Sorry, I know I’m a newbie, but where do I find these feed URLs to begin with?!

  5. Hethir says:

    Thank you so much for this. I just discovered Pipes after the RSS combiner I was using shut down. Your tutorial helped so much, esp. with having the feed post in the correct order/date.

  6. ChattNow says:

    You are an angel. Thank you for saving my sanity with this! It seems so obvious that the Sort, Truncate and Unique are the most common things that anyone would want to do, yet I was in despair of ever finding this out from other tutorial sources.

    THANK YOU.

  7. Jacob says:

    Great post, and very useful. I’ve been trying to find a way to combine feeds for a while, and Yahoo Pipes was lost to me until I found your post. Thanks!

  8. Pedro says:

    Hello Peter, i’m having some problems to do this.
    Everything is supposed to be fine, the problem is at the end. When I run the pipe looks ok, but when i paste the rss feed address into my twitterfeed account, it doesn’t validates it, and I have to keep change the way it posts the feed every time, i even go to feed validator and it says that it won’t valide cause it has some errors due to the news, because i’m using about 200 sources, i don’t know what to do :(

    • Peter says:

      Contacting you by email, Pedro

    • Peter says:

      Think we resolved this issue via email. The main problem is that Yahoo Pipes can’t really deal with more than +- 40 input RSS feeds at the same time.
      If you use more, the output RSS feed from Pipes times out.

      So the solution is to split the big Pipe in several small ones.

      Peter

  9. plumbing says:

    For me, yahoo pipes is very common today. Thank you so much for teaching us how to combine rss feeds.

  10. Janna says:

    Great tutorial, thank you! I love how you start with the easy steps and make the more complicated stuff optional at the end. Well done.

  11. jean says:

    How do you control “Read timed out” error?

    I occasionally bump into a feed that becomes unresponsive, then “Read timed out” error occur and everything stops.
    So a single “Read timed out” error in the 1000 feed and the whole thing is gone, you can’t get any result past that feed.

    Doesn’t happen so often, but it’s very annoying

    • Peter says:

      Normally Pipes v2 skips the feeds that don’t respond in time, or give errors, and continue with the rest.
      Are you sure the “Read timed out” is for a single feed?

      I am having problems with v2 when reading more than 25 feeds in a single pipe. It reads about 15-20 feeds and then skips the rest. It seems v2 is not handling massive feeds well.
      I found no work around in Pipes. For my aggregators pulling together many feeds, I now have a (hidden) wordpress blog which does nothing else but read all the feeds, and output a single (aggregated) feed. For large amount of feeds, it seems much more reliable than Pipes.

      Peter

      • jean says:

        thanks for sharing Peter, it seems Pipes is not reliable enough for what I have to do. If WordPress can do it, all good!

        I read RSS urls from a shared spreadsheet, and use a Pipes Loop on them for aggregation. It chokes with “read timed out” sometimes.
        I never got it working properly in fact!

        • Peter says:

          Hi Jean,

          I do exactly the same: read my feed URLs from a Google spreadsheet.
          See this message on the Pipes discussion board: From the 100 feeds in the spreadsheet, only about 20-25 get processed.
          Look at the answer I got: it seems the problem is with the max run time for Yahoo Pipes.

          I do it easier now: I run a VPS server, and import 1,000 feeds into a (hidden) WordPress blog, using a plugin called FeedWordPress. Because it runs on my server, I can tweak the timeouts.
          For mass-aggregation, I found this mechanism fully reliable…

          Peter

  12. Mark says:

    How often does Pipes regenerate (re-run) itself to update the pipe? Is there a setting for that?

    • Peter says:

      Hi Mark,

      There are 3 answers to that question, dependent on the way you look at it:

      1/ Refresh of input feeds:

      The refresh rate depends on the RSS feeds you aggregate.

      Pipes caches all the feeds that it visits and tries hard not to download feeds from the same site too often. To help Pipes out, be sure your feed is properly setting the Last-Modified, ETag or Expires fields. Pipes will honor any of these standard HTTP headers.

      if the feed does not have any of these headers, this post claims, the default is about 30 minutes. Even though this is old (2009) information, that seems about right in my experience.

      There is no variable you can set to force more frequent polling.

      2/ Any changes you make to the pipe algorythm
      On the other hand, if you make changes in your pipe algorythm, e.g. when you debug a pipe, the output will automatically change, according to the edits you made (taking the same input pipes).

      3/ RSS output from your pipe
      While the browser output from your pipe changes immediately, dependent on your edits, the RSS they generate from that output (if you use an RSS output), seems to be cached, and only refreshed with the newest feedinput/pipe-edits every 15-30 minutes, it seems.

      In general, my experience has been, with the 1,200 feeds I currently aggregate, seldomly any new post was missed. The only real limitation is not the time interval, but the number of feeds you aggregate. One pipe can only deal with 20-30 input RSS feeds at the same time. If you put in more, it wont give an error, but it will not poll the “overflow”.

      Hope this helps.

      Peter

  13. Osi says:

    As of the time of this comment. Yahoo Pipes crashes after you add in 4 RSS feeds and click Save.

    This makes the product Alpha/Beta stage at the most.

    • Peter says:

      Strange, since the V2 release a year ago, Pipes has been very stable for me. Suggest to check on the Pipes discussion forum if anyone else has this problem or if it is related to your pipe particularly..

  14. Jules says:

    Hi Peter

    I have 11 feeds going into various unique filters which then go through a union module, sort and truncate.

    I would like to have the name of the source displayed in my feed output. Is this possible – if so how do i go about this?

    Here is my feed is that helps:

    feed://pipes.yahoo.com/pipes/pipe.run?_id=edc623e83e5fdb17f7778f4d25230fbe&_render=rss

    PIPE is – swanseacitynewspod

    Any help is appreciated.

    Thanks
    Jules

  15. Gordana says:

    Hello Peter!

    Thanks for this very good post. I want to organize my fetch feed boxes, so I want to ask you is it possible to give them names? Also, can I connect two or more pipes into one?

    Thank you in advance,
    Gordana

    • Peter says:

      Dear Gordana… no, you can not give names to fetch feed boxes, unfortunately.
      In terms of “connecting” pipes, do you mean combining the output of two separate pipes?
      Easiest way is to take the RSS output of two pipes and create a 3rd pipe that combines those two feeds…

      Peter

  16. Pete says:

    Hi Peter,

    Am new to Yahoo Pipes and have copied an existing working pipe for a Twitter bot that retweets a hashtag from a Twitter list.

    Only problem is, the pipe I’ve created here:

    http://pipes.yahoo.com/pipes/pipe.info?_id=fd8d903424fe1c20ee6c242fb9ad6ef9

    The debugger returns the following…

    RT:@EDPchrislakey/statuses/231001610451566593EDPchrislakey: Seen the Canary mag? #pinkun contribution of PL teams is a terrific guide to the top flight. Rest of it isn’t shabby either.

    What do I need to add/remove so that:

    RT:@EDPchrislakey/statuses/231001610451566593EDPchrislakey:

    appears as:

    RT: @EDPchrislakey

    Many thanks,

    Pete

  17. Shawn says:

    This looks so close to what I need ‘but’ I have a couple of odd requirements:

    1. My feeds are secure/requiring passwords – they reside in SharePoint based on business data
    2. My CEO wants a rollup of feeds from SharePoint to give status on projects
    2a. He would love to get them in EMAIL…

    Thought/Suggestions?

  18. Angela says:

    Hi, I followed your instructions to mashup two feeds, but it won’t order by date even with the sort module in there.

    Is this sort function not working anymore?

    http://pipes.yahoo.com/pipes/pipe.run?_id=8c60b5dfac34d9b8299c77bf4ddef850&_render=rss

    • Peter says:

      Those posts seem perfectly sorted to me, Angela.. One thing though: if you change a feed, it takes about 30 minutes to see the output change in the RSS output!

  19. manuel says:

    Hi I would like to know why Union when pipe goes straight to URL output shows results but when I connect Flickr with output pipe junction and does not show me anything. and where I could find that data you enter and exit data of each module to see if I can connections and what not.

    • Peter says:

      Hi Manuel,

      For support questions on Yahoo Pipes, it would be best to go directly to their support forum. However, to answer one of your questions: you can see the output of each module when you click on the top line (the name line) of the module. It will appear in orange, and the output of that module will be shown in the debug window, at the bottom of the page.

      Hope this helps,

      Peter

  20. jags says:

    The images could be bigger to see something. I don’t know why is so difficult to read and to see nothing in a tutorial.

    Mistery…

  21. Thank you for this article. I’ve never used Yahoo Pipes before so this will be interesting. Hopefully tomorrow when I check my email the feed is actually how my sites are today :) . I’ll know tomorrow for sure, but do I have to manually update the feed or will Yahoo Pipes ping the feeds?

Leave a Comment

*