If you have a selfhosted WordPress blog (WordPress.org), take urgent measures to secure your site from a recently discovered vulnerability.

Many WordPress themes and plug-ins use a script called “timthumb” (timthumb.php). This is the most common code used to create thumbnails from pictures.

End July, a vulnerability surfaced showing external users could dump malicious code onto your site. Typically, a short piece of .php code is uploaded via a timthumb backdoor. This hacking code then creates a wider backdoor to gain pretty much full access to your site.

It looks like the hackers were on holiday too, and are only gearing up their activity right now. Many sites were hacked in the last couple of days. As many sites use timthumb.php, we can foresee a major hacking spree in the next weeks.

So it is high time to secure your selfhosted WordPress site now.

How to check if you have been hacked via timthumb.php?

There is not one specific signature to this hack, contrary to the shared hosting hack last year, but here are some common things that seem to happen:

• First… Check if you actually use “timthumb.php” on your site. It does not come with the default WordPress installation, so check if any of your uploaded themes contain the file “timthumb.php”.
  Do a site wide search (with SSH or SFTP). Some popular plug-ins that use timthumb.php are “WordPress Popular Posts” and “WP Mobile Detector”.
  Many themes use timthumb.php, or a variation of it. E.g. the widely used “Thesis” theme uses it as “thumb.php”.
• If you find the timthumb.php in your plugin or themes directory, you’d better give your site a thorough check, so check further
• The hackers often upload .php files in the timthumb upload directory “/cache” (a subdirectory from the one where the timthumb script is stored). You should check that directory, and delete any non-picture files (.html .php,…)
• Often hackers upload .php files to several other subdirectories within your WordPress installation. I have seen them in the “/upload” “/supercache” directories (and their subdirectories) as well as in the directories for plugins and themes. Delete them.
• Recently, the hackers got bolder and entire subdirectories were uploaded. First a .zip file would be uploaded, it would be unzipped and an entire sub-site was installed in one of the WordPress directories. I have seen zipfiles called halifaxsecurity.zip, hal.zip, studentloanupdate.zip, student.zip. Malicious subdirectories I detected on other sites, were called /halifaxsecurity, /hal and /studentloanupdate. Delete those, if you find them.
• People also report direct hacks in .php files and style sheets, adding malicious code (similar to the last year’s hacks).
• Check your .htaccess files
• ..

Check also Sucuri’s blog for more hack signatures and scripts, and Mark Maunder’s blogpost for a full description of the timthumb vulnerability.
List of themes and plugins (non-exhaustive, though) using timthumb.php, you can find on Big Webmaster and Sucuri’s blog.

Deleting those malicious files is not sufficient, as it still leaves the backdoor open for future hacks, so you need to secure your timthumb.php code NOW ! Read on:

How to secure timthumb.php against hacks?

1. Locate all instances of timthumb.php (or any renames of it) on your site.
2. Download the newest timthumb.php code (Check also the plug-in’s home page)
3. Replace the old timthumb.php with your downloaded code.
4. While the new code is already secure, I strongly suggest to limit the access from external sites.
   Replace the line:
   define ('ALLOW_EXTERNAL', TRUE);
   with:
   define ('ALLOW_EXTERNAL', FALSE);

Good luck!

Picture courtesy TGDaily
With thanks to Michael Marus

You can classify blogs into categories using many different criteria. From a blog-administrator’s point of view, the main classification whether a blog is self-hosted on your own server (such as WordPress.org) or hosted by the blogging service itself (such as Blogger, Tumblr, Posterous, WordPress.com,…)

Blogs hosted by the blogging service make you dependent on their uptime and functionality. Self-hosting blogs allow you to take the tiller in your own hands, giving you much more freedom to sculpture the blog as you see best fit. But it also dumps all the work to backup, upgrade and maintain your blog into your lap. “With freedom also comes responsibility”, did your parents not tell you that when you were allowed to go to a party for the first time?

I have posted many times before about the work involved to keep self-hosted blogs up-to-date and up-and-running.

I just wrapped up yet another adventure with one of my selfhosted blogs. Let me share it with you, as this is yet another reminder to all those of you thinkering about self-hosting your blog.

When things go wrong…

News On Green is my main aggregator for environmental news. It is hosted on my Hostgator VPS server, like most of my other blogs, and has about 130,000 blogposts. I monitor this server’s performance continuously and three days ago, I saw the resource hogging going haywire. The amount of CPU time consumed went through the roof, free memory got exhausted and everything slowed down up to the point where I could not even log into the server anymore.

I had seen instances like that before, when the caching broke on some of the blogs. As my sites get a lot of traffic (currently around 70,000 visits per month, excluding all the RSS feeds polls, search engine crawlers,..), proper caching is critical. If caching breaks on even one of the sites, the server has to go to its MySQL database for every single page visit, and the server goes belly up under the CPU load.

This is apparently what happened on News On Green, where I use the WP Supercache plugin for caching. For each individual page or post on your blog, Supercache creates a subdirectory in the /cache/supercache directory. That’s where it stores a “pre-cooked” .html file, rather than executing a MySQL query for each page. HTML files are “served” much faster than MySQL queries.

Debugging, once more…

To help debugging, Supercache also puts a comment line at the bottom of the HTML page for each post. It contains some statistics or error messages. This time, I saw the error “can not create /cache/supercache/subdirectory/xxx.tmp”.

The error.log came up with all sorts of errors, including:

PHP Notice: Undefined index: HTTP_ACCEPT in xxx/wp-content/plugins/wp-super-cache/wp-cache-phase1.php on line 415

So Supercache was no longer caching the pages, so far was clear. But the question was why.

I checked the /supercache directory for the blog, and found that I had 31,998 cached pages, thus 31,998 directories. One subdirectory for each cached page or post. When I see numbers anywhere close to a magic “^2″ figure, I get suspicious: 32, 64, 16,000, 32,0000… And as “31,9998″ pages was close to one of the magic “^2″ figures (32,000), I suspected a system or account limitation on my server.

I called up my hosting company, Hostgator via an online chat channel. The technician was really patient. Whereas on other hosting services, they would have answered “This is a WordPress problem”, their support took the trouble of checking several system parameters. But we could not find anything limiting the number of files or sub directories per directory…

I was still convinced I hit some hard threshold. As an experiment, I deleted some 100 cached pages, and lo and behold, the cache started working again. Eureka, workaround found! But what was the cause of the problem?

While online, I searched the internet for “maximum Linux subdirectories”, and found this post. There seems to be Linux system parameter (for filesystems EXT4, EXT3, EXT2 – which are used on most Linux hosts). The maximum number of Linux subdirectories in one single directory is… 31,998.

Bingo! That was exact the amount of subdirectories I had in my cache directory.

The solution…

So what was the solution? I deleted the oldest 5,000 cached pages. As if by miracle, the crippled server resurrected like a phoenix.

The permanent solution will be, of course, to automate this check: a CRON job should automatically trim the oldest cached pages, once I come close to the limit of 32,000. Or 31,998 to be exact. Work to do in the next days.

Now how is that for a nerdy story, hey?

 
Picture courtesy Kaboodle

A few months ago, I migrated 7 blogs with 350,000 blogposts from Tumblr to WordPress on my VPS server.

Together with some other blogs and websites, that server happily processed about 50,000 visitors per month, with a steady increase of about 10% per month. All sites and the SQLserver were properly tuned, and the server purred happily like a cat next to a stove.

Until this week. Help!

All of sudden, I could see an excessive amount of PHP requests coming in, and the server’s load average went from 1-2 to 10-20. That means that at any moment, 20 processes were waiting for CPU time… Everything slowed down. What now?

It turned out I had two problems:

The cache broke on one of the blogs

AidNews, one of my main aggregator blogs, showed to give an excessive PHP load. That wasn’t normal, as most content was cached, so no need to call PHP…?!

When I loaded some blogposts, and looked at the source file, I could see a debugging error, stating SuperCache could not cache the page. So, one of my most busiest sites no longer cached its pages, making each page load go to the SQL database via a PHP call.

I changed several cache settings, but in vain. What worked on the other blogs no longer worked on this blog. The cache went RIP. In the end, I had to delete the plug-in, erase all cache directories and reinstalled the plug-in.

Problem cured. At least one of the problems. The high server load continued, even without the excessive PHP load from that single blog.

Excessive crawler traffic

After installing the new blogs, I did a good effort to tune the SEO (Search Engine Optimization). Nightly sitemaps are submitted to several search engines, and the crawlers were happy crawling my pages.

But all too happily it seemed. When I checked the crawler stats for each of my sites in Google Webmaster, I found the Google crawler daily visited 5,000 to 10,000 posts for each of the sites. That makes close to 100,000 page visits per day, only for Google.
I mean I like a good crawler rate, but that was excessive. Many of the posts I generate with most aggregation blogs I have on the server, never change. So there is no reason for the crawlers to revisit many posts. Using the same method I described earlier to increase the crawler rate, I forced the rate down.

Voila: a “robots.txt” with two lines code went onto each blog’s root directory, and within 12 hours, the server started purring again… Like a cat next to a stove.

Update Oct 7 2011:
Over the past 14 days, I have been monitoring my server performance more closely, as there were sudden mysterious spikes in activity. Last night was one of those occasions. I took a snapshot of the server, and with the folks over at Hostgator discovered crawlers are having a feast on my websites. At one single moment crawlers were accessing over 7,000 pages simultaneously on one site. 7,000! Few servers will sustain that…
Again, I love crawlers, but I like a fast website more. So now I tweaked the robot.txt to only allow the few popular crawlers, and not more. 95% of the search traffic comes from Google anyway, so why would I care for dozens of unknown crawlers?

Lessons learned?

Well, if you self-host a blog then it’s not “just a matter of configuring the server and the blogs”. You need to keep a close eye on things. From time to time something goes haywire, and unless you act fast, all runs out of control. One more reason to really think hard if you want to switch from a hosted blog platform, to a shared or dedicated server… You need to be ready to spend more time on technical stuff, than on blogging.

Happy blogging!
Picture courtesy Bogenfeund

Moving blogs: labour intensive, but fun!

I had seven blogs on Tumblr which aggregate news. Using a technique I described earlier, they take RSS feeds from over 1,000 carefully selected websites and blogs, filter them, clean them up, and feed them into the different Tumblr blogs. I used the unique feature built into Tumblr to convert RSS feeds into posts. All automatically. Pretty neat. Until it stopped working…

Two months ago Tumblr’s autoimport feature started to hiccup. Tumblr support said “We are aware and working on it”, but could not give me any estimate when it would be fixed.

A month later, still nothing. So what to do? I rely on this Tumblr feature, for my blogs. In two years, those blogs collected 350,000 news articles. Quite a resource library which I did not want to give up.

So I decided to use the Christmas holidays to migrate these blogs from Tumblr onto WordPress, on my HostGator VPS server. It was an interesting process, involving many different techniques and debugging efforts:

1. How to export a Tumblr blog into WordPress

Ben Ward's Tumblr2WordPress utility

I pretty much described the process and the technique to export a Tumblr blog in an earlier post: Using Tumblr2WordPress, a neat PHP program by Ben Ward. It uses the Tumblr API to export blogposts, and to create an .XML file, which I could import into WordPress. An API that Tumblr disables every US afternoon and evening, by the way.

While Ben’s program works well to export smaller Tumblr blogs, I had biiiiig blogs, so I had to adapt the PHP code. As the code is public domain and available on Github, I downloaded it and installed it on my server. I changed the PHP parameters to allocate a massive chunk of memory, and allow the export routine to run longer than a standard PHP program. Tip: change the parameters only for that routine, not for your whole server!

Update March 1, 2010:
Ben’s source code is still available, but the executable program is no longer available on the link I provided. You can still run Tumble to WordPress routines based on the same engine from Tumblr2WP or Tumble2WordPress – With thanks to Parneix and Aaron for the updates)

I also patched Ben’s original code to work around a smaller problem I discovered on “published dates” and “categories”. Pretty easy, even for a PHP novice like me, as the code is well documented.

As WordPress can only import .XML files smaller than 8Mb, I split up the first exported blog manually into smaller chunks. Took me two hours for the smallest of the seven blogs. I decided once again to delve into the code, and wrote a small patch that allowed me to export 5,000 blog posts at a time. Each export file now was smaller than 8 Mbyte.

Cool. Exported all blogs, and there I sat on 70 files, about half a gigabyte worth of .XML files.

2. Importing 350,000 posts into seven new WordPress blogs

The WordPress Import routine rocks!

For the seven blogs, I created seven new accounts on my HostGator VPS server. Some of my Tumblr blogs were using custom domains, so I changed the DNS, pointing to my HostGator VPS server, and created seven new WordPress blogs. While I was at it, I registered two new domains. ChangeThru.Info became AidResources.org and Youandusand.me became NewsOnGreen.org. Wanted to do that a long time ago, so now was the right time. I kept the old domains live on Tumblr, so I did not lose any traffic.

Installing a new WordPress blog was easy to do with the “Fantastico” program in the server’s Cpanel. I choose a neat and simple magazine template and added the usual plugins I always use for caching, automated blog backup, etc…

Then, one by one, I imported the 500 Mbyte of export .XML files. Worked flawlessly, but took about two days. Not a single error, not a single problem. I should say: WordPress impressed me once more.

Done. Well at least with importing the old posts. How to feed in new posts using my myriad of RSS feeds?

3. Implementing an “RSS to blogpost” routine in WordPress

FeedWordPress, you rock!

FeedWordPress made my day. This neat plugin imports RSS feeds and converts them into WordPress blogposts. And it does so very well. The plugin is well designed, easy to use, and has a lot of options to customize the import process.
It also has add-ons that allow you limit the size of the imported post, add text in the title or in the body of the imports. Realllllly neat!

I configured the different feeds I process via Yahoo Pipes, and ran a CRON job to import the blog posts. About two hours work per blog, and the whole cake went into the oven and started cooking: FeedWordPress neatly imported the posts.

You would think I was done. The real work had not even started.

4. The need for speed

From the beginning until the end, including the customization of the template etc.. the export from Tumblr and import into WordPress took me about a week. Fine-tuning the blogs and the server took another two weeks.

As “Good is Fine, Perfect is Best”, I saw some formatting deficiencies I could not live with and needed modifications to the feeds and template. And even worse, much much much worse, my poor server went through its knees with the extra load.. The new blogs demanded so much CPU time from Apache and the MySQL server that everything slowed down to a snail’s pace.

Time to get geeky!

5. Tuning the XML sitemaps

XML Sitemap: Tune it!

It only takes one dumb sysadmin to make the fastest server to go slow, I realized while monitored the CPU load with the Linux “TOP” command. I saw the “load average” to peak way above “10″, meaning there were at least 10 processes queueing up for CPU time.
So I looked at several WordPress plugins that might cause the problem.

The first problem I saw was the XML sitemap generator. There was one option, which I had overlooked: “Rebuild sitemap if you change the content of your blog”. Might be fine on smaller blogs, but the automatic feedimporters were putting up new blogposts at a rate of 100 per hour. So the server was pretty much doing nothing else but generating sitemaps.

I disabled the feature, and scheduled a CRON job to regenerate a sitemap once a day, at night-time. The server load went down significantly.

Oh, by the way, if you have a huge blog, limit the number of posts to include in the sitemap to 10,000 , Google’s maximum limit for sitemaps!

6. Tuning WP Supercache

WP SuperCache: Tune it!

As I mentioned before: any blogger using a selfhosted blog without a cache, should appear before court for “blog neglect”. So I use caching extensively, with WP Supercache, my preferred plugin.

But it only takes a stupid blog administrator to make even the best plugin not to work properly. Supercache needed tuning:

• Unchecked the option “Clear all cache files when a post or page is published” (I published 100 posts per hour, so the cache was always invalidated)
• Pre-loaded the last 10% of blogposts, but put “Refresh preloaded cache” to “0″ (as once a post is imported from its RSS feed, I don’t update it anymore, so it can remain in cache forever). This means I only had to pre-load a massive amount of blogposts once, and it was done.
• For the same reason, I put the “expiry time” to “0″, as once cached after a preload cycle, I want the page to remain in cache. It generates a LOT of cached files, but I have plenty of disk space on my server.
  If you put “expiry time” to a value > 30 minutes, garbage collection is done every 10 minutes, which generates a lot of load on your server.
• As now I had caches with an eternal life time, I needed to ensure the homepage, feeds, archives were NOT cached, otherwise visitors never got an updated overview of the latest posts.
  As I discovered that AFTER I preloaded the posts, and had put the caching to “eternity”, I had to manually delete the cached files for the home page, searches and the running month’s archives.
• To further reduce the load on the PHP server, I choose the option to use “mod_rewrite to serve cache files”.
• And by the way, if you don’t cache the home page, the “Cache Tester” will give an error – as it tests caching on… the home page. So ignore that error, and just look at the source of any random page, to see if, at the bottom of the source, you have a date/time for the cache generation, which is in the past.

7. Trashing “Most Popular Posts”

As describe in this post, one plugin meant to show “the most read posts”, also logged every single access to the SQL database, and effectively slowed down my server. Had to trash it.

8. Tuning FeedWordPress

I spent quite a bit of time to tune FeedWordPress, to balance how often feeds were to be imported with the success rate of each import cycle. I combine 1,000+ feeds into about 20 Yahoo Pipes feeds. These are large and complex feeds, which take a lot of time to fetch. Many times the import of a feed would time out.

At first I worked around that problem, by refreshing all feed imports every 10 minutes. But once again, that put a lot of pressure on the server. As you can deal with any problem either by working around it, or by addressing the cause of it, it was time to look for the source of the problem. In the process of doing so, distinguish well between what is “a cause” and what is “a symptom”. Often we try to solve the latter, while we should address the former. Think about that. That is deeeeep!

So the symptom I saw was the feeds timing out. As I know the Yahoo Pipes’ feeds often take very long to refresh, even interactively, I had to patch FeedWordPress with a timeout of 60 seconds to deal with the Yahoo Pipes’ lack of speed. Cool. But that caused a dreaded SQL error “My SQL server has gone away” to appear more frequently in my CRON log files. Beh.

To make a long story short, the solution was to also change the PHP parameter “wait_timeout” from the default of “30″ seconds to “240″. Changed that in /etc/my.cnf and restarted SQL server. Problem solved.

As this solved the timeout when reading RSS feeds, I could also decrease the frequency of the FeedWordPress CRON jobs. And that once again made my server very happy. I like happy servers…!

9. Server tuning

Tuning a server: not for the faint of heart...

Depending on what exactly you do on your blogs, for large and heavy traffic blogs like the seven I had just migrated, the SQL server might need tuning. This is not for the faint-of-heart, and requires patience and caution. With one wrong setting, you can cause more damage than good.

The first indication that my SQL parameters might need tuning, was simply the fact that SQL took up so much CPU time on my server. phpMyAdmin, a routine available to about every selfhosted server, has a neat feature called “Status”, which gave me an overview of the parameters which might need changing. But I was not sure. So I installed mySQLTuner: using SSH, I logged into my server’s root. With just three commands, I got a better overview of the parameters I needed in three commands:

wget mysqltuner.pl
chmod 775 mysqltuner.pl
perl mysqltuner.pl 

I changed one parameter at the time, and waited for 12-24 hours to see its effect.

It seems the two most important parameters to tune are “key_buffer_size” and “table_cache”. It was advised to tune these first before touching the others. Which I did. “key_buffer_size” was ok on its default value of 48M, but “table_cache” needed 4,096 instead of the default 1,024.

The rest of the parameters I changed over time:

tmp_table_size: from 32M to 64M
max_heap_table_size: from 32M to 64M
sort_buffer_size: from 1M to 4M

While I am still observing the server and tuning bits and pieces, it looked by now that, ladies and gentlemen, we have a happy server, purring  like a happy cat. Sure enough I still get peak loads, with “load average” of 3-4, most of the time it stays at “1″ or below. And that is good. I like happy servers.

10. Template tuning

For the seven blogs, I use a neat little magazine template, called Magazine-Basic by Bavotasan. It comes for free, and has some basic functions that I like.

I customized the CSS and some of the functions. I sinned heavily by patching the original template, rather than using a child theme, but that is just because I made so many changes. On top of that, for my main blogs, “speed” is important, I did not want every page refresh to read several CSS files. So patching, it was. I will live with the fact that I can not upgrade the template automatically later on, but hey, I learned that upgrading templates is always a pain, and often causes more problems than it is worth. So I avoid theme upgrades like the pest. My opinion, punto.

One of the main challenges I faced was that the template, by default, puts the first image if finds in the post as a thumbnail on the home and archive pages. And often that image was junk (a “Retweet” button, or a Feedburner “Email this” image).. So I had to tweak all my Yahoo Pipes feeds to delete those images. Took me a week. Some feeds were too complex to delete all images, so for some of the blogs, I just disabled the thumbnails in the template. I mean, I got to sleep too, so can’t keep on tuning all feeds to find all kinds of combinations of dummy images..

Maybe that will be work for next Xmas!

11. Installing the mobile theme

And to finish it all in beauty, I installed a plugin to enable a mobile theme to be displayed when visitors access my blog using a mobile phone.

12. And here is the end result of about four weeks of work:

From Tumblr to WordPress. With Love

Check them out on your mobile. Browse through the posts to see if you like the download speed (remember the homepage and archives are not cached, and thus a bit slower!):  AidNews, AidResources, News On Green, AidBlogs, The NonProfit Blogs, The Weird Bit and Blogging Today.
You like?

Cartoon courtesy Mark Lowe

Mea Culpa! Drupal is fast, but I made it slow...

Remember my rumbling about how one plug-in can make your WordPress blog slow? Well, here is a similar story about my Drupal news aggregation site.

Before I begin, let me re-state that Drupal by itself, is fast. I mean real fast. That’s why I gave Drupal two thumbs up in my 2010 blogger’s wrapup. But it only takes half a dumb system administrator (or web admin) to make any fast system slow. And I succeeded very well in that.

As I was debugging my server’s CPU load for my WordPress blogs, I saw that my Drupal site, hosted on the same machine, was putting a lot of pressure on my SQL and Apache servers.

Well, the site as such was not slow. As I used extensive caching, a visitor did not see anything. But boy, the site really loaded my server!

I did not understand why. I mean, sure, I have a lot of traffic, but as with most news site, 90% of all traffic goes to the same most recent posts. And that was caught by the cache, so that traffic should not even touch PHP or SQL. Proof of the matter was that as a visitor, the site was very fast, taking only one or two seconds to load.

So why did the site put that much load on the server? I found out be pure luck.

I was checking Google Webmasters how the sitemaps for my different blogs were going, and noticed my Drupal site’s sitemap never gets submitted to Google. So I looked at the module settings. Something I should have done much earlier.

Apart from the module having apparent problems with high volume sites, I also misconfigured it: I had set the module to regenerate a sitemap of the latest 2,000 posts at every CRON (the internal scheduler) run.
On my site, CRON is set to run at every 1o minutes, as I use CRON to trigger the import over 1,000 RSS feeds which need very regular refreshing.

So in short, stupid me, I was forcing Drupal to scan through 2,000 posts every 10 minutes and generate a new sitemap.xml file. You can imagine how many SQL queries that generated unnecessarily.

I disabled the module while I monitored the CPU load interactively. Almost on the spot, I could see the load on the Apache and SQL server going down. And the server has been purring happily ever since.

Since that moment, I am sitting in the corner of my room, with ashes on my head, crying “Mea Culpa, Mea Culpa, Mea Maxima Culpa”* (* “It is my fault, my fault, my freaking fault”), and hope the Server Gods won’t look badly upon me on my Digital Judgement day.

Bottomline:

For selfhosted Content Management Systems like Drupal or WordPress alike: check each of your plugins and their settings. It takes only one plugin or one wrongly configured plugin to either slow down your site, or slow down your server.

Unless if you want to be a stupid web admin like myself, of course.

Many bloggers use RSS feeds to check on the latest posts from different websites and blogs. Bloggers often use widgets to display RSS feeds from related blogs, or their Twitter and Delicious updates. Or they might simply pull RSS feeds into an RSS reader like Google Reader.

Comes a time where it might be useful to combine different RSS feeds into one feed. Imagine you have a blog about fundraising, and you have bookmarked 10 different fundraising sites of interest to you. You would like to display in an RSS widget on your blog with the latest updates on those blogs. You would not want to have 10 widgets, each for one RSS feed, would you?

The easiest is to combine different feeds into one. And Yahoo Pipes is still the best tool to do so.

Yahoo Pipes is a versatile tool you can use to combine different content, mash it up to create new content, but here is a step by step tutorial for the most simple thing you can do with Yahoo Pipes: combine different feeds.

Step 1: Getting started with Yahoo Pipes

1. Register for a Yahoo account
2. Go to Yahoo Pipes and log in with your Yahoo account

Step 2: Create a Pipe

A “Pipe” is a workflow you create to take RSS (or other) input, mix it up, and create a new output.

On the home page of Yahoo Pipes, click on “Create a Pipe”. This will give you a worksheet to construct a pipe, using a nice graphical user interface.
In the left column, you will see a collapsible menu with the main working modules, and the main part of the screen is your worksheet:

Step 3. Create a workflow for your pipe

1. The first thing we have to do, is to tell Yahoo Pipes where to get the input RSS feeds from, the feeds we want to combine. From the left module list, drag the module “Fetch Feed” (from the “Sources” list) into your work sheet on the right:
   
2. Note that once you dragged the “Fetch Feed” module, Yahoo Pipes automatically also put a second module “Pipe Output” in your worksheet. Leave it there for the time being.
3. The “Fetch Feed” module will open up, and you can fill in the different feeds you want to combine.
   As an example, let’s put two feeds in here, one from BlogTips and one from Blogging Today. In this case, both are Feedburner feeds, but it does not matter. You can take Atom feeds, RSS 1.0 or RSS 2.0. Yahoo Pipes will process them properly.
4. After filling in the first feed URL, click the “+” icon and fill in the second feed.
   
5. At any time during the process of creating a pipe, you can click on a module, and as it gets highlighted in orange, the debugging window at the bottom of the sheet will change when you hit “refresh”.
   The debugging module basically shows the output of that module. You can click the triangle icons to see the different internal components.
   This is very useful if you create complex pipes, but for now, we will leave it aside.
6. The next step is to connect the “Fetch Feed” module to the “Pipe Output” module: Click on the blue dot (the connection dot) at the bottom of the “Fetch Feed” module, and drag it to the blue dot at the top of the “Pipe Output” module:
   
7. And in principle you are done now.

Step 4. How to combine many many many RSS feeds

While each “Fetch Feed” module will let you enter about 20 feeds, you might need to combine more feeds than that. Easy!

1. Drag another “Fetch Feed” module from the left “Sources” list into your worksheet
2. Fill in your feeds.
3. If you still have the first “Fetch Feed” module already connected to your “Pipe Output” module, disconnect it: Just hover over the connecting blue line, and scissors will appear. Click on the scissors, and the connection line will disappear.
   
4. Now you need to combine both “Fetch Feed” modules into one. Drag the “Union” module from the “Operators” module list on the left, into your worksheet.
   
5. Now connect the two “Fetch Feed” modules to the input connectors of the “Union” module (at the top) and connect the output connector of the “Union” module to the “Pipe Output” module.
   Remember, you only need to do this, if you have a lot of input feeds. Each “Fetch Feed” module will easily cater for 20 input feeds.
6. If need be, you can add more “Fetch Feed” modules and connect them all to the “Union” module. I suggest to limit to 60-80 input feeds per pipe, otherwise the pipe becomes too slow.

Step 5. Save your pipe

Congratulations, you have just constructed a pipe which combines your RSS feeds. Now save the Pipe.

1. Click on the tab “Untitled” at the top left of your screen, and give the pipe a name, and click “OK”
   
2. Now click on the “Save” button in the right corner
3. If you have many input feeds in your pipe, saving might take a while.
   There have been times where Yahoo Pipes had performance problems and saving just hung, or gave obscure error messages, but it seems those times are now past. (touch wood)

Step 6. Create an RSS output from your pipe

While you have now successfully created a Pipes workflow, you still don’t have the combined RSS output of your pipe yet, do you? Two simple steps to get there:

1. After you saved your pipe, click on “Run Pipe”
2. A new window will open, and a screen will appear showing “Running Pipe”
   
3. Dependent on how many feeds you combine, this might take 2 or take 60 seconds, before the items in your feed will appear.
   Do check the bottom of your feed to see if there were any errors. You might have entered a wrong feed, or entered a home page URL instead of a feed, a common mistake!
4. Your pipe, combining your different RSS feeds, can be used with different tools (add it to My Yahoo!, to your Google Home page, to your Google Reader, get a badge to put the output on your blog, etc…). If you just want the link to the RSS output, click on “Get as RSS”
   
5. And voila, there is the RSS output. Copy the URL of the RSS output and you can reuse it anywhere you want.
   

Step 7. Refining your Yahoo Pipes output

Now we’re done. Well… except for one thing. Well, except for two things. Well, except for three things:

1. As it stands right now, the RSS output does not have the feed items sorted by date, which is no good. We will need to sort the RSS feed, and publish it with the most recent RSS items first.
2. If you combine many different feeds, the RSS output might contain many items, and that could clog up wherever you want to reuse the feed. We will need to limit the amount of items in the Yahoo Pipes’ RSS feed.
3. And again, if you use many different RSS feeds as input, one input blog might have re-used input from the other, so we will want to take out duplicate entries.

So roll up your sleeves once more for the “final act”:

1. Disconnect the “Union” and the “Pipe Output” module
2. From the “Operators” menu on the left, drag the modules “Sort”, “Unique”, “Truncate” onto your worksheet.
3. Connect them in this order, and fill in the different fields:
   
4. In the “Sort” module, select “item.pubDate” and “descending” order
5. In the “Unique” module, select “item.title”
6. In the “Truncate” module, fill in the maximum number of items in your output field. We used 40 in our example.
7. Connect the different modules (From “Union” or “Fetch Feed” if you don’t use the “Union” module, to the “Sort”, the “Unique”, the “Truncate” and the “Output Pipe”.
8. Save the pipe, and you are done. Yahoo Pipes will automatically change your output RSS according to the modifications you saved from your worksheet.

So now you are really done. Unless if you want to…

Step 8. Do more sophisticated stuff with Yahoo Pipes. (optional)

Yahoo Pipes allows you to do many many many different things. You can take input and mash it up to create virtually new content. You can check out the pipes others have created clicking “Browse” on Pipes’ home page. People come up with the weirdest and most sophisticated ideas, using translators, automatically integrating FLICKR pictures in a pipe, etc…

However, the most commonly needed option, is to change or filter out some of the content from the input pipes. For that, the “Regex” module from the “Operators” menu is the most used workhorse.

There are few “programming syntaxes” which are as geeky as “Regex”, so beware, this is not for the “faint of heart”. But just one example: Let’s say you want to delete all images from the input feeds, and generate a “text only” output feed. And as we started this post saying you have different blogs and sites about “fundraising”, let’s change the output marking any occurrence of the word “fundraising” in bold:

1. Disconnect the “Truncate” module from the “Pipe Output” module.
2. Drag the “Regex” module from the “Operators” menu on the left, into your worksheet
3. In the “In” field, select which part of the RSS feed you want to work with. In most feeds, the actual content of every RSS item is stored in the “item.description” field, so select that field
   Some RSS feeds store their content in other fields, so you might want to use the debugger view of Pipes to find out where the content is stored.
4. To replace all images, we need to take out the “img” tag from the content. So we want to replace anything from the beginning of the img tag until the end, with “nothing”. The Regex formula for that is:(<img.*?>)
   …which means: take any string, beginning with “<img,” following by anything (the “.*?” part) and ending with “>”. We need to replace that with “nothing” so leave the “with” field empty.
   And we want to replace all the occurrences of this in the input, so we click the “g” option in the regex module.
5. We will now look for all occurences of the word “fundraising” (including those starting with a capital “F”), and put the “bold” tags around it. Pfft, easy!
6. Our regex module will then look like this:
   
7. Don’t forget to connect the “Truncate”, “Regex” and “Pipes Output” module..
8. Save your pipe, and once again, you are done.

As said, you can go as complex as you want. If you get into trouble, call on your peers for support in the Yahoo Pipes Discussion Forum. People will be ready to help you.

9. What I use Yahoo Pipes for?

I use Yahoo Pipes for many different things.

I have different pipes where I take feeds from dozens of news sites and combine them into a single RSS output, which I read using Google Reader, both on my laptop, iPad and iPhone.

I also use Pipes to channel different feeds onto several Twitter and Facebook accounts, using a series different techniques.

But I mostly use Yahoo Pipes to combine, process and mashup feeds from over 1,000 sites and blogs about aid, development, humanitarian issues and the environment. I republish summaries, as I describe in an earlier post.
Check out the end result in Humanitarian News, AidNews, AidResources, News On Green, AidBlogs and The NonProfit Blogs.
In Blogging Today I also aggregate the latest updates from “Blogs about Blogging”, and The Weird News keeps a smile on my face…

Oh, and as a closing tip: if you want to go over “How to create a Yahoo Pipe” in a video, check this out:

Have fun with Pipes!

Picture courtesy Rajtilak

Well I’ll be darned. I am never too old to learn.

Over the past two weeks, I migrated seven blogs from Tumblr to WordPress, onto my VPS (Virtual Private Server) on Hostgator. Concerned about the performance of the server, I scrupulously monitored the CPU, memory and traffic load, as I redirected the domains and as the traffic started to flow in.

There is quite a bit of traffic on those seven blogs, so I use aggressive caching on WordPress, using WP Super Cache.

WP Super Cache has the option to pre-load a set number of posts into a cache: At pre-set intervals, it reads the posts from the SQL database, and converts them to plain HTML files. This way, any visitor gets the pre-cooked HTML file. This creates quite a bit of files, but offloads the server as it does not have to look up each page on the SQLserver. Important for me, as space and bandwidth is not a problem. CPU power is.

That being said, I was astonished to see the CPU load of the server to grow up to the point where everything slowed down. The “Load Average” on the server showed an average of six to ten process waiting to be served. Monitoring the load, I could still see a lot of SQL access happening.

How was that possible? I cached every single page, so no SQL access was needed. I checked and double checked everything. Could not see what caused it these SQL queries. I had everything cached?!?!

Until I found the culprit. I had a plugin “WordPress Popular Posts”, which checked which posts were being accessed, so I could put a “Most Popular Posts” widget in my side column. Thought that would be a cool idea and installed it on all seven migrated blogs.

A cool idea it is, as I use it here on BlogTips. But that is not a cool idea apparently for a high volume site, with limited resources. My poor server was sweating like hell, as apparently all visits are being logged into the database..

I disabled the plugin, cleared the caches, rebuilt them, and voilà.. Almost on the spot the CPU load went down from six to ten processes in the waiting queue to less than two. As anything less than four queued processes per CPU (I have one CPU) is acceptable, my server is a happy camper, I am a happy camper, and my visitors are served with faster websites.

Lessons learned: blog performance is sometimes a matter of plain logical thinking. And often the solution is not in “blaming it on the server”, but looking at the blog in front of you. The “adder under the stone” might be right in front of you, with the venom in juuuust a small plug-in.

And I don’t mean to blame this “Popular Post” plugin. It does what it is supposed to do. But it does not work for high volume sites.

I feel like standing in front of a rail crossing, when the red lights just won’t go off. Is it worth driving to the next rail crossing, just a minute further down the road? Are the lights defective, or is the crossing closed for a true reason? The longer I wait, the less it will be worth driving off.

As a metaphor, that is how I feel waiting for Tumblr to get its act together after month’s of outages. I have a dozen blogs on Tumblr, many of them are aggregators, creating blogposts from imported RSS feeds. I know WordPress.org (the self-hosted version) has a cute aggregation plug-in that does the same job, so should I move to WordPress, or wait for Tumblr to solve its problems?

Each of my Tumblr blogs has thousands of posts, so migrating them won’t be a small feat. Plus I will have to install a new blog on my server, redo the theme-ing, install the plug-ins etc.. before the blog goes live. Maybe Tumblr will solve its problems tomorrow? Or the day after, or the next? Or maybe I will discover new problems with the WordPress aggregator tool that will keep me busy for days too…?

Well, ladies and gentlemen, after standing in front of the rail crossing for weeks, I decided to move one blog as a trial. The original blog (I left its remains on Tumblr) contained 20,000 blogposts ( ! ), and I did not want to lose those. So one of the main challenges to move my Tumblr blog to WordPress was to migrate all blogposts. All 20,000 of them…

Here is how I did it:

There are three steps in migrating your Tumblr blogposts to WordPress:

Step 1: Export your Tumblr posts
Step 2: Process your exported Tumblr posts (optional)
Step 3: Import the export file into WordPress

Step one. How to export your Tumblr blogposts?

Tumblr does not feature an “export function”. I found a list of possibilities, but none really suited me, until I stumbled upon  Tumblr2WordPress (by Ben Ward). And Ben saved my day.

Just run Tumblr2WordPress, enter your blog’s Tumblr subdomain (don’t use your custom domain), select if you want to export to WordPress.com (the WordPress hosted blogs) or WordPress.org (for selfhosted blogs) and … click export. The exported posts will be downloaded onto your PC, as an .XML file…
That should do it for most Tumblr blogs.

(Update March 1, 2010: Ben’s source code is still available, but the executable program is no longer available on this link. You can still run similar code from Tumblr2WP or Tumble2WordPress – With thanks to Aaron and Parneix for the updates)

If you get an error “Tumblr API Request Failed”, this means -once again- Tumblr is failing (as it does frequently in the past months), and the API request to export the posts gives an error. Try it out manually with a command like: http://yourtumblrblog.tumblr.com/api/read?start=0&num=50 – If you get an error, the only thing you can is “try again later”.
At the time of writing, it seems Tumblr is blocking API-calls during the US-day time (afternoon and evenings mostly)…

If you are a freak, like me, and have 20,000 posts in your blog, Ben’s routine might give a time-out. I had to download the PHP source code and install it on one of my servers, so I could dramatically increase the system resources. For the nerds amongst you, I put the PHP code in a subdirectory, and added a php.ini file to it, with the following parameters:

upload_max_filesize = 20M
post_max_size = 30M
memory_limit = 400M
max_execution_time = 600

… but again, 99% of you might not have 20,000 blogposts, so Ben’s hosted routine will do just fine.

Step 2: Processing your exported Tumblr posts

In normal circumstances, you can skip this step, but if you are a purist, like me, you might want to clean up the .XML file a bit to avoid some issues when importing the file in step 3.

You can edit the .XML file with a normal ASCII editor (WordPad does just fine for me, the simple Windows XP user). Each post is stored between <item>... </item> tags.

For each post, you will need to clean up two things with a simple search and replace:

One: Clean up the category tags
In some cases, the WordPress importer will create a single category for each imported post. Import 100 posts, and you will get 100 junk categories. While those are easy to clean up after importing the posts, it is better to avoid the problem than curing it.
The only thing you need to do, is to delete the two category tag lines, for each post:

<category><![CDATA[link]]></category>
<category domain="category" nicename="link"><![CDATA[link]]></category>

Two: clean up the date warnings.
Under some circumstances, you will get a date warning in the .XML file:

<wp:post_date><br />
<b>Warning</b>: date() [<a href='function.date'>function.date</a>]: It is not safe (blabla)
2010-12-24 12:00:58</wp:post_date>

Just search for that string, and replace it with the date you find for each post, for example:

<wp:post_date>2010-12-24 12:00:58</wp:post_date>

Three: Split up the file in smaller chunks
Oh, and yes, there is a third thing, before I forget: WordPress can not import file larger than 8 Mb. So if your .XML export file is larger than 8 Mb, split it into individual small files.

Beware: Each file should contain the header section, which starts with

<?xml version="1.0" encoding="UTF-8" ?>

and ends with:

<wp:category>
<wp:category_nicename>uncategorized</wp:category_nicename>
<wp:category_parent></wp:category_parent>
<wp:cat_name><![CDATA[Uncategorized]]></wp:cat_name>
</wp:category>

and each file should end with:

</channel>
</rss>

In other words: create a series of small files, which contain all posts (between <item>... </item> ) and paste the file header and end tags.

Hey, and forget all of this, if your .XML file is smaller than 8 Mb !

Step 3: Import your .XML file into WordPress

Now for the fun (and easy part): Import your .XML file with the WordPress Importer utility ( Dashboard > Tools > Import ).

The importer gives you a series of input formats. Select “WordPress”. And if you don’t have that plugin, you will have to install it first, with one click (don’t you just love WordPress? In Drupal, that would cost you four hours of work.. )…

Now you are ready to import your .XML file:

Ready for the WordPress magic?

WordPress will give you the option to define the “blog user” name you want as the author for the imported posts, chew on things for a while, and in the end, list you the names for all the posts it has imported.

And ready you are…

In my case, I converted this Tumblr blog into this WordPress blog in a day, including the import of 20,000 blogposts, the theme-ing and all other WordPress goodies involved in starting a new selfhosted blog…

Pretty neat, no?

Update Jan 18, 2010:
Important remark (as per comment from Parneix below):
Unlike WordPress to WordPress import where media (video, pictures) which are in the input blog’s media library are also imported into the new blog’s media library, images in posts exported from Tumblr and imported to WordPress will be “hotlinked”. This means they won’t be copied into the new WordPress blog’s media library, and will actually continue to link to the Tumblr media library.
As long as you keep your old Tumblr blog alive (don’t delete the Tumblr blog nor any of the posts), that should not be a problem, even though it is a bit of a drag…
The workaround is to automatically import hotlinked images to your local media library using this plugin for instance.

Thanks Parneix for the rectification!

Update March 1, 2010:
Ben’s source code is still available, but the executable program is no longer available on the link I provided in this post. You can still run similar code from Tumblr2WP or Tumble2WordPress.
- With thanks to Parneix and Aaron for the updates

Mystery, Hitchcock, PHP: Sometimes "ugly" can be "beautiful" too

Remember I once wrote about the choice of selfhosting your blog or not? No? Well I did. And if you choose to selfhost a blog, some of the issues you need to be aware of are (drum roll):

• the work involved to maintain a selfhosted blog,
• protecting your blog from hackers, and
• dealing with (g)hosting issues

Well, here is another example how a problem with self-hosted blogs can send you on a debugging trail as vicious, addictive and poisoning as a true mystery story, in the true art of a Hitchcock or Sherlock Holmes thriller:

I use permalinks in all my WordPress blogs to convert the ugly and SEO-unfriendly default WordPress URLs:

http://www.blogtips.org/?p=123

into something nice and smiling, like:

http://www.blogtips.org/how-to-evaluate-a-blog-introduction

But since I moved BlogTips from its old GoDaddy (or “SlowDaddy”) to its new Hostgator host, I noticed that the permalinks settings page returned empty.

The Mystery of the Disappearing Permalinks

This was the start of my latest online mystery adventure.

As picked up my pipe and magnifying glass, I realized: “I must to exclude all variables (as a good IT debugger should do)”. Thus, I installed a new plain vanilla WordPress test site, without any plug-ins. And sure enough, the permalinks settings page was blank.

To my mind did not come the thought “Man, That Sucks”, but “Good, now I know it had nothing to do with any weird plug-in..”. So what caused it then, The Damned Permalinks Page to Disappear?

Googling “WordPress Permalinks page blank”, I found this post on the WordPress forum, which seems to point to a PHP problem:

Had this same problem, took us forever to get our issue resolved. We did this: Rebuilt apache (using easyapache within cPanel) with version 5.2.x instead of 5.3.x

So I narrowed my search, and googled “wordpress permalinks page blank, PHP 5.3″ …to find this post on the forum:

I have the same problem here (only the pdo_sqlite.so error), apparently it is a php5.3 bug.

So maybe I should have a look what PHP version I am running on my server…

Never done that before? What? Saying you never created a script to check your PHP settings is like saying.. “I never had sex under the shower”. It is easy, and fun: Create a info.php file on the root directory of your blog containing:

<?php
phpinfo();
?>

.. and execute it as: http://yourblog.com/info.php.

You’ll get a (whooooopy!) long list of all PHP settings, including the version (at the top. No, not “on top”, “at the top”) Mine said:

Yikes, PHP 5.3.2? Well, in the previous forum post, they talked about a “pdo_sqlite.so error”, so I wondered if I could find any error on the PHP error log (significantly called “error_log”), on my server, located in the directory /var/log/httpd/
And indeed, I found the error: /usr/local/lib/php/extensions/no-debug-non-zts-20090626/pdo_sqlite.so: undefined symbol: sqlite3_libversion in Unknown

Cool. I found the problem. Now how to solve it?

Googling: “undefined symbol: sqlite3_libversion”, I found this post on the Cpanels forum. Gee, these people seem to know what they are talking about, using terms like: “EasyApache”, “build profiles” and “mismatched shared extension”. One post said:

The only solution I am aware of at the present time is to either disable the PDO SQLite3 extension, by editing the system PHP configuration file “php.ini” (at “/usr/local/lib/php.ini”), or downgrade to PHP 5.2

I checked with my hosting support, and they recommended downgrading to PHP 5.2… Now, Watson, downgrading software is never as easy as upgrading. I did not want to introduce new unknown problems, while solving known problems. Like digging a new hole to fill an old hole, right?

So I had a look at the system php.ini file. On my server, it is located at: /usr/local/lib/php.ini

At the bottom, I found something like this:

extension=pdo_sqlite.so
extension=sqlite.so

I commented it out:

;;extension=pdo_sqlite.so
;;extension=sqlite.so

And saved the file. Reloaded the permalink settings page and voila:

It worked…. The end of the Sherlock Holmes Permalinks murder story: If your permalinks page returns blank, and you use PHP 5.3.2, just comment out the “sqlite.so” settings in your system php.ini file. Mystery solved.

- ….?
- What?
- ….
- “Sometime ugly can be beautiful too”?
- …
- Oh, right, what I meant with the caption from the title page “Sometime ugly can be beautiful too”?
- …!
- Right, sorry, forgot about that.

What I meant was: I had a server problem and solved that problem, after a long search. But by solving that problem, I also resolved another problem: For ages the thumbnails did not appear anymore on my blog’s archive page. Never been able to find that one. By solving the permalinks problem, it seemed the thumbnails re-appeared.

And that is the beauty of problems with self-hosting: by solving one problem, you might incidentally also solve another.

- …!
- You are welcome!

After the massive hacks injecting malware into shared hosted sites from several providers back in April and May, it seems they are back at work.

Many sites hosted by GoDaddy are being hacked at the moment I am writing this post. Two of mine were affected an hour ago.

Update: Hit again this morning (Sept 21).
Here is a record of the September virus spree, as I saw on my sites (all CET – Central European Time):

• Friday Sept 17, 2010 – 23:30 CET
• Tuesday Sept 21, 2010 – 08:30 CET

The scenario is the same as a few months ago: Malware is injected into the .php files on the hosted sites, and the visitors of a site are getting redirected to a third website which injects a virus into the visitors’ computer.

At this moment, it seems also other hosting providers were/are attacked, so monitor your blogs. Check if it is infected regularly during the next days. If you get infected, run the script from this post, and your site will be cured in a minute.
You can also use the same script to verify if your site was infected. If you get the message

0 Infected Files in ./

… then your site is clean. If you get a list of infected files, click “Fix Files”, and within a few seconds, your site will be cleaned up. If you use a cache-plugin, don’t forget to clear your cache!

Note that if your site was infected, and you loaded the site yourself, your computer might be infected too. Many antivirus (MacAfee, Norton,..) programmes will NOT catch the infection. Download the free malware scanner from MalWareBytes to verify and cure the infection.

Best of luck to you.

Picture courtesy thenewnewinternet