I have blogged before how I don’t like Google’s growing monopoly on the web, and how they increasingly block non-Chrome browsers when you use their own web applications.

It looks like Google is now moving into more unsound web practices, using their web crawling power and abilities to unfair, unethical and illegal purposes.

Google pays bloggers for Chrome links

Earlier this month, news broke that Google paid bloggers to write about Chrome, their own browser. Embedding paid advertisements, and, in some cases, hiding it as genuine blog content, has always been something I found highly unethical.
OK, in some cases during Google’s campaign, the paid blogs were clearly marked as such, but the post embedded links to Chrome’s download site. “Paid” links is something Google themselves have always battled against, saying it cheats search engines. Now they are doing it themselves, to market their own products.

Verdict: unethical and unfair competition.

Google crawls for data for their own profit

Today, I read this interesting and in-depth story where Google crawled data from Mocality, a small Kenyan company publishing an online Kenya business directory.

In their article, the Mocality folks detail how Google crawls their business directory, with the sole purpose to contact the listed companies, convincing them to convert to a Google business directory. The article features recording of telephone calls from Google employees to the listed companies, which also reveals other unclear and illegal business practices.
Even worse, after the story broke out, Google willingly switched the crawler’s IP address, and continued their malpractice in crawling Mocality’s data. This shows a lack of ignorance, and clear mal-intent.

Verdict: unethical, unfair competition, commercial spying.

My own thing with Google

Web crawlers, scanning your web content, are to obey rules web admins set in “robots.txt”. This file defines what crawlers can scan, what they should not, and at what rate they can scan.
I always had a problem that for the Google crawler, some settings, like the Google’s crawler rate, can only be set in the Google Web Master tools, ignoring the settings in “robots.txt”. Even worse, every three months, Google resets their crawler rates. If you want to spare your server from excessive Google crawling, you have to manually reset the crawler rates for each of your site, again. Every three months.

Today, I discovered that Google’s crawlers also ignore other, more basic rules:

On “Humanitarian News“, one of my sites, the “robots.txt” explicitly disallows crawlers to access the search facility:

Disallow: /search/
Disallow: /opensearch/

The reason is that with the rate Google crawls, and the amount of search combinations possible, my server performance goes down each time Google spams my site with excessive crawls. Even worse, this “opensearch” string calls for SOLR search, which invokes a JAVA script offering advanced RSS features for “live” users. Needless to say this sucks up server resources, certainly if you shoot off opensearch requests at a rate Google’s crawler does. This is the reason why I block crawlers from accessing the search, in the first place.

This morning, while verifying my log, I found a string of searches which caught my attention:

search 13 Jan 2012 – 12:02 somalia (Search). Anonymous results
search 13 Jan 2012 – 12:02 somalia (Search). Anonymous results
search 13 Jan 2012 – 12:02 somalia (Search). Anonymous results
search 13 Jan 2012 – 12:02 somalia (Search). Anonymous results
search 13 Jan 2012 – 12:02 somalia (Search). Anonymous results
etc etc etc..

Looking into it further, the search strings were of the format:

http://humanitariannews.org/search/apachesolr_search/somalia?page=1442

And who generated those searches? They all come from IP 66.249.72.105
Which is (source)…:

OrgName: Google Inc.
OrgId: GOGL
Address: 1600 Amphitheatre Parkway
City: Mountain View
StateProv: CA
PostalCode: 94043
Country: US

So…

1. My robots.txt clearly blocks /search access.
2. Google ignores that rule, and as such ignores a basic rule to protect certain web content from crawling.
3. Going over my log, I find repetitive Google crawlers of the search
4. On top of that, there is not much relevant data to be found in the search which can not be found in a normal crawl of my site. And certainly not on (as in the example) on “page 1,442″ of a search (see above example).

The problem is that other than blocking their crawler’s IP address, you can’t do much about it. And if you block their crawler’s IP address, as a web manager, you can just as well throw yourself under a train: The Google search results for your site, would disappear. – do I smell a monopoly here?-

Verdict: monopoly, unethical business practice, bad example for anyone else on the web.

Is it time for an #OccupyGoogle ? I think it is.
PS: If anyone has bright ideas how to tweak the robots.txt to disallow crawler access, I’d like to hear about it.
Angry face picture courtesy SearchEnginePeople

If you have a selfhosted WordPress blog (WordPress.org), take urgent measures to secure your site from a recently discovered vulnerability.

Many WordPress themes and plug-ins use a script called “timthumb” (timthumb.php). This is the most common code used to create thumbnails from pictures.

End July, a vulnerability surfaced showing external users could dump malicious code onto your site. Typically, a short piece of .php code is uploaded via a timthumb backdoor. This hacking code then creates a wider backdoor to gain pretty much full access to your site.

It looks like the hackers were on holiday too, and are only gearing up their activity right now. Many sites were hacked in the last couple of days. As many sites use timthumb.php, we can foresee a major hacking spree in the next weeks.

So it is high time to secure your selfhosted WordPress site now.

How to check if you have been hacked via timthumb.php?

There is not one specific signature to this hack, contrary to the shared hosting hack last year, but here are some common things that seem to happen:

• First… Check if you actually use “timthumb.php” on your site. It does not come with the default WordPress installation, so check if any of your uploaded themes contain the file “timthumb.php”.
  Do a site wide search (with SSH or SFTP). Some popular plug-ins that use timthumb.php are “WordPress Popular Posts” and “WP Mobile Detector”.
  Many themes use timthumb.php, or a variation of it. E.g. the widely used “Thesis” theme uses it as “thumb.php”.
• If you find the timthumb.php in your plugin or themes directory, you’d better give your site a thorough check, so check further
• The hackers often upload .php files in the timthumb upload directory “/cache” (a subdirectory from the one where the timthumb script is stored). You should check that directory, and delete any non-picture files (.html .php,…)
• Often hackers upload .php files to several other subdirectories within your WordPress installation. I have seen them in the “/upload” “/supercache” directories (and their subdirectories) as well as in the directories for plugins and themes. Delete them.
• Recently, the hackers got bolder and entire subdirectories were uploaded. First a .zip file would be uploaded, it would be unzipped and an entire sub-site was installed in one of the WordPress directories. I have seen zipfiles called halifaxsecurity.zip, hal.zip, studentloanupdate.zip, student.zip. Malicious subdirectories I detected on other sites, were called /halifaxsecurity, /hal and /studentloanupdate. Delete those, if you find them.
• People also report direct hacks in .php files and style sheets, adding malicious code (similar to the last year’s hacks).
• Check your .htaccess files
• ..

Check also Sucuri’s blog for more hack signatures and scripts, and Mark Maunder’s blogpost for a full description of the timthumb vulnerability.
List of themes and plugins (non-exhaustive, though) using timthumb.php, you can find on Big Webmaster and Sucuri’s blog.

Deleting those malicious files is not sufficient, as it still leaves the backdoor open for future hacks, so you need to secure your timthumb.php code NOW ! Read on:

How to secure timthumb.php against hacks?

1. Locate all instances of timthumb.php (or any renames of it) on your site.
2. Download the newest timthumb.php code (Check also the plug-in’s home page)
3. Replace the old timthumb.php with your downloaded code.
4. While the new code is already secure, I strongly suggest to limit the access from external sites.
   Replace the line:
   define ('ALLOW_EXTERNAL', TRUE);
   with:
   define ('ALLOW_EXTERNAL', FALSE);

Good luck!

Picture courtesy TGDaily
With thanks to Michael Marus

Yahoo Pipes is a powerful and free tool, which allows you to combine and manipulate RSS feeds. I use it extensively for many of my news aggregation blogs.

In the past, Yahoo Pipes has been suffering from regular downtime, and just over one year ago, Yahoo decided it was time to overhaul the Pipes’ core engine entirely: “Pipes Version 2″, or V2 was born.

Well, it was a difficult birth it seems.

Since June last year, Pipes V2 was available in beta for its end-users. The initial testing revealed so many bugs that hardly anyone bothered to report problems anymore on the V2-discussion forum.

A couple of months ago, it seems Yahoo put new steam in the V2-migration. Several software engineers started to actively engage on the support forum. But still, bugs were plentiful. Many answers were the same: “Known issue. Working on it”. I tried my share, but did not succeed to migrate any of my pipes successfully to V2. So hardly anyone migrated. And if they did, most switched back to the V1-engine immediately.

That was until end June when Yahoo announced the “switch back” would no longer be available. And that all pipes would be converted to V2 as of early August.

Panic!

Everyone is now scrambling to convert their pipes to V2. Luckily, Yahoo beefed up their presence on the discussion forum, and a number of active and user-oriented software engineers were braving the wave of complaints.

It was not until last week when a real stable V2-version was released (that was one day before a 20 hour long Pipes melt-down, but that is just a detail).

A few nights ago, I tested and migrated all my 30-odd pipes one by one, and found that with one or two exceptions, all pipes migrated just fine. The few errors I found, have been fixed by the Pipes team in the mean time, or I was able to work around them.

So, my advice: if you are using Yahoo Pipes, better test and convert those Pipes NOW, before you are left with no choice. Deadline: August 1, remember?

PS: If you are interested in this powerful RSS manipulation tool, check out my simple Yahoo Pipes Tutorial.

Picture courtesy Photo Dictionary

GoDaddy versus DreamHost: and the winner is...

After my debacle with GoDaddy, I moved most of my high volume blogs to a HostGator Virtual Private Server. I still had some test blogs on GoDaddy’s shared servers, but as the speed and uptime of GoDaddy went from bad to worse, I recently moved several of these blogs to a DreamHost shared server.

Here are my impressions of DreamHost versus GoDaddy shared servers:

DreamHost versus GoDaddy shared hosting speed

After two years of fighting GoDaddy’s slow speed (I mean REAL slow), it was a relief to see how fast the DreamHost shared hosting really was.

Take this test site, which I recently made for a client. While this WordPress blog is not cached (it is used for an interactive re-theming of an existing site), it is about as fast as one can get. It outperforms any heavily cached site I have on GoDaddy.

Kudos, DreamHost! – I am impressed…

DreamHost versus GoDaddy shared hosting prices

As I have several blogs and domains running on the same shared hosting account, I had a GoDaddy Deluxe hosting, which is comparable to the standard DreamHost Web Hosting features (see also the next chapter).
GoDaddy advertises Deluxe Hosting at $7.99 per month, DreamHost goes at $8.95 per month. What GoDaddy does not tell you, though, is that when you renew hosting, the price goes up drastically. GoDaddy Deluxe Hosting renewal goes at $150.96 per year. DreamHost Webhosting goes at US$119.40 per year…

By the way, GoDaddy Economy Hosting, which can only host a SINGLE domain and offers FAR fewer features/storage/databases than their Deluxe Hosting, costs $101.80/year for a renewal.

I do also have an issue with GoDaddy’s mal-advertising: there is nowhere on their website (I checked with their sales department) where you can find the cost for renewals. Nor do they mention that the cost for hosting renewal is different from the initial fee. Their sales person told me bluntly: “Yeah, but you need to understand, this is how we attract new customers.” To which I answered: “…And how you lose existing customers.”
Nothing short of cheating, if you ask me.

So DreamHost is far cheaper than GoDaddy

GoDaddy versus DreamHost shared hosting capacity features

GoDaddy Deluxe Hosting offers:

• 150 GB storage
• unlimited websites (domain hosting)
• unlimited bandwidth
• 500 Email Accounts
• 25 MySQL Databases (of max 1 GB each)
• 1 GB max database size to be included in backups

DreamHost Web Hosting offers:

• unlimited storage
• unlimited websites (domain hosting)
• unlimited bandwidth
• unlimited Email Accounts
• unlimited MySQL Databases (with unlimited storage each)
• 4 GB max database size to be included in backups

GoDaddy’s claim for “unlimited” websites hosting is bullocks, as they only support up to 25 MySQL databases. This means you can only host up to 25 blogs (assuming all blogs use MySQL databases) on one Deluxe Hosting. That to me, smell like false advertising, once more!

So once again, DreamHost beats GoDaddy.

GoDaddy versus DreamHost support

GoDaddy only offers Email and telephone support. I wrote before about my issues with GoDaddy support: Email support is almost a no-go. Most of the time I get a pre-cooked boiler plate Email answer, which looks like they did not really read my question or issue. Each exchange you have with them, you get another support person answering, who clearly does not go through the history of that particular support ticket.

GoDaddy’s telephone support is slightly better than their Email support, but it really depends on the person you get on the line. I would say 90% of the time, the person “had to check” with a 2nd line of support. And most of the time, the answer was “Yep, apparently there is a problem on the shared server your blog runs on, but they are working on it”. They have no possibility of warning you when the issue is resolved. Some issues were never solved.

DreamHost has Email and telephone support, but also feature an online chat support. This is great if you have a quick question, and saved me a lot of time at many occasions. I can’t believe GoDaddy still does not offer this service.

For all my interactions with DreamHost, I’d rate their level of support much higher than GoDaddy.

GoDaddy versus DreamHost shared hosting features

I was very surprised to find the DreamHost Web Hosting features to be more similar to a VPS (Virtual Private Server) hosting, than a shared hosting:

• On DreamHost, you can define an unlimited number of user accounts, each with their own log-in, disk space, domain, etc…: This allows you to subdivide your account for each of the blogs you host. It is also an additional security feature.
  None of that on GoDaddy: one account, one login on Deluxe Hosting
• On DreamHost you have a full featured SSH (terminal emulation) access: Most Linux commands can be used on DreamHost. Even the “TOP” command will show you the processes that run on your account, once again resembling the features of VPS hosting.
  On GoDaddy, many Linux commands are blocked.
• DreamHost has a more integrated control panel, giving you access to all account, domain, email and DNS functions.
  In GoDaddy, many of these functions are split off in different submenus and sub-websites are often slow, and difficult to use.
• DreamHost allows 50 Gb of backup storage, which is configured as a separate user. This can be used to store an off-site backup of your laptop, or backups from another server.
  GoDaddy does not allow their Deluxe Hosting server to be used as backup server.
• Both DreamHost and GoDaddy provide “One-Click” installs for the most popular public domain CMS (Content Management Systems) and blogs. I should say that I did not like the WordPress One-Click install of DreamHost: it was not a standard installation. It also includes 30-odd themes, which is just an overhead if you want to keep your site lean and mean, and easy to backup
• Talking about backups: GoDaddy has a MySQL backup function (even though it is hidden in some obscure sub-sub-sub menu’s icon). DreamHost has a one-click backup function, for your “whole account in one”. You can only take one backup per month. I’d like to see a more robust and easier backup feature, though: an automated daily/weekly/monthly backup would be nice…
• There are several additional features I like on DreamHost Webhosting: They support Email discussion and Email broadcast lists, one-click site imports from another host’s CPanel, and software development version control, to name a few…

GoDaddy versus DreamHost shared hosting, which is best?

DreamHost, without any doubt. Better price, far better performance, better support and more features.

And no… I am not sponsored by DreamHost!

Changing the Google crawl rate will have an immediate effect

As I explained in my tutorial “How to analyse your visitors’ traffic figures”, the search engine traffic is important to have people discover your blog. So we have to do everything we can to make search engines discover our blog’s content, through SEO or Search Engine Optimization.

While there is much written about SEO, any honest expert will confess that there are many mysteries surrounding this subject. That’s why in BlogTips, I only share those techniques I experienced to work: In previous posts, I have proven how changing from one blog platform to another, and generating a site map can dramatically increase the amount of posts a crawler “discovers”. And as crawlers like fast sites, I showed the difference of crawlers’ access speed when switching from shared to dedicated servers.

But… these techniques still let crawlers decide how deep they want to dig into your site. If you have a website, or a blog, with a lot of content, much of it might remain undiscovered. And if crawlers don’t check for your content, people won’t be able to find it through the search engines.

By default, Google determines automatically their crawl rate for your site. The “crawl rate” defines many how pages from your site, or posts from your blog, they’ll check on “each visit”. If the crawl rate is too low, a lot of content might go undiscovered and un-indexed.

So how can we force Google crawler (or gently ask it) to crawl more frequently?

1. How many pages does Google crawl on your blog?

As explained in Five things to do after creating a new blog, it is good practice to submit your site to Google Webmasters.
In the Webmasters “Diagnostics Menu”, check the “Crawl stats”, to find out how many pages Google crawls per day. If, according to the amount of blogposts you have, this is too low, you will need to increase the crawl rate.

2. Using Google Webmasters tools to increase the crawl rate

For any site, even if you don’t selfhost your blog, you can adjust the Google crawl rate using the same Google Webmasters tool: Go to “Site Configuration” – “Settings” and click on “Set custom crawl rate”:

Adjust the slider to increase the crawl rate, save the setting and you are done.
This is not a permanent setting. It is only valid for 90 days. Also if you have a robot.txt on your selfhosted blog’s root (see point 3), than the most restrictive setting will apply.

Beware, this ONLY changes the Google crawlrate, and not the crawlrate for any other search engine. Another thing to keep in mind: adjusting the Google crawlrate will only remain valid for three months. You will have to re-adjust it after that period.

3. Using robots.txt to increase the crawl rate

This solution is only valid if you selfhost your blog, running it on your own server.
“robots.txt” is a simple text file with “instructions” for crawlers. It can restrict the access (for all crawlers or only for specific crawlers) to parts of your website, to protect non-public parts, scripts, etc.. But it can also define the rate in which your site is crawled. SearchEngineLand has a simple tutorial if you want to learn more about robots.txt

In short, to make a robots.txt, to change the crawl rate (say to 60 seconds), for all crawlers, something like this is sufficient: (“#” lines are comments)

#
# Change the crawlrate for any crawler
#
User-agent: *
Crawl-delay: 60

Save it in a text file (“robots.txt” – all lower case) and put it on the root directory of your blog, and you’re done!

Attention! All crawlers accept the robot.txt “crawl-delay” setting, except Google. The Google crawl-delay can only be adjusted using Google’s webmaster tools (as described in the previous paragraph).

And does it work?

You bet! AidNews, one of me humanitarian news aggregators, has over 100,000 posts. While I have a lot of traffic on this site, the amount of visitors coming in via Google was way too low. I changed the crawler delay with a robots.txt from 300 to 60, and see how, the same day, the crawler responded on the graph at the top of this post.
Now, the only thing I can do, is wait until the search engine traffic picks up.

Happy crawling!

Alright! You recognized your organisation can benefit from a blog and you know what you will blog about. But… which “software” should you use to blog? From the many different blogplatforms on the market, which suits your needs?

In this tutorial, I explain the questions you need to ask yourself, the basic choices you will have to make and will give you practical tips to choose the blogging tool which is the best suited for your needs:

1. Why is it important to choose the right blog platform? What are the different blog platforms on the market? If you make the wrong choice, can you still switch?
2. Should you selfhost your blog or not? Do you want to use the free blog blogplatforms that host the blog for you, or do you want to run the blog on your own server? Is this linked to having your own domain? What are the advantages and pitfalls of this choice?
3. What functionality do the different blog platforms offer? Looking deeper at the most popular blog software and blogging platforms, are there special niches they are made for? What functionality do you want?
4. Which blog platform is the easiest to use and offers the best support? Or: what is the difference between heaven and hell?
5. Which blog platforms are the most flexible to configure your layout, design and navigation? Appeal, graphic layout and usability are sometimes hampered by limitations imposed by the blogging software. Which blogging software is the most “feature rich”?
6. Which blog platform is the easiest to customize? If a blog software does not offer you what you want and how you want it, what can you customize? Which is the most flexible blog software?
7. Summarizing: So which blogging software is your final choice? A summary of the pro’s and con’s of the most popular blog packages.

So… WordPress, Typepad, Blogger or Tumblr, what is your choice?

Writing this series, I got significant help from Dave Barnhart, who filled in the blanks on Typepad and Movable Type.
Dave is a social media strategy consultant, founder of Business Blogging Pros, and a gourmet chef. He and his firm have been helping companies use social media since 2005.
He blogs at Business Blogging Pros and Fumbling Foodie. Check out some of the blogs he has created.

A serious blog is geared towards its audience. As a serious blogger, it is important you understand your audience, your readers.
Simple and free tools like Google Analytics, will give you heaps of figures, but how do you make sense out of all that? What do these figures mean and what can I do with this?

In this tutorial I use a practical case study to help you analyse your traffic and answer some basic questions:

• Analysing the traffic quantity: Where does my visitors’ traffic come from? How much traffic do I get from search engines and social bookmarking sites? Should I do efforts to increase that traffic? Why?
• Analysing the traffic quality: The difference between new visitors and returning visitors? Why should I turn an occasional visitor into a returning visitor? How much time do visitors spend on my blog, and why is that important?
• Concluding – How to get more quality traffic to my blog: What is the most effective way to find new and returning visitors for my blog? Should I spend more time on social bookmarking, on search engine optimization or on discussion forums to get new visitors?

Picture courtesy Walid Hassanein

"Pending upgrades"... have the effect of a red cloth on a bull.

Recently WordPress 3.1 came out. While the (major) upgrade from WordPress 2.9 to 3.0 was in general seen as a success, 3.1 seems to be different.

There are a number of known problems with the WordPress 3.1 upgrade. Most of them have to do with incompatibilities with certain plugins and themes. There are also issues with custom post types and with plugins using jQuery functions (picture sliders for instance).

I upgraded two of my fifteen WordPress sites (including BlogTips) successfully, but got stuck with my third site. After upgrading, my admin screens would work well, but the site itself returned a blank screen.

Panic.

That particular WordPress site used about ten different plugins, which I disabled one by one, but in vain. I started to backup the upgraded site and to retrieve a backup copy of my old site, looking forward for a loooong day of downgrading (upgrading is a flip of a switch, downgrading is a pain).

In the process of doing so, I remembered one of the golden rules of debugging: “switch back to the default Twenty-Ten theme”! When I checked, it seemed that the upgrade had already switched me to the Twenty-Ten theme. I re-enabled my normal theme, and voila, the site was back up.

I re-enabled all plugins one by one, restored the cache files (which had all been deleted when I disabled Supercache), and the site was back up.

I am yet to analyse exactly what went wrong, but I suspect a conflict between WordPress 3.1 and one of the plugins. Which? That is the one million dollar question.

My advise is to wait it out for a few more weeks. Let the plug-in developers debug 3.1 incompatibilities first.

Two tips on upgrading (be it the WordPress core, themes or plugins):

1. If you have an important site (which site is not important, hey?), then keep a mirror test site, which is a working copy of your production site. Test any upgrade first on your mirror test site. If the upgrade works, and continues to work after a couple of days, then consider upgrading your production site.
2. Before you upgrade, make sure you have a backup of both your files, and your SQL database. If all goes haywire, downgrading will only be possible using your backups. otherwise -as they say in French- you’ll be stuffed.

Wishing you luck. I am waiting it out.

Update April 19, 2011:
WordPress released version 3.1.1 which seems to cure most problems of 3.1… I have upgraded most of my sites to 3.1.1, and everything seems stable now. My advise now: go for it!

Moving blogs: labour intensive, but fun!

I had seven blogs on Tumblr which aggregate news. Using a technique I described earlier, they take RSS feeds from over 1,000 carefully selected websites and blogs, filter them, clean them up, and feed them into the different Tumblr blogs. I used the unique feature built into Tumblr to convert RSS feeds into posts. All automatically. Pretty neat. Until it stopped working…

Two months ago Tumblr’s autoimport feature started to hiccup. Tumblr support said “We are aware and working on it”, but could not give me any estimate when it would be fixed.

A month later, still nothing. So what to do? I rely on this Tumblr feature, for my blogs. In two years, those blogs collected 350,000 news articles. Quite a resource library which I did not want to give up.

So I decided to use the Christmas holidays to migrate these blogs from Tumblr onto WordPress, on my HostGator VPS server. It was an interesting process, involving many different techniques and debugging efforts:

1. How to export a Tumblr blog into WordPress

Ben Ward's Tumblr2WordPress utility

I pretty much described the process and the technique to export a Tumblr blog in an earlier post: Using Tumblr2WordPress, a neat PHP program by Ben Ward. It uses the Tumblr API to export blogposts, and to create an .XML file, which I could import into WordPress. An API that Tumblr disables every US afternoon and evening, by the way.

While Ben’s program works well to export smaller Tumblr blogs, I had biiiiig blogs, so I had to adapt the PHP code. As the code is public domain and available on Github, I downloaded it and installed it on my server. I changed the PHP parameters to allocate a massive chunk of memory, and allow the export routine to run longer than a standard PHP program. Tip: change the parameters only for that routine, not for your whole server!

Update March 1, 2010:
Ben’s source code is still available, but the executable program is no longer available on the link I provided. You can still run Tumble to WordPress routines based on the same engine from Tumblr2WP or Tumble2WordPress – With thanks to Parneix and Aaron for the updates)

I also patched Ben’s original code to work around a smaller problem I discovered on “published dates” and “categories”. Pretty easy, even for a PHP novice like me, as the code is well documented.

As WordPress can only import .XML files smaller than 8Mb, I split up the first exported blog manually into smaller chunks. Took me two hours for the smallest of the seven blogs. I decided once again to delve into the code, and wrote a small patch that allowed me to export 5,000 blog posts at a time. Each export file now was smaller than 8 Mbyte.

Cool. Exported all blogs, and there I sat on 70 files, about half a gigabyte worth of .XML files.

2. Importing 350,000 posts into seven new WordPress blogs

The WordPress Import routine rocks!

For the seven blogs, I created seven new accounts on my HostGator VPS server. Some of my Tumblr blogs were using custom domains, so I changed the DNS, pointing to my HostGator VPS server, and created seven new WordPress blogs. While I was at it, I registered two new domains. ChangeThru.Info became AidResources.org and Youandusand.me became NewsOnGreen.org. Wanted to do that a long time ago, so now was the right time. I kept the old domains live on Tumblr, so I did not lose any traffic.

Installing a new WordPress blog was easy to do with the “Fantastico” program in the server’s Cpanel. I choose a neat and simple magazine template and added the usual plugins I always use for caching, automated blog backup, etc…

Then, one by one, I imported the 500 Mbyte of export .XML files. Worked flawlessly, but took about two days. Not a single error, not a single problem. I should say: WordPress impressed me once more.

Done. Well at least with importing the old posts. How to feed in new posts using my myriad of RSS feeds?

3. Implementing an “RSS to blogpost” routine in WordPress

FeedWordPress, you rock!

FeedWordPress made my day. This neat plugin imports RSS feeds and converts them into WordPress blogposts. And it does so very well. The plugin is well designed, easy to use, and has a lot of options to customize the import process.
It also has add-ons that allow you limit the size of the imported post, add text in the title or in the body of the imports. Realllllly neat!

I configured the different feeds I process via Yahoo Pipes, and ran a CRON job to import the blog posts. About two hours work per blog, and the whole cake went into the oven and started cooking: FeedWordPress neatly imported the posts.

You would think I was done. The real work had not even started.

4. The need for speed

From the beginning until the end, including the customization of the template etc.. the export from Tumblr and import into WordPress took me about a week. Fine-tuning the blogs and the server took another two weeks.

As “Good is Fine, Perfect is Best”, I saw some formatting deficiencies I could not live with and needed modifications to the feeds and template. And even worse, much much much worse, my poor server went through its knees with the extra load.. The new blogs demanded so much CPU time from Apache and the MySQL server that everything slowed down to a snail’s pace.

Time to get geeky!

5. Tuning the XML sitemaps

XML Sitemap: Tune it!

It only takes one dumb sysadmin to make the fastest server to go slow, I realized while monitored the CPU load with the Linux “TOP” command. I saw the “load average” to peak way above “10″, meaning there were at least 10 processes queueing up for CPU time.
So I looked at several WordPress plugins that might cause the problem.

The first problem I saw was the XML sitemap generator. There was one option, which I had overlooked: “Rebuild sitemap if you change the content of your blog”. Might be fine on smaller blogs, but the automatic feedimporters were putting up new blogposts at a rate of 100 per hour. So the server was pretty much doing nothing else but generating sitemaps.

I disabled the feature, and scheduled a CRON job to regenerate a sitemap once a day, at night-time. The server load went down significantly.

Oh, by the way, if you have a huge blog, limit the number of posts to include in the sitemap to 10,000 , Google’s maximum limit for sitemaps!

6. Tuning WP Supercache

WP SuperCache: Tune it!

As I mentioned before: any blogger using a selfhosted blog without a cache, should appear before court for “blog neglect”. So I use caching extensively, with WP Supercache, my preferred plugin.

But it only takes a stupid blog administrator to make even the best plugin not to work properly. Supercache needed tuning:

• Unchecked the option “Clear all cache files when a post or page is published” (I published 100 posts per hour, so the cache was always invalidated)
• Pre-loaded the last 10% of blogposts, but put “Refresh preloaded cache” to “0″ (as once a post is imported from its RSS feed, I don’t update it anymore, so it can remain in cache forever). This means I only had to pre-load a massive amount of blogposts once, and it was done.
• For the same reason, I put the “expiry time” to “0″, as once cached after a preload cycle, I want the page to remain in cache. It generates a LOT of cached files, but I have plenty of disk space on my server.
  If you put “expiry time” to a value > 30 minutes, garbage collection is done every 10 minutes, which generates a lot of load on your server.
• As now I had caches with an eternal life time, I needed to ensure the homepage, feeds, archives were NOT cached, otherwise visitors never got an updated overview of the latest posts.
  As I discovered that AFTER I preloaded the posts, and had put the caching to “eternity”, I had to manually delete the cached files for the home page, searches and the running month’s archives.
• To further reduce the load on the PHP server, I choose the option to use “mod_rewrite to serve cache files”.
• And by the way, if you don’t cache the home page, the “Cache Tester” will give an error – as it tests caching on… the home page. So ignore that error, and just look at the source of any random page, to see if, at the bottom of the source, you have a date/time for the cache generation, which is in the past.

7. Trashing “Most Popular Posts”

As describe in this post, one plugin meant to show “the most read posts”, also logged every single access to the SQL database, and effectively slowed down my server. Had to trash it.

8. Tuning FeedWordPress

I spent quite a bit of time to tune FeedWordPress, to balance how often feeds were to be imported with the success rate of each import cycle. I combine 1,000+ feeds into about 20 Yahoo Pipes feeds. These are large and complex feeds, which take a lot of time to fetch. Many times the import of a feed would time out.

At first I worked around that problem, by refreshing all feed imports every 10 minutes. But once again, that put a lot of pressure on the server. As you can deal with any problem either by working around it, or by addressing the cause of it, it was time to look for the source of the problem. In the process of doing so, distinguish well between what is “a cause” and what is “a symptom”. Often we try to solve the latter, while we should address the former. Think about that. That is deeeeep!

So the symptom I saw was the feeds timing out. As I know the Yahoo Pipes’ feeds often take very long to refresh, even interactively, I had to patch FeedWordPress with a timeout of 60 seconds to deal with the Yahoo Pipes’ lack of speed. Cool. But that caused a dreaded SQL error “My SQL server has gone away” to appear more frequently in my CRON log files. Beh.

To make a long story short, the solution was to also change the PHP parameter “wait_timeout” from the default of “30″ seconds to “240″. Changed that in /etc/my.cnf and restarted SQL server. Problem solved.

As this solved the timeout when reading RSS feeds, I could also decrease the frequency of the FeedWordPress CRON jobs. And that once again made my server very happy. I like happy servers…!

9. Server tuning

Tuning a server: not for the faint of heart...

Depending on what exactly you do on your blogs, for large and heavy traffic blogs like the seven I had just migrated, the SQL server might need tuning. This is not for the faint-of-heart, and requires patience and caution. With one wrong setting, you can cause more damage than good.

The first indication that my SQL parameters might need tuning, was simply the fact that SQL took up so much CPU time on my server. phpMyAdmin, a routine available to about every selfhosted server, has a neat feature called “Status”, which gave me an overview of the parameters which might need changing. But I was not sure. So I installed mySQLTuner: using SSH, I logged into my server’s root. With just three commands, I got a better overview of the parameters I needed in three commands:

wget mysqltuner.pl
chmod 775 mysqltuner.pl
perl mysqltuner.pl 

I changed one parameter at the time, and waited for 12-24 hours to see its effect.

It seems the two most important parameters to tune are “key_buffer_size” and “table_cache”. It was advised to tune these first before touching the others. Which I did. “key_buffer_size” was ok on its default value of 48M, but “table_cache” needed 4,096 instead of the default 1,024.

The rest of the parameters I changed over time:

tmp_table_size: from 32M to 64M
max_heap_table_size: from 32M to 64M
sort_buffer_size: from 1M to 4M

While I am still observing the server and tuning bits and pieces, it looked by now that, ladies and gentlemen, we have a happy server, purring  like a happy cat. Sure enough I still get peak loads, with “load average” of 3-4, most of the time it stays at “1″ or below. And that is good. I like happy servers.

10. Template tuning

For the seven blogs, I use a neat little magazine template, called Magazine-Basic by Bavotasan. It comes for free, and has some basic functions that I like.

I customized the CSS and some of the functions. I sinned heavily by patching the original template, rather than using a child theme, but that is just because I made so many changes. On top of that, for my main blogs, “speed” is important, I did not want every page refresh to read several CSS files. So patching, it was. I will live with the fact that I can not upgrade the template automatically later on, but hey, I learned that upgrading templates is always a pain, and often causes more problems than it is worth. So I avoid theme upgrades like the pest. My opinion, punto.

One of the main challenges I faced was that the template, by default, puts the first image if finds in the post as a thumbnail on the home and archive pages. And often that image was junk (a “Retweet” button, or a Feedburner “Email this” image).. So I had to tweak all my Yahoo Pipes feeds to delete those images. Took me a week. Some feeds were too complex to delete all images, so for some of the blogs, I just disabled the thumbnails in the template. I mean, I got to sleep too, so can’t keep on tuning all feeds to find all kinds of combinations of dummy images..

Maybe that will be work for next Xmas!

11. Installing the mobile theme

And to finish it all in beauty, I installed a plugin to enable a mobile theme to be displayed when visitors access my blog using a mobile phone.

12. And here is the end result of about four weeks of work:

From Tumblr to WordPress. With Love

Check them out on your mobile. Browse through the posts to see if you like the download speed (remember the homepage and archives are not cached, and thus a bit slower!):  AidNews, AidResources, News On Green, AidBlogs, The NonProfit Blogs, The Weird Bit and Blogging Today.
You like?

Cartoon courtesy Mark Lowe

Many bloggers use RSS feeds to check on the latest posts from different websites and blogs. Bloggers often use widgets to display RSS feeds from related blogs, or their Twitter and Delicious updates. Or they might simply pull RSS feeds into an RSS reader like Google Reader.

Comes a time where it might be useful to combine different RSS feeds into one feed. Imagine you have a blog about fundraising, and you have bookmarked 10 different fundraising sites of interest to you. You would like to display in an RSS widget on your blog with the latest updates on those blogs. You would not want to have 10 widgets, each for one RSS feed, would you?

The easiest is to combine different feeds into one. And Yahoo Pipes is still the best tool to do so.

Yahoo Pipes is a versatile tool you can use to combine different content, mash it up to create new content, but here is a step by step tutorial for the most simple thing you can do with Yahoo Pipes: combine different feeds.

Step 1: Getting started with Yahoo Pipes

1. Register for a Yahoo account
2. Go to Yahoo Pipes and log in with your Yahoo account

Step 2: Create a Pipe

A “Pipe” is a workflow you create to take RSS (or other) input, mix it up, and create a new output.

On the home page of Yahoo Pipes, click on “Create a Pipe”. This will give you a worksheet to construct a pipe, using a nice graphical user interface.
In the left column, you will see a collapsible menu with the main working modules, and the main part of the screen is your worksheet:

Step 3. Create a workflow for your pipe

1. The first thing we have to do, is to tell Yahoo Pipes where to get the input RSS feeds from, the feeds we want to combine. From the left module list, drag the module “Fetch Feed” (from the “Sources” list) into your work sheet on the right:
   
2. Note that once you dragged the “Fetch Feed” module, Yahoo Pipes automatically also put a second module “Pipe Output” in your worksheet. Leave it there for the time being.
3. The “Fetch Feed” module will open up, and you can fill in the different feeds you want to combine.
   As an example, let’s put two feeds in here, one from BlogTips and one from Blogging Today. In this case, both are Feedburner feeds, but it does not matter. You can take Atom feeds, RSS 1.0 or RSS 2.0. Yahoo Pipes will process them properly.
4. After filling in the first feed URL, click the “+” icon and fill in the second feed.
   
5. At any time during the process of creating a pipe, you can click on a module, and as it gets highlighted in orange, the debugging window at the bottom of the sheet will change when you hit “refresh”.
   The debugging module basically shows the output of that module. You can click the triangle icons to see the different internal components.
   This is very useful if you create complex pipes, but for now, we will leave it aside.
6. The next step is to connect the “Fetch Feed” module to the “Pipe Output” module: Click on the blue dot (the connection dot) at the bottom of the “Fetch Feed” module, and drag it to the blue dot at the top of the “Pipe Output” module:
   
7. And in principle you are done now.

Step 4. How to combine many many many RSS feeds

While each “Fetch Feed” module will let you enter about 20 feeds, you might need to combine more feeds than that. Easy!

1. Drag another “Fetch Feed” module from the left “Sources” list into your worksheet
2. Fill in your feeds.
3. If you still have the first “Fetch Feed” module already connected to your “Pipe Output” module, disconnect it: Just hover over the connecting blue line, and scissors will appear. Click on the scissors, and the connection line will disappear.
   
4. Now you need to combine both “Fetch Feed” modules into one. Drag the “Union” module from the “Operators” module list on the left, into your worksheet.
   
5. Now connect the two “Fetch Feed” modules to the input connectors of the “Union” module (at the top) and connect the output connector of the “Union” module to the “Pipe Output” module.
   Remember, you only need to do this, if you have a lot of input feeds. Each “Fetch Feed” module will easily cater for 20 input feeds.
6. If need be, you can add more “Fetch Feed” modules and connect them all to the “Union” module. I suggest to limit to 60-80 input feeds per pipe, otherwise the pipe becomes too slow.

Step 5. Save your pipe

Congratulations, you have just constructed a pipe which combines your RSS feeds. Now save the Pipe.

1. Click on the tab “Untitled” at the top left of your screen, and give the pipe a name, and click “OK”
   
2. Now click on the “Save” button in the right corner
3. If you have many input feeds in your pipe, saving might take a while.
   There have been times where Yahoo Pipes had performance problems and saving just hung, or gave obscure error messages, but it seems those times are now past. (touch wood)

Step 6. Create an RSS output from your pipe

While you have now successfully created a Pipes workflow, you still don’t have the combined RSS output of your pipe yet, do you? Two simple steps to get there:

1. After you saved your pipe, click on “Run Pipe”
2. A new window will open, and a screen will appear showing “Running Pipe”
   
3. Dependent on how many feeds you combine, this might take 2 or take 60 seconds, before the items in your feed will appear.
   Do check the bottom of your feed to see if there were any errors. You might have entered a wrong feed, or entered a home page URL instead of a feed, a common mistake!
4. Your pipe, combining your different RSS feeds, can be used with different tools (add it to My Yahoo!, to your Google Home page, to your Google Reader, get a badge to put the output on your blog, etc…). If you just want the link to the RSS output, click on “Get as RSS”
   
5. And voila, there is the RSS output. Copy the URL of the RSS output and you can reuse it anywhere you want.
   

Step 7. Refining your Yahoo Pipes output

Now we’re done. Well… except for one thing. Well, except for two things. Well, except for three things:

1. As it stands right now, the RSS output does not have the feed items sorted by date, which is no good. We will need to sort the RSS feed, and publish it with the most recent RSS items first.
2. If you combine many different feeds, the RSS output might contain many items, and that could clog up wherever you want to reuse the feed. We will need to limit the amount of items in the Yahoo Pipes’ RSS feed.
3. And again, if you use many different RSS feeds as input, one input blog might have re-used input from the other, so we will want to take out duplicate entries.

So roll up your sleeves once more for the “final act”:

1. Disconnect the “Union” and the “Pipe Output” module
2. From the “Operators” menu on the left, drag the modules “Sort”, “Unique”, “Truncate” onto your worksheet.
3. Connect them in this order, and fill in the different fields:
   
4. In the “Sort” module, select “item.pubDate” and “descending” order
5. In the “Unique” module, select “item.title”
6. In the “Truncate” module, fill in the maximum number of items in your output field. We used 40 in our example.
7. Connect the different modules (From “Union” or “Fetch Feed” if you don’t use the “Union” module, to the “Sort”, the “Unique”, the “Truncate” and the “Output Pipe”.
8. Save the pipe, and you are done. Yahoo Pipes will automatically change your output RSS according to the modifications you saved from your worksheet.

So now you are really done. Unless if you want to…

Step 8. Do more sophisticated stuff with Yahoo Pipes. (optional)

Yahoo Pipes allows you to do many many many different things. You can take input and mash it up to create virtually new content. You can check out the pipes others have created clicking “Browse” on Pipes’ home page. People come up with the weirdest and most sophisticated ideas, using translators, automatically integrating FLICKR pictures in a pipe, etc…

However, the most commonly needed option, is to change or filter out some of the content from the input pipes. For that, the “Regex” module from the “Operators” menu is the most used workhorse.

There are few “programming syntaxes” which are as geeky as “Regex”, so beware, this is not for the “faint of heart”. But just one example: Let’s say you want to delete all images from the input feeds, and generate a “text only” output feed. And as we started this post saying you have different blogs and sites about “fundraising”, let’s change the output marking any occurrence of the word “fundraising” in bold:

1. Disconnect the “Truncate” module from the “Pipe Output” module.
2. Drag the “Regex” module from the “Operators” menu on the left, into your worksheet
3. In the “In” field, select which part of the RSS feed you want to work with. In most feeds, the actual content of every RSS item is stored in the “item.description” field, so select that field
   Some RSS feeds store their content in other fields, so you might want to use the debugger view of Pipes to find out where the content is stored.
4. To replace all images, we need to take out the “img” tag from the content. So we want to replace anything from the beginning of the img tag until the end, with “nothing”. The Regex formula for that is:(<img.*?>)
   …which means: take any string, beginning with “<img,” following by anything (the “.*?” part) and ending with “>”. We need to replace that with “nothing” so leave the “with” field empty.
   And we want to replace all the occurrences of this in the input, so we click the “g” option in the regex module.
5. We will now look for all occurences of the word “fundraising” (including those starting with a capital “F”), and put the “bold” tags around it. Pfft, easy!
6. Our regex module will then look like this:
   
7. Don’t forget to connect the “Truncate”, “Regex” and “Pipes Output” module..
8. Save your pipe, and once again, you are done.

As said, you can go as complex as you want. If you get into trouble, call on your peers for support in the Yahoo Pipes Discussion Forum. People will be ready to help you.

9. What I use Yahoo Pipes for?

I use Yahoo Pipes for many different things.

I have different pipes where I take feeds from dozens of news sites and combine them into a single RSS output, which I read using Google Reader, both on my laptop, iPad and iPhone.

I also use Pipes to channel different feeds onto several Twitter and Facebook accounts, using a series different techniques.

But I mostly use Yahoo Pipes to combine, process and mashup feeds from over 1,000 sites and blogs about aid, development, humanitarian issues and the environment. I republish summaries, as I describe in an earlier post.
Check out the end result in Humanitarian News, AidNews, AidResources, News On Green, AidBlogs and The NonProfit Blogs.
In Blogging Today I also aggregate the latest updates from “Blogs about Blogging”, and The Weird News keeps a smile on my face…

Oh, and as a closing tip: if you want to go over “How to create a Yahoo Pipe” in a video, check this out:

Have fun with Pipes!

Picture courtesy Rajtilak