Reading Graywolf’s blog, I was reminded to watch out for duplicate content issues and Wordpress. It turns out that the wordpress default doesn’t nofollow “tags”.

Because bloggers who tag posts tend to create zillions of tags, they often end up with exactly one post in a many individual “/tag/” directories. This nearly always create duplicate content, which is not a good thing.

You’ll want to fix this; it’s fairly easy. I fixed the issue by modifying my robots.txt file.

What’s a Robot.txt file?

The robots.txt file is a plain text file you place in your root directory. It tells robots not to crawl specific files thereby eliminating the duplicate content issue.

The robot.txt file for BigBucksBlogger now reads like this:

User-agent: *
Disallow: /*.js
Disallow: /*.png
Disallow: /*trackback
Disallow: /*.css
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /tag/
Disallow: /author/
Disallow: /comments/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /wp-admin/
Disallow: /*?*
Disallow: /*?

I’m keeping robots out of my tag, author, comments and wp-admin directories, out of a number of subdirectories in wp-contents, and out of javascript, png, css, trackback files.

Because I don’t use the Wordpress default permalinks, I’ve also blocked robots from crawling addresses with query strings. Do this only if you aren’t using query strings in your permalinks otherwise, you’ll block the ‘bots from your whole site. You really don’t want to do that. Permitting the bot to index duplicate content is bad; indexing no content is worse.

I could block the bots from other files, but there is generally no need to block the robots from any file that is never linked. Also, you never want to block robots from your index.php file, sitemap or ‘.php’ files in general.

Once you create the robot.txt file, just save it with the name robots.txt file and drop it in my root directory.

Should you block from categories?

I don’t block bots from my categories directory because I only post excerpts on those pages. So, I don’t worry too much about duplicate content on those pages. In fact, since I find the ‘bots often index those and bring me traffic, I want the ‘bots to crawl those. (I am planning to change things to post 10 excerpts per page; I think that will make it easier for people who look at categories to find what they want to find.)

If you run full articles in your categories, you may wish to block ‘bot from those. The same holds for all your archives. (I’m going to be modifying my template to show nothing but excerpts at all addresses except the front page and the individula posts. This should prevent a lot of duplicate content issues without the need to block ‘bots.)

Is there anything you can do to avoid duplicate content?

Sure. In fact, there is a great Meta Robots WordPress plugin available that lets you tailor no-follows on your blog. One of the options is adding nofollow metatags to the headers of files in your “tag” directories. This will eliminate the duplicate content penalty as well.

Using the plugin also permits you to precisely tailor link-juice flow around your blog. I’ll be using this soon and explaining many of my “nofollow” decisions after I install the plugin.

I just got an email from an advertiser wanting me to remove a post from September 13.

1. What would be the logic behind that?
2. Can I delete posts, and if so, how old do they need to be?

Why ask to remove the link?

Theoretically, the advertiser was walloped by Google for paid links, and is trying to correct the issue. They are now writing posties asking them to remove posts, as required to get Google to consider re-inclusion in search results.

Will Posties remove the links?

Probably. Few want to screw over their paying customers.

Still, the responses can be a bit funny. Here’s a tongue-in-cheek quip:

“*chuckle* . wonder if they would pay you to remove it Wink”

Ouch! And who is to say it’s entirely unfair?

Here’s a refusal:

[...] the say they don’t want links from blogs anymore. Well………..I’m a tough love bitty, and my post is good and I don’t delete content.

So, I guess if you pay for links in content, you may lose control!

Two other posties suggest additional evil spins on the request:

A funny/evil thing to do would be to change the links to a competitor!

That would be a smarter choice, because if I was an evil competitor, I would nicely start emailing all the posties to get rid of the post of my competitor Twisted Evil

Which prompts at least three Posties to point out that such requests should be funneled through PPP. After all, how is the Postie to verify who is asking them to remove the links?!

Presumably, PPP will figure out a procedure to deal with these novel requests. Who’d a thunk this would happen?

We regret to inform you that your BlogRush Account Is Currently INACTIVE. Your blog(s) did not pass our Quality Review criteria. You will find instructions below for making your account active again. You will notice that the widget no longer loads on your pages — please remove the BlogRush code from your blog for now.

Having discovered “The Tampon Blog” through BlogRush, I was curious to see if it got banned.

Behold! “The Tampon Blog” appears to meet the Blogrush Quality guidelines. Here’s a screen shot of The Blogrush Widget which still displays and functions!

What type of blogs got banned? Blogs like Tamar Weinberg’s blog (for not displaying the widget), SEO Pedia, John Cow, Steve Olson, Ramblings of an Affiliate Marketer, Ryan Parker.net, Geology Joe, Dawn and Jimmy, Just a Girl in Short Shorts Talkin’ and SEO Cog who had previously tried, but failed, to quit Blogrush.

Hmmm…..

Daegan wrote a fantastic post today on “Dear John Reese: Are Tampon Sample â

Admittedly, the author is a bit muddled about my name, but they dropped me a link. Isn’t that exciting?! …Well….No. I know if I leave the trackback, my blog risks a Google penalty!

That trackback will soon link into a “bad neighborhood”. If you read further, you’ll discover why out-link to bad neighborhood’s could cause Google’s algorithm to suspect a blogger of posting paid links resulting in Google penalty!

But first: How to identify “Daegan Spam”:

1. A trackback will quote some of your post, nearly always attributing your words to someone else. I’ve been called “Daegan”, “AnnPlugged” and a variety of other names.
2. Often, the “Blog” consists of nothing but very, very short quotes. They don’t violate copyright and won’t cause you duplicate content issues. This may lead you to shrug it off and permit the trackback.
3. On some blogs — when the spam is fresh– the collection of links are sort of interesting.

   For example, today, a link lead me to a forum where someone advertised a LinkShield Link Protection, a service that will cloak affiliate links. That’s a type of product I might want to learn more about.

So, if you are aware of the danger, the spam is pretty easy to identify. Yet, it seems innocuous doesn’t it?

How could a link to that blog result in a Google penalty?

As the spammy trackback ages, the blog content becomes truly spammy. Idea Hustle, which originally left a trackback on my Duplicate Content Plugin, now redirects to a probably worthless paintball directory site, which could become anything in future.

If I leave this link in place and allow it to “follow”, I suspect I will soon be linking into into a site Google considers to fall in a “bad neighborhood”.

That’s bad for me because we know:

1. Google’s algorithm examines what we link to determine our trust and page ranks.
2. That paintball link look very spammy; after all, bloggers are often paid to link to these sorts of products.
3. Google has been penalizing even very good sites for anything Google considers or suspect to be a paid link.

So, clearly we need to keep links like that off our blogs — especially if we “dofollow”. Luckily, the solution is easy!

Send “Daegen Spam” to Akismet!

Sending these to Akismet gets it of my blog, and also protects all blogs from this sort of spam.

Better yet, even if I miss one or two of these, I never actually give these places a dofollow. L’s Linky Love won’t give the follow until they drop at least 3 trackbacks. And after I get up date all my plugins (insert some cursing at WP here), I’m going to add a feature to LLL so I can periodically review sites that are currently “dofollowed”, that way, I can catch things retroactively. (I keep saying this… some day I’ll even do it! )

I decided to just test, and slapped a video in here.

All I’m seeing in preview is a link to “Watch the latest videos on YouTube.com”. Hhhmmm…. Maybe a video will appear after they search my post? Well, I need to attend my Thursday night knitting group. If something cool appears, I guess I’ll see it when I get back.

Helio of Goteborg’s History wrote asking me how to get more links for his Swedish history page. I’ll admit I’m not an expert, but my last name sounds scandinavian, so I’m motivated to give a nice Swede the few tips I know. The first tips applies to non-English language bloggers, but read generally, the other five apply to everyone.

Here goes:

1. Provide content in English. Blogging in Swedish limits your readership and linkage; English is much more widely spoken and understood.

   A number of translation plugins exist for Wordpress. I haven’t tried them; maybe I should. You definitely should!

   Polyglot is a Wordpress plugin can help you provide content in a variety of languages, but you need to supply the content in both languages. Unfortunately, writing copy in both English and Swedish is work. Try robo-translation using Word Translation Plugin; it’s free. If free doesn’t work, you can pay for Angsumans’ Översättare Plugg Pro 4.0 Befriaren also known as “Angsuman’s Translator Plugin Pro 4.0 Released”.

2. Link or comment to niche blogs: True niche bloggers love getting links and they link back to interesting niche content. The challenge is to think of “niche” just broadly enough and to avoid becoming an obnoxious spammer.

   Helio, let’s take your blog as an example: you write about “The history of ‘X’ in Sweden”. So, you can link to a) history blogs, b) Swedish culture blogs and most importantly c) niche blogs about ‘X’.

   Your best bet is to find niche ‘X’ bloggers who seem interested in Sweden. These exist!

   Need proof? Do you see Lucy, the adorably cute cat to the left? Well, Wendy Johnson is attached to the hands holding Lucy. Wendy not only knits a lot; she sometimes knits Scandinavian styles.

   What good does this do you link-wise? There is a strong possibility Wendy might be interested in articles about the history of Scandinavian knitting. (Or she may not.)

   How might you, Helio, get a link from a blogger like Wendy? There is no sure way, but you could try this: post an interesting article about the history of Swedish knitting mentioning the Bohus Tradition; mention modern day knitters who still knit these things. Link Wendy’s Bohus sweater. (BTW: It’s a great sweater; she’s a famous American knitter and knitting book author. Your readers will probably be interested in the link!)

   If your article is interests Wendy, she might link back. (BTW: be sure to use WP-Cache if she does. Click to read why.)

   Now, don’t focus on Wendy; she’s an example. And let’s face it; she doesn’t want to be spammed.

   I want you to generalize the advice I gave you. Do you plan a post about The History of Luttefisk? Find A list (or B or C list) food bloggers who might be interested in Luttefisk. (Or better yet, blog about something edible.) Swedish Furniture? Find bloggers who discuss Swedish furniture or interior design.

   What can bloggers other than Helio do: Think about their niche broadly. Try to link to people in their tight niche, then try to find creative ways to link to people who overlap their niche.

3. Out link a lot. When first starting out, linking one person per post probably won’t do the trick. Make your goal to link to at least five bloggers who might be interested in your post.

   I’m bad. Finding five people is work. Looking at the links I’ve dropped so far, no one is likely to link back!

4. Identify your “link-niche”. How? Ask other people who visit your blog. You, Helio, may think your niche is History. It is, but are historians going to link you a lot?

   I think your “link-niche” is also Sweden! Swedish Americans; are dotty for Sweden. Find them; they will probably link more than historians. Or not– still, it’s worth aiming for links from both niches.

   I have no idea how you can attract historian bloggers. But, with a name like mine, I know loads of people who are dotty about Swedish culture.

   If you want to find niche bloggers are interested in Swedish culture, try Googling. Add Swedish term to the “niche” you are connecting too. When you eventually blog about the history of Swedish food, try “food scandinavian”. It won’t hurt to try “luttefisk blogger”, or “crafts Swedish”. To find Swedish groups, Google “Swedish Days” of “Swedish Museum”. (Heck, try Andersenville, Chicago.)

   This will take time, but if you find a blogger who is posting regularly, and mentions anything Swedish, finding a reasonable, non-spammy excuse to link them would be helpful.

   Your goal is to get these bloggers to notice your post. Their response is in their hands!

5. Always verify you pinged or tracked-back. If you mess up a link, linking won’t leave a trackback or pingback. If the blogger’s blog doesn’t post trackbacks, you won’t leave a trackback. If the blogger’s blog doesn’t let Wordpress autodiscover the trackback, you might not leave a trackback.

   You really need that trackback. So, after you hit publish, always go back to “edit” your post. Scroll down and check whether or not a trackback was sent. If it was not, check your link is correct. If the link is correct, visit the page and search for “trackback”, find the url, add it to the trackback box at Wordpress. Save your post again.

   After that, trackbacks are in the hands of the blogger you pinged or tracked back.

6. Use StumbleUpon. This is for traffic– but you need traffic to get links. Besides, I suspect StumbleUpon friends are often bloggers: they are more likely to drop a link than non-stumblers.

   So, join StumbleUpon. Install the toolbar. Make a decent profile. Apply a few of my tips.

   Spend one or two hours a week stumbling content in your niches. Find friends in your niche, defined broadly: History is a niche. Sweden is another niche. Afterwards, find the correct niches for your individual posts. If necessary, self stumble– but it’s best to be patient and wait for someone else to stumble because you’ll get more traffic, get more friends and fans.

   Heck, write a good post about the history Bohus knitting, let me know and I’ll Stumble the English language version and tag it “knitting”.

Summary of How to Get Links

So, the short advice is: provide content in English, link to bloggers, link to a lot of bloggers, develop a method of identifying bloggers who are interested in your niche (who you can later link and who will link you), make sure you they know you linked them, use StumbleUpon to make friends with… uhmm… interested bloggers!

Well, remember that there are people who like to report paid links to Google’s snitch service. Some may know enough to load your plugin folder by typing http://yourpaidlinkblog.com/wp-content/plugins/ into their browser.

If they do, will they see a list of all your plugins?

What’s bad about letting Google snitches see this list?

Well, if you’ve got the “wrong” kind of plugin, the snitch may report you to Google for taking paid posts! (And the snitch is probably correct about the paid links. After all, why else have you installed AutoPaidLinkInsertion.php? )

Google may or may not spank your Page Rank for this, but your advertisers would probably prefer Google didn’t know you were selling paid ads. (And hey, who knows? If the word gets out, some advertisers may start checking your plugins folder to figure out if Google is likely to know you are a link farm. Hiding this list could mean more money for you.)

So, maybe you’d like to prevent snoops and snitches from seeing that list of files?
Here are two easy ways to do it:

1. Upload an index.html file to wp-content/plugins. This will hide that file list in /wp-content/plugins. However, you may need to repeat this when you upgrade Wordpress. Also, you won’t hide the listing in any other folders. That may not bother you– unless you are storing something you’d rather keep under wraps.
2. Add one or two lines to the .htaccess file. Modifying .htaccess in the root directory can keep snoops from viewing listings in every directory on your site. If you think these snoops don’t exist, read Voyeur Heaven, which I discovered while Stumbling. Obviously, people snoop for many reasons including industrial espionage, curiosity, and, a desire to find porn.

How to modify the .htaccess file

Now, visit your web site using ftp. Find the ‘.htaccess’ file; the ‘dot’ in front is important. Now insert these two lines at the end of the file.

#prevent people from viewing directory listings
Options -Indexes

The first line is an optional comment. The second line prevents people from reading the list of all files when a directory that doesn’t include ‘index.php’ or ‘index.html’ file.

Now, save. Then visit your blog to make sure the blog loads. If it doesn’t, fix the .htaccess file immediately.

You’re done!

From now on, no-one can surf google for ‘index.php’ and discover stuff you might not wish them to see. Click to see how well it worked for me. http://money.bigbucksblogger.com/wp-content/plugins/.

I’ll explain:

1. How to create undetectable cloaked nofollows in Wordpress .
2. Why it is almost always a bad idea to do this.

There are other, easier ways, to make it hard for advertisers to see them– but they aren’t truly invisible. You can detect these other ways using the methods described in Two Quick Ways to Detect Cloaked Nofollows.

How to cloak nofollows so only the Googlebot sees them.

Use this two step method:

1. Prevent anyone other than Google from loading your site using the Google user agent. The purpose of this step will be to prevent detection. This can be done using BadBehavior or Noobliminal’s script to verify spiders.
2. Insert a few lines PHP code at the top of your Header.php code. This code a) detects the Google user agent, b) checks if this is the page you want to cloak and c) if both occur, it adds the new Robots Exclusion tag.

   Here’s the code to cloack-nofollow my ‘page 3′ in a Wordpress blog.

   $user_agent = $_SERVER["HTTP_USER_AGENT"];
   if(preg_match(”/(google|mediapartners)/i”,$userAgent) && is_single(’3′)){
   header(”X-Robots-Tag: nofollow”);
   }

After you’ve done this, the only visitor that can “see” the nofollow is Google.

Because the nofollow is sent by HTTP headers, it won’t display in Google’s cache. So no one will be able to visit Google, read the html in the Google cache and see you are nofollowing that page.

This method can also be adapted to apply cloaked no-follows to whole categories, archives or whatever you wish to cloak-no-follow.

Cloaked nofollows using “X-Robots-Tag: nofollow” is a bad idea.

Now that I’ve explained how to cloak the nofollows so only Google can see them, let me give a few reasons why it’s a verybad idea:

1. It’s unethical: There are legitimate reasons to nofollow links. However, only reasons I can think of to cloak a nofollow tag are either to deceive advertisers who pay you are to gip people you promised link exchanges. If want to “nofollow” every link on page 3 without cloaking, delete this bit in the if statement: “preg_match(”/(google|mediapartners)/i”,$userAgent) &&”. That will nofollow the link for everyone.

   But if you are a bit evil, you’re still tempted, right?

2. You will probably make your blog drop out of the SERPS! Why? Well, when you nofollow this way, you no follow every link on a page. This means you will nofollow links you want followed.

   If you nofollow only one page, this probably isn’t an issue. But suppose you begin trying to get clever and nofollow your whole sponsored category? Or every page that contains at least one paid link? Yes, you’ll avoid any Google penalty for paid links. But Google will stop following your internal links. Then, unless you are very, very careful, and very very clever, you will soon find your blog dropping out of SERPS for everything.

   So, massive nofollowing using the “X-Robots-Tag: nofollow” will probably hurt you more than anyone else!

3. Cloaking violates Google’s guidelines. Now, this might seem like an odd reason. After all, the reason one might wish to nofollow is to obey the guideline of “nofollowing” all paid links. But you don’t need to cloak to obey that, right? You can do nofollow publically– letting everyone including advertisers know what’s up.

   But, if you try to conceal this from the advertisers by cloaking, you violate a second Google guideline. Only Google can catch you, and we don’t know what they would do. But, given the negative impact that will come from nofollowing all your internal links, why risk getting an additional penalty for cloaking?

That’s how you cloak nofollows… and why you shouldn’t!

So, now you know how to cloak a nofollow directive so only Google can see it. But you also know that method nofollows every link on the page and so can screw up Google’s ability to crawl your blog. This means if you do it wantonly, you will hurt yourself more than you could ever hurt any advertiser or the person who gave you a reciprocal link hoping for one in return.

I was visiting your blog today, and I noticed that Advertising Age today and noticed Mark Simon criticized Mahalo. Evidently, he’s can’t believe that human powered search can work. He specifically says:

6. “HUMAN-POWERED” SEARCH ENGINES.
The reason search engines are much better places to find information than directories is because they leverage automation to do the grunt work that human editors used to do.

Well, Jason, I can’t help but agree. As I’ve pointed out, the guides you have working on Mahalo are having some difficulty applying the style guide, catching dead links, and catching sneaky redirects that appear after the page is published. Still, I know you plan to persist in the insane endeavor you call “Mahalo”.

As long as you are, I’d like to make a few suggestions. I think you will recognize the underlying goal of each: Let computers and ‘bot do what they do well, let humans do what they do well!

So, here goes:

1. Create ‘bots to count ads:
   Problem: Currently, your guides evidently need to sift through hundreds and hundreds of ad laden pages to find the ones that aren’t full of ads. This is a costly waste of time. Their time would be better spent comparing the less ad-intensive sites and deciding which is best.

   Solution: Write a ‘bot that permits the user to enter a Google search. After the Google search is run, let the ‘bot load the content of each page, identifies each (using the same method available to the FireFox extension AdBlockerPlus) and count the number of ads. Then return a link results page that includes only pages with limited numbers of ads.

   To catch cloaked affiliate ads, you may need to extend the ‘bot to follow all links and watch whether the link passes through a redirect. Flag those as suspicious.

2. Create ‘bots to discover redirects on Mahalo links:Problem: Aggressive marketers will identify Mahalo pages, and persuade the page owner to allow them to redirect the page to a substitute page. Josiah Cole notes this happened to Mahalo’s Jesus in Food page.

   Solution: Store the content of original pages. Write a ‘bot that loads each link on Mahalo. Record the url of the final destination page. If the url has changed, have this noted in a database. Send humans to check that page promptly.

3. Create ‘bots to notice changes in content.Problem: Links die. This has already happened. I identified five several dead-ish links at new sources on your Change page. They weren’t entirely dead, but the story had moved.

   Solution: Cache the source of all pages linked on Mahalo. Write a ‘bot that compare the current version of pages to old pages using the sorts of algorithms used by plagiarism checkers. (Andy Beard described a Iplagiarism.) When the content of the new site deviates substantially from the old content, send a human to check the page.

4. Create online rubrics to ensure style guides are followed.Problem: Those styles guides are long. Sometimes they are ambiguous. There is some inconsistency in applying the rules.

   Solution: Have the guides fill out a checklist covering the top ten spammy features only humans can find. No content above the fold? Check a box. Found contact email? Check. Phone? Check. That sort of thing. If the page has to many danger flags, the rubric will catch it.

5. Create ‘bots that check for spam words:Problem: Yes, searches for “Jason Calacanis” point to links about lesbians making out.

   Solution: When that ‘bot I suggested above loads all the Google search results, have it compare the content to a blacklist of spam terms. Lesbians is in my SpamKarma blacklist– put it in your Mahalo blacklist. If the ‘bot finds a bunch of porn terms, you can be pretty sure it’s a porn site. The guides don’t need to waste their time checking it!

6. Send me a huge box of luxury yarn. Or maybe season tickets to the Lyric Opera. Or something. Ok. This won’t help you at all. But you keep joking you should put me on the payroll.

   What you really should do is find someone whose first reaction to criticism of Mahalo is to assume there is a problem, and who then says: “If this is a problem, how could we fix it?”

‘Bot can help this insane idea scale.

Heck, if you write the correct ‘bot to help the humans, there is a chance you can actually get Mahalo to work within some reasonable budget. Let’s face it, you can hope the public will report your dead links. But in reality, you’ll find more often “the public” will be very eager to suggest their own sites. (For example: my knitting blog is the best blog and belongs on Mahalo. Period. )

Heck, with luck, you’ll find these ‘bots will help you explain why my knitting blog is not the best one (even though it is.)

P.S. I thought explaining how the guides can use javascript to quickly find the duplicate content I told you about before was just the right solution. Not too complicated, implemented quickly. I still think Mahalo isn’t any good yet, but I can now see why they pay you the big bucks.

P.S.S. To add to the list– there’s no reason you can’t write the bot to do the plagiarism checks before the human ever loads the page!