• TWE Home
• Forum Home

1. 

   Sabina
   Member

   Hi everyone!

   I just launched my first blog, http://www.solofemaletraveler.com. I love it, but it is innundated by spiders (bots). There are lots of "good" spiders like google and yahoo but there's also a massive tangle of bad spiders whom I've googled to discover they're known e-mail harvesters and content thieves. I've created a robots.txt page and run it through a filter which tells me I have it set up correctly, but still the bad spiders return because there is apparently no way to stop them, you can only ask them to please go away with the robots.txt page. And then, of course, because they're bad spiders they don't.

   So my questions are: Is anyone else bothered by the cacophony of bad bots on your site? Do you have any advice for making them more effectively go away than the robots.txt page or by paying someone to write code to try to keep them away? Also, do you know of anyone who has actually found that they have had content stolen from their blog and posted elsewhere?

   Thanks a lot,

   Sabina

   Posted 1 year ago #
2. 

   Trisha Miller
   Administrator

   Hi Sabina

   I'm heaving a huge sigh of frustration for you right now, because you've run into one of the worst parts of the internet.....and sadly there is not much you can do about it. I say not much, because while you can try things, in the end there isn't much to truly stop them.

   You're right that bad bots simply ignore the robots.txt file. And you can try messing with your .htaccess file (very carefully!) to deny access from IP addresses, but the bots may or may not leave you alone - when they get denied from too many sites they simply switch IP addresses, and THAT assumes that they even stick with the same ones - often they switch around their IP addresses so much that they can be hard to track down and deny them. AND that's not even the only way to have your content stolen, there is also your RSS feed, which site scrapers take advantage of.

   And yes, many, many bloggers have had their content stolen and posted elsewhere. The only thing you can do (when you find it) is to complain to the ISP of the offending site and hope that they will deal with the site owner.

   On VERY rare occasions you can contact a site owner - usually they don't give any clues how to contact them and use private domain registration, but sometimes they are just really stupid people who don't realize that what they're doing is wrong. When you can reach them and ask them to remove the stolen content, they will generally comply. But this is a very rare occurrence.

   While I don't advocate giving up and not dealing with it at all, I do encourage you to not let it disturb you too much.

   Posted 1 year ago #
3. 

   Sabina
   Member

   Thanks, Trisha.

   I'm glad you gave me the heads up on stolen content. I read about something - copyscape or something like that - that I'm sure you know about, which you can utilize in some way to try to prevent copying, or at least locate it easily when it occurs. I suppose I'll go that route and hope it's effective.

   Now, back to watch the spiders seeping in the crevices of my blog...

   Posted 1 year ago #
4. 

   Amandah Blackwell
   Moderator

   Trisha, I'm glad you understand the "techie" side. This was helpful information about the "robots."

   Posted 1 year ago #
5. 

   Trisha Miller
   Administrator

   Sabina I just ran into an issue with your site - you're using a plugin that disables the right-click function which some people think will stop others from copying your content.

   It actually doesn't do anything of the sort. The people who want your content can easily get it from your RSS feed, or a variety of other ways (spiders and bots don't use right click anyway, they just copy from the source code, easy to do).

   Right-clicking is a valuable tool for many other functions though, that I frequently use, such as opening links in a new window, instead of a new Tab (or sometimes, in a new Tab instead of a new window), adding sites to bookmarks, emailing a link to a post for someone else to check out, etc. I have many functions that are in a right-click contextual menu that have nothing to do with copying, but can't use them on sites that use that particular plugin. I particularly like to use right-click to retweet posts in a new window so I can continue to browse someone's site without having to use a back button. Many browser add-ons and accessbility apps also use the right click menu to add their own functions.

   It also causes me to be unable to click anywhere in a comment to edit what I've written before I submit it (like to correct a typo or change a phrase). This leaves me so frustrated that I generally just leave the site without commenting or referring it to anyone. I just don't have time to be forced into doing things the way a site wants me to, without regard for the fact that my way is faster and easier, just because the plugin makes the often-erroneous assumption that everyone who right-clicks is stealing.

   I'm just curious to know if you're really seeing any benefit from using this plugin - I realize that I may be alone in this opinion, so I'd like to hear from others if they really believe it does anything for them.

   Posted 1 year ago #

RSS feed for this topic

Reply

You must log in to post.