When Web Crawlers Attack

Web crawlers, or search bots, are very popular beasts of the Internet. They allow your site to be automatically scanned and indexed. The main advantage is that people may find your site through these indexes and visit your site. The main disadvantages is that your content is copied somewhere else (where you have no control over it) and that the bots take your server resources and bandwidth.

On my site for collectors, I have created a pretty extensive robots.txt file to prevent some nicer bots from scanning parts of the site they shouldn't and blocking semi-nice bots. In addition, server rules to block some less than nice bots out there were added.

The biggest problem left unanswered is what to do when the supposedly nice bots attack your site. The web's most-popular bots is probably GoogleBot, create and operated by Google. Obviously, it brings traffic and is a good bot that should be allowed to scan the site. However, more and more frequently I see that the bot is looking for more and more URLs that NEVER existed on the site. Atop of that, since the site supports 35 languages, the bot even made up language-specific URLs. For some reason, it decided I should have a /en/phone page and so it also tries to fetch /es/phone, de/phone and so on.

So why is that so annoying? Two main reasons:

1/ It appears in my logs. I check these for errors and end up spending time on it.
2/ The bot is not giving up on these URLs although a proper 404 code is returned. It tries them over and over and over and over again.

Any suggestions? Seems to me that modifying robots.txt with 35 new URLs each time GoogleBot makes up a URL isn't the easiest solution.

The problem is not unique to GoogleBot. I have completely blocked Alexa's ia_archiver which is making up URLs like crazy.

Are there any reasons for inventing NEVER-existing URLs? Probably broken HTML files or invalid links from somewhere. Sometimes, wrong interpretation of JavaScript code (do they really HAVE TO follow every nofollow link as well???) seems to be the reason.

2009/04/15 - Read the update

Colnect Rising on Compete

Though I update about trends in site metrics for Colnect, I'm not really sure what they mean as they don't always coincide with my Analytics results. You're welcomed to check Colnect's rankings on Compete. It has risen 34% in the last month. Pretty nice :)

GMail turn 5 - still BETA??? Colnect will not follow.

Gmail's official blog announced that Gmail celebrates it's 5th birthday. 5 years is not a short amount of time. However, GMail is still in BETA. It seems that Google has changed the common meaning of "BETA" from "publicly available product about to go fully public when final fixes and additions are made" into "fully fledged public product that is expected to sometimes fail and we won't take responsibility for it when it does".
Google even created the 'beta' mark trend in logos of companies and services.

I personally find it rediculous and unfair to the customers. Of course products sometimes fail but we cannot abuse the term "BETA" for 5 (FIVE!!!) years.

Colnect has been marked as beta for less than 6 months since it went public before all key features were ready and prior to proper testing. Raising a site from grass-roots up is not a simple task. However, as of today, since Colnect is relatively stable and many of its key features (a lot more is to come but I'll elaborate on that another time) are ready and publicly available, the BETA mark will be removed.
Yes, my system may sometimes fail. Yes, it's not as perfect as I'd like it to be. However, it's public, it's working, it makes many people using it happy so it's not a beta anymore.

Buying and selling collectibles

A recent addition to premium members of Colnect are the buy and sell lists. You can read all about them here.

Buy List / Sell List

These lists are available with Premium Membership. Unlike Custom Personal Lists, collectibles added to these lists appear on the Collectors inventory information section of each single collectible item page.

When adding collectibles to these lists it is best to put the relevant price in the public note box. We suggest using world-popular currencies and use their 3 letter code rather than symbol. Example: USD is always US dollar, but the $ sign has different meaning in different countries.

NOTE! Prices you quote must be valid. You may add details regarding trades on your personal page under My Account. Complaints received regarding invalid prices (for example: you offered to sell an item for a certain price but later asked for a higher price) will be investigated. If you are found dishonest, your Colnect account may be deactivated without any refunds.

Japanese and Lithuanian languages added

Colnect is now available in 35 languages. The latest two additions are Japanese and Lithuanian.

Translations on Colnect are performed manually by volunteering translators who are members of the site. Whenever a phrase is not properly translated they can translate it easily. It's all explained here.

A recent addition is the use of automated suggestions. When a phrase has not yet been translated, it'll first be translated with an automated suggestion. An icon telling the translator he should translate (or confirm) that phrase still exists. The use of suggestions is intended for the period of time after a new content is published on Colnect (which is quite often) until a translator actually gets to translate it.

Yes, automated translations sometimes suck really bad. For example "FREE trial - 1 month" had a Hebrew suggestion that can be translated back to English as "Free trial - 1 year". What?!?! How did a month become a year? That is quite dangerous and I hope these mistakes are not too frequent. I hope that the automated suggestions many times "get over the net", meaning they are understood by the reader although acknowledged as improper language use.

Japanese is currently the only language for which Colnect yet has no translator and so we rely on the automatic suggestions. It's a sort of pilot to see if it can attract Japanese collectors and hopefully one of them will agree to become a translator. If this experiment succeeds, other languages may be added this way. A warning message will be displayed with languages that are not completely manually translated.

You're welcomed to check Transposh for translation solutions.

Twitter fails as a promotional tool?

Colnect has joined Twitter less than a week ago using a few Twitter profiles:
* Colnect news Twitter @colnect
* A personal Twitter for Colnect's manager @AmirWald
* Automated feed reporting new collectibles in Colnect's catalogs @ColnectCatalogs
* Automated feed on Colnect's catalog edits @ColnectEdits

During these days 28 visits came to Colnect from Twitter, a meager amount in comparison to the number of "followers" and energy invested. The bounce rate (visitors seeing a single page and leaving the site) was incredibly high as well. In comparison, a few posts on a relevant forum resulted in hundreds of relevant visits (with much lesser bounce rate).

It seems a lot of people use Twitter to self promote and so it's more of a bubble where "followers" is a rough indication of the number of people who will actually read anything of what you write. My guess is that for most people, a small percent of their "followers" actually read more than 5% of their tweets. Though some people think of it as a useful personal tool, it doesn't seem like they dominate Twitter.

Though less than a week may be a too short amount of time for a verdict, results so far are very unsatisfying. In the future, the automated feeds may be of use to some of the addict collectors on Colnect. Let's see what the future brings.

Colnect on Twitter

Giving up to the fad? Possibly...
Easier than blog posts? Obviously...
Useful? hmmm.....

Anyway, you're welcomed to follow the official Colnect on Twitter
All public updates regarding Colnect may be there before anywhere else.

Custom Personal Lists on Colnect

A new much-needed premium service had just been added to Colnect.

Custom Personal Lists

In addition to the common inventory lists on Colnect (Collection / Swap / Wish), premium members can create more lists. When custom lists are added, you can add and remove item to and from them in the same way you do for the common lists.

Common Uses

• Reserving items for a specific swap
• Offering many item for a single sale
• Marking items you wish to attend to later
• Creating a list to be shown to collectors on other sites

Gmail: back to the future

The following post has nothing to do with this blog but it's just something amusing I've recently seen while using my GMail. After sending a message, it tells me my message has been sent -1 minutes ago. Are they trying to say it will be sent in 1 minute? Is it yet another bug in Gmail?


Here's the attached picture. My interface is in Spanish so "minutos antes" means "minutes ago".

Biggest phone cards catalog for collectors - 142,000+ phone cards listed

For a long time, Colnect provides the world's most extensive collectible phone cards catalog. However, the phone card collectors on Colnect keep adding information to the catalog. It has just passed over 142,000 listed phonecards. The catalog is very organic and changes as the editors on Colnect receive feedbacks from collectors. It's freely available for browsing even without the need to register. Free registration to Colnect allows collectors to also manage their personal collection.

Coins catalog: over 14,000 coins

Quietly but surely, Colnect's coins catalog, being built by coin collectors for coin collectors, has surpassed 14,000 listed coins. It's freely available for browsing even without the need to register. Free registration to Colnect allows collectors to manage their personal collection.

Colnect's coins catalog is currently the world's biggest freely available resource for coin pictures and information.

Colnect rises on Compete / Quantcast / Alexa

I've been updating about Colnect's rank rising on Alexa and have now decided to include links to other rankings. So you can check out Colnect's rankings on Compete and Quantcast and start wondering what all these ratings are actually worth. So although it's nice to see Colnect gains momentum on all non-related meters, it's interesting to note the difference between what they report and between the reality. By "reality" I usually mean my reports from Google Analytics and, when really bored, the server's logs can be inspected. There's little connection between the real graphs and the estimated ones by these services. Colnect has been growing and growing throughout. Sometimes slower, lately faster. The graphs by Compete and Quantcast actually show a very inaccurate picture. Oh well...

Collectors premium services on Colnect

A few days ago, Colnect's premium services for collectors have finally opened up. Colnect has already been offering a respectable set of free services to collectors from around the world. These free services will continue to be offered for the benefit of the entire collectors community.

The announcement has been made:
"Premium membership gives you access to several features that will enhance your Colnect experience. If you regularly use Colnect, please consider supporting us by becoming a premium member."

Currently, not many premium services are offered but these will be added with time according to collector demands.

Happy collecting :)