Bogus Requests From Msnbot/Bing Crawler Resulting in Error 404

Apologies to the collectors on Colnect, this post is a technical post not related to collecting and collectibles.

Recently, we've started seeing a lot of bogus requests from msnbot. These were requests to URLs that never existed on our website and were filling up our error logs. Internet searches have led to nothing which is the reason for this blog post.

Suggested solution: disable msnbot on your robots.txt file and submit a ticket to their support. While it did take them some time to respond, they finally did and the issue has been resolved. We've then removed the block on our robots.txt and for over a week now, everything seems fine.

When asked about the reasons for these problem, the support team provided NO EXPLANATION as to what might have caused the problem:

"Thank you for patiently waiting. The product group have resolved this request but no additional information was provided."

"We are sorry to inform you that we cannot provide any information regarding this. We hope you understand."

Which leads us to believe the reason is a bug in msnbot or an attack on msnbot which messed up URLs between websites.

Here are a few such bogus requests:
[27-Dec-2013 03:48:51 GMT+0] REQUEST[/content/dam/aba/global/global_nav_logo.png] REFERER[]
USER[0] AGENT[msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)] ADDR[65.55.215.33]
[27-Dec-2013 03:49:36 GMT+0]
REQUEST[/bin/dashboard/default-avatar-unknown.png] REFERER[]
USER[0] AGENT[msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)] ADDR[65.55.215.39]
[27-Dec-2013 02:50:38 GMT+0] REQUEST[/enclosure.jpg] REFERER[]
USER[0] AGENT[msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)] ADDR[199.30.20.57]
[27-Dec-2013 02:53:48 GMT+0]
REQUEST[/user_data/packages/ginichi/img/common/btn_en.jpg] REFERER[]
USER[0] AGENT[msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)] ADDR[65.55.215.71]
[27-Dec-2013 02:54:14 GMT+0]
REQUEST[/DesktopModules/CATALooKStore/MakeThumbImage.aspx?fileticket=ikazmqBNAjM%3d&PORTALID=0&W=120&H=120] REFERER[]
USER[0] AGENT[msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)] ADDR[199.30.16.47]
[27-Dec-2013 03:33:21 GMT+0]
REQUEST[/upload/save_image/category/c1177.jpg] REFERER[]
USER[0] AGENT[msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)] ADDR[65.55.215.74]
[27-Dec-2013 03:49:20 GMT+0]
REQUEST[/wvc-1365778711/wordansfiles/images/2013/4/12/186393/186393_340.jpg] REFERER[]
USER[0] AGENT[msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)] ADDR[199.30.16.30]
[27-Dec-2013 03:52:45 GMT+0]
REQUEST[/core/media/media.nl?id=343820&c=337772&h=2fd9dd88833b005b614a] REFERER[]
USER[0] AGENT[msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)] ADDR[199.30.16.14]

Paper Clips Collectors Welcome to Colnect

First in a batch  of new categories joining the Colnect catalogs is Paper Clips. These sweet, shiny and sometimes colorful devices will be gracing the Colnect pages from now on. We are calling all collectors of Paper Clips to come, take a look, let us know what you think, and help the catalog grow :)

Axion Paper Clip, newly added on the Colnect catalog

We wish the new category coordinator, Bruun Visser [Bruun01], good luck with his new role and thank him for the efforts to create this catalog for the benefit of all collectors.

Wing Paper Clip, Newly added on the Colnect catalog

This category is starting out with 2,390 Paper Clips and with your help it will grow soon to be more. Read here how to add Information to the Colnect catalogs.

Dutch Paper Clip, Newly added on the Colnect catalog

The category has started, like all categories on Colnect, at the imitative of enthusiastic collectors on the forum. You are welcome to read how it all came about on the Paper Clips forum thread.

Promo Paper Clip, newly added on the Colnect catalog

Your collectible category is not yet on Colnect? Would like to create a new collectible catalog? Read more on how to add More Collectibles, take the reigns and make your suggestion on the New Category Forum.

Wing Paper Clip, newly added on the Colnect catalog

Colnect 2013 was Fun - version 2014 is going to be so much more!

They say that the way you start a year, is the way it will be for the rest of it. If that saying has any truth to it, judging by the amount of activity and newly added improvements, it seems Colnect is getting ready for a blast of a year to come :)

Lottery Ticket "Happy New Year!"

So what did we have in this year of 2013?

This year must have gotten some good luck when we celebrated our catalogs reaching 1,111,111 collectibles, which is seven lucky numbers one. Since then almost 300,000 additional items have been published on the catalogs.

This year had many new categories joining, from Lottery Tickets to Medals, from Drink Labels to Video Games, and from a familiar game changing Functional Cards to a the sister catalog to Teabags - Tea Labels.

Drink Label "Kruger Pilsner", from the new catalog

The New Year 2014 is now expected to bring with it New Categories that were waiting in the wings to be launched. We will have the highly anticipated Sports Cards category, and also Paper Clips, Milk Cup Lids, Kids Meal Toys and Music CDs. Many more were already suggested and are sure to join in the upcoming year.

Functional Card "Happy New Year", from the new catalog

There could never be too many helpful features and additions. In everything having to do with collectors and Colnect, the more helpful tools is definitely the merrier. We've just now added some cool helpful hands to the Inventory Notes. That's not the only feature we've added though, we did much more throughout the year. We've opened up the Personal Customs lists to all, so you can put your collection any way you'd like. We parsed the seas and divided the collectibles into Continents. We've made it easy to find items through Face Value. We sped things up with Quick Filter options. We've diversified the catalogs with adding Variant support. We even put a looking glass right into the catalog and added a Zoom into the pictures.

Medal "Royal Military", from the new catalog

Each collectible category is unique and has it's own specific needs. Many categories got tweaked and improved, decorated and streamlined. You can see that on the Postcards category that finally had Theme Collectors give out a sigh of relief, Phonecards that has split in two, following much debate, and Hotel Key Cards that got a cool new Aggregation Feature added to it's catalog.

Our community kept reaching new heights, we have marked the occasion of having 56789 collectors on Colnect, and had the opportunity to thank the 666 contributors on Colnect making it a better place for collectors every day.

Tea Label "1 De Beste", on the new catalog

Finally, December ended on an especially heart warming note, when the wonderful Colnect Coordinators, were able to gift Free Premium Memberships to some 100 happy collectors.

Lottery Ticket "Happy New Year!", from the new catalog

Colnect has been preparing to welcome the New Year in style and festivities. It has been an eventful December, that will undoubtedly lead to an exciting January 2014, full of new features, new categories, more options and more collecting.