Question

I am working towards building an index of URLs. The objective is to build and store a data structure which will have key as a domain URL (eg. www.nytimes.com) and the value will be a set of features associated with that URL. I am looking for your suggestions for this set of features. For example I would like to store www.nytimes.com as following:

[www.nytimes.com: [lang:en, alexa_rank:96, content_type:news, spam_probability: 0.0001, etc..]

Why I am building this? Well the ultimate goal is to do some interesting things with this index, for example I may do clustering on this index and find interesting groups etc. I have with me a whole lot of text which was generated by whole lot URLs over a period of whole lot of time :) So data is not a problem.

Any kind of suggestions are very welcome.

Answer 1

Make it work first with what you ve already suggested. Then start adding features suggested by everybody else.

ideas are worth nothing unless executed.

-- http://www.codinghorror.com/blog/2010/01/cultivate-teams-not-ideas.html

Answer 2

I would maybe start here: Google white papers on IR

Then also search for white papers on IR on Google maybe?

Also a few things to add to your index:

Subdomains associated with the domain
IP addresses associated with the domain
Average page speed
Links pointing at the domain in Yahoo - e.g link:nytimes.com or search on yahoo
Number of pages on the domain - site:nytimes.com on Google
traffic nos on compete.com or google trends
whois info e.g. age of domain, length of time registered for etc.

Some other places to research - http://www.majesticseo.com/, http://www.opensearch.org/Home and http://www.seomoz.org they all have their own indexes

I m sure theres plenty more but hopefully the IR stuff will get the cogs whirring :)

友情链接