algorithm - how to solve different url forward to same page in web robot application -


i have problem web robot application.

url a:http://www.domain.com/path?id=1

url b:http://www.domain.com/path?id=1&sessionid=xxxxxx

there 2 urls , forward same page.robot application download page twice.

in robot application, 2 url convert md5 value check visited . url string changed, md5 value changed. visited cache can not hit.

have better solution?

if i'd use algorithm calculate similarity of content, , if similar configured threshold, consider them same document. checking absolute equality (like md5sum), won't work because dynamic contents (like timestamp) break such scheme.

using document similarity common approach in web crawling prevent robots downloading same content on , over.

a simple similarity algorithm levenshtein distance job, cosine similarity better suited this.


Comments

Post a Comment

Popular posts from this blog

SPSS keyboard combination alters encoding -

my question not have programming practical issue on spss. accident keyword combination cannot remember because did not intend press changed encoding in spss , text in data in different language english shown small boxes. nevertheless, when kind of analysis performed different language text shown in output. does know reason? tried change encoding did not produce result. first, @ edit > options , see whether setting unicode or locale. might fix it. if have text in multiple languages, want unicode. otherwise, may have specific locale set wrong. can change set locale, need know locale want use.
Read more

Socket.connect doesn't throw exception in Android -

if put nonsense url no exception thrown , none of rest of code executed not rest of asynctask called method connects. try { socketcliente.connect(new inetsocketaddress("f", port), 2000); } catch (unknownhostexception e) { geterror("host doesn't exist"); return -1; } catch (ioexception e) { geterror("could not connect: host down"); return -1; } add catch statement. catch ( exception e ) { log.d(tag, e.getmessage(),e); toast.maketext(getapplicationcontext(), "unexpected error:" + e.getmessage(), toast.length_long).show(); } this write log , pop toast tell what's going on.
Read more

iphone - How do I keep MDScrollView from truncating my row headers and making my cells look bad? -

Image
here's screenshot show describing: and relevant code: - (nsinteger)spreadview:(mdspreadview *)aspreadview numberofcolumnsinsection:(nsinteger)section{ return self.viewmodel.matrix.width - 1; } - (nsinteger)spreadview:(mdspreadview *)aspreadview numberofrowsinsection:(nsinteger)section{ return self.viewmodel.matrix.height - 1; } -(mdspreadviewcell *)spreadview:(mdspreadview *)aspreadview cellforrowatindexpath:(mdindexpath *)rowpath forcolumnatindexpath:(mdindexpath *)columnpath{ mdspreadviewcell *cell = [aspreadview dequeuereusablecellwithidentifier:@"infocell"]; if (cell == nil){ cell = [[[mdspreadviewcell alloc] initwithstyle:mdspreadviewcellstyledefault reuseidentifier:@"infocell"] autorelease]; } cell.textlabel.text = ((dtvendor *)self.vendors[rowpath.row]).name; return cell; } -(cgfloat)spreadview:(mdspreadview *)aspreadview heightforrowheaderinsection:(nsinteger)rowsection{ return 200; } -(cgfloat)spreadview:(mdspreadview...
Read more