Scraping HTML in lisp -


my question related question found here scraping html table in common lisp?

i trying extract data webpage in common lisp. using drakma send http request, , i'm trying use chtml extract data looking for. webpage i'm trying scrap http://erg.delph-in.net/logon, here code

(defun send-request (sentence)  "sends sentence in http request logon parsing, , recieves   webpage containing mrs output"  (drakma:http-request "http://erg.delph-in.net/logon"                     :method :post                     :parameters `(("input" . ,sentence)                                  ("task" . "analyze")                                  ("roots" . "sentences")                                  ("output" . "mrs")                                  ("exhaustivep" . "best")                                  ("nresults" . "1"))))

and here's function having trouble with

(defun get-mrs (sentence)     (let* (        (str (send-request sentence))        (document (chtml:parse str (cxml-stp:make-builder))))       (stp:filter-recursively (stp:of-name "mrsfeaturetop") document)))

basically data need extract in html table, it's big paste here though. in get-mrs function, trying tag name mrsfeaturetop, not sure if correct though since getting error: not ncname 'onclick. scraping table appreciated. thank you.

ancient question, know. 1 that defeated me long time. it's true lot of webpages rubish, entire 2.0 build upon screen scraping, integrating heterogeneous websites hack upon hack -- should ideal application lisp!

the key (in addition drakma) lquery allows access pages contents using lispy transliteration of css selectors (what jquery uses).

let's links media strip on google's news page! if open https://news.google.com in browser , view source. you'll overwhelmed complexity of page. if view page in browsers development panel (firefox: f12, inspector) you'll see page has logic it. use search box find .media-strip-table element contain images want. open you're favourite repl. (well, let's honest here, emacs: m-x slime)

(ql:quickload :drakma) (ql:quickload :lquery)  ;;; links media strip on google's news page. (defparameter response  (drakma:http-request "https://news.google.com/"))  ;;; lquery parses page , gets ready queried. (lquery:$ (initialize http-response))

now let's explore results

;;; package qualified '$' opperator, barbaric!   ;;; use (use-package :lquery) omit package prefix. (lquery:$ ".media-strip-table" (html))

wow! that's tiny section of page? ok, how first element?

(elt (lquery:$ ".media-strip-table" (html)) 0)

ok, that's little more manageable. let's see if there's image tag in there somewhere, emacs: c-s img yay! there is.

(lquery:$ ".media-strip-table img" (html))

hmmm... it's finding something, returning empty text... oh yeah, image tags supposed empty!

(lquery:$ ".media-strip-table img" (attr :src))

holy crap! gif's aren't used unfunny, grainy animations?


Comments

Post a Comment

Popular posts from this blog

SPSS keyboard combination alters encoding -

my question not have programming practical issue on spss. accident keyword combination cannot remember because did not intend press changed encoding in spss , text in data in different language english shown small boxes. nevertheless, when kind of analysis performed different language text shown in output. does know reason? tried change encoding did not produce result. first, @ edit > options , see whether setting unicode or locale. might fix it. if have text in multiple languages, want unicode. otherwise, may have specific locale set wrong. can change set locale, need know locale want use.
Read more

Add new record to the table by click on the button in Microsoft Access -

in ms access form want implement separate button, adds new record table. in order added button , attached button event: private sub btnaddrec_click() refresh codecontextobject on error resume next docmd.gotorecord , , acnewrec if err.number <> 0 btnaddrec.enabled = false end if end end sub everything ok when open window , click btnaddrec button, problem when first of perform navigation through existed records , after click on button. got runtime error: 2105: «you can't go specified record. may @ end of recordset» . how solve issue, need have ability add new record on click on specific button, no matter, have walked or not walked through records before. thanks. i created simple form field call description (and autonumber) , created button code follows click event. filled number of records , navigated through them, clicked addnewrec button. form navigated new record without issues. able click addnewrec button directly after
Read more

javascript - jQuery .height() return 0 when visible but non-0 when hidden -

i have problem here right opposite others: in jquery cannot height of element when it's visible, when hidden. have html code: <div id="filtering-filter-holder"> <div id="filtering-filter-holder-inner"></div> </div> basic css it: #filtering-filter-holder { display: none; position: absolute; width: 420px; min-height: 100px; padding: 5px 5px 5px 5px; background-color: #ffffff; border: 1px solid #164372; } and finally, js in document.ready(): $('div.filtering-filter-headline').mouseenter(function() { var el = $(this).parent().find('div.filtering-filter-content'); if ($('div#filtering-filter-holder').is(':visible')) { $('div#filtering-filter-holder-inner').stop(true, true).fadeout(300, function() { $('div#filtering-filter-holder-inner').html($(el).html()); var height = $('div#filtering-filter-holder-inner&#
Read more