fill-spidered-result ( spider spider-result -- )

Vocabulary

spider.private

Inputs

spider	an object
spider-result	an object

Outputs

None

Definition

USING: accessors assocs calendar html.parser
html.parser.analyzer http.client kernel tools.time ;

IN: spider.private

:: fill-spidered-result ( spider spider-result -- )
    spider-result url>> :> url f url spider spidered>> set-at
    [ url http-get ] benchmark :> ( headers html fetched-in ) [
        url url-html? [
            html parse-html spider currently-spidering>> over
            find-all-links normalize-hrefs
        ] [ f { } ] if
    ] benchmark :> ( parsed-html links processed-in )
    spider-result headers >>headers fetched-in >>fetched-in
    parsed-html >>parsed-html links >>links
    processed-in >>processed-in now >>fetched-at drop ;