diff options
| author | ale <ale@incal.net> | 2015-06-29 10:07:40 +0100 |
|---|---|---|
| committer | ale <ale@incal.net> | 2015-06-29 10:07:40 +0100 |
| commit | 9fbc656c6cd2ad610986a265c6b346bc234bb881 (patch) | |
| tree | a5aa8a44c63b239f194617dd09cfa92cf47495e0 /analysis | |
| parent | 63bd51e06b32d48878da68df8931809d42996df1 (diff) | |
| download | crawl-9fbc656c6cd2ad610986a265c6b346bc234bb881.tar.gz crawl-9fbc656c6cd2ad610986a265c6b346bc234bb881.zip | |
improve queue code; golint fixes
The queuing code now performs proper lease accounting, and it will not
return a URL twice if the page load is slow.
Diffstat (limited to 'analysis')
| -rw-r--r-- | analysis/links.go | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/analysis/links.go b/analysis/links.go index 22bcb80..a51a467 100644 --- a/analysis/links.go +++ b/analysis/links.go @@ -27,6 +27,8 @@ var ( } ) +// GetLinks returns all the links found in a document. Currently only +// parses HTML pages and CSS stylesheets. func GetLinks(resp *http.Response) ([]*url.URL, error) { var outlinks []string |