Keep Server Online
If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.
or
A donation makes a contribution towards the costs, the time and effort that's going in this site and building.
Thank You! Steffen
Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
| |
|
 Topic: web crawler |
|
| Author |
|
sailor
Joined: 17 Apr 2015
Posts: 82
Location: US
|
 Posted: Tue 14 May '24 14:50 Post subject: web crawler |
 |
|
I need a web crawler that I can run from Windows or maybe a RedHat Linux 9 to prime a content security policy report.
It seems like curl and wget (which shows last updated in 2008), might work, but not sure if they are good enough. Also, some of the sites require a logon. |
|
| Back to top |
|
James Blond
Moderator
Joined: 19 Jan 2006
Posts: 7371
Location: Germany, Next to Hamburg
|
| Posted: Wed 15 May '24 9:53 Post subject: |
|
|
| What are the specifications? Do you want to see the web headers? Download the website? Test SSL? |
|
| Back to top |
|
sailor
Joined: 17 Apr 2015
Posts: 82
Location: US
|
| Posted: Thu 16 May '24 15:38 Post subject: |
|
|
| I have content-security-policy setup in Apache with a report to url and want to prime the reporting location (and not wait many many days). I don't think I need the headers. It would be good to crawl the entire site. Yes, definitely by SSL. |
|
| Back to top |
|
|
|
|
|
|
