Using cURL to retrieve multiple URLs from a text file

curl -K params.txt

params.txt contents:

url = http://tomholland.me.uk/about

url = http://tomholland.me.uk/publications

output = file1.html

output = file2.html

Will create files named:

file1.html
file2.html

curl -K params.txt

params.txt contents:

url = http://tomholland.me.uk/{about}

url = http://tomholland.me.uk/{publications}

output = #1.html

output = #1.html

Will create files named:

about.html
publications.html

Update:

An alternative to the second method:

xargs -n1 curl -o "#1.html" < urls.txt

urls.txt contents:

http://tomholland.me.uk/{about}

http://tomholland.me.uk/{publications}

Will create files named:

about.html
publications.html

If you're running Tiger (Mac OS 10.4) or Leopard (Mac OS 10.5) then curl and xargs are there by default (I'm unsure about Jaguar or Panther).

Blog posts tagged "cURL"