How to access and download NOAA@NSIDC data
NOAA@NSIDC data recently transitioned to HTTPS. The directory structure has NOT changed, but the beginning of the URL has changed from:
ftp://sidads.colorado.edu/pub/DATASETS/
to:
https://noaadata.apps.nsidc.org/NOAA/
For example, if you wish to access daily GeoTIFFs for the northern hemisphere from the Sea Ice Index data set the URL is now:
https://noaadata.apps.nsidc.org/NOAA/G02135/north/daily/geotiff/
Or for masked files from the SNODAS (G02158) data set the URL is:
https://noaadata.apps.nsidc.org/NOAA/G02158/masked/
Below, we describe workflows for accessing the data via HTTPS that are equivalent to those used for accessing the data on the FTP server:
- If you accessed the data using an FTP client, there isn't a direct equivalent work for HTTPS. We recommend looking at the How to access data using a web browser section below.
- If you accessed the data by logging in to the FTP server from the command lines, there isn't a direct equivalent workflow for HTTPS. We recommend looking at the How to access data using a web browser or How to access data using wget sections below as they are similar.
- If you accessed the data using a script in python or R, see the How to access the data using a Python or R script section below.
- If you accessed the data using wget either from the command line or in a bash script, see the How to access data using wget section below.
How to access data using a web browser
Data can be accessed by inputting the following URL into a web browser:
https://noaadata.apps.nsidc.org/NOAA
It will bring you to a page like the one in the screenshot below:
From this page you can navigate to the relevant data set by clicking on the data set ID and the subsequent directories. To download a file, click on the filename. To go back to the previous directory, click on the '../' at the top left of the page or click on the back button on your browser.
If you wish to download multiple files, there is an extension for the Chrome web browser called Simple mass downloader. This will load all the links on the page and you can select which ones you wish to download (see screenshot below). This is only available for the Chrome web browser and we do not provide technical support for using this third party software.
How to access data using wget
Downloading a single file
To download a single file from the HTTPS server, you can use the following wget command either from the command line or in a bash script:
wget -nd --no-check-certificate --reject "index.html*" -np -e robots=off https://noaadata.apps.nsidc.org/NOAA/<path to file>
where <path to file> is replaced by the path to the file you wish to download.
Below is an example for downloading a Sea Ice Index daily data file for the northern hemisphere:
wget -nd --no-check-certificate --reject "index.html*" -np -e robots=off https://noaadata.apps.nsidc.org/NOAA/G02135/north/daily/data/N_sea_ice_extent_daily_v3.0.csv Downloading multiple files
To download multiple files, you can use the following wget command either from the command line or in a bash script:
wget -nd --no-check-certificate --reject "index.html*" -np -e robots=off -i <links_text_file>
where <links_text_file> is the name of a text file that contains a list of the HTTPS URLs to the data files you wish to download.
Downloading a directory
To download all the files in one directory, you can use the following wget command either from the command line or in a bash script:
wget -r -nd --no-check-certificate --reject "index.html*" -np -e robots=off https://noaadata.apps.nsidc.org/NOAA/<path to directory>
where <path to directory> is the path to the directory you wish to download.
Below is an example for downloading all the 4 km GeoTIFFs for 2023 from the IMS Daily Northern Hemisphere Snow and Ice Analysis at 1 km, 4 km, and 24 km Resolutions data set:
wget -r -nd --no-check-certificate --reject "index.html*" -np -e robots=off https://noaadata.apps.nsidc.org/NOAA/G02156/GIS/4km/2023/How to access data using a Python or R script
Below are code snippets for both Python and R which demonstrate downloading a single file and all the files in a directory.
Python
Download a single file
This code snippet demonstrates how to download a single CSV file from the Sea Ice Index data set.
import requests
file_url = "https://noaadata.apps.nsidc.org/NOAA/G02135/north/daily/data/N_seaice_extent_daily_v3.0.csv"
r = requests.get(file_url)
r.raise_for_status()
except requests.exceptions.HTTPError as err:
raise SystemExit(err)
with open("N_seaice_extent_daily_v3.0.csv", "wb") as f:
f.write(r.content)
Download a directory
This code snippet demonstrates how to download all the files in a directory from the Sea Ice Index data set, in this example it is all the daily GeoTIFFs for the northern hemisphere in October 1978.
import requests
from bs4 import BeautifulSoup
archive_url = "https://noaadata.apps.nsidc.org/NOAA/G02135/north/daily/geotiff/1978/10_Oct/"
r = requests.get(archive_url)
data = BeautifulSoup(r.text, "html.parser")
for l in data.find_all("a")[1:]:
r = requests.get(archive_url + l["href"])
with open(l["href"], "wb") as f:
f.write(r.content)R
Download a single file
This code snippet demonstrates how to download a single CSV file from the Sea Ice Index data set.
setwd("<path to directory you wish to download the data to>")
url <- "https://noaadata.apps.nsidc.org/NOAA/G02135/south/daily/geotiff/2023/05_May/S_20230501_concentration_v3.0.tif"
destination <- "S_20230501_concentration_v3.0.tif"
download.file(url, destination, mode = "wb")Download a directory
This code snippet demonstrates how to download all the files from a directory for the Sea Ice Index data set, in this example it is all the daily GeoTIFFs for the southern hemisphere for May 2023.
library(rvest)
setwd("<path to directory you wish to download the data to>")
url <- "https://noaadata.apps.nsidc.org/NOAA/G02135/south/daily/geotiff/2023/05_May"
page <- read_html(url)
files <- page %>% html_nodes("a") %>% html_attr("href")
for(i in 2:length(files)){
u <- paste(url,files[i], sep="/")
download.file(u,files[i], mode = "wb")