Basic Usage: Search and download products¶

This example demonstrates the standalone usage of gportal package.

Make sure to install gportal package, and import it.

$ pip install gportal

In [1]:

import gportal

import gportal

Search products via Catalogue Service¶

G-Portal provides OGC Catalogue Service API.

For API details, see Appendix 7 of G-Portal User's Manual.

Get dataset IDs¶

Dataset IDs are required to search products. You can get them by gportal.datasets.

In [2]:

datasets = gportal.datasets()

datasets = gportal.datasets()

The return value is a tree-like dictionary, which structure corresponds to the "spacecraft / sensor" search tree of the G-Portal Web UI, and the leaves are dataset IDs.

In [3]:

datasets.keys()

datasets.keys()

Out[3]:

dict_keys(['GCOM-C/SGLI', 'GCOM-W/AMSR2', 'GPM', 'GPM Constellation satellites', 'GSMaP', 'TRMM_GPMFormat', 'ALOS', 'ALOS-2', 'CIRC', 'ADEOS', 'ADEOS-II', 'AQUA', 'TRMM', 'JERS-1', 'MOS-1', 'MOS-1b', 'NASA-CMR', 'SLATS'])

In [4]:

datasets["GCOM-C/SGLI"]["LEVEL1"]

datasets["GCOM-C/SGLI"]["LEVEL1"]

Out[4]:

{'L1A-Visible & Near Infrared, VNR': ['10001000'],
 'L1A-Visible & Near Infrared, POL': ['10001001'],
 'L1A-SWI & TIR': ['10001002'],
 'L1B-Visible & Near Infrared, VNR': ['10001003'],
 'L1B-Visible & Near Infrared, POL': ['10001004'],
 'L1B-SWI & TIR': ['10001005']}

Build a search query¶

Let's search GCOM-C Level 2 SST (Sea Surface Temperature) products near Japan on 2023-01-01.

In [5]:

datasets["GCOM-C/SGLI"]["LEVEL2"]["Oceanic sphere"]["L2-SST"]

datasets["GCOM-C/SGLI"]["LEVEL2"]["Oceanic sphere"]["L2-SST"]

Out[5]:

['10002002']

In [6]:

res = gportal.search(
    dataset_ids=datasets["GCOM-C/SGLI"]["LEVEL2"]["Oceanic sphere"]["L2-SST"],
    start_time="2023-01-01T00:00:00",
    end_time="2023-01-01T23:59:59",
    bbox=[140, 30, 145, 35],
    params={
        # You can add more parameters.
        # See Appendix 7 of G-Portal User's Manual for details:
        # https://gportal.jaxa.jp/gpr/assets/mng_upload/COMMON/upload/GPortalUserManual_en.pdf
    },
)

res = gportal.search( dataset_ids=datasets["GCOM-C/SGLI"]["LEVEL2"]["Oceanic sphere"]["L2-SST"], start_time="2023-01-01T00:00:00", end_time="2023-01-01T23:59:59", bbox=[140, 30, 145, 35], params={ # You can add more parameters. # See Appendix 7 of G-Portal User's Manual for details: # https://gportal.jaxa.jp/gpr/assets/mng_upload/COMMON/upload/GPortalUserManual_en.pdf }, )

gportal.search returns a Search object.

In [7]:

res

res

Out[7]:

<gportal.Search params={'datasetId': '10002002', 'bbox': '140,30,145,35', 'startTime': '2023-01-01T00:00:00', 'endTime': '2023-01-01T23:59:59', 'count': 100}>

Search.matched returns the number of matched products.

In [8]:

res.matched()

res.matched()

Out[8]:

8

Note that any requests are not invoked with gportal.search but only invoked with Search.matched.

Search class also provides a method to iterate through the resulting products with automatic pagination.

In [9]:

products = list(res.products())
products

products = list(res.products()) products

Out[9]:

[<gportal.Product id=GC1SG1_202301010122E04710_L2SG_SSTDK_3001>,
 <gportal.Product id=GC1SG1_202301010126J04711_L2SG_SSTDK_3001>,
 <gportal.Product id=GC1SG1_202301010122E04710_L2SG_SSTDQ_3001>,
 <gportal.Product id=GC1SG1_202301010126J04711_L2SG_SSTDQ_3001>,
 <gportal.Product id=GC1SG1_202301011239J28503_L2SG_SSTNK_3001>,
 <gportal.Product id=GC1SG1_202301011235E28502_L2SG_SSTNK_3001>,
 <gportal.Product id=GC1SG1_202301011239J28503_L2SG_SSTNQ_3001>,
 <gportal.Product id=GC1SG1_202301011235E28502_L2SG_SSTNQ_3001>]

Examine metadata of resulting products¶

Product wraps a GeoJSON in search responses.

In [10]:

product = products[0]
product.to_dict()

product = products[0] product.to_dict()

Out[10]:

{'type': 'Feature',
 'geometry': {'type': 'Polygon',
  'coordinates': [[[134.51026916503906, 46.44524002075195],
    [140.54034423828125, 45.92348480224609],
    [144.48358154296875, 45.40235137939453],
    [148.24114990234375, 44.76785278320313],
    [153.3236541748047, 43.68489837646484],
    [154.52767944335938, 43.38881301879883],
    [153.22018432617188, 40.91696929931641],
    [152.01185607910156, 38.42909622192383],
    [150.8895721435547, 35.92721176147461],
    [149.84219360351562, 33.41301345825195],
    [148.86044311523438, 30.8879337310791],
    [147.93641662597656, 28.35320091247559],
    [147.46324157714844, 26.99820137023926],
    [142.8500518798828, 27.90709114074707],
    [139.73326110839844, 28.42703247070313],
    [136.67523193359375, 28.86391067504883],
    [132.38002014160156, 29.35588836669922],
    [131.33270263671875, 29.45441246032715],
    [131.81683349609375, 32.05686187744141],
    [132.29971313476562, 34.65885543823242],
    [132.7825469970703, 37.2602424621582],
    [133.26670837402344, 39.86090850830078],
    [133.75375366210938, 42.46072006225586],
    [134.24545288085938, 45.05959320068359],
    [134.51026916503906, 46.44524002075195]]]},
 'properties': {'identifier': 'GC1SG1_202301010122E04710_L2SG_SSTDK_3001',
  'acquisitionType': 'NOMINAL',
  'imageQualityDegradation': None,
  'imageQualityDegradationQuotationMode': None,
  'processingDate': '2023-01-01T15:20:47.00Z',
  'processingLevel': 'L2',
  'ProductType': 'S',
  'status': 'ARCHIVED',
  'beginPosition': '2023-01-01T01:21:56.94Z',
  'endPosition': '2023-01-01T01:26:44.51Z',
  'platformShortName': 'GCOM-C',
  'instrumentShortName': 'SGLI',
  'sensorType': 'OPTICAL',
  'operationalMode': 'NOMINAL',
  'resolution': None,
  'orbitNumber': 47,
  'lastOrbitNumber': None,
  'DayNight': None,
  'illuminationAzimuthAngle': None,
  'illuminationElevationAngle': None,
  'multiExtentOf': '134.51026916503906 46.44524002075195 140.54034423828125 45.92348480224609 144.48358154296875 45.40235137939453 148.24114990234375 44.76785278320313 153.32365417480469 43.68489837646484 154.52767944335938 43.38881301879883 153.22018432617188 40.91696929931641 152.01185607910156 38.42909622192383 150.88957214355469 35.92721176147461 149.84219360351563 33.41301345825195 148.86044311523438 30.88793373107910 147.93641662597656 28.35320091247559 147.46324157714844 26.99820137023926 142.85005187988281 27.90709114074707 139.73326110839844 28.42703247070313 136.67523193359375 28.86391067504883 132.38002014160156 29.35588836669922 131.33270263671875 29.45441246032715 131.81683349609375 32.05686187744141 132.29971313476563 34.65885543823242 132.78254699707031 37.26024246215820 133.26670837402344 39.86090850830078 133.75375366210938 42.46072006225586 134.24545288085938 45.05959320068359 134.51026916503906 46.44524002075195',
  'centerOf': '141.85800170898438 36.94295883178711',
  'product': {'fileName': 'https://gportal.jaxa.jp/download/standard/GCOM-C/GCOM-C.SGLI/L2.OCEAN.SST_/3/2023/01/01/GC1SG1_202301010122E04710_L2SG_SSTDK_3001.h5',
   'size': 4194304,
   'DataFormatType': 'HDF5',
   'version': 3},
  'browse': [{'type': 'QUICKLOOK',
    'fileName': 'https://gportal.jaxa.jp/gpr/img/br/GC1SG1_202301010122E04710_L2SG_SSTDK_3001/GC1SG1_202301010122E04710_L2SG_SSTDK_3001.jpg'},
   {'type': 'QUICKLOOK',
    'fileName': 'https://gportal.jaxa.jp/gpr/img/sb/GC1SG1_202301010122E04710_L2SG_SSTDK_3001/GC1SG1_202301010122E04710_L2SG_SSTDK_3001-000.jpg'},
   {'type': 'THUMBNAIL',
    'fileName': 'https://gportal.jaxa.jp/gpr/img/thm/GC1SG1_202301010122E04710_L2SG_SSTDK_3001/GC1SG1_202301010122E04710_L2SG_SSTDK_3001_thm.jpg'}],
  'gpp': {'datasetId': 10002002,
   'totalQualityCode': 'Good',
   'cloudCoverPercentage': 68,
   'operatorComment': None,
   'compressFlag': 1,
   'physicalQuantity': 'SGLI SST',
   'Resolution': '1km',
   'browseImageSize': 813,
   'parameterVersion': 1,
   'algorithmVersion': 3,
   'numberPixels': None,
   'numberLines': None,
   'numberBands': None,
   'numberMissingData': None,
   'sceneNumber': 10,
   'startPathNumber': 47,
   'endPathNumber': 47,
   'startArgumentLat': None,
   'endArgumentLat': None,
   'mapProjection': None,
   'mapDirection': None,
   'orbitDirection': 'Descending',
   'tileNo': None,
   'channels': None,
   'bandWidth': None,
   'downlinkSegmentNumber': None,
   'sensorOffPeriod': None,
   'sceneCenterTime': '2023-01-01T01:24:20.73Z',
   'startSubsatellitePosition': None,
   'endSubsatellitePosition': None,
   'ProcessTimeUnit': None,
   'tileHNo': None,
   'tileVNo': None,
   'topicCategory': '-',
   'organizationName': 'JAXA',
   'pseq': 'EQ',
   'hasProduct': True}}}

Product has shorthand accessors for GeoJSON properties.

In [11]:

print("Granule ID:", product.id)
print("Product URL:", product.data_url)
print("Observation Start:", product.get_as_datetime("beginPosition"))
print("Observation End", product.get_as_datetime("endPosition"))
print("Orbit Direction:", product["orbitDirection"])
print("Resolution:", product["Resolution"])
print("Cloud Cover (%):", product["cloudCoverPercentage"])

print("Granule ID:", product.id) print("Product URL:", product.data_url) print("Observation Start:", product.get_as_datetime("beginPosition")) print("Observation End", product.get_as_datetime("endPosition")) print("Orbit Direction:", product["orbitDirection"]) print("Resolution:", product["Resolution"]) print("Cloud Cover (%):", product["cloudCoverPercentage"])

Granule ID: GC1SG1_202301010122E04710_L2SG_SSTDK_3001
Product URL: https://gportal.jaxa.jp/download/standard/GCOM-C/GCOM-C.SGLI/L2.OCEAN.SST_/3/2023/01/01/GC1SG1_202301010122E04710_L2SG_SSTDK_3001.h5
Observation Start: 2023-01-01 01:21:56.940000+00:00
Observation End 2023-01-01 01:26:44.510000+00:00
Orbit Direction: Descending
Resolution: 1km
Cloud Cover (%): 68

Let's see the browse image.

In [12]:

from IPython.display import Image

Image(url=product.quicklook_url)

from IPython.display import Image Image(url=product.quicklook_url)

Out[12]:

Download products via SFTP¶

G-Portal provides an SFTP server for product files distribution.

For more details, see Chapter 3 of G-Portal User's Manual.

Download search results¶

username and password must be set to connect the G-Portal SFTP server.

In [13]:

import getpass

gportal.username = "sankichi92"  # Replace with your username
gportal.password = getpass.getpass()

import getpass gportal.username = "sankichi92" # Replace with your username gportal.password = getpass.getpass()

Give Product objects and a download directory to gportal.download.

In [14]:

targets = [p for p in products if p["Resolution"] == "1km"]
gportal.download(targets, local_dir="./downloads")

targets = [p for p in products if p["Resolution"] == "1km"] gportal.download(targets, local_dir="./downloads")

Out[14]:

['./downloads/GC1SG1_202301010122E04710_L2SG_SSTDK_3001.h5',
 './downloads/GC1SG1_202301010126J04711_L2SG_SSTDK_3001.h5',
 './downloads/GC1SG1_202301011239J28503_L2SG_SSTNK_3001.h5',
 './downloads/GC1SG1_202301011235E28502_L2SG_SSTNK_3001.h5']

Explore the SFTP server¶

SFTP.connect starts an SFTP session and returns SFTP object.

In [15]:

sftp = gportal.SFTP.connect()
sftp

sftp = gportal.SFTP.connect() sftp

Out[15]:

<gportal.sftp.SFTP at 0x112a83790>

SFTP.listdir lists directory contents like ls command.

In [16]:

sftp.listdir()

sftp.listdir()

Out[16]:

['nrt', 'process', 'order', 'standard']

You can filter the results by giving a regular expression as filter_pattern.

In [17]:

sftp.listdir("standard/GCOM-C/GCOM-C.SGLI", filter_pattern=r"SST")

sftp.listdir("standard/GCOM-C/GCOM-C.SGLI", filter_pattern=r"SST")

Out[17]:

['L2.OCEAN.SST_', 'L3.OCEAN.SST_']

The download method also accepts file paths on the SFTP server. They must be full paths from the root.

In [18]:

targets = sftp.listdir(
    "standard/GCOM-C/GCOM-C.SGLI/L3.OCEAN.SST_/3/2023/01",
    filter_pattern=r"A08D_D0000",
    fullpath=True,
)
sftp.download(targets, local_dir="./downloads")

targets = sftp.listdir( "standard/GCOM-C/GCOM-C.SGLI/L3.OCEAN.SST_/3/2023/01", filter_pattern=r"A08D_D0000", fullpath=True, ) sftp.download(targets, local_dir="./downloads")

Out[18]:

['./downloads/GC1SG1_20230101A08D_D0000_3MSG_SST_F_3000.h5',
 './downloads/GC1SG1_20230109A08D_D0000_3MSG_SST_F_3000.h5',
 './downloads/GC1SG1_20230117A08D_D0000_3MSG_SST_F_3000.h5',
 './downloads/GC1SG1_20230125A08D_D0000_3MSG_SST_F_3000.h5']

Close the SFTP session.

In [19]:

sftp.close()

sftp.close()

It's good idea to use with statement.

with gportal.SFTP.connect() as sftp:
    targets = sftp.listdir("...", fullpath=True)
    sftp.download(targets, local_dir="./downloads")