KRSTN

Isolate Conda on Linux Servers

Sun, 25 Nov 2018 00:00:00 +0000

I’m a huge fan of conda (especially on Windows) for any scientific Python work - it is a cross-platform package manager and virtual environment manager with batteries included. Conda is especially helpful for getting my colleagues set-up and ready to go and share environments. There is one big downside I encountered so far: installing it on a Linux systems has a weird default - it appends itself to the system Path and overwrites your systems Python alias. This is highly problematic in any (server) environment that depends on a stable Python version.

The easiest way I found to isolate conda from the system Python is by creating a new alias in my .bashrc that activates conda whenever I need it but leaves the system Python untouched for anything else.

# use alias to switch to Anaconda Python distribution
# keeps "python" pointing to system distribution
alias conda_on='export PATH="/users/krstn/miniconda3/bin:$PATH"'

Usage:

krstn@server:~$ conda_on
krstn@server:~$ source activate my_conda_env

Likelyhood of being hit on the head by a falling rocket on new years eve

Wed, 03 Jan 2018 00:00:00 +0000

This new years eve a friend of mine was hit on the head by the debris of a rocket and had to spend the evening in the hospital. Being hit on the head seemed very unlikely to me, but how unlikely exactly? As a fan of the TheyDidTheMath subreddit I calculated how unlikely her new years eve was. It was also a great excuse to read up on probablity theory which I haven’t used since I graduated school 10 years ago.

tl;dr

It is pretty unlikely at circa 0.07 percent.

Rockets sold in Germany

As a starting point we take the number of rockets sold in Germany and expect all of them to be fired on new years eve. The expected sale of fireworks this year was 137,000,000€ of which 20% are rockets. Assuming an average price of 0.60€ per rocket this gives us:

\[rockets_{Germany} = 137000000 * 0.2 * 0.6 = 16440000\]

Source: VPI - Verband der Pyrotechnischen Industrie (Association of the pyrotechnical industry)

Rockets in Berlin-Kreuzberg

My friend was hit on the head on a balcony in Berlin-Kreuzberg. Assuming the rockets are distributed evenly according to the population we can calculate the number of rockets in Kreuzberg as

\[rockets_{Kreuzberg} = \frac{rockets_{Germany}}{\frac{population_{Germany}}{population_{Kreuzberg}}}\]

Filling in the current population numbers we get the number of rockets fired in Berlin-Kreuzberg on new years eve 2018.

\[rockets_{Kreuzberg} = \frac{16440000}{\frac{82670000}{153887}} = 30602\]

Likelyhood of being hit on the head

We assume all debris from the rockets fired in Kreuzberg will also come back down on Kreuzberg. This should be reasonably close to the reality as most central Berlin districts have comparable population densities.

We assume an average hit area (\(area_{hit}\)) of \(0.25 m^2\). Due to the tumbling fall of the wooden sticks this should give a reasonable overlap with the head area viewed from above to ensure a hit.

Furthermore I expect my friend to have stayed outside until all rockets have been fired (pretty unlikely but estimating a firing frequency in dependency of the time is a can of worms I won’t open).

We take the basic formula for estimating the likelihood of a hit on the head from the excellent source What is the mathematical probability of getting pooped on by a pigeon in NYC?

\[1-(1-\frac{a}{A})^N\]

and include our values

\[1-(1-\frac{area_{hit}}{area_{Kreuzberg}})^{rockets_{Kreuzberg}}\]

which gives us

\[1-(1-\frac{0.25}{10380000})^{30602} = 0.000737\]

The probability of being hit on the head be a falling rocket in Berlin-Kreuzberg is 0.000737 or circa 0.07 percent.

Aftermath

For those of you wondering what damage one of those puny little sticks can do. Turns out - quite a lot actually.

Parallel Pandas

Sun, 10 Dec 2017 00:00:00 +0000

Using pandas performance is usually not an issue when you use the well optimized internal functions. However, sometimes you have to a perform a lot of calculations column wise on a large dataframe. I recently ran into this issue while calculating time series features. I increased the speed of the calculation 5x by chunking the dataframe and using parallel processing with Pythons multiprocessing library.

import numpy as np
import pandas as pd

Create a test dataset. The real dataset I am working on is a set of daily satellite measurements (from Copernicus Sentinel-1) ranging from ca. -25 to 0.

ts_df = pd.DataFrame(np.random.random(size=(365, 3000)))

I want to calculate a number of temporal features to be used as input for a regression analysis. These will be calculated for each column. The features themselves are straightforward multi-temporal features such as percentiles, using a lagged time series and some based on Fourier transformation.

def feature_calculation(df):
    # create DataFrame and populate with stdDev
    result = pd.DataFrame(df.std(axis=0))
    result.columns = ["stdDev"]
    
    # mean
    result["mean"] = df.mean(axis=0)

    # percentiles
    for i in [0.1, 0.25, 0.5, 0.75, 0.9]:
        result[str(int(i*100)) + "perc"] = df.quantile(q=i)

    # percentile differences / amplitudes
    result["diff_90perc10perc"] = (result["10perc"] - result["90perc"])
    result["diff_75perc25perc"] = (result["75perc"] - result["25perc"])

    # percentiles of lagged time-series
    for lag in [10, 20, 30, 40, 50]:
        for i in [0.1, 0.25, 0.5, 0.75, 0.9]:
            result["lag" + str(lag) + "_" + str(int(i*100)) + "perc"] = (df - df.shift(lag)).quantile(q=i)

    # fft
    df_fft = np.fft.fft(df, axis=0)  # fourier transform only along time axis
    result["fft_angle_mean"] = np.mean(np.angle(df_fft, deg=True), axis=0)
    result["fft_angle_min"] = np.min(np.angle(df_fft, deg=True), axis=0)
    result["fft_angle_max"] = np.max(np.angle(df_fft, deg=True), axis=0)
    
    return result

Testing how long the calculation takes for a small test dataset.

%%timeit -n 3 -r 3
ts_features = feature_calculation(ts_df)

11.4 s ± 86.3 ms per loop (mean ± std. dev. of 3 runs, 3 loops each)

The calculation takes quite some time and increases linear with the number of columns. My real dataset has more than 700k columns instead of the 3000 we use here. I started looking into optimizing the feature calculation when I found out that my script spent 70% of the time calculating the features.

During the calculation only one core is used. As the calculation is performed for each column we can split the dataframe into a number of subsets and utilize multiple cores to calculate the features - making this an embarassingly parallel problem.

from multiprocessing import Pool

def parallel_feature_calculation(df, partitions=10, processes=4):
    # calculate features in parallel by splitting the dataframe into partitions and using parallel processes
    
    pool = Pool(processes)
    
    df_split = np.array_split(df, partitions, axis=1)  # split dataframe into partitions column wise
    
    df = pd.concat(pool.map(feature_calculation, df_split))
    pool.close()
    pool.join()
    
    return df

%%timeit -n 3 -r 3
ts_features_parallel = parallel_feature_calculation(ts_df, partitions=14, processes=7)

2.06 s ± 15.4 ms per loop (mean ± std. dev. of 3 runs, 3 loops each)

Compare the two results to make sure we get identical results using both feature calculation functions.

ts_features.equals(ts_features_parallel)

True

Using a simple parallelization routine the time series features are now calculated about 5 times faster - a significant time saving when working with large dataframes.

Update - 2017-12-11:

Jonas (@j08lue) pointed out:

You might save some extra overhead replacing multiprocessing.Pool with concurrent.futures.ProcessPoolExecutor…

Let’s try it but this time using a 10x larger dataset.

ts_df = pd.DataFrame(np.random.random(size=(365, 30000)))

%%timeit -n 3 -r 3
ts_features_parallel = parallel_feature_calculation(ts_df, partitions=100, processes=7)

19.6 s ± 618 ms per loop (mean ± std. dev. of 3 runs, 3 loops each)

Define the new function using ProcessPoolExecutor.

# using different pool to save overhead
from concurrent.futures import ProcessPoolExecutor

def parallel_feature_calculation_ppe(df, partitions=10, processes=4):
    # calculate features in paralell by splitting the dataframe into partitions and using paralell processes
    
    df_split = np.array_split(df, partitions, axis=1)  # split dataframe into partitions column wise
    
    with ProcessPoolExecutor(processes) as pool:        
        df = pd.concat(pool.map(feature_calculation, df_split))
    
    return df

%%timeit -n 3 -r 3
ts_features_parallel_ppe = parallel_feature_calculation_ppe(ts_df, partitions=100, processes=7)

18.4 s ± 376 ms per loop (mean ± std. dev. of 3 runs, 3 loops each)

Employing this tip we can indeed save about 6% execution time. Just keep in mind that this requires Python 3.2 or newer.

Awesome Sentinel

Wed, 11 Jan 2017 00:00:00 +0000

The Copernicus Sentinel satellites are an amazing opportunity for people all over the world to explore unprecedented amounts of remote sensing data under a free and open license. The collaborative nature of the Copernicus programme means that there are multiple ways to get access to Sentinel satellite data and the open license allows anyone to host mirrors of the data. The definitive data access portal for Sentinel data is the Copernicus Open Access Hub aka. SciHub. It contains all Sentinel-1, -2 and -3 data since their respective launches and offers a graphical search interface as well as an API. It is based on the open-source data hub software, which is also employed by most national mirrors. There are however a wide number of mirrors and tools to get the data and process it. This post is a curated list of awesome tools, tutorials and APIs related to the Copernicus Sentinel Satellites.

Data Access

Data Hubs and National Mirrors

Official datahubs and mirrors by the Copernicus partners and Collaborative Ground Segment members.

Partial Mirrors

Initiatives to integrate specific Sentinel data into existing search and discovery platforms.

Alaska Satellite Facility (Sentinel-1)
Centre for Environmental Data Analysis - CEDA (Sentinel-1, -2)
Theia (Sentinel-2)
- atmospherically corrected L2A products covering several European countries and areas proposed by scientists
- published less than two days after L1C is available
USGS EarthExplorer (Sentinel-2)
EUMETSAT CODA (Sentinel-3 Marine Products)
- 14 day rolling archive of Sentinel-3 L1 and L2 marine products in near real-time (NRT), short time critical (STC) and non time critical (NTC) latency mode
DLR Geoservice (Sentinel-2)
- Download 2 years rolling archive of MAJA-corrected Sentinel-2 scenes covering Germany
NOAA CoastWatch
- Sentinel-3 OLCI and Sentinel-2 over United States coasts
NASA Earthdata
- search NASA mirrors for Sentinel-1, Sentinel-3, and Sentinel-5P

Cloud Providers

Providers that host Copernicus Sentinel data and allow you to bring your own code to process it without the need to download the data.

Open data on AWS
- Sentinel-2 L1C and L2A hosted in region eu-central-1 (Frankfurt), requester-pays S3 buckets
- Sentinel-2 L2A Cloud-Optimized GeoTIFFs, hosted in region us-west-2 (Oregon), S3 buckets
  - STAC Browser for Sentinel-2 COG hosted on AWS
- Sentinel-1 GRD hosted in region eu-central-1 (Frankfurt), requester-pays S3 buckets
- Sentinel-1 ARD CONUS - analysis ready dataset of Sentinel-1, tiled COGs, for contiguous United States
- Sentinel-3 NRT, STC, NTC, COG - all Sentinel-3 products for near-real time, short time critical and not time critical, all products also available converted to COGs
- Sentinel-5P L2 - all Level-2 products from Sentinel-5P, also available as COG converted data
Google (Sentinel-2)
- public Google Cloud Storage bucket, .SAFE format, EU region
Planet
- Sentinel-2 included in commercial API

DIAS

Data and Information Access Services (DIAS), funded by the European Commission “providing centralised access to Copernicus data and information, as well as to processing tools”

CREODIAS
- full Sentinel archive, free visualization and download through data discovery portal and CREODIAS Browser
- large EO data archive including Landsat, Envisat and others, next to Copernicus data
MUNDI
- Sentinel archive, free visualization and download through mundi web services
ONDA DIAS
- VM Infrastructure as a Service, with an API for access to hosted Copernicus data
sobloo
- on-demand processing of thematic products with link to other data-sets (i.e. geo-marketing)
WEkEO
- harmonised data access with a REST API, hosted VM options

Tools

Specific to Copernicus Sentinel data discovery, download and processing.

Search & Download

sentinelsat
- search and download from any DHuS-powered Datahub. Comes with an intuitive command line and a flexible Python API.
Sentinel-download
- download Sentinel-2 data from Copernicus SciHub. Supports download of sub-tiles in the old product format (PDS <14).
peps_download
- download data from the French National Mirror (PEPS).
Sentinel2ProductIngestor
- ingest Sentinel-2 data from SciHub into S3. Used by Sinergise to populate the AWS Sentinel-2 mirror
sat-download
- download Sentinel-2 data from AWS
sat-api
- query Sentinel-2 data on AWS using APIGateWay
- deployed by Development Seed at https://api.developmentseed.org/satellites
awsdownload
- downloader for Sentinel-2 products from Amazon or SciHub
sentinelhub-py
- Python library for downloading Sentinel-2 data from Amazon into ESA .SAFE format and interface Sentinel Hub OGC services
aws-sat-api
- Simple Serverless API for satellite data hosted on AWS Public Dataset
sentinel2-search-api
- query Sentinel-2 data hosted on AWS by MGRS tile
- API deployed at https://sentinel2.satgateway.com, tile preview front-end deployed at https://s2viewer.satgateway.com
sentinel2_aws
- Ruby gem for parsing Sentinel-2 metadata from AWS
eodag
- command line tool and plugin-oriented Python framework for search and download from multiple providers including all DIAS
sentinelloader
- Sentinel-2 satellite tiles images downloader from Copernicus. Minimizes data download and combines multiple tiles to return a single area of interest

Viewers & Portals

AWS/Sinergise “Sentinel Image Browser”
- search Sentinel-2 data available on Amazon Webservices
EOS “Land Viewer”
- viewer for Landsat-8/7, MODIS and Sentinel-2 data hosted by AWS
- visualize band combinations on-the-fly
jeobrowser “Rocket”
- viewer for Sentinel (1,2,3), Landsat-8, SPOT and Pleiades imagery
- based on resto search engine and used as frontend for PEPS
mundialis “EO-me”
- viewer for Sentinel-2 and Landsat-8 data with custom metadata filters
- satellite tiles enriched with additional metadata (e.g. terrain statistics, NDVI at overpass, climatic parameters, population count)
OceanDataLab
- portals focussing on Ocean Remote Sensing data, including Sentinel-1 and 3
RemotePixel “Viewer”
- open source viewer for Landsat-8, Sentinel-2 and CBERS-4 data hosted by AWS
- uses sentinel-tiler (tiles server based on AWS Lambda)
RemotePixel “Satellite Search”
- open source Browser for Landsat-8 and Sentinel-2 data hosted by AWS
- supports on-the-fly display and calculation of band combinations
- uses remotepixel-api (based on AWS Lambda)
Research and User Support (RUS)
- service portal to promote the uptake of Copernicus data and scaling of R&D activities
- provides training and computing environments
Sinergise “Sentinel Playground”
- visualize AWS Sentinel-2 data in different band combinations
- offers a WMS/WMTS service.
Sinergise “Sentinel-Hub”
- search Sentinel-1, 2, 3 and other free satellite data
- supports pixel based band-math operations and simple data processing
SnapPlanet
- Android / iOS App to to view Sentinel-2 images, compare changes and share
Spectator
- real-time tracking of EO satellites, set-up custom channels to track ROI overpass and preview images
Thematic Exploitation Platforms “TEPs”
- platforms for finding and processing (Sentinel) data relating to a thematic topic
- available platforms: Coastal, Forestry, Geohazards, Hydrology, Polar, Urban, Food Security
USGS “Sentinel2Look”
- variant of the LandsatLook Viewer to search and download Sentinel-2 data from the USGS archive
ESRI Sentinel-2 Explorer
- view Sentinel-2 data rendered with a number of indexes
- available as ArcGIS ImageServer (REST)

Processing

SNAP (Sentinel Application Plattform)
- (pre-)process any Sentinel data
- also available as docker
ARCSI (Atmospheric and Radiometric Correction of Satellite Imagery)
- atmospheric correction of Sentinel-2 data
Google Earth Engine
- process the global Sentinel archives directly on Google’s servers
EOS Processing
- workflow library for thematic processing of (Sentinel-2) satellite data
iCOR
- atmospheric correction of Sentinel-2 data
- available as SNAP plugin
MAJA (MACCS ATCOR Joint Algorithm)
- atmospheric correction of Sentinel-2 data using time series
- used for Theia and Sen2-Agri
Sen2-Agri
- toolbox for processing images for agricultural purposes
- includes modules for atmospheric correction, monthly syntheses, biophysical variables, crop mask, crop-type classification and an orchestrator
s2cloudless
- single scene, pixel-based cloud detection algorithm used at Sentinel-Hub
- accompanying write-up with performance comparison to other cloud detection algorithms
Sen2Cor
- atmospheric correction of Sentinel-2 data
- basis for L2A data published on Copernicus Open Access Hub
sen2r
- R toolbox to search, download and pre-process Sentinel-2 data
ACOLITE
- atmospheric correction algorithms for aquatic applications of Landsat and Sentinel-2
C2RCC
- atmospheric correction of Sentinel-3 and -2 for coast colour applications
- included in the SNAP toolbox for Sentinel-3
i.sentinel.mask
- GRASS GIS addon for atmospheric correction of Sentinel-2 including cloud and shadow detection
sat-stac-sentinel
- convert original Sentinel-1 and -2 metadata into STAC items
EOReader
- Opensource Python library reading Sentinel-1, 2, 3, and other optical and SAR sensors - loading and stacking bands in a sensor-agnostic way
xsar
- read Sentinel-1 data into xarray for further processing

Products

Products, datasets and applications generated from Copernicus Sentinel data.

EOX “Sentinel-2 cloudless”
- cloudless, medium brightness, global Sentinel-2 composite
- also provided as WMTS Layer under CC BY 4.0
- original GeoTIFF tiles provided on AWS S3 bucket
S2GLC Land Cover Map of Europe 2017
- tiles in MGRS (Sentinel-2) available at CREODIAS Finder (collection: S2GLC)

last edit: 2024-03-08 - changelog

Know another awesome API, mirror or app that is missing from this list? Send it to me via Email, Twitter or open a pull request on Github.

Beautiful Map Backgrounds - Slope Hillshades

Sat, 30 Jul 2016 00:00:00 +0000

In the field of remote sensing you often have to present classification results as maps and need a nice, unobtrusive background that gives the viewer an idea where everything is located without being as distracting as a RGB satellite image. A nice choice for this is a grey hillshading or slope relief background that adds some texture information to your map by using a Digital Elevation Model.

Downloading the DEM

To get the result pictured above you need a DEM as a starting point. A popular (and free) choice for this would be the SRTM 1-arc second DEM which is available for almost all regions throughout the world.

Download can be either done through EarthExplorer or directly from http://e4ftl01.cr.usgs.gov/. The easiest way for multiple tiles is finding out their names on EarthExplorer and then download them from e4ftl01 through wget.

This example will use four tiles placed in the Hanzhong Plain, China.

# download
curl -O http://e4ftl01.cr.usgs.gov/SRTM/SRTMGL1.003/2000.02.11/N32E106.SRTMGL1.hgt.zip
curl -O http://e4ftl01.cr.usgs.gov/SRTM/SRTMGL1.003/2000.02.11/N32E107.SRTMGL1.hgt.zip
curl -O http://e4ftl01.cr.usgs.gov/SRTM/SRTMGL1.003/2000.02.11/N33E106.SRTMGL1.hgt.zip
curl -O http://e4ftl01.cr.usgs.gov/SRTM/SRTMGL1.003/2000.02.11/N33E107.SRTMGL1.hgt.zip

# unzip and remove
unzip /data/China/slope_relief/\*.zip -d /data/China/slope_relief/
rm /data/China/slope_relief/\*.zip

Merge

In a next step we merge the .hgt DEMs into a single GeoTiff. This step could also be used for warping into a different projection, resolution, etc.

# merge
cd /data/China/slope_relief/
gdal_merge.py -o hanzhong-dem.tif *.hgt

Calculate Slope

For this we will use gdaldem with the additional -s parameter to account for the difference in horizontal units (degree) and vertical units (meters). Also we want to compute at raster edges and near no_data values and therefore invoke -compute_edges. If you want the slope in percent instead of degree use -p.

cd /data/China/slope_relief/
gdaldem slope -s 111120 -compute_edges hanzhong-dem.tif hanzhong-slope.tif

Coloring

In a last step we want to color our results. gdaldem color-relief is usually used to color DEMs in green, brown and yellow. We will use it to color our slopes from white (0 degrees) to black (90 degrees).

Therefore prepare a slope_color.txt with the content:

0 255 255 255
90 0 0 0

And finally create the finished result.

cd /data/China/slope_relief/
gdaldem color-relief hanzhong-slope.tif slope_color.txt hanzhong-slope-relief.tif

Prototyping time-series analysis with Google Earth Engine

Fri, 06 May 2016 00:00:00 +0000

Sometimes you are struck by an idea how to analyze satellite images, but how fast can you get from a remote sensing idea to a prototype of the result? The sheer size and amount of openly available satellite images makes developing prototypes rather cumbersome. To get from an idea to a prototype product you need to download gigabytes of data, pre-process it, write the code that calculates your prototype product and save it somewhere. Oftentimes you’ll end up spending more time on downloading and pre-processing than the actual prototyping and calculation. More often than not I discarded ideas because I had no time to test them. Today I’ll show an example of how Google’s Earth Engine can be used to put the rapid back into rapid prototyping for remote sensing.

Goal: Find agricultural areas with Sentinel-1

My goal for this example is to detect agricultural areas in the Mekong Delta in Vietnam, one of the largest rice growing regions in the world. The data I want to use are the Sentinel-1 GRD products, published by ESA. These are Ground Range Detected Synthetic Aperture Radar Intensity Images - in other words images that measure how much of the radar signal emitted by the satellite arrives back at it.

The idea for this prototype is rather simple: Use the complete 2015 time-series of Sentinel-1 images, calculate the amplitude of the signal and find a threshold to distinguish agriculture from other land cover. The principle behind this idea is, that plant growth influences the intensity of SAR backscatter, especially with rice plants that are planted on flooded paddy fields.

Developing a prototype to test this idea on my local PC would mean I needed to download 243 Gb of satellite images, pre-process them to sigma0 (the backscatter intensity) and write code that calculates the amplitude at each pixel’s location and finally apply my threshold. We’re looking at weeks of work to test my idea. In order to develop a rapid prototype I turned to Google’s Earth Engine.

Earth Engine

Google’s Earth Engine is a platform that allows you to calculate remote sensing products on Google’s cloud infrastructure. In other words - their engineers handle the large datasets, complex server infrastructure and processing framework and you just provide code to calculate the results you want.

They ingested a large number of datasets, including the complete Landsat archive and both Sentinel archives. New Landsat and Sentinel data is also ingested shortly after publication. The processing framework is exposed with a well documented API that can be accessed with JavaScript or Python. The cherry on top is an online code editor that stores and versions all your code. All that is distributed for the low price of free - you just need to register for a free account.

Sentinel-1 time-series

Datasets are stored as collections (basically time-series stacks). We’ll start by loading the Sentinel-1 image collection and filtering it to all VH polarized, GRD images acquired in 2015 in Descending orbit direction.

// Load the Sentinel-1 ImageCollection.
var sentinel1 = ee.ImageCollection('COPERNICUS/S1_GRD');

// Filter VH, IW
var vh = sentinel1
  // Filter to get images with VV and VH dual polarization.
  .filter(ee.Filter.listContains('transmitterReceiverPolarisation', 'VH'))
  // Filter to get images collected in interferometric wide swath mode.
  .filter(ee.Filter.eq('instrumentMode', 'IW'))
  // reduce to VH polarization
  .select('VH')
  // filter 10m resolution
  .filter(ee.Filter.eq('resolution_meters', 10));
// Filter to orbitdirection Descending
var vhDescending = vh.filter(ee.Filter.eq('orbitProperties_pass', 'DESCENDING'));
// Filter time 2015
var vhDesc2015 = vhDescending.filterDate(ee.Date('2015-01-01'), ee.Date('2015-12-31'));

The image collection spans the globe, or at least the parts that Sentinel-1 regularly covers. I am only interested in the Mekong Delta for now, so I’ll clip the image collection with a polygon of my Region-of-Interest.

var roi = ee.Geometry.Polygon(
        [[[104.39967317142975, 8.648797299726574],
          [104.97136816724105, 8.42750154669911],
          [105.44386722890897, 8.433739470974832],
          [106.66892316475673, 9.522314436594979],
          [107.25187226402818, 10.336446225915548],
          [106.80624398899499, 11.138468981755926],
          [104.38860737239793, 10.944315053122876]]]);
// Filter to MKD roi
var s1_mkd = vhDesc2015.filterBounds(roi);

Next I’ll calculate the amplitude. Since I am not sure if the data contains outliers I’ll use the 90th and 10th percentile.

var p90 = s1_mkd.reduce(ee.Reducer.percentile([90]));
var p10 = s1_mkd.reduce(ee.Reducer.percentile([10]));
var s1_mkd_perc_diff = p90.subtract(p10);

This type of calculation of features over a time-series is blazingly fast and it only takes seconds to visualize the results for this. Areas with a high amplitude are bright and areas with low amplitude are dark. Near the city of Rach Gia we can already see some structures from this amplitude image. Some (presumeably) forest and urban areas in dark color, some very bright areas, which might be rice fields or other agriculture.

Map.addLayer(s1_mkd_perc_diff, {min: [2], max: [17]}, 'p90-p10 2015', 1);

Following the initial idea, that agriculture has a higher backscatter amplitude than other land cover, I’ll mask all areas with a backscatter amplitude larger than 7.5 dB.

var amplitude_mask = s1_mkd_perc_diff.gt(7.5);
var amplitude_mask = amplitude_mask.updateMask(amplitude_mask);
Map.addLayer(amplitude_mask, {palette: 'FF0000'}, 'amplitude_mask', 1);

We can see that a lot of structures are coming out nicely. The rectangular shape of the masked areas in red and their high backscatter amplitude mean that they are likely agriculture fields, which in this region is very likely rice. This prototype also allows me to see the limitations of the initial idea. A lot of area over the sea is masked as well, because waves create pixels with a high backscatter amplitude as well. Also near the coast are some small rectangular masked areas which I suspect to be aquaculture, as it would be unusual to have fields this close to the sea.

time well spent

All in all creating this prototype took me three days, which included registering for an account and learning basic JavaScript (I mostly dabble in Python). This sort of rapid prototyping enables me to test the validity of ideas I’d usually have discarded due to the time it would take to create the results. For Remote Sensing scientists and enthusiasts the Earth Engine is a nice playground to effectively process large amounts of data, if your processing chain works with the data and API methods they supply.

Download Copernicus Sentinel-2 images

Sat, 23 Jan 2016 00:00:00 +0000

The Sentinel satellites are an amazing opportunity for scientists all over the world to explore unprecedented amounts of remote sensing data free of charge. I am genuinely happy that this is one of the first large remote sensing missions that has Open Data and Open Access baked in right from the start. All Sentinel data can be accessed through Copernicus Open Access Hub. Searching and downloading of multiple scenes however is not very user friendly. Fortunately the Hub also provides an API we can use to search and download multiple scenes at once. This post aims to show a simple workflow of searching and downloading multiple Sentinel-2 scenes using the Python package sentinelsat

There are three steps necessary before you can start downloading Sentinel data.

Register at the Hub
Install sentinelsat with pip install sentinelsat
Create a GeoJSON polygon of your area of interest.

Once you’ve taken care of 1 and 2 head over to Geojson.io (provided by the awesome folks over at @mapbox) and draw your Area-of-Interest.

Here is what I am looking at today - the surroundings of Lake Tonle Sap in Cambodia, a region rich in rice agriculture. Just draw a polygon around your search area and save it as Geojson.

Command Line

If you just want to search for and download scenes the easiest way is through sentinelsats command line interface. From the command line this is what I would execute to get an overview of the available data for Lake Tonle Sap between October 2015 and February 2016.

sentinelsat -u <user> -p <password> -g tonle_sap.geojson --sentinel 2 --cloud 40 -s 20151001 -e 20160201

Explaining the parts of this command.

sentinelsat
-u <user> -p <password> username and password you registered
-g tonle_sap.geojson geometry of our search area
--sentinel 2 limit the search to Sentinel-2
--cloud 40 limit the maximum cloud cover to 40%, since this is a cloud prone area
-s 20151001 -e 20160201 start and end date of our search formatted as YYYY-MM-DD

You can use the --footprints flag to create a GeoJSON search_footprints.geojson showing you which scenes fulfilled your search criteria, their respective download links and more metadata. This is a good starting point to get an overview of the available data.

To download all the scenes simply add the option -d and make sure you have enough diskspace, as most scenes from that time are 5-7Gb each. It is also a good idea to use the provided MD5 checksum and verify the integrity of the downloaded files with the --md5 flag.

sentinelsat -u <user> -p <password> -g tonle_sap.geojson --sentinel 2 --cloud 40 -s 20151001 -e 20160201

Python API

All this can also be done from within Python.

Set the connection details for the Hub:

from sentinelsat.sentinel import SentinelAPI

s2_api = SentinelAPI(
    user="username",
    password="password",
    api_url="https://scihub.copernicus.eu/apihub/"
)

Query products in our AOI.

from sentinelsat import read_geojson, geojson_to_wkt

products = s2_api.query(
    area = geojson_to_wkt(read_geojson('tonle_sap.geojson'))
    date = ("20151001", "20160201"),
    platformname = "Sentinel-2",
    cloudcoverpercentage="(0,40)"
)

All products from the query are stored in a Python dictionary.

print(s2_api.get_products())

You can convert the results into a GeoJSON FeatureCollection or a GeoPandas GeoDataFrame to manipulate further.

gj = api.to_geojson(products)
geo_df = s2_api.to_geopandas(products)

Download all products from the query.

s2_api.download_all(products)

For more examples on the download options and the Python functions head over to the sentinelsat Github repository or check the documentation.

last update: 2017-10-08 to change syntax to sentinelsat v0.12

How to fill a Donut

Thu, 31 Dec 2015 00:00:00 +0000

When you are classifying pixels in satellite images you often encounter the dreaded artefact commonly referred to as Donut. A good example is the classification of lakes where you are not able to correctly classify the complete shoreline (resulting in non-closed circles or Open Donuts) and remaining pixels in the wrong class which make up the Hole inside the Donut. Recently a question popped up on GIS.Stackexchange on How to transform raster donuts to circles This problem can be solved with the use of mathematical morphology.

Donuts

Here we are working with an example image but really any dataset that can be read into a binary numpy array will work.

%matplotlib inline
from scipy.ndimage import imread
import matplotlib.pyplot as plt

donut_image = imread(fname="Donuts.png", flatten=True)
donut_image[donut_image == 0] = 1
donut_image[donut_image == 255] = 0
donut_image = donut_image.astype(np.int8)

plt.imshow(donut_image)

You can see there are open structures, closed ones - artefacts just like you’d expect when you try to classify lakes in an image.

Closing

First we are looking to close the gaps in the outer rings. Working with a binary image any package including mathematical morphology algorithms should work. This is an example using scipy.ndimage.morphology.

from scipy.ndimage.morphology import binary_closing

donut_closed = binary_closing(donut_image, structure=np.ones((3,2)))
plt.imshow(donut_closed)

All the gaps in the outer borders are now closed, but the holes of the donut are still intact. The structure used to dilate and erode the structures needs to be adapted to the size of the gaps - this might involve a bit of trial and error.

Filling

In a last step we just need to fill the remaining Donut Holes and get the desired result.

from scipy.ndimage.morphology import binary_closing

donut_filled = binary_fill_holes(donut_closed)
plt.imshow(donut_filled)

np.nanpercentile() - there has to be a faster way!

Thu, 17 Sep 2015 00:00:00 +0000

Recently I was trying to calculate the quantiles of the vegetation index of an area over time. For this I have a time-series of satellite raster images of a certain region that cover identical extents. This is represented as a numpy.ndarray of the shape(96, 4800, 4800) - in other words 96 satellite images each measuring 4800 by 4800 pixels. I want to calculate the 10th, 25th, 50th, 75th and 90th quantile along the time/z-axis, which can be done easily with np.percentile(a, q=[10,25,50,75,90], axis=0). The data I am working with contains no_data areas due to residual cloud cover, rainfall, etc. represented as np.NaN. Naturally I was turning to numpys np.nanpercentile(a, q=[10, 25, 50, 75, 90], axis=0). Unfortunately np.nanpercentile() was ~300x slower on my dataset than np.percentile() so I had to find a way to speed it up.

np.nanpercentile() speed

Generate some test data and benchmark numpys function to see what we are working with:

# create array of shape(5,100,100) - image of size 10x10 with 5 layers
test_arr = np.random.randint(0, 10000, 50000).reshape(5,100,100).astype(np.float32)
np.random.shuffle(test_arr)
# place random NaN
rand_NaN = np.random.randint(0, 50000, 500).astype(np.float32)
for r in rand_NaN:
    test_arr[test_arr == r] = np.NaN

%timeit np.nanpercentile(test_arr, q=25, axis=0)
100 loops, best of 3: 5.92 ms per loop

The reason np.nanpercentile() is so slow in my case can be found in the source code. Numpys implementation includes a private function to calculate the percentile along a 1D array while ignoring NaNs. This means that it has to loop through all pixels resulting in 4800 x 4800 native Python loops. Not exactly a recipe for speed.

The alternative: a pure Python implementation of the quantile calculation

Since we are working with just 3 dimensions we can use a pure Python implementation of the quantile function and should see a significant increase in speed (sounds wrong, I know!). The code for percentile calculation was posted in a stackoverflow answer.

The general idea is to:

find the number of valid observations (non NaN)
replace NaN with maximum value of array
sort values along axis
find position of quantile regarding number of valid observations
linear interpolation if the desired quantile is inbetween two positions (like numpys linear interpolation)

def nan_percentile(arr, q):
    # valid (non NaN) observations along the first axis
    valid_obs = np.sum(np.isfinite(arr), axis=0)
    # replace NaN with maximum
    max_val = np.nanmax(arr)
    arr[np.isnan(arr)] = max_val
    # sort - former NaNs will move to the end
    arr = np.sort(arr, axis=0)

    # loop over requested quantiles
    if type(q) is list:
        qs = []
        qs.extend(q)
    else:
        qs = [q]
    if len(qs) < 2:
        quant_arr = np.zeros(shape=(arr.shape[1], arr.shape[2]))
    else:
        quant_arr = np.zeros(shape=(len(qs), arr.shape[1], arr.shape[2]))

    result = []
    for i in range(len(qs)):
        quant = qs[i]
        # desired position as well as floor and ceiling of it
        k_arr = (valid_obs - 1) * (quant / 100.0)
        f_arr = np.floor(k_arr).astype(np.int32)
        c_arr = np.ceil(k_arr).astype(np.int32)
        fc_equal_k_mask = f_arr == c_arr

        # linear interpolation (like numpy percentile) takes the fractional part of desired position
        floor_val = _zvalue_from_index(arr=arr, ind=f_arr) * (c_arr - k_arr)
        ceil_val = _zvalue_from_index(arr=arr, ind=c_arr) * (k_arr - f_arr)

        quant_arr = floor_val + ceil_val
        quant_arr[fc_equal_k_mask] = _zvalue_from_index(arr=arr, ind=k_arr.astype(np.int32))[fc_equal_k_mask]  # if floor == ceiling take floor value

        result.append(quant_arr)

    return result

One issue to work around is numpys implementation of np.choose() which only allows for 32 choices, meaning it failed on my dataset with with 96 raster images. Luckily my question on StackOverflow, Index 3D array with index of last axis stored in 2D array was answered with a workaround to indexing the 3D array without using np.choose(). This is achieved with a small helper function.

def _zvalue_from_index(arr, ind):
    """private helper function to work around the limitation of np.choose() by employing np.take()
    arr has to be a 3D array
    ind has to be a 2D array containing values for z-indicies to take from arr
    See: http://stackoverflow.com/a/32091712/4169585
    This is faster and more memory efficient than using the ogrid based solution with fancy indexing.
    """
    # get number of columns and rows
    _,nC,nR = arr.shape

    # get linear indices and extract elements with np.take()
    idx = nC*nR*ind + nR*np.arange(nR)[:,None] + np.arange(nC)
    return np.take(arr, idx)

edit (2018-10-22): This only works on evenly spaced rasters. Jean-Francois Bourdon pointed out, that it works on rasters with uneven dimensions by changing the second to last line. Thanks for the tip Jean-Francois!

idx = nC*nR*ind + nC*np.arange(nR)[:,None] + np.arange(nC)

edit (2018-11-08): Sam Bowers contacted me with a further suggestion on improving the function. I’m fascinated that this blog post is still gathering attention. Thanks for the suggestion Sam!

idx = nC*nR*ind + np.arange(nC*nR).reshape((nC,nR))

edit (2019-11-13): Ben Sugerman porposed an extension to the code to support ‘nearest neighbour’, ‘lowest’ and ‘highest’ interpolation options present in np.percentile().

if interpolation=='nearest':
    f_arr = np.around(k_arr).astype(np.int32)
    quant_arr=_zvalue_from_index(arr=arr, ind=f_arr) 
elif interpolation=='lowest':
    f_arr = np.floor(k_arr).astype(np.int32)
    quant_arr=_zvalue_from_index(arr=arr, ind=f_arr) 
elif interpolation=='highest':
    f_arr = np.ceiling(k_arr).astype(np.int32)
    quant_arr=_zvalue_from_index(arr=arr, ind=f_arr) 

Result and speed comparison

Let’s see if our new functions produces the same results as np.nanpercentile()

input_arr = np.array(test_arr, copy=True)
old_func = np.nanpercentile(input_arr, q=range(0,100), axis=0)
new_func = nan_percentile(input_arr, q=range(0,100))

np.allclose(new_func, old_func)
True

So far so good. Let’s see what we achieved in terms of speed.

input_arr = np.array(test_arr, copy=True)
%timeit np.nanpercentile(input_arr, q=[10,25,50,75,90], axis=0)
1 loops, best of 3: 603 ms per loop

input_arr = np.array(test_arr, copy=True)
%timeit nan_percentile(test_arr, q=[10,25,50,75,90])
100 loops, best of 3: 3.81 ms per loop

The new function is roughly 160x faster than np.nanpercentile(), so it was time well spent looking for a faster way.

Fast linear 1D interpolation with numba

Mon, 06 Jul 2015 00:00:00 +0000

I am currently doing time-series analysis on MODIS derived vegetation index data. In order to get a reliable signal from the data outliers need to be removed and the resulting gaps interpolated/filled before further filtering/smoothing of the signal. The time-series for one tile, covering 10° by 10°, spans roughly 14 years with 46 images per year. Each image weighs in at around 70-100 Mb. If you are processing, say, Africa you are looking at roughly 2.3 Terrabyte of input data. Interpolation of such massive amounts of data begs teh question - What is the fastest way to do it?

The Question: What is the fastest way to interpolate such a massive dataset?

Each time-series needs to be interpolated in the time domain. Since every tile consists of 4800 by 4800 pixels this means the task is to interpolate 23040000 1D numpy arrays containing 644 evenly spaced data points.

My first attempts at this focussed on scipy.interp1d and the Pandas wrapper for it.

Unfortunately they turned out to be too slow to be feasible, which lead to me asking for help on StackOverflow.

Example dataset

The example dataset reflects the input data rather well. Random Integers from -10000 to 10000 are like the NDVI dataset. There are areas where the complete z-axis is NaN (for example over water in the original data) and z-axis where only some values are NaN, just like after outlier removal along the time domain. There is high correlation along the time/z-axis. There might be some correlation in the x,y dimensions as well (Toplers Law) but this should not be employed for interpolation.

 # import necessary libraries
 import numpy as np
 import pandas as pd
 import matplotlib.pyplot as plt
 %matplotlib inline

# create example data, original is (644, 4800, 4800)
test_arr = np.random.randint(low=-10000, high=10000, size=(92, 480, 480))
test_arr[1:90:7, :, :] = -32768  # NaN fill value in original data
test_arr[2,:,:] = -32768
test_arr[:, 1:479:6, 1:479:8] = -32768

# show the example time-series at location (:,3,4)
plt.plot(test_arr[:,3,4])

Scipy.interp1d interpolation

Uitilizes scipy.signal.interp1d wrapper in pandas. This has the advantage of interpolating over a consecutive number of NaN up to a given limit with an interpolation method of choice. It is very convenient but it is rather slow.

def interpolate_nan(arr, method="linear", limit=3):
    """return array interpolated along time-axis to fill missing values"""
    result = np.zeros_like(arr, dtype=np.int16)

    for i in range(arr.shape[1]):
        # slice along y axis, interpolate with pandas wrapper to interp1d
        line_stack = pd.DataFrame(data=arr[:,i,:], dtype=np.float32)
        line_stack.replace(to_replace=-37268, value=np.NaN, inplace=True)
        line_stack.interpolate(method=method, axis=0, inplace=True, limit=limit)
        line_stack.replace(to_replace=np.NaN, value=-37268, inplace=True)
        result[:, i, :] = line_stack.values.astype(np.int16)
    return result

While this is very convenient it is way too slow for my purposes. Interpolation of my input data would require upwards of 3 weeks. I started looking for ways to speed this up. Since numpy has no fast 1D interpolation function and writing C code or learn Cython would also cost me quite some time I turned towards numba.

1D interpolation with numba

The idea is to loop through all 644x4800x4800 pixels and replace it with the mean of it’s neighbours in the z-axis. This kind of loop would be horribly slow in pure Python. Numba compiles this function once and thus speeds up the loop drastically.

This is done with the @jit decorator before the function. This function can also be nested into other functions as long as each one uses the decorator.

Numba is only faster than Python if it is not run in object mode. The standard behaviour is to fall back into object mode if the function can’t be compiled to low level code. The nopython=True argument supresses this behaviour and returns an exception if the code can’t be compiled. Functions have to be written in basic syntax with standard Python operations and boundary conditions have to be explicitly implemented.

This is less convenient and functional but pretty fast.

from numba import jit

@jit(nopython=True)
def interpolate_numba(arr, no_data=-32768):
    """return array interpolated along time-axis to fill missing values"""
    result = np.zeros_like(arr, dtype=np.int16)

    for x in range(arr.shape[2]):
        # slice along x axis
        for y in range(arr.shape[1]):
            # slice along y axis
            for z in range(arr.shape[0]):
                value = arr[z,y,x]
                if z == 0:  # don't interpolate first value
                    new_value = value
                elif z == len(arr[:,0,0])-1:  # don't interpolate last value
                    new_value = value

                elif value == no_data:  # interpolate

                    left = arr[z-1,y,x]
                    right = arr[z+1,y,x]
                    # look for valid neighbours
                    if left != no_data and right != no_data:  # left and right are valid
                        new_value = (left + right) / 2

                    elif left == no_data and z == 1:  # boundary condition left
                        new_value = value
                    elif right == no_data and z == len(arr[:,0,0])-2:  # boundary condition right
                        new_value = value

                    elif left == no_data and right != no_data:  # take second neighbour to the left
                        more_left = arr[z-2,y,x]
                        if more_left == no_data:
                            new_value = value
                        else:
                            new_value = (more_left + right) / 2

                    elif left != no_data and right == no_data:  # take second neighbour to the right
                        more_right = arr[z+2,y,x]
                        if more_right == no_data:
                            new_value = value
                        else:
                            new_value = (more_right + left) / 2

                    elif left == no_data and right == no_data:  # take second neighbour on both sides
                        more_left = arr[z-2,y,x]
                        more_right = arr[z+2,y,x]
                        if more_left != no_data and more_right != no_data:
                            new_value = (more_left + more_right) / 2
                        else:
                            new_value = value
                    else:
                        new_value = value
                else:
                    new_value = value
                result[z,y,x] = int(new_value)
    return result

Comparing performance

Testing the performance of both functions on the example dataset shows that the numba function is more than 20 times faster. While it is less convenient than SciPy’s function it is easy to write a function and use numbas LLVM magic to reach speeds close to native C Code without the hassle of having to actually learn C.

%timeit interpolate_nan(test_arr)
1 loops, best of 3: 11 s per loop

%timeit interpolate_numba(test_arr)
1 loops, best of 3: 558 ms per loop