ECMWF Open Data#

This tutorial demonstrates how to use the ECMWF Real-Time Open Data. This data is freely available from ECMWF in GRIB2 format (👀 Read more).

“ The data that are becoming available are based on a range of high-resolution forecasts (HRES – 9 km horizontal resolution) and ensemble forecasts (ENS – 18 km horizontal resolution). They will be made accessible at a resolution of 0.4 x 0.4 degrees”.

- Media Statement

Products

`product=`	Product Description	Available model runs
`"oper"`	operational high-resolution forecast, atmospheric fields	`00z`, `12z`,
`"wave"`	wave forecasts	`00z`, `12z`,
`"scda"`	short cut-off high-resolution forecast, atmospheric fields (also known a high-frequency products)”,	`06z`, `18z`
`"scwv"`	short cut-off high-resolution forecast, ocean wave fields (also known a high-frequency products)”,	`06z`, `18z`
`"enfo"`	ensemble forecast, atmospheric fields	`00z`, `06z`, `12z`, `18z`
`"waef"`	ensemble forecast, ocean wave fields,	`00z`, `06z`, `12z`, `18z`
`"mmsf"`	multi-model seasonal forecasts fields from the ECMWF model only.	?

Model Initialization

00z, 06z, 12z, 18z. Not all products are available every hour.

[1]:

from herbie import Herbie

import cartopy.crs as ccrs
import matplotlib.pyplot as plt
import numpy as np

from paint.standard2 import cm_tmp, cm_wind, cm_wave_height
from toolbox import EasyMap, pc

[2]:

H = Herbie("2023-02-01", model="ecmwf", product="oper", fxx=12)

✅ Found ┊ model=ecmwf ┊ product=oper ┊ 2023-Feb-01 00:00 UTC F12 ┊ GRIB2 @ azure ┊ IDX @ azure

[4]:

H = Herbie("2023-02-01 06", model="ecmwf", product="oper", fxx=12)
H.grib

✅ Found ┊ model=ecmwf ┊ product=oper ┊ 2023-Feb-01 06:00 UTC F12 ┊ GRIB2 @ azure-scda ┊ IDX @ azure-scda

[4]:

'https://ai4edataeuwest.blob.core.windows.net/ecmwf/20230201/06z/0p4-beta/scda/20230201060000-12h-scda-fc.grib2'

[20]:

H = Herbie("2023-1-23", model="ecmwf", product="oper", fxx=12, priority="azure")

✅ Found ┊ model=ecmwf ┊ product=oper ┊ 2023-Jan-23 00:00 UTC F12 ┊ GRIB2 @ azure ┊ IDX @ azure

[4]:

H.xarray(":2t:")

👨🏻‍🏭 Created directory: [/home/blaylock/data/ecmwf/20230201]
curl -s --range 16444658-17053704 "https://ai4edataeuwest.blob.core.windows.net/ecmwf/20230201/00z/0p4-beta/oper/20230201000000-12h-oper-fc.grib2" > "/home/blaylock/data/ecmwf/20230201/subset_d712f340__20230201000000-12h-oper-fc.grib2"

[ ]:

H = Herbie("2022-01-26", model="ecmwf", product="oper", fxx=12)

Unique Index Files#

The ECMWF index files are different than the wgrib2-style index files, so pay close attention to how you should select the field you want.

[3]:

# Show the searchString_help
print(H.searchString_help)


Use regular expression to search for lines in the index file.
Here are some examples you can use for the ecCodes-style `searchString`

Look at the ECMWF GRIB Parameter Database
https://apps.ecmwf.int/codes/grib/param-db

======================== ==============================================
searchString (oper/enso) Messages that will be downloaded
======================== ==============================================
":2t:"                   2-m temperature
":10u:"                  10-m u wind vector
":10v:"                  10-m v wind vector
":10(u|v):               **10m u and 10m v wind**
":d:"                    Divergence (all levels)
":gh:"                   geopotential height (all levels)
":gh:500"                geopotential height only at 500 hPa
":st:"                   soil temperature
":tp:"                   total precipitation
":msl:"                  mean sea level pressure
":q:"                    Specific Humidity
":r:"                    relative humidity
":ro:"                   Runn-off
":skt:"                  skin temperature
":sp:"                   surface pressure
":t:"                    temperature
":tcwv:"                 Total column vertically integrated water vapor
":vo:"                   Relative vorticity
":v:"                    v wind vector
":u:"                    u wind vector
":(t|u|v|r):"            Temp, u/v wind, RH (all levels)
":500:"                  All variables on the 500 hPa level

======================== ==============================================
searchString (wave/waef) Messages that will be downloaded
======================== ==============================================
":swh:"                  Significant height of wind waves + swell
":mwp:"                  Mean wave period
":mwd:"                  Mean wave direction
":pp1d:"                 Peak wave period
":mp2:"                  Mean zero-crossing wave period

If you need help with regular expression, search the web or look at
this cheatsheet: https://www.petefreitag.com/cheatsheets/regex/.

[4]:

# Pay attention to the "search_this" column
H.inventory()

[4]:

	grib_message	start_byte	end_byte	range	reference_time	valid_time	step	param	levelist	levtype	number	domain	expver	class	type	stream	search_this
0	1	0	609046	0-609046	2022-01-26	2022-01-26 12:00:00	0 days 12:00:00	skt	NaN	sfc	NaN	g	0001	od	fc	oper	:skt:sfc:g:0001:od:fc:oper
1	2	609046	1218092	609046-1218092	2022-01-26	2022-01-26 12:00:00	0 days 12:00:00	st	NaN	sfc	NaN	g	0001	od	fc	oper	:st:sfc:g:0001:od:fc:oper
2	3	1218092	1827138	1218092-1827138	2022-01-26	2022-01-26 12:00:00	0 days 12:00:00	gh	1000	pl	NaN	g	0001	od	fc	oper	:gh:1000:pl:g:0001:od:fc:oper
3	4	1827138	2436184	1827138-2436184	2022-01-26	2022-01-26 12:00:00	0 days 12:00:00	gh	925	pl	NaN	g	0001	od	fc	oper	:gh:925:pl:g:0001:od:fc:oper
4	5	2436184	2842280	2436184-2842280	2022-01-26	2022-01-26 12:00:00	0 days 12:00:00	r	300	pl	NaN	g	0001	od	fc	oper	:r:300:pl:g:0001:od:fc:oper
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
77	78	48114266	48926262	48114266-48926262	2022-01-26	2022-01-26 12:00:00	0 days 12:00:00	d	250	pl	NaN	g	0001	od	fc	oper	:d:250:pl:g:0001:od:fc:oper
78	79	48926262	49738258	48926262-49738258	2022-01-26	2022-01-26 12:00:00	0 days 12:00:00	vo	250	pl	NaN	g	0001	od	fc	oper	:vo:250:pl:g:0001:od:fc:oper
79	80	49738258	50550254	49738258-50550254	2022-01-26	2022-01-26 12:00:00	0 days 12:00:00	vo	50	pl	NaN	g	0001	od	fc	oper	:vo:50:pl:g:0001:od:fc:oper
80	81	50550254	51159324	50550254-51159324	2022-01-26	2022-01-26 12:00:00	0 days 12:00:00	ro	NaN	sfc	NaN	g	0001	od	fc	oper	:ro:sfc:g:0001:od:fc:oper
81	82	51159324	51971320	51159324-51971320	2022-01-26	2022-01-26 12:00:00	0 days 12:00:00	d	50	pl	NaN	g	0001	od	fc	oper	:d:50:pl:g:0001:od:fc:oper

82 rows × 17 columns

Ok, now that we have some understanding of the index file, we can read the 2-m temperature data.

[6]:

ds = H.xarray(":2t:")
ds

[7]:

ax = EasyMap("50m", crs=ds.herbie.crs, figsize=[10, 10]).STATES().BORDERS().ax
p = ax.pcolormesh(
    ds.longitude, ds.latitude, ds.t2m, transform=pc, **cm_tmp(units="K").cmap_kwargs
)
plt.colorbar(
    p, ax=ax, orientation="horizontal", pad=0.05, **cm_tmp(units="K").cbar_kwargs
)

ax.set_title(
    f"{ds.model.upper()}: {H.product_description}\nValid: {ds.valid_time.dt.strftime('%H:%M UTC %d %b %Y').item()}",
    loc="left",
)
ax.set_title(ds.t2m.GRIB_name, loc="right")

[7]:

Text(1.0, 1.0, '2 metre temperature')

c:\Users\blaylock\Miniconda3\envs\herbie-dev\lib\site-packages\cartopy\io\__init__.py:241: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/50m_cultural/ne_50m_admin_0_boundary_lines_land.zip
  warnings.warn(f'Downloading: {url}', DownloadWarning)

../../_images/user_guide__model_notebooks_ecmwf_12_2.png

Now the same, but for wind.

[8]:

H = Herbie("2022-01-26", model="ecmwf", product="oper")

# Get u and v wind component
ds = H.xarray(":10(u|v):")

# Compute the wind speed
ds["spd"] = np.sqrt(ds["u10"] ** 2 + ds["v10"] ** 2)

# without too much thought, just quickly copy attributes
ds["spd"].attrs = ds["u10"].attrs.copy()
ds["spd"].attrs["standard_name"] = "wind_speed"
ds["spd"].attrs["long_name"] = "10 m wind speed"
ds["spd"].attrs["GRIB_name"] = "10 m Wind Speed"

ds

✅ Found ┊ model=ecmwf ┊ product=oper ┊ 2022-Jan-26 00:00 UTC F00 ┊ GRIB2 @ azure ┊ IDX @ azure

C:\Users\blaylock\_GITHUB\Herbie\herbie\archive.py:722: UserWarning: This pattern is interpreted as a regular expression, and has match groups. To actually get the groups, use str.extract.
  logic = df.search_this.str.contains(searchString)
C:\Users\blaylock\_GITHUB\Herbie\herbie\archive.py:1041: UserWarning: sorry, on windows I couldn't remove the file.
  warnings.warn("sorry, on windows I couldn't remove the file.")

[9]:

ax = EasyMap("50m", crs=ds.herbie.crs, figsize=[10, 10]).STATES().BORDERS().ax
p = ax.pcolormesh(
    ds.longitude, ds.latitude, ds.spd, transform=pc, **cm_wind().cmap_kwargs
)
plt.colorbar(p, ax=ax, orientation="horizontal", pad=0.05, **cm_wind().cbar_kwargs)

ax.set_title(
    f"{ds.model.upper()}: {H.product_description}\nValid: {ds.valid_time.dt.strftime('%H:%M UTC %d %b %Y').item()}",
    loc="left",
)
ax.set_title(ds.spd.GRIB_name, loc="right")

[9]:

Text(1.0, 1.0, '10 m Wind Speed')

../../_images/user_guide__model_notebooks_ecmwf_15_1.png

Now lets get the humidity and geopotential height at 500 hPa

[10]:

ds = H.xarray(":(q|gh):500")
ds

C:\Users\blaylock\_GITHUB\Herbie\herbie\archive.py:722: UserWarning: This pattern is interpreted as a regular expression, and has match groups. To actually get the groups, use str.extract.
  logic = df.search_this.str.contains(searchString)
C:\Users\blaylock\_GITHUB\Herbie\herbie\archive.py:1041: UserWarning: sorry, on windows I couldn't remove the file.
  warnings.warn("sorry, on windows I couldn't remove the file.")

[11]:

ax = EasyMap("50m", crs=ccrs.Robinson(), figsize=[10, 10]).STATES().BORDERS().ax

# Color shade by specific humidity
p = ax.pcolormesh(ds.longitude, ds.latitude, ds.q, transform=pc, cmap="Greens")

plt.colorbar(
    p,
    ax=ax,
    orientation="horizontal",
    pad=0.05,
    label=f"{ds.q.GRIB_name} ({ds.q.units})",
)

# Contours for geopotential height
ax.contour(
    ds.longitude,
    ds.latitude,
    ds.gh,
    transform=pc,
    colors="k",
    linewidths=0.5,
    levels=range(0, 10_000, 60 * 2),
)


ax.set_title(
    f"{ds.model.upper()}: {H.product_description}\nValid: {ds.valid_time.dt.strftime('%H:%M UTC %d %b %Y').item()}",
    loc="left",
)
ax.set_title(
    f"{ds.isobaricInhPa.item()} {ds.isobaricInhPa.units}\n{ds.q.GRIB_name}/{ds.gh.GRIB_name}",
    loc="right",
)

[11]:

Text(1.0, 1.0, '500.0 hPa\nSpecific humidity/Geopotential height')

../../_images/user_guide__model_notebooks_ecmwf_18_1.png

Ensemble Forecast Products#

[ ]:

H = Herbie("2022-01-26 00:00", model="ecmwf", product="enfo")
ds = H.xarray(":2t:")
ds

[ ]:

# Dataset with all 50 members
ds[0]

[ ]:

# This Dataset is of the mean of all the members, right?
ds[1]

[ ]:

H.idx

Ensemble Wave Products#

[ ]:

H = Herbie("2022-01-26 00:00", model="ecmwf", product="waef")
ds = H.xarray(None)
ds

[ ]:

len(ds)

[ ]:

ds[0]

[ ]:

ds[1]

[ ]:

H = Herbie("2022-01-26", model="ecmwf", product="enfo")
H.inventory()

[ ]:

Here is another examle, just for fun

[ ]:

H = Herbie("2022-01-26", model="ecmwf", product="oper", fxx=12)

[ ]:

# Download the full grib2 file
H.download()

[ ]:

# Download just the 10-m u and v winds
H.download(searchString=":10(u|v):")

[ ]:

# Retrieve the 500 hPa temperature as an xarray.Dataset
ds = H.xarray(searchString=":t:500:")

[ ]:

ds

[ ]:

ds.t.plot()

Attribution

Source www.ecmwf.int
Licence Statement: This data is published under a Creative Commons Attribution 4.0 International (CC BY 4.0). https://creativecommons.org/licenses/by/4.0/
Disclaimer: ECMWF does not accept any liability whatsoever for any error or omission in the data, their availability, or for any loss or damage arising from their use.

ECMWF Open Data#

Unique Index Files#

ECMWF Wave Output#

Ensemble Forecast Products#

Ensemble Wave Products#