The objective of this repository is to share with an MIT license the visualization tools used with public data and developed by the Wholesale Markets Surveillance Directorate (DSMG) of the Regulatory Commission of Energy (CRE). It can be used by final users such as developers and energy analysts.
To obtain the proposed interactive visualizations, the user should :
- install the package,
- download the input data,
- run one of the scripts with ipython and her own choice of parameters.
All suggestions are welcome at opensource[at]cre.fr.
It can be installed with :
cd ~/Downloads
git clone https://github.com/cre-dev/pub-data-visualization.git
cd pub-data-visualization
python3 -m venv venv
source venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -e .
The installation can then be tested with one of the following :
python3 scripts/weather/main_curve.py
or
python3 scripts/load/main_forecasting_error.py
The scripts should terminate without any error. They will create 3 folders in the home directory :
~/
├── _energy_plots/ # for the plots
├── _energy_public_data/ # for the raw data
├── _energy_tmp_data/ # for the transformed data
However, ipython should then be preferred for interactive plots.
The data used for the visualizations proposed in this repository come from different public data sources. only the data from eCO2mix and Météo-France are downloaded automatically. Data from RTE and ENTSO-E should be downloaded manually.
Data about the supply and demand equilibrium and provided by Réseau de Transport d’Electricité (RTE) through eCO2mix allow to illustrate the production and the consumption on the French electricity network. They can be downloaded automatically. No account is necessary.
The European Network of Transmission System Operators for Electricity (ENTSO-E) publishes fundamental data on its transparency platform. The source files used for the visualizations in this repository currently have to be downloaded manually with the SFTP share. An account is necessary.
As the French national meteorological service, Météo-France provides observation data extracted from the Global Telecommunication System (GTS) of the World Meteorological Organization (WMO). The data can be downloaded automatically. No account is necessary.
RTE publishes fundamental data about the French electricity transmission system. The files currently have to be downloaded manually on the platform RTE services portal. An account is necessary.
The data have to be stored as follows :
~/_energy_public_data/
├── 11_ENTSOE/
│ ├── ActualGenerationOutputPerGenerationUnit_16.1.A/
│ │ ├── 2021_01_ActualGenerationOutputPerGenerationUnit_16.1.A.csv
│ │ ├── …
│ ├── ActualTotalLoad_6.1.A/
│ │ ├── 2021_01_ActualTotalLoad_6.1.A.csv
│ │ ├── …
│ ├── DayAheadPrices_12.1.D/
│ │ ├── 2021_01_DayAheadPrices_12.1.D.csv
│ │ ├── …
│ ├── Outages/
│ │ ├── UnavailabilityOfGenerationUnits_15.1.A_B/
│ │ │ ├── 2021_01_UnavailabilityOfGenerationUnits_15.1.A_B.csv
│ │ │ ├── …
│ │ ├── UnavailabilityOfProductionUnits_15.1.C_D/
│ │ │ ├── 2021_01_UnavailabilityOfProductionUnits_15.1.C_D.csv
│ │ │ ├── …
├──20_MeteoFrance/
│ ├── synop/
│ │ ├── postesSynop.csv
│ │ ├── synop.201001.csv
│ │ ├── …
├──24_RTE/
│ ├── Centrales_production_reference/
│ │ ├── Centrales_production_reference.xls
│ ├── DonneesIndisponibilitesProduction/
│ │ ├── DonneesIndisponibilitesProduction_2010.xls
│ │ ├── …
│ ├── eCO2mix_RTE/
│ │ ├── eCO2mix_RTE_Annuel-Definitif_2012.xls
│ │ ├── …
│ ├── ProductionGroupe/
│ │ ├── ProductionGroupe_2012/
│ │ │ ├── ProductionGroupe_2012-semestre1.xls
│ │ │ ├── …
│ │ ├── …
In this repository, we propose a set of modules that read, format, transform and plot the input data from different public sources. We also provide ready-to-run visualization scripts as illustrated below. The parameters therein can be modified by the user.
This figure, that represents the fixing prices of the day-ahead auctions, is obtained by running scripts/indices/main_price.py.
The data, provided by ENTSO-E, currently have to be downloaded manually.
This figure, obtained by running scripts/load/main_power.py, is a mere representation of the load curve.
Data from eCO2mix and ENTSO-E can serve as inputs.
This figure represents the national load forecasting error and is obtained by running scripts/load/main_forecasting_error.py.
The data are provided by eCO2mix.
This figure is obtained by running scripts/outages/main_animated_availability.py.
The data have to be downloaded manually from ENTSO-E or RTE platforms.
This figure is obtained by running scripts/outages/main_evolution_mean_availability.py.
The data have to be downloaded manually from ENTSO-E or RTE platforms.
This figure is obtained by running scripts/outages/main_expected_program.py.
The data have to be downloaded manually from ENTSO-E or RTE platforms.
This figure is obtained by running scripts/outages/main_incremental_programs.py.
The data have to be downloaded manually from ENTSO-E or RTE platforms.
This figure is obtained by running scripts/outages/main_regression_delays.py.
The data have to be downloaded manually from ENTSO-E or RTE platforms.
It displays a linear fit between the initially announced and the finally observed length of the outages.
The coefficient being obtained with the minimization of a squared error, outliers have a significant effect.
This figure is obtained by running scripts/production/main_power.py.
It can be used with data provided by eCO2mix, ENTSO-E or RTE.
This figure is obtained by running scripts/weather/main_curve.py.
The data, provided by Météo-France, are downloaded automatically.
This figure is obtained by running scripts/weather/main_distribution.py.
The data, provided by Météo-France, are downloaded automatically.
This figure is obtained by running scripts/multiplots/main_spot_report.py.
As it mixes data from different sources, the data from ENTSO-E and RTE have to be downloaded manually.
This figure is obtained by running scripts/multiplots/main_transparent_production.py.
The data from RTE have to be downloaded manually.
This figure is obtained by running scripts/multiplots/main_scatter_price_load.py.
The data from ENTSO-E and RTE have to be downloaded manually.
This figure is obtained by running scripts/multiplots/main_scatter_price_production.py.
The data from ENTSO-E and RTE have to be downloaded manually.
This figure is obtained by running scripts/multiplots/main_scatter_price_weather.py.
The data from ENTSO-E have to be downloaded manually.
An html documentation generated with Sphinx can be found in ./doc/pub_data_visualization.html.
| Variable name | Type | Possible values | Purpose |
|---|---|---|---|
| close | bool | True; False | Close the figure after saving |
| contract_delivery_begin_year | int | 2018; … | - |
| contract_delivery_period_index | int | depends on the selected contract_product | - |
| contract_frequency | string | "M"; "Q"; … | - |
| contract_profile | string | "BASE"; "PEAK" … | - |
| data_source_auctions | string | "ENTSOE" | - |
| data_source_load | string | "eCO2mix"; "ENTSOE" | - |
| data_source_outages | string | "ENTSOE"; "RTE" | - |
| data_source_production | string | "eCO2mix"; "ENTSOE"; "RTE" | - |
| data_source_weather | string | "MétéoFrance" | - |
| delivery_begin_dt_max | None or localized pd.Timestamp | - | - |
| delivery_end_dt_min | None or localized pd.Timestamp | - | - |
| diff_init | bool | True; False | Plot the differences between pairs of dates |
| figsize | (int,int) | (8,6) | Figure size |
| folder_out | path | global_var.path_plots | Plots output |
| load_nature | string | "load forecast D-1 (GW)";"load forecast D-0 (GW)"; "load (GW)" | - |
| map_code | string or list of strings | "FR"; … or ["FR", "GB", "BE"…] | - |
| map_code_auctions | string or list of strings | "FR"; … or ["FR", "GB", "BE"…] | - |
| producer_outages | None or string | any producer that publishes | - |
| production_dt_max | None or localized pd.Timestamp | - | - |
| production_dt_min | None or localized pd.Timestamp | - | - |
| production_nature | string | "production (GW)" | - |
| production_source | None or string | "biomass"; "solar"; … | - |
| publication_dt_max | None or localized pd.Timestamp | - | - |
| publication_dt_min | None or localized pd.Timestamp | - | - |
| smoother | string or pd.Timedelta | "basic"; | For aesthetic purposes |
| unit_name | None or string | any production unit | - |
| viewpoint_dt_extrapolate | list of localized pd.Timestamp | - | Position plotted as seen from these dates |
| weather_nature | string | "observation" | - |
| weather_quantity | string | "nebulosity (%)"; '"temperature (°C)"; "wind_speed (m/s)" | - |