Data Import and Export

See the module pypsa.io.

PyPSA is intended to be data format agnostic, but given the reliance internally on pandas DataFrames, it is natural to use comma-separated-variable (CSV) files.

Import from folder of CSV files

Create a folder with CSVs for each component type (e.g. generators.csv, storage_units.csv), then a CSV for each time-dependent variable (e.g. generators-p_max_pu.csv, loads-p_set.csv).

Then run network.import_from_csv_folder(csv_folder_name).

Network.import_from_csv_folder(csv_folder_name, encoding=None, skip_time=False)

Import network data from CSVs in a folder.

The CSVs must follow the standard form, see pypsa/examples.

Parameters
  • csv_folder_name (string) – Name of folder

  • encoding (str, default None) – Encoding to use for UTF when reading (ex. ‘utf-8’). List of Python standard encodings

  • skip_time (bool, default False) – Skip reading in time dependent attributes

Examples

>>> network.import_from_csv_folder(csv_folder_name)

Note

Note that is is NOT necessary to add every single column, only those where values differ from the defaults listed in Components. All empty values/columns are filled with the defaults.

Export to folder of CSV files

The network can be exported as a folder of csv files with network.export_to_csv_folder(csv_folder_name).

Network.export_to_csv_folder(csv_folder_name, encoding=None, export_standard_types=False)

Export network and components to a folder of CSVs.

Both static and series attributes of all components are exported, but only if they have non-default values.

If csv_folder_name does not already exist, it is created.

Static attributes are exported in one CSV file per component, e.g. generators.csv.

Series attributes are exported in one CSV file per component per attribute, e.g. generators-p_set.csv.

Parameters
  • csv_folder_name (string) – Name of folder to which to export.

  • encoding (str, default None) –

    Encoding to use for UTF when reading (ex. ‘utf-8’). List of Python standard encodings

  • export_standard_types (boolean, default False) – If True, then standard types are exported too (upon reimporting you should then set “ignore_standard_types” when initialising the network).

Examples

>>> network.export_to_csv_folder(csv_folder_name)

Adding and removing components one-by-one

Networks can also be built step-by-step for each component by calling

Network.add(class_name, name, **kwargs)

Add a single component to the network.

Adds it to component DataFrame.

Any attributes which are not specified will be given the default value from Components.

This method is slow for many components; instead use madd or import_components_from_dataframe (see below).

Parameters
  • class_name (string) – Component class name in [“Bus”,”Generator”,”Load”,”StorageUnit”,”Store”,”ShuntImpedance”,”Line”,”Transformer”,”Link”]

  • name (string) – Component name

  • kwargs – Component attributes, e.g. x=0.1, length=123

Examples

>>> network.add("Bus","my_bus_0")
>>> network.add("Bus","my_bus_1",v_nom=380)
>>> network.add("Line","my_line_name",bus0="my_bus_0",bus1="my_bus_1",length=34,r=2,x=4)

Likewise, components can also be removed with

Network.remove(class_name, name)

Removes a single component from the network.

Removes it from component DataFrames.

Parameters
  • class_name (string) – Component class name

  • name (string) – Component name

Examples

>>> network.remove("Line","my_line 12345")

Adding and removing multiple components

Multiple components can be added by calling

Network.madd(class_name, names, suffix='', **kwargs)

Add multiple components to the network, along with their attributes.

Make sure when adding static attributes as pandas Series that they are indexed by names. Make sure when adding time-varying attributes as pandas DataFrames that their index is a superset of network.snapshots and their columns are a subset of names.

Any attributes which are not specified will be given the default value from Components.

Parameters
  • class_name (string) – Component class name in [“Bus”,”Generator”,”Load”,”StorageUnit”,”Store”,”ShuntImpedance”,”Line”,”Transformer”,”Link”]

  • names (list-like or pandas.Index) – Component names

  • suffix (string, default '') – All components are named after names with this added suffix. It is assumed that all Series and DataFrames are indexed by the original names.

  • kwargs – Component attributes, e.g. x=[0.1,0.2], can be list, pandas.Series of pandas.DataFrame for time-varying

Returns

new_names – Names of new components (including suffix)

Return type

pandas.index

Examples

Short Example:

>>> network.madd("Load", ["load 1", "load 2"],
...        bus=["1","2"],
...        p_set=np.random.rand(len(network.snapshots),2))

Long Example:

>>> import pandas as pd, numpy as np
>>> buses = range(13)
>>> snapshots = range(7)
>>> n = pypsa.Network()
>>> n.set_snapshots(snapshots)
>>> n.madd("Bus", buses)
>>> # add load as numpy array
>>> n.madd("Load",
...        n.buses.index + " load",
...        bus=buses,
...        p_set=np.random.rand(len(snapshots),len(buses)))
>>> # add wind availability as pandas DataFrame
>>> wind = pd.DataFrame(np.random.rand(len(snapshots),len(buses)),
...        index=n.snapshots,
...        columns=buses)
>>> #use a suffix to avoid boilerplate to rename everything
>>> n.madd("Generator",
...        buses,
...        suffix=' wind',
...        bus=buses,
...        p_nom_extendable=True,
...        capital_cost=1e5,
...        p_max_pu=wind)

Multiple components can be removed by calling

Network.mremove(class_name, names)

Removes multiple components from the network.

Removes them from component DataFrames.

Parameters
  • class_name (string) – Component class name

  • name (list-like) – Component names

Examples

>>> network.mremove("Line", ["line x", "line y"])

Adding components using pandas DataFrames

To add multiple components whose static attributes are given in a pandas DataFrame, use

Network.import_components_from_dataframe(dataframe, cls_name)

Import components from a pandas DataFrame.

If columns are missing then defaults are used.

If extra columns are added, these are left in the resulting component dataframe.

Parameters
  • dataframe (pandas.DataFrame) – A DataFrame whose index is the names of the components and whose columns are the non-default attributes.

  • cls_name (string) – Name of class of component, e.g. "Line","Bus","Generator", "StorageUnit"

Examples

>>> import pandas as pd
>>> buses = ['Berlin', 'Frankfurt', 'Munich', 'Hamburg']
>>> network.import_components_from_dataframe(
        pd.DataFrame({"v_nom" : 380, "control" : 'PV'},
                    index=buses),
                    "Bus")
>>> network.import_components_from_dataframe(
        pd.DataFrame({"carrier" : "solar", "bus" : buses, "p_nom_extendable" : True},
                    index=[b+" PV" for b in buses]),
                    "Generator")

To import time-varying information use

Network.import_series_from_dataframe(dataframe, cls_name, attr)

Import time series from a pandas DataFrame.

Parameters
  • dataframe (pandas.DataFrame) – A DataFrame whose index is network.snapshots and whose columns are a subset of the relevant components.

  • cls_name (string) – Name of class of component

  • attr (string) – Name of time-varying series attribute

Examples

>>> import numpy as np
>>> network.set_snapshots(range(10))
>>> network.import_series_from_dataframe(
        pd.DataFrame(np.random.rand(10,4),
            columns=network.generators.index,
                        index=range(10)),
                    "Generator",
                    "p_max_pu")

Export to netCDF

netCDF files take up less space than CSV files and are faster to load.

netCDF is also preferred over HDF5 because netCDF is structured more cleanly, is easier to use from other programming languages, can limit float precision to save space and supports lazy loading.

To export network and components to a netCDF file run network.export_to_netcdf('file.nc').

Network.export_to_netcdf(path=None, export_standard_types=False, least_significant_digit=None)

Export network and components to a netCDF file.

Both static and series attributes of components are exported, but only if they have non-default values.

If path does not already exist, it is created.

If no path is passed, no file is exported, but the xarray.Dataset is still returned.

Be aware that this cannot export boolean attributes on the Network class, e.g. network.my_bool = False is not supported by netCDF.

Parameters
  • path (string|None) – Name of netCDF file to which to export (if it exists, it is overwritten); if None is passed, no file is exported.

  • export_standard_types (boolean, default False) – If True, then standard types are exported too (upon reimporting you should then set “ignore_standard_types” when initialising the network).

  • least_significant_digit – This is passed to the netCDF exporter, but currently makes no difference to file size or float accuracy. We’re working on improving this…

Returns

ds

Return type

xarray.Dataset

Examples

>>> network.export_to_netcdf("my_file.nc")

Import from netCDF

To import network data from netCDF file run network.import_from_netcdf(file.nc).

Network.import_from_netcdf(path, skip_time=False)

Import network data from netCDF file or xarray Dataset at path.

Parameters
  • path (string|xr.Dataset) – Path to netCDF dataset or instance of xarray Dataset

  • skip_time (bool, default False) – Skip reading in time dependent attributes

Export to HDF5

Note

netCDF is preferred over HDF5 because netCDF is structured more cleanly, is easier to use from other programming languages, can limit float precision to save space and supports lazy loading.

To export network and components to an HDF store run network.export_to_hdf5(path).

Network.export_to_hdf5(path, export_standard_types=False, **kwargs)

Export network and components to an HDF store.

Both static and series attributes of components are exported, but only if they have non-default values.

If path does not already exist, it is created.

Parameters
  • path (string) – Name of hdf5 file to which to export (if it exists, it is overwritten)

  • export_standard_types (boolean, default False) – If True, then standard types are exported too (upon reimporting you should then set “ignore_standard_types” when initialising the network).

  • **kwargs – Extra arguments for pd.HDFStore to specify f.i. compression (default: complevel=4)

Examples

>>> network.export_to_hdf5(filename)

Import from HDF5

To import network data from HDF5 store at path run network.import_from_hdf5(path).

Network.import_from_hdf5(path, skip_time=False)

Import network data from HDF5 store at path.

Parameters
  • path (string) – Name of HDF5 store

  • skip_time (bool, default False) – Skip reading in time dependent attributes

Import from Pypower

PyPSA supports import from Pypower’s ppc dictionary/numpy.array format version 2.

Network.import_from_pypower_ppc(ppc, overwrite_zero_s_nom=None)

Import network from PYPOWER PPC dictionary format version 2.

Converts all baseMVA to base power of 1 MVA.

For the meaning of the pypower indices, see also pypower/idx_*.

Parameters
  • ppc (PYPOWER PPC dict) –

  • overwrite_zero_s_nom (Float or None, default None) –

Examples

>>> from pypower.api import case30
>>> ppc = case30()
>>> network.import_from_pypower_ppc(ppc)

Import from Pandapower

Warning

Importing from pandapower is still in beta; not all pandapower data is supported.

PyPSA supports import from pandapower.

Network.import_from_pandapower_net(net, extra_line_data=False)

Import network from pandapower net.

Importing from pandapower is still in beta; not all pandapower data is supported.

Unsupported features include: - three-winding transformers - switches - in_service status, - shunt impedances, and - tap positions of transformers.”

Parameters
  • net (pandapower network) –

  • extra_line_data (boolean, default: False) – if True, the line data for all parameters is imported instead of only the type

Examples

>>> network.import_from_pandapower_net(net)
OR
>>> import pandapower as pp
>>> import pandapower.networks as pn
>>> net = pn.create_cigre_network_mv(with_der='all')
>>> pp.runpp(net)
>>> network.import_from_pandapower_net(net, extra_line_data=True)