Data Import and Export
See the module pypsa.io
.
PyPSA is intended to be data format agnostic, but given the reliance internally on pandas DataFrames, it is natural to use comma-separated-variable (CSV) files.
Import from folder of CSV files
Create a folder with CSVs for each component type
(e.g. generators.csv, storage_units.csv
), then a CSV for each
time-dependent variable (e.g. generators-p_max_pu.csv,
loads-p_set.csv
).
Then run network.import_from_csv_folder(csv_folder_name)
.
- Network.import_from_csv_folder(csv_folder_name, encoding=None, skip_time=False)
Import network data from CSVs in a folder.
The CSVs must follow the standard form, see
pypsa/examples
.- Parameters
csv_folder_name (string) – Name of folder
encoding (str, default None) – Encoding to use for UTF when reading (ex. ‘utf-8’). List of Python standard encodings
skip_time (bool, default False) – Skip reading in time dependent attributes
Examples
>>> network.import_from_csv_folder(csv_folder_name)
Note
Note that is is NOT necessary to add every single column, only those where values differ from the defaults listed in Components. All empty values/columns are filled with the defaults.
Export to folder of CSV files
The network can be exported as a folder of csv files with network.export_to_csv_folder(csv_folder_name)
.
- Network.export_to_csv_folder(csv_folder_name, encoding=None, export_standard_types=False)
Export network and components to a folder of CSVs.
Both static and series attributes of all components are exported, but only if they have non-default values.
If
csv_folder_name
does not already exist, it is created.Static attributes are exported in one CSV file per component, e.g.
generators.csv
.Series attributes are exported in one CSV file per component per attribute, e.g.
generators-p_set.csv
.- Parameters
csv_folder_name (string) – Name of folder to which to export.
encoding (str, default None) –
Encoding to use for UTF when reading (ex. ‘utf-8’). List of Python standard encodings
export_standard_types (boolean, default False) – If True, then standard types are exported too (upon reimporting you should then set “ignore_standard_types” when initialising the network).
Examples
>>> network.export_to_csv_folder(csv_folder_name)
Adding and removing components one-by-one
Networks can also be built step-by-step for each component by calling
- Network.add(class_name, name, **kwargs)
Add a single component to the network.
Adds it to component DataFrame.
Any attributes which are not specified will be given the default value from Components.
This method is slow for many components; instead use
madd
orimport_components_from_dataframe
(see below).- Parameters
class_name (string) – Component class name in [“Bus”,”Generator”,”Load”,”StorageUnit”,”Store”,”ShuntImpedance”,”Line”,”Transformer”,”Link”]
name (string) – Component name
kwargs – Component attributes, e.g. x=0.1, length=123
Examples
>>> network.add("Bus","my_bus_0") >>> network.add("Bus","my_bus_1",v_nom=380) >>> network.add("Line","my_line_name",bus0="my_bus_0",bus1="my_bus_1",length=34,r=2,x=4)
Likewise, components can also be removed with
- Network.remove(class_name, name)
Removes a single component from the network.
Removes it from component DataFrames.
- Parameters
class_name (string) – Component class name
name (string) – Component name
Examples
>>> network.remove("Line","my_line 12345")
Adding and removing multiple components
Multiple components can be added by calling
- Network.madd(class_name, names, suffix='', **kwargs)
Add multiple components to the network, along with their attributes.
Make sure when adding static attributes as pandas Series that they are indexed by names. Make sure when adding time-varying attributes as pandas DataFrames that their index is a superset of network.snapshots and their columns are a subset of names.
Any attributes which are not specified will be given the default value from Components.
- Parameters
class_name (string) – Component class name in [“Bus”,”Generator”,”Load”,”StorageUnit”,”Store”,”ShuntImpedance”,”Line”,”Transformer”,”Link”]
names (list-like or pandas.Index) – Component names
suffix (string, default '') – All components are named after names with this added suffix. It is assumed that all Series and DataFrames are indexed by the original names.
kwargs – Component attributes, e.g. x=[0.1,0.2], can be list, pandas.Series of pandas.DataFrame for time-varying
- Returns
new_names – Names of new components (including suffix)
- Return type
pandas.index
Examples
Short Example:
>>> network.madd("Load", ["load 1", "load 2"], ... bus=["1","2"], ... p_set=np.random.rand(len(network.snapshots),2))
Long Example:
>>> import pandas as pd, numpy as np >>> buses = range(13) >>> snapshots = range(7) >>> n = pypsa.Network() >>> n.set_snapshots(snapshots) >>> n.madd("Bus", buses) >>> # add load as numpy array >>> n.madd("Load", ... n.buses.index + " load", ... bus=buses, ... p_set=np.random.rand(len(snapshots),len(buses))) >>> # add wind availability as pandas DataFrame >>> wind = pd.DataFrame(np.random.rand(len(snapshots),len(buses)), ... index=n.snapshots, ... columns=buses) >>> #use a suffix to avoid boilerplate to rename everything >>> n.madd("Generator", ... buses, ... suffix=' wind', ... bus=buses, ... p_nom_extendable=True, ... capital_cost=1e5, ... p_max_pu=wind)
Multiple components can be removed by calling
- Network.mremove(class_name, names)
Removes multiple components from the network.
Removes them from component DataFrames.
- Parameters
class_name (string) – Component class name
name (list-like) – Component names
Examples
>>> network.mremove("Line", ["line x", "line y"])
Adding components using pandas DataFrames
To add multiple components whose static attributes are given in a pandas DataFrame, use
- Network.import_components_from_dataframe(dataframe, cls_name)
Import components from a pandas DataFrame.
If columns are missing then defaults are used.
If extra columns are added, these are left in the resulting component dataframe.
- Parameters
dataframe (pandas.DataFrame) – A DataFrame whose index is the names of the components and whose columns are the non-default attributes.
cls_name (string) – Name of class of component, e.g.
"Line","Bus","Generator", "StorageUnit"
Examples
>>> import pandas as pd >>> buses = ['Berlin', 'Frankfurt', 'Munich', 'Hamburg'] >>> network.import_components_from_dataframe( pd.DataFrame({"v_nom" : 380, "control" : 'PV'}, index=buses), "Bus") >>> network.import_components_from_dataframe( pd.DataFrame({"carrier" : "solar", "bus" : buses, "p_nom_extendable" : True}, index=[b+" PV" for b in buses]), "Generator")
See also
To import time-varying information use
- Network.import_series_from_dataframe(dataframe, cls_name, attr)
Import time series from a pandas DataFrame.
- Parameters
dataframe (pandas.DataFrame) – A DataFrame whose index is
network.snapshots
and whose columns are a subset of the relevant components.cls_name (string) – Name of class of component
attr (string) – Name of time-varying series attribute
Examples
>>> import numpy as np >>> network.set_snapshots(range(10)) >>> network.import_series_from_dataframe( pd.DataFrame(np.random.rand(10,4), columns=network.generators.index, index=range(10)), "Generator", "p_max_pu")
See also
Export to netCDF
netCDF files take up less space than CSV files and are faster to load.
netCDF is also preferred over HDF5 because netCDF is structured more cleanly, is easier to use from other programming languages, can limit float precision to save space and supports lazy loading.
To export network and components to a netCDF file run
network.export_to_netcdf('file.nc')
.
- Network.export_to_netcdf(path=None, export_standard_types=False, least_significant_digit=None)
Export network and components to a netCDF file.
Both static and series attributes of components are exported, but only if they have non-default values.
If path does not already exist, it is created.
If no path is passed, no file is exported, but the xarray.Dataset is still returned.
Be aware that this cannot export boolean attributes on the Network class, e.g. network.my_bool = False is not supported by netCDF.
- Parameters
path (string|None) – Name of netCDF file to which to export (if it exists, it is overwritten); if None is passed, no file is exported.
export_standard_types (boolean, default False) – If True, then standard types are exported too (upon reimporting you should then set “ignore_standard_types” when initialising the network).
least_significant_digit – This is passed to the netCDF exporter, but currently makes no difference to file size or float accuracy. We’re working on improving this…
- Returns
ds
- Return type
xarray.Dataset
Examples
>>> network.export_to_netcdf("my_file.nc")
Import from netCDF
To import network data from netCDF file run
network.import_from_netcdf(file.nc)
.
Export to HDF5
Note
netCDF is preferred over HDF5 because netCDF is structured more cleanly, is easier to use from other programming languages, can limit float precision to save space and supports lazy loading.
To export network and components to an HDF store run
network.export_to_hdf5(path)
.
- Network.export_to_hdf5(path, export_standard_types=False, **kwargs)
Export network and components to an HDF store.
Both static and series attributes of components are exported, but only if they have non-default values.
If path does not already exist, it is created.
- Parameters
path (string) – Name of hdf5 file to which to export (if it exists, it is overwritten)
export_standard_types (boolean, default False) – If True, then standard types are exported too (upon reimporting you should then set “ignore_standard_types” when initialising the network).
**kwargs – Extra arguments for pd.HDFStore to specify f.i. compression (default: complevel=4)
Examples
>>> network.export_to_hdf5(filename)
Import from HDF5
To import network data from HDF5 store at path
run
network.import_from_hdf5(path)
.
Import from Pypower
PyPSA supports import from Pypower’s ppc dictionary/numpy.array format version 2.
- Network.import_from_pypower_ppc(ppc, overwrite_zero_s_nom=None)
Import network from PYPOWER PPC dictionary format version 2.
Converts all baseMVA to base power of 1 MVA.
For the meaning of the pypower indices, see also pypower/idx_*.
- Parameters
ppc (PYPOWER PPC dict) –
overwrite_zero_s_nom (Float or None, default None) –
Examples
>>> from pypower.api import case30 >>> ppc = case30() >>> network.import_from_pypower_ppc(ppc)
Import from Pandapower
Warning
Importing from pandapower is still in beta; not all pandapower data is supported.
PyPSA supports import from pandapower.
- Network.import_from_pandapower_net(net, extra_line_data=False)
Import network from pandapower net.
Importing from pandapower is still in beta; not all pandapower data is supported.
Unsupported features include: - three-winding transformers - switches - in_service status, - shunt impedances, and - tap positions of transformers.”
- Parameters
net (pandapower network) –
extra_line_data (boolean, default: False) – if True, the line data for all parameters is imported instead of only the type
Examples
>>> network.import_from_pandapower_net(net) OR >>> import pandapower as pp >>> import pandapower.networks as pn >>> net = pn.create_cigre_network_mv(with_der='all') >>> pp.runpp(net) >>> network.import_from_pandapower_net(net, extra_line_data=True)