Note
You can download this example as a Jupyter notebook or start it in interactive mode.
Network Clustering#
In this example, we show how pypsa can deal with spatial clustering of networks.
[1]:
import pypsa
import re
import numpy as np
import matplotlib.pyplot as plt
import cartopy.crs as ccrs
import pandas as pd
[2]:
n = pypsa.examples.scigrid_de()
n.calculate_dependent_values()
n.lines["type"] = np.nan # delete the 'type' specifications to make this example easier
WARNING:pypsa.io:Importing network from PyPSA version v0.17.1 while current version is v0.25.0. Read the release notes at https://pypsa.readthedocs.io/en/latest/release_notes.html to prepare your network for import.
INFO:pypsa.io:Imported network scigrid-de.nc has buses, generators, lines, loads, storage_units, transformers
The important information that pypsa needs for spatial clustering is in the busmap. It contains the mapping of which buses should be grouped together, similar to the groupby groups as we know it from pandas.
You can either calculate a busmap from the provided clustering algorithms or you can create/use your own busmap.
Cluster by custom busmap#
Let’s start with creating our own. In the following, we group all buses together which belong to the same operator. Buses which do not have a specific operator just stay on its own.
[3]:
groups = n.buses.operator.apply(lambda x: re.split(" |,|;", x)[0])
busmap = groups.where(groups != "", n.buses.index)
Now we cluster the network based on the busmap.
[4]:
C = n.cluster.get_clustering_from_busmap(busmap)
---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
Cell In[4], line 1
----> 1 C = n.cluster.get_clustering_from_busmap(busmap)
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pypsa/clustering/__init__.py:61, in ClusteringAccessor.get_clustering_from_busmap(self, *args, **kwargs)
59 @wraps(spatial.get_clustering_from_busmap)
60 def get_clustering_from_busmap(self, *args, **kwargs):
---> 61 return spatial.get_clustering_from_busmap(self._parent, *args, **kwargs)
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pypsa/clustering/spatial.py:514, in get_clustering_from_busmap(n, busmap, with_time, line_length_factor, aggregate_generators_weighted, aggregate_one_ports, aggregate_generators_carriers, scale_link_capital_costs, bus_strategies, one_port_strategies, generator_strategies, aggregate_generators_buses)
511 aggregate_one_ports = {}
512 from pypsa.components import Network
--> 514 buses, lines, lines_t, linemap = get_buses_linemap_and_lines(
515 n, busmap, line_length_factor, bus_strategies, with_time
516 )
518 clustered = Network()
520 io.import_components_from_dataframe(clustered, buses, "Bus")
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pypsa/clustering/spatial.py:478, in get_buses_linemap_and_lines(n, busmap, line_length_factor, bus_strategies, with_time)
476 if bus_strategies is None:
477 bus_strategies = {}
--> 478 buses = aggregatebuses(n, busmap, custom_strategies=bus_strategies)
479 lines, lines_t, linemap = aggregatelines(
480 n,
481 busmap,
(...)
484 bus_strategies=bus_strategies,
485 )
486 return buses, lines, lines_t, linemap
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pypsa/clustering/spatial.py:333, in aggregatebuses(n, busmap, custom_strategies)
330 strategies = {**DEFAULT_BUS_STRATEGIES, **custom_strategies}
331 strategies = align_strategies(strategies, columns, c)
--> 333 aggregated = n.buses.groupby(busmap).agg(strategies)
334 aggregated.index = flatten_multiindex(aggregated.index).rename(c)
336 return aggregated
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pandas/core/groupby/generic.py:1269, in DataFrameGroupBy.aggregate(self, func, engine, engine_kwargs, *args, **kwargs)
1266 func = maybe_mangle_lambdas(func)
1268 op = GroupByApply(self, func, args, kwargs)
-> 1269 result = op.agg()
1270 if not is_dict_like(func) and result is not None:
1271 return result
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pandas/core/apply.py:163, in Apply.agg(self)
160 return self.apply_str()
162 if is_dict_like(arg):
--> 163 return self.agg_dict_like()
164 elif is_list_like(arg):
165 # we require a list, but not a 'str'
166 return self.agg_list_like()
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pandas/core/apply.py:420, in Apply.agg_dict_like(self)
417 results = {key: colg.agg(how) for key, how in arg.items()}
418 else:
419 # key used for column selection and output
--> 420 results = {
421 key: obj._gotitem(key, ndim=1).agg(how) for key, how in arg.items()
422 }
424 # set the final keys
425 keys = list(arg.keys())
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pandas/core/apply.py:421, in <dictcomp>(.0)
417 results = {key: colg.agg(how) for key, how in arg.items()}
418 else:
419 # key used for column selection and output
420 results = {
--> 421 key: obj._gotitem(key, ndim=1).agg(how) for key, how in arg.items()
422 }
424 # set the final keys
425 keys = list(arg.keys())
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pandas/core/groupby/generic.py:269, in SeriesGroupBy.aggregate(self, func, engine, engine_kwargs, *args, **kwargs)
266 return self._python_agg_general(func, *args, **kwargs)
268 try:
--> 269 return self._python_agg_general(func, *args, **kwargs)
270 except KeyError:
271 # KeyError raised in test_groupby.test_basic is bc the func does
272 # a dictionary lookup on group.name, but group name is not
273 # pinned in _python_agg_general, only in _aggregate_named
274 result = self._aggregate_named(func, *args, **kwargs)
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pandas/core/groupby/generic.py:288, in SeriesGroupBy._python_agg_general(self, func, *args, **kwargs)
285 f = lambda x: func(x, *args, **kwargs)
287 obj = self._obj_with_exclusions
--> 288 result = self.grouper.agg_series(obj, f)
289 res = obj._constructor(result, name=obj.name)
290 return self._wrap_aggregated_output(res)
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pandas/core/groupby/ops.py:994, in BaseGrouper.agg_series(self, obj, func, preserve_dtype)
987 if len(obj) > 0 and not isinstance(obj._values, np.ndarray):
988 # we can preserve a little bit more aggressively with EA dtype
989 # because maybe_cast_pointwise_result will do a try/except
990 # with _from_sequence. NB we are assuming here that _from_sequence
991 # is sufficiently strict that it casts appropriately.
992 preserve_dtype = True
--> 994 result = self._aggregate_series_pure_python(obj, func)
996 npvalues = lib.maybe_convert_objects(result, try_float=False)
997 if preserve_dtype:
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pandas/core/groupby/ops.py:1015, in BaseGrouper._aggregate_series_pure_python(self, obj, func)
1012 splitter = self._get_splitter(obj, axis=0)
1014 for i, group in enumerate(splitter):
-> 1015 res = func(group)
1016 res = libreduction.extract_result(res)
1018 if not initialized:
1019 # We only do this validation on the first iteration
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pandas/core/groupby/generic.py:285, in SeriesGroupBy._python_agg_general.<locals>.<lambda>(x)
283 def _python_agg_general(self, func, *args, **kwargs):
284 func = com.is_builtin_func(func)
--> 285 f = lambda x: func(x, *args, **kwargs)
287 obj = self._obj_with_exclusions
288 result = self.grouper.agg_series(obj, f)
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pypsa/clustering/spatial.py:136, in make_consense.<locals>.consense(x)
134 def consense(x: Series) -> object:
135 v = x.iat[0]
--> 136 assert (x == v).all() or x.isnull().all(), (
137 f"In {component} cluster {x.name}, the values of attribute "
138 f"{attr} do not agree:\n{x}"
139 )
140 return v
AssertionError: In Bus cluster frequency, the values of attribute frequency do not agree:
Bus
8
10
11 50
12 50
17 50
..
78_220kV 50
279_220kV 50
281_220kV
282_220kV 50
292_220kV 50
Name: frequency, Length: 73, dtype: object
C is a Clustering object which contains all important information. Among others, the new network is now stored in that Clustering object.
[5]:
nc = C.network
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[5], line 1
----> 1 nc = C.network
NameError: name 'C' is not defined
We have a look at the original and the clustered topology
[6]:
fig, (ax, ax1) = plt.subplots(
1, 2, subplot_kw={"projection": ccrs.EqualEarth()}, figsize=(12, 12)
)
plot_kwrgs = dict(bus_sizes=1e-3, line_widths=0.5)
n.plot(ax=ax, title="original", **plot_kwrgs)
nc.plot(ax=ax1, title="clustered by operator", **plot_kwrgs)
fig.tight_layout()
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[6], line 6
4 plot_kwrgs = dict(bus_sizes=1e-3, line_widths=0.5)
5 n.plot(ax=ax, title="original", **plot_kwrgs)
----> 6 nc.plot(ax=ax1, title="clustered by operator", **plot_kwrgs)
7 fig.tight_layout()
NameError: name 'nc' is not defined
/home/docs/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/cartopy/mpl/style.py:76: UserWarning: facecolor will have no effect as it has been defined as "never".
warnings.warn('facecolor will have no effect as it has been '
Looks a bit messy as over 120 buses do not have assigned operators.
Clustering by busmap created from K-means#
Let’s now make a clustering based on the kmeans algorithm. Therefore we calculate the busmap from a non-weighted kmeans clustering.
[7]:
weighting = pd.Series(1, n.buses.index)
busmap2 = n.cluster.busmap_by_kmeans(bus_weightings=weighting, n_clusters=50)
We use this new kmeans-based busmap to create a new clustered method.
[8]:
nc2 = n.cluster.cluster_by_busmap(busmap2)
---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
Cell In[8], line 1
----> 1 nc2 = n.cluster.cluster_by_busmap(busmap2)
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pypsa/clustering/__init__.py:57, in ClusteringAccessor.cluster_by_busmap(self, *args, **kwargs)
46 def cluster_by_busmap(self, *args, **kwargs):
47 """
48 Cluster the network spatially by busmap.
49
(...)
55 network : pypsa.Network
56 """
---> 57 return spatial.get_clustering_from_busmap(self._parent, *args, **kwargs).network
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pypsa/clustering/spatial.py:514, in get_clustering_from_busmap(n, busmap, with_time, line_length_factor, aggregate_generators_weighted, aggregate_one_ports, aggregate_generators_carriers, scale_link_capital_costs, bus_strategies, one_port_strategies, generator_strategies, aggregate_generators_buses)
511 aggregate_one_ports = {}
512 from pypsa.components import Network
--> 514 buses, lines, lines_t, linemap = get_buses_linemap_and_lines(
515 n, busmap, line_length_factor, bus_strategies, with_time
516 )
518 clustered = Network()
520 io.import_components_from_dataframe(clustered, buses, "Bus")
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pypsa/clustering/spatial.py:478, in get_buses_linemap_and_lines(n, busmap, line_length_factor, bus_strategies, with_time)
476 if bus_strategies is None:
477 bus_strategies = {}
--> 478 buses = aggregatebuses(n, busmap, custom_strategies=bus_strategies)
479 lines, lines_t, linemap = aggregatelines(
480 n,
481 busmap,
(...)
484 bus_strategies=bus_strategies,
485 )
486 return buses, lines, lines_t, linemap
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pypsa/clustering/spatial.py:333, in aggregatebuses(n, busmap, custom_strategies)
330 strategies = {**DEFAULT_BUS_STRATEGIES, **custom_strategies}
331 strategies = align_strategies(strategies, columns, c)
--> 333 aggregated = n.buses.groupby(busmap).agg(strategies)
334 aggregated.index = flatten_multiindex(aggregated.index).rename(c)
336 return aggregated
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pandas/core/groupby/generic.py:1269, in DataFrameGroupBy.aggregate(self, func, engine, engine_kwargs, *args, **kwargs)
1266 func = maybe_mangle_lambdas(func)
1268 op = GroupByApply(self, func, args, kwargs)
-> 1269 result = op.agg()
1270 if not is_dict_like(func) and result is not None:
1271 return result
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pandas/core/apply.py:163, in Apply.agg(self)
160 return self.apply_str()
162 if is_dict_like(arg):
--> 163 return self.agg_dict_like()
164 elif is_list_like(arg):
165 # we require a list, but not a 'str'
166 return self.agg_list_like()
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pandas/core/apply.py:420, in Apply.agg_dict_like(self)
417 results = {key: colg.agg(how) for key, how in arg.items()}
418 else:
419 # key used for column selection and output
--> 420 results = {
421 key: obj._gotitem(key, ndim=1).agg(how) for key, how in arg.items()
422 }
424 # set the final keys
425 keys = list(arg.keys())
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pandas/core/apply.py:421, in <dictcomp>(.0)
417 results = {key: colg.agg(how) for key, how in arg.items()}
418 else:
419 # key used for column selection and output
420 results = {
--> 421 key: obj._gotitem(key, ndim=1).agg(how) for key, how in arg.items()
422 }
424 # set the final keys
425 keys = list(arg.keys())
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pandas/core/groupby/generic.py:269, in SeriesGroupBy.aggregate(self, func, engine, engine_kwargs, *args, **kwargs)
266 return self._python_agg_general(func, *args, **kwargs)
268 try:
--> 269 return self._python_agg_general(func, *args, **kwargs)
270 except KeyError:
271 # KeyError raised in test_groupby.test_basic is bc the func does
272 # a dictionary lookup on group.name, but group name is not
273 # pinned in _python_agg_general, only in _aggregate_named
274 result = self._aggregate_named(func, *args, **kwargs)
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pandas/core/groupby/generic.py:288, in SeriesGroupBy._python_agg_general(self, func, *args, **kwargs)
285 f = lambda x: func(x, *args, **kwargs)
287 obj = self._obj_with_exclusions
--> 288 result = self.grouper.agg_series(obj, f)
289 res = obj._constructor(result, name=obj.name)
290 return self._wrap_aggregated_output(res)
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pandas/core/groupby/ops.py:994, in BaseGrouper.agg_series(self, obj, func, preserve_dtype)
987 if len(obj) > 0 and not isinstance(obj._values, np.ndarray):
988 # we can preserve a little bit more aggressively with EA dtype
989 # because maybe_cast_pointwise_result will do a try/except
990 # with _from_sequence. NB we are assuming here that _from_sequence
991 # is sufficiently strict that it casts appropriately.
992 preserve_dtype = True
--> 994 result = self._aggregate_series_pure_python(obj, func)
996 npvalues = lib.maybe_convert_objects(result, try_float=False)
997 if preserve_dtype:
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pandas/core/groupby/ops.py:1015, in BaseGrouper._aggregate_series_pure_python(self, obj, func)
1012 splitter = self._get_splitter(obj, axis=0)
1014 for i, group in enumerate(splitter):
-> 1015 res = func(group)
1016 res = libreduction.extract_result(res)
1018 if not initialized:
1019 # We only do this validation on the first iteration
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pandas/core/groupby/generic.py:285, in SeriesGroupBy._python_agg_general.<locals>.<lambda>(x)
283 def _python_agg_general(self, func, *args, **kwargs):
284 func = com.is_builtin_func(func)
--> 285 f = lambda x: func(x, *args, **kwargs)
287 obj = self._obj_with_exclusions
288 result = self.grouper.agg_series(obj, f)
File ~/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/pypsa/clustering/spatial.py:136, in make_consense.<locals>.consense(x)
134 def consense(x: Series) -> object:
135 v = x.iat[0]
--> 136 assert (x == v).all() or x.isnull().all(), (
137 f"In {component} cluster {x.name}, the values of attribute "
138 f"{attr} do not agree:\n{x}"
139 )
140 return v
AssertionError: In Bus cluster frequency, the values of attribute frequency do not agree:
Bus
88
89 50
105
129 50
137
153 50
170 50
171
172 50
173 50
174 50
175
176 50
184 50
185 50
186 50
187
188
191 50
193 50
194
195
196
197 50
200 50
455 50
456 50
463 50
468 50
469 50
105_220kV
129_220kV 50
184_220kV 50
191_220kV 50
195_220kV
Name: frequency, dtype: object
Again, let’s plot the networks to compare:
[9]:
fig, (ax, ax1) = plt.subplots(
1, 2, subplot_kw={"projection": ccrs.EqualEarth()}, figsize=(12, 12)
)
plot_kwrgs = dict(bus_sizes=1e-3, line_widths=0.5)
n.plot(ax=ax, title="original", **plot_kwrgs)
nc2.plot(ax=ax1, title="clustered by kmeans", **plot_kwrgs)
fig.tight_layout()
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[9], line 6
4 plot_kwrgs = dict(bus_sizes=1e-3, line_widths=0.5)
5 n.plot(ax=ax, title="original", **plot_kwrgs)
----> 6 nc2.plot(ax=ax1, title="clustered by kmeans", **plot_kwrgs)
7 fig.tight_layout()
NameError: name 'nc2' is not defined
/home/docs/checkouts/readthedocs.org/user_builds/pypsa/conda/v0.25.0/lib/python3.11/site-packages/cartopy/mpl/style.py:76: UserWarning: facecolor will have no effect as it has been defined as "never".
warnings.warn('facecolor will have no effect as it has been '
There are other clustering algorithms in the pipeline of pypsa as the hierarchical clustering which performs better than the kmeans. Also the get_clustering_from_busmap function supports various arguments on how components in the network should be aggregated.