clinicadl.metrics.MetricsHandler¶

class clinicadl.metrics.MetricsHandler(metrics_on_cpu: bool = True, **metrics: Metric | MetricConfig)[source]¶

To handle all the Metric computed during an evaluation phase.

MetricsHandler is itself a callable that works like monai.metrics.CumulativeIterationMetric, with reset() and aggregate() methods. So, it can be used like a raw Metric object.

The results are stored in DataFrames (df and detailed_df), that can be saved with save().

Parameters:

metrics_on_cpu (boo, True) – Whether to necessarily apply metrics computation on CPU. If False, postprocessing will be applied on the device where are currently the data
**metrics (MetricOrConfig) – Metrics to add to the MetricsHandler. They must be passed as Metric or configuration classes.

Examples

from clinicadl.metrics import MetricsHandler, config
from clinicadl.data.structures.examples import Colin27DataPoint
from clinicadl.data.dataloader import Batch
import torch

metrics = MetricsHandler(
    mse=config.MSEMetricConfig(label_key="ground_truth"),
    mae=config.MAEMetricConfig(label_key="ground_truth"),
)
metrics.init_metrics()

torch.manual_seed(0)
batch = Batch(
    [
        Colin27DataPoint(output=torch.randn(1, 10), ground_truth=torch.randn(1, 10)),
        Colin27DataPoint(output=torch.randn(1, 10), ground_truth=torch.randn(1, 10)),
    ]
)

>>> metrics(batch)
   epoch participant_id session_id       mse       mae
0      1        sub-000   ses-M000  1.155020  0.871509
1      1        sub-001   ses-M000  1.540214  0.835756
>>> metrics.detailed_df
   epoch participant_id session_id       mse       mae
0      1        sub-000   ses-M000  1.155020  0.871509
1      1        sub-001   ses-M000  1.540214  0.835756
>>> metrics.aggregate(epoch=1)
>>> metrics.df
   epoch       mse       mae
0      1  1.347617  0.853633

init_metrics(model: Model | None = None) → None[source]¶

Instantiates the metrics from their configuration classes.

Parameters:: model (Optional[Model], default=None) – The model that contains the potential losses to compute on the validation set (defined in Model.get_loss_functions).

property metrics: dict[str, Metric] | None¶: The metrics currently in the MetricsHandler. If None, it means that init_metrics() must be called.

property df: DataFrame¶: The pandas.DataFrame containing the aggregated results, i.e. the results on the whole dataset obtained by calling aggregate().

property detailed_df: DataFrame¶: The pandas.DataFrame containing the detailed results, i.e. the results for each image.

add_metrics(**metrics: Metric | MetricConfig) → None[source]¶

Adds metrics to the MetricsHandler instance.

Warning

To be sure that all the metrics are computed on the same dataset, add_metrics will reset all the present metrics.

Parameters:: **metrics (MetricConfig) – Metrics to add to the MetricsHandler. They must be passed as Metric or configuration classes.

remove_metrics(metrics: str | Sequence[str]) → None[source]¶

Removes metrics from the MetricsHandler instance.

Parameters:: metrics (Union[str, Sequence[str]]) – The name of the metric(s) to remove.

reset(reset_df: bool = False) → None[source]¶

Reset all metric states.

Parameters:: reset_df (bool, default=False) – If True, also reset the DataFrames containing the results.