Examples

The general command format for running a benchmark is:

thunder benchmark <MODEL> <DATASET> <TASK> [OPTIONS]

In the following examples, we illustrate how to use THUNDER through both the command-line interface (CLI) and the Python API. As example, we will use Hoptimus1 as the model, MHIST as the classification dataset, and PanNuke for segmentation.

---

View Benchmark Help

CLI

thunder benchmark --help

---

Download Datasets

CLI

thunder download-datasets mhist pannuke

---

Generate Data Splits

CLI

thunder generate-data-splits pannuke mhist

---

Download Pretrained Models

CLI

thunder download-models hoptimus1

---

Benchmarking Examples

Below are examples for multiple tasks using the Hoptimus1 model.

---

Pre-computing Embeddings

CLI

thunder benchmark hoptimus1 mhist pre_computing_embeddings
thunder benchmark hoptimus1 pannuke pre_computing_embeddings

API

from thunder import benchmark

benchmark(
    "hoptimus1",
    dataset="mhist",
    task="pre_computing_embeddings",
)

benchmark(
    "hoptimus1",
    dataset="pannuke",
    task="pre_computing_embeddings",
)

---

K-NN Classification

CLI

thunder benchmark hoptimus1 mhist knn

API

from thunder import benchmark

benchmark(
    "hoptimus1",
    dataset="mhist",
    task="knn",
)

---

Linear Probing

Using pre-loaded embeddings.

CLI

thunder benchmark hoptimus1 mhist linear_probing --loading-mode=embedding_pre_loading

API

from thunder import benchmark

benchmark(
    "hoptimus1",
    dataset="mhist",
    task="linear_probing",
    loading_mode="embedding_pre_loading",
)

---

Segmentation

CLI

thunder benchmark hoptimus1 pannuke segmentation --loading-mode=embedding_pre_loading

API

from thunder import benchmark

benchmark(
    "hoptimus1",
    dataset="pannuke",
    task="segmentation",
    loading_mode="embedding_pre_loading",
)

---

SimpleShot Classification

CLI

thunder benchmark hoptimus1 mhist simple_shot --loading-mode=embedding_pre_loading

API

from thunder import benchmark

benchmark(
    "hoptimus1",
    dataset="mhist",
    task="simple_shot",
    loading_mode="embedding_pre_loading",
)

---

Transformation Invariance

CLI

thunder benchmark hoptimus1 mhist transformation_invariance

API

from thunder import benchmark

benchmark(
    "hoptimus1",
    dataset="mhist",
    task="transformation_invariance",
)

---

Adversarial Attack Robustness

CLI

thunder benchmark hoptimus1 mhist adversarial_attack

API

from thunder import benchmark

benchmark(
    "hoptimus1",
    dataset="mhist",
    task="adversarial_attack",
)

---

Results Summary

CLI

To display all collected benchmark results, simply run the command below after your experiments have finished.

thunder results-summary