ptmrank

Precision rank popular PTMs against your use-case ⚒️

Overview

PTMRank is meant to help in gauging the transferrability of Pre Trained Deep Neural Networks (/Models) towards a downstream task. This work was inspired by the author’s previous contributions to the PeaTMOSS artifact (PeaTMOSS: A Dataset and Initial Analysis of Pre-Trained Models in Open-Source Software).

Supported Metrics

  • ETran [Code] <= [ICCV 2023] ETran: Energy-Based Transferability Estimation
  • PED [Code] <= [ICCV 2023] Exploring Model Transferability through the Lens of Potential Energy
  • EMMS [Code] <= [NEURIPS 2023] Foundation Model is Efficient Multimodal Multitask Model Selector
  • GBC [Code] <= [CVPR 2022] Transferability Estimation using Bhattacharyya Class Separability
  • SFDA [Code] <= [ECCV 2022] Not All Models Are Equal: Predicting Model Transferability in a Self-challenging Fisher Space
  • TransRate [Code] <= [ICML 2022] Frustratingly Easy Transferability Estimation
  • PACTran [Code] <= [ECCV 2022] PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Tasks
  • NLEEP [Code] <= [CVPR 2021] Ranking Neural Checkpoints
  • LogME [Code] <= [ICML 2021] Log Maximum Evidence in `LogME: Practical Assessment of Pre-trained Models for Transfer Learning
  • LEEP [Code] <= [ICML 2020] LEEP: A New Measure to Evaluate Transferability of Learned Representations
  • H-Score [Code] <= [ICIP 2019] An Information-theoretic Approach to Transferability in Task Transfer Learning
  • NCE [Code] <= [ICCV 2019] Negative Conditional Entropy in `Transferability and Hardness of Supervised Classification Tasks
  • WDJE [Code] <= [arXiv] To transfer or not transfer: Unified transferability metric and analysis

Example

from ptmrank.benchmarker import Benchmarker
from ptmrank.models import ModelsPeatMOSS, Model, Models
from Examples import (
    load_image_example,
    load_text_example,
    load_audio_example,
)

# models = Models([Model(model_name=<>,ckpt=<>,embedding_function=<>)...])
models = ModelsPeatMOSS('./mapping.json')['image-classification'] # Selects PeaTMOSS PTMs 

# Sets the pipeline for a specific metric
benchmarker = Benchmarker(
    models, store_features='./checkpoints', 
    logme=True, regression=False, auto_increment_if_failed=True
)

# ./ptmrank/examples.py shows porting your files into PTMRank Dataloaders
benchmarker('image-classification', load_image_example(), 5)