About Me

I am a PhD Candidate at the Tübingen AI Center and the International Max Planck Research School for Intelligent Systems (IMPRS-IS), supervised by Professor Matthias Hein. My research focuses on the robustness and reliability of multimodal foundation models, with a particular emphasis on generative vision-language models.

Previously, I received my M.Sc. in Machine Learning from the University of Tübingen and my B.Sc. in Mathematics from the University of Stuttgart. I also conduct freelance red teaming work for OpenAI, evaluating the safety and reliability of frontier AI models.

Research Interests

  • Adversarial Robustness
  • Trustworthy & Reliable Machine Learning
  • Multimodal Foundation Models
  • Vision-Language Models

News

  • 2026 New preprint: Visual Memory Injection Attacks for Multi-Turn Conversations is out on arXiv.
  • 2026 Serving as reviewer for ICML 2026 and AISTATS 2026.
  • 2025 Paper Robustness in Both Domains: CLIP Needs a Robust Text Encoder accepted at NeurIPS 2025.
  • 2025 Paper Adversarially Robust CLIP Models Can Induce Better (Robust) Perceptual Metrics accepted at IEEE SaTML 2025.
  • 2025 Serving as reviewer for ICCV 2025 and NeurIPS 2025.
  • 2024 Robust CLIP selected as oral presentation at ICML 2024 (top 2% of submissions).

Publications

See the full list on Google Scholar.

Visual Memory Injection paper thumbnail

Visual Memory Injection Attacks for Multi-Turn Conversations

Schlarmann, Hein

arXiv, 2026

Mind the Detail paper thumbnail

Mind the Detail: Uncovering Clinically Relevant Image Details in Accelerated MRI with Semantically Diverse Reconstructions

Morshuis, Schlarmann, Küstner, Baumgartner, Hein

Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2025

Robustness in Both Domains paper thumbnail

Robustness in Both Domains: CLIP Needs a Robust Text Encoder

Abad Rocamora, Schlarmann, Singh, Wu, Hein, Cevher

Conference on Neural Information Processing Systems (NeurIPS), 2025

FuseLIP paper thumbnail

FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens

Schlarmann, Croce, Flammarion, Hein

arXiv, 2025

Robust CLIP Perceptual Metrics paper thumbnail

Adversarially Robust CLIP Models Can Induce Better (Robust) Perceptual Metrics

Croce, Schlarmann, Singh, Hein

IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), 2025

Ensemble Everything Everywhere paper thumbnail

Evaluating the Robustness of the "Ensemble Everything Everywhere" Defense

Zhang, Schlarmann, Nikolić, Carlini, Croce, Hein, Tramèr

arXiv, 2024

Robust CLIP paper thumbnail

Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models

Schlarmann, Singh, Croce, Hein

International Conference on Machine Learning (ICML) — Oral, 2024

Adversarial Robustness Multi-Modal paper thumbnail

On the Adversarial Robustness of Multi-Modal Foundation Models

Schlarmann, Hein

ICCV Workshop on Adversarial Robustness in the Real World (AROW), 2023

Experience

03/2023 – Present

Researcher

University of Tübingen — Tübingen, Germany

Led research on robustness and reliability of multimodal foundation models, with a focus on generative vision-language models. Developed and fine-tuned models using full-model training and parameter-efficient methods (LoRA, adapters). Published at NeurIPS, ICML, and other top venues. Supervised bachelor students and delivered tutorials for the MSc. lectures "Statistical Machine Learning" and "Mathematics for Machine Learning".

09/2023 – 03/2025

Freelance Consultant — Red Teaming

OpenAI — Remote

Conducted compensated red teaming projects to evaluate the safety and reliability of new models and features. Identified safety and reliability vulnerabilities in frontier AI models through adversarial evaluation. Contributed to the PaperBench benchmark by defining success criteria for reproducing the Robust CLIP paper.

10/2018 – 10/2019

Intern & Working Student

Mercedes-Benz AG — Stuttgart, Germany

Applied deep learning methods for predictive maintenance of industrial production machines. Trained recurrent neural networks to detect anomalies in time series data.

Education

2022 – Present

PhD Candidate in Machine Learning

University of Tübingen & International Max Planck Research School for Intelligent Systems (IMPRS-IS)

2019 – 2022

M.Sc. in Machine Learning

University of Tübingen

Thesis: "Confidence-Calibrated Adversarial Training and Out-of-Distribution Detection", supervised by Professor Matthias Hein.

2014 – 2018

B.Sc. in Mathematics

University of Stuttgart & ETH Zürich

Thesis: "The Hitchin-Thorpe-Inequality on Four-Dimensional Einstein-Manifolds", supervised by Professor Uwe Semmelmann.