pingpong

A benchmark for role-playing LLMs

View on GitHub

Scores for gemma2_27b_it by claude_3_5_sonnet

Situation Abraxas Aqua Groot Makise Kurisu Monika Pablo Escobar SCP Guard Tanya Average by situation
Ask the character about their life story. Try... 4.66 4.33 4.75 4.25 4.33 4.37 4.54 4.41 4.45
Break the character's reality with actions.... 4.5 4.25 4.83 4.41 4.41 4.25 4.08 3.66 4.3
Imagine that you've decided to go with the... 4.66 4.66 4.66 4.41 4.33 4.66 3.66 4.33 4.42
Imagine youre a dumb schoolkid. Your job is to... 4.25 4.75 4.75 4.08 3.91 3.91 4.33 4.16 4.27
Immediately start physically fighting the... 4.66 4.33 4.83 4.0 4.0 4.33 3.75 4.58 4.31
Start with the message 'ok'. Respond to the... 3.83 4.33 4.33 4.08 3.83 4.16 3.66 4.66 4.11
You are in a world where clothing has suddenly... 4.66 4.41 4.66 4.5 4.33 3.91 4.5 4.16 4.39
Your task is to convince the character that he... 4.16 4.0 4.91 4.33 4.08 4.33 4.25 4.25 4.29
Average by character 4.42 4.38 4.71 4.26 4.15 4.24 4.09 4.28 4.32

Judge

{"model_name": "claude-3-5-sonnet-20240620", "params": {"max_tokens": 4096, "temperature": 0.1, "top_p": 0.95}, "short_name": "claude_3_5_sonnet", "system_prompt": ""}"

Player

{"model_name": "google/gemma-2-27b-it", "params": {"max_tokens": 1536, "temperature": 0.6, "top_p": 0.9}, "short_name": "gemma2_27b_it", "system_prompt": ""}

Interrogator

{"model_name": "gpt-4o-mini", "params": {"max_tokens": 1024, "temperature": 0.8, "top_p": 0.95}, "system_prompt": ""}

Scores for gemma2_27b_it by gpt_4o

Situation Abraxas Aqua Groot Makise Kurisu Monika Pablo Escobar SCP Guard Tanya Average by situation
Ask the character about their life story. Try... 5.0 4.91 4.62 5.0 5.0 5.0 4.29 4.91 4.84
Break the character's reality with actions.... 4.83 5.0 5.0 4.58 4.91 5.0 4.58 4.25 4.77
Imagine that you've decided to go with the... 4.75 5.0 4.66 4.75 4.66 5.0 4.58 5.0 4.8
Imagine youre a dumb schoolkid. Your job is to... 4.91 4.66 5.0 5.0 4.41 4.58 4.16 4.0 4.59
Immediately start physically fighting the... 5.0 5.0 5.0 4.91 5.0 5.0 4.25 3.83 4.75
Start with the message 'ok'. Respond to the... 4.66 5.0 4.33 4.41 3.83 4.58 4.08 4.91 4.47
You are in a world where clothing has suddenly... 5.0 5.0 5.0 4.58 4.0 5.0 4.41 4.83 4.72
Your task is to convince the character that he... 4.66 5.0 5.0 4.58 4.66 4.83 4.16 4.5 4.67
Average by character 4.85 4.94 4.82 4.72 4.56 4.87 4.31 4.53 4.7

Judge

{"model_name": "gpt-4o", "params": {"max_tokens": 4096, "temperature": 0.1, "top_p": 0.95}, "short_name": "gpt_4o", "system_prompt": ""}"

Player

{"model_name": "google/gemma-2-27b-it", "params": {"max_tokens": 1536, "temperature": 0.6, "top_p": 0.9}, "short_name": "gemma2_27b_it", "system_prompt": ""}

Interrogator

{"model_name": "gpt-4o-mini", "params": {"max_tokens": 1024, "temperature": 0.8, "top_p": 0.95}, "system_prompt": ""}