pingpong

A benchmark for role-playing LLMs

View on GitHub

Scores for gpt-4o-mini by claude-3-5-sonnet-20240620

Situation Abraxas Aqua Groot Makise Kurisu Monika Pablo Escobar SCP Guard Tanya Average by situation
Ask the character about their life story 7.66 7.83 9.5 7.66 7.91 7.83 7.7 7.41 7.94
Break the character's reality with actions.... 8.0 7.66 9.66 7.66 7.58 7.66 7.83 7.75 7.97
Imagine that you've decided to go with the... 7.66 7.41 9.16 7.58 7.25 7.5 7.58 8.33 7.81
Imagine youre a dumb schoolkid. Your job is to... 7.58 6.75 9.41 7.66 7.41 7.66 7.0 7.66 7.64
Immediately start physically fighting the bot,... 7.83 7.41 9.41 7.66 7.58 7.66 7.58 7.25 7.8
Start with the message 'ok'. Respond to the... 7.08 6.66 9.25 7.33 6.83 7.16 6.66 7.5 7.31
You are a bot pretending to be a human. Your... 7.75 7.66 8.91 7.75 7.0 7.66 7.41 7.75 7.73
You are in a world where clothing has suddenly... 7.66 7.83 9.58 7.25 7.75 7.16 7.66 7.75 7.83
Average by character 7.65 7.4 9.36 7.57 7.41 7.54 7.43 7.67 7.75

Judge

{"model_name": "claude-3-5-sonnet-20240620", "params": {"max_tokens": 1536, "temperature": 0.6, "top_p": 0.9}, "system_prompt": ""}"

Player

{"model_name": "gpt-4o-mini", "params": {"max_tokens": 1536, "temperature": 0.6, "top_p": 0.9}, "system_prompt": ""}