AI Radar

JetBrains tests 12 AI models on PHP and Python

JetBrains asked 12 recent AI models for a controversial programming-language opinion, 50 times per prompt and two prompts per model, 1,200 requests in all. Each request used a fresh conversation with no system prompt. The goal was to see whether models have distinct takes or repeat the same one.

Who should care
Developers curious about how consistent LLM answers are on opinion prompts.

Key facts

12 models from 10 labs in three countries
1,200 requests, all returned
Two prompts, 50 runs each per model
Collecting the responses cost USD 7.68
Models include Claude Opus 5.5, Gemini 4 Argon, Grok 4.7, DeepSeek V4 Pro, Mistral Large 4
Each prompt asked for 50 to 90 words

Timeline

Official

Evidence

High evidence: the claim comes from an official source (the company or project itself), or at least two independent sites reported it.

Details written Nov 15, 23:39 UTC from the sources listed above.