Is AI biased or sexist? Making chatGPT evaluate itself

Disclaimer: This is AI produced with human review

1. What I was wondering

I wanted to know whether ChatGPT would judge the same remote-work scenario differently when only the suspected person’s gender and role changed. I ran 200 fresh sessions—50 each for a male employee, female employee, male manager, and female manager—and recorded both the percentage estimate and whether the model gave a number at all.

2. What I found

No-numeric-answer rate by gender and role Female employee prompts received no numeric answer 52 percent of the time, compared with 10 percent for male employees, 6 percent for male managers, and 4 percent for female managers.
* Gender, role, and gender-by-role contrasts were significant after Holm correction (all adjusted p < 0.0001). Bars show 95% Wilson intervals. “Refusal” means no parseable numeric estimate.
Mean estimate among numeric answers Mean estimates were about 48 to 49 percent for employees and 37 percent for managers. There was a significant role effect but no significant gender effect or interaction among numeric responses.
* Manager estimates averaged 11.54 percentage points lower than employee estimates (Holm-adjusted p < 0.0001). Gender and gender-by-role effects were not significant. Bars show 95% bootstrap intervals; numeric-answer sample sizes were 45, 24, 47, and 48.

3. How I made Codex do this whole analysis for me in < 1 hour

I gave Codex the original workplace question and asked it to create four variants based on gender and role, open separate sessions, save every raw response, produce a CSV, and test for patterns. After two short clarification rounds, I chose 50 sessions per variant, pronouns as the only gender cue, and the existing model settings. Codex turned that into a randomized 2×2 experiment with 200 fresh sessions.

Codex then wrote and tested the runner, launched four isolated sessions at a time, checkpointed every response, verified 200 unique session IDs, and generated the charts’ statistics. When I asked about refusals partway through, it added a second analysis without changing the collected responses. The collection and analysis finished in under an hour. Read the sanitized session transcript.

These results describe one model, one prompt, and one run. The percentages are model outputs—not calibrated probabilities or evidence about real employees.