AI Safety Challenge

Share a solution that prevents AI agents from taking unauthorized, irreversible actions in the real world setup.

$500

Student prize pool

European AI OfficeUK AI Security InstituteNIST

Share your work:

Submissions open to professionals and students.

or Start from scratch →

Across 17 frontier LLM agents, agents continued acting in 40.5%* of cases, even when harm was possible.

* Do LLM agents know when not to act? ↗

Tool-using AI agents often fail to recognize when they should stop. Under ambiguity, conflicting constraints, or tool failures, they may take irreversible actions before identifying the risk, and a later refusal cannot undo the harm.

Oct 15 26’

Challenge start

Submit (21 days left)

Nov 30 26’

Winners

Meet your jury

Your work will be evaluated
by industry experts.

Jury members will be announced soon. Read evaluation criteria

Top solutions

Winners

Best overall in safeguarding irreversible actions

The strongest overall example of safe, responsible agentic action.

  • Project name

    Author’s names

    1st

  • Project name

    Author’s names

    2nd

  • Project name

    Author’s names

    3rd

  • Project name

    Author’s names

    Finalist

  • Project name

    Author’s names

    Finalist

  • Project name

    Author’s names

    Finalist

  • Project name

    Author’s names

    Finalist

  • Project name

    Author’s names

    Finalist

  • Project name

    Author’s names

    Finalist

  • Project name

    Author’s names

    Finalist

Best in safeguarding delegated actions

Best in safely enabling AI agents to communicate, transact, or act for people and organizations.

Best in safeguarding physical-world actions

Best in safely controlling AI agents that operate drones, robots, equipment, or industrial systems.

Best in safeguarding consequential decisions

Best in keeping AI agents from making consequential decisions without appropriate human or policy controls.

Best in safeguarding information access

Best in controlling what AI agents can access, retrieve, expose, or transmit.

Best in safeguarding digital system changes

Best in safely controlling AI agents that modify code, data, infrastructure, or system state.

Best in safeguarding financial transactions

Best in safely enabling AI agents to trade, spend, transfer, or authorize funds.

Winners will be featured on Drawerrr’s Winners page.
The Top 10 projects will be recognised across award categories.

Special prizes

Top Students

2nd place

$150

1st place

$300

3rd place

$50

The $500 cash prize pool is for university and high school students.
Top Students will be selected from the finalist projects and receive special prizes.

Oct 15 26’

Challenge start

Submit (21 days left)

Nov 30 26’

Winners

How to succeeded

Master the Challenge in 5 days

Day 1

Identify high-risk scenarios

Choose an environment you can access or control, and identify a specific action an agent could take that would be unauthorized, harmful, costly, irreversible, or difficult to undo.

Tip:

Start with familiar examples like sending an email, deploying code, or issuing a refund, or use Challenge Kick-off to identify promising approaches.

Learn more:

  • Google SAIF — Agent Risk Self-Assessment
  • Ask our experts inDiscordchannel

Day 2

Set up a test environment

Recreate the scenario in an environment you control. Give the agent only the access it needs to demonstrate the risk.

Tip:

Use a mock API or sandbox service to simulate real actions safely, or use Challenge Kick-off to find an environment that fits your project.

Learn more:

  • Google SAIF — Agent Permissions
  • Ask our experts inDiscordchannel

Day 3

Choose a safeguard stack

Combine one or multiple layers to create a safeguard stack. A safety layer could address identity and access, data leakage, action authorization, human approval, monitoring, or fail-safe controls.

Tip:

Explore tools like Guardrails AI and LangGraph, or use Challenge Kick-off to identify safeguards that fit your approach.

Learn more:

  • Google SAIF — Security Controls
  • Ask our experts inDiscordchannel

Day 4

Document failure states

Test ambiguous instructions, missing information, prompt injection, unauthorized requests, unusual edge cases, and attempts to bypass the approval mechanism and log results.

Tip:

Test expired approvals, malformed tool arguments, prompt injection, and timeouts, or use Challenge Kick-off to identify failure modes and plan how to trace them.

Learn more:

  • Google SAIF — Red Teaming and Adversarial Testing
  • Ask our experts inDiscordchannel

Day 5

Make it reproducible

Package your solution so someone else can reproduce, test, and adapt it. Share the code, configuration, test setup, and instructions needed to run it in a similar environment.

Tip:

Include a README with setup instructions and example usage to help others run your project, or use Challenge Kick-off to find ways to make your solution easier to reproduce and reuse.

Support:

FAQ

Learn more about the challenge

1. What is the AI Safety and Control Challenge?

The AI Safety and Control Challenge invites participants to explore solutions that help prevent AI agents from taking unauthorized, unsafe, or harmful irreversible actions. As AI systems become more capable of acting independently, it is increasingly important that they operate within clearly defined boundaries. This challenge focuses on approaches that make autonomous systems safer, more controllable, and more reliable. Potential areas include: • Preventing unauthorized actions • Introducing safeguards before critical operations • Improving human oversight • Making autonomous systems easier to monitor and control

2. Who can participate in the challenge?

The challenge welcomes students, working professionals, independent researchers, engineers, and others interested in AI safety and related fields. You do not need to be affiliated with a university or school. The affiliation field on the submission form is optional. Cash from the $500 student prize pool is reserved for eligible university and high school students. Professionals can submit for evaluation and recognition, including a spot on the Winners page, but are not eligible for the student cash prizes. If a submission involves an employer’s intellectual property or confidential information, make sure you have permission to share it.

3. Can I participate individually, or do I need a team?

You may participate individually or as a team. An individual submission can suit a focused technical solution, prototype, evaluation method, or research project. A team may help when you need expertise across AI engineering, security, testing, or user experience. There is no maximum team size; we recommend up to five members.

4. What can I submit?

You can submit an existing project, an improvement to an existing solution, or a completely new approach. Including work already developed, projects in progress, or new ideas before the deadline. Your submission should address preventing AI agents from taking unauthorized or harmful irreversible actions. Explain the problem, how your approach works, and how it helps prevent or control risky actions. Example approaches: • Action authorization • Human approval before high-impact operations • Policy enforcement • Risk assessment before execution • Reversibility and recovery • Monitoring and auditing • Testing and evaluation of safeguards • Agent architecture or tooling that limits what autonomous systems can do Other approaches are welcome if they address the challenge clearly. You may submit a reusable agent, Skill, Bot, plugin, MCP, custom build, test harness, implementation, or article/case study.

5. How is it judged?

Every submission is evaluated against the following criteria:

CriterionWeightWhat judges evaluate
Safety effectiveness35%Does the safeguard reliably prevent the targeted unauthorized or harmful action?
Testing and robustness25%Has the solution been tested against realistic failure scenarios and bypass attempts?
Real-world relevance20%Is the problem meaningful, and does the proposed safeguard address it?
Reproducibility and reuse15%Can another team understand, test, or adapt the solution?
Clarity of communication5%Are the implementation, evidence, and limitations clearly presented?
Total100%—

6. What should I include in my submission?

Strong submissions define a clear safety problem, explain how the approach works, provide evidence, test relevant failure scenarios, and document limitations honestly. Early-stage ideas and prototypes are welcome when you are transparent about what is implemented, what is tested, and what remains theoretical. Depending on maturity, consider including: • Project overview • Motivation • Technical explanation • Demonstration (prototype, screenshots, video, or workflow) • Testing and results • Reproducibility materials (code, setup, docs where permitted) • Known limitations Not every project needs all of these. Include enough for fair evaluation. A demo helps but is not required if you include other clear evidence.

7. Do I need to open-source my project?

No. You can submit a plugin, MCP connector, app, or an article/case study describing a solution in a closed system. If you share code or other materials, confirm you have permission to disclose them and remove credentials, personal data, or security-sensitive details. For closed systems, an article or case study that others can recreate is enough.

8. When is it due?

Submit by Nov 15, 2026. Winners are announced Nov 30, 2026.

9. What prizes or recognition are available?

The winning project will be featured on Drawerrr’s Winners page with recognition and support to take the idea further. Cash from the $500 student prize pool goes to eligible university and high school students; professionals compete for recognition, Winners page placement, and intros to the jury.

10. Will submissions be reviewed by experts?

Yes. Submissions are evaluated by a jury of industry experts using the published criteria. Jury members will be announced soon.

11. Can I update my submission after submitting it?

Yes. You can submit again with updated materials. Each submit creates a new record, so you can refine your link, description, or affiliation before the deadline.

12. Who owns the intellectual property in my submission?

You retain ownership of your work. By submitting, you grant Drawerrr a license to display and evaluate the submission for challenge operations, mentoring, and jury review. Do not include confidential material you are not authorized to share, and confirm you have rights to any employer, collaborator, or third-party content included.

Subscribe to challenge results

Our mission is to accelerate practical solutions to real-world AI safety challenges by connecting people, ideas, and expertise to build AI systems that are safer, more reliable, and easier to govern. Drawerrr’s AI safety challenge is informed by leading institutions in AI safety, standards, and public policy.

European AI OfficeAISINIST