
AI Safety Challenge
Share a solution that prevents AI agents from taking unauthorized, irreversible actions in the real world setup.
Share your work:
Submissions open to professionals and students.

Across 17 frontier LLM agents, agents continued acting in 40.5%* of cases, even when harm was possible.
* Do LLM agents know when not to act? ↗Tool-using AI agents often fail to recognize when they should stop. Under ambiguity, conflicting constraints, or tool failures, they may take irreversible actions before identifying the risk, and a later refusal cannot undo the harm.
Oct 15 26’
Challenge start
Submit (12 days left)
Nov 15 26’
Deadline
Meet your jury
Your work will be evaluated
by industry experts.
More jury members will be announced as they confirm. Read evaluation criteria
Top solutions
Winners
Best overall in safeguarding irreversible actions
The strongest overall example of safe, responsible agentic action.
Project name
Author’s names
1st
Project name
Author’s names
2nd
Project name
Author’s names
3rd
Project name
Author’s names
Finalist
Project name
Author’s names
Finalist
Project name
Author’s names
Finalist
Project name
Author’s names
Finalist
Project name
Author’s names
Finalist
Project name
Author’s names
Finalist
Project name
Author’s names
Finalist
Best in safeguarding delegated actions
Best in safely enabling AI agents to communicate, transact, or act for people and organizations.
Best in safeguarding physical-world actions
Best in safely controlling AI agents that operate drones, robots, equipment, or industrial systems.
Best in safeguarding consequential decisions
Best in keeping AI agents from making consequential decisions without appropriate human or policy controls.
Best in safeguarding information access
Best in controlling what AI agents can access, retrieve, expose, or transmit.
Best in safeguarding digital system changes
Best in safely controlling AI agents that modify code, data, infrastructure, or system state.
Best in safeguarding financial transactions
Best in safely enabling AI agents to trade, spend, transfer, or authorize funds.
Winners will be featured on Drawerrr’s Winners page.
The Top 10 projects will be recognised across award categories.
Special prizes
Top Students
2nd place
$150
1st place
$300
3rd place
$50
The $500 cash prize pool is for university and high school students.
Top Students will be selected from the finalist projects and receive special prizes.
Oct 15 26’
Challenge start
Submit (12 days left)
Nov 15 26’
Deadline
How to succeeded
Master the Challenge in 5 days

Day 1
Identify high-risk scenarios
Choose an environment you can access or control, and identify a specific action an agent could take that would be unauthorized, harmful, costly, irreversible, or difficult to undo.
Tip:
Start with familiar examples like sending an email, deploying code, or issuing a refund, or use Challenge Kick-off to identify promising approaches.
Learn more:
- Google SAIF — Agent Risk Self-Assessment
- Ask our experts in
Discordchannel
Day 2
Set up a test environment
Recreate the scenario in an environment you control. Give the agent only the access it needs to demonstrate the risk.
Tip:
Use a mock API or sandbox service to simulate real actions safely, or use Challenge Kick-off to find an environment that fits your project.
Learn more:
- Google SAIF — Agent Permissions
- Ask our experts in
Discordchannel


Day 3
Choose a safeguard stack
Combine one or multiple layers to create a safeguard stack. A safety layer could address identity and access, data leakage, action authorization, human approval, monitoring, or fail-safe controls.
Tip:
Explore tools like Guardrails AI and LangGraph, or use Challenge Kick-off to identify safeguards that fit your approach.
Learn more:
- Google SAIF — Security Controls
- Ask our experts in
Discordchannel
Day 4
Document failure states
Test ambiguous instructions, missing information, prompt injection, unauthorized requests, unusual edge cases, and attempts to bypass the approval mechanism and log results.
Tip:
Test expired approvals, malformed tool arguments, prompt injection, and timeouts, or use Challenge Kick-off to identify failure modes and plan how to trace them.
Learn more:
- Google SAIF — Red Teaming and Adversarial Testing
- Ask our experts in
Discordchannel


Day 5
Make it reproducible
Package your solution so someone else can reproduce, test, and adapt it. Share the code, configuration, test setup, and instructions needed to run it in a similar environment.
Tip:
Include a README with setup instructions and example usage to help others run your project, or use Challenge Kick-off to find ways to make your solution easier to reproduce and reuse.
Support:
- Ask our experts in
Discordchannel
FAQ
Learn more about the challenge
1. What is the AI Safety and Control Challenge?
The AI Safety and Control Challenge invites participants to explore solutions that help prevent AI agents from taking unauthorized, unsafe, or harmful irreversible actions.
As AI systems become more capable of acting independently, it is increasingly important that they operate within clearly defined boundaries.
This challenge focuses on approaches that make autonomous systems safer, more controllable, and more reliable.
Potential areas include:
• Preventing unauthorized actions
• Introducing safeguards before critical operations
• Improving human oversight
• Making autonomous systems easier to monitor and control
2. Who can participate in the challenge?
The challenge welcomes students, working professionals, independent researchers, engineers, and others interested in AI safety and related fields.
You do not need to be affiliated with a university or school. The affiliation field on the submission form is optional.
Cash from the $500 student prize pool is reserved for eligible university and high school students.
Professionals can submit for evaluation and recognition, including a spot on the Winners page, but are not eligible for the student cash prizes.
If a submission involves an employer’s intellectual property or confidential information, make sure you have permission to share it.
3. Can I participate individually, or do I need a team?
You may participate individually or as a team.
An individual submission can suit a focused technical solution, prototype, evaluation method, or research project.
A team may help when you need expertise across AI engineering, security, testing, or user experience.
There is no maximum team size; we recommend up to five members.
4. What can I submit?
You can submit an existing project, an improvement to an existing solution, or a completely new approach.
Including work already developed, projects in progress, or new ideas before the deadline.
Your submission should address preventing AI agents from taking unauthorized or harmful irreversible actions. Explain the problem, how your approach works, and how it helps prevent or control risky actions.
Example approaches:
• Action authorization
• Human approval before high-impact operations
• Policy enforcement
• Risk assessment before execution
• Reversibility and recovery
• Monitoring and auditing
• Testing and evaluation of safeguards
• Agent architecture or tooling that limits what autonomous systems can do
Other approaches are welcome if they address the challenge clearly.
You may submit a reusable agent, Skill, Bot, plugin, MCP, custom build, test harness, implementation, or article/case study.
5. How is it judged?
Every submission is evaluated against the following criteria:
| Criterion | Weight | What judges evaluate |
|---|---|---|
| Safety effectiveness | 35% | Does the safeguard reliably prevent the targeted unauthorized or harmful action? |
| Testing and robustness | 25% | Has the solution been tested against realistic failure scenarios and bypass attempts? |
| Real-world relevance | 20% | Is the problem meaningful, and does the proposed safeguard address it? |
| Reproducibility and reuse | 15% | Can another team understand, test, or adapt the solution? |
| Clarity of communication | 5% | Are the implementation, evidence, and limitations clearly presented? |
| Total | 100% | — |
6. What should I include in my submission?
Strong submissions define a clear safety problem, explain how the approach works, provide evidence, test relevant failure scenarios, and document limitations honestly.
Early-stage ideas and prototypes are welcome when you are transparent about what is implemented, what is tested, and what remains theoretical.
Depending on maturity, consider including:
• Project overview
• Motivation
• Technical explanation
• Demonstration (prototype, screenshots, video, or workflow)
• Testing and results
• Reproducibility materials (code, setup, docs where permitted)
• Known limitations
Not every project needs all of these.
Include enough for fair evaluation.
A demo helps but is not required if you include other clear evidence.
7. Do I need to open-source my project?
No. You can submit a plugin, MCP connector, app, or an article/case study describing a solution in a closed system.
If you share code or other materials, confirm you have permission to disclose them and remove credentials, personal data, or security-sensitive details.
For closed systems, an article or case study that others can recreate is enough.
8. When is it due?
Submit by Nov 15, 2026. Winners are announced Nov 30, 2026.
9. What prizes or recognition are available?
The winning project will be featured on Drawerrr’s Winners page with recognition and support to take the idea further. Cash from the $500 student prize pool goes to eligible university and high school students; professionals compete for recognition, Winners page placement, and intros to the jury.
10. Will submissions be reviewed by experts?
Yes. Submissions are evaluated by a jury of industry experts using the published criteria.
More jury members will be announced as they confirm.
11. Can I update my submission after submitting it?
Yes. You can submit again with updated materials.
Each submit creates a new record, so you can refine your link, description, or affiliation before the deadline.
12. Who owns the intellectual property in my submission?
You retain ownership of your work.
By submitting, you grant Drawerrr a license to display and evaluate the submission for challenge operations, mentoring, and jury review.
Do not include confidential material you are not authorized to share, and confirm you have rights to any employer, collaborator, or third-party content included.
Subscribe to challenge results
Our mission is to accelerate practical solutions to real-world AI safety challenges by connecting people, ideas, and expertise to build AI systems that are safer, more reliable, and easier to govern. Drawerrr’s AI safety challenge is informed by leading institutions in AI safety, standards, and public policy.








