This is a concrete win: LLMs as circuit designers, not as general problem-solvers. The framework is simple, the results are measurable, and the tasks span real domains from chemistry to optimization. For builders exploring LLM-guided synthesis in other hardware domains, this is a template worth studying.

arXiv cs.AI · 10h ago

This is the first empirical proof that agent guard evaluations are language-specific in ways that matter. If your deployment touches non-English users, your guard's verdicts are not portable. Run the same audit on your guard stack now, especially if you inherit a guard from another vendor or a benchmark. The implication for vendors: safety evaluation needs to be multilingual from day one or your guard is selling false confidence.

arXiv cs.CL · 12h ago

This closes a real gap. Existing agent benchmarks test clean task specifications; real field deployment demands ambiguity handling, live system manipulation, and stakeholder coordination. The grading on customer success metric (not task completion) changes what winning means. If you're building agents for customer-facing work, this benchmark matters now.

arXiv cs.CL · 12h ago