Review Instructions
Guidelines for CAIS 2026 Program Committee Members
About CAIS
CAIS 2026 is the first ACM conference dedicated to AI and Agentic Systems. These are AI systems that advance the state of the art through principled composition of modular components and specifications. This includes agents that orchestrate tool use, retrieval-augmented pipelines, multi-model ensembles, and systems that combine LLMs with symbolic code execution, verifiers, planners, and external knowledge. This has now become a dominant paradigm in deployed AI but, until now, no venue has been purpose-built for the research community studying and building these systems.
As a member of the program committee, your reviews will shape what this field counts as a contribution, what evidence is sufficient, and how you weigh novelty against impact. We ask you to approach this responsibility with care and openness.
What Makes CAIS Different
CAIS sits at the intersection of systems (SOSP/OSDI/VLDB), machine learning (NeurIPS/ICML), natural language processing (ACL/EMNLP/SIGIR), software engineering (ICSE), programming languages (OOPSLA/ASPLOS), and security (CCS). It is none of these conferences. Reviewing for CAIS requires calibration that no existing conference's norms can provide.
1. Practical impact is a first-class contribution
A deployment study that reveals what actually works (or fails) in production has equal standing with a paper proposing a novel architecture. We value both algorithmic innovations and practical tradeoffs.
Impact that can lead to acceptance includes: advancing scientific understanding of agentic AI systems; novel tools, frameworks, or infrastructure; demonstrating what works and what doesn't with novel evidence; and establishing new research directions like new problems or novel abstractions.
2. The field is still defining itself
AI systems and compound or agentic systems as a research area are young and rapidly evolving. Be open to unconventional framings, mixed methodologies, and papers that don't fit neatly into existing categories. A paper that opens a new direction is valuable even if it doesn't conclusively resolve it.
3. Methodological breadth is expected
You will encounter formal verification, GPU kernel optimizations, user studies, benchmark construction, multi-agent protocol design, and production retrospectives, sometimes in the same review batch. You are not expected to be an expert in all of these. Report your confidence transparently.
4. Industry and academic perspectives are valued equally
Our PC includes professors and industry practitioners (from OpenAI, Oracle, AWS, IBM, Google, and others). Evaluate each paper on the merits of its own contribution type: a paper from an industry team describing deployment lessons learned is not "just engineering," and neither is an academic paper formalizing agent communication "just theory."
If a paper provides concrete, transferable lessons from building or operating an AI system in production, it is making a genuine research contribution in the evidence and the lessons, even if not in algorithmic invention. However, we do expect that papers will accurately identify their own contributions and your feedback should help authors do this well.
Research Pillars
CAIS 2026 organizes research broadly around five pillars. Many papers span multiple pillars.
| Pillar | % of Submissions | Topics |
|---|---|---|
| Architectural Patterns & Composition | 42% | Multi-agent systems, RAG, chains, tool use, orchestration patterns, memory architectures, agent communication protocols |
| Evaluation & Benchmarking | 18% | Benchmarks, metrics, human evaluation, LLM-as-judge, red teaming, testing methodology for compound systems |
| Security & Privacy | 16% | Prompt injection, data poisoning, guardrails, trust boundaries, privacy, alignment, safety of composed systems |
| System Optimization & Efficiency | 13% | Latency, throughput, cost optimization, caching, distillation, quantization, serving, scheduling |
| Engineering & Operations | 12% | MLOps, deployment, monitoring, CI/CD for AI, frameworks, reliability, developer tools |
Contribution Types & How to Assess Them
CAIS does not require authors to declare a fixed paper type, and you should not force papers into a single category. However, understanding the range of contributions helps you calibrate what evidence is appropriate for each.
1. Novel System Architectures
A new way of composing AI components to solve a problem.
Look for: Well-motivated problem. Principled composition (not arbitrary complexity). Evaluation showing that design choices matter, and ideally ablations or comparisons that isolate the effect of the architecture. Clear description of when this approach is and isn't appropriate.
2. System Optimizations
Improving the efficiency, cost, latency, or throughput of AI systems.
Look for: Rigorous measurement methodology. Practical significance (not just microbenchmark improvements). Fundamental insight into why the optimization works, not just that it does. Fair baselines.
3. Benchmark & Evaluation Methodology
New ways to measure compound AI system behavior.
Look for: Addressing a real evaluation gap. Sound task design with clear rationale. Community utility: will others actually use this? Informative baselines that contextualize results. Discussion of what the benchmark does and doesn't measure.
4. Production Deployment & Experience Report
Lessons from building or operating compound AI systems at scale.
Look for: Genuine lessons learned (not a product pitch). Concrete and specific details: architecture decisions, failure modes, performance characteristics. Transferable insight that others can apply. Honest discussion of what didn't work. Do NOT penalize for "lacking novelty." The novelty is in the evidence and the lessons.
5. Formal Methods, Verification & PL Approaches
Formal guarantees for compound AI system behavior.
Look for: Meaningful guarantees for realistic AI systems (not just toy settings). Bridges theory to practice, either by demonstrating on real systems or clearly arguing feasibility. Accessible framing for the compound AI audience.
6. Frameworks, Specifications & Protocols
Infrastructure for building, composing, or interoperating compound AI components.
Look for: Addressing a real interoperability or usability problem. Well-reasoned design choices with alternatives considered. Evidence of utility (adoption, case studies, or compelling argument). Not just "we built a wrapper."
7. Empirical Studies & Analyses
Systematic investigation of compound AI system behavior.
Look for: Clear research question stated up front. Sound methodology (appropriate controls, sufficient scale, valid measures). Actionable findings: what should practitioners or researchers do differently based on this? Transparent about limitations of the study design.
8. Security Attacks or Defenses
Identifying or mitigating vulnerabilities in compound AI systems.
Look for: Realistic threat model with clearly stated assumptions. For attacks: adaptive adversary evaluation (not just attacking a straw-man defense). For defenses: evaluation against adaptive adversaries (not just the attack that motivated the defense). Discussion of the arms-race dynamics.
Timeline & Rebuttal
| Reviews due | April 3, 2026 (AoE) |
| Rebuttal opens | April 6, 2026 |
| Rebuttal closes | April 17, 2026 (AoE) |
| Author notification | April 21, 2026 |
After reviews are submitted, there will be a brief author rebuttal period. The rebuttal is designed for a single response from authors to reviewer questions. Think of it as the place where misunderstandings get worked out. This means: write clear, specific questions. The more precise your questions, the more useful the rebuttal will be for everyone.
