Noah Shinn

Contact

San Francisco, CA
Email: noahrshinn[at]gmail[dot]com
Github
Google scholar

About

I am a research scientist at Sierra. Previously, I worked on type prediction and code generation in the Programming Language Group at Northeastern University, and excited-state molecular dynamics in the Computational Photochemistry Group. I was also a member of the avionics team in the NU Aerospace group.

Papers

τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Shunyu Yao, Noah Shinn, Pedram Razavi, Karthik Narasimhan
In submission, NeurIPS 2024
paper | code
Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions
Federico Cassano, Luisa Li, Akul Sethi, Noah Shinn, Abby Brennan-Jones, Edward Berman, George Chakhnashvili, Anton Lozhkov, Carolyn Anderson, Arjun Guha
Conference on Language Modeling (COLM) 2024
paper | code
Type Prediction With Program Decomposition and Fill-in-the-Type Training
Federico Cassano, Ming-Ho Yee, Noah Shinn, Arjun Guha, Steven Holtzen
Arxiv
paper | code
Reflexion: Language Agents with Verbal Reinforcement Learning
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao
Neural Information Processing Systems (NeurIPS) 2023
paper | code

Other

web-browser: A tool that automates web search, traversal, and extraction of information on unstructured web pages
public-web: A prototype of a public network of APIs
gitm: A search tool for git and Github
The Collaborative Browser: a web browser for all users
llm-chat: Question answering in your terminal
gcl: Tool for git commits and GitHub issues
bashllm: Tool for auto cli suggestions
A blog post written with Ashwin Gopinath
kstateagent: Classification + q-learning written in OCaml and Rust
erica: A project from a hackathon written in Rust and Python
Leetcode Hard Gym: A hard benchmark for programming tests

Last updated: 12/2024