Noah Shinn

Contact

San Francisco, CA
Email: noahrshinn[at]gmail[dot]com
Github
Google scholar

About

I am a research scientist at Sierra. Previously, I worked on equivariant graph neural networks for accelerating excited-state molecular dynamics programs, type prediction for programming languages, and code generation with large language models. I am broadly interested in programming language affordances, acquisition, and expressibility, and its influences on language modeling. (I am currently on-leave from my undergraduate studies at Northeastern University, where I studied computer science and chemistry.)

Papers

τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Shunyu Yao, Noah Shinn, Pedram Razavi, Karthik Narasimhan
In submission, NeurIPS 2024
paper | code
Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions
Federico Cassano, Luisa Li, Akul Sethi, Noah Shinn, Abby Brennan-Jones, Edward Berman, George Chakhnashvili, Anton Lozhkov, Carolyn Anderson, Arjun Guha
Conference on Language Modeling (COLM) 2024
paper | code
Type Prediction With Program Decomposition and Fill-in-the-Type Training
Federico Cassano, Ming-Ho Yee, Noah Shinn, Arjun Guha, Steven Holtzen
Arxiv
paper | code
Reflexion: Language Agents with Verbal Reinforcement Learning
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao
Neural Information Processing Systems (NeurIPS) 2023
paper | code

Other

pubapi-web: A prototype of a web of public APIs
gitm: A search tool for git and Github
The Collaborative Browser: a web browser for all users
llm-chat: Question answering in your terminal
gcl: Tool for git commits and GitHub issues
bashllm: Tool for auto cli suggestions
A blog post written with Ashwin Gopinath
kstateagent: Classification + q-learning written in OCaml and Rust
erica: A project from a hackathon written in Rust and Python
Leetcode Hard Gym: A hard benchmark for programming tests

Last updated: 09/2024