Reviewer

Baz Reviewer performs a comprehensive code review of open pull requests in GitHub.

How it Works

Baz Reviewer uses a retrieval and analysis system to process code changes within the context of the entire codebase. Code is divided into manageable chunks using a LangChain-based approach, with tree-sitter handling parsing for supported languages like Python. For embedding and similarity search, Baz relies on Voyage-Code-3, a model optimized for code representation. This setup enables Baz to analyze pull requests while accounting for dependencies and broader repository context, identifying issues such as breaking changes, outdated comments, and log errors.

Baz automates several steps in the code review process by integrating directly with GitHub. It evaluates outdated comments based on commit metadata and prior comment payloads, determining whether issues have been resolved. Log errors are identified by parsing GitHub Actions logs and attaching detailed comments at the relevant lines. Baz also identifies specific issues like typos, generic variable names, missing test assertions, and type mismatches. These insights are delivered as structured comments, enabling developers to address them directly in the GitHub interface.

The system is designed for efficient processing and scalability. Repository and organization data are stored in a single multi-tenant table, filtered by organization ID, repository name, and file path. Embeddings are stored in a pgvector database, enabling similarity searches to locate relevant code sections. When files are updated, Baz reprocesses only the changed files, ensuring minimal overhead while maintaining up-to-date insights. This approach supports a wide range of use cases and scales to handle large repositories effectively.

Data Model

Baz Reviewer’s is underpinned by a structured data model that maps API elements to their corresponding code definitions. This model ensures that Baz can accurately trace, detect, and evaluate changes across the codebase.

1. API Endpoint Mapping Each API endpoint is identified and linked to its corresponding entry point in the code, such as function definitions or class methods. This mapping is established through in-file connections (e.g., callables linked to function definitions) and cross-module imports.

2. Parameter and Return Type Linking Function parameters and return types are traced back to their definitions, accounting for alias imports, re-exports, and class hierarchies. This linkage supports complex data structures like destructured TypeScript parameters or JSON payloads in Rust.

3. Change Identification Every change in the codebase is associated with an element ID that corresponds to a function, parameter, or return type. Baz identifies the enclosing range of the change and evaluates whether the affected element is API-related.

4. Change Evaluation Changes are evaluated using an LLM, with Baz determining whether a modification is breaking. The data model supports both naïve and pre-processed approaches, such as recursively checking relationships or pre-marking API-relevant elements.

Working with Reviewer

Baz Reviewer identifies and comments on a range of issues within code changes to streamline reviews. It starts by analyzing outdated comments, using commit history to determine if previous feedback has been addressed. For example, it might note, "This comment has been addressed. 🤘" or, if unresolved, "This comment hasn’t been addressed yet. ✋."

Log errors from GitHub Actions are another focus. Baz pinpoints the exact line of failure and provides a concise comment such as, "A failed check needs your attention. 🛠 I’ve attached the log error for your convenience." Breaking changes are flagged with contextual details, including code snippets and descriptions like, "Breaking change identified for API. ⚠️."

Smaller but crucial issues like typos and generic variable names are flagged with suggestions, such as, "Did you mean corrected_name?" or, "Consider changing generic_name to something more descriptive." It also identifies commented-out code, recommending cleanup with comments like, "There is commented-out code in the file(s) X, Y, Z. Consider removing it."

For tests, Baz highlights missing or removed assertions, suggesting improvements like, "Consider adding assertions—currently some cases might be missed." It also flags type change mistakes, directing reviewers to the relevant lines with explanations about the discrepancy. Each of these findings is designed to provide clear, actionable feedback, ensuring reviews are thorough and efficient.

Examples

Baz Reviewer identifies a range of issues to help maintain software stability and compatibility during ongoing development. These include:

Endpoint Modifications

Changes such as removing or renaming API endpoints can render existing integrations invalid. Baz detects these modifications by analyzing endpoint definitions and their usage across the codebase.

HTTP Method Changes

Altering or removing HTTP methods for an endpoint, such as changing POST to GET, can break client expectations and lead to errors. Baz flags these changes to ensure consistent API behavior.

Parameter Adjustments

Modifications to parameters, such as renaming, removing, or making a previously optional parameter required, can disrupt how clients interact with the API. Baz tracks these adjustments and highlights their impact.

Response Changes

Altering the structure or type of response fields, such as changing a string to an integer, can cause downstream failures. Baz identifies such changes to help developers address them proactively.

Enum Adjustments

Removing or renaming enum values used in API inputs or outputs can lead to invalid requests or unexpected client behavior. Baz detects these changes and flags their potential impact.

Validation Rules

Adding stricter validation rules to parameters, such as introducing new constraints or formats, can reject previously valid requests. Baz analyzes these rule changes to ensure compatibility.

Authorization Changes

Modifying authentication or authorization requirements, such as requiring additional tokens or altering roles, can block access for clients relying on older configurations. Baz surfaces these changes for review.

Behavior Changes

Subtle logic changes, such as modifying the processing order or introducing new side effects, can alter expected outcomes without changing the API signature. Baz captures these behavior shifts to reduce unintended consequences.