Reviewer
Baz Reviewer performs a comprehensive code review of open pull requests in GitHub.
Last updated
Baz Reviewer performs a comprehensive code review of open pull requests in GitHub.
Last updated
Baz Reviewer uses a retrieval and analysis system to process code changes within the context of the entire codebase. Code is divided into manageable chunks using a LangChain-based approach, with tree-sitter handling parsing for supported languages like Python. For embedding and similarity search, Baz relies on Voyage-Code-3, a model optimized for code representation. This setup enables Baz to analyze pull requests while accounting for dependencies and broader repository context, identifying issues such as breaking changes, outdated comments, and log errors.
Baz automates several steps in the code review process by integrating directly with GitHub. It evaluates outdated comments based on commit metadata and prior comment payloads, determining whether issues have been resolved. Log errors are identified by parsing GitHub Actions logs and attaching detailed comments at the relevant lines. Baz also identifies specific issues like typos, generic variable names, missing test assertions, and type mismatches. These insights are delivered as structured comments, enabling developers to address them directly in the GitHub interface.
The system is designed for efficient processing and scalability. Repository and organization data are stored in a single multi-tenant table, filtered by organization ID, repository name, and file path. Embeddings are stored in a pgvector database, enabling similarity searches to locate relevant code sections. When files are updated, Baz reprocesses only the changed files, ensuring minimal overhead while maintaining up-to-date insights. This approach supports a wide range of use cases and scales to handle large repositories effectively.
Baz Reviewer’s is underpinned by a structured data model that maps API elements to their corresponding code definitions. This model ensures that Baz can accurately trace, detect, and evaluate changes across the codebase.
API Endpoint Mapping Each API endpoint is identified and linked to its corresponding entry point in the code, such as function definitions or class methods. This mapping is established through in-file connections (e.g., callables linked to function definitions) and cross-module imports.
Parameter and Return Type Linking Function parameters and return types are traced back to their definitions, accounting for alias imports, re-exports, and class hierarchies. This linkage supports complex data structures like destructured TypeScript parameters or JSON payloads in Rust.
Change Identification Every change in the codebase is associated with an element ID that corresponds to a function, parameter, or return type. Baz identifies the enclosing range of the change and evaluates whether the affected element is API-related.
Change Evaluation Changes are evaluated using an LLM, with Baz determining whether a modification is breaking. The data model supports both naïve and pre-processed approaches, such as recursively checking relationships or pre-marking API-relevant elements.
Baz now allows organization admins to configure Reviewer workflows, enabling or disabling specific checks to better align with their team’s review process.
How to Configure Workflows:
Navigate to Baz Settings → Preferences
Under Customize Reviewer Workflows, toggle workflows on or off based on your team’s needs
Changes apply to all reviewers in the organization
Additionally, you can also specify paths you would like to exclude using Glob patterns.
Important Notes:
Breaking Changes and CI Errors workflows cannot be disabled, as they highlight critical risks.
Baz Reviewer analyzes extended diffs and leverages the AST of each programming language to identify non-trivial issues, such as:
Highlights addressed and unresolved comments for improved visibility.
Identifies breaking changes to prevent potential disruptions.
Detects typos and overly generic names for clearer, more professional code.
Ensures proper code hygiene by adhering to best practices.
Verifies that tests include meaningful assertions.
Flags type inconsistencies to maintain code integrity and prevent errors.
Recognizes opportunities to rewrite code more concisely for better readability and efficiency.
Reviewer also analyzes every commit to determine if previous feedback has been addressed. For example, it might note, "This comment has been addressed. 🤘" or, if unresolved, "This comment hasn’t been addressed yet. ✋."
Log errors from GitHub Actions are another focus. Baz pinpoints the exact line of failure and provides a concise comment such as, "A failed check needs your attention. 🛠 I’ve attached the log error for your convenience." Breaking changes are flagged with contextual details, including code snippets and descriptions like, "Breaking change identified for API. ⚠️."
Smaller but crucial issues like typos and generic variable names are flagged with suggestions, such as, "Did you mean corrected_name?" or, "Consider changing generic_name to something more descriptive." It also identifies commented-out code, recommending cleanup with comments like, "There is commented-out code in the file(s) X, Y, Z. Consider removing it."
For tests, Baz highlights missing or removed assertions, suggesting improvements like, "Consider adding assertions—currently some cases might be missed." It also flags type change mistakes, directing reviewers to the relevant lines with explanations about the discrepancy. Each of these findings is designed to provide clear, actionable feedback, ensuring reviews are thorough and efficient.
Just like a human reviewer, you can ask follow up questions, request clarifications or even correct Reviewer if you believe it made a mistake. Simply start a chat with any Reviewer finding to start a conversation. Just as any comment, you can select to interact with Reviewer direcrly in GitHub or in Baz.
Baz Reviewer also identifies a range of issues to help maintain software stability and compatibility during ongoing development. These include:
Changes such as removing or renaming API endpoints can render existing integrations invalid. Baz detects these modifications by analyzing endpoint definitions and their usage across the codebase.
Altering or removing HTTP methods for an endpoint, such as changing POST to GET, can break client expectations and lead to errors. Baz flags these changes to ensure consistent API behavior.
Modifications to parameters, such as renaming, removing, or making a previously optional parameter required, can disrupt how clients interact with the API. Baz tracks these adjustments and highlights their impact.
Altering the structure or type of response fields, such as changing a string to an integer, can cause downstream failures. Baz identifies such changes to help developers address them proactively.
Removing or renaming enum values used in API inputs or outputs can lead to invalid requests or unexpected client behavior. Baz detects these changes and flags their potential impact.
Adding stricter validation rules to parameters, such as introducing new constraints or formats, can reject previously valid requests. Baz analyzes these rule changes to ensure compatibility.
Modifying authentication or authorization requirements, such as requiring additional tokens or altering roles, can block access for clients relying on older configurations. Baz surfaces these changes for review.
Subtle logic changes, such as modifying the processing order or introducing new side effects, can alter expected outcomes without changing the API signature. Baz captures these behavior shifts to reduce unintended consequences.
