Benchmark explorer

Back to the report

Role definitions and legend
How to read this
Roles and models
RoleModel classTools the role may call
builderbuilder (qwen2.5-coder-yarn:3b)read_file, write_file, edit_file, search_files, search_graph, get_code_snippet
databuilder (qwen2.5-coder-yarn:3b)read_file, list_directory
analyzerthinker (llama3.2:3b-16k)read_file, search_graph, get_code_snippet, trace_path, git_diff, ast-grep
securitythinker (llama3.2:3b-16k)read_file, search_files, ast-grep, search_sonar_issues_in_projects, search_security_hotspots, show_rule
reviewerthinker (llama3.2:3b-16k)read_file, git_diff, get_code_snippet, trace_path, ast-grep
debuggerthinker (llama3.2:3b-16k)read_file, search_graph, trace_path, git_log, sequentialthinking, execute_code
docsthinker (llama3.2:3b-16k)read_file, list_directory, git_log, fetch
designerthinker (llama3.2:3b-16k)high_level_overview, penpot_api_info, execute_code, export_shape
browserthinker (llama3.2:3b-16k)navigate_page, take_snapshot, take_screenshot, list_console_messages, list_network_requests, lighthouse_audit
trackerthinker (llama3.2:3b-16k)plan_next, plan_claim, plan_status, plan_blockers, plan_done, plan_note
github-readthinker (llama3.2:3b-16k)list_pull_requests, pull_request_read, issue_read, list_issues, get_file_contents
github-actions-readthinker (llama3.2:3b-16k)actions_list, actions_get, get_job_logs
devopsbuilder (qwen2.5-coder-yarn:3b)read_file, write_file, edit_file, git_diff, git_log, execute_code
financethinker (llama3.2:3b-16k)read_file, list_directory, fetch, web_search, crawl_markdown
researchthinker (llama3.2:3b-16k)web_search, crawl_markdown, crawl_links

Column groups: builder cases = write_function, fix_bug, write_tests, rename_across_files. Thinker cases = find_symbol, review_sqli, debug_trace, summarize_readme, code_graph_search, list_project_files, read_docs_file, security_file_read, design_overview, browser_navigate_snapshot, plan_status, web_crawl.

Charts (linked to the filters and table)