This paper introduces DatBench, a novel evaluation framework designed to address the challenges in evaluating Vision-Language Models (VLMs). Existing evaluation methods suffer from issues such as multiple-choice formats that encourage guessing, questions solvable without images, and mislabeled or ambiguous samples, failing to accurately reflect a model's true capabilities. Furthermore, the computational cost of evaluation has become prohibitive. DatBench aims to resolve these issues by transforming and filtering existing benchmarks to enhance discriminability and faithfulness while improving computational efficiency. Specifically, converting multiple-choice questions to generative tasks revealed capability drops of up to 35%. Filtering out blindly solvable and mislabeled samples improved discriminative power and reduced computational cost. DatBench-Full comprises 33 datasets, and DatBench achieves an average 13x speedup (up to 50x) while closely matching the discriminative power of the original datasets. This work outlines a path toward rigorous and sustainable evaluation practices for scaling VLMs.

This study investigates whether Transformer models can mimic biological learning processes, specifically the early developmental stages of newborn visual systems. Unlike typical Transformers trained on biologically implausible datasets, this research simulated prenatal visual input using retinal waves and trained Transformers via self-supervised learning. The results showed that Transformers spontaneously self-organized in a manner analogous to newborn visual systems. Specifically, early layers became specialized in edge detection, later layers in shape detection, and receptive fields expanded across layers, mirroring biological developmental patterns. This convergence suggests that brains and Transformers may learn through common principles and fitting mechanisms.

This study presents the first comprehensive evaluation comparing AI agents against human cybersecurity professionals in a live enterprise environment. Ten cybersecurity professionals and six existing AI agents, along with ARTEMIS (a new agent scaffold), were evaluated on a large university network of approximately 8,000 hosts. ARTEMIS features dynamic prompt generation, arbitrary sub-agents, and automatic vulnerability triaging. In the comparative study, ARTEMIS ranked second overall, discovering 9 valid vulnerabilities with an 82% valid submission rate, outperforming 9 out of 10 human participants. While existing scaffolds like Codex and CyAgent underperformed relative to most humans, ARTEMIS demonstrated technical sophistication and submission quality comparable to the strongest participants. AI agents offer advantages in systematic enumeration, parallel exploitation, and cost (certain ARTEMIS variants cost $18/hour versus $60/hour for professional penetration testers). Key capability gaps were also identified, including higher false-positive rates and struggles with GUI-based tasks for AI agents.