Zhiyuan Tao

Advisor: Prof. Wei-Lun Chao · Collaborator: Dr. Jianyang Gu

• Investigated whether hierarchical consistency in vision-language representations requires explicit parent–child constraints or can emerge from image–text alignment alone.
• Conceived and implemented level-restricted contrastive learning, which prevents valid labels at different semantic levels from acting as false negatives and balances optimization across granularities.
• Built on BioCLIP and trained on TreeOfLife-10M; improved hierarchical consistency in Euclidean and hyperbolic spaces, with a 30.47% relative gain in zero-shot average accuracy across levels over the prior SOTA (RCME) on iNat21.
• Led method development, implementation, experimental analysis, and contribution design; presented the work orally at the CVPR 2026 FGVC Workshop.

Multi-Granularity Vision–Language Learning

• Independently conceived and led, from formulation to writing, a controlled study of whether multi-granularity vision–language learning benefits from true semantic-level identities or from the candidate-pool structure induced by level restriction.
• Constructed VG-MG50K, a 50,000-image Visual Genome benchmark with four strictly nested captions per image, controlling semantic-level identity, candidate-pool balance, and training objective while preserving image–caption membership.
• Showed across ten paired seeds that balanced random partitions match true semantic levels in retrieval (equivalent within ±0.30 mean recall) once candidate-pool structure is matched, locating the restricted-softmax gain in balanced per-image pool construction.
• Developed a membership-only sigmoid objective that uses only image–caption membership (no level labels), reaching the strongest controlled result: 30.41 vs. 30.00 mean recall for level-restricted softmax.

Compositional Visual Execution

• Investigated why learned visual operations that are individually executable fail when composed into unseen programs, using controlled grid-to-grid visual reasoning tasks.
• Developed an input-aligned execution strategy that returns each intermediate grid through the visual interface consumed by the next operation, instead of keeping task state implicit in latent activations.
• Under parameter-matched training, raised exact accuracy on held-out primitive pairs from 43.5% to 88.2%, persisting on longer programs.
• Isolated the interface effect in a frozen executor: changing only the state handoff raised held-out-pair accuracy from 29.2% to 87.6%.
• Showed with probes and boundary controls that latent states encode the intermediate grid yet fail to support composition; extended results to grounded-object operations and demonstration-recovered programs.

Advisor: Prof. Qingyun Wang

• Formulated a controlled test of whether increasing the difficulty of an independent benign task makes vision-language models less safe under a fixed harmful request, via a benign-Task1 / harmful-Task2 framework.
• Built the full inference and evaluation pipeline across three VLMs, two safety benchmarks, and multiple benign task families.
• Found that benign-task performance can drop by up to 82.5 points while attack success changes by at most 2.8 points; matched protocol controls revealed that response formatting alone shifts ASR by 14–15 points on VLJailbreakBench.

Contributed to data-processing infrastructure for the TreeOfLife-200M biodiversity dataset; acknowledged by the dataset authors.

BioCLIP 2 project page →

The Ohio State University

Assisted with course instruction and student support.

The Ohio State University, Columbus, OH · Advisor: Wei-Lun Chao

Nanjing University of Posts and Telecommunications, Nanjing, China