Hey everyone,

I did a small personal benchmark on using local models to detect UI icons from application screenshots. English is not my first language, so sorry for any grammar mistakes! I just wanted to share what I found in case it helps someone doing similar stuff.

Models includes(none quantization):

• Gemma4-31B-it
• Qwen3.5-27B
• Qwen3.6-35B-A3B

Approach:

I feed the app screenshot into the LLM and ask it to recognize the UI icons and return the bbox_2d coordinates. After it gives me the coordinates, I use supervision to draw red bounding boxes on the image. Finally, I just check the results manually by eye.

For the setup, I used the newest vLLM v0.19.1 doing offline inference. I set the starting temperature to 0 because I want the most confident output. If the model returns 0 icons, I gradually increase the temperature: 0 -> 0.3 -> 0.6 -> 0.9.

Overall Results:

Overall, the Dense model is much better than the MoE model for this task. My ranking: Qwen3.5 > Qwen3.6 ≈ Gemma4

Some specific findings:

• Gemma4 and Qwen3.6 are both tied for last place. They are noticeably worse than Qwen3.5.
• Gemma4 completely failed on the Cursor IDE screenshot. I tried 4 times, everytime pushing the temperature all the way to 0.9, and it still couldn't detect a single icon.
• Qwen3.6 did something really funny on the Photoshop screenshot. It basically recognized the whole entire image as one giant icon and drew a massive box around the screen. 😅
• For the other app scenarios, you can check the comparison pictures below.

Here are the detail vllm parameters:

 - name: gemma-4-31B-it family: gemma4 params_b: 31 vllm_kwargs: model: google/gemma-4-31B-it tensor_parallel_size: 8 max_model_len: 8192 max_num_seqs: 1 gpu_memory_utilization: 0.85 limit_mm_per_prompt: image: 1 audio: 0 video: 0 mm_processor_cache_gb: 0 skip_mm_profiling: true mm_processor_kwargs: max_soft_tokens: 1120 - name: qwen3.5-27b family: qwen3.5 params_b: 27 vllm_kwargs: model: Qwen/Qwen3.5-27B tensor_parallel_size: 8 max_model_len: 32768 max_num_seqs: 1 gpu_memory_utilization: 0.9 limit_mm_per_prompt: image: 1 audio: 0 video: 0 mm_processor_cache_gb: 0 mm_encoder_tp_mode: data skip_mm_profiling: true - name: qwen3.6-35b-a3b family: qwen3.5 params_b: 35 vllm_kwargs: model: Qwen/Qwen3.6-35B-A3B tensor_parallel_size: 8 max_model_len: 32768 max_num_seqs: 1 gpu_memory_utilization: 0.9 limit_mm_per_prompt: image: 1 audio: 0 video: 0 mm_processor_cache_gb: 0 mm_encoder_tp_mode: data skip_mm_profiling: true 

Has anyone else tried UI element detection with local models recently? Curious if you guys have any tricks for getting better bounding boxes.