(APIServer pid=1) INFO 09-18 02:55:12 [api_utils.py:347] (APIServer pid=1) INFO 09-18 02:55:12 [api_utils.py:347] █ █ █▄ ▄█ (APIServer pid=1) INFO 09-18 02:55:12 [api_utils.py:347] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.3.1.dev3+g0bfc7a15d (APIServer pid=1) INFO 09-18 02:55:12 [api_utils.py:347] █▄█▀ █ █ █ █ model /model (APIServer pid=1) INFO 09-18 02:55:12 [api_utils.py:347] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ (APIServer pid=1) INFO 09-18 02:55:12 [api_utils.py:347] (APIServer pid=1) INFO 09-18 02:55:12 [api_utils.py:286] non-default args: {'model_tag': '/model', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_xml', 'host': '0.0.0.0', 'api_key': '***', 'model': '/model', 'dtype': 'bfloat16', 'max_model_len': 131072, 'served_model_name': ['qwen3.8-flash-next'], 'load_format': 'safetensors', 'reasoning_parser': 'qwen3', 'gpu_memory_utilization': 0.985, 'kv_cache_dtype': 'fp8', 'enable_prefix_caching': True, 'max_num_batched_tokens': 2048, 'max_num_seqs': 1, 'enable_chunked_prefill': True, 'enable_flashinfer_autotune': False, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 2}, 'compilation_config': {'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': [], 'ir_enable_torch_wrap': None, 'splitting_ops': None, 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': None, 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [1, 3], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': None, 'pass_config': {}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': None, 'static_all_moe_layers': []}, 'engram_config': EngramConfig(cpu_offload=True, embedding_across_dp=False, dp_shared_memory=False)} (APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (APIServer pid=1) INFO 09-18 02:55:12 [model.py:691] Resolved architecture: Qwen4ExpForConditionalGeneration (APIServer pid=1) INFO 09-18 02:55:12 [model.py:2024] Using max model len 131072 (APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (APIServer pid=1) INFO 09-18 02:55:15 [cache.py:345] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor (APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (APIServer pid=1) INFO 09-18 02:55:15 [model.py:691] Resolved architecture: Qwen4ExpMTP (APIServer pid=1) INFO 09-18 02:55:15 [model.py:2024] Using max model len 262144 (APIServer pid=1) WARNING 09-18 02:55:15 [speculative.py:1360] Enabling num_speculative_tokens > 1 will run multiple times of forward on same MTP layer,which may result in lower acceptance rate (APIServer pid=1) INFO 09-18 02:55:15 [speculative.py:1653] Overriding draft model max model len from 262144 to 131072 (APIServer pid=1) INFO 09-18 02:55:15 [config.py:625] Mamba cache mode is set to 'align' for Qwen4ExpForConditionalGeneration by default when prefix caching is enabled (APIServer pid=1) INFO 09-18 02:55:15 [vllm.py:1271] Resolved Engram configuration: EngramConfig(cpu_offload=True, embedding_across_dp=False, dp_shared_memory=False) (APIServer pid=1) INFO 09-18 02:55:15 [vllm.py:781] Auto-enabling VLLM_USE_BREAKABLE_CUDAGRAPH=1. Set VLLM_USE_BREAKABLE_CUDAGRAPH=0 to opt out. (APIServer pid=1) INFO 09-18 02:55:15 [kernel.py:408] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'], gelu_and_mul_sparse=['triton', 'native']) (APIServer pid=1) WARNING 09-18 02:55:15 [vllm.py:2152] max_num_scheduled_tokens is set to 2048 based on the speculative decoding settings. This may lead to suboptimal performance. Consider increasing max_num_batched_tokens to accommodate the additional draft token slots, or decrease num_speculative_tokens. (APIServer pid=1) INFO 09-18 02:55:15 [compilation.py:331] Enabled custom fusions: norm_quant, act_quant (APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (APIServer pid=1) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`. (APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (EngineCore pid=112) INFO 09-18 02:55:28 [core.py:123] Initializing a V1 LLM engine (v0.3.1.dev3+g0bfc7a15d) with config: model='/model', speculative_config=SpeculativeConfig(method='mtp', model='/model', num_spec_tokens=2), tokenizer='/model', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=safetensors, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=modelopt_mixed, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=qwen3.8-flash-next, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+quant_fp8', 'all', '+quant_fp8'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 3], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 3, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) (APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (EngineCore pid=112) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (EngineCore pid=112) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (EngineCore pid=112) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (EngineCore pid=112) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (EngineCore pid=112) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (EngineCore pid=112) INFO 09-18 02:55:29 [parallel_state.py:1825] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_12361652d17045ad82ebf3d8d70939b6 backend=nccl (EngineCore pid=112) INFO 09-18 02:55:29 [parallel_state.py:2269] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank 0, EPLB rank N/A (EngineCore pid=112) INFO 09-18 02:55:29 [gpu_worker.py:441] Using V2 Model Runner (EngineCore pid=112) INFO 09-18 02:55:30 [model_runner.py:387] Loading model from scratch... (EngineCore pid=112) INFO 09-18 02:55:30 [cuda.py:595] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention (EngineCore pid=112) INFO 09-18 02:55:30 [mm_encoder_attention.py:372] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention. (EngineCore pid=112) INFO 09-18 02:55:30 [qwen_gdn_linear_attn.py:176] Using FlashInfer GDN prefill kernel (requested=auto, head_k_dim=128). (EngineCore pid=112) INFO 09-18 02:55:30 [qwen_gdn_linear_attn.py:528] GDN decode kernel: cuda (EngineCore pid=112) INFO 09-18 02:55:32 [nvfp4.py:302] Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend out of potential backends: ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTEDSL_BATCHED', 'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'HUMMING', 'EMULATION']. (APIServer pid=1) [transformers] Qwen3VL video processing does not apply the per-frame pixel cap the reference implementation (qwen-vl-utils) applies, so some videos cost far more tokens than they would there. In v5.22 the capped behavior will become the default and `cap_pixels_per_frame` will be removed. Pass `cap_pixels_per_frame=True` to adopt the reference behavior now, or `False` to keep the current behavior and silence this warning. (APIServer pid=1) INFO 09-18 02:55:34 [base.py:261] Multi-modal warmup completed in 12.188s (APIServer pid=1) INFO 09-18 02:55:35 [base.py:261] Readonly multi-modal warmup completed in 1.309s (EngineCore pid=112) INFO 09-18 02:56:05 [ngram_embedding.py:720] Initialized PLE embedding language_model.model.layers.1.ple.ple_embedding.ngram_embedding: quantization_method=Qwen4ExpPLEFp8EmbeddingMethod, weight_dtype=torch.float8_e4m3fn, weight_device=cpu, pinned=True (EngineCore pid=112) INFO 09-18 02:56:05 [flash_attn.py:1115] Using FlashAttention version 2 (EngineCore pid=112) WARNING 09-18 02:56:06 [compilation.py:1350] Op 'quant_fp8' not present in model, enabling with '+quant_fp8' has no effect (EngineCore pid=112) INFO 09-18 02:56:06 [weight_utils.py:900] Filesystem type for checkpoints: EXT4. Checkpoint size: 123.57 GiB. Available RAM: 142.29 GiB. (EngineCore pid=112) INFO 09-18 02:56:06 [weight_utils.py:923] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. (EngineCore pid=112) Loading safetensors checkpoint shards: 0% Completed | 0/11 [00:00= mamba page size. (EngineCore pid=112) INFO 09-18 02:56:57 [interface.py:942] Padding mamba page size by 0.38% to ensure that mamba page size and attention page size are exactly equal. (EngineCore pid=112) INFO 09-18 02:56:57 [utils.py:320] Using BLNHC KV cache layout. (EngineCore pid=112) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (EngineCore pid=112) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (EngineCore pid=112) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`. (EngineCore pid=112) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (EngineCore pid=112) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'} (EngineCore pid=112) INFO 09-18 02:57:01 [encoder_runner.py:131] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size. (EngineCore pid=112) WARNING 09-18 02:57:23 [kv_cache_utils.py:2219] Speculative decoding (method=mtp) is enabled but no KV cache group could be identified as the draft model's. (EngineCore pid=112) WARNING 09-18 02:57:23 [compilation.py:1415] CUDAGraphMode.FULL is not supported with GDNAttentionBackend backend (support: AttentionCGSupport.UNIFORM_BATCH); setting cudagraph_mode=FULL_DECODE_ONLY (EngineCore pid=112) INFO 09-18 02:57:23 [speculator.py:119] Fused multi-step draft decode is not supported by attention backend(s) QWEN4_EXP_EXP_QSA_STATE; falling back to rebuilding attention metadata between draft steps. (EngineCore pid=112) Capturing CUDA graphs (FULL): 0%| | 0/1 [00:00 262144). Running this sequence through the model will result in indexing errors (APIServer pid=1) INFO: 172.21.0.1:47602 - "POST /tokenize HTTP/1.1" 200 OK (APIServer pid=1) INFO: 172.21.0.1:47608 - "POST /tokenize HTTP/1.1" 200 OK (APIServer pid=1) INFO: 172.21.0.1:47610 - "POST /tokenize HTTP/1.1" 200 OK (APIServer pid=1) INFO: 172.21.0.1:47624 - "POST /tokenize HTTP/1.1" 200 OK (APIServer pid=1) INFO: 172.21.0.1:47634 - "POST /tokenize HTTP/1.1" 200 OK (APIServer pid=1) INFO: 172.21.0.1:47640 - "POST /tokenize HTTP/1.1" 200 OK (APIServer pid=1) INFO: 172.21.0.1:47650 - "POST /tokenize HTTP/1.1" 200 OK (APIServer pid=1) INFO: 172.21.0.1:47662 - "POST /tokenize HTTP/1.1" 200 OK (APIServer pid=1) INFO: 172.21.0.1:47678 - "POST /tokenize HTTP/1.1" 200 OK (APIServer pid=1) INFO: 172.21.0.1:47688 - "POST /tokenize HTTP/1.1" 200 OK (APIServer pid=1) INFO: 172.21.0.1:47698 - "POST /tokenize HTTP/1.1" 200 OK (APIServer pid=1) INFO: 172.21.0.1:47712 - "POST /tokenize HTTP/1.1" 200 OK (APIServer pid=1) INFO: 172.21.0.1:47716 - "POST /tokenize HTTP/1.1" 200 OK (APIServer pid=1) INFO: 172.21.0.1:47726 - "POST /tokenize HTTP/1.1" 200 OK (APIServer pid=1) INFO: 172.21.0.1:47732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 09-18 02:58:05 [loggers.py:323] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 60.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0% (APIServer pid=1) INFO 09-18 02:58:05 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.77, Accepted throughput: 6.20 tokens/s, Drafted throughput: 7.00 tokens/s, Accepted: 62 tokens, Drafted: 70 tokens, Per-position acceptance rate: 0.971, 0.800, Avg Draft acceptance rate: 88.6% (APIServer pid=1) INFO: 172.21.0.1:36086 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 172.21.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 09-18 02:58:15 [loggers.py:323] Engine 000: Avg prompt throughput: 13184.9 tokens/s, Avg generation throughput: 34.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.5%, Prefix cache hit rate: 48.3%, MM cache hit rate: 0.0% (APIServer pid=1) INFO 09-18 02:58:15 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.73, Accepted throughput: 21.60 tokens/s, Drafted throughput: 25.00 tokens/s, Accepted: 216 tokens, Drafted: 250 tokens, Per-position acceptance rate: 0.936, 0.792, Avg Draft acceptance rate: 86.4% (APIServer pid=1) INFO: 172.21.0.1:33248 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:45376 - "GET /health HTTP/1.1" 200 OK (APIServer pid=1) INFO: 172.21.0.1:33262 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 172.21.0.1:33272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 09-18 02:58:25 [loggers.py:323] Engine 000: Avg prompt throughput: 17.4 tokens/s, Avg generation throughput: 79.7 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 48.3%, MM cache hit rate: 0.0% (APIServer pid=1) INFO 09-18 02:58:25 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.68, Accepted throughput: 49.80 tokens/s, Drafted throughput: 59.20 tokens/s, Accepted: 498 tokens, Drafted: 592 tokens, Per-position acceptance rate: 0.926, 0.757, Avg Draft acceptance rate: 84.1% (APIServer pid=1) INFO 09-18 02:58:35 [loggers.py:323] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 48.3%, MM cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:60766 - "GET /health HTTP/1.1" 200 OK