(APIServer pid=1) INFO 09-18 14:54:04 [api_utils.py:347]
(APIServer pid=1) INFO 09-18 14:54:04 [api_utils.py:347]        █     █     █▄   ▄█
(APIServer pid=1) INFO 09-18 14:54:04 [api_utils.py:347]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.3.1.dev3+g0bfc7a15d
(APIServer pid=1) INFO 09-18 14:54:04 [api_utils.py:347]   █▄█▀ █     █     █     █  model   /model
(APIServer pid=1) INFO 09-18 14:54:04 [api_utils.py:347]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=1) INFO 09-18 14:54:04 [api_utils.py:347]
(APIServer pid=1) INFO 09-18 14:54:04 [api_utils.py:286] non-default args: {'model_tag': '/model', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_xml', 'host': '0.0.0.0', 'api_key': '***', 'model': '/model', 'dtype': 'bfloat16', 'max_model_len': 131072, 'served_model_name': ['qwen3.8-flash-next'], 'load_format': 'safetensors', 'reasoning_parser': 'qwen3', 'gpu_memory_utilization': 0.985, 'kv_cache_dtype': 'fp8', 'enable_prefix_caching': True, 'max_num_batched_tokens': 2048, 'max_num_seqs': 1, 'enable_chunked_prefill': True, 'enable_flashinfer_autotune': False, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 1}, 'compilation_config': {'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': [], 'ir_enable_torch_wrap': None, 'splitting_ops': None, 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': None, 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL: 2>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [1, 2], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': None, 'pass_config': {}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': None, 'static_all_moe_layers': []}, 'engram_config': EngramConfig(cpu_offload=True, embedding_across_dp=False, dp_shared_memory=False)}
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=1) INFO 09-18 14:54:04 [model.py:691] Resolved architecture: Qwen4ExpForConditionalGeneration
(APIServer pid=1) INFO 09-18 14:54:04 [model.py:2024] Using max model len 131072
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=1) INFO 09-18 14:54:07 [cache.py:345] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=1) INFO 09-18 14:54:07 [model.py:691] Resolved architecture: Qwen4ExpMTP
(APIServer pid=1) INFO 09-18 14:54:07 [model.py:2024] Using max model len 262144
(APIServer pid=1) INFO 09-18 14:54:07 [speculative.py:1653] Overriding draft model max model len from 262144 to 131072
(APIServer pid=1) INFO 09-18 14:54:07 [config.py:625] Mamba cache mode is set to 'align' for Qwen4ExpForConditionalGeneration by default when prefix caching is enabled
(APIServer pid=1) INFO 09-18 14:54:07 [vllm.py:1271] Resolved Engram configuration: EngramConfig(cpu_offload=True, embedding_across_dp=False, dp_shared_memory=False)
(APIServer pid=1) INFO 09-18 14:54:07 [vllm.py:781] Auto-enabling VLLM_USE_BREAKABLE_CUDAGRAPH=1. Set VLLM_USE_BREAKABLE_CUDAGRAPH=0 to opt out.
(APIServer pid=1) INFO 09-18 14:54:07 [kernel.py:408] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'], gelu_and_mul_sparse=['triton', 'native'])
(APIServer pid=1) WARNING 09-18 14:54:07 [vllm.py:2152] max_num_scheduled_tokens is set to 2048 based on the speculative decoding settings. This may lead to suboptimal performance. Consider increasing max_num_batched_tokens to accommodate the additional draft token slots, or decrease num_speculative_tokens.
(APIServer pid=1) INFO 09-18 14:54:07 [compilation.py:331] Enabled custom fusions: norm_quant, act_quant
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=1) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(EngineCore pid=112) INFO 09-18 14:54:20 [core.py:123] Initializing a V1 LLM engine (v0.3.1.dev3+g0bfc7a15d) with config: model='/model', speculative_config=SpeculativeConfig(method='mtp', model='/model', num_spec_tokens=1), tokenizer='/model', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=safetensors, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=modelopt_mixed, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=qwen3.8-flash-next, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+quant_fp8', 'all', '+quant_fp8'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL: 2>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 2, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None)
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(EngineCore pid=112) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
(EngineCore pid=112) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
(EngineCore pid=112) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
(EngineCore pid=112) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
(EngineCore pid=112) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
(EngineCore pid=112) INFO 09-18 14:54:21 [parallel_state.py:1825] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_d75d110cee7b4c8d8fc41c851dee4d8c backend=nccl
(EngineCore pid=112) INFO 09-18 14:54:21 [parallel_state.py:2269] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=112) INFO 09-18 14:54:21 [gpu_worker.py:441] Using V2 Model Runner
(EngineCore pid=112) INFO 09-18 14:54:22 [model_runner.py:387] Loading model from scratch...
(EngineCore pid=112) INFO 09-18 14:54:22 [cuda.py:595] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=112) INFO 09-18 14:54:22 [mm_encoder_attention.py:372] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=112) INFO 09-18 14:54:22 [qwen_gdn_linear_attn.py:176] Using FlashInfer GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=112) INFO 09-18 14:54:22 [qwen_gdn_linear_attn.py:528] GDN decode kernel: cuda
(EngineCore pid=112) INFO 09-18 14:54:24 [nvfp4.py:302] Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend out of potential backends: ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTEDSL_BATCHED', 'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'HUMMING', 'EMULATION'].
(APIServer pid=1) [transformers] Qwen3VL video processing does not apply the per-frame pixel cap the reference implementation (qwen-vl-utils) applies, so some videos cost far more tokens than they would there. In v5.22 the capped behavior will become the default and `cap_pixels_per_frame` will be removed. Pass `cap_pixels_per_frame=True` to adopt the reference behavior now, or `False` to keep the current behavior and silence this warning.
(APIServer pid=1) INFO 09-18 14:54:26 [base.py:261] Multi-modal warmup completed in 12.108s
(APIServer pid=1) INFO 09-18 14:54:27 [base.py:261] Readonly multi-modal warmup completed in 1.295s
(EngineCore pid=112) INFO 09-18 14:54:58 [ngram_embedding.py:720] Initialized PLE embedding language_model.model.layers.1.ple.ple_embedding.ngram_embedding: quantization_method=Qwen4ExpPLEFp8EmbeddingMethod, weight_dtype=torch.float8_e4m3fn, weight_device=cpu, pinned=True
(EngineCore pid=112) INFO 09-18 14:54:58 [flash_attn.py:1115] Using FlashAttention version 2
(EngineCore pid=112) WARNING 09-18 14:54:58 [compilation.py:1350] Op 'quant_fp8' not present in model, enabling with '+quant_fp8' has no effect
(EngineCore pid=112) INFO 09-18 14:54:58 [weight_utils.py:900] Filesystem type for checkpoints: EXT4. Checkpoint size: 123.57 GiB. Available RAM: 142.31 GiB.
(EngineCore pid=112) INFO 09-18 14:54:58 [weight_utils.py:923] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(EngineCore pid=112)
Loading safetensors checkpoint shards:   0% Completed | 0/11 [00:00<?, ?it/s]
(EngineCore pid=112)
Loading safetensors checkpoint shards:   9% Completed | 1/11 [00:01<00:17,  1.73s/it]
(EngineCore pid=112)
Loading safetensors checkpoint shards:  18% Completed | 2/11 [00:07<00:35,  3.98s/it]
(EngineCore pid=112)
Loading safetensors checkpoint shards:  27% Completed | 3/11 [00:13<00:38,  4.80s/it]
(EngineCore pid=112)
Loading safetensors checkpoint shards:  36% Completed | 4/11 [00:18<00:36,  5.19s/it]
(EngineCore pid=112)
Loading safetensors checkpoint shards:  45% Completed | 5/11 [00:24<00:32,  5.45s/it]
(EngineCore pid=112)
Loading safetensors checkpoint shards:  55% Completed | 6/11 [00:30<00:28,  5.61s/it]
(EngineCore pid=112)
Loading safetensors checkpoint shards:  64% Completed | 7/11 [00:36<00:22,  5.67s/it]
(EngineCore pid=112)
Loading safetensors checkpoint shards:  73% Completed | 8/11 [00:42<00:17,  5.74s/it]
(EngineCore pid=112)
Loading safetensors checkpoint shards:  82% Completed | 9/11 [00:43<00:08,  4.41s/it]
(EngineCore pid=112)
Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:44<00:00,  2.51s/it]
(EngineCore pid=112)
Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:44<00:00,  4.06s/it]
(EngineCore pid=112)
(EngineCore pid=112) INFO 09-18 14:55:43 [default_loader.py:430] Loading weights took 44.78 seconds
(EngineCore pid=112) INFO 09-18 14:55:43 [nvfp4.py:611] Using MoEPrepareAndFinalizeNoDPEPModular
(EngineCore pid=112) INFO 09-18 14:55:43 [vllm.py:1271] Resolved Engram configuration: EngramConfig(cpu_offload=True, embedding_across_dp=False, dp_shared_memory=False)
(EngineCore pid=112) INFO 09-18 14:55:43 [kernel.py:408] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'], gelu_and_mul_sparse=['triton', 'native'])
(EngineCore pid=112) WARNING 09-18 14:55:43 [vllm.py:2152] max_num_scheduled_tokens is set to 2048 based on the speculative decoding settings. This may lead to suboptimal performance. Consider increasing max_num_batched_tokens to accommodate the additional draft token slots, or decrease num_speculative_tokens.
(EngineCore pid=112) INFO 09-18 14:55:43 [compilation.py:331] Enabled custom fusions: norm_quant, act_quant
(EngineCore pid=112) INFO 09-18 14:55:44 [fp8.py:433] Using DEEPGEMM Fp8 MoE backend out of potential backends: ['AITER', 'FLASHINFER_TRTLLM', 'FLASHINFER_CUTLASS', 'DEEPGEMM', 'TRITON', 'MARLIN', 'HUMMING', 'BATCHED_DEEPGEMM', 'BATCHED_TRITON', 'XPU', 'CPU', 'HPC'].
(EngineCore pid=112) INFO 09-18 14:55:44 [weight_utils.py:900] Filesystem type for checkpoints: EXT4. Checkpoint size: 123.57 GiB. Available RAM: 142.17 GiB.
(EngineCore pid=112)
Loading safetensors checkpoint shards:   0% Completed | 0/11 [00:00<?, ?it/s]
(EngineCore pid=112)
Loading safetensors checkpoint shards:  18% Completed | 2/11 [00:00<00:01,  5.16it/s]
(EngineCore pid=112)
Loading safetensors checkpoint shards:  27% Completed | 3/11 [00:00<00:01,  4.04it/s]
(EngineCore pid=112)
Loading safetensors checkpoint shards:  36% Completed | 4/11 [00:01<00:01,  3.65it/s]
(EngineCore pid=112)
Loading safetensors checkpoint shards:  45% Completed | 5/11 [00:01<00:01,  3.44it/s]
(EngineCore pid=112)
Loading safetensors checkpoint shards:  55% Completed | 6/11 [00:01<00:01,  3.32it/s]
(EngineCore pid=112)
Loading safetensors checkpoint shards:  64% Completed | 7/11 [00:01<00:01,  3.26it/s]
(EngineCore pid=112)
Loading safetensors checkpoint shards:  73% Completed | 8/11 [00:02<00:00,  3.22it/s]
(EngineCore pid=112)
Loading safetensors checkpoint shards:  82% Completed | 9/11 [00:02<00:00,  3.72it/s]
(EngineCore pid=112)
Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:02<00:00,  3.91it/s]
(EngineCore pid=112)
Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:02<00:00,  3.70it/s]
(EngineCore pid=112)
(EngineCore pid=112) INFO 09-18 14:55:47 [default_loader.py:430] Loading weights took 2.98 seconds
(EngineCore pid=112) INFO 09-18 14:55:47 [deep_gemm.py:196] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(EngineCore pid=112) INFO 09-18 14:55:47 [deep_gemm.py:223] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(EngineCore pid=112) INFO 09-18 14:55:47 [deep_gemm.py:136] DeepGEMM E8M0 enabled on current platform.
(EngineCore pid=112) INFO 09-18 14:55:50 [fp8.py:733] Using MoEPrepareAndFinalizeNoDPEPModular
(EngineCore pid=112) WARNING 09-18 14:55:50 [speculator.py:235] Draft model Qwen4ExpMTP does not support external multimodal embeddings. Embeddings from the target model will not be passed to the drafter; using text-only draft inputs instead.
(EngineCore pid=112) INFO 09-18 14:55:51 [model_runner.py:419] Model loading took 76.36 GiB memory and 89.100896 seconds
(EngineCore pid=112) INFO 09-18 14:55:51 [topk_topp_sampler.py:78] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=112) INFO 09-18 14:55:51 [interface.py:918] Setting attention block size to 3152 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=112) INFO 09-18 14:55:51 [utils.py:320] Using BLNHC KV cache layout.
(EngineCore pid=112) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
(EngineCore pid=112) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
(EngineCore pid=112) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore pid=112) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
(EngineCore pid=112) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
(EngineCore pid=112) INFO 09-18 14:55:55 [encoder_runner.py:131] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(EngineCore pid=112) WARNING 09-18 14:56:17 [kv_cache_utils.py:2219] Speculative decoding (method=mtp) is enabled but no KV cache group could be identified as the draft model's.
(EngineCore pid=112) WARNING 09-18 14:56:17 [compilation.py:1415] CUDAGraphMode.FULL is not supported with GDNAttentionBackend backend (support: AttentionCGSupport.UNIFORM_BATCH); setting cudagraph_mode=FULL_DECODE_ONLY
(EngineCore pid=112)
Capturing CUDA graphs (FULL):   0%|          | 0/1 [00:00<?, ?it/s]
Capturing CUDA graphs (FULL): 100%|██████████| 1/1 [00:01<00:00,  1.81s/it]
Capturing CUDA graphs (FULL): 100%|██████████| 1/1 [00:01<00:00,  1.81s/it]
(EngineCore pid=112) INFO 09-18 14:56:19 [speculator.py:150] Capturing model for speculator...
(EngineCore pid=112)
Capturing prefill CUDA graphs (FULL):   0%|          | 0/1 [00:00<?, ?it/s]
Capturing prefill CUDA graphs (FULL): 100%|██████████| 1/1 [00:00<00:00,  1.95it/s]
Capturing prefill CUDA graphs (FULL): 100%|██████████| 1/1 [00:00<00:00,  1.95it/s]
(EngineCore pid=112) INFO 09-18 14:56:20 [model_runner.py:1057] Graph capturing finished in 3 secs, took 0.14 GiB
(EngineCore pid=112) INFO 09-18 14:56:21 [gpu_worker.py:641] Available KV cache memory: 2.78 GiB
(EngineCore pid=112) INFO 09-18 14:56:21 [gpu_worker.py:656] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9850 is equivalent to --gpu-memory-utilization=0.9833 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9867. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=112) WARNING 09-18 14:56:21 [kv_cache_utils.py:2219] Speculative decoding (method=mtp) is enabled but no KV cache group could be identified as the draft model's.
(EngineCore pid=112) INFO 09-18 14:56:21 [kv_cache_utils.py:2404] GPU KV cache size: 159,669 tokens, Maximum concurrency for 131,072 tokens per request: 1.22x
(EngineCore pid=112) INFO 09-18 14:56:21 [kernel_warmup.py:171] JIT kernel warmup starting.
(EngineCore pid=112) INFO 09-18 14:56:21 [kernel_warmup.py:184] JIT kernel warmup finished in 0.00s.
(EngineCore pid=112) INFO 09-18 14:56:21 [qwen_vl_triton_warmup.py:57] Warmed position embedding and vision rotary kernels on grids=[(1, 16, 16), (1, 16, 2), (1, 2, 16), (1, 2, 2)].
(EngineCore pid=112) INFO 09-18 14:56:21 [qwen_vl_triton_warmup.py:98] Warmed M-RoPE Triton kernels.
(EngineCore pid=112) INFO 09-18 14:56:21 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel.
(EngineCore pid=112) INFO 09-18 14:56:22 [qwen4_exp_qsa_warmup.py:71] Warmed up Qwen4Exp QSA decode kernels: ((1, 1), (2, 1)).
(EngineCore pid=112) INFO 09-18 14:56:27 [qwen4_exp_qsa_warmup.py:85] Warmed up Qwen4Exp QSA sparse attention kernels: ((32, 2, 1), (32, 4, 4), (64, 1, 2), (64, 4, 4), (64, 8, 4), (64, 33, 8), (128, 4, 4), (128, 8, 4)).
(EngineCore pid=112) INFO 09-18 14:56:27 [kernel_warmup.py:254] Skipping FlashInfer autotune because it is disabled.
(EngineCore pid=112)
Capturing CUDA graphs (FULL):   0%|          | 0/1 [00:00<?, ?it/s]
Capturing CUDA graphs (FULL): 100%|██████████| 1/1 [00:00<00:00,  9.68it/s]
Capturing CUDA graphs (FULL): 100%|██████████| 1/1 [00:00<00:00,  9.67it/s]
(EngineCore pid=112) INFO 09-18 14:56:36 [speculator.py:150] Capturing model for speculator...
(EngineCore pid=112)
Capturing prefill CUDA graphs (FULL):   0%|          | 0/1 [00:00<?, ?it/s]
Capturing prefill CUDA graphs (FULL): 100%|██████████| 1/1 [00:00<00:00, 120.04it/s]
(EngineCore pid=112) INFO 09-18 14:56:36 [model_runner.py:1057] Graph capturing finished in 1 secs, took 0.10 GiB
(EngineCore pid=112) INFO 09-18 14:56:36 [gpu_worker.py:824] CUDA graph pool memory: 0.1 GiB (actual), 0.14 GiB (estimated), difference: 0.04 GiB (37.7%).
(EngineCore pid=112) INFO 09-18 14:56:36 [gpu_worker.py:887] Free memory on device (82.59/83.05 GiB) on startup. Desired GPU memory utilization is (0.985, 81.8 GiB). Actual usage is 77.82 GiB for consumed memory (weights + non-torch), 1.2 GiB for peak activation, and 0.1 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=2718838744` (2.53 GiB) to fit into requested memory, or `--kv-cache-memory=3565505536` (3.32 GiB) to fully utilize gpu memory. Current kv cache memory in use is 2.78 GiB.
(EngineCore pid=112) INFO 09-18 14:56:37 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=112) WARNING 09-18 14:56:38 [torch_utils.py:274] OMP_NUM_THREADS=8 is set; leaving Torch threads at 8 for serving. Multi-threaded torch CPU ops during serving can degrade performance through spin-wait contention and cgroup CPU-quota throttling.
(EngineCore pid=112) INFO 09-18 14:56:38 [core.py:380] init engine (profile, create kv cache, warmup model) took 47.02 s
(EngineCore pid=112) INFO 09-18 14:56:38 [kv_cache_utils.py:762] kv cache group sizes [3152, 3152, 3152, 3152, 8, 3152]
(EngineCore pid=112) INFO 09-18 14:56:38 [kv_cache_utils.py:763] kv lcm block sizes 3152
(EngineCore pid=112) INFO 09-18 14:56:38 [kernel.py:408] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'], gelu_and_mul_sparse=['triton', 'native'])
(APIServer pid=1) INFO 09-18 14:56:38 [entry.py:132] Supported tasks: ['generate']
(APIServer pid=1) INFO 09-18 14:56:38 [factories.py:76] Scale-out endpoints are disabled. Set --enable-scale-out to enable them.
(APIServer pid=1) INFO 09-18 14:56:38 [parser_manager.py:34] "auto" tool choice has been enabled.
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=1) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=1) INFO 09-18 14:56:38 [hf.py:642] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this.
(APIServer pid=1) WARNING 09-18 14:56:38 [model.py:1769] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 1.0, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=1) INFO 09-18 14:56:38 [entry.py:136] Starting vLLM server on http://0.0.0.0:8000
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:60] Available routes are:
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /docs, Methods: HEAD, GET
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /redoc, Methods: HEAD, GET
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /load, Methods: GET
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /version, Methods: GET
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /health, Methods: GET
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /metrics, Methods: GET
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /tokenize, Methods: POST
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /detokenize, Methods: POST
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /v1/models, Methods: GET
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /ping, Methods: GET
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /ping, Methods: POST
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /invocations, Methods: POST
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /v1/chat/completions, Methods: POST
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /v1/responses, Methods: POST
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /v1/completions, Methods: POST
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /v1/messages, Methods: POST
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /generative_scoring, Methods: POST
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=1) INFO 09-18 14:56:38 [launcher.py:69] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=1) INFO:     Started server process [1]
(APIServer pid=1) INFO:     Waiting for application startup.
(APIServer pid=1) INFO:     Application startup complete.
(APIServer pid=1) INFO:     172.21.0.1:38360 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:38376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(EngineCore pid=112) WARNING 09-18 14:56:41 [jit_monitor.py:140] Triton kernel JIT compilation during inference: layer_norm_fwd_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=112) WARNING 09-18 14:56:41 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _count_expert_num_tokens. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=112) WARNING 09-18 14:56:41 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _compute_local_logits_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=112) WARNING 09-18 14:56:41 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _rejection_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=112) WARNING 09-18 14:56:41 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _resample_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=1) INFO:     172.21.0.1:38386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:38400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     127.0.0.1:41714 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:38404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:38416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:52596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:52608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:52624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:52626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:52628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:52642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:52646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 14:56:49 [loggers.py:323] Engine 000: Avg prompt throughput: 64.7 tokens/s, Avg generation throughput: 73.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 09-18 14:56:49 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.99, Accepted throughput: 36.01 tokens/s, Drafted throughput: 36.56 tokens/s, Accepted: 396 tokens, Drafted: 402 tokens, Per-position acceptance rate: 0.985, Avg Draft acceptance rate: 98.5%
(APIServer pid=1) INFO:     172.21.0.1:52658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:52666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:52672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:52688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[rank0]:[W918 14:56:53.391378074 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 377487360 bytes (free: 83820544, total: 89173131264).
(APIServer pid=1) INFO:     172.21.0.1:53008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:53016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 14:56:59 [loggers.py:323] Engine 000: Avg prompt throughput: 2778.3 tokens/s, Avg generation throughput: 59.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 66.7%
(APIServer pid=1) INFO 09-18 14:56:59 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.90, Accepted throughput: 28.10 tokens/s, Drafted throughput: 31.20 tokens/s, Accepted: 281 tokens, Drafted: 312 tokens, Per-position acceptance rate: 0.901, Avg Draft acceptance rate: 90.1%
(APIServer pid=1) INFO:     172.21.0.1:53018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 14:57:09 [loggers.py:323] Engine 000: Avg prompt throughput: 7.7 tokens/s, Avg generation throughput: 109.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 66.7%
(APIServer pid=1) INFO 09-18 14:57:09 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.75, Accepted throughput: 46.60 tokens/s, Drafted throughput: 62.40 tokens/s, Accepted: 466 tokens, Drafted: 624 tokens, Per-position acceptance rate: 0.747, Avg Draft acceptance rate: 74.7%
(APIServer pid=1) INFO:     127.0.0.1:34068 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 14:57:19 [loggers.py:323] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 108.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 66.7%
(APIServer pid=1) INFO 09-18 14:57:19 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.72, Accepted throughput: 45.60 tokens/s, Drafted throughput: 62.90 tokens/s, Accepted: 456 tokens, Drafted: 629 tokens, Per-position acceptance rate: 0.725, Avg Draft acceptance rate: 72.5%
(APIServer pid=1) INFO:     172.21.0.1:44752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 14:57:29 [loggers.py:323] Engine 000: Avg prompt throughput: 7.7 tokens/s, Avg generation throughput: 111.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 66.7%
(APIServer pid=1) INFO 09-18 14:57:29 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.77, Accepted throughput: 48.40 tokens/s, Drafted throughput: 62.60 tokens/s, Accepted: 484 tokens, Drafted: 626 tokens, Per-position acceptance rate: 0.773, Avg Draft acceptance rate: 77.3%
(APIServer pid=1) INFO:     172.21.0.1:55170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 14:57:39 [loggers.py:323] Engine 000: Avg prompt throughput: 7.7 tokens/s, Avg generation throughput: 107.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 66.7%
(APIServer pid=1) INFO 09-18 14:57:39 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.72, Accepted throughput: 44.80 tokens/s, Drafted throughput: 62.40 tokens/s, Accepted: 448 tokens, Drafted: 624 tokens, Per-position acceptance rate: 0.718, Avg Draft acceptance rate: 71.8%
(APIServer pid=1) INFO:     127.0.0.1:39416 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 14:57:49 [loggers.py:323] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 111.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 66.7%
(APIServer pid=1) INFO 09-18 14:57:49 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.77, Accepted throughput: 48.30 tokens/s, Drafted throughput: 63.00 tokens/s, Accepted: 483 tokens, Drafted: 630 tokens, Per-position acceptance rate: 0.767, Avg Draft acceptance rate: 76.7%
(APIServer pid=1) INFO:     172.21.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:49968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 14:57:59 [loggers.py:323] Engine 000: Avg prompt throughput: 15.8 tokens/s, Avg generation throughput: 105.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 66.7%
(APIServer pid=1) INFO 09-18 14:57:59 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.71, Accepted throughput: 43.90 tokens/s, Drafted throughput: 61.80 tokens/s, Accepted: 439 tokens, Drafted: 618 tokens, Per-position acceptance rate: 0.710, Avg Draft acceptance rate: 71.0%
(APIServer pid=1) INFO 09-18 14:58:09 [loggers.py:323] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 113.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 66.7%
(APIServer pid=1) INFO 09-18 14:58:09 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.80, Accepted throughput: 50.20 tokens/s, Drafted throughput: 63.10 tokens/s, Accepted: 502 tokens, Drafted: 631 tokens, Per-position acceptance rate: 0.796, Avg Draft acceptance rate: 79.6%
(APIServer pid=1) INFO:     127.0.0.1:41032 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:35070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 14:58:19 [loggers.py:323] Engine 000: Avg prompt throughput: 7.9 tokens/s, Avg generation throughput: 110.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 66.7%
(APIServer pid=1) INFO 09-18 14:58:19 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.77, Accepted throughput: 48.20 tokens/s, Drafted throughput: 62.50 tokens/s, Accepted: 482 tokens, Drafted: 625 tokens, Per-position acceptance rate: 0.771, Avg Draft acceptance rate: 77.1%
(APIServer pid=1) INFO 09-18 14:58:29 [loggers.py:323] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 113.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 66.7%
(APIServer pid=1) INFO 09-18 14:58:29 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.80, Accepted throughput: 50.49 tokens/s, Drafted throughput: 63.09 tokens/s, Accepted: 505 tokens, Drafted: 631 tokens, Per-position acceptance rate: 0.800, Avg Draft acceptance rate: 80.0%
(APIServer pid=1) INFO:     172.21.0.1:36238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 14:58:39 [loggers.py:323] Engine 000: Avg prompt throughput: 7.9 tokens/s, Avg generation throughput: 110.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 66.7%
(APIServer pid=1) INFO 09-18 14:58:39 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.77, Accepted throughput: 48.00 tokens/s, Drafted throughput: 62.60 tokens/s, Accepted: 480 tokens, Drafted: 626 tokens, Per-position acceptance rate: 0.767, Avg Draft acceptance rate: 76.7%
(APIServer pid=1) INFO:     127.0.0.1:47540 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 14:58:49 [loggers.py:323] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 113.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 66.7%
(APIServer pid=1) INFO 09-18 14:58:49 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.81, Accepted throughput: 50.80 tokens/s, Drafted throughput: 63.09 tokens/s, Accepted: 508 tokens, Drafted: 631 tokens, Per-position acceptance rate: 0.805, Avg Draft acceptance rate: 80.5%
(APIServer pid=1) INFO:     172.21.0.1:46184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:34866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 14:58:59 [loggers.py:323] Engine 000: Avg prompt throughput: 15.8 tokens/s, Avg generation throughput: 109.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 66.7%
(APIServer pid=1) INFO 09-18 14:58:59 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.75, Accepted throughput: 46.80 tokens/s, Drafted throughput: 62.10 tokens/s, Accepted: 468 tokens, Drafted: 621 tokens, Per-position acceptance rate: 0.754, Avg Draft acceptance rate: 75.4%
(APIServer pid=1) INFO 09-18 14:59:09 [loggers.py:323] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 108.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 66.7%
(APIServer pid=1) INFO 09-18 14:59:09 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.71, Accepted throughput: 45.00 tokens/s, Drafted throughput: 63.00 tokens/s, Accepted: 450 tokens, Drafted: 630 tokens, Per-position acceptance rate: 0.714, Avg Draft acceptance rate: 71.4%
(APIServer pid=1) INFO:     127.0.0.1:32954 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:54998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 14:59:19 [loggers.py:323] Engine 000: Avg prompt throughput: 7.9 tokens/s, Avg generation throughput: 108.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 66.7%
(APIServer pid=1) INFO 09-18 14:59:19 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.74, Accepted throughput: 46.20 tokens/s, Drafted throughput: 62.50 tokens/s, Accepted: 462 tokens, Drafted: 625 tokens, Per-position acceptance rate: 0.739, Avg Draft acceptance rate: 73.9%
(APIServer pid=1) INFO 09-18 14:59:29 [loggers.py:323] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 107.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 66.7%
(APIServer pid=1) INFO 09-18 14:59:29 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.71, Accepted throughput: 44.49 tokens/s, Drafted throughput: 62.99 tokens/s, Accepted: 445 tokens, Drafted: 630 tokens, Per-position acceptance rate: 0.706, Avg Draft acceptance rate: 70.6%
(APIServer pid=1) INFO:     172.21.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 14:59:39 [loggers.py:323] Engine 000: Avg prompt throughput: 7.9 tokens/s, Avg generation throughput: 108.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 66.7%
(APIServer pid=1) INFO 09-18 14:59:39 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.74, Accepted throughput: 46.10 tokens/s, Drafted throughput: 62.50 tokens/s, Accepted: 461 tokens, Drafted: 625 tokens, Per-position acceptance rate: 0.738, Avg Draft acceptance rate: 73.8%
(APIServer pid=1) INFO:     127.0.0.1:53944 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 14:59:49 [loggers.py:323] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 107.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 66.7%
(APIServer pid=1) INFO 09-18 14:59:49 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.70, Accepted throughput: 44.30 tokens/s, Drafted throughput: 63.00 tokens/s, Accepted: 443 tokens, Drafted: 630 tokens, Per-position acceptance rate: 0.703, Avg Draft acceptance rate: 70.3%
(APIServer pid=1) [transformers] Token indices sequence length is longer than the specified maximum sequence length for this model (268298 > 262144). Running this sequence through the model will result in indexing errors
(APIServer pid=1) INFO:     172.21.0.1:43784 - "POST /tokenize HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:43794 - "POST /tokenize HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:43806 - "POST /tokenize HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:43814 - "POST /tokenize HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:43818 - "POST /tokenize HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:40940 - "POST /tokenize HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:40946 - "POST /tokenize HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:40952 - "POST /tokenize HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:40964 - "POST /tokenize HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:40968 - "POST /tokenize HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:40970 - "POST /tokenize HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:40984 - "POST /tokenize HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:40986 - "POST /tokenize HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:41002 - "POST /tokenize HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:41010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 14:59:59 [loggers.py:323] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 43.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 39.4%, Prefix cache hit rate: 0.0%, MM cache hit rate: 66.7%
(APIServer pid=1) INFO 09-18 14:59:59 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.72, Accepted throughput: 18.10 tokens/s, Drafted throughput: 25.00 tokens/s, Accepted: 181 tokens, Drafted: 250 tokens, Per-position acceptance rate: 0.724, Avg Draft acceptance rate: 72.4%
(APIServer pid=1) INFO 09-18 15:00:09 [loggers.py:323] Engine 000: Avg prompt throughput: 12599.4 tokens/s, Avg generation throughput: 0.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 66.7%
(APIServer pid=1) INFO 09-18 15:00:09 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.00, Accepted throughput: 0.30 tokens/s, Drafted throughput: 0.30 tokens/s, Accepted: 3 tokens, Drafted: 3 tokens, Per-position acceptance rate: 1.000, Avg Draft acceptance rate: 100.0%
(APIServer pid=1) INFO:     172.21.0.1:52464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:52476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     127.0.0.1:57076 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:52546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 15:00:19 [loggers.py:323] Engine 000: Avg prompt throughput: 1243.7 tokens/s, Avg generation throughput: 39.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 42.4%, Prefix cache hit rate: 46.2%, MM cache hit rate: 71.4%
(APIServer pid=1) INFO 09-18 15:00:19 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.92, Accepted throughput: 18.90 tokens/s, Drafted throughput: 20.60 tokens/s, Accepted: 189 tokens, Drafted: 206 tokens, Per-position acceptance rate: 0.917, Avg Draft acceptance rate: 91.7%
(APIServer pid=1) INFO:     172.21.0.1:38136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:38150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:38158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 15:00:29 [loggers.py:323] Engine 000: Avg prompt throughput: 11171.3 tokens/s, Avg generation throughput: 24.7 tokens/s, Running: 1 reqs, Waiting: 3 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 46.2%, MM cache hit rate: 71.4%
(APIServer pid=1) INFO 09-18 15:00:29 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.87, Accepted throughput: 11.40 tokens/s, Drafted throughput: 13.10 tokens/s, Accepted: 114 tokens, Drafted: 131 tokens, Per-position acceptance rate: 0.870, Avg Draft acceptance rate: 87.0%
(APIServer pid=1) INFO:     172.21.0.1:38172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:38188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:34860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:34870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 15:00:39 [loggers.py:323] Engine 000: Avg prompt throughput: 58.8 tokens/s, Avg generation throughput: 114.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 46.1%, MM cache hit rate: 71.4%
(APIServer pid=1) INFO 09-18 15:00:39 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.95, Accepted throughput: 55.30 tokens/s, Drafted throughput: 58.40 tokens/s, Accepted: 553 tokens, Drafted: 584 tokens, Per-position acceptance rate: 0.947, Avg Draft acceptance rate: 94.7%
(APIServer pid=1) INFO:     127.0.0.1:44446 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:34880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 15:00:49 [loggers.py:323] Engine 000: Avg prompt throughput: 9.2 tokens/s, Avg generation throughput: 117.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 46.1%, MM cache hit rate: 71.4%
(APIServer pid=1) INFO 09-18 15:00:49 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.89, Accepted throughput: 55.10 tokens/s, Drafted throughput: 62.10 tokens/s, Accepted: 551 tokens, Drafted: 621 tokens, Per-position acceptance rate: 0.887, Avg Draft acceptance rate: 88.7%
(APIServer pid=1) INFO:     172.21.0.1:34804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.21.0.1:34816 - "POST /v1/chat/completions HTTP/1.1" 200 OK