Run Qwen3 Next 80B A3B on Windows: 2025 Guide

Running the Qwen3 Next 80B A3B model—a cutting-edge large language AI—on Windows is now achievable thanks to WSL2, NVIDIA GPU acceleration, and Docker containerization.

This guide provides a step-by-step walkthrough to install, configure, and run Qwen3 Next 80B A3B on Windows 11, including practical examples, API usage, optimizations, and troubleshooting tips.

Introduction to Qwen3 Next 80B A3B

Qwen3 Next 80B A3B is an instruction-optimized, sparse Mixture-of-Experts (MoE) LLM developed by Alibaba AI Research. Despite having 80 billion parameters, it activates only ~3B per inference, allowing high throughput while using fewer resources. Key features include:

• Sparse MoE Architecture: 512 experts, routing ~10 per token.
• FP8 Quantization: Reduces VRAM requirements for efficient GPU usage.
• Extended Context Length: Supports 262K tokens natively, extendable to ~1M with YaRN.
• Multi-Token Prediction: Generates multiple tokens in parallel, boosting inference speed.
• Performance: Matches dense models in quality but requires less compute.

Running this model locally requires NVIDIA GPUs with CUDA support, WSL2, and proper environment setup.

Prerequisites

Hardware Requirements

• NVIDIA GPU: RTX 30/40 series or professional GPUs like RTX PRO 6000 or A100. Minimum 48GB VRAM; 75GB+ recommended for FP8 weights.
• CPU: Multi-core x64 processor for smooth virtualization.
• RAM: 64GB+ recommended for model caching.

Software Requirements

• Windows 11 (latest build)
• WSL2 with Ubuntu 22.04+
• NVIDIA Driver + CUDA Toolkit 12.1
• Docker Desktop (Linux containers + GPU integration)
• Python 3.10+
• vLLM Serving Framework (for OpenAI-compatible API)
• FlashInfer Library (CUDA graph optimized for LLMs)
• Qwen3 Next 80B A3B Model Weights (FP8 quantized recommended)

Benchmarking Overview

Tested on RTX 4090 Ti, CUDA 12.1, vLLM 0.10.2.

Step-by-Step Installation

Step 1: Setup WSL2 with Ubuntu

# Enable WSL and Virtual Machine Platform
wsl --install
dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart
dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart

# Set default WSL version
wsl --set-default-version 2

1. Install Ubuntu 22.04 LTS from Microsoft Store.
2. Update packages:

sudo apt update && sudo apt upgrade -y
sudo apt install -y build-essential curl git python3 python3-pip

Step 2: Install NVIDIA Drivers and CUDA

1. Install NVIDIA driver for WSL.
2. Verify GPU inside WSL:

nvidia-smi

3. Install CUDA Toolkit 12.1 and PyTorch with CUDA support:

pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu121

# Verify GPU
nvidia-smi

# Add NVIDIA Docker repository
distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt update
sudo apt install -y nvidia-docker2
sudo systemctl restart docker

Step 3: Install Docker Desktop with GPU Support

1. Install Docker Desktop and enable WSL2 integration.
2. Install NVIDIA Container Toolkit inside Ubuntu:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt update
sudo apt install -y nvidia-docker2
sudo systemctl restart docker

3. Test GPU access in Docker:

docker run --rm --gpus all nvidia/cuda:12.1-base nvidia-smi

Step 4: Prepare Python Environment

pip3 install --upgrade pip
pip3 install transformers vllm>=0.10.2 flashinfer>=0.3.1

• Download or clone Qwen3 Next weights (FP8 recommended).

pip3 install --upgrade pip
pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu121
pip3 install vllm flashinfer transformers

Step 5: Run Qwen3 Next 80B A3B via Docker and vLLM

docker run -d --gpus all --name qwen3next80b -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface:rw \
  -e HF_HOME=/root/.cache/huggingface \
  vllm/vllm-openai:v0.10.2 \
  --model Qwen/Qwen3-Next-80B-A3B-Instruct \
  --host 0.0.0.0 --port 8000 \
  --async-scheduling --tensor-parallel-size=4 \
  --trust-remote-code

mkdir -p ~/models/qwen3 && cd ~/models/qwen3
git clone https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct

Notes:

• Mount Huggingface cache for efficient downloads.
• Adjust tensor-parallel-size for multiple GPUs.
• Server accessible at http://localhost:8000.

Launch the Model

1. Using vLLM Docker

docker run -d --gpus all --name qwen3next80b -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_HOME=/root/.cache/huggingface \
  vllm/vllm-openai:v0.10.2 \
  --model Qwen/Qwen3-Next-80B-A3B-Instruct \
  --host 0.0.0.0 --port 8000 \
  --async-scheduling --tensor-parallel-size=4 \
  --trust-remote-code

2. Using SGLang

pip install 'sglang[all]>=0.5.2'
sglang launch \
  --model-path ~/models/qwen3/Qwen3-Next-80B-A3B-Instruct \
  --port 8080 \
  --max-context-length 262144 \
  --speculative-algo NEXTN \
  --speculative-num-steps 3 \
  --speculative-num-draft-tokens 4

Model Architecture

1. Sparse Mixture-of-Experts (MoE) Design

• 80B Total Parameters: Distributed across 512 experts.
• 3B Active Parameters: ~10 experts routed per token, optimizing compute.
• Router Module: Dynamically dispatches tokens to experts for efficient inference.

2. FP8 Quantization

• Weight Compression: 8-bit floating-point precision reduces VRAM by ~60%.
• FlashInfer Integration: CUDA-graph optimized for NVIDIA Blackwell/Hopper GPUs.

3. Extended Context Handling

• Native 262K Tokens: Efficient long-sequence processing.
• YaRN Rope Scaling: Extend context up to 1M tokens with minimal accuracy loss.

4. Multi-Token and Speculative Execution

• Multi-Token Prediction (MTP): Generates multiple tokens per forward pass.
• Speculative Decoding: Works with vLLM asynchronous scheduler to boost throughput.

Performance Tuning

• Adjust --tensor-parallel-size to match GPU count.
• Configure speculative decoding with vLLM --speculative-config.
• Batch multiple prompts for improved throughput.
• Preload frequent prompts to warm GPU cache.
• Enable FLASHINFER_USE_CUDA_GRAPH=1 for CUDA graph optimization.

Advanced Use Cases

1. Chained-Task Pipelines

# Extract entities
entities = generate(model, tokenizer, prompt="Extract key entities from this text: ...")

# Generate summary
summary = generate(model, tokenizer, prompt=f"Summarize these entities: {entities}")

2. Multi-Modal Input with Vision Adapter

pip3 install transformers[vision]

from transformers import AutoProcessor, AutoModelForVision2Seq
processor = AutoProcessor.from_pretrained("Your/VisionAdapter-Qwen3Next")
model = AutoModelForVision2Seq.from_pretrained("Your/VisionAdapter-Qwen3Next")

inputs = processor(images=pil_image, text="Describe this scene:", return_tensors="pt")
outputs = model.generate(**inputs)
print(processor.decode(outputs[0], skip_special_tokens=True))

Practical Examples

Example 1: Legal Contract Analysis

curl -X POST http://localhost:8000/v1/chat/completions \
 -H "Content-Type: application/json" \
 -d '{ "model":"Qwen3-Next-80B-A3B-Instruct", "messages":[{"role":"user","content":"Identify risks in this contract: [contract text]"}] }'

Example 2: Code Review Assistant

from vllm import Client
client = Client("http://localhost:8000")

response = client.chat(
    model="Qwen3-Next-80B-A3B-Instruct",
    messages=[{"role":"user","content":"Review this Python function and suggest improvements:\n``````"}]
)
print(response.choices[0].message.content)

Example 3: Query via Curl

curl -X POST "http://localhost:8000/v1/chat/completions" \
-H "Content-Type: application/json" \
-d '{
  "model": "Qwen/Qwen3-Next-80B-A3B-Instruct",
  "messages": [
    {"role": "user", "content": "Explain the benefits of using WSL2 for AI model deployment."}
  ]
}'

Example 4: Query via Python Client

import requests

url = "http://localhost:8000/v1/chat/completions"
headers = {"Content-Type": "application/json"}
data = {
  "model": "Qwen/Qwen3-Next-80B-A3B-Instruct",
  "messages": [{"role": "user", "content": "Summarize advantages of sparse MoE models."}]
}

response = requests.post(url, headers=headers, json=data)
print(response.json()["choices"][0]["message"]["content"])

Run:

python3 query_qwen.py

Optimization Tips

• GPU Memory: Use FP8 weights or tensor parallelism for multi-GPU setups.
• FlashInfer: Version >=0.3.1 fixes CUDA graph errors.
• CUDA & PyTorch: Use compatible versions (CUDA 12.1 + CU121 PyTorch).
• Caching: Keep Huggingface cache consistent to avoid repeated downloads.
• Verbose Logs: Enable in vLLM for debugging.
• Context Length: Modify config for extended context via YaRN scaling.

Troubleshooting

• Model fails to load: Check GPU recognition with nvidia-smi and CUDA compatibility.
• WSL2 issues: Update Linux kernel to latest stable release.
• Slow inference: Ensure FP8 quantized model is used, and async scheduling is enabled.
• Container errors: Verify Docker GPU passthrough and environment variables.

Summary

Running Qwen3 Next 80B A3B on Windows 11 is now practical with WSL2, Docker, and NVIDIA GPU acceleration. With FP8 quantization, sparse MoE architecture, and extended context support, you can deploy large-scale, instruction-optimized AI models locally for research, NLP, multi-modal projects, and advanced chained-task pipelines.

References

1. Running Qwen3 8B on Windows: A Comprehensive Guide
2. Run Qwen 3 8B on Mac: An Installation Guide
3. Run Qwen3-8B on Ubuntu
4. Gemma 3 vs Qwen 3
5. Run Qwen3 Next 80B A3B on macOS