Getting Started with ChatGLM3 6B: Run Local AI on Consumer Hardware

ChatGLM3-6B is optimized for efficiency, running smoothly on consumer GPUs with at least 6GB of VRAM (8GB+ recommended for better performance). First, ensure you have Python 3.8 or higher installed. Open your terminal and create a dedicated virtual environment to avoid dependency conflicts. Run `python -m venv glm_env` followed by `source glm_env/bin/activate` (on macOS/Linux) or `glm_env\Scripts\activate` (on Windows). Next, install the required libraries. The core dependency is `transformers`, along with `torch` for CUDA support. Execute `pip install transformers torch sentencepiece protobuf`. If you are using an NVIDIA GPU, ensure your CUDA toolkit is compatible with your PyTorch version. For CPU-only inference, the model will still run but significantly slower. Verify your installation by running `python -c "import transformers; print(transformers.__version__)` to ensure the library loads without errors.

Navigate to the official Hugging Face repository for ChatGLM3-6B. You can download the model directly using the `huggingface_hub` library, which is often faster than the default `transformers` download method due to parallel downloads. Install the hub package with `pip install huggingface_hub`. Then, use the following Python script to download the model files to a local directory, such as `./chatglm3-6b`. This approach allows you to reuse the model files across different projects without re-downloading. The total size is approximately 12-13GB. Ensure you have sufficient disk space. If you are in a region with slow access to Hugging Face, consider using a mirror or a third-party download service compatible with the `hf.co` repository structure. Once downloaded, you will see folders like `model-00001-of-00002.safetensors` and configuration files like `config.json`.

Create a Python file named `run_glm.py`. Import the necessary classes: `AutoTokenizer` and `AutoModel` from `transformers`. Since ChatGLM3 uses a custom tokenizer, ensure you use the correct path. The model supports both FP16 and INT4 quantization. For most consumer GPUs with 8GB VRAM, loading the FP16 model is feasible. For 6GB VRAM, use INT4 quantization. Here is the code snippet: ```python from transformers import AutoTokenizer, AutoModel import torch model_path = './chatglm3-6b' tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) model = AutoModel.from_pretrained(model_path, trust_remote_code=True).quantize(4).cuda() model = model.eval() ``` The `trust_remote_code=True` argument is crucial because ChatGLM3 uses custom modeling code not present in the standard transformers library. The `.quantize(4)` method reduces memory usage by half, allowing the model to fit in smaller VRAM. If you have a high-end GPU (24GB+), you can remove `.quantize(4)` for higher quality outputs.

ChatGLM3 supports multi-turn conversations. To interact with the model, you need to format your input correctly. The model expects a list of messages or a specific prompt format. For a single turn, you can pass a string. For multi-turn, use the `chat()` method. Add this code to your script: ```python def chat_with_glm(query): response, history = model.chat(tokenizer, query, history=[]) return response print(chat_with_glm("Hello, tell me about AI.")) ``` The `history` parameter stores previous interactions. To maintain context in a multi-turn session, pass the `history` list returned by the previous `chat()` call into the next call. For example: ```python query1 = "Who is the founder of Apple?" response1, history = model.chat(tokenizer, query1, history=[]) print(response1) query2 = "What year was he born?" response2, history = model.chat(tokenizer, query2, history=history) print(response2) ``` This structure allows the AI to remember previous questions, enabling coherent dialogue. The model generates tokens sequentially, so you might want to implement streaming output for a better user experience in production apps.

To use ChatGLM3 in web applications or other services, you can expose it via a FastAPI server. This allows you to send HTTP requests to your local AI. Create a new file `api_server.py`. Install FastAPI and Uvicorn with `pip install fastapi uvicorn`. Use the following code structure: ```python from fastapi import FastAPI, Request from transformers import AutoTokenizer, AutoModel app = FastAPI() # Load model once at startup tokenizer = AutoTokenizer.from_pretrained('./chatglm3-6b', trust_remote_code=True) model = AutoModel.from_pretrained('./chatglm3-6b', trust_remote_code=True).quantize(4).cuda().eval() @app.post("/chat") async def chat(request: Request): data = await request.json() query = data.get('query', '') response, history = model.chat(tokenizer, query, history=[]) return {"response": response} ``` Run the server with `uvicorn api_server:app --reload --port 8000`. Now, you can test it using `curl -X POST http://localhost:8000/chat -d '{"query":"Hello"}' -H 'Content-Type: application/json'`. This setup decouples the AI logic from your frontend, allowing you to build chat interfaces in React, Vue, or mobile apps that connect to your local AI instance.

If you experience high memory usage or slow inference, consider adjusting the `max_length` parameter in the `chat` method to limit output length. Additionally, ensure your GPU drivers are up to date. If you encounter CUDA out-of-memory errors, reduce the batch size or use INT4 quantization even if you have 8GB VRAM. For CPU inference, remove the `.cuda()` call and ensure your CPU supports AVX2 instructions for faster computation. You can also explore using `vLLM` or `TextGenerationInference` for more efficient batched serving if you plan to handle multiple concurrent requests. Remember that ChatGLM3 is trained primarily on Chinese and English data; it may perform less reliably in other languages. Fine-tuning with specific domain data can improve accuracy for specialized tasks.