What to know

  • Microsoft’s Phi-3 is a small but powerful AI model that you can run locally on Windows.
  • Install Ollama, then run the command ollama run phi3 on a terminal app (like CMD). Once phi-3 is downloaded, you’ll be able to chat with the AI within the terminal itself.
  • You can also use a software like LM Studio for a graphical interface to chat with Phi-3 locally. Download the Phi-3 guff file separately and save it within LM Studio’s directory. Then load the model within LM Studio and start chatting with Phi-3 on Windows locally. 

Microsoft’s Phi-3 family of language models are finally here. For their size, they’re definitely a class apart and are already proving much better than other recently released models like Llama 3 and Mixtral on many fronts. Thanks to its tiny size, Phi-3 can easily run on your Windows PC locally. Here’s how you can do so using Ollama and LM Studio. 

How to run Microsoft’s Phi-3 on Windows using Ollama

Ollama is a software framework that lets you run and experiment with LLMs. Here’s how to use it to run Microsoft’s Phi-3 on Windows locally.

Step 1: Download and install Ollama

Firstly, let’s download and install Ollama. Here’s how:

  1. Use the link mentioned above and click on Download for Windows (Preview).
  2. Once downloaded, run the setup file.
  3. Click on Install and install Ollama.

Step 2: Run Phi-3 command and download the LLM

Next, let’s download the Phi-3 model using Ollama.

  1. Open Ollama.com and click on Models.
  2. Scroll down and click on phi3. If you don’t see it at first, you can search for it as well.
  3. Here, copy the command to download phi3. 
  4. Next, open the Command Prompt (or any terminal app of your choice) from the Start menu. 
  5. Here, paste the copied command.
  6. Hit Enter and wait for Phi-3 to download on your machine.
  7. Once you see the “Send a message” message, you’re ready to start chatting with the model locally.

Step 3: Start chatting with Microsoft’s Phi-3 LLM

You can start chatting within the terminal app itself. Simply type a prompt and hit Enter. Here are a few areas we tested the model in.

Testing censorship resistance

Testing understanding of complex topics

Testing for hallucinations

Testing for creativity

How to run Microsoft’s Phi-3 on Windows using LM Studio

If you don’t like chatting with Microsoft’s Phi-3 on Windows using your terminal app and would rather have a dedicated interface for it, there’s nothing better than LM Studio. Here’s how to set up Phi-3 on LM Studio and start chatting with the model locally.

Step 1: Install LM Studio 

  1. Use the link above and click on LM Studio for Windows to download it.
  2. Once downloaded, run the installer and let the LM Studio install.

Step 2: Download Phi-3 gguf file

You won’t be able to search for and download Phi-3 from within LM Studio itself. You’ll need to get the Phi-3 guff file separately. Here’s how:

  1. Use the link given above and click on Files.
  2. Here, you’ll find two versions of the Phi-3 model. Select one. For our purposes, we’re selecting the smaller version.
  3. Click on Download.
  4. Then save it in a convenient location.

Step 3: Load the Phi-3 model

Next, we’re going to load the downloaded Phi-3 model. Follow the steps to do so:

  1. Open LM Studio and click on My Models on the left.
  2. Take note of the ‘Local models folder’. This is where we need to move the downloaded Phi-3 guff file. Click on Show in File Explorer to open the directory.
  3. Here, create a new folder titled Microsoft.
  4. Within the Microsoft folder, create another folder titled Phi-3.
  5. Paste the downloaded Phi-3 guff file within the Phi-3 folder.
  6. Once you’ve moved the Phi-3 guff file, it will appear in the LM Studio as well.

    You may need to restart LM Studio for it to recognize Phi-3 in its directory.
  7. To load the Phi-3 model, click on the AI Chat option on the left.
  8. Click on Select a model to load.
  9. And select the Phi-3 model.
  10. Wait for it to load. Once done, you can start chatting with Phi-3. However, we recommend offloading the model to the GPU so your CPU doesn’t come under undue pressure. To do so, under ‘System Prompt’ on the right, click on Advanced Configuration > Hardware Settings.
  11. Under ‘GPU Acceleration’, click on Max.
  12. Click on Reload model to apply configuration.
  13. Once the model loads, you can start chatting with Phi-3. 

Step 4: Start chatting with Microsoft’s Phi-3 LLM

And that’s about it. Go ahead and enter a prompt. Regardless of whether or not you have an internet connection, you’ll now be able to chat with Microsoft’s Phi-3 model on your Windows PC locally. 

FAQ

Let’s consider a few commonly asked questions about running Microsoft’s Phi-3 locally on your Windows PC.

How to fix Ollama stuck at downloading Phi-3?

If you encounter issues downloading Phi-3 via Ollama on the command prompt, simply enter the command ollama run phi3 again. The download will resume from the point where it was struck before. 

We hope this guide helped you run Microsoft’s Phi-3 model locally on your Windows PC via Ollama and LM Studio. Until next time!