I really like having a coding assistant like GitHub’s CoPilot helping out with syntax and making library suggestions, but it doesn’t take me long to run through the trial tokens. Even after I upgraded to the ‘diet plan’ for 20 bucks a month, and was careful with how much I used it I still managed to burn through my quota before month’s end.

It was really exciting to stumble upon Tabby ML, the Open Source AI coding assistant that runs on my hardware so not only am I not going broke paying someone else for tokens, all of my code stays private.

Getting Started:

The quick start guide on Tabby ML’s docsite got me going in no time flat. Here’s what I did to install Tabby on a GPU-less Fedora 44 system:

• Download the CPU-only release from GitHub. For me, at the time of this writing was tabby_x86_64-manylinux_2_28.tar.gz.
• Install Katana – I will be beefing up Tabby’s mind with outside sources and Katana is the web-crawler engine that makes that happen. I followed Tabby’s instructions and downloaded the Katana executable for my platform (https://github.com/projectdiscovery/katana/releases/download/v1.1.2/katana_1.1.2_linux_amd64.zip), extracted it, and moved it into /usr/bin in my filesystem.
• Install Tabby – I extracted Tabby’s harness binary https://github.com/TabbyML/tabby/releases/download/v0.32.0/tabby_x86_64-manylinux_2_28.tar.gz into my home directory which created a directory named tabby_x86_64-manylinux_2_28.
• I changed into this directory and made the two binaries in that directory (llama-server and tabby) executable.
• Finally I started Tabby, and asked it to use Starcoder-1B as the model to use for code-completion and Quen for the chatbot:
  • `./tabby serve –model StarCoder-1B –chat-model Qwen2-1.5B-Instruct`
• It took a little longer to get things going on the first run since both StarCoder and Quen models needed to be downloaded at about 1.5GB apiece, but shortly after downloads completed, a giant figlet-generated “TABBY” in my terminal let me know that startup had completed and that Tabby’s WebUI was available at: localhost:8080.
• I opened up a web browser on http://localhost:8080 and configured my username/password and was off to the races.

The first thing I did after getting Tabby installed and running was get it integrated with VSCode. Tabby helps out with this by providing a VSCode icon in the drop down menu that becomes visible when you click on your avatar image in the WebUI’s upper right corner. Clicking the VSCode’s icon brings you to the homepage of the Tabby extension for VSCode.

Also note that your endpoint URL and access token is displayed when you click on your avatar image. You’ll need these to configure the VSCode extension to call Tabby.

Impressions:

With a CPU-only setup this coding assistant isn’t going to win any races, but it’s fairly impressive what it can do with limited resources. I’m impressed enough to want to see what can be done with a proper GPU integrated. The model I chose is also pretty under-powered and requires a lot of checking up on. Again, with better hardware, I’ll be able to use a more capable model from Tabby’s model registry.