Let's install a completely local ChatGPT-like bot on an Android phone - with a few commands, sorry iPhone people..

Quick Demo

This is running on better phone than the one in the guide and it has a different operating system if you want to know more about that let me know!

Setup

Okay so in this post we will utilize some of the things we learned already like Nix and F-Droid.

In case you wanted to do this on a MacOS, BSD or a Linux laptop just get the Nix Package Manager one line installer from their website and you're good to go. You can just run the few commands after that. The power of Nix..

Now since we want to do this on our Android phone we have to download the Nix-On-Droid app, and to do that we will need F-Droid Open-Source app "store". ( Everything is free there guys I always look to bring it to people's attention. And consider supporting the developers if you see something useful. )

Let's get F-Droid from their page or Git repository if you really want to build it and push it on your phone yourself, but if you know how to do that much you probably didn't need me to tell you about this even.

Now your phone will probably ask for permissions to install and start panicing about the file even before that.

It does that for anything outside of the Playstore, even if it's Google Chrome...

The phone should still walk you through enabling it, just click on whatever message you get

If you are worried about this I encourage you to do your own research on everything, always, not just the stuff we teach.

It might take a little bit to for the app to setup and link everything up. Next step is to find Nix-on-Droid.

This is basically a terminal emulator and the Nix Package Manager configured and ready to use.

You'll probably need to enable F-droid to download APKs now

Now that that's done we need to open the app and it's gonna prompt us for a location, I haven't been playing around with it but in theory if you have a rooted phone ( administrator access ) you should be able to install it anywhere. Either way I haven't tested anything myself so I suggest using the default suggested location to avoid any issues with access.

Give it time whatever it takes and whatever we are doing especially if you have an older phone and / or a slower internet connection.

I suggest disabling flake support for quicker and easier inital setup, there's less hoops to jump this way and then later if you want to play around with some projects yourself, which hopefully you will, you can redo it then.

I enabled it now because I will play around with it and I don't mind waiting to get this running. BUT it will be a very looong time you will probably forget you started doing this if you decide to enable flakes...

Taking a long time getting some errors...

And then when you forgot what you started doing in the first place..

Finally we've jumped all those hoops let's do the few commands but first let's make sure Nix is running properly ( There shouldn't be any issues but paste the errors into DuckDuckGo or ChatGPT if you have any )

# Paste in the text / commands individually and press Enter to run them (In case some people never did something like this before)

whoami 

ls -a

nix-shell -p htop tmux

# nix-shell only installs the packages for the current shell / session. If you exit the app you won't have these apps again.. Unless you run the command obviously

# We don't need htop but it will display the current resource usage and we will need tmux for the LLMs, it allows us run more terminal session on any terminal emulator ( Nix-On-Droid is built on top of Termux, which doesn't have this feature. Some have this feature built in but let's not get too much off topic )

htop

Visually appealing

Now if that's all nice and daisey let's finally do the few commands

The Magic

nix-env -iA nixpkgs.ollama 

# nix-env installs the packages for all the future shells / sessions. Basically you don't have to run nix-shell -p every time you want to run things.

# I recommend doing this with tmux as well

# This is a topic for it's own blog post but it's ollama is software that allows you to run model inference.

Now if you're in the same shell where you installed htop and tmux run tmux if not make tmux available to you and run it.

tmux

Now you want to press CTRL+b+c on that small virtual keyboard ( All the inputs for tmux-commands are lower case ). Okay if that went well you should have 2 windows now ( 0 and 1 )

Now run :

ollama serve

Now press CTRL+b but this time press n or p after that run :

# This is a small model and it should run on quiet a few phones!

ollama run tinyllama:1.1b-chat-v0.6-q4_K_M # You can try q4_K_S for an even smaller model
ollama run tinyllama:1.1b-chat-v0.6-q3_K_S
# This one should run on almost any device, fast.
# Very small. I used it to make the video.

# Or run :

ollama run tinyllama:1.1b-chat-v0.6-q5_K_S # For a larger model keep in mind the larger the models slower it will be.

What you actually came for!

First setup the enviroment: Get the F-Droid Open-Source App"store" and the Nix-on-Droid app. Or on other devices just get the nix package manager from their website and you're good to go.

Second run a few commands to configured the LLM prompt

nix-env -iA nixpkgs.ollama nixpkgs.tmux

tmux # CTRL+b and c on the virtual keyboard to get 2 windows

ollama serve # in one window then CTRL+b and n to change to the next the window

Finally we run one more command

ollama run tinyllama:1.1b-chat-v0.6-q3_K_S # And you wait for a little bit for the model to download and prompt you - depending on your connection.

I have made the guide and I run models on an old Sony L2 ( it is slow but it runs ), I made the actual video of running the LLM on a little bit better phone because I needed to record the screen as well. If you want a guide on how to setup everything the way I have it let me know.

I know I probably made it a bit long but I hope everyone kinda had an idea of what was going on and it all worked for you!