Explore the VAE

from diffusers import AutoencoderKL
vae = AutoencoderKL.from_pretrained("CompVis/stable-diffusion-v1-4", subfolder="vae")

/usr/local/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py:205: UserWarning: The `local_dir_use_symlinks` argument is deprecated and ignored in `hf_hub_download`. Downloading to a local directory does not use symlinks anymore.
  warnings.warn(
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

!curl --output macaw.jpg 'https://lafeber.com/pet-birds/wp-content/uploads/2018/06/Scarlet-Macaw-2.jpg'

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 58602  100 58602    0     0  1014k      0 --:--:-- --:--:-- --:--:-- 1021k

from PIL import Image
img = Image.open('macaw.jpg').resize((512,512)); img

The image needs to be converted to a PyTorch tensor.

import torch
from torchvision.transforms import v2
v2.Compose([v2.ToImage(), v2.ToDtype(torch.float32, scale=True)])(img)

Image([[[0.1804, 0.1804, 0.1804,  ..., 0.2392, 0.2392, 0.2431],
        [0.1843, 0.1843, 0.1843,  ..., 0.2392, 0.2392, 0.2431],
        [0.1843, 0.1882, 0.1882,  ..., 0.2431, 0.2431, 0.2431],
        ...,
        [0.8471, 0.8471, 0.8471,  ..., 0.5373, 0.5490, 0.5686],
        [0.8471, 0.8471, 0.8471,  ..., 0.5333, 0.5451, 0.5608],
        [0.8471, 0.8471, 0.8471,  ..., 0.5294, 0.5451, 0.5569]],

       [[0.1529, 0.1529, 0.1529,  ..., 0.2118, 0.2118, 0.2157],
        [0.1569, 0.1569, 0.1569,  ..., 0.2118, 0.2118, 0.2157],
        [0.1569, 0.1608, 0.1608,  ..., 0.2157, 0.2157, 0.2157],
        ...,
        [0.8196, 0.8196, 0.8196,  ..., 0.5569, 0.5647, 0.5843],
        [0.8196, 0.8196, 0.8196,  ..., 0.5569, 0.5647, 0.5765],
        [0.8196, 0.8196, 0.8196,  ..., 0.5529, 0.5608, 0.5725]],

       [[0.1843, 0.1843, 0.1843,  ..., 0.2431, 0.2431, 0.2471],
        [0.1882, 0.1882, 0.1882,  ..., 0.2431, 0.2431, 0.2471],
        [0.1882, 0.1922, 0.1922,  ..., 0.2471, 0.2471, 0.2471],
        ...,
        [0.7098, 0.7098, 0.7216,  ..., 0.5569, 0.5686, 0.5882],
        [0.7098, 0.7098, 0.7216,  ..., 0.5569, 0.5647, 0.5804],
        [0.7098, 0.7098, 0.7216,  ..., 0.5529, 0.5647, 0.5765]]], )

ToTensor = v2.Compose([v2.ToImage(), v2.ToDtype(torch.float32, scale=True)]); ToTensor

Compose(
      ToImage()
      ToDtype(scale=True)
)

ToTensor(img).shape, ToTensor(img)[None,...].shape

(torch.Size([3, 512, 512]), torch.Size([1, 3, 512, 512]))

v2.ToDtype needs to be usedd. Otherwise directly using v2.ToImage results in a uint8 tensor

import torch
from torchvision.transforms import v2
with torch.no_grad():
    lat = vae.encode(ToTensor(img)[None,...]*2-1)
    lat = 0.18215 * lat.latent_dist.sample()

lat.shape, lat[0].shape

(torch.Size([1, 4, 64, 64]), torch.Size([4, 64, 64]))

import matplotlib.pyplot as plt
fig,axs = plt.subplots(1,4, figsize=(16,4))
for c in range(4): 
    axs[c].imshow(lat[0][c], cmap='Greys')
    axs[c].axis('off')

APP

The app will be written with FastHTML. I will be previewing the app inline within this dialog. For this to work, I first tweak a Starlette setting so cookies also work within the inline previews.

from starlette.middleware.sessions import SessionMiddleware

class PartitionedSessionMiddleware(SessionMiddleware):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.security_flags += "; partitioned"

Now I define the app.

from fasthtml.common import *

app = FastHTML(hdrs=(picolink, MarkdownJS(), HighlightJS()), debug=True, session_cookie='vae_app', same_site='none', sess_https_only=True, sess_cls=PartitionedSessionMiddleware)
rt = app.route

Now I write the code that will allow me to preview the app inline.

from functools import partial
from fasthtml.jupyter import *

port = 8002
url = 'the-vae-visualizer.solveit.pub'
server = JupyUvi(app, port=port)

def get_preview(app): return partial(HTMX, app=app, host=url, port=443)
prev = get_preview(app); prev()

def p(component):
    if hasattr(component, 'app'): return prev(component)
    else:
        @rt
        def _test(): return component
        return prev(_test)
p(Input())

render_ft()

render_ft enables rendering of HTML components inline.

from contextvars import ContextVar
current_lang = ContextVar('lang', default='zh'); current_lang

<ContextVar name='lang' default='zh' at 0x75bbc4baab10>

@rt
def setsession(session):
    current_lang.set(session.setdefault('lang', 'zh'))

T = {
    'zh': {
        'title': 'VAE 压缩探索器',
        'process': '开始处理',
        'advanced_settings': '高级设置',
        'image_label': '图片',
        'vae_scale_label': 'VAE 缩放',
        'vae_scale_tip': '归一化潜在空间，匹配噪声调度器的尺度',
        'norm_one_label': '归一化因子1',
        'norm_two_label': '归一化因子2',
        'norm_tip': '将像素值从 [0,1] 映射到 [-1,1]，匹配 VAE 训练分布',
        'no_result': '这里还空着呢～',
        'error_msg': '❌ 错误：',
        'vae_scale_desc': 'VAE 缩放系数继承自 Stable Diffusion 1.4。它对潜在空间进行缩放，使其方差与噪声调度器的预期尺度保持一致，防止迭代采样过程中的漂移。',
        'norm_desc': 'Norm 参数负责将像素值从 [0,1] 重映射到 [-1,1]，以匹配 VAE 训练时使用的像素值。如果有一组 0 到 1 之间的值，乘以 2 再减去 1，就能将它们转换到 -1 到 1 的范围。',
        'footer': '版本 0.0.1 | 使用 ',
        'crafted_by': ' 制作',
        'name': '纳克威·萨尔曼'
    },
    'en': {
        'title': 'VAE Compression Explorer',
        'process': 'Process',
        'advanced_settings': 'Advanced Settings',
        'image_label': 'Image',
        'vae_scale_label': 'VAE Scale',
        'vae_scale_tip': "Normalizes latent variance to align with the noise scheduler's scale",
        'norm_one_label': 'Norm 1',
        'norm_two_label': 'Norm 2',
        'norm_tip': 'Rescales pixel values from [0,1] to [-1,1] to match VAE training distribution',
        'no_result': 'Nothing here yet!',
        'error_msg': '❌ Error: ',
        'vae_scale_desc': 'The VAE Scale is a variance normalization constant inherited from Stable Diffusion 1.4. It scales the latent so its variance is consistent with the noise scheduler\'s expected scale. This prevents drift during the iterative sampling process.',
        'norm_desc': 'The "Norm" parameters are responsible for remapping pixel values from [0,1] to [-1,1], matching the pixel values the VAE was trained on. If we have a set of values betweeen 0 and 1, multiply them by 2, and then subtract one, we\'ve transformed that set of values to be between -1 and 1.',
        'footer': 'Version 0.0.1 | Crafted with ',
        'crafted_by': ' by ',
        'name': 'Salman Naqvi'
    }
}

def t(key, session=None):
    lang = session.get('lang', 'zh') if session else current_lang.get()
    return T[lang].get(key, key)

t('title')

'VAE 压缩探索器'

def img2lat(img, scale, nconst1, nconst2):
    with torch.no_grad():
        lat = vae.encode(ToTensor(img)[None, ...]*nconst1-nconst2)
        return scale * lat.latent_dist.sample()

def lat2img(lat, scale, nconst1, nconst2):
    lat *= 1/scale
    with torch.no_grad(): img = vae.decode(lat).sample
    img = ((img+nconst2)/nconst1).clamp(0,1)
    img = img.detach().permute(0,2,3,1).numpy()
    return Image.fromarray((img*255).round().astype('uint8')[0])

def lat2plot(lat):
    fig,axs = plt.subplots(1,4, figsize=(16,4))
    for c in range(4):
        axs[c].imshow(lat[0][c], cmap='Greys')
        axs[c].axis('off')
    return fig

from io import BytesIO
from base64 import b64encode
def process(fbytes, scale=0.18215, nconst1=2, nconst2=1):
    oimg = Image.open(BytesIO(fbytes)).resize((512,512))
    lat = img2lat(oimg, scale, nconst1, nconst2)
    fig = lat2plot(lat)
    # pimg = lat2img(lat, scale, nconst1, nconst2)
        
    buf1, buf2 = BytesIO(), BytesIO()
    fig .savefig(buf1, format='png')
    # pimg.save   (buf2, format='png')
    oimg.save   (buf2, format='png')
    
    def buf2b64(buf): buf.seek(0); return b64encode(buf.read()).decode()
    return mapt(buf2b64, [buf1,buf2])

buf = BytesIO()
img.save(buf, format='png')
buf.seek(0)
a, b = process(fbytes=buf.getvalue())

a[:200], b[:200]

('iVBORw0KGgoAAAANSUhEUgAABkAAAAGQCAYAAAD2lawGAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjgsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvwVt1zgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAsqJJREFUeJztvXvwntdZnvuEHjgUEmM7',
 'iVBORw0KGgoAAAANSUhEUgAAAgAAAAIACAIAAAB7GkOtAAEAAElEQVR4nKz9Sbdlu44uhn0f5toRcc65WTy9J0vDsp6b6tn//6e442F37DGeLOWtThGxFwE3QIBgMdfeN1Mz88ZZe04WIIiKIAjy//a//N8BACBJEuVZ3uTvpVi+MbO9lpmJiKrW6lkS98/Staqamb9c')

from base64 import b64encode
from io import BytesIO
from PIL import Image
@rt
async def upload(file:UploadFile, vae_scale:float, norm_one:float, norm_two:float): 
    try:
        fbytes = await file.read()
        lat_b64, proc_b64 = process(fbytes, vae_scale, norm_one, norm_two)
        return Div(style="display:flex; flex-direction:column")(
            Img(src=f'data:image/jpeg;base64,{lat_b64}', style='max-width:800px; width: 100%'),
            Img(src=f'data:image/jpeg;base64,{proc_b64}', style='max-width:800px; width: 50%; display:block; margin:auto'),
        )
    except Exception as e: return Article(f"{t('error_msg')}{e}", style="color: var(--pico-del-color); border-color: var(--pico-del-color);")

from starlette.datastructures import UploadFile
buf.seek(0)
uf = UploadFile(file=buf); uf

UploadFile(filename=None, size=None, headers=Headers({}))

# 
# await upload(uf, 0.18215, 2, 1)

import fasthtml.components as fc
@delegates(fc.Input)
def LabelledInput(lbl=None, tip=None, id=None, name=None, **kwargs): 
    assert id is not None or name is not None, "id or name required"
    return Div(style='flex-direction:column')(
        fc.Label(lbl, data_tooltip=tip, fr=f'inp-{id or name}', id=f'lbl-{id or name}'), 
        fc.Input(id=f'inp-{id or name}', name=name, **kwargs)
    )

p(LabelledInput(placeholder='hehe', name='testing', tip='LOL', lbl='haha'))

def Dashboard():
    return Card(style='flex:2')(
        Form(hx_post=upload, hx_target='#res', hx_swap='innerHTML',
             hx_indicator='#loading', hx_disabled_elt='#process-btn, #upload-inp',
             enctype="multipart/form-data")(
            UploadWidget(),
            Div(style='margin-top: 1rem')(SettingsWidget()),
        )
    )

def UploadWidget():
    return Div(style='display:flex; align-items:center; gap:0.5rem')(
        UploadInput(t('image_label'), name='file'),
        Button(t('process'), type='submit', id='process-btn', style='margin-bottom:0;'),
    )

@delegates(fc.Input)
def UploadInput(lbl=None, id=None, name=None, **kwargs): 
    assert id is not None or name is not None, "id or name required"
    return Div()(
        fc.Label(style='margin-bottom:0', role='button')(lbl, id=f"lbl-{id or name}", fr=f"inp-{id or name}"),
        fc.Input(id=f"inp-{id or name}", name=name, type='file', accept='image/*', style='display:none; margin-bottom:0', onchange="this.previousElementSibling.style.backgroundColor = this.files[0] ? 'green' : ''", **kwargs),
    )

p(UploadInput('test', id='test'))

p(UploadWidget())

def SettingsWidget():
    return Details(
        Summary(t('advanced_settings'), id='adv-summary'),
        Div(style='display:flex; gap:1rem;')(
            LabelledInput(type='number', name='vae_scale', value=0.18215, step='any',
                lbl=t('vae_scale_label'), tip=t('vae_scale_tip')),
            LabelledInput(type='number', name='norm_one', value=2,
                lbl=t('norm_one_label'), tip=t('norm_tip')),
            LabelledInput(type='number', name='norm_two', value=1,
                lbl=t('norm_two_label'), tip=t('norm_tip')),
        ),
        P(t('vae_scale_desc'), id='vae-scale-desc'),
        P(t('norm_desc'), id='norm-desc')
    )

p(SettingsWidget())

p(Dashboard())

@rt
def toggle_lang(session):
    lang = 'en' if session.get('lang') == 'zh' else 'zh'
    current_lang.set(lang)
    session['lang'] = lang
    return (
        Button(t('process'), type='submit', id='process-btn', hx_swap_oob='true', style='margin-bottom:0'),
        H1(t('title'), id='title', hx_swap_oob='true'),
        Summary(t('advanced_settings'), id='adv-summary', hx_swap_oob='true'),
        Div(id='res', style='display:flex; align-items:center; justify-content:center; flex:1', hx_swap_oob='true')(t('no_result')),
        Label(t('image_label'), id='lbl-file', fr='inp-file', role='button', hx_swap_oob='true', style='margin-bottom:0'),
        Label(t('vae_scale_label'), id='lbl-vae_scale', fr='vae_scale', hx_swap_oob='true', data_tooltip=t('vae_scale_tip')),
        Label(t('norm_one_label'), id='lbl-norm_one', fr='norm_one', hx_swap_oob='true', data_tooltip=t('norm_tip')),
        Label(t('norm_two_label'), id='lbl-norm_two', fr='norm_two', hx_swap_oob='true', data_tooltip=t('norm_tip')),
        P(t('vae_scale_desc'), id='vae-scale-desc', hx_swap_oob='true'),
        P(t('norm_desc'), id='norm-desc', hx_swap_oob='true'),
        Footer(id='footer', hx_swap_oob='true', style='color: var(--pico-muted-color);')(t('footer'), A('FastHTML', href='https://fastht.ml'), t('crafted_by'), A(t('name'), href='https://forbo7.github.io/'))
    )

@rt
def index(session): 
    setsession(session)
    return Card(style="padding: 2rem; max-width: 1200px; margin: 2rem auto;")(
        Div(style="display:flex; align-items:center; gap:1rem; margin-bottom:2rem")(
            H1(t('title'), id='title'),
            Button('🌐', hx_post=toggle_lang, hx_swap='none'),
            Div(id='loading', aria_busy='true', cls='htmx-indicator')(''),
        ),
        Div(style="display:flex; gap:1rem; align-items:stretch")(
            Dashboard(),
            Card(style='flex:3; margin-top:1rem; display:flex; flex-direction:column; align-items:center; min-height: 200px;')(
                Div(id='res', style='display:flex; align-items:center; justify-content:center; flex:1')(t('no_result')),
            )
        ),
        Footer(id='footer', style='color: var(--pico-muted-color);')(t('footer'), A('FastHTML', href='https://fastht.ml'), t('crafted_by'), A(t('name'), href='https://forbo7.github.io/'))
    )

prev(index)

On SolveIt

We need to first setup SSH Key Authentication. On the SolveIt instance, generate a new SSH key and note/copy the public key.

ssh-keygen -t ed25519
cat ~/.ssh/id_ed25519.pub

-t ed25519 generates a key using the ed25519 algorithm. This is supposedly safer, quicker, and a more concise alternative to the traditional RSA algorithm.

On Your Machine

First ensure that remote access is enabled in System Settings on your local machine.

Next, install bore-cli. If you haven’t, also install tmux as it helps keep SSH connections persistent.

brew install bore-cli
brew install tmux

Then, start a new tmux session. Here, I’ve called the session bore.

tmux new -s bore

Take the public key you generated on SolveIt. Then, on the local machine, add it to the list of authorized keys.

echo "YOUR_PUBLIC_KEY_HERE" >> ~/.ssh/authorized_keys

Now it’s time to run bore. Note down the port that bore returns.

bore local 22 --to bore.pub

This command exposes port 22 on your local machine to the bore public servers through an encrypted tunnel. Why 22? Because that is the default port macOS uses for SSH after enabling remote access. The port you get back is the port bore randomly allocates on its public servers.

Testing the Connection from SolveIt

And that should be it! Test the connection from SolveIt:

ssh -o StrictHostKeyChecking=no YOUR_USERNAME@bore.pub -p YOUR_PORT "echo Connection successful!"

If you are running this command outside a Jupyter Notebook or a SolveIt Dialog, you can omit the -o StrictHostKeyChecking=no parameter. This parameter allows us to bypass the prompt that asks us to verify the connection.

fastcore has a run_cmd function we can use to abstract how we run commands. We could also use it to produce a tool the LLM can use.

from fastcore.all import *
?run_cmd

def run_cmd(
    cmd:str, # The command name to run
    argstr:str='', # All args to the command, will be split with shlex
    disallow_re:str=None, # optional regex which, if matched on argstr, will disallow the command
    allow_re:str=None, # optional regex which, if not matched on argstr, will disallow the command
):

Run `cmd` passing split `argstr`, optionally checking for allowed argstr

def mac(cmd:str, port:str): 
    "Execute bash commands on the user's local Mac."
    return print(run_cmd('ssh', f'YOUR_USERNAME@bore.pub -p {port} {cmd}'))

Now you’re able to control your local machine directly from SolveIt. You could also pass it as a tool to the LLM with &`mac`.

If you have any comments, questions, suggestions, feedback, criticisms, or corrections, please do post them down in the comment section below!

GitHubq

GitHubq is a web service from Answer.Ai that allows you to easily make GitHub repos available as context for LLMs. The basic way to use the service is to change github.com to githubq.com.

You can change what you get back with various parameters. For instance,

• exts=py returns only files ending with .py
• sigs_only=True returns only function signatures and their docstrings.

toolslm.xml

See the full documentation of Answer.AI’s toolslm library here.

from toolslm.xml import repo2ctx
print(repo2ctx('https://github.com/AnswerDotAI/shell_sage', exts='py')[:100])

<documents title="GitHub repository contents from AnswerDotAI/shell_sage/main (filters applied -- ex

?repo2ctx

def repo2ctx(
    owner:str, # GitHub repo owner or "owner/repo" or a full github URL
    repo:str=None, # GitHub repo name (leave empty if using "owner/repo" or URL format for owner param)
    ref:str=None, # Git ref (branch/tag/sha) (get from URL not provided); defaults to repo's default branch
    folder:str=None, # Only include files under this path (get from URL not provided)
    show_filters:bool=True, # Include filter info in title?
    token:str=None, # GitHub token (uses GITHUB_TOKEN env var if None)
    prefix:bool=False, # Include Anthropic's suggested prose intro?
    out:bool=True, # Include notebook cell outputs?
    include_base:bool=True, # Include full path in src?
    title:str=None, # Optional title attr for Documents element
    max_size:int=100000, # Skip files larger than this (bytes)
    max_total:int=10000000, # Max total output size in bytes
    readme_first:bool=False, # Prioritize README files at start of context?
    files_only:bool=False, # Return dict of {filename: size} instead of context?
    sigs_only:bool=False, # Return signatures instead of full text for python files?
    ids:bool=True, # Include cell ids in notebooks?
    recursive:bool=True, # search subfolders
    symlinks:bool=True, # follow symlinks?
    file_glob:str=None, # Only include files matching glob
    file_re:str=None, # Only include files matching regex
    folder_re:str=None, # Only enter folders matching regex
    skip_file_glob:str=None, # Skip files matching glob
    skip_file_re:str=None, # Skip files matching regex
    skip_folder_re:str=None, # Skip folders matching regex,
    ret_folders:bool=False, # return folders, not just files
    sort:bool=True, # sort files by name within each folder
    types:str \| list=None, # list or comma-separated str of ext types from: py, js, java, c, cpp, rb, r, ex, sh, web, doc, cfg
    exts:str \| list=None, # list or comma-separated str of exts to include
)->Union: # XML for LM context, or dict of file sizes

Convert GitHub repo to XML context without cloning

!ls 

def folder2ctx(
    path:Union, # Folder to read
    prefix:bool=False, # Include Anthropic's suggested prose intro?
    out:bool=True, # Include notebook cell outputs?
    include_base:bool=True, # Include full path in src?
    title:str=None, # Optional title attr for Documents element
    max_size:int=100000, # Skip files larger than this (bytes)
    max_total:int=10000000, # Max total output size in bytes
    readme_first:bool=False, # Prioritize README files at start of context?
    files_only:bool=False, # Return dict of {filename: size} instead of context?
    sigs_only:bool=False, # Return signatures instead of full text for python files?
    ids:bool=True, # Include cell ids in notebooks?
    recursive:bool=True, # search subfolders
    symlinks:bool=True, # follow symlinks?
    file_glob:str=None, # Only include files matching glob
    file_re:str=None, # Only include files matching regex
    folder_re:str=None, # Only enter folders matching regex
    skip_file_glob:str=None, # Skip files matching glob
    skip_file_re:str=None, # Skip files matching regex
    skip_folder_re:str=None, # Skip folders matching regex,
    ret_folders:bool=False, # return folders, not just files
    sort:bool=True, # sort files by name within each folder
    types:str \| list=None, # list or comma-separated str of ext types from: py, js, java, c, cpp, rb, r, ex, sh, web, doc, cfg
    exts:str \| list=None, # list or comma-separated str of exts to include
)->Union:

Convert folder contents to XML context, handling notebooks

!ls

'00-09 系统'   70ec03a05ceeffe770a8d00297fa241a   solveit_settings.json
'20-29 大学'   jobs.csv                 test
'30–39 学习'   shell_sage           暂时对话

from toolslm.xml import folder2ctx
print(folder2ctx('shell_sage', exts='py')[:250])

<documents><document index="1"><src>
shell_sage/setup.py
</src><document-content>
from pkg_resources import parse_version
from configparser import ConfigParser
import setuptools, shlex
assert parse_version(setuptools.__version__)>=parse_version('36.2

from toolslm.xml import py2sigs
print(py2sigs('shell_sage/shell_sage/core.py')[:250])

def __rich_console__(self:CodeBlock, console, options):

def Chat(*arg, **kw):
    "Lazy load lisette to make ssage more responsive"

def _aliases(shell):

def _sys_info():

def get_pane(n, pid=None):
    "Get output from a tmux pane"

def get_panes(

print(sym2pkgctx(sc.ShellSageConfig)[:250])

<documents><document index="1"><src>
/usr/local/lib/python3.12/site-packages/shell_sage/__init__.py
</src><document-content>
__version__ = "1.0.4"
</document-content></document><document index="2"><src>
/usr/local/lib/python3.12/site-packages/shell_s

from toolslm.xml import sym2folderctx
print(sym2folderctx(sc.ShellSageConfig)[:250])

<documents><document index="1"><src>
/usr/local/lib/python3.12/site-packages/shell_sage/__init__.py
</src><document-content>
__version__ = "1.0.4"
</document-content></document><document index="2"><src>
/usr/local/lib/python3.12/site-packages/shell_s

?sym2pkgctx

def sym2pkgctx(
    sym,
    types:str \| list='py', # list or comma-separated str of ext types from: py, js, java, c, cpp, rb, r, ex, sh, web, doc, cfg
    skip_file_re:str='^_mod', prefix:bool=False, # Include Anthropic's suggested prose intro?
    out:bool=True, # Include notebook cell outputs?
    include_base:bool=True, # Include full path in src?
    title:str=None, # Optional title attr for Documents element
    max_size:int=100000, # Skip files larger than this (bytes)
    max_total:int=10000000, # Max total output size in bytes
    readme_first:bool=False, # Prioritize README files at start of context?
    files_only:bool=False, # Return dict of {filename: size} instead of context?
    sigs_only:bool=False, # Return signatures instead of full text for python files?
    ids:bool=True, # Include cell ids in notebooks?
    recursive:bool=True, # search subfolders
    symlinks:bool=True, # follow symlinks?
    file_glob:str=None, # Only include files matching glob
    file_re:str=None, # Only include files matching regex
    folder_re:str=None, # Only enter folders matching regex
    skip_file_glob:str=None, # Skip files matching glob
    skip_folder_re:str=None, # Skip folders matching regex,
    ret_folders:bool=False, # return folders, not just files
    sort:bool=True, # sort files by name within each folder
    exts:str \| list=None, # list or comma-separated str of exts to include
):

Return repo context for a symbol's root package

contextkit

Next up is contextkit’s read module. However, this approach is much more hands on as the module’s functions do not return XML formatted contents. Though this does allow you to perform your own processing/handling. You will need to have a SSH key set up between your machine and GitHub as the functions in read create temporary clones of the GitHub repo.

contextkit.reads read_gh_repo returns a dictionary in the format of {'path': 'contents'}.

import contextkit.read as rd
rd.read_gh_repo('https://github.com/AnswerDotAI/shell_sage').keys()

dict_keys(['/app/data/.cache/contextkit_git_clones/shell_sage/settings.ini', '/app/data/.cache/contextkit_git_clones/shell_sage/CONTRIBUTING.md', '/app/data/.cache/contextkit_git_clones/shell_sage/LICENSE', '/app/data/.cache/contextkit_git_clones/shell_sage/README.md', '/app/data/.cache/contextkit_git_clones/shell_sage/pyproject.toml', '/app/data/.cache/contextkit_git_clones/shell_sage/uv.lock', '/app/data/.cache/contextkit_git_clones/shell_sage/MANIFEST.in', '/app/data/.cache/contextkit_git_clones/shell_sage/setup.py', '/app/data/.cache/contextkit_git_clones/shell_sage/CHANGELOG.md', '/app/data/.cache/contextkit_git_clones/shell_sage/ipython/00-init.py', '/app/data/.cache/contextkit_git_clones/shell_sage/nbs/_quarto.yml', '/app/data/.cache/contextkit_git_clones/shell_sage/nbs/CNAME', '/app/data/.cache/contextkit_git_clones/shell_sage/nbs/sidebar.yml', '/app/data/.cache/contextkit_git_clones/shell_sage/nbs/tmux.conf', '/app/data/.cache/contextkit_git_clones/shell_sage/nbs/01_config.ipynb', '/app/data/.cache/contextkit_git_clones/shell_sage/nbs/00_core.ipynb', '/app/data/.cache/contextkit_git_clones/shell_sage/nbs/styles.css', '/app/data/.cache/contextkit_git_clones/shell_sage/nbs/index.ipynb', '/app/data/.cache/contextkit_git_clones/shell_sage/nbs/nbdev.yml', '/app/data/.cache/contextkit_git_clones/shell_sage/shell_sage/core.py', '/app/data/.cache/contextkit_git_clones/shell_sage/shell_sage/__init__.py', '/app/data/.cache/contextkit_git_clones/shell_sage/shell_sage/_modidx.py', '/app/data/.cache/contextkit_git_clones/shell_sage/shell_sage/config.py', '/app/data/.cache/contextkit_git_clones/shell_sage/debug/aliases.py', '/app/data/.cache/contextkit_git_clones/shell_sage/debug/safe_aliases.py', '/app/data/.cache/contextkit_git_clones/shell_sage/debug/probe_tty.py'])

read_gh_file returns the contents of the file as a string.

rd.read_gh_file('https://github.com/AnswerDotAI/shell_sage/blob/main/shell_sage/core.py')[:250]

"# AUTOGENERATED! DO NOT EDIT! File to edit: ../nbs/00_core.ipynb.\n\n# %% auto 0\n__all__ = ['console', 'print', 'sp', 'ssp', 'default_cfg', 'tools', 'sps', 'log_path', 'Chat', 'get_pane', 'get_panes',\n           'tmux_history_lim', 'get_history', 'get_"

print(rd.read_gh_file('https://github.com/AnswerDotAI/shell_sage/blob/main/shell_sage/core.py')[:250])

# AUTOGENERATED! DO NOT EDIT! File to edit: ../nbs/00_core.ipynb.

# %% auto 0
__all__ = ['console', 'print', 'sp', 'ssp', 'default_cfg', 'tools', 'sps', 'log_path', 'Chat', 'get_pane', 'get_panes',
           'tmux_history_lim', 'get_history', 'get_

See the full documentation of Answer.AI’s contextkit library here.

SolveIt

If you’re in SolveIt, you can also change directory and ask the LLM to directly read through a particular directory for you with its builtin tools. Note that you should use % rather than ! so the directory switches in a non-temporary manner.

%cd shell_sage

/app/data/shell_sage

Can you look through the directory and tell me what are the key functions of the shell sage library?

{
  "id": "toolu_01TcJme7ugD8Xzq3rBZLjinR",
  "call": {
    "function": "view",
    "arguments": {
      "path": "."
    }
  },
  "result": "'Directory contents of /app/<TRUNCATED>"
}

{
  "id": "toolu_01N9i48X1GWBFrBj9awo8qzk",
  "call": {
    "function": "view",
    "arguments": {
      "path": "shell_sage/core.py",
      "view_range": "[1, 50]"
    }
  },
  "result": "'# AUTOGENERATED! DO NOT EDIT! File to edit: ../nbs/00_core.ipynb.\\n\\n# %% auto 0\\n__all__ = [\\'console\\', \\'print\\', \\'sp\\', \\'ssp\\', \\'default_cfg\\', \\'tools\\', \\'sps\\', \\'log_path\\', \\'Chat\\', \\'get_pane\\', \\'get_panes\\',\\n           \\'tmux_history_lim\\', \\'get_history\\', \\'get_opts\\', \\'with_permission\\', \\'get_sage\\', \\'get_res\\', \\'Log\\', \\'mk_db\\',\\n           \\'main\\', \\'extract_cf\\', \\'extract\\']\\n\\n# %% ../nbs/00_core.ipynb 3\\nfrom datetime import datetime\\nfrom itertools import accumulate\\nfrom fastcore.script import *\\nfrom fastcore.tools import *\\nfrom fastcore.utils import *\\nfro<TRUNCATED>"
}

{
  "id": "toolu_011DD4XT7Yge4mwqi49s3u6Z",
  "call": {
    "function": "read_url",
    "arguments": {
      "url": "https://github.com/AnswerDotAI/shell_sage"
    }
  },
  "result": "'# AnswerDotAI/shell_sage\\nShellSage saves sysadmins\u2019 sanity by solving shell script snafus super swiftly\\n\\n## / Files\\n- \ud83d\udcc1 .github\\n- .gitignore\\n- CHANGELOG.md\\n- CONTRIBUTING.md\\n- LICENSE\\n- MANIFEST.in\\n- README.md\\n- \ud83d\udcc1 debug\\n- \ud83d\udcc1 ipython\\n- \ud83d\udcc1 nbs\\n- pyproject.toml\\n- \ud83d\udcc1 screenshots\\n- settings.ini\\n- setup.py\\n- \ud83d\udcc1 shell_sage\\n- uv.lock\\n\\n## README\\n# ShellSage\\n\\n\\n<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->\\n\\nShellSage is an AI-powered command-line assistant that integrates\\nseamlessly with your terminal workflow through tmux. It provides\\ncontextual help for shell ope<TRUNCATED>"
}

pip install shell-sage

At an Overview

Other toolslm.xml Goodies

Beyond the functions we’ve covered, toolslm.xml has other interesting functions:

Additional Functions: - files2ctx — Pass in a specific list of files rather than an entire folder - nb2xml — Convert Jupyter notebooks to XML format - json_to_xml — Convert JSON or Python dicts to XML - sym2file — Get the source file contents for any class or function

Useful Parameters: - prefix=True — Prepends Anthropic’s recommended preamble for document context - nums=True — Adds line numbers to code (handy for referencing specific lines)

Command Line Support: Both folder2ctx and repo2ctx are available as CLI tools, so you can use them directly from your terminal without writing Python.

There’s plenty more to discover. Explore the full module at the docs by pulling it into a LLM with the techniques we’ve learned here!

Conclusion

If you have any comments, questions, suggestions, feedback, criticisms, or corrections, please do post them down in the comment section below!

Settings and Headers

::: {#243ffab3 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:33:56.461282+00:00’}

from fasthtml.common import *

:::

::: {#f9ccd095 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:33:56.462308+00:00’}

bsc_hdrs = (
    Script(src="https://cdn.jsdelivr.net/npm/@tailwindcss/browser@4"),  # Need to download first.
    Link(rel='stylesheet', href='https://cdn.jsdelivr.net/npm/basecoat-css@0.3.6/dist/basecoat.cdn.min.css'),
    Script(src='https://cdn.jsdelivr.net/npm/basecoat-css@0.3.6/dist/js/all.min.js', defer=True),
)

:::

::: {#7d1e280a .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:33:56.482447+00:00’}

app = FastHTML(hdrs=(bsc_hdrs, MarkdownJS(), HighlightJS(langs=['python'])), session_cookie='mysessioncookie', debug=True)
rt = app.route

:::

A custom session cookie needs to be set for FastHTML sessions to work in SolveIt as SolveIt currently uses the default one.

Preview

Here I create a preview helper function that will allow me to interactively preview various components of the app.

from functools import partial
from fasthtml.jupyter import *

server = JupyUvi(app)

def get_preview(app):
    return partial(HTMX, app=app, host=None, port=None)
prev = get_preview(app); prev()

render_ft()

render_ft will render any output FastHTML. prev will display it in the context of the FastHTML app.

Database

Creating a database to store users and their dialogs, with dialogs linked to users via uid.

::: {#1f39f676 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:33:56.637473+00:00’}

from fastlite import *

:::

::: {#88d6aa36 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:33:56.668265+00:00’}

db = database('solvish.db')

:::

db

<Database <apsw.Connection object "/app/data/forbo7.github.io/forblog/posts/solvish.db" at 0x70ee23760500>>

::: {#dd112d66 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:33:57.092178+00:00’}

class User: id:int; restore_code:str
class Dialog: id:int; uid:int; dialog_num:int; name:str; messages:str; server_session:str

:::

::: {#6bd3be8e .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:33:57.093191+00:00’}

users = db.create(User, transform=True)
dialogs = db.create(Dialog, pk=('uid', 'dialog_num'), transform=True)

:::

Tables will not be overwritten as db.create() is idempotent.

users.delete_where()
dialogs.delete_where()

<Table dialog (id, uid, dialog_num, name, messages, server_session)>

.delete_where with no arguments passed will drop all rows from the table.

New Dialog

Creating a new dialog first auto-increments dialog_num based on the user’s existing dialogs, then sets session['current_dialog'] to track which dialog is active. The dialog is stored in the dialogs table for persistence across sessions, while also being loaded into user_chats as a live chat object for the current server session.

::: {#b7b8f919 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:33:57.163073+00:00’}

from lisette.core import *
from toolslm.shell import get_shell

sp = 'Respond in the language the user uses; The user has access to a Python interpreter that you can view; NEVER copy the format of the code interpreter messsages in your response.'
def make_chat(hist=[]): return Chat('moonshot/kimi-k2-0711-preview', hist=hist, sp=sp), get_shell()

:::

make_chat()

(<lisette.core.Chat at 0x70ee1a849eb0>,
 <IPython.terminal.interactiveshell.TerminalInteractiveShell at 0x70ee1a849f40>)

As my Kimi API key is compatable with servers in China, I have to change the Moonshoot endpoint.

::: {#35143833 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:33:58.702646+00:00’}

os.environ['MOONSHOT_API_BASE'] = 'https://api.moonshot.cn/v1'

:::

# from cachy import enable_cachy; enable_cachy()

cachy will cache my API calls.

make_chat()[0]('what do you have access to?')

::: {#50efa44f .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:34:01.492003+00:00’}

from coolname import generate_slug

@rt
def create_dialog(session):
    uid = session.get('userid')
    existing = dialogs(where=f'uid={uid}')
    dnum = max((d.dialog_num for d in existing), default=0) + 1
    session['current_dialog'] = dnum
    dialogs.insert(uid=uid, dialog_num=dnum, name=generate_slug(2), messages='[]')
    user_chats[(uid,dnum)] = make_chat()
    return Div(hx_get=f'/dialog?dnum={dnum}', hx_trigger='load', hx_target='#content', hx_swap='innerHTML')

:::

Session Persistance

::: {#67b4dc4e .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:34:01.506942+00:00’}

user_chats = {}

:::

I’ll be handling session persistance with this user_chats dictionary. If my server restarts, it means user_chats will be erased and thus a new session will be created for all users. Users can input their restore code to retrieve back their data.

::: {#89ceda80 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:34:01.511262+00:00’}

def ensure_user(session):
    uid = session.get('userid')
    if not uid or uid not in users:
        u = users.insert(restore_code=generate_slug(3))
        session['userid'] = u.id
    return session['userid']

:::

fake_session = {}

uid = ensure_user(fake_session)
fake_session, uid

({'userid': 1}, 1)

::: {#9ac5a9ef .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:34:01.526400+00:00’}

from fastcore.all import *
import json

def ensure_dialog(session):
    uid, dnum = map(session.get, ('userid', 'current_dialog'))
    if (uid, dnum) in user_chats: return
    elif dnum and (uid, dnum) in dialogs: 
        d = dialogs[(uid, dnum)]
        user_chats[(uid, dnum)] = make_chat(json.loads(d.messages))
    else: create_dialog(session)

:::

ensure_dialog acts as a lazy loader for chat sessions. It first checks whether the user’s current dialog is already active in memory—if so, there’s nothing to do. If the dialog exists in the database but hasn’t been loaded yet, it retrieves the stored messages and reconstructs the chat object. Otherwise, when no dialog exists at all, it creates a fresh one for the user.

ensure_dialog(fake_session)
user_chats

{(1, 1): (<lisette.core.Chat at 0x70ee1a6efda0>,
  <IPython.terminal.interactiveshell.TerminalInteractiveShell at 0x70ee225f31a0>)}

::: {#f4bf5236 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:34:01.546425+00:00’}

from contextvars import ContextVar
current_lang = ContextVar('lang', default='en'); current_lang

<ContextVar name='lang' default='en' at 0x70ee19141df0>

:::

Every time a request is made, a new ContextVar is created. This is a way I can keep language preferences seperate for each user.

::: {#f4830cbf .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:34:01.551464+00:00’}

@rt
def setsession(session):
    current_lang.set(session.setdefault('lang', 'en'))
    uid = ensure_user(session)
    ensure_dialog(session)

:::

prev(setsession)

The cookie only stores the user id. The data for the user is stored inside the current Python runtime. Once the runtime ends, the data is lost. The database permanently stores everything.

users()

[User(id=1, restore_code='provocative-lori-of-acceptance')]

dialogs()

[Dialog(id=None, uid=1, dialog_num=1, name='viridian-pegasus', messages='[]', server_session=None)]

user_chats

{(1, 1): (<lisette.core.Chat at 0x70ee1a6efda0>,
  <IPython.terminal.interactiveshell.TerminalInteractiveShell at 0x70ee225f31a0>)}

@rt
def debug(session):
    return Pre(str(dict(session)))
prev(debug)

@rt
def reset(session):
    session.clear()
    users.delete_where()
    dialogs.delete_where()
    return session

I won’t be exporting these two routes into the final application, else it’d allow users to mess around with other users!

Localization

A translation dictionary stores the strings for both Chinese and English. The helper function t() performs the localization.

::: {#95173331 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ hide_input=‘true’ time_run=‘2026-01-22T00:34:01.589178+00:00’}

T = {
    'zh': {
        'input_placeholder': '输入内容···',
        'submit': '提交',
        'ask_model': '问问模型···',
        'your_code': '你的代码···',
        'any_thoughts': '有啥想法···',
        'enter_code': '输入代码···',
        'code': '代码',
        'prompt': '题词',
        'note': '笔记',
        'tab': '标签',
        'all_dialogs': '所有对话',
        'new_dialog': '新对话',
        'delete': '删除',
        'save': '保存',
        'solveit_lite': 'solveit-轻量版',
        'footer': '由 FastHTML 匠心打造',
        'restore_code': '恢复码',
        'restore_failed': '‼ 恢复失败',
        'restore_error': '找不到与此恢复码关联的账户，请检查后重试。',
        'enter_restore_code': '输入恢复码···',
    },
    'en': {
        'input_placeholder': 'Enter content...',
        'submit': 'Submit',
        'ask_model': 'Ask the model...',
        'your_code': 'Your code...',
        'any_thoughts': 'Any thoughts...',
        'enter_code': 'Enter code...',
        'code': 'Code',
        'prompt': 'Prompt',
        'note': 'Note',
        'tab': 'Tab',
        'all_dialogs': 'All Dialogs',
        'new_dialog': 'New Dialog',
        'delete': 'Delete',
        'save': 'Save',
        'solveit_lite': 'solveit-lite',
        'footer': 'Crafted with FastHTML',
        'restore_code': 'Restore Code',
        'restore_failed': '‼ Restore Failed',
        'restore_error': 'No account found with this restore code. Please check and try again.',
        'enter_restore_code': 'Enter restore code...',
    }
}

:::

::: {#f66dc855 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:34:01.593643+00:00’}

def t(key, session=None): 
    lang = session.get('lang', 'zh') if session else current_lang.get()
    return T[lang].get(key, key)

:::

t('ask_model', {'lang':'zh'})

'问问模型···'

Chat Bubbles

Bubbles that display the dialog content. Each bubble type has its own color scheme—blue for code, red for prompts, and green for notes.

::: {#4a5b2004 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:34:01.603161+00:00’}

def Bubble(text, color, type='q', marked=True):
    colors = {
        'code': 'border-blue-500 bg-blue-50',
        'note': 'border-green-500 bg-green-50',
        'prompt': 'border-red-500 bg-red-50'
    }
    cls = f"{'marked' if marked else ''} {'ml-6' if type=='r' else ''} flex w-max max-w-[75%] flex-col gap-2 rounded-lg px-3 py-2 text-sm border-2 {colors[color]}"
    return Div(text, cls=cls)

:::

prev(Bubble('**嗨喽**', 'note'))

prev(Div(
    Bubble('嗨！', 'prompt'),
    Bubble('哈喽！', 'prompt', 'r')
))

prev(Bubble('''
```py
print('嗨！！！')
''', 'code'))

Component Wrappers

::: {#06830b37 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:34:01.636405+00:00’}

import fasthtml.components as fc

:::

::: {#0db9e155 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:34:01.640925+00:00’}

def mk_comp(name, def_cls):
    comp = getattr(fc, name)
    globals()[name] = lambda *args, var='', cls='', **kwargs: comp(*args, cls=f'{def_cls}{"-"+var if var else ""} {cls}', **kwargs)

:::

This wrapper will allow us to define HTML components with Basecoat styling without redundantly stating what that component is.

Before:

prev(Button('🔘', cls='btn'))

prev(Button('🔘', cls='btn-destructive'))

::: {#2e0f1420 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:34:01.661349+00:00’}

mk_comp('Button', 'btn')
mk_comp('Input', 'input')
mk_comp('Textarea', 'textarea')
mk_comp('Label', 'label')

:::

After:

prev(Button('🔘'))

prev(Button('🔘', var='destructive'))

to_xml(Button('🔘')), to_xml(Button('🔘', var='destructive'))

('<button class="btn ">🔘</button>',
 '<button class="btn-destructive ">🔘</button>')

Input Box

The input area that will be used for entering messages to the dialog.

::: {#ae807178 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:34:01.687299+00:00’}

def Inp(type='text', placeholder='输入内容···', id='inp', oob=False, **kwargs): 
    return Textarea(type='text', rows=1, placeholder=placeholder, id=id, cls='font-mono', **(dict(hx_swap_oob='true') if oob else {}), **kwargs)

:::

prev(Inp())

to_xml(Inp())

'<textarea type="text" rows="1" placeholder="输入内容···" id="inp" class="textarea font-mono" name="inp"></textarea>'

Button

prev(Button('提交', type='submit'))

Route Helper Functions

Helper functions for managing chat sessions and rendering responses. get_chat retrieves the active chat and shell for the current user’s dialog. sync_hist persists the conversation history to the database. mk_reply generates the UI response—a pair of bubbles (user input and model output) along with an OOB swap to clear the input field.

user_chats

{(1, 1): (<lisette.core.Chat at 0x70ee1a6efda0>,
  <IPython.terminal.interactiveshell.TerminalInteractiveShell at 0x70ee225f31a0>)}

fake_session = {'userid':1, 'current_dialog':1}

::: {#7eba2cc7 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:34:01.725230+00:00’}

def get_chat(session): return user_chats[(session['userid'], session['current_dialog'])]

:::

get_chat(fake_session)

(<lisette.core.Chat at 0x70ee1a6efda0>,
 <IPython.terminal.interactiveshell.TerminalInteractiveShell at 0x70ee225f31a0>)

::: {#649b261d .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:34:01.734433+00:00’}

def sync_hist(session, hist): dialogs.update(uid=session['userid'], dialog_num=session['current_dialog'], messages=json.dumps([h.model_dump() if hasattr(h, 'model_dump') else h for h in hist]))

:::

c = Chat('moonshot/kimi-k2-0711-preview', sp=sp)
c('你是谁？'); c.hist

[{'role': 'user', 'content': '你是谁？'},
 Message(content='我是Kimi，由月之暗面科技有限公司训练的大语言模型。很高兴为你提供帮助！', role='assistant', tool_calls=None, function_call=None, provider_specific_fields={'refusal': None})]

sync_hist(fake_session, c.hist)

json.loads(dialogs[(1, 1)].messages)

[{'role': 'user', 'content': '你是谁？'},
 {'content': '我是Kimi，由月之暗面科技有限公司训练的大语言模型。很高兴为你提供帮助！',
  'role': 'assistant',
  'tool_calls': None,
  'function_call': None,
  'provider_specific_fields': {'refusal': None}}]

::: {#c862e101 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:34:02.717276+00:00’}

def mk_reply(inp, out, color, name='inp', placeholder='输入内容···'): return Inp(oob=True, name=name, placeholder=placeholder, onkeydown="if(event.shiftKey && event.key==='Enter') { event.preventDefault(); this.form.requestSubmit(); }"), Div(cls='space-y-0.5')(
    Bubble(inp, color), 
    Bubble(out, color, 'r') if out else None)

:::

prev(mk_reply('呵呵', '哈哈', 'prompt'))

LLM Route

r = c('嗨'); r.choices[0].message.content

'嗨！有什么我可以帮你的吗？'

::: {#744b0ef0 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:34:03.555060+00:00’}

def add_hist(hist, type, content='', role='user'):
    hist.append(dict(role=role, content=content, type=type))

:::

I’ve added a type key to allow me to know in what style to render the message (blue for 'code', red for 'prompt', and green for 'note') when I load them.

add_hist(c.hist, 'note', content='这只是一张便条')
c.hist

[{'role': 'user', 'content': '你是谁？'},
 Message(content='我是Kimi，由月之暗面科技有限公司训练的大语言模型。很高兴为你提供帮助！', role='assistant', tool_calls=None, function_call=None, provider_specific_fields={'refusal': None}),
 {'role': 'user', 'content': '嗨'},
 Message(content='嗨！有什么我可以帮你的吗？', role='assistant', tool_calls=None, function_call=None, provider_specific_fields={'refusal': None}),
 {'role': 'user', 'content': '这只是一张便条', 'type': 'note'}]

::: {#e8fb36a9 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:34:03.565395+00:00’}

import json
@rt
def ask_llm(qry:str, session):
    c,sh = get_chat(session)
    res = c(qry)
    content = res.choices[0].message.content
    c.hist[-2]['type'] = 'prompt'
    c.hist[-1]['type'] = 'prompt'
    sync_hist(session, c.hist)
    return mk_reply(qry, content, 'prompt', 'qry', t('ask_model'))

:::

prev('/ask_llm?qry=你好')

ask_llm, alongside quering the model, stores the conversation with a type marker for rendering and syncs the history to the database.

::: {#a3bcde66 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:34:03.574612+00:00’}

def test(inp):
    return Div(Div(id='msgs'),
    Form(hx_post=ask_llm, hx_target='#msgs', hx_swap='beforeend', cls='flex items-center space-x-2')(inp, Button('提交', type='submit')))

:::

prev(test(Inp(id='qry')))

user_chats

{(1, 1): (<lisette.core.Chat at 0x70ee1a6efda0>,
  <IPython.terminal.interactiveshell.TerminalInteractiveShell at 0x70ee225f31a0>)}

Code Route

sh = get_shell(); sh

<IPython.terminal.interactiveshell.TerminalInteractiveShell at 0x70ee190c31d0>

??get_shell

def get_shell()->TerminalInteractiveShell:
    "Get a `TerminalInteractiveShell` with minimal functionality"
    sh = TerminalInteractiveShell()
    sh.logger.log_output = sh.history_manager.enabled = False
    dh = sh.displayhook
    dh.finish_displayhook = dh.write_output_prompt = dh.start_displayhook = lambda: None
    dh.write_format_data = lambda format_dict, md_dict=None: None
    sh.logstart = sh.automagic = sh.autoindent = False
    sh.autocall = 0
    sh.system = lambda cmd: None
    return sh

get_shell relies on IPython under the hood.

::: {#0f20abdc .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:34:23.268921+00:00’}

def ex(code, sh):
    res = sh.run_cell(code)
    return res.result if res.result else res.stdout

:::

::: {#3cd307bc .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:34:23.654002+00:00’}

@rt
def ex_code(code:str, session):
    c,sh = get_chat(session)
    add_hist(c.hist, 'code', f'[INTERPRETER INPUT]\n```py\n{code}\n```')
    res = ex(code, sh)
    add_hist(c.hist, 'code', f'[INTERPRETER OUTPUT]\n```py\n{res}\n```', role='assistant')
    sync_hist(session, c.hist)
    return mk_reply(inp=f'```py\n{code}\n```', out= f'```py\n{res}\n```', color='code', name='code', placeholder=t('your_code'))

:::

ex_code similarly also stores the conversation with a type marker and syncs with the database. A special '[INTERPRETER INPUT/OUTPUT]' marker is also added as indicators for the LLM.

prev(Div(
    Div(id='msgs'),
    Form(hx_post=ex_code, hx_target='#msgs', hx_swap='beforeend', cls='flex items-center space-x-2')(Inp(name='code'), Button('提交', type='submit'))
))

Note Route

::: {#a591691c .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:34:34.562110+00:00’}

@rt
def add_note(note:str, session):
    c,sh = get_chat(session)
    add_hist(c.hist, 'note', content=note,)
    sync_hist(session, c.hist)
    return mk_reply(note, '', 'note', 'note', t('any_thoughts'))

:::

def test(inp):
    return Div(Div(id='msgs'),
    Form(hx_post=add_note, hx_target='#msgs', hx_swap='beforeend', cls='flex items-center space-x-2')(inp, Button('提交', type='submit')))

prev(test(Inp(id='note')))

Editor Component

Three editor components for the different message types. Each editor is a form that posts to its corresponding route (ex_code, ask_llm, or add_note) and appends the result to the message area.

::: {#3da6f419 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:34:55.717291+00:00’}

@rt
def code_editor(session): return Form(hx_post=ex_code, hx_target='#msgs', hx_swap='beforeend', cls='flex items-stretch space-x-2')(Inp(placeholder=t('enter_code', session), name='code', onkeydown="if(event.shiftKey && event.key==='Enter') { event.preventDefault(); this.form.requestSubmit(); }"), Button(t('submit', session), type='submit', cls='self-stretch h-auto'))

:::

The editor routes need the session cookie for translations because each editor is loaded in a separate request. Since ContextVar resets with each new request, we need to re-set the language from the session at the start of each request.

prev(Div(Div(id='msgs'),Div(hx_get=code_editor, hx_trigger='load')))

::: {#e85cd35e .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:35:18.553389+00:00’}

@rt
def llm_editor(session): return Form(hx_post=ask_llm, hx_target='#msgs', hx_swap='beforeend', cls='flex items-center space-x-2')(Inp(placeholder=t('ask_model', session), name='qry', onkeydown="if(event.shiftKey && event.key==='Enter') { event.preventDefault(); this.form.requestSubmit(); }"), Button(t('submit', session), type='submit', cls='self-stretch h-auto'))

:::

prev(llm_editor)

::: {#3f0d14d8 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:35:37.352665+00:00’}

@rt
def note_editor(session): return Form(hx_post=add_note, hx_target='#msgs', hx_swap='beforeend', cls='flex items-center space-x-2')(Inp(placeholder=t('any_thoughts', session), name='note', onkeydown="if(event.shiftKey && event.key==='Enter') { event.preventDefault(); this.form.requestSubmit(); }"), Button(t('submit', session), type='submit', cls='self-stretch h-auto'))

:::

prev(note_editor)

![](../images/31_solveit_in_solveit/4.png)

/bin/bash: -c: line 1: syntax error near unexpected token `../images/31_solveit_in_solveit/4.png'
/bin/bash: -c: line 1: `[](../images/31_solveit_in_solveit/4.png)'

Tabs

A tabbed navigation bar that lets users switch between the three editor types—code, prompt, and note. Each tab triggers an HTMX request to load the corresponding editor form into the editor container.

::: {#e2b96dec .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:39:32.407645+00:00’}

def Tab(title, id, hx_post, **kwargs): return fc.Button(title, id=f'{id}-tab', type='button', role='tab', hx_post=hx_post, hx_target='#editor', hx_swap='innerHTML', **kwargs)

:::

prev(Nav(Tab('标签', id='test', hx_post=None), cls='w-full', role='tablist'))

::: {#355bf665 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:39:37.560871+00:00’}

def TabBar(cls='', *args, **kwargs): 
    return Nav(
        Tab(t('code'), id='code', hx_post=code_editor),
        Tab(t('prompt'), id='qry', hx_post=llm_editor),
        Tab(t('note'), id='note', hx_post=note_editor),
        cls=f'w-full {cls}', role='tablist', *args, **kwargs
    )

:::

prev(Div(TabBar(), Div(id='editor'), cls='tabs w-full'))

Render Existing Messages

::: {#764bf26a .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:39:42.067948+00:00’}

@rt
def load_messages(session):
    c,sh = get_chat(session)
    for i in c.hist:
        role = i.get('role') if isinstance(i, dict) else i.role
        if role=='user':      yield Div(Bubble(i['content'].replace('[INTERPRETER INPUT]\n', ''), i['type']), cls='space-y-0.5')
        if role=='assistant': yield Div(Bubble(i['content'].replace('[INTERPRETER OUTPUT]\n', ''), i['type'], 'r'), cls='space-y-0.5')

:::

prev(f'/load_messages')

Dialog

The dialog component ties everything together—a title bar that supports inline editing, a message display area that loads existing conversation history, and the tabbed editor for adding new messages. Clicking a dialog name makes it editable, and saving updates the database immediately.

::: {#64ce33d2 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:40:04.037914+00:00’}

@rt
def edit_dname(session):
    dname = dialogs[(session['userid'], session['current_dialog'])].name
    return Form(id='dname', hx_post=save_dname, hx_target='#dname', hx_swap='outerHTML')(
        Input(value=dname, name='newname', autofocus=True, onfocus='this.select()', required=True, cls='text-2xl font-bold outline-none'),
        Button(t('save', session), type='submit')
    )

:::

::: {#9d6ae097 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:40:04.608020+00:00’}

@rt 
def save_dname(session, newname:str):
    dialogs.update(uid=session['userid'], dialog_num=session['current_dialog'], name=newname)
    return DialogTitle(newname)

:::

::: {#dfb739b4 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:40:05.386759+00:00’}

def DialogTitle(name): return H1(name, cls='text-2xl font-bold', id='dname', hx_get=edit_dname, hx_target='#dname', hx_swap='outerHTML')

:::

prev(DialogTitle('对话'))

::: {#5f7ef951 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:40:10.636548+00:00’}

@rt
def dialog(dnum:int, session): 
    session['current_dialog'] = dnum
    return Div(
        DialogTitle(dialogs[(session['userid'], dnum)].name),
        Div(id='msgs', cls='p-8 max-h-96 overflow-y-auto', hx_get=load_messages, hx_trigger='load'),
        TabBar(id='tabbar'),
        Div(id='editor'),
        cls='tabs w-full'
    )

:::

prev('dialog?dnum=1')

Delete Dialog

::: {#493dcf03 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:40:29.905945+00:00’}

@rt
def delete_dialog(session, dnum:int):
    uid = session.get('userid')
    dialogs.delete((uid, dnum))
    del user_chats[(uid, dnum)]

:::

fake_session

{'userid': 1, 'current_dialog': 1}

users(), dialogs()

([User(id=1, restore_code='provocative-lori-of-acceptance')],
 [Dialog(id=None, uid=1, dialog_num=1, name='viridian-pegasus', messages='[{"role": "user", "content": "\\u4f60\\u597d", "type": "prompt"}, {"content": "\\u4f60\\u597d\\uff01\\u6709\\u4ec0\\u4e48\\u6211\\u53ef\\u4ee5\\u5e2e\\u60a8\\u7684\\u5417\\uff1f", "role": "assistant", "tool_calls": null, "function_call": null, "provider_specific_fields": {"refusal": null}, "type": "prompt"}, {"role": "user", "content": "\\u54c8\\u55bd", "type": "prompt"}, {"content": "\\u54c8\\u55bd\\uff01\\u5f88\\u9ad8\\u5174\\u53c8\\u89c1\\u5230\\u4f60\\uff5e  \\n\\u4eca\\u5929\\u60f3\\u804a\\u70b9\\u4ec0\\u4e48\\uff0c\\u6216\\u8005\\u6709\\u4ec0\\u4e48\\u9700\\u8981\\u6211\\u5e2e\\u5fd9\\u7684\\uff1f", "role": "assistant", "tool_calls": null, "function_call": null, "provider_specific_fields": {"refusal": null}, "type": "prompt"}, {"role": "user", "content": "[INTERPRETER INPUT]\\n```py\\n1+1\\n```", "type": "code"}, {"role": "assistant", "content": "[INTERPRETER OUTPUT]\\n```py\\n2\\n```", "type": "code"}, {"role": "user", "content": "\\u4e00\\u5f20\\u4fbf\\u8d34", "type": "note"}, {"role": "user", "content": "[INTERPRETER INPUT]\\n```py\\n5**2\\n```", "type": "code"}, {"role": "assistant", "content": "[INTERPRETER OUTPUT]\\n```py\\n25\\n```", "type": "code"}, {"role": "user", "content": "\\u4f60\\u597d", "type": "prompt"}, {"content": "\\u4f60\\u597d\\uff01\\u770b\\u8d77\\u6765\\u4f60\\u521a\\u521a\\u8bd5\\u4e86 1+1 \\u548c 5**2\\uff0c\\u7ed3\\u679c\\u5206\\u522b\\u662f 2 \\u548c 25\\u3002  \\n\\u201c\\u4e00\\u5f20\\u4fbf\\u8d34\\u201d\\u662f\\u60f3\\u8ba9\\u6211\\u5e2e\\u4f60\\u5199\\u70b9\\u4ec0\\u4e48\\u5417\\uff1f\\u8fd8\\u662f\\u6709\\u4ec0\\u4e48\\u522b\\u7684\\u9700\\u6c42\\uff1f", "role": "assistant", "tool_calls": null, "function_call": null, "provider_specific_fields": {"refusal": null}, "type": "prompt"}]', server_session=None)])

user_chats

{(1, 1): (<lisette.core.Chat at 0x70ee1a6efda0>,
  <IPython.terminal.interactiveshell.TerminalInteractiveShell at 0x70ee225f31a0>)}

delete_dialog(fake_session, 1)

users(), dialogs()

([User(id=1, restore_code='provocative-lori-of-acceptance')], [])

user_chats

{}

Restore Dialog

The restore session feature allows users to recover their data if the server restarts or they switch devices. Users enter their unique restore code (generated at account creation), which looks up their account in the database. If found, it restores their session by setting the user ID, loading their first dialog, and redirecting to the main page. A destructive alert displays if the restore code is invalid.

::: {#b7ebe597 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:40:39.407474+00:00’}

insolveit = 'IN_SOLVEIT' in os.environ

:::

insolveit

True

I’ve created the insolveit check to determine whether the app is running in my SolveIt instance or whether as a deployed app.

::: {#7c59ce58 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:40:41.808713+00:00’}

@rt
def restore_session(restore_code:str, session):
    if (u:=first(users('restore_code=?', [restore_code]))):
        session['userid'] = u.id
        existing = dialogs(where=f'uid={u.id}')
        if existing: session['current_dialog'] = existing[0].dialog_num
        setsession(session)
        return HtmxResponseHeaders(redirect='/main' if insolveit else '/')
    else: return Div(cls='alert-destructive')(H2(t('restore_failed')), Section(t('restore_error')))

    session['userid'] = first(users('restore_code=?', [restore_code])).id
    setsession(session)

:::

prev('/restore_session?restore_code=haha')

user_chats

{}

users[1].restore_code

'provocative-lori-of-acceptance'

user_chats = {}
prev(restore_session.to(restore_code=users[1].restore_code))

user_chats

{(1, 1): (<lisette.core.Chat at 0x70ee0bf336e0>,
  <IPython.terminal.interactiveshell.TerminalInteractiveShell at 0x70ee19188fb0>)}

prev(Div(
    Form(hx_post=restore_session, hx_target='#alert-area', cls='flex items-end gap-2')(
        Div()(
            Label('恢复码', fr='restore_code', cls='mb-2'),
            Input(placeholder='looks-like-this', type='text', name='restore_code')
        ),
        Button('提交', type='submit')
    ),
    Div(id='alert-area')
))

::: {#b997e64f .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:42:07.092837+00:00’}

def RestoreForm(session, id='restore-form', *args, **kwargs):
    return Form(hx_post=restore_session, hx_target='#alert-box', cls='flex items-end gap-2', id=id, *args, **kwargs)(
        Div(
            Label(t('restore_code'), ' · ', users[session['userid']].restore_code, fr='restore_code', cls='mb-2'),
            Input(type='text', name='restore_code', placeholder=t('enter_restore_code'))
        ),
        Button(t('submit'), type='submit')
    )

:::

prev(RestoreForm(fake_session))

Dialog List

The dialog list component displays all dialogs belonging to the current user. The display_dialogs route fetches all user dialogs from the database and includes a “new dialog” card that triggers dialog creation. The restore code form is also embedded here, allowing users to recover their session from a different device.

::: {#7fde0067 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:42:13.946257+00:00’}

@rt
def current_dialog(session): return dialog(session.get('current_dialog'), session)

:::

::: {#d1206a9d .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:42:14.672987+00:00’}

def DialogCard(d): return Div(id=f'dialog-{d.uid}{d.dialog_num}', cls='card p-4 hover:bg-gray-100 flex flex-row justify-between items-center')(
    Span(hx_get=dialog.to(dnum=d.dialog_num), hx_target='#content', cls='cursor-pointer')('💬 ', d.name), 
    Button(t('delete'), var='destructive', hx_post=delete_dialog.to(dnum=d.dialog_num), hx_swap='delete', hx_target=f'#dialog-{d.uid}{d.dialog_num}')
)

:::

prev(DialogCard(first(dialogs())))

::: {#4151e92e .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:42:23.354678+00:00’}

@rt
def display_dialogs(session):
    ds = L(dialogs('uid=?', [session.get('userid')]))
    ds = ds.map(lambda d: DialogCard(d))
    return Div(cls='flex flex-col gap-2 max-w-xl')(
        Div(cls='flex items-center justify-between gap-4')(H1(t('all_dialogs', session), cls='text-2xl font-bold', id='all-dialogs-title'), RestoreForm(session)), 
        Div(id='alert-box'),
        Div('➕', Span(t('new_dialog', session), id='new-dialog-text'), cls='card p-4 hover:bg-gray-100 cursor-pointer border-2 border-dashed flex flex-row items-center gap-2', hx_post=create_dialog, hx_target='#content'),
        Div(*ds, cls='flex flex-col gap-2 max-h-64 overflow-y-auto')
    )

:::

prev(Div(
    Div(id='content')(
        display_dialogs({'userid': 1, 'current_dialog': 11})
    )
))

Main Page

::: {#886bc7d5 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:42:30.692188+00:00’}

@rt
def toggle_lang(session):
    lang = 'zh' if session.get('lang') == 'en' else 'en'
    current_lang.set(lang)
    session['lang'] = lang
    return (
        H2(t('solveit_lite'), id='title', hx_swap_oob='true'),
        Button(t('all_dialogs'), id='dialogs-btn', hx_post=display_dialogs, hx_target='#content', var='secondary', hx_swap_oob='true'),
        Span(t('code'), hx_swap_oob='innerHTML:#code-tab'),
        Span(t('prompt'), hx_swap_oob='innerHTML:#qry-tab'),
        Span(t('note'), hx_swap_oob='innerHTML:#note-tab'),
        Div(id='editor', hx_swap_oob='true'),
        Footer(t('footer'), id='footer', cls='text-sm text-gray-400', hx_swap_oob='true'),
        RestoreForm(session, hx_swap_oob='true', id='restore-form'),
        H1(t('all_dialogs'), cls='text-2xl font-bold', id='all-dialogs-title', hx_swap_oob='true'),
        Span(t('new_dialog'), id='new-dialog-text', hx_swap_oob='true')
    )

:::

This route allows users to switch between Chinese and English with OOB swaps to update all translatable text across the page simultaneously.

::: {#e98d1f46 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’ time_run=‘2026-01-22T00:42:32.887451+00:00’}

@rt('/main' if insolveit else '/')
def main(session):
    setsession(session)
    return Div(id='body', cls='card p-8 max-w-4xl mx-auto my-4')(
        Header(cls='flex justify-between items-center')(H2(
            t('solveit_lite'), id='title'), 
            Span(
                Button('🌐', var='secondary', hx_post=toggle_lang, hx_swap='none'), 
                Button(t('all_dialogs'), id='dialogs-btn', hx_post=display_dialogs, hx_target='#content', var='secondary')
            )
        ),
        Section(id='content')(
            current_dialog(session)
        ),
        Footer(t('footer'), id='footer', cls='text-sm text-gray-400')
    )

:::

prev('/main')

Deploy

::: {#b05651d9 .cell 0=‘e’ 1=‘x’ 2=‘p’ 3=‘o’ 4=‘r’ 5=‘t’}

if not insolveit: serve()

:::

Path('requirements.txt').write_text('''\
python-fasthtml
lisette
coolname
ipython
''')
!cat requirements.txt

python-fasthtml
lisette
coolname
ipython

Path('.plash').write_text('export PLASH_APP_NAME=solveit-lite-unofficial')
!cat .plash

export PLASH_APP_NAME=solveit-lite-unofficial

Path('plash.env').write_text(f'export MOONSHOT_API_KEY={os.environ.get('MOONSHOT_API_KEY')}')

75

import plash_cli as pc
pc.deploy()

'https://solveit-lite-unofficial.pla.sh'

print(pc.logs())

Build Start Time: 2026-01-16 01:30:30.917195+00:00
Step 1/14 : FROM plash:latest
 ---> 8c65e51fa2d2
Step 2/14 : ARG USER_UID=1000
 ---> Using cache
 ---> 781580c8e262
Step 3/14 : ARG USER_GID=1000
 ---> Using cache
 ---> dcf3956b8c50
Step 4/14 : RUN groupadd -g $USER_GID plash && useradd -u $USER_UID -g $USER_GID -m plash
 ---> Using cache
 ---> fe019dc49177
Step 5/14 : ENV PLASH_PRODUCTION=1
 ---> Using cache
 ---> a86e28497fc0
Step 6/14 : COPY .dockerignore setup.sh* ./
 ---> Using cache
 ---> 786f3c3d2c09
Step 7/14 : RUN if [ -f ./setup.sh ]; then echo "Found setup.sh, executing..." && chmod +x ./setup.sh && ./setup.sh; fi
 ---> Using cache
 ---> 3f80b2a299d3
Step 8/14 : COPY .dockerignore requirements.txt* ./
 ---> Using cache
 ---> dea62d9dd70b
Step 9/14 : RUN if [ -f ./requirements.txt ]; then echo "Found requirements.txt, executing..." && uv pip install -r ./requirements.txt --system; fi
 ---> Using cache
 ---> 241057bc8973
Step 10/14 : RUN chown -R plash:plash /app
 ---> Using cache
 ---> eb5b444c975f
Step 11/14 : USER plash
 ---> Using cache
 ---> d2eda950b2e1
Step 12/14 : ENV PATH="/home/plash/.local/bin:$PATH"
 ---> Using cache
 ---> 5631843ecbf7
Step 13/14 : EXPOSE 5001
 ---> Using cache
 ---> b1e69db470b1
Step 14/14 : ENTRYPOINT ["bash", "-c", "if [ -f ./plash.env ]; then . ./plash.env; fi && python main.py"]
 ---> Using cache
 ---> 5ad4e58136f3
Successfully built 5ad4e58136f3
Successfully tagged 9d102cbb-e493-4ce7-9110-09dbf5c01851:latest

Build End Time: 2026-01-16 01:30:31.194696+00:00

print(pc.logs(mode='app'))

INFO:     Will watch for changes in these directories: ['/app']
INFO:     Uvicorn running on http://0.0.0.0:5001 (Press CTRL+C to quit)
INFO:     Started reloader process [1] using WatchFiles
Link: http://localhost:5001
INFO:     Started server process [14]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

import httpx
print(httpx.get('https://solveit-lite-unofficial.pla.sh').text)

 <!doctype html>
 <html>
   <head>
     <title>FastHTML page</title>
     <link rel="canonical" href="https://solveit-lite-unofficial.pla.sh/">
     <meta charset="utf-8">
     <meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
<script src="https://cdn.jsdelivr.net/npm/htmx.org@2.0.7/dist/htmx.js"></script><script src="https://cdn.jsdelivr.net/gh/answerdotai/fasthtml-js@1.0.12/fasthtml.js"></script><script src="https://cdn.jsdelivr.net/gh/answerdotai/surreal@main/surreal.js"></script><script src="https://cdn.jsdelivr.net/gh/gnat/css-scope-inline@main/script.js"></script><script src="https://cdn.jsdelivr.net/npm/@tailwindcss/browser@4"></script>     <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/basecoat-css@0.3.6/dist/basecoat.cdn.min.css">
<script src="https://cdn.jsdelivr.net/npm/basecoat-css@0.3.6/dist/js/all.min.js" defer></script><script type="module">import { marked } from "https://cdn.jsdelivr.net/npm/marked/lib/marked.esm.js";
proc_htmx('.marked', e => e.innerHTML = marked.parse(e.textContent));</script>     <link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/highlightjs/cdn-release/build/styles/atom-one-dark.css" media="(prefers-color-scheme: dark)">
     <link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/highlightjs/cdn-release/build/styles/atom-one-light.css" media="(prefers-color-scheme: light)">
<script src="https://cdn.jsdelivr.net/gh/highlightjs/cdn-release/build/highlight.min.js"></script><script src="https://cdn.jsdelivr.net/gh/arronhunt/highlightjs-copy/dist/highlightjs-copy.min.js"></script>     <link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/arronhunt/highlightjs-copy/dist/highlightjs-copy.min.css">
<script src="https://cdn.jsdelivr.net/gh/highlightjs/cdn-release/build/languages/python.min.js"></script><script type="module">
hljs.addPlugin(new CopyButtonPlugin());
hljs.configure({'cssSelector': 'pre code:not([data-highlighted="yes"])'});
htmx.onLoad(hljs.highlightAll);</script>   </head>
   <body>
     <div id="body" class="card p-8 max-w-4xl mx-auto my-4">
       <header class="flex justify-between items-center">
         <h2 id="title">solveit-lite</h2>
<span><button hx-post="/toggle_lang" hx-swap="none" class="btn-secondary ">🌐</button><button hx-post="/display_dialogs" hx-target="#content" id="dialogs-btn" class="btn-secondary " name="dialogs-btn">All Dialogs</button></span>       </header>
       <section id="content">
         <div class="tabs w-full">
           <h1 class="text-2xl font-bold" id="dname" hx-get="/edit_dname" hx-target="#dname" hx-swap="outerHTML">intrepid-turaco</h1>
           <div id="msgs" class="p-8 max-h-96 overflow-y-auto" hx-get="/load_messages" hx-trigger="load"></div>
           <nav class="w-full " role="tablist" id="tabbar">
<button type="button" role="tab" hx-post="/code_editor" hx-swap="innerHTML" hx-target="#editor" id="code-tab" name="code-tab">Code</button><button type="button" role="tab" hx-post="/llm_editor" hx-swap="innerHTML" hx-target="#editor" id="qry-tab" name="qry-tab">Prompt</button><button type="button" role="tab" hx-post="/note_editor" hx-swap="innerHTML" hx-target="#editor" id="note-tab" name="note-tab">Note</button>           </nav>
           <div id="editor"></div>
         </div>
       </section>
       <footer id="footer" class="text-sm text-gray-400">Crafted with FastHTML</footer>
     </div>
   </body>
 </html>

If you have any comments, questions, suggestions, feedback, criticisms, or corrections, please do post them down in the comment section below!

Introduction

In this notebook I’ll be demonstrating how one can update messages in SolveIt dialogues and do further work with them. At the same time also demonstrating how SolveIt dialogue messages can be translated with a LLM.

I’m currently teaching myself Chinese, and I’ve found to learn a language, one has to use the language. So I make the LLM respond in Chinese.

I’ll be using lisette for working with the LLM, DeepSeek as the model, and dialoghelper to process the dialogue messages.

Sample Dialogue

I’ll create a rough, short dialogue that I can work upon.

You’ll see that the English translations are already appended. See the SolveIt dialogue here.

Hey there, who are you?

Translation

from dialoghelper import *

?find_msgs

Signature:
find_msgs(
    re_pattern: str = '',
    msg_type: str = None,
    limit: int = None,
    include_output: bool = True,
    dname: str = '',
)
Docstring: Find `list[dict]` of messages in current specific dialog that contain the given information. To refer to a message found later, use its `id` field.
File:      /usr/local/lib/python3.12/site-packages/dialoghelper/core.py
Type:      function

We can use find_msgs to filter out those message types that are prompts.

o = find_msgs(msg_type='prompt'); o[0]

{'id': '_4dcda3fa',
 'input_tokens': 10,
 'output_tokens': 94,
 'time_run': '',
 'is_exported': False,
 'skipped': False,
 'hidden': False,
 'bookmark': None,
 'i_collapsed': False,
 'o_collapsed': False,
 'heading_collapsed': False,
 'i_clamp': False,
 'o_clamp': False,
 'pinned': False,
 'use_thinking': True,
 'attachments': [],
 'oob': None,
 'run': None,
 'scroll': False,
 'order': 3,
 'content': 'Hey there, who are you?',
 'output': '🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠\n\n你好！我是 solveit，一个学习助手。我可以帮你学习各种各样的东西——数学、编程、写作，或者别的什么。\n\n你想学什么呢？你对什么感兴趣？告诉我一点儿关于你自己的情况吧，这样我可以更好地帮助你。',
 'msg_type': 'prompt'}

o[0]['output']

'🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠\n\n你好！我是 solveit，一个学习助手。我可以帮你学习各种各样的东西——数学、编程、写作，或者别的什么。\n\n你想学什么呢？你对什么感兴趣？告诉我一点儿关于你自己的情况吧，这样我可以更好地帮助你。'

And then when we are iterating through all the prompt messages we can fetch the respective message ID and update it using update_msg.

?update_msg

Signature:
update_msg(
    msgid: str = None,
    msg: Optional[Dict] = None,
    dname: str = '',
    *,
    content: str | None = None,
    msg_type: str | None = None,
    output: str | None = None,
    time_run: str | None = None,
    is_exported: int | None = None,
    skipped: int | None = None,
    i_collapsed: int | None = None,
    o_collapsed: int | None = None,
    heading_collapsed: int | None = None,
    pinned: int | None = None,
)
Docstring:
Update an existing message. Provide either `msg` OR field key/values to update.
- Use `content` param to update contents.
- Only include parameters to update--missing ones will be left unchanged.
File:      /usr/local/lib/python3.12/site-packages/dialoghelper/core.py
Type:      function

I’ll be formatting the new updated prompt output as original + '\n\n---\n\n' + translated.

In a nutshell, the core loop is:

1. Fetch a prompt message.
2. Access the output content of that message.
3. Translate.
4. Reformat output content
5. Update the respective message.

I’ll begin with a single example.

from lisette.core import *
sys_prmpt = "You'll receive a series of messages that are responses in a dialogue. I need you to aptly translate those responses to English, whilst maintaining the same tone/style and minimal loss of information. If you encoutner any brain emojies, skip them. Do not re-output the original message."
chat = Chat('deepseek/deepseek-chat', hist=[sys_prmpt]); chat

<lisette.core.Chat at 0x7da9f8871160>

res = chat(o[0]['output'])
display(res)

display(chat(o[1]['output']))

So that’s how the translations look. Let’s now update the messages themselves.

??update_msg

@delegates(_umsg)
def update_msg(
    msgid:str=None, # id of message to update (if None, uses current message)
    msg:Optional[Dict]=None, # Dictionary of field keys/values to update
    dname:str='', # Running dialog to get info for; defaults to current dialog
    **kwargs):
    """Update an existing message. Provide either `msg` OR field key/values to update.
    - Use `content` param to update contents.
    - Only include parameters to update--missing ones will be left unchanged."""
    if not msgid and not msg: raise TypeError("update_msg needs either a dict message or `msgid=`")
    res = call_endp('update_msg_', dname, msgid=msgid, **kwargs)
    set_var('__msg_id', res)
    return res

trans = res.choices[0].message.content; trans

"Hello! I'm solveit, a learning assistant. I can help you learn all sorts of things—math, programming, writing, or anything else.\n\nWhat would you like to learn? What are you interested in? Tell me a little about yourself so I can help you better."

new_out = o[0]['output']+'\n\n---\n\n'+trans; new_out

"🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠\n\n你好！我是 solveit，一个学习助手。我可以帮你学习各种各样的东西——数学、编程、写作，或者别的什么。\n\n你想学什么呢？你对什么感兴趣？告诉我一点儿关于你自己的情况吧，这样我可以更好地帮助你。\n\n---\n\nHello! I'm solveit, a learning assistant. I can help you learn all sorts of things—math, programming, writing, or anything else.\n\nWhat would you like to learn? What are you interested in? Tell me a little about yourself so I can help you better."

update_msg(msgid=o[0]['id'], output=new_out)

'_4dcda3fa'

o[0]

{'id': '_4dcda3fa',
 'input_tokens': 10,
 'output_tokens': 94,
 'time_run': '',
 'is_exported': False,
 'skipped': False,
 'hidden': False,
 'bookmark': None,
 'i_collapsed': False,
 'o_collapsed': False,
 'heading_collapsed': False,
 'i_clamp': False,
 'o_clamp': False,
 'pinned': False,
 'use_thinking': True,
 'attachments': [],
 'oob': None,
 'run': None,
 'scroll': False,
 'order': 3,
 'content': 'Hey there, who are you?',
 'output': '🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠\n\n你好！我是 solveit，一个学习助手。我可以帮你学习各种各样的东西——数学、编程、写作，或者别的什么。\n\n你想学什么呢？你对什么感兴趣？告诉我一点儿关于你自己的情况吧，这样我可以更好地帮助你。',
 'msg_type': 'prompt'}

The message has been updated! Click here to jump to it and see the update.

Now we can package everything up into a loop.

chat = Chat('deepseek/deepseek-chat', hist=[sys_prmpt])
outs = find_msgs(msg_type='prompt')
for o in outs:
    r = chat(o['output'])
    trans = r.choices[0].message.content
    new_o = o['output']+'\n\n---\n\n'+trans
    update_msg(msgid=o['id'], output=new_o)

Some areas for improvement: - [ ] The reformatted output includes collapsable/folding blocks - [ ] Figure out how to use fastprogress view translation progress.

Conclusion

And it’s as simple as that!

If you have any comments, questions, suggestions, feedback, criticisms, or corrections, please do post them down in the comment section below!

The Requirement

The bar is low. It looks complicated, but it’s not. This is the reality to get started making your own AI models:

Math: None. Really.

Data: You can go miles with only 50 items.

Computing power: You can get state of the art computing for free.

Experience: If you can write basic code and understand what a function is, you’re ready.

Just kidding–these aren’t the real requirements. The real requirement is being okay with not understanding everything immediately. You’ll learn by doing first, understanding later.

When you play soccer, do you sit in a physics classroom or starting kicking a ball?

Let’s Kick the Ball

To get started:

• Head to Kaggle, sign up, and verify your phone number (that’s how you get access to fancy GPUs)
• Create a notebook
• Then in the first cell, run !uv pip install ddg–this is just setting up the equipment in the soccer pitch

In 2015, this XKCD comic made a joke: distinguishing birds from non-birds in photos was “nearly impossible” for computers. Everyone laughed because it was obvioulsy true.

Let’s build that exact system, in minutes. If you think it’ll be hard, it’ll be hard. If you think it’ll be easy, it’ll be easy.

We’ll need to:

1. Get some data
2. Process the data
3. Fine-tune an AI model
4. Use the model

! uv pip install ddgs

Using Python 3.11.13 environment at: /usr

Resolved 17 packages in 370ms                                        

Prepared 4 packages in 229ms                                             

Uninstalled 1 package in 21ms

Installed 4 packages in 11ms                                

 + ddgs==9.8.0

 - lxml==5.4.0

 + lxml==6.0.2

 + primp==0.15.0

 + socksio==1.0.0

import logging
logging.disable(logging.WARNING)

from ddgs import DDGS
from fastcore.all import *

def search_images(keywords, max_images=200):
    return L(DDGS().images(keywords, max_results=max_images)).itemgot('image')

urls = search_images('bird photos', max_images=1)

from fastdownload import download_url
from fastai.vision.all import *

download_url(urls[0], 'bird.jpg', show_progress=False)
im = Image.open('bird.jpg')
im.to_thumb(256,256)

/usr/local/lib/python3.11/dist-packages/pydantic/_internal/_generate_schema.py:2249: UnsupportedFieldAttributeWarning: The 'repr' attribute with value False was provided to the `Field()` function, which has no effect in the context it was used. 'repr' is field-specific metadata, and can only be attached to a model field using `Annotated` metadata or by assignment. This may have happened because an `Annotated` type alias using the `type` statement was used, or if the `Field()` function was attached to a single member of a union type.
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/pydantic/_internal/_generate_schema.py:2249: UnsupportedFieldAttributeWarning: The 'frozen' attribute with value True was provided to the `Field()` function, which has no effect in the context it was used. 'frozen' is field-specific metadata, and can only be attached to a model field using `Annotated` metadata or by assignment. This may have happened because an `Annotated` type alias using the `type` statement was used, or if the `Field()` function was attached to a single member of a union type.
  warnings.warn(

searches = 'forest', 'bird'
path = Path('bird_or_not')

for o in searches:
    dest = (path/o)
    dest.mkdir(exist_ok=True, parents=True)
    download_images(dest, urls=search_images(f'{o} photo'))
    resize_images(path/o, max_size=400, dest=path/o)

dls = DataBlock(
    blocks=(ImageBlock, CategoryBlock), 
    get_items=get_image_files, 
    splitter=RandomSplitter(valid_pct=0.2, seed=42),
    get_y=parent_label,
    item_tfms=[Resize(192, method='squish')]
).dataloaders(path, bs=32)

dls.show_batch(max_n=6)

learn = vision_learner(dls, resnet18, metrics=error_rate)
learn.fine_tune(3)

Downloading: "https://download.pytorch.org/models/resnet18-f37072fd.pth" to /root/.cache/torch/hub/checkpoints/resnet18-f37072fd.pth
100%|██████████| 44.7M/44.7M [00:00<00:00, 149MB/s] 

im.to_thumb(256,256)

is_bird,_,probs = learn.predict(PILImage.create('bird.jpg'))
print(f"This is a: {is_bird}.")
print(f"Probability it's a bird: {probs[0]:.4f}")

This is a: bird.
Probability it's a bird: 1.0000

What Just Happened?

Datasets and splits

We used DuckDuckGo to find image URLs, then downloaded them. You created your very own dataset.

searches = 'forest','bird'
path = Path('bird_or_not')

for o in searches:
    dest = (path/o)
    dest.mkdir(exist_ok=True, parents=True)
    download_images(dest, urls=search_images(f'{o} photo'))
    resize_images(path/o, max_size=400, dest=path/o)

Machine learning models need two things: data to learn from (training set) and data to test accuracy (validation set). We split our 400 images 80/20:

splitter=RandomSplitter(valid_pct=0.2, seed=42)

The model never sees validation images during training. This tells us if it actually learned or just memorized.

DataBlocks and DataLoaders

A DataBlock is a block of data. It says “here’s how to handle my data”:

DataBlock(
    blocks=(ImageBlock, CategoryBlock),  # what type of data
    get_items=get_image_files,           # how to find it
    splitter=RandomSplitter(...),        # how to split it
    get_y=parent_label,                  # how to label it
    item_tfms=[Resize(192, ...)]         # how to transform it
)

Think of it like a crate with items inside. The crate has a sticker informing us how to handle the items within.

Think of .dataloaders like the dude unpacking the crate.

DataBlock(
    ...
).dataloaders(path, bs=32)

bs, short for batch size, tells the dude how many items to unpack from the crate and put at the shelf at any given moment.

More technically, DataLoaders turns this block into actual iterators that feed batches of data to your model during training.

Pre-trained models

You may have noticed that our vision_learner was passed resnet18. This is an already existing model, already trained on ImageNet—1 million images across 1000 categories. It already knows edges, textures, shapes, even some animals.

learn = vision_learner(dls, resnet18, metrics=error_rate)

When you create a learner, fastai downloads those pre-trained weights automatically.

Fine-tuning

You’re not training from scratch. You’re adjusting an expert:

learn.fine_tune(3)

Models have something called weights. They guide the model to the correct answer. For our case, that’s bird or forest. In three loops, the model tweaks the weights so it getes better at determining what’s a bird and what’s a forest.

The model already knows “bird-like features.” You’re just teaching it: “these specific birds, not those specific forests.”

That’s why it can take 30 seconds instead of 30 hours.

Predictions

is_bird, _, probs = learn.predict(PILImage.create('bird.jpg'))

Returns three things: the predicted label, the label index, and probabilities for each category.

The process, start to finish:

1. Collect data – Download images, organize into folders by category
2. Process Data – Tell fastai: input type, output type, how to find files, how to split, how to label
3. Load Data – Package the data into batches the model can consume
4. Create learner – Combine your data with a pre-trained model (like resnet18)
5. Fine-tune – Adjust the pre-trained weights to your specific task
6. Predict – Use the trained model on new images

This pattern works for almost any image classification problem. Different images, same steps.

Beyond Birds

The same six-step pattern works for completely different problems.

Segmentation – Label every pixel in an image by what it represents. Point at a road scene and the model colors cars brown, roads gray, buildings red, sky blue. Same process: organize pixel-labeled images, create DataLoaders, use a segmentation learner, fine-tune.

Tabular data – Predict outcomes from spreadsheets. Will this customer churn? What salary range fits these demographics? Organize your CSV, tell fastai which columns are categories vs numbers, create a tabular learner, fit. No pre-trained models here—every table is unique.

Collaborative filtering – Recommend products based on what similar users liked. “Users who rated movie X highly also liked movie Y.” Organize user-item-rating data, create collaborative filtering DataLoaders, fine-tune. This powers Netflix, Spotify, Amazon.

Text classification – Detect sentiment, filter spam, categorize documents. Organize labeled text, create text DataLoaders, use a text classifier (pre-trained on Wikipedia), fine-tune to your specific task.

Different domains, same structure. The pattern is portable.

For detailed examples, see the fastai docs.

Your Turn

You now know enough to build your own classifier. Pick something you care about:

• Healthy vs diseased plants
• Your handwriting vs someone else’s
• Cats vs dogs (classic, but effective)
• Different types of clouds
• Your face vs not your face

The steps are the same:

1. Search and download 50-200 images per category
2. Create a DataBlock
3. Make DataLoaders and check with show_batch()
4. Create a vision learner with resnet18
5. Fine-tune for 3-5 epochs
6. Test predictions

Start simple. Two categories. Small dataset. Get something working in the next hour.

What to do next:

• Read Chapter 1 of the fastai book - same material, different angle
• Try the official fastai course for video walkthroughs
• Share your project on the fastai forums - the community is helpful and encouraging

Don’t wait until you understand everything. Build first, understand later. That’s how you actually learn.

If you have any comments, questions, suggestions, feedback, criticisms, or corrections, please do post them down in the comment section below!

What are Agents?

“An AI agent is just an LLM in a while loop.”

That’s it. The LLM requests to run a function (or in other words, the tool), you run it, give back the result, and repeat until the LLM no longer requests. No fancy frameworks needed.

flowchart TD
    Start([Start]) --> Call[Call LLM with message + tools]
    Call --> Check{Tool requested?}
    Check -->|Yes| Execute[Execute tool function]
    Execute --> Add[Add result to history]
    Add --> Call
    Check -->|No| Done([Done])

def multer(a:float,b:float)->float:
    'Take the product of two numbers'
    return a*b
multer(2,3)

6

def pow(a:float,b:float)->float:
    'Return a to the power of b'
    return a**b
pow(2,3)

8

tools = [multer,pow]; tools

[<function __main__.multer(a: float, b: float) -> float>,
 <function __main__.pow(a: float, b: float) -> float>]

from lisette import *

schemas = [lite_mk_func(t) for t in tools]; schemas

[{'type': 'function',
  'function': {'name': 'multer',
   'description': 'Take the product of two numbers\n\nReturns:\n- type: number',
   'parameters': {'type': 'object',
    'properties': {'a': {'type': 'number', 'description': ''},
     'b': {'type': 'number', 'description': ''}},
    'required': ['a', 'b']}}},
 {'type': 'function',
  'function': {'name': 'pow',
   'description': 'Return a to the power of b\n\nReturns:\n- type: number',
   'parameters': {'type': 'object',
    'properties': {'a': {'type': 'number', 'description': ''},
     'b': {'type': 'number', 'description': ''}},
    'required': ['a', 'b']}}}]

??lite_mk_func

def lite_mk_func(f):
    if isinstance(f, dict): return f
    return {'type':'function', 'function':get_schema(f, pname='parameters')}

from toolslm.funccall import mk_ns

ns = mk_ns(tools); ns

{'multer': <function __main__.multer(a: float, b: float) -> float>,
 'pow': <function __main__.pow(a: float, b: float) -> float>}

??mk_ns

def mk_ns(fs):
    if isinstance(fs, abc.Mapping): return fs
    merged = {}
    for o in listify(fs):
        if isinstance(o, dict): merged |= o
        elif callable(o) and hasattr(o, '__name__'): merged |= {o.__name__: o}
    return merged

from fastcore.all import *
os.environ['MOONSHOT_API_BASE'] = 'https://api.moonshot.cn/v1'

from litellm import completion
hist = [{'role': 'user', 'content': 'Calculate 2*2353**5*1/2'}]
r = completion(model='moonshot/kimi-k2-0711-preview', messages=hist, tools=schemas); r

print(r)

ModelResponse(id='chatcmpl-6923a5f5ce7148a79dfc95ba', created=1763943925, model='moonshot/kimi-k2-0711-preview', object='chat.completion', system_fingerprint=None, choices=[Choices(finish_reason='tool_calls', index=0, message=Message(content="I need to calculate 2 * 2353^5 * 1/2.\n\nLet me break this down:\n- First, I'll calculate 2353^5\n- Then multiply by 2\n- Then multiply by 1/2 (which is the same as dividing by 2)\n\nLet me start with 2353^5:", role='assistant', tool_calls=[ChatCompletionMessageToolCall(index=0, function=Function(arguments='{"a": 2353, "b": 5}', name='pow'), id='pow:0', type='function')], function_call=None, provider_specific_fields={'refusal': None}), provider_specific_fields={})], usage=Usage(completion_tokens=93, prompt_tokens=111, total_tokens=204, completion_tokens_details=None, prompt_tokens_details=None), service_tier=None)

r.choices[0].finish_reason

'tool_calls'

tc = r.choices[0].message.tool_calls[0].to_dict(); tc

{'index': 0,
 'function': {'arguments': '{"a": 2353, "b": 5}', 'name': 'pow'},
 'id': 'pow:0',
 'type': 'function'}

import json
res = ns[tc['function']['name']](**json.loads(tc['function']['arguments'])); res

72128954282026993

hist.append(r.choices[0].message)
hist.append(mk_tc_result(tc, str(res))); hist

[{'role': 'user', 'content': 'Calculate 2*2353**5*1/2'},
 Message(content="I need to calculate 2 * 2353^5 * 1/2.\n\nLet me break this down:\n- First, I'll calculate 2353^5\n- Then multiply by 2\n- Then multiply by 1/2 (which is the same as dividing by 2)\n\nLet me start with 2353^5:", role='assistant', tool_calls=[ChatCompletionMessageToolCall(index=0, function=Function(arguments='{"a": 2353, "b": 5}', name='pow'), id='pow:0', type='function')], function_call=None, provider_specific_fields={'refusal': None}),
 {'tool_call_id': 'pow:0',
  'role': 'tool',
  'name': 'pow',
  'content': '72128954282026993'}]

while r.choices[0].finish_reason!='stop':
    mtcd = r.choices[0].message.tool_calls[0].to_dict()
    o = ns[mtcd['function']['name']](**json.loads(mtcd['function']['arguments']))
    tc_res = mk_tc_result(mtcd, str(o))
    hist.append(r.choices[0].message)
    hist.append(tc_res)
    r = completion(model='moonshot/kimi-k2-0711-preview', messages=hist, tools=schemas)

r.choices[0].finish_reason

'stop'

from pprint import pprint as pp
pp(hist)

[{'content': 'Calculate 2*2353**5*1/2', 'role': 'user'},
 Message(content="I need to calculate 2 * 2353^5 * 1/2.\n\nLet me break this down:\n- First, I'll calculate 2353^5\n- Then multiply by 2\n- Then multiply by 1/2 (which is the same as dividing by 2)\n\nLet me start with 2353^5:", role='assistant', tool_calls=[{'index': 0, 'function': {'arguments': '{"a": 2353, "b": 5}', 'name': 'pow'}, 'id': 'pow:0', 'type': 'function'}], function_call=None, provider_specific_fields={'refusal': None}),
 {'content': '72128954282026993',
  'name': 'pow',
  'role': 'tool',
  'tool_call_id': 'pow:0'},
 Message(content="I need to calculate 2 * 2353^5 * 1/2.\n\nLet me break this down:\n- First, I'll calculate 2353^5\n- Then multiply by 2\n- Then multiply by 1/2 (which is the same as dividing by 2)\n\nLet me start with 2353^5:", role='assistant', tool_calls=[{'index': 0, 'function': {'arguments': '{"a": 2353, "b": 5}', 'name': 'pow'}, 'id': 'pow:0', 'type': 'function'}], function_call=None, provider_specific_fields={'refusal': None}),
 {'content': '72128954282026993',
  'name': 'pow',
  'role': 'tool',
  'tool_call_id': 'pow:0'}]

def set_tools(tools):
    tools = listify(tools)
    schemas = [lite_mk_func(t) for t in tools]
    ns = mk_ns(tools)
    return tools, schemas, ns

def make_query(m,h,s):
    return completion(model=m, messages=h, tools=s)

def tc2dict(r):
    return r.choices[0].message.tool_calls[0].to_dict()

def run_tool(tc,ns):
    return ns[tc['function']['name']](**json.loads(tc['function']['arguments']))

def build_hist(h,r,tc_res):
    h.append(r.choices[0].message)
    h.append(mk_tc_result(r.choices[0].message.tool_calls[0],str(tc_res)))

hist = [{'role':'user', 'content':'Determine 2*3**4**5'}]
tools, schemas, ns = set_tools([multer,pow])

r = make_query('moonshot/kimi-k2-0711-preview',hist,schemas); r

while r.choices[0].finish_reason != 'stop':
    tc = tc2dict(r)
    tc_res = run_tool(tc, ns)
    build_hist(hist,r,tc_res)
    r = make_query('moonshot/kimi-k2-0711-preview', hist, schemas)

hist

[{'role': 'user', 'content': 'Determine 2*3**4**5'},
 Message(content="I'll solve this step by step, following the order of operations for exponentiation (which is right-associative). The expression 2*3**4**5 means 2×(3^(4^5)).", role='assistant', tool_calls=[{'index': 0, 'function': {'arguments': '{"a": 4, "b": 5}', 'name': 'pow'}, 'id': 'pow:0', 'type': 'function'}], function_call=None, provider_specific_fields={'refusal': None}),
 {'tool_call_id': 'pow:0', 'role': 'tool', 'name': 'pow', 'content': '1024'},
 Message(content="Now I'll calculate 3^1024:", role='assistant', tool_calls=[{'index': 0, 'function': {'arguments': '{"a": 3, "b": 1024}', 'name': 'pow'}, 'id': 'pow:1', 'type': 'function'}], function_call=None, provider_specific_fields={'refusal': None}),
 {'tool_call_id': 'pow:1',
  'role': 'tool',
  'name': 'pow',
  'content': '373391848741020043532959754184866588225409776783734007750636931722079040617265251229993688938803977220468765065431475158108727054592160858581351336982809187314191748594262580938807019951956404285571818041046681288797402925517668012340617298396574731619152386723046235125934896058590588284654793540505936202376547807442730582144527058988756251452817793413352141920744623027518729185432862375737063985485319476416926263819972887006907013899256524297198527698749274196276811060702333710356481'},
 Message(content="Finally, I'll multiply this by 2:", role='assistant', tool_calls=[{'index': 0, 'function': {'arguments': '{"a": 2, "b": 373391848741020043532959754184866588225409776783734007750636931722079040617265251229993688938803977220468765065431475158108727054592160858581351336982809187314191748594262580938807019951956404285571818041046681288797402925517668012340617298396574731619152386723046235125934896058590588284654793540505936202376547807442730582144527058988756251452817793413352141920744623027518729185432862375737063985485319476416926263819972887006907013899256524297198527698749274196276811060702333710356481}', 'name': 'multer'}, 'id': 'multer:2', 'type': 'function'}], function_call=None, provider_specific_fields={'refusal': None}),
 {'tool_call_id': 'multer:2',
  'role': 'tool',
  'name': 'multer',
  'content': '746783697482040087065919508369733176450819553567468015501273863444158081234530502459987377877607954440937530130862950316217454109184321717162702673965618374628383497188525161877614039903912808571143636082093362577594805851035336024681234596793149463238304773446092470251869792117181176569309587081011872404753095614885461164289054117977512502905635586826704283841489246055037458370865724751474127970970638952833852527639945774013814027798513048594397055397498548392553622121404667420712962'}]

Using Lisette

Lisette is pretty concise. In only 2 lines of code, you can have a tool-loop up and running

m = 'moonshot/kimi-k2-0711-preview'
c = Chat(m); c

<lisette.core.Chat at 0x7f45865da300>

r = c('Who are you?')

from IPython.display import display
c = Chat(m, tools=[multer,pow])
o = c("What's the result of (13**2)*25.5?", max_steps=10, return_all=True); display(*o)

{'tool_call_id': 'pow:0', 'role': 'tool', 'name': 'pow', 'content': '169'}

{'tool_call_id': 'multer:1',
 'role': 'tool',
 'name': 'multer',
 'content': '4309.5'}

The only thing to note here is the max_steps parameter. That determines the length of the tool-loop.

Conclusion

Building a LLM agent is simpler than it looks seem—at its core, it’s just a LLM calling functions in a loop.

By repeatedly calling the model, executing requested tools, and feeding results back into the conversation history, you create a system that can solve complex multi-step problems.

If you have any comments, questions, suggestions, feedback, criticisms, or corrections, please do post them down in the comment section below!

The Confusion of Two Attentions

We observed the code first calculating an llm_attn_matrix. At the time, we thought: “Great, this must cover all attention, including both text and images.” But then, the code computed a separate vis_attn_matrix.

“Why recalculate visual attention?” you asked. “Didn’t we already calculate attention for all tokens, including the image tokens?” This was a brilliant question—and the key to understanding the entire process. We gradually realized these two attention types originate from different parts of the model and address fundamentally distinct questions.

The Language Model, the Clever “Manager”

We first grasped the role of the language model’s (LLM) attention. It acts like a strategic manager. When the model generates a word like “cat,” this “manager” examines the question and image, then decides: “To output ‘cat,’ the image token representing ‘ears’ is most critical, while ‘whiskers’ is moderately important.” Thus, the LLM’s attention answers the “which” question. It assigns an importance score to each image token. But here was the catch: these scores are “coarse.” They reveal which large patch matters but not what within the patch the model focuses on. Plotting this alone would color entire patches uniformly, yielding unusable visualizations.

The Vision Encoder, the Diligent “Analyst”

So what does the vision encoder’s (ViT) attention do? Initially, we mistakenly assumed it also knew the question. But we soon learned it works like a meticulous “analyst” that completes its job before the question ever arrives. Its task is to analyze the image in extreme detail, completely independent of the question. It examines every small image patch and generates a “map” for each, revealing how that patch relates to all others. The ViT’s attention thus answers the “what” question—outputting detailed maps without scores. “But if it doesn’t know the question,” you asked, “how would it highlight relevant features? It could focus on irrelevant elements.” You were absolutely right. The ViT alone cannot determine what matters for the question; it merely catalogues all interesting structures in the image.

The Power of Collaboration

Finally, we understood how everything integrates. It’s a two-step process, like manager and analyst collaborating:

1. The diligent analyst (ViT) first prepares a detailed image analysis report—containing a unique attention “map” for every patch.
2. The clever manager (LLM) reads the question, consults this report, and selects which maps to use (and their weights).
3. We, as external observers, then combine these selected maps according to the LLM’s weights to produce the final, clear attention heatmap.

The resulting visualization isn’t generated by any single component—it emerges from the symbiosis of both systems, harmonized by our interpretation.

End of Narration

If you have any comments, questions, suggestions, feedback, criticisms, or corrections, please do post them down in the comment section below!

Frequently Asked Questions (FAQ)

Q: Why do we need two attention mechanisms (LLM’s and ViT’s)? A: Because they answer two distinct but necessary questions. LLM attention is “coarse,” identifying which patch matters for word generation. ViT attention is “fine,” providing spatial “maps” for each patch to reveal internal image structures.

Q: If the vision encoder doesn’t know my question, isn’t its attention potentially irrelevant? A: Correct. The ViT objectively analyzes the image, cataloguing all salient structures unrelated to the question. The LLM then acts as a filter, selecting only ViT outputs relevant to the question.

Q: Why not use only the LLM’s attention for plotting? Doesn’t it already know what’s important? A: LLM attention assigns scores to entire patches but lacks spatial resolution. Plotting it alone would color entire patches uniformly, producing low-detail visualizations. ViT maps provide the necessary granularity.

Q: So the vision encoder generates one attention map per image patch? How do we display so many maps? A: Exactly. For a 576-patch image, the ViT produces 576 maps. The final heatmap is a weighted average of all 576 maps, with weights derived from the LLM’s importance scores for each patch.

RoboOS: A 3-piece kit instead of a 300-piece puzzle

RoboOS decomposes the generalizability problem of robotics into three clean layers:

• The Embodied Brain: A cloud-deployed MLLM that thinks globally–planning, replanning, and correcting errors in real time.
• The Cerebellum Skill Library: A plug-and-play toolkit that enables various actions, allowing the robot to perform different functions.
• Real-Time Shared Memory: Helps maintain spatial memory, temporal memory, and embodiment memory.

There are three manageable chunks that allows for the creation of systems that are general enough for specialized tasks, but not overly general that its bad at everything. It might not allow for novel actions, but it can allow for known actions. And it allows for each of these pieces to be iterated on through their own merits.

In addition, RoboOS allows for multiple different robots to coordinate with each other. What occured in my mind is that that doesn’t have to be individual separate robots; it could be different pieces of the same robot (e.g., the legs and the hands of a robot).

RoboBrain: solve the foundations, and work with the data

Many general purpose robotic solutions try to remedy issues with VLMs with all sorts of pipelines, gadgets, and gizmos. Rather than actually solving the foundational issue that is the VLM itself. Many solutions use VLMs for purposes that they are exactly bad for. VLMs are langauge biased, spatially blind, and temporally naive.

RoboBrain 1.0 tackled:

• task planning by using QA pairs that decompose long horizon tasks (“make coffee”) into atomic steps (“reach mug → grasp handle → lift → …”)
• affordance identification by using 6k images with bounding boxes of “where you’d actually touch the mug” and LoRA-tuning the LLM
• trajectory planning by feeding the LLM start / goal 2-D points and asking it to regress waypoints

RoboBrain 2.0 doubles down on this philosophy with improvements in three aspects:

• Spatial understanding with multiview static images, visual grounding, object pointing, and spatial referring examples
• Temporal modeling by linking video frames with positional encoding that tracks both the space and time of each frame, multi-robot planning examples, close-loop interaction examples, and ego-view planning
• Reasoning chains

What was done for both models was construct a dataset that forced the model to learn these skills.

In addition, the RoboBrain papers did a bunch of data curating so as to ensure desired outputs were obtained: truncating examples, increasing throughput of other examples, limiting certain annotations, ensuring diversity of certain examples, mixing data from various stages, and more. In addition, RoboBrain 2.0 is able to output in a wide variety of modalities, making it suitable for a wide range of embodied robotic tasks.

Conclusion

The most prominent parts of these papers that stuck out to me is the focus on decomposing problems into much smaller manageable ones, and fixing problems by working with the data.

• RoboOS: a minimal, composable OS
• RoboBrain: a VLM trained for robotics. Not a VLM adapted to robotics.

Maybe this is the viable road to allow enough generalizability in robots.

If you have any comments, questions, suggestions, feedback, criticisms, or corrections, please do post them down in the comment section below!

Links

• BAAI (English)
• BAAI (中文)
• RoboOS
• Robobrain
• RoboBrain 2.0

In addition, RoboBrain 2 contains information about their training and inference infrastructure.

3D Understanding and Reconstruction

• 3D geometric encoding for monocular images: Use a 3D geometric encoder to produce 3D tokens from monocular images (RGB only images; no depth map). VLM-3R: Vision language models, augmented with instruction aligned 3D reconstruction
  
• Evaluating 3D awareness in image models: Explores whether image models can “see” in 3D. Finds self-supervised models are good at single-view awareness but fail at multiview awareness. Probing the 3D awareness of Visual Foundation Models
  
• CLIP-based 3D-text embedding mapping: Using CLIP, create an embedding that maps text to 3D point clouds. Unified Representation Space for 3D Visual Grounding
  
• Dual-encoder system for 2D semantics and 3D structure: Enhances spatial understanding with a 2D encoder (for visual semantics) and a 3D encoder (for 3D structure), eliminating the need for 3D data like depth maps, and “intelligently” selecting key frames. Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence
  
• Intermediate perception token generation: Trains a VLM to output intermediate perception tokens (e.g., depth tokens) to facilitate its reasoning process. Perception tokens and enhance visual reasoning in multimodal language models
  
• Enhanced CLIP with intermediate layers and video support: A stronger CLIP variant that works on video, uses intermediate layers, scales well, aligns for VL and spatial tasks, and includes a dataset. Perception Encoder: The best visual embeddings are not at the output of the network
  
• View-aware spatial reasoning for vision-language models: Enables vision-language models to answer spatial questions about 3D scenes by generating and using imagined views from different perspectives, paired with a world model for view synthesis. Mental Exploration: Augmenting Vision-Language Models with View Synthesis for Spatial Reasoning

Spatial Reasoning Enhancements

• Incorporating horizontal lines and row-wise scanning: Introduce horizontal lines to input images, and prompt to scan row-by-row to improve spatial performance. Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs
  
• Implicit Perception Loss for policy optimization: Extends GRPO by introducing “Implicit Perception Loss” to encourage the model to rely more on visual input. Perception-Aware Policy Optimization for Multimodal Reasoning
  
• Latent visual token-based reasoning: Enables VLMs to “imagine” scenes using latent visual tokens (without outputting pixels). Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokens

Datasets and Benchmarks for Spatial Understanding

• Spatial detail-rich QnA dataset finetuning: Mixes existing VLMs with a custom image QnA dataset containing object details (presence, depth, 3D positions). SpatialVLM: Endowing VLMs with Spatial Reasoning Capabilities
  
• Spatial relationship dataset for robotics: A dataset to help VLMs learn spatial relationships better. RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics
  
• Simulated dynamic spatial dataset: A dataset produced from simulations to help VLMs improve understanding of space and movement. SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models
  
• Multi-modal spatial perception benchmark: Tests how well VLMs understand objects in space and their relation to other images. MIRAGE: A Multi-modal Benchmark for Spatial Perception, Reasoning, and Intelligence
  
• Multi-image spatial intelligence benchmark: Evaluates VLM spatial understanding using multiple images of the same scene. MMSI-Bench: A benchmark for multi-image spatial intelligence
  
• Spatial intelligence evaluation framework: A benchmark to test VLM spatial intelligence. SITE: Towards spatial intelligence through evaluation
  
• Multi-view understanding evaluation: A benchmark to assess VLM performance across varied angles of the same scene, finding VLMs struggle with this. Seeing from Another Perspective: Evaluating Mutli-View Understanding in MLLMs
  
• Hierarchical spatial weakness evaluation: A framework to identify VLM weaknesses, finding VLMs struggle with depth perception, view switching (first/third-person), and physical reasoning (e.g., occlusion). SPHERE: Unveiling Spatial Blindspots in VLMs through Hierarchical Evaluation
  
• Depth and object annotation dataset for 3D finetuning: Creates a custom dataset to finetune a VLM on depth images and object annotations. MM-Spatial: Exploring 3D Spatial Understanding in MLLMs
  
• Depth map-annotated dataset finetuning: Finetunes a VLM on a custom dataset of annotated RGB images + depth maps. SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models
• RGBD QnA dataset finetuning: Finetunes a VLM on a custom dataset of annotated RGB images + depth maps + QnA. SpatialBot: Precise Spatial Understanding with Vision Language Models

Attention Mechanisms and Visual Feature Utilization

• Attention logit temperature adjustment: Guides attention to relevant image parts by increasing the temperature of attention logits in confident answers. Why is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas
  
• Improving visual feature utilization: Highlights that VLM vision encoders outperform the VLM itself, as VLMs fail to effectively use vision encoder outputs, relying instead on language biases. Hidden in plain sight: VLMs overlook their visual representations
  
• Teacher-student training with refiner modules: Refines CLIP’s spatial awareness using a teacher/student model and a refiner module to remove semantic contamination (noise from irrelevant context). Refining CLIP’s spatial awareness: A Visual-centric Perspective

Multimodal and Temporal Spatial Reasoning

• Temporal event prediction and segmentation: Teaches a model to infer missing parts in video events via cause and effect, and to split videos into non-overlapping events with detailed timestamps. Tempura: Temporal event masked prediction and understanding for reasoning in action
  
• Multiframe depth and correspondence learning: Improves VLMs’ understanding of spatial relationships between frames by teaching depth perception, visual correspondence, and dynamic perception. Multi-SpatialMLLM: Multiframe Spatial Understanding with MLLMs
  
• Human motion and video finetuning: Finetunes a VLM on videos of human actions + skeleton sequences or 3D human movement models, enabling it to output text describing motions. MotionLLM: Understanding Human Behaviors from Human Motions and Videos

Novel Prompting and Interaction Methods

• Iterative visual prompting (“hot and cold”): Elicits actionable knowledge from VLMs by having them iteratively select from the best visual suggestions (e.g., arrows/markers on images) instead of outputting actions directly. PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
  
• VLM cropping (scanning-like behavior): A technique reminiscent of CNN scanning. VLM Cropping

Cognitive and Mental Models of Space

• Cognitive map exploration: Explores “cognitive maps” as a way for VLMs to visualize and encode space, studying how they see, remember, and recall spaces. Thinking in Space: How MLLMs See, Remember, and Recall Spaces
  
• Latent visual token-based imagination: Enables VLMs to “imagine” scenes using latent visual tokens (without outputting pixels). Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokens

Conclusion

If you have any comments, questions, suggestions, feedback, criticisms, or corrections, please do post them down in the comment section below!

Foreword

In this post, I have annotated a presentation I gave at my univesity’s multimodal intelligence lab. I’m a newcomer to general purpose robotics, so I had a different mindset and perspective going in.

At the end of this post, I provide another perspective that arose as I wrote these annotations.

Annotated Presentation

Here, I introduce the task assigned to me, which focused on addressing the limitations of VLMs in understanding object poses through fine-tuning on annotated videos.

The core problem is that current vision-language models (VLMs) struggle with understanding object poses. When queried about an object’s orientation or directional changes, VLMs often fail when even simple complexity is introduced or they forget the object’s prior states.

The goal is to fine-tune a VLM on videos annotated with pose information to improve its performance in understanding and reasoning about object poses.

Two main obstacles exist: data and pose models.

A major obstacle is the lack of existing annotated video datasets with object pose information. Self-recorded samples suffer from limited diversity and aren’t suitable for achieving generalizability. While off-the-shelf video datasets exist, they can’t be easily processed due to pose model requirements.

Pose estimation models present significant friction due to their specialized input requirements (camera data, reference images, 3D models) and fragmented ecosystem. There’s no standardized text-to-pose interface, requiring manual masking and setup of different libraries/environments for each model.

The current workflow involves too much friction and insufficient iteration, leading to a lack of confidence in the self-created dataset approach.

Quick iteration is key. Quick iteration is key. Quick iteration is key.

The workflow involves multiple steps with numerous variables: recording videos (sample count, duration, frames, pose complexity, object count), segmenting videos into frames, pose annotation (RGB, depth, mask frames, camera metadata), text annotation (QnA pairs), and finally fine-tuning. Each step introduces significant time investment and uncertainty.

The creation of this dataset is a whole project of its own, with the pose model itself being the core component.

So I suggested a different direction: why not test the underlying theory but with those pieces of data that have less friction.

Given the challenges, a different direction maintaining the same core theory is proposed: testing whether annotating images (with RGBD or other data) improves VLM performance on specific tasks, rather than focusing on pose annotations.

I also did a few experiments of my own, before I was assigned this project, on RGBD images to test for spatial understanding. An off-the-shelf VLM scored ~28% on a given benchmark, a 100 QnA pair finetune scored ~34%, and a 2000 QnA pair finetune scored ~12%.

My current intuition for the discrepency on the larger sample size is that while there were 2000 samples, there were only 300 unique RGBD images. This is contrary to the 100 QnA pair set which had 100 unique images. The suggests diversity is indeed important, as I pointed out earlier in my presentation.

Benefits a switching approaches is quicker iteration.

This new direction enables quicker iteration, allowing me to try more ideas out. I had set up a pipeline in which I was able to conduct those 3 prior benchmarks in a couple of days. With quicker iteration, I was able to already test for diversity. With quicker iteration, I could also test out other ideas, such as: should I keep the RGB and depth images separate? should I overlay the depth image over the RGB image? does using multiple view points of a scene help increase spatial understanding?

So I am brought to my current perspective on the current state of general purpose robotics: better, more impactful directions exist. Directions that have small steps, but allow for interesting, quick iterations and explorations.

This is how I currently see the field. I see to many papers and people jumping to fancy pipelines, rather than iterating on the core pieces that underpin those fancy pipelines.

I’m surprised that the primary reason that these general purpose pipelines use VLMs for suck at the very specific reason. (I know this meme doesn’t match the original intent of the xkcd meme, but it appeared in my mind.)

The field seems to be jumping to complex pipeline integrations rather than conducting small, meaningful iteration experiments. There’s a lack of basic infrastructure for quick experimentation, such as pose estimation models being problematic. Without these basic components and pieces being ironed out, the more interesting directions cannot be explored.

Better directions exist that focus on fundamental improvements through systematic testing rather than complex integrations.

Closing Words

As I annotated this presention, it just occured to me: overlay pose annotations on videos perhaps might not be needed at all. Current vision models and VLMs can pick out objects in images without the need for, say, a bounding box and label around the object in question. So perhaps, what needs to be done is that, perhaps, QnA pairs are created in a way that allows for the VLM to learn the concept of spaces, depth, poses, etc. The answer describes the manipulation of the object, the change in pose of the object, and so on.

If you have any comments, questions, suggestions, feedback, criticisms, or corrections, please do post them down in the comment section below!

算术运算

• 是 “n 加 m”.
• 是 “n 减 m”.

乘除法

• 标准说法：
  • 是 “n 乘以 m”
  • 是 “n 除以 m”
• 传统用法：
  • “n 乘 m” 可能表示
  • “n 除 m” 可能表示

等式/相等

• 是 “等于”
• 是 “约等于”
• 是 “不等于”
• 是 “小于”
• 是 “大于”
• 是 “小于等于”
• 是 “小于等于”

负数

• 是 “负 4 点 87”

几何

• 是 “角 ABC”
• 是 “三角形 ABC”

微积分

• 是 “dy 除以 dx” 或者 “y 对 x 的导数”
• 是 “f(x) 的积分”

集合论

• 是 “属于”
• 是 “子集”

组合数学

• 是 “n 取 m” 或者 “从 n 个元素中取 m 个元素的组合数”

阶乘与连乘

• 是 “1 乘 2 乘 3 乘 4 一直乘下去”
• 是 “1 乘 2 乘 3 一直乘到 n” 或者 “n 的阶乘”

分数

• 是 “b 分之 a”
• 是 “c 又 b 分之 a”

指数与根式

• 是 “n 次方”
• 是 “根号 n”
• 是 “n 的 m 次方根”

如果你有任何意见、问题、建议、反馈、批评、或更正，请在下方评论区留言！

In a Nutshell

• R1-Zero is a model created purely through reinforcement learning (RL), with Group Relative Policy Optimization (GRPO) as the policy.
• R1 is a model created with a combination of multiple RL and supervised finetuning (SFT) stages.

R1: The Hybrid Training Pipeline

In order to remedy these issues, a multi-stage training pipeline was introduced.

flowchart LR
  A[SFT]-->B[RL]-->C[SFT]-->D[RL]
  style A fill:#b3e6ff,stroke:#333
  style C fill:#b3e6ff,stroke:#333
  style B fill:#ffb3b3,stroke:#333
  style D fill:#ffb3b3,stroke:#333

This process arose from wondering whether performance could be improved, convergence accelerated, or a coherent chain-of-thought (CoT) be produced through a small amount of high quality data.

Conclusion

If you have any comments, questions, suggestions, feedback, criticisms, or corrections, please do post them down in the comment section below!

Read the Deepseek R1 paper here..

Training Strategies

I know it sounds really obvious, but one should never start the finetuning process on the full dataset with the largest model variant right out of the box. There will be issues. One may be under the assumption that they’ve implemented the tweaks they needed to on the model, in the first attempt. Or that the data is correctly processed and formatted. However, chances are that they’re not. Save time, money, and resources by finetuning on on a small subset of the data with a smaller model variant. Once it is known the whole process works correctly and as expected, then go ahead with the full finetune.

BPE Tokenizers

Byte Pair Encoding (BPE) tokenizers are “dynamic” tokenizers. That is, they do not tokenize text according to a preset vocabulary or set of rules. Rather, they tokenize according to an algorithm. In short, frequent sequences of characters are merged together to form tokens. These tokenizers also store their vocabulary in a data type known as byte strings.

GPUs

There are various types of GPUs. I got to try out the mighty A100 all the way to reliable T4. I found the L4 and 4090 to be the sweet spot, though I’d lean towards the latter. Also, if you’re simply going after a GPU, don’t go for the major cloud providers such as Google Cloud. The only reason our company was using Google Cloud was because they had obtained free credits. Otherwise, it’s, well, a rip off when compared to other providers (in the context of research).

Forward Hooks

A hook is a piece of code that can be inserted at predefined places of an algorithm to achieve extra functionality. In the context of deep learning, a forward hook is a piece of code that is run after the forward pass of a model. By creating a forward hook, one can directly play around with and tweak the computed logits before the loss is calculated.

Token Generation

With Hugging Face, at least, the way models generate tokens during training and inference is different. During training, all output tokens are generated simultaneously whereas during inference, generation occurs token by token.

However, as I dug behind the scenes of the builtin generate method, I found that during inference, all tokens were actually generated simultaneously–in a way. Say, if 10 tokens were to be generated, for each token, 10 tokens would be generated with the remaining 9 being “filler” tokens.

Logit Processors

Logit processors are nifty objects that Hugging Face provides to be able to conveniently tweak a model’s logits during inference with the builtin generate method.

Handling Mistakes

Mistakes happen. Especially for challenging tasks. That’s normal. Mistakes should happen. What’s important is that one learns from their mistakes.

FSDP

Fully Sharded Data Parallel is a technique for splitting a model across multiple GPUs. Imagine literally splitting a model’s parameters into a bunch of different groups, and then putting those groups onto multiple GPUs that run simultaneously. When a layer is being computed on a GPU, all relevant parameters are copied to that GPU.

LoRA

Low Rank Adaptation (LoRA) is a LLM finetuning technique in which new layers are added onto the head of the existing model. These new layers are then trained to achieve a specific task, and form an, well, adapter of sorts. Think of a screwdriver handle onto which you can attach various heads, each head achieving a specific task. In the context of LLMs, these specific tasks could be increasing the performance of a model, reducing its resource usage, or changing what sort of output it generates.

Quantization

Quantization is a LLM inference technique whereby the weights in the remaining few layers of the model are stored in 4 or fewer bits, as opposed to the more typical 16 or 32 bits. This allows more weights to be stored in the same amount of memory, and larger models to be fit on smaller hardware, all whilst retaining the relative same level of performance.

Reading Papers

When browsing papers, one typically wants to read the abstract, headings, figures, and results. From this, one can determine whether a given paper is worthy for their goal or not.

When reading a paper, one wants to do it iteratively. In the first iteration, also only read the abstract, figures, and results. In the next iteration, read the paper but not with too much focus. Don’t understand something? That’s fine, move on. In the third iteration, start reading more closely and try to make sense of what is going on. Get bogged down? Spend some more time on it, but not too much more–save it for the next iteration. And so on. As you do more iterations, it may be helpful to rephrase or summarize what each section discusses. This allows you to find gaps in your knowledge and determine whether you understand what you just wrote.

Looping and Slicing

Loops can be immensely slow. I managed to deflate my training time from 10+ hours to around 2 hours by replacing two nested for loops with a single slicing operation.

LLM Training Data Formats

There are different ways LLM training data can be formatted. The Alpaca format uses '###' as the delimiter, while ChatML uses the tokens '<|im_start|> and '<|im_end|>' as the delimiters.

Best Performing Open Source Models

The best performing open source models come from the other side of the Pacific ocean; from China. The AI scene there is quite vibrant, with most of the actual advances happening there. If you look at the latest papers, most of the authors will be from there too. During my internships, I came across so many models I never had heard off that were well performers or had interesting perks or quirks. Qwen is the best performing open source model. Deepseek is the most cost effective endpoint that exists. Then there are so many other models such as InternLM, Yi, PhotoMaker, and more.

Refactoring

Refactored code is abstracted code. Abstracted code is maintainable code. Abstracted code is understandable code. Abstracted code is good code.

Data

It’s stating the obvious, but data matters. It actually does influence the quality or manner of the model output.

Conclusion