AbstractPhil/geoclip-vit-base-patch-32-512d

Preface

A first experiment to test and convert clip-vit-base-patch32 into a geometric model by using only a classification head.

Below is GPT 5's auto-generated dictation based on the notebook. I have included the entire notebook 6 for posterity.

The question was simple; can linear layers learn geometric?

The answer is... maybe. More research required.

Reasoning

I used the 32 dim geometric vocab; as it seemed to be the weakest with flow-match euler-discreet to test the hypothesis that a small dimensional geometry could in fact be used in substitution of a high-geometric variation.

For obvious reasons the 512 dim vocab managed to handle a higher accuracy than the 32d; however not for the reason you would think.

The output model is much larger than I wanted; which defeats the purpose of the overall structure - but it's paired directly at the knee with clip-vit-base-patch32, so I'll prepare a decoupled version here in a bit.

Why clip-vit instead of just vit?

I believe the clip-vit variations have more utility overall so I wanted to ensure a fair target was assessed.

Notebook-6 · Crystal-CLIP CIFAR-100

One-vector image embeddings (HF CLIP) + pentachora vocabulary anchors → cosine-similarity classifier for CIFAR-100. This repo hosts the trained crystal classification head (+ run configs/metrics) built in Notebook 6.

---

OVERVIEW

• Vision encoder: openai/clip-vit-base-patch32 (Hugging Face transformers), frozen by default. Produces exactly one L2-normalized embedding per image (image_embeds, dim=512).
• Vocabulary: AbstractPhil/geometric-vocab-512d (pentachora crystals). For CIFAR-100 class names, any missing tokens are deterministically synthesized via the unicode path to guarantee 100/100 coverage and preserve class ordering.
• Head: projects both image embeddings (De=512) and role-selected class anchors (Dv=512) into a shared symbol space (crystal_dims=128), L2-normalizes, and computes cosine logits divided by T (temperature).
• Training: Cross-Entropy on CIFAR-100, AdamW, optional AMP, cosine LR with warmup. Best checkpoint is saved and (optionally) pushed to Hugging Face.

---

MODEL CARD

• Task: Image Classification (CIFAR-100)
• Backbone: openai/clip-vit-base-patch32 (vision-only)
• Head: Crystal projection head (image 512→128, anchor 512→128) + cosine logits (temperature)
• Vocabulary: AbstractPhil/geometric-vocab-512d (wordnet_eng split + deterministic unicode synth for gaps)
• Metrics: Top-1 = [80~], Top-3 = [90>]
• License: MIT

---

FILES IN THIS REPO

• _best.safetensors — weights for:
  • head::* (crystal classifier head)
  • encoder::* (optional, if you chose to unfreeze/fine-tune)
• _best.config.json — full CONFIG used for the run
• _best.metrics.json — summary metrics for the best epoch
• Optionally: _latest. variants if you pushed latest per-epoch artifacts.

Note: If you only want to ship the head, you can also include a stripped crystal_head.safetensors (head-only state_dict). The snippets below handle either format.

---

QUICKSTART (Inference)

1. Load CLIP vision (frozen) and processor HF_CLIP_ID = "openai/clip-vit-base-patch32" Processor = AutoImageProcessor.from_pretrained(HF_CLIP_ID) Encoder = CLIPVisionModelWithProjection.from_pretrained(HF_CLIP_ID).eval().to("cuda")

2. Build the crystal head (same shape as training) image_dim = Encoder.config.projection_dim # 512 crystal_dim = 512 # vocab repo uses 512D anchors sym_dim = 128 # crystal_dims from CONFIG temperature = 0.07 # from CONFIG

   class CrystalHead(torch.nn.Module): def init(self, De, Dv, Dsym, T): super().init() self.proj_img = torch.nn.Linear(De, Dsym, bias=True) self.proj_anc = torch.nn.Linear(Dv, Dsym, bias=False) self.T = T self.register_buffer("anchors_vocab", torch.empty(0, Dv), persistent=False) def set_anchors(self, anchors): # [C, Dv] self.anchors_vocab = anchors.contiguous() def forward(self, image_embeds): # [B, De] (L2 ok) z = torch.nn.functional.normalize(self.proj_img(image_embeds), dim=-1) a = torch.nn.functional.normalize(self.proj_anc(self.anchors_vocab), dim=-1) return (z @ a.T) / max(1e-8, self.T) # [B, C] head = CrystalHead(De=image_dim, Dv=crystal_dim, Dsym=sym_dim, T=temperature).to("cuda")

3. Load weights (handles prefixed multi-module .safetensors) state = safetensors.torch.load_file("_best.safetensors") head_state = {k.split("head::",1)[1]: v for k,v in state.items() if k.startswith("head::")} head.load_state_dict(head_state, strict=True)

4. Prepare anchors from your vocabulary (same order as training) You likely already exported anchors or can rebuild them exactly as in Notebook 6. anchors: torch.Tensor of shape [100, 512] head.set_anchors(anchors.to("cuda"))

5. Inference on a batch of images (PIL or ndarray) imgs = [PIL.Image.open("example_0.png").convert("RGB"), PIL.Image.open("example_1.png").convert("RGB")] batch = Processor(images=imgs, return_tensors="pt").to("cuda") with torch.no_grad(): out = Encoder(pixel_values=batch["pixel_values"], return_dict=True) z = torch.nn.functional.normalize(out.image_embeds, dim=-1) # [B, 512] logits = head(z) # [B, 100] pred = logits.argmax(dim=-1).tolist() print("pred:", pred)

Note: The head expects the same class order used at training time. Save and ship class_names.json (CIFAR-100 labels) and the exact anchors_vocab.pt you used (or rebuild deterministically with the vocab + synth step).

---

REPRODUCE (Notebook 6)

1. Config only (single source of truth): image size, CLIP stats, dataset, temperature, crystal dims, etc.
2. Cell 5 – HF CLIP vision loader (one embedding per image).
3. Cell 6 – Vocabulary interface; synth any missing CIFAR tokens, cache crystals, select role anchors.
4. Cell 8 – Crystal head (image+anchor projections → cosine logits / T).
5. Cell 9 – Trainer (AdamW + AMP + cosine LR). Saves latest/best, pushes to HF if enabled.

Replace with your final numbers after the run completes.

---

ACKNOWLEDGEMENTS

• CLIP ViT-B/32: OpenAI (openai/clip-vit-base-patch32) via Hugging Face transformers.
• Pentachora Vocabulary: AbstractPhil/geometric-vocab-512d.
• Built in Notebook 6 (CONFIG-first, deterministic synth for gaps, head-only training).