Model index

Independent models. Shared research lineage.

ORIS is a collection of independent experiments rather than one architecture. The current work is increasingly about the interaction between architecture, data quality and how much useful knowledge a very small model can actually afford to represent.

Data, data, and data again.

Testing the small-model line in the Polish SLM Benchmark kept pointing to the same conclusion: at this scale, architecture alone is not the bottleneck. A small model can waste a huge fraction of its parameters memorizing boilerplate, repeated web patterns and low-value structure. Current work therefore spends at least as much effort on corpus curation and the distribution of knowledge over a limited parameter budget as on the model itself.

Working assumption

These models are too small to be good at everything, but making them useful for only one narrow task is often equally uninteresting. The research problem is to find the useful middle: a model with a clear role, enough breadth to generalize, and a corpus shaped around what its parameter budget can realistically retain.

Completed

Finished experimental branches

2024 · completed

PolMATH

Custom numerical-representation research and the project where much of the early end-to-end model work was developed.

custom architecturenumeric channelcompleted

2026 · completed

ORIS BERT

Compact Polish BERT-style encoder developed for local filtering, scoring, embeddings and downstream fine-tuning.

25.41Mlocal/global attentioncompleted

2026 · completed

Koliber 1.0

A ~126M Polish causal language model used as the first stable Koliber baseline and as a reference point for later architecture and data experiments.

~126M12 layersGQAcompleted

Current

What is being trained now

2026 · active research

Koliber 1.1

An experimental continuation of Koliber 1.0 focused on reducing semantic drift and overconfident web/SEO-like continuations by correcting parts of the model's token distribution during continued training.

~126M classbase LMlogit correction researchactive

Abandoned / paused indefinitely

Useful experiments that are not being continued

2025–2026 · abandoned

ORIS 660M / Qwen 0.8B

Structural recovery, tokenizer replacement and partial-freezing experiments that produced useful diagnostics but are no longer an active training branch.

ORIS 660MQwen 0.8Babandoned

2026 · abandoned

Vyuhu

Supernetwork research with deterministic compute profiles and exact physical extraction. The architecture-validation work remains documented, but the planned full branch is not being continued.

4 profilesexact extraction0.834B validation runabandoned

Planned

Projects kept on the roadmap

planned

Vidar-VL

Planned ORIS vision-language model for understanding images and interacting with visual content in Polish.

image understandingOCR / documentsmultimodal reasoning

planned

ORIS Vision

Compact visual encoder planned as the visual counterpart to ORIS BERT for similarity, retrieval, scoring and dataset filtering.

vision encoderembeddingsvisual filtering

PolMATH

Numerical representation research

PolMATH

A custom language-model experiment that became the main place to learn the complete model pipeline in practice — from data formatting and tokenizer training, through embeddings and transformer blocks, to forward passes, custom heads, decoding and the awkward reality of exporting non-standard solutions to Hugging Face. The numerical idea mattered, but so did learning how all the pieces actually fit together.

2024

Initial architecture and training experimentsCustom numerical channel, numerical decoding, routing and number↔text association tests.

Completed

Original PolMATH research branch finishedThe existing experiment is treated as complete; any future publication-oriented rework would be a separate continuation.

Why it existed

Value and notation were treated as related, but not identical.

PolMATH did not begin as an adapter attached to an existing pretrained model. The transformer stack, tokenizer behaviour, numerical codec, embedding fusion, auxiliary losses, routing logic and numerical decoding path were implemented as one experimental architecture.

Text could contain a dedicated numerical position while the actual value travelled through a structured numerical representation. That representation could encode integer and fractional digits, lengths, decimal form and exponent structure rather than collapsing every number into one scalar or leaving it entirely to BPE tokenization.

This made the experiment less about “doing mathematics” and more about asking whether a language model benefits from separating what a number means from how that number happened to be written.

Architecture

Language decides that a number belongs here. A numerical path decides which number.

Tokenizer

[NUM] position

Written numbers could be normalized into a dedicated numerical position while preserving linguistic context around them.

Numeric codec

Structured representation

Integer digits, fractional digits, notation flags and exponent structure were represented explicitly.

Fusion

Gated numerical injection

Numerical features were projected into the token representation and gated per position instead of affecting every token indiscriminately.

Objectives

Multiple numerical heads

Categorical structure prediction and a bucket/regression path provided complementary ways of reconstructing values.

Alignment

Numeric-only pressure

An auxiliary objective aligned text and numerical representations at number positions and penalized numerical leakage outside them.

Generation

Two-stage output

The LM could first decide that a numerical token belongs in the sequence, then the numerical head could reconstruct the value.

Technical notebook

The useful part is in the plumbing.

PolMATH was the project where the author spent the most time taking the pipeline apart and putting it back together. The point of the notes below is not to dump the repository onto a webpage, but to expose the decisions that normally disappear behind AutoModel.from_pretrained().

Tokenizer · ByteLevel-BPE + a real numerical position

The tokenizer branch trains a ByteLevel-BPE vocabulary while keeping [PAD], [BOS], [EOS], [NUM] and the ten digits available explicitly. A regex detects ordinary decimals, decimal commas, scientific notation and suffixes such as 1930s or 10th. The numeric magnitude is moved out of the text stream and the text receives [NUM] in its place.

pre, spans = self._preprocess_text_for_numbers(raw_text)
enc = self._tok.encode(pre)
# token stream:  "Temperature is [NUM] degrees"
# side channel:  {position_of_NUM: "23.5"}

This makes the tokenizer responsible for linguistic placement, while the numeric path can carry the value itself.

Number codec · structure instead of one floating-point scalar

The categorical codec decomposes a number into notation flags, integer/fraction/exponent lengths, exponent sign and digit classes. In the current magnitude-only branch the leading value sign is intentionally separated from the numeric magnitude.

Integer digits

up to 10 explicit digit slots

Fraction

up to 6 digit slots + decimal comma flag

Exponent

up to 3 digits + exponent sign

Notation flags

scientific / fraction / decimal form

1939.0  → int=[1,9,3,9], frac=[0]
1.939e3 → int=[1], frac=[9,3,9], exp=[3]
# different notation, recoverable structure

Embedding fusion · let the model decide how much number to inject

The normal token, position and token-type embeddings are built first. Numeric features are projected to hidden size. In the gated variant a small MLP sees both representations and produces a per-position gate; the numeric mask suppresses the channel everywhere except numerical positions.

proj = self.numeric_proj(numeric_features)
g = sigmoid(self.gate_mlp(cat([x_base, proj])))
g = g * numeric_mask.unsqueeze(-1)
x = x_base + g * proj

An auxiliary term experiments with aligning text and numeric representations where a number exists and penalizing numeric leakage elsewhere.

Transformer body and forward · deliberately ordinary where it should be ordinary

The language backbone is intentionally understandable: causal scaled dot-product self-attention, a triangular mask, pre-norm residual blocks and a GELU MLP. One default configuration in the branch uses a 32k vocabulary, 768 hidden width, 12 layers, 12 attention heads, 3072 intermediate width and 2048 positions.

h = self.ln1(x)
x = x + self.attn(h, attn_mask=attention_mask)
h2 = self.ln2(x)
x = x + self.mlp(h2)

This was useful pedagogically: custom behaviour was isolated around numbers rather than hiding every component behind another abstraction.

Heads + router · categorical reconstruction versus bucket/regression

The model can emit ordinary vocabulary logits and numerical predictions from the same hidden states. The categorical head predicts the components needed to reconstruct notation. An optional mixed head adds bucket classification and regression. A learned two-way router tests whether the model can choose which numerical path should dominate instead of hard-coding the choice.

logits   = self.lm_head(x)
pred_cat = self.numeric_head_cat(x)
pred_mix = self.numeric_head_mix(x)
gate     = softmax(self.router(x), dim=-1)

Tests · what gets checked before a long run

The tokenizer script includes a direct sanity check on special-token IDs, digit IDs, encoded tokens, numeric positions and reconstruction. One test string deliberately mixes arithmetic, a negative decimal, a year-like form and an ordinal:

"8+9 is 17. Temperature is -23.5 degrees.
In the 1930s, 10th place was okay."

Useful test families for this branch include round-trip encode/decode, notation-equivalence pairs, sign and suffix edge cases, scientific notation, numeric-mask leakage, gate behaviour, shape checks for every head, causal-mask checks, and reversed association prompts such as fact → year versus year → fact.

Hugging Face exports · where custom ideas stop being cute

A large part of the learning curve was not the forward pass itself but packaging a custom configuration, tokenizer behaviour, extra numeric tensors and custom model outputs so that they behaved predictably outside the local training script. Early exports were inconsistent: a model can train locally and still fail the practical save → load → tokenize → forward → generate path expected by Hugging Face tooling.

The lesson from PolMATH was blunt: a custom architecture is not finished when loss.backward() works. Serialization, configuration fields, tokenizer special tokens, output schemas and reload tests are part of the architecture too.

Observed training behaviour

The numerical channel became usable rather than decorative.

The training pipeline reached the intended qualitative behaviours: matching number words to numeric values and back, deciding when a numerical position should appear despite ordinary text tokens dominating the corpus, and learning useful associations between values and facts.

One of the useful examples was the relation between 1939 and the outbreak of the Second World War. The interesting part was not only producing the number after the fact, but also retaining the association when the direction of the prompt was reversed.

Different written forms such as 1939, 1939.0 and 1.939e3 could be learned as related expressions of the same value rather than as unrelated text fragments.

Important limitation

PolMATH was not a general-purpose mathematical reasoner. Its value was narrower: it demonstrated that an explicit numerical path could coexist with ordinary language modelling and learn useful number-related behaviour.

Roadmap

The original research branch is complete.

2024

Initial PolMATH

Custom architecture and structured numerical representation experiments.

2024–25

Training and data experiments

Numeric placement, notation equivalence, routing, generation and association tests.

Completed

Original branch finished

The Polish experimental branch is treated as complete rather than as a checkpoint waiting for more training.

Publication-oriented continuation

A cleaner experimental protocol and an English-language iteration are planned so the idea can be evaluated beyond a Polish-only setting.

ORIS 660M / Qwen 0.8B

Structural compression and recovery

ORIS 660M / Qwen 0.8B

A pair of practical model-training experiments. ORIS 660M is the structural-compression and recovery branch described below; the earlier Qwen 0.8B track focused on changing the tokenizer and continuing training while freezing part of the pretrained model.

660.13Mparameters

32 → 12transformer blocks

621.984Mbenchmarked recovery labels

Abandonedbranch status

07 Aug 2026

ORIS 660M beginsDepth-reduction experiment from Bielik-1.5B-v3.

Recovery

Selected-vs-uniform architecture testsLayer identity proved important to recovery trajectory.

Later work

Knowledge and data-control experimentsInherited-vs-random controls, Polish diagnostics and tighter data-control experiments.

2025 · Qwen 0.8B track

Change the tokenizer, freeze part of the model, then see what actually adapts.

The Qwen 0.8B experiment was a more conventional pretrained-model surgery than ORIS 660M. The working test replaced the tokenizer and continued training while keeping part of the inherited weights frozen. The practical question was how much of a pretrained model can be preserved when the interface between raw text and embeddings changes.

This branch was useful less as a final model and more as training practice: vocabulary replacement, embedding adaptation, deciding which modules should remain frozen, checkpoint compatibility and observing where transfer breaks when the tokenizer no longer matches the one used during pretraining.

Scope of the note

The uploaded material documents the ORIS 660M branch in detail; the tokenizer-replacement / partial-freezing Qwen description here follows the project history supplied by the author, not a benchmark table from the provided files.

Origin

Not a conventionally scaled-down 660M transformer.

The lineage is Qwen 2.5 → Bielik-1.5B-v3 → ORIS 660M. The starting teacher had 32 transformer blocks. ORIS retained only 12 while preserving the teacher's hidden width, attention geometry, tokenizer, embeddings, feed-forward dimensions and LM head.

The selected blocks were 0, 1, 2, 3, 4, 21, 23, 24, 25, 29, 30 and 31. They were not chosen by simply keeping every second or third layer. Small ablations and calibration tests were used to identify less-sensitive regions and compare candidate 9–12 layer structures.

The extreme jump from teacher layer 4 to layer 21 means that downstream blocks receive representations unlike those they originally saw during pretraining. That mismatch is one of the central features of the experiment.

Step 0

The student became faster and lighter — and language modelling collapsed.

Model	Parameters	Loss	Perplexity	Throughput	Peak VRAM
ORIS 660M step0	660M	7.1613	1288.58	17.29k tok/s	1.54 GiB
Bielik-1.5B	1.596B	2.2692	9.67	6.69k tok/s	3.29 GiB

Fixed evaluation on 128 sequences of length 1024.

The point was not pruning without damage.

The network was strongly damaged. The experiment became: can an inherited but structurally broken network reorganize itself through ordinary next-token training?

Architecture selection

The identity of retained layers materially changed recovery.

Architecture	Initial validation loss	After ~2.03M training tokens
Selected 12-layer ORIS	7.2737	4.3134
Uniform 12-layer control	8.9560	5.4880
Selected 10-layer candidate	—	~4.9100

These experiments were intentionally small and do not establish that the chosen structure is globally optimal. They do show that equal depth does not imply equal recoverability.

Main recovery used standard causal language modelling. Teacher/student KL was useful for screening candidate structures, but teacher logits were not the training objective during the main continuation run.

589.248Mearlier recovery labels

32.736MV2.5A continuation labels

621.984Mcumulative benchmarked labels

~1×roughly one recovery label per student parameter

Generation behaviour

Loss recovered faster than stable generation.

Continued training restored grammatical Polish, reasonable local continuations and recognizable semantic associations, but free generation remained unstable.

Observed issues included greedy repetition loops, topic drift, weak long-range coherence, hallucinated dates and numerical sequences, web/forum residue, metadata-like fragments, premature EOS at some checkpoints and high sensitivity to sampling strategy.

One of the strongest lessons was that lower loss did not translate monotonically into better generation. Syntax, factual accessibility, calibration, semantic control and free-generation stability could improve at different rates.

V2.5A benchmark snapshot

A recovery snapshot, not a leaderboard claim.

Task	ORIS V2.5A
Belebele accuracy	0.2256
8tags accuracy	0.1757
PoLeMo2 in accuracy	0.4155
PoLeMo2 out accuracy	0.3684
DYK binary F1	0.1268
PSC binary F1	0.1627
PPC accuracy	0.4180
CBD macro F1	0.1291
KLEJ NER accuracy	0.1025
PolQA reranking	0.5157

Later diagnostics

Competitive signals and very obvious failure modes appeared together.

Project-specific MC / continuation probe

Model	Params	Raw MC	Normalized MC	Continuation NLL
ORIS 660M	660M	0.750	0.500	3.3525
Qra-1b	~1.10B	0.917	0.583	1.5715

Qra remained clearly stronger; the comparison tested whether ORIS had returned to a meaningful operating regime.

SpeakLeash polish_mc diagnostic — 0-shot, 100 examples per task

Model	Parameters	Accuracy	Normalized accuracy	F1
ORIS 660M	660M	0.399	0.386	0.0116
APT3-1B-Base	~1B	0.330	0.300	0.3215

Task	ORIS	APT3
PoLeMo2 in accuracy	0.43	0.34
PoLeMo2 out accuracy	0.33	0.38
8tags accuracy	0.12	0.20
Belebele accuracy	0.23	0.25
KLEJ NER accuracy	0.31	0.05
PolQA reranking accuracy	0.62	0.39
PPC accuracy	0.44	0.42
PSC accuracy	0.68	0.32

The shape of the errors mattered more than the aggregate.

ORIS could show surprisingly high accuracy on some tasks while binary F1 collapsed because of severe class bias. That is exactly why these results are diagnostic rather than a claim of general superiority over APT3.

Knowledge recovery

Was knowledge destroyed, degraded, inaccessible — or relearned?

An inherited 12-layer ORIS student was compared against an architecture-matched model initialized from scratch. Both had comparable structural capacity; only one inherited the teacher's pretrained parameter structure.

24.7 → 35.0%inherited fact-level majority accuracy

0.154 → ~0.366teacher-ranking Spearman

~10%random control accuracy

-0.109random control Spearman

ENTITY_ONLY subset	Before	After
Accuracy	20.0%	41.5%
Mean factual margin	-0.840	+0.120

The result does not prove that complete symbolic facts remain intact inside individual weights. It does suggest that the inherited network begins recovery from a qualitatively different state than a random model with the same architecture.

A small generation probe captured the ambiguity: “Polska jest…” remained approximately correct, while “Kopernik był…” produced “Kopernik był w kosmosie.” The statement is false, but the astronomy/space semantic neighborhood survived.

Case 1

Accessible

Knowledge survives and remains directly usable.

Case 2

Degraded

The semantic region survives while precise retrieval fails.

Case 3

Latent

Useful structure may survive but no longer be reachable through the altered path.

Case 4

Relearned

The information genuinely has to be reacquired during continued training.

Data research

The bottleneck moved from model architecture to controlling what the recovering model sees.

Later generations repeatedly exposed artifacts from web-derived training data: forum fragments, metadata, SEO text, navigation elements, date-heavy sequences, list structures, transcripts, document templates and poorly separated topic boundaries.

The newer pipeline therefore separates linguistic quality, structural quality, corruption, noise, difficulty, domain, information density and knowledge value rather than compressing all of them into one score.

A difficult scientific or legal document can have high perplexity and many rare tokens while still carrying high knowledge value. A fluent generic paragraph can look clean while contributing almost none.

Canonical ingest

Stable IDs, provenance, validation, sharding and raw preservation.

Repair & structure

Encoding repair, paragraph preservation, segmentation and conservative local deduplication.

Signal extraction

Language confidence, coherence, repetition, information density, rare-token statistics and perplexity.

Multi-axis classification

Separate estimates for quality, noise, knowledge value and reconstruction decisions.

Knowledge structure

Domain hierarchies, subdomains and knowledge flags.

Corpus curation

Global exact/fuzzy deduplication, redundancy control and later curriculum balancing.

Data direction

Data, data and data again.

A model is, to a large extent, what you feed it. At small scale this becomes especially visible: every repeated pattern, low-value fragment and badly balanced domain competes for a limited parameter budget with the information you actually want the model to retain.

The current focus is therefore less on pushing arbitrary token counts and more on controlling what enters the corpus, how often it appears and what kind of knowledge each sequence is teaching.

Why the branch was stopped

Because simply feeding the checkpoint more web text would answer the least interesting questions.

ORIS is an independent, self-funded project developed primarily on privately owned compute. A substantial portion of pruning, validation, training and evaluation was performed locally, including on an RTX 5060 Ti.

The existing experiments suggest that a severely depth-reduced Polish model can recover useful language-model behaviour. The harder questions are now what exactly survives structural reduction, what is genuinely relearned, whether inherited knowledge can be distinguished from reacquired knowledge, and whether recovery can be controlled through better data selection, ordering and curriculum design.

Stopped does not mean uninformative.

The experiment clarified the next bottleneck. The branch itself is no longer being continued; its useful results were folded into later work on data infrastructure, evaluation methodology and smaller-model research.

ORIS BERT

Compact Polish encoder

ORIS BERT

ORIS BERT is the continuation of the model originally developed under the working name ORIS Bert Small C. It uses BERT-style masked-language pretraining and is treated primarily as a compact encoder / embedding model, not a stock scaled-down BERT.

25.41Mparameters

6layers

128KPolish-oriented BPE

8.00Binput pretraining tokens

16 Aug 2026

Small C checkpoint completedFrom-scratch 8B-token MLM pretraining completed.

Downstream

KLEJ-style and document-filtering testsEstablished both the model's limitations and its practical pipeline advantage.

Efficiency

Local encoder benchmarkingMeasured strong throughput and VRAM advantages on RTX 5060 Ti / Blackwell-oriented workloads.

Why this model exists

A production bottleneck accidentally became an architecture project.

Dataset preparation for ORIS 660M became a throughput bottleneck, so a smaller encoder was built specifically for local filtering, categorization and scoring on NVIDIA Blackwell hardware.

What began as an internal pipeline tool became a compact Polish encoder intended primarily as a backbone for task-specific fine-tuning.

Not a sentence-embedding model out of the box.

Raw mean-pooled embeddings are strongly anisotropic and are not recommended for zero-shot semantic search without additional contrastive or task-specific fine-tuning.

Architecture

The “BERT” part describes the training style more than the architecture.

Property	Value
Model type	Custom Transformer encoder
Hidden size	384
Token embedding size	128
Attention heads	6
Context length	1024
Attention layout	256, 256, 1024, 256, 256, 256
Normalization	RMSNorm
Objective	Masked Language Modeling
Initialization	From random initialization

Five of six layers use local 256-token attention. The third layer performs full 1024-token attention and acts as the main global mixing layer.

The vocabulary is intentionally large, but the token embedding dimension is only 128 while the encoder hidden size is 384. This factorization keeps the 128K vocabulary from dominating the parameter budget.

During MLM pretraining, vocabulary logits are calculated only for masked positions. Those hidden states are projected from 384 dimensions back into the 128-dimensional token space and scored using the tied token-embedding matrix.

Pretraining

8.00B input tokens from scratch on a single local GPU.

Setting	Value
Sequence length	1024
Micro-batch size	16
Gradient accumulation	8
Tokens / optimizer update	131,072
Optimizer updates	61,036
Peak learning rate	3e-4
Mask probability	15%
Precision	BF16 autocast
GPU	RTX 5060 Ti 16GB
Training time	~13.44 h
Average throughput	~165.4K input tok/s

Training data was primarily Polish MADLAD with an auxiliary mixture containing Wikipedia, OpenSubtitles PL, balanced NKJP and Polish legal/judicial text. The auxiliary pool represented roughly 15% of generated training sequences.

The tokenizer is a custom 128K BPE with NFKC normalization and Metaspace pre-tokenization. It is Polish-oriented but includes a broad Unicode alphabet for noisy web text.

Sentence-space limitation

The raw encoder space is highly anisotropic under mean pooling.

Model	Mean cosine for unrelated texts
ORIS BERT	~0.987
HerBERT	~0.90
PolDense	~0.26

Internal diagnostic only; not a general encoder-quality benchmark.

Polish downstream benchmarks

Competitive on several tasks, clearly weaker on others — at about one quarter of PolBERTa's size.

Task	Metric	ORIS BERT	PolBERTa base
NKJP-NER	Macro-F1	75.52	84.36
CDSC-E	Accuracy	91.30	91.00
CDSC-R	Spearman	88.18	88.97
CBD	F1(+)	50.24	43.75
PolEmo2.0-IN	Accuracy	83.33	85.32
PolEmo2.0-OUT	Accuracy	65.59	63.77
DYK	F1(+)	37.86	46.31
PSC	Macro-F1	57.28	85.87
AR	MAE ↓	0.5929	0.5753

Local evaluation using the same fixed procedure for ORIS BERT and PolBERTa base; not an official KLEJ leaderboard submission.

Document filtering

The workload it was originally built for is where the design makes the most sense.

Metric	mmBERT-base	ORIS BERT
Decision Macro-F1	0.4334	0.5015
Decision accuracy	0.5185	0.6296
Training time	583.2 s	124.4 s
Peak VRAM	5.83 GiB	0.52 GiB

Production-style full pipeline

Metric	mmBERT-base	ORIS BERT
Full pipeline time	21.732 s	4.408 s
Documents / second	11.78	58.07
Mean latency / document	84.89 ms	17.22 ms
Peak VRAM	1.806 GiB	0.252 GiB

~4.93× more documents per second in this workload.

This is a task-specific pipeline result, not a claim that Small C is universally superior to larger encoders.

Encoder efficiency

The compact architecture also showed a large raw forward-pass advantage.

Setting	ORIS BERT	mmBERT-small	Advantage
Batch 1, 128 tokens	3.828 ms	17.153 ms	4.48×
Batch 1, 1024 tokens	3.940 ms	17.229 ms	4.37×
Batch 8, 1024 tokens	1.25M tok/s	159K tok/s	7.85×

Different tokenizers mean cross-model tokens/s should be interpreted carefully.

Roadmap and limitations

Small C is useful precisely because its limits are explicit.

Raw mean-pooled sentence representations have poor cosine-space separation. Retrieval, ranking and sentence similarity need additional fine-tuning. Larger Polish encoders remain stronger on several benchmarks, and five local-attention layers mean cross-window communication relies heavily on one global mixing layer.

The model is therefore best understood as a compact MLM-pretrained backbone for supervised tasks, filtering and feature extraction — not as a universal zero-shot embedding model.

Origin

ORIS 660M pipeline bottleneck

Built to make local filtering and scoring fast enough to matter.

16 Aug 2026

Small C checkpoint completed

8.00B-token from-scratch pretraining completed.

Current

Gated research release

Used as a practical encoder backbone while downstream and representation-space behaviour are evaluated.

Possible next

Broader ORIS encoder line

Further architecture, training and evaluation work may turn the concept-stage checkpoint into a fuller encoder family.

Vyuhu

Abandoned architecture research

Vyuhu

One shot, one compute, many possibilities. One jointly trained model exposes four deterministic compute profiles and can later be physically reduced into smaller standalone models without changing what a given profile computes.

4 + 1four compute profiles + the shared supernetwork

0.834Btokens in the completed architecture-validation run

Δlogit = 0exact profile-vs-extracted-model equivalence

~11.8k tok/s~500M smoke / training-path throughput on RTX 5060 Ti 16 GB

validated

0.834B-token architecture runCompute ordering, profile separation and exact physical extraction all passed the intended checks.

cancelled

~280M family targetThe planned full family run was not continued after the architecture-validation phase.

smoke

~500M scale testThe architecture and training path were also instantiated at ~501.8M parameters and sustained roughly 11.8k tok/s in the current optimized path.

The idea

Not MoE. Not token routing. One deterministic family inside one trained network.

Vyuhu does not learn a router that decides which experts receive each token. A profile is selected before the forward pass. Mandatory anchor blocks are always present; optional heavy compute is executed according to a fixed profile schedule. Cheap transfer paths and low-rank controllers keep representations compatible when compute is skipped.

The result is one training lineage with several explicit operating points: Vasudeva, Sankarshana, Pradyumna and Aniruddha. The fifth object in “4 in one? Nah, 5.” is the complete shared supernetwork from which those paths come.

Vasudeva

Full path

The largest deterministic path. Nothing optional is withheld.

Sankarshana

Middle-high compute

A reduced path intended to land near the ~175M class in the first ~280M family run.

Pradyumna

Middle-low compute

A smaller operating point targeted near the ~125M class.

Aniruddha

Anchor-first minimum

The smallest deterministic path, targeted near ~100M in the first full family run.

One shot, one compute, many possibilities.

Train the shared system once. Select a deterministic compute level at inference — or extract that level into its own standalone model later.

Architecture validation

The 0.834B-token run passed every architectural check it was designed to answer.

The validation run was not treated as a final quality leaderboard. Its purpose was narrower and more important: determine whether one jointly trained network could preserve a stable compute→quality ordering across deterministic profiles, and whether a selected profile could be physically removed from the supernetwork without silently changing its function.

At roughly 0.832B evaluated tokens, quality remained monotonic with compute: the full Vasudeva path was best, followed by Sankarshana, Pradyumna and Aniruddha. No profile inversion appeared in the final evaluation.

Profile	Schedule	Physical extracted size	Eval loss	Perplexity
Vasudeva	[4, 4, 4]	125.721M	3.2705	26.32
Sankarshana	[1, 2, 1]	75.602M	3.3174	27.59
Pradyumna	[0, 1, 0]	56.723M	3.4138	30.38
Aniruddha	[0, 0, 0]	50.430M	3.5446	34.63

V > S > P > Amonotonic compute-quality ordering

4 / 4profiles physically extracted

0.0maximum profile-vs-extracted logit delta in exact extraction test

100%of the targeted architecture checks passed in this run

“Extraction” is not another approximation step.

The extracted model reproduces the selected profile exactly: Δlogit = 0 in the extraction test. A smaller extracted artifact can therefore be deployed without an additional quality loss caused by the act of removing inactive structure. The profiles themselves still have different quality because they intentionally use different amounts of compute.

First full run

A ~280M supernetwork that behaves like a small Polish model family.

The next target is deliberately smaller than the 500M engineering scale test. Width is reduced while the proven four-profile topology is retained, with the middle profiles raised by one heavy block so the family lands near four useful deployment classes rather than clustering too tightly at the bottom.

Profile	Planned schedule	Approx. family class	Role
Vasudeva	[4, 4, 4]	~275–285M	full model
Sankarshana	[1, 3, 1]	~175M	balanced middle-high path
Pradyumna	[0, 2, 0]	~125M	compact middle-low path
Aniruddha	[0, 0, 0]	~100M	minimum anchor path

Sizes are current design targets for the first ~280M run, not yet final checkpoint counts.

Tokenizer

VYUHU32k

32K corpus-specific byte-level tokenizer prepared for the Polish training corpus and exact Unicode round-trip behaviour.

Hidden width

1152

Hardware-friendly width with 64-dimensional attention heads.

Attention

GQA anchors

18 query heads / 6 KV heads in the current ~280M design. Optional heavy blocks use the elastic mixer rather than full global attention.

FFN

3584 · SwiGLU

Dense SwiGLU channel capacity retained across anchor and elastic compute.

Optimizer

Muon + fused AdamW

Hybrid optimizer path: Muon for selected 2D hidden matrices, fused AdamW for the remaining parameter groups.

Some internals

:)

Controller/bypass details, training mixture and a few run-level knobs stay intentionally unpublished until the experiment is complete.

Scale reference

The first Vyuhu run sits in the same broad size class as a conventional ~275M MHA-style Polish LM — but spends parameters differently.

A useful conventional reference is the published ~275M configuration shown below. It is not presented as a controlled Vyuhu benchmark: the architectures, optimizer path, tokenizer and training data differ. The point is simply to make the scale tangible before the first ~280M Vyuhu run exists.

Hyperparameter	Conventional ~275M reference	Vyuhu ~280M target
Model parameters	275M	~275–285M supernetwork target
Sequence length	1024	1024
Vocabulary	31,980	32,000 · VYUHU32k
Transformer layers	32	4 mandatory GQA anchors + elastic staged compute
Attention heads	16 MHA	18 Q / 6 KV in GQA anchors
Head dimension	64	64
Model width	768	1152
Intermediate size	2048	3584
Positional encoding	RoPE	RoPE
Activation	SwiGLU	SwiGLU
Normalization	RMSNorm · ε 1e-6	RMSNorm · ε 1e-6
Dropout / bias	0.0 / no	0.0 / no
Optimizer	AdamW	Muon + fused AdamW
Exact MB / GA / LR schedule	13 / 40 / 4e-4 → 2e-5	:)

Different objective, not just a strange way to build another 280M model.

A conventional dense 275M checkpoint is one operating point. Vyuhu is being designed so a single training run can yield a ~280M full path and useful ~175M, ~125M and ~100M deterministic descendants — with exact extraction rather than post-hoc pruning.

Scale smoke test

The design was also instantiated at ~501.8M parameters before committing to the smaller final run.

The ~500M branch was used as an engineering stress test for memory, optimizer geometry, profiling and custom-kernel work on a single RTX 5060 Ti 16 GB. The current training path reached roughly 11.8k tokens/s at sequence length 1024 with MB 8 / GA 16.

This is a throughput smoke test, not a quality claim for a finished 500M checkpoint. It demonstrated that the architecture, hybrid optimizer and current runtime can execute at this scale on the target desktop GPU.

501.83Mengineering-scale parameter count

1024sequence length

MB 8 / GA 16measured training geometry

~11.8k tok/soptimized full training path

The riddle survived

4 in one? Nah, 5.

Vyuhu

What are you?

Vasudeva: The form that answers when nothing needs to be left behind.

Sankarshana: The same answer, after deciding some of itself can remain silent.

Pradyumna: What appears when less is allowed to be enough.

Aniruddha: The version that still reaches the answer after forgetting how much it was supposed to carry.

Vyuhu

Which one of you is Vyuhu?

Vasudeva: I.

Sankarshana: I.

Pradyumna: I.

Aniruddha: I.

Supernetwork: You trained me once.

Vyuhu · architecture-validation work retained as documentation · full branch abandoned

Koliber 1.0

Completed Polish causal LM baseline

Koliber 1.0

The first stable Koliber baseline: a small decoder-only Polish language model trained from scratch and later used as the control point for architecture, context-length and data-quality experiments.

~126Mparameters

12decoder layers

1536training context

32Kvocabulary

Role in the research line

A baseline that made the next questions measurable.

Koliber 1.0 uses an intentionally conventional causal-LM core: 768 hidden width, 12 layers, 12 query heads, 2 KV heads, SwiGLU, RMSNorm, RoPE and tied embeddings. Its value now is less as a moving target and more as a stable reference for later changes.

Longer free generation exposed a recurring failure mode: the model could begin on-topic and then gradually drift, paraphrase itself or fall into locally plausible but globally weaker continuations. That observation motivated the context-representation experiments in Koliber 1.1.

Koliber 1.1

Current small-LM research

Koliber 1.1

Koliber 1.1 is an experimental continuation of Koliber 1.0. The current work is less about making the model larger and more about changing how a very small language model behaves when its own token distribution starts drifting toward patterns that were learned too strongly during pretraining.

~126M class continued training distribution correction experimental

Why 1.1 exists

Koliber 1.0 often knows the direction. The problem is what happens after that.

Koliber 1.0 is already a useful baseline for its size. In many raw completions it starts in a plausible semantic direction and can produce a coherent first sentence or short paragraph. The more visible weakness appears later: probability can become concentrated around locally familiar continuations that are only weakly related to the text that came before.

In practice this can look like a normal continuation suddenly turning into a page title, a ranking, a navigation fragment, an SEO transition, a scraped-document boundary or another pattern that appeared often enough in web data to become disproportionately attractive to a small model.

This is not the same as saying that Koliber 1.0 has no representation of the subject. The model can be moving in roughly the right semantic direction and still make a poor local decision a few tokens later. Once that decision changes the style or document mode, subsequent next-token predictions reinforce the new path and the continuation can drift quickly.

A small model has a real capacity ceiling.

The goal is not to pretend that a model around 126M parameters can be turned into a 126B model that remembers half of the internet. At this scale, repeated templates, boilerplate and accidental web regularities can consume meaningful capacity. The more practical question is whether some of those behaviours can be made less attractive without destroying the useful language knowledge the model already has.

Observed in Koliber 1.0

Coherent beginnings can collapse into memorized web structure.

The examples below are raw Koliber 1.0 Base continuations. They are intentionally left uncorrected because the failure itself is the useful part of the example.

Example 1 · news-like continuation → unrelated site residue

Prompt

Wczoraj wieczorem mieszkańcy miasta zauważyli, że

Koliber v1.0 Base

Wczoraj wieczorem mieszkańcy miasta zauważyli, że przy ulicy 3 Maja w tym rejonie znajduje się sklep spożywczy. Ulicami tej ulicy przejeżdżało kilka samochodów i zjeżdżały tam samochody.

W ciągu kilku godzin na ulicy 3 Maja zostały uszkodzone szyby samochodów. Wypadek miał miejsce na rondzie oraz ulicy 3 Maja - informuje policja.Humor: The Last of Us - film.gildia.pl - serial, newsy, recenzje

Zjawa: W tle pogłoski, o

Example 2 · city topic → list / SEO-like transition

Prompt

Największym problemem współczesnych miast jest

Koliber v1.0 Base

Największym problemem współczesnych miast jest ich bezpieczeństwo. Według statystyk w Polsce z powodu wypadków drogowych i kolizji drogowych zginęło co najmniej 12 osób, a na 100 z nich zginęły cztery osoby.TOP 10 najlepszych blogerów, którzy czytają książki: Zdradza (nie)zabawki

Zdradza (nie)zabawki

"Zdradza", czyli "Zdrada" to najnowsza edycja bestsellera autorki. I nie chodzi tu o takie banały jak "Wymysły

Example 3 · plausible scientific lead → invented local narrative

Prompt

Naukowcy odkryli nową metodę, która

Koliber v1.0 Base

Naukowcy odkryli nową metodę, która pozwoli na zmniejszenie strat ciepła w budynkach.

W praktyce, nie tylko na zewnątrz budynku, ale i na zewnątrz, to właśnie dzięki wykorzystaniu światła słonecznego. Dzięki temu nasze domy są bardziej ekologiczne, mniej zużywane przez wiele lat – powiedział Daniel Knopfman, dyrektor generalny firmy Reverend Systems.

O tym, że zastosowanie takich technologii może przyczynić się do poprawy wydajności energetycznej budynków, można dowiedzieć się z najnowszej konferencji „Ultrasolar Foundation

The complete Koliber 1.0 model card, architecture, loading instructions and additional raw generation examples are available on Hugging Face · OrisTeam/Koliber-v1.0-Base.

What is being tested

Correct the distribution rather than simply adding more tokens.

The current 1.1 experiments keep ordinary next-token training as the base objective, but add an external reference signal during continued training. That reference comes from a teacher outside the ORIS model family.

The teacher is not intended to replace Koliber's own representation and is not part of the inference model. It is used as a second opinion when the local probability distribution becomes strongly attracted to continuations that the external model does not support in the same way.

The working idea is deliberately narrower than full model distillation. Koliber should remain Koliber: its existing language knowledge and its own probability distribution are still the starting point. The experiment asks whether selected parts of that distribution can be pushed away from overlearned or low-value local behaviours while preserving ordinary language modelling.

The exact teacher, mapping strategy, correction schedule and loss construction are not being published during the preview stage. Those details may also change as the experiment develops.

This is a behaviour-correction experiment, not a claim of transferring a much larger model into 126M parameters.

If it works, the useful result would be modest but practical: fewer abrupt transitions into boilerplate or web residue, less confidence in some memorized local patterns, and more probability left for continuations that remain compatible with the preceding text.

Data work

The corpus is treated as an iterative filtering problem.

The broader project is also moving away from a one-pass view of dataset cleaning. Web residue, repeated structures and ambiguous examples are repeatedly sampled, reviewed and fed back into the filtering process.

Larger language models, compact models and manually reviewed examples are being used as complementary signals. The exact production filtering stack is still changing, so the preview does not describe a fixed final pipeline.

The purpose is not to manufacture a perfectly clean corpus — that is unrealistic — but to reduce the amount of model capacity spent on patterns that are frequent because of how webpages are built rather than because they are useful language knowledge.

Current status

A controlled experiment, not a finished release.

Koliber 1.1 is still in active testing. Current runs are being compared against the released Koliber 1.0 Base on fixed language-model evaluation data and downstream Polish benchmarks.

The main question is whether changes in continuation behaviour correspond to measurable improvements rather than simply producing a different writing style. A useful result must preserve the strengths of the base model while reducing some of its recurring failure modes.

Not every mechanism tested under the 1.1 name will necessarily appear in the final release.

Reference model: OrisTeam/Koliber-v1.0-Base.

Vidar-VL

Planned vision-language research

Vidar-VL

A planned ORIS vision-language model focused on understanding images and interacting with visual content in Polish.

image understanding visual question answering OCR / documents Polish visual interaction multimodal reasoning

Status

Planned and currently under development.

Vidar-VL is intended as the ORIS vision-language branch for image understanding, visual question answering, OCR-oriented interaction and broader multimodal reasoning.

Further technical details will be published when the project reaches its first experimental release.

ORIS Vision

Planned visual representation research

ORIS Vision

A planned compact visual embedding model — the visual counterpart to ORIS BERT — for similarity, retrieval, scoring, analysis and filtering.

visual embeddings image scoring similarity retrieval quality filtering

Status

The visual half of the compact embedding-model pair.

ORIS Vision is intended as the visual counterpart to ORIS BERT: both are compact encoder-style models aimed at representations rather than conversational generation. Planned uses include image embeddings, similarity, retrieval, scoring and image filtering.

Architecture, training sources and implementation details are intentionally not disclosed at this stage.