NSRL Diagnostic Evals

HEADLINE

NSRL-MME v0 Multimodal LLM Eval

failed

Headline score: 37.1%
Target: 70.0%
MME artifact: present
Quality report: present
Objective coverage: present

Sampler, replay, browser-probe, and memory-assisted sample metrics are diagnostics only; they do not define the headline score.

text prompt -> symbolic image plan

43.9%

432 rows / confidence_trace.directional_native_eval.groups.text_prompt_to_image_plan.stats

seal image -> identity / attributes / source text

43.4%

720 rows / confidence_trace.directional_native_eval.groups.seal_image_to_text.stats

text + seal -> grounded explanation / match

37.1%

576 rows / confidence_trace.directional_native_eval.groups.text_and_seal_to_explanation.stats

prompt/name -> identity / source binding

37.5%

720 rows / confidence_trace.directional_native_eval.groups.identity_source_binding.stats

match / no-match hard-negative agreement

100.0%

313 rows / confidence_trace.cross_modal_agreement

green quality report with confidence trace

not green

Required gate for passing NSRL-MME v0.

local objective coverage proof

green

Required gate for passing NSRL-MME v0.

source-grounded text/image evidence

green

Required gate for passing NSRL-MME v0.

held-out generated output integrity

not green

Required gate for passing NSRL-MME v0.

SCOREBOARD

Models On Our Evals

4 model paths

NSRLLMM1

attention joint text/image-token model

probe-gated + raw control

Current browser artifact path. The probe suite exercises prompt-scoped text, embedded image memory, raw name logits, and source-specific body-start logits. The raw no-memory row is a free-running negative control and should not be read as browser-path quality.

Artifact probe suite: 72/72
Constrained name-logit top-1: 72/72
Constrained body-start top-1: 72/72
Raw no-memory name match: 0.0%
Raw text quality score: 27/100
Raw scaffold outputs: 76.3%

scripts/check-solomon-attention-web-quality.mjs --all-names --summary
scripts/probe-solomon-attention-raw-rank.mjs --all-names --summary
scripts/probe-solomon-attention-body-start-rank.mjs --summary
docs/solomon-attention-eval.tsv

NSRLLAT1

prompt-to-layout latent prior

published evals

Prompt/layout prior measured on checked-in train/eval/gold prompt partitions and the clean 72 held-out generation prompt set. The clean-72 row is the fast planner target to improve before rerunning bitmap generation.

Clean-72 planner top-1: 4.1%
Clean-72 planner top-5: 8.3%
Best eval top-1: 20.0%
Best eval top-5: 46.2%
Best gold top-1: 25.0%
Best shape-probe eval top-1: 23.7%

docs/solomon-latent-planner-eval.tsv
docs/solomon-eval-scaling-curve.tsv
docs/solomon-text-feature-shape-probe.tsv

NSRLTCH

text-conditioned bitmap denoiser

published evals

The denoiser artifact is scored on learned latent-conditioned generations. The oracle-conditioning diagnostic is an upper bound only: it gives the denoiser the true 16x16 target plan to separate planner failure from renderer/retrieval failure.

Published artifact: present
Best generated top-1: 4.1%
Best generated top-5: 4.1%
Best latent top-1: 8.3%
Oracle-plan signature top-1: 98.6%
Oracle-plan retrieval top-1: 18.0%
Best target distance: 5,849,621

web/assets/solomon-model.nsrltch
docs/solomon-generative-eval.tsv
docs/solomon-oracle-condition-diagnostic.tsv

NSRLMOD1

coarse joint text/image-token model

published evals

The multimodal fallback artifact is scored on tracked corpus replay: next-token ranks over prompt bytes, generated text bytes, marker tokens, and 16x16 image bins. This is artifact-native replay quality, not broad free-running quality.

Published artifact: present
Overall replay top-1: 99.4%
Text replay top-1: 99.6%
Image-token top-1: 100.0%
Exact examples: 0.3%

web/assets/solomon-multimodal.nsrlmod
docs/solomon-multimodal-eval.tsv
scripts/run-solomon-multimodal-eval.mjs

SAMPLES

Representative Eval Panels

5 panels

NSRLLAT1

Latent Prior Routing

Best checked-in routing row

Prompt-to-layout routing is scored on held-out Solomon prompt partitions. This panel shows the strongest eval top-1 row now published.

Eval top-1: 20.0%
Eval top-5: 46.2%
Gold top-5: 27.7%

docs/solomon-eval-scaling-curve.tsv
576 prompt rows / latent 32 / text 512 / 0ce31085

NSRLTCH

Generated Seal Retrieval

Class-head generation eval

Generated bitmap seals are checked against target retrieval. The image panel is the tracked visual sample; the numbers come from the generative eval table.

Generated top-1: 4.1%
Generated top-5: 4.1%
Latent top-1: 8.3%

docs/solomon-generative-eval.tsv
docs/assets/solomon-text-conditioned-seals.png

NSRLTCH

Oracle Conditioning Split

Diagnostic upper bound; not headline evidence

The denoiser receives the true held-out 16x16 target plan here. This isolates the current failure: the learned planner is the first blocker, while rendered-image retrieval remains a separate weak gate.

Signature top-1: 98.6%
Signature top-5: 100.0%
Retrieval top-1: 18.0%
Retrieval top-5: 52.7%

docs/solomon-oracle-condition-diagnostic.tsv
scripts/run-solomon-oracle-condition-diagnostic.mjs

NSRLMOD1

Text/Image Token Replay

Tracked corpus replay

The multimodal artifact is scored on next-token replay over prompt bytes, generated text bytes, marker tokens, and 16x16 image bins.

Overall top-1: 99.4%
Text top-1: 99.6%
Image top-1: 100.0%
Context hit: 100.0%

docs/solomon-multimodal-eval.tsv
deployed / 313 examples / 0x8057e681de909abc

NSRLLMM1

Probe-Gated Artifact, Raw Control

Browser probes pass; raw no-memory control fails name binding

The browser artifact path is probe-gated separately from native free-running sampling with memory and conditioning disabled.

seal of Bael

---He is of the Goetia and teacheth with his ART in LINE.

Artifact prompts: 72/72
Name-logit top-1: 72/72
Body-start top-1: 72/72
Raw name match: 0.0%
Raw text score: 27/100

docs/solomon-attention-eval.tsv
scripts/check-solomon-attention-web-quality.mjs --all-names --summary

Solomon selects Murmur: and appeareth in the Form of a Warrior riding upon a Gryphon, with a Ducal Crown upon his Head.

Prompt kind: held-out
Text source: lm strict
Image source: image memory

trace

EVIDENCE

Published Eval Inputs

31 table rows

Headline NSRL-MME v0

37.1%

Status: failed. Floor score across model-native multimodal task families.

Honesty guard

passed

CI fails when claimed model coverage lacks checked-in rows, probes, assets, or samples.

Prompt sample gallery

6 rows

Fixed NSRLLMM1 prompt samples with copied PNGs and a TSV trace.

Attention artifact probes

3/3 passed

Prompt/text/image checks run against the browser artifact.

Attention raw eval table

1 rows

NSRLLMM1 no-memory native sampling control row.

Latent prior tables

19 rows

Prompt routing, shape, and text-feature sweeps.

Generative eval table

2 rows

Held-out generated bitmap eval rows, when checked in.

Multimodal eval table

1 rows

NSRLMOD1 prompt/text/image-token replay rows.

ATTENTION ARTIFACT

Probe Details

3/3 passed

passed

Browser artifact quality

Prompts: 72
All Names: true

passed

Raw prompt-name rank

Prompts: 72
Ranked: 72
Top 1: 72
Top 5: 72
Top 10: 72
Median Rank: 1
Worst Rank: 1
Median Margin Q8: 4064
Worst Margin Q8: 4064

passed

Body-start rank

Prompts: 72
Ranked: 72
Top 1: 72
Top 5: 72
Top 10: 72
Median Rank: 1
Worst Rank: 1
Median Margin Q8: 1391
Worst Margin Q8: 1

ARTIFACTS

Published Files

Artifact	Status	Bytes	SHA-256
NSRLLMM1 attention artifact	present	62.1 KiB	60826f8b823f3bc6
NSRLMOD1 multimodal artifact	present	17.1 MiB	ef57fc5ad94cda62
NSRLTCH denoiser artifact	present	10.3 KiB	a60497cd37201ede
Solomon text signature index	present	104.8 KiB	34c0227d0f54ab5c

Prior Scaling Curve

12 rows

eval_top1_per_mille

200

576 prompt rows / latent 32 / text 512 / 0ce31085

eval_top5_per_mille

462

576 prompt rows / latent 32 / text 512 / 0ce31085

gold_top1_per_mille

250

288 prompt rows / latent 32 / text 512 / c98fc237

prompt_rows	n_train_prompts	latent_dim	text_features	eval_top1_per_mille	eval_top5_per_mille	gold_top1_per_mille	gold_top5_per_mille	model_hash
288	115	32	512	34	275	250	506	c98fc237
288	115	64	512	0	241	104	326	ca117469
288	115	128	512	34	241	97	326	f67981cf
576	352	32	512	200	462	76	277	0ce31085
576	352	64	512	137	362	97	229	be26f0d7
576	352	128	512	187	362	83	222	57d13997
1152	822	32	512	86	231	62	222	65047fbd
1152	822	64	512	64	215	76	256	a2a510ad
1152	822	128	512	75	231	104	256	e8f9bf91
1425	1040	32	512	66	240	41	222	d3ea03af
1425	1040	64	512	49	244	62	243	134230e1
1425	1040	128	512	49	219	97	284	4e63b0ad

Text Feature Shape Probe

6 rows

eval_top1_per_mille

237

576 prompt rows / latent 128 / text 8192 / 4fcc02bd

novel_top1_per_mille

291

576 prompt rows / latent 128 / text 8192 / 4fcc02bd

gold_top5_per_mille

416

576 prompt rows / latent 32 / text 2048 / 04993d13

prompt_rows	latent_dim	text_features	eval_top1_per_mille	eval_top5_per_mille	novel_top1_per_mille	gold_top1_per_mille	gold_top5_per_mille	model_hash
576	32	2048	225	537	208	166	416	04993d13
576	32	8192	212	537	208	173	388	b801e8fd
576	64	2048	187	487	125	159	333	127e2a41
576	64	8192	175	437	166	152	319	5e6dfa75
576	128	2048	137	437	166	145	326	d1b4e7ed
576	128	8192	237	462	291	138	277	4fcc02bd

Text Feature Probe

1 rows

eval_top1_per_mille

212

576 prompt rows / latent 32 / text 8192 / b801e8fd

novel_top1_per_mille

208

576 prompt rows / latent 32 / text 8192 / b801e8fd

gold_top5_per_mille

388

576 prompt rows / latent 32 / text 8192 / b801e8fd

prompt_rows	latent_dim	text_features	eval_top1_per_mille	eval_top5_per_mille	novel_top1_per_mille	gold_top1_per_mille	gold_top5_per_mille	model_hash
576	32	8192	212	537	208	173	388	b801e8fd

Generative Eval

2 rows

top1_per_mille

41

centroid16 / 24 prompts

top5_per_mille

41

centroid16 / 24 prompts

latent_top1_per_mille

83

centroid16 / 24 prompts

model	prompts	top1_per_mille	top5_per_mille	latent_top1_per_mille	latent_top5_per_mille	mean_generated_target_distance_q8	text_weight
centroid16	24	41	41	83	125	6062784	96
aws-e8	24	41	41	41	41	5849621	96

Latent Planner Eval

1 rows

top1_per_mille

41

ld128tf1024 / 72 prompts

top5_per_mille

83

ld128tf1024 / 72 prompts

run_id	model	prompts	top1_per_mille	top5_per_mille	mean_rank_q8	mean_target_distance_q8	latent_model_hash	conclusion
clean72-ld128tf1024	ld128tf1024	72	41	83	9255	4265852	0xa620d349f9f336ad	learned decoded latent plans only identify 3 of 72 held-out prompts at top-1, confirming the prompt-to-signature planner is the first blocker before bitmap rendering

Oracle Conditioning Diagnostic

1 rows

signature_top1_per_mille

986

72 prompts

retrieval_top1_per_mille

180

72 prompts

retrieval_top5_per_mille

527

72 prompts

run_id	diagnostic_only	prompts	signature_top1_per_mille	signature_top5_per_mille	retrieval_top1_per_mille	retrieval_top5_per_mille	mean_signature_rank_q8	mean_retrieval_rank_q8	conclusion
v1-oracle-l72-c1-p1	true	72	986	1000	180	527	259	2549	true 16x16 conditioning nearly solves signature identity, so the learned prompt-to-signature planner is the first blocker; rendered retrieval identity remains weak and separate

Multimodal Eval

1 rows

overall_top1_per_mille

994

deployed / 313 examples / 0x8057e681de909abc

text_top1_per_mille

996

deployed / 313 examples / 0x8057e681de909abc

image_top1_per_mille

1000

deployed / 313 examples / 0x8057e681de909abc

model	eval_scope	examples	overall_top1_per_mille	text_top1_per_mille	image_top1_per_mille	prompt_top1_per_mille	exact_examples_per_mille	context_hit_per_mille	model_hash
deployed	tracked_corpus_replay	313	994	996	1000	882	3	1000	0x8057e681de909abc

Attention Raw Eval

1 rows

prompt_name_match_per_mille

0

raw-no-memory / 72 prompts / 0xff009204cd3d1767

mean_raw_quality_score

27

raw-no-memory / 72 prompts / 0xff009204cd3d1767

distinct_texts

4

raw-no-memory / 72 prompts / 0xff009204cd3d1767

model	eval_scope	prompts	prompt_name_match_per_mille	mean_raw_quality_score	min_raw_quality_score	distinct_texts	scaffold_output_per_mille	model_hash
raw-no-memory	native_raw_no_memory_no_conditioning	72	0	27	15	4	763	0xff009204cd3d1767

Prompt Sample Gallery

6 rows

model	prompt_id	prompt_kind	prompt	text_source	image_source	text_lm_fallback	text	image_path	model_hash
NSRLLMM1	bael	fixed-known	seal of Bael	embedded_text_lm_strict	embedded_image_memory_strict	false	Solomon selects Bael: He maketh thee to go Invisible.	docs/assets/results-samples/bael.png	0xff009204cd3d1767
NSRLLMM1	stolas	fixed-known	seal of Stolas	embedded_text_lm_strict	embedded_image_memory_strict	false	Solomon selects Stolas: He teacheth the Art of Astronomy, and the Virtues of Herbs and Precious Stones.	docs/assets/results-samples/stolas.png	0xff009204cd3d1767
NSRLLMM1	marbas	fixed-known	seal of Marbas	embedded_text_memory_guard	embedded_image_memory_strict	true	Solomon selects Marbas: He is a Great President, and appeareth at first in the form of a Great Lion, but afterwards, at the request of the Master, he putteth on Human Shape.	docs/assets/results-samples/marbas.png	0xff009204cd3d1767
NSRLLMM1	generic	fixed-generic	king solomon seal	embedded_text_lm_strict	embedded_image_memory_strict	false	Solomon selects Ronove: He teacheth the Art of Rhetoric very well and giveth Good Servants, Knowledge of Tongues, and Favours with Friends or Foes.	docs/assets/results-samples/generic.png	0xff009204cd3d1767
NSRLLMM1	eastern-king	fixed-held-out-phrase	eastern king invisible spirit	embedded_text_memory_guard	embedded_image_memory_strict	true	Solomon selects Marbas: He is a Great President, and appeareth at first in the form of a Great Lion, but afterwards, at the request of the Master, he putteth on Human Shape.	docs/assets/results-samples/eastern-king.png	0xff009204cd3d1767
NSRLLMM1	owl-astronomy	fixed-held-out-phrase	owl astronomy seal	embedded_text_lm_strict	embedded_image_memory_strict	false	Solomon selects Murmur: and appeareth in the Form of a Warrior riding upon a Gryphon, with a Ducal Crown upon his Head.	docs/assets/results-samples/owl-astronomy.png	0xff009204cd3d1767