Conversation
* feat(jacobian_lens): add occupancy and fraction-of-variance Two J-space profiling statistics on top of the sparse decomposition, following Gurnee et al. (2026): - estimate_occupancy / JacobianLens.occupancy: how many J-lens vectors are meaningfully active in an activation, via the step of maximum separation between the real and a random-control cumulative captured-variance curve (deterministic, threshold-free). - JacobianLens.fraction_of_variance / JSpaceVarianceProfile: the J-space share of activation variance over a corpus, per layer as the median and pooled ratio of ||j_space_component||^2 / ||activation||^2 -- the selected-support span projection, not the nonnegative reconstruction. Model-free unit tests (planted-sparsity recovery, determinism, input validation, and the fraction_of_variance no-sample NaN contract) plus gpt2-small integration tests (occupancy seed-determinism, the fraction positions= override, and the corpus median/pooled path); documented in jacobian_lens_fitting.md as an open-weight, shape-only observation. * fix(jacobian_lens): harden occupancy estimator contract Resolve PR #1676 review comments 1 and 2 as one model-free occupancy contract update. Validation: - Reject non-finite and zero-norm targets before projection math. - Reject complex inputs instead of silently discarding imaginary values. - Reject target and dictionary norm overflow. - Add focused regressions for every new validation path. - Correct existing validation tests so they reach the intended checks instead of passing through an invalid default max_atoms value. Algorithm documentation: - Describe occupancy's unconstrained span-projection residual recurrence. - Clarify that occupancy and sparse decomposition share the per-step correlation rule but can select different supports. - State that occupancy always selects exactly max_atoms atoms and does not use sparse decomposition's early stopping. - Keep the max-separation statistic and random-control behavior unchanged. Verification: - Decomposition and public-import unit surface: 182 passed. - black, isort, and pycln clean on the touched Python files. - mypy clean on the touched source. - Sphinx build and git diff --check clean. * fix(jacobian_lens): validate variance profile sampling inputs Resolve PR #1676 review comments 3 and 4 as one corpus-sampling contract update. Token shape validation: - Reject token tensors that are not exactly [1, seq] before run_with_cache. - The batched case ([2, seq]) previously ran silently and profiled only the first prompt because the implementation reads cache[hook][0]. - Add parameterized regressions for missing batch dimension, multiple prompts, and extra dimension. Negative skip_first: - Reject skip_first < 0 unconditionally, including when explicit positions override sampling. - Previously range(skip_first, seq_len) accepted negative starts and indexed activations from the end. - Add parameterized regressions for default and explicit-positions paths. Documentation: - Update fraction_of_variance docstring: prompts, skip_first, and Raises. - Add token-shape and non-negative skip_first contract to public docs page. Verification: - Wrapper plus decomposition unit surface: 251 passed. - Lockfile-pinned black, isort, and pycln clean on the touched Python files. - mypy clean on the touched source. - Focused Sphinx page build and git diff --check clean.
Padding-aware TransformerBridge loss: -inf x 0 = nan no longer leaks into the reduced loss (lm_utils masked_fill), fully-masked padding rows no longer softmax to NaN in native attention, and forward() threads attention_mask into loss_fn for return_type='loss'/'both' with 2D/4D mask reduction. (cherry picked from commit 7ebeab9, adapted for dev: bridge_core/transformer_bridge hunks applied to bridge.py's inlined dispatch; RemoteBridge portions dropped — class absent on dev) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…v-4.x a1fbe1e (#1613) forward(labels=...) now computes shifted causal CE against the labels (ignore_index=-100, attention-mask aware) instead of silently scoring input_ids; encoder-decoder models require labels for loss and use HF's native unshifted objective; benchmark loss equivalence supplies tokenized self-labels. Brings make_tiny_pair helper the tests depend on. (reimplemented from commit a1fbe1e for dev: _finalize_return logic applied to bridge.py's inlined dispatch; encoder-decoder detection reads self.original_model.config instead of 4.x's _driver; RemoteBridge/transformers_driver portions dropped. Dev-specific addition the upstream diff never needed: with labels forwarded, HF tuple outputs are (loss, logits, ...), so the tuple fallback now indexes logits at [1]) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…input — mirror of dev-4.x 21993bf (#1610) Masked-out tokens silently shifted the absolute position of every real token after them (no error, wrong logits/loss; gpt2 loss 4.50 -> 11.15 with three left pads). Positions are now derived from the mask via utils.get_offset_position_ids, per row, only when a masked token precedes a real one; explicit position_ids still wins; cached decode slices the derived positions back to the passed tokens. _accepts_derived_position_ids gates injection off fixed-signature forwards, mRoPE models that own their position derivation, and mask-consuming positional embeddings (OPT). (reimplemented from commit 21993bf for dev: the gate introspects self.original_model — dev has no _driver abstraction; the gate unit test's stand-ins adapted to the same shape) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ts — mirror of dev-4.x 7db5f8d (#1617) generate() on an already-padded token tensor treated pads as real context, shifting every real token's position and changing the continuation. generate(attention_mask=...) states the prompt padding directly and is extended one attended column per generated token; the cached step derives the new token's position from the running mask instead of total_len - 1 (which counts pad slots); generate(padding_side=...) now raises when no tokenizer/pad id exists to read the padding from instead of being inert. (path-retargeted from commit 7db5f8dc21657444886659f56ba13910eb45408c: transformer_bridge.py hunks applied to bridge.py; no semantic changes) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rget model — mirror of dev-4.x c5967ea (#1627) The batched-list branch and every cached-step branch handed derived position_ids to the model unchecked: a fixed-signature forward (LLaDA) raised TypeError where it used to return logits, and gating only the prompt derivation would divert refused models into the total_len - 1 fallback, which counts pad slots and is wrong per row for left-padded batches (tiny-random-OPT cached-vs-uncached drift 7.45e-08 -> 2.98e-01). Both sites now consult _accepts_derived_position_ids(); models that own their position derivation receive the mask alone. (path-retargeted from commit c5967ea: transformer_bridge.py hunks applied to bridge.py; no semantic changes) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GLM-4V/Qwen-VL models cache rope_deltas on the HF module; a text-only prefill never refreshes them, so generate() after a multimodal forward on the same bridge added the stale image-sequence delta to every cached-step position (RuntimeError in apply_rotary_pos_emb). Masked before the position_ids gating landed because explicit cached-step position_ids short-circuited HF's rope_deltas path entirely. HF's generate recomputes the deltas at prefill via prepare_inputs_for_generation, which the hooked loop bypasses — clearing them for gate-refused models matches that. Latent on dev-4.x as well (same loop, same gate); flagged for upstream. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Loss Masking Mirror
…1629) get_tokens_with_bos_removed assumed a bos_token_id exists. Callers gate it on cfg.tokenizer_prepends_bos, which detect_tokenizer_bos_eos only sets when the tokenizer has one, but the flag goes stale on a bridge built via build_bridge_from_module(tokenizer=None) and given a tokenizer afterwards: the setter re-runs configure_tokenizer only on reassignment, so the config default of True survives. Trusting it then does damage in both directions. Under right padding, the default, the helper drops the first token unconditionally, silently removing [CLS] from a BERT tokenizer's output and returning a plausible-looking wrong result. Under left padding it evaluates (tokens == None).int() and raises AttributeError: 'bool' object has no attribute 'int', which names neither the tokenizer nor the flag. Return the tokens unchanged when there is no bos_token_id: with no BOS there is nothing to remove, which is correct however the config got out of sync. Reproduced with two off-the-shelf tokenizers, bert-base-cased and t5-small, both of which have bos_token_id None. The normal boot_transformers path is unaffected, since detection runs there. The root cause is the reassignment test in bridge_core.py, left alone deliberately. Correcting the flag would route to_tokens(prepend_bos=True) into the manual-prepend branch at transformer_bridge.py:716, which calls get_input_with_manually_prepended_bos(tokenizer.bos_token, ...) and raises TypeError on a None bos_token. The stale flag currently masks that, so the root-cause fix needs the prepend path hardened first. Fixes #1628 Co-authored-by: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 4574892)
#1634) get_input_with_manually_prepended_bos concatenated bos_token + input unconditionally, which is None + str for a tokenizer with no BOS token and raises TypeError naming neither the tokenizer nor the flag that caused it. With no BOS token there is nothing to prepend, so return the input unchanged. Reached when a tokenizer skips setup_tokenizer's bos_token = eos_token backfill, which is the initial-assignment branch at bridge_core.py:113, and tokenizer_prepends_bos is then corrected to False. That is the follow-up scoped out of #1628; hardening it here unblocks the root-cause fix. The guard sits in the shared helper, so all three call sites are covered: transformer_bridge.py:717, HookedTransformer.py:850, remote_bridge.py:216. bos_token widens to Optional[str] because beartype rejects None against the old str annotation, so the runtime guard alone would still fail under test. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 71e3b30; dev adaptation: added Optional to tokenize_utils typing imports — already present on 4.x)
…ation Padding Generation Fixes from 4.x
…f dev-4.x cd89db7 (#1569) Reassigning bridge.tokenizer was a plain attribute write: d_vocab, BOS/EOS detection and padding setup all kept the old tokenizer's values, silently desyncing cfg from the tokenizer actually in use. tokenizer is now a property whose setter re-runs the wiring on reassignment and infers d_vocab on first assignment exactly as __init__ did. (reimplemented from commit cd89db7f0b9f5eb4784edd318102626f7f56b656 for dev: 4.x's configure_tokenizer does not exist here — the setter inlines dev's own setup_tokenizer + detect_tokenizer_bos_eos, the same pair dev's boot path runs; the test's BridgeCore/driver scaffolding became a bare TransformerBridge following test_generation_benchmark_mechanics's construction pattern)
… mirror of dev-4.x 956c989 (#1625) run_with_cache's string path tokenized without prepend_bos, silently dropping the caller's choice: the kwarg was consumed but never reached to_tokens, so cached runs on strings always used the default BOS policy. Both string sites now pop prepend_bos and thread it through. (reimplemented from commit 956c989 for dev: bridge_core.py hunks applied to bridge.py's run_with_cache string sites, which differ in device-placement context; test lands verbatim thanks to the tokenizer property mirrored just before — dev adaptation: added pytest import to test_boot_native.py, already present on 4.x)
…or of dev-4.x a6e0033 (#1598) load_state_dict silently dropped TL-format keys (the very format state_dict() emits): unknown keys passed through to the wrapped model, strict was quietly downgraded, and aliased parameters (e.g. GPT-2's q/k/v views into c_attn) were only partially written. _tl_key_to_actual_keys inverts the state_dict() renaming with every alias; load accepts TL, raw, and stripped key formats; missing_keys treats an alias group as satisfied when any alias was written; strict raises with proper missing/unexpected lists. (path-retargeted from commit a6e003309760a34a20b16ab6b8ff8e04adcf6b6f: transformer_bridge.py hunks applied to bridge.py; no semantic changes)
…rror of dev-4.x 80d9f36 (#1661) Composing a bridge inside a parent nn.Module broke state_dict round trips three ways: attn._ln1_module lazily registered ln1's raw module as a child at first forward (parent state_dict keys changed after forward), and the joint QKV / gate-up components filtered their combined weights out of state_dict() but rejected their own output under strict load. ln1's execution reference now lives outside the ownership tree and is wired at construction; both joint components register load pre-hooks that restore the filtered keys for strict matching. (mirrored from commit 80d9f36bb63f92b4e2f0d6acdb7963069c30a49a: bridge.py hunks path-retargeted from transformer_bridge.py; block.py applied into dev's _maybe_wire_capture_hooks, which merged 4.x's _maybe_wire_pre_ln_capture. Note for the audit record: the component_setup.py and joint_gate_up_mlp.py hunks — marked not-needed-on-dev in the mirror audit — ARE needed; upstream's own tests fail on dev without them)
…or of dev-4.x ea11a86 + 3350142 (#1671, #1664) Parameters and buffers owned by unmapped container modules (BERT's embeddings LayerNorm and token type embeddings, AST's classifier LayerNorm, ViT/BERT pooler and task heads) were unreachable through the bridge: absent from state_dict traversal and unhookable. component_setup now registers container state owners (_container_state_owners submodule) for anything the mapping doesn't cover, adapters map the missing components (embed_ln/mlm_head/nsp_head/pooler, classifier_ln, token_type_embed), and parent-module traversal round-trips identically whether the bridge is wrapped or direct. (mirrored from commits ea11a860bd0e001b8083e94086c50e0e83f0a1c9 and 335014240ef468334e70195b6b40b334b19d75fd — the token_type_embed mapping was catalogued as a hook enhancement by the mirror audit but is load-bearing for parameter coverage: upstream's traversal test fails on dev without it. bridge.py hunks path-retargeted from transformer_bridge.py; dev defines _BLOCK_LIST_ATTRS locally so 4.x's bridge_core import was dropped) Co-authored-by: Liang Hu <35699841+LarryHu0217@users.noreply.github.com>
…A expansion — mirror of dev-4.x 2985420 + 82104b8 hunks (#1603, #1614) get_bridge_params read raw module weights with shape-guessing reshapes: processed/converted layouts were misread, GQA models' grouped K/V never expanded (SVDInterpreter saw n_kv_heads-shaped W_K/W_V), and absent weights zero-filled silently with no signal. The rewrite reads the components' TL-layout accessors (W_Q/W_in/... , post-conversion and post-processing), expands grouped K/V and biases to n_heads via repeat_interleave, emits ln params so consumers can detect fold state, warns on zero-fills, and raises on cfg/blocks inconsistency. (extracted from mixed commits 2985420 and 82104b8; the file is taken at dev-4.x HEAD, which already merges dev's interleaved-MoE dense handling from #1666 — upstream resolved that merge and this reuses its resolution. tests/unit/model_bridge/test_get_params_util.py taken at the same point. Dev's test_get_params_multi_query_attention_reshaping is removed as upstream did in reanchoring: it monkeypatched raw weights against the old shape-guesser and is superseded by test_grouped_kv_expanded_to_n_heads / test_grouped_biases_expanded under the accessor contract)
…odule semantics — mirror of dev-4.x de7531b + d92683d hunks (#1335, #1538) reset_hooks only walked GeneralizedComponent children, so hook points that exist solely in the hook registry (alias-registered and scanned points outside the component tree) silently kept their hooks across resets — the session-fixture leak class. The registry is now cleared directly, with HookedRootModule-parity parameters (direction/including_permanent/level) and clear_contexts/ remove_all_hook_fns/hook_points helpers. (extracted from mixed commits de7531b and d92683d; dev divergence from 4.x HEAD: the component walk is kept alongside the registry pass on full resets — 4.x asserts its registry is canonical, dev's is not verified to be, so both passes run. Guard tests are new: neither origin commit shipped one)
* Raise clear error when adding hooks to gated-off hook points * Warn instead of erroring for filter/callable matches on gated hooks; only warn in run_with_cache when names_filter is explicit * Fix duplicated apply_hooks function and misplaced warning message causing mypy failures * Fix run_with_cache gated hook handling * Update gated MLP hook compatibility test * Format gated MLP hook test --------- Co-authored-by: Jonah Larson <jonahalarson@comcast.net>
Tokenizer BOS mirror
* Fix batched TransformerBridge padding semantics * Make batched cache acceptance padding-aware * Preserve causal right-padding compatibility Co-authored-by: tandede <1090179959@qq.com> --------- Co-authored-by: tandede <1090179959@qq.com>
…aversal `state_dict` Traversal System updates Mirror
…dev-4.x 2985420 hunks (#1603) remove_batch_dim and apply_slice_to_batch_dim assumed every cache entry leads with the batch dim: position-indexed entries (T5-style relative position bias, leading dim = seq len) and broadcast entries (leading 1) were stripped or sliced as if batched — silent corruption — and a cache with true batch > 1 plus any dim-1 entry passed the old any() gate. _batch_size() takes the mode of leading dims; removal asserts mode == 1 and strips only dim-1 entries; slicing skips non-batch entries. (extracted from mixed commit 2985420; its test edits are reanchoring and stay behind — guard tests are new, dev had no coverage of the mixed-shape scenario)
(cherry picked from commit cb34fbf)
…/pretrain — mirror of dev-4.x 1b2eaa4 hunks (#1550) Five adapter fixes with live consumers on dev: Cohere2's NoPE layers warned spuriously about missing RoPE on every forward (rope_optional=True — nulling position_embeddings there is the design); Dream and GIDD silently computed shifted causal CE on bidirectional diffusion objectives (supports_causal_loss=False makes return_type='loss'/'both' raise); Raven's depth-recurrent core cannot use HF past_key_values stepping (supports_kv_cache/supports_batched_generation=False); pretrain never set uses_rms_norm, so norm bridges mean-centered RMS intermediates whenever the wrapped norm's class name didn't say RMSNorm (_set_rms_rotary_defaults, which dev already provides, sets it). (extracted from mixed commit 1b2eaa4; its remote-code-compat refactor and adapter rewrites stay behind — only the confirmed fix lines are mirrored. Guard tests are new and exercise the consuming code paths: the forward loss guard, _resolve_generation_caching, and a from-config Cohere2 forward)
…2ab5 (#1573) boot_native given a HookedTransformerConfig (or anything else) crashed deep in adapter dispatch with an opaque error; it now raises a pointing TypeError up front. The impl signature widens to Any with @overload stubs so the guard is reachable — a Union hint would have beartype reject foreign configs with its own violation first. (reimplemented from commit b232ab5f66e434e2519478d02cef2f18ec2b8b16 for dev: guard applied to TransformerBridge.boot_native, dev's home for the native boot path; the 4.x message's deprecation wording dropped — legacy configs are not deprecated on dev)
…— mirror of dev-4.x b27b7c6 + 82104b8 hunks (#1577, #1614) boot_native ignored the config's initialization contract: the -1.0 initializer_range sentinel fell through to std=0.02 instead of 0.8/sqrt(d_model) in gpt2 mode, xavier/kaiming ignored initializer_range as gain, and init_weights=False still ran custom init. The sentinel now resolves at point of use (cfg keeps -1.0, matching HookedTransformerConfig consumers), gain threads through _NON_RESIDUAL_MODES, init_weights gates initialization, and bridge.init_weights() reinitializes natives in place. Residual 1/sqrt(2*n_layers) output scaling is kept and documented as an intentional delta from HookedTransformer. (mirrored from commits b27b7c6 and 82104b8's native hunks; boot changes applied to TransformerBridge.boot_native — dev's home for the native boot path. LNPre/RMSPre tests from the 4.x context stay behind with their unmirrored feature. test_native_init_semantics.py taken at dev-4.x HEAD) Co-authored-by: Guan Tong <tguan@gmail.com> Co-authored-by: Jonah Larson <jonahalarson@comcast.net>
…ts — mirror of dev-4.x 6773425 (#1593) QK/OV factored-matrix accessors and all_composition_scores crashed on GQA models: grouped W_K/W_V (n_kv_heads) never expanded to n_heads before the einsums. _expand_kv_heads repeat_interleaves K/V weights and biases at the accessor boundary; MHA models are a documented no-op (bit-identical outputs). (path-retargeted from commit 677342504975da414d59b7292342966fa4f28d43: transformer_bridge.py hunks applied to bridge.py; test module imports retargeted to dev's bridge module)
…259a (#1633) stop_at_layer never fired on native bridges: blocks are named top-level ("layers.0") and the leading-dot pattern missed them, so forwards ran the whole stack and returned final-layer residuals as "stopped" output; negative stop indices were also compared unnormalized. The layer-name pattern is now anchored alternation, negative stops normalize against len(blocks), and NativeModel accepts inputs_embeds for the resumed path. (mirrored from commit 0d1259ad20fcfeb2e0995126db0522b90014f2a6: transformer_bridge.py hunk applied to bridge.py; block.py's regex fix applied inside dev's name-guard shape. The input_to_embed round-trip test from the 4.x context stays behind — that API does not exist on dev)
…apter (#1602) NanogptArchitectureAdapter.convert_weights ended in super().convert_weights(remote_module), but ArchitectureAdapter has had no such method since 3efbd6e ("Cleanup (#1129)"), so every call raised AttributeError. The failure was invisible to CI because of the # type: ignore[misc] on that line; without it mypy reports '"convert_weights" undefined in superclass'. The override had no callers anywhere in the tree, and its _orig_mod. prefix strip was a no-op regardless: nn.Module.state_dict() returns a fresh dict, so the loop mutated a throwaway copy before passing the original module to super(). Removing it cannot regress behaviour, since every path through it raised. Dropping the ignore leaves the CI type-check job as the guard against reintroduction. Not re-homed into preprocess_weights: that hook runs on self.state_dict() inside process_weights, i.e. after load with TL-renamed keys, so it never observes _orig_mod.-prefixed checkpoint keys. (cherry picked from commit 29a6baf)
…ev-4.x 1fdc955 hunks The BERT.ipynb NSP cell called nsp(...) without token_type_ids, so both sentences sat in segment 0 and the demo asserted on a prediction the model never actually made for sentence pairs; the migration guide still claimed NSP was not portable to the bridge. The notebook passes token_type_ids and the guide now shows the working model_class recipe (verified end-to-end: 'The sentences are sequential'). (extracted from commit 1fdc955; its deprecation-wording hunks are 4.x divergence and stay behind)
…-path rule — mirror of dev-4.x 82104b8 hunks (#1614) train() wrote its resolved device and wandb defaults back onto the caller's config object — a silent side effect (dataclasses.replace copy). Grokking_Demo hardcoded key_freqs/key_freq_indices from a past run, wrong for any new seed or training length — now derived from the run's own Fourier norms — and its training loop tqdm now throttles below Jupyter's IOPub rate limit. doc_sanitize gains a tmp-path masking rule. (extracted from mixed commit 82104b8: training.py hunk applied to dev's train.py — the tools/ relocation is 4.x-only; the sanitize rule lands as [regex8] because dev already has an unrelated [regex7]; the notebook's HookedTransformer->Bridge migration stays behind as reanchoring. Guard test is new; the notebook fix is untestable in CI where Grokking is disabled)
…ference — mirror of dev-4.x 1b2eaa4 hunks (#1550) update_model_registry stamped STATUS_VERIFIED unconditionally: failing phase scores and structural-only runs (no HF numerical reference) both recorded as verified, and every run appended a verification record that VerificationHistory.is_verified() then trusted. Status now derives from the shared verify_models threshold/note logic (FAILED on threshold misses, PROVISIONAL without an HF reference, no history record for provisional runs), and main() forwards --no-hf-reference. The registry validator also checks phase4/7/8/9 scores it previously ignored. (extracted from mixed commit 1b2eaa4: dev keeps extract_phase_scores/ pass_status as verify_models privates rather than promoting them to registry_io; the adapter phase-applicability gating from the same commit is NOT mirrored — it depends on a registry_io TEXT_PHASES promotion and was not a confirmed audit unit. Upstream's gating tests land verbatim)
* kv cache layer_past issue, MoE norm gains * Bloom fixes, qwen fixes, gemma norm values * Improved backwards gradient testing
Native Adapter Mirror
Activation Cache Mirror
Docs & Tooling Mirror
Co-authored-by: nightcityblade <nightcityblade@gmail.com>
Co-authored-by: nightcityblade <nightcityblade@gmail.com>
* BERT verification flaw fixes * Refresh stale BERT demo outputs Cell 7's saved predictions were captured before the fallback-BOS fix, when '<|endoftext|>' was WordPiece-shredded into 8 tokens ahead of every prompt and skewed the masked predictions. Re-running the cell on the fixed code gives sun/went/caught. * test fix for bert verification * Improved fidelity on this test
* Fix direct Bridge config flag assignment * Address Bridge config assignment review feedback * Warn on reused live Bridge configs
… xavier/kaiming (#1685) * Fix native init: resolve initializer_range sentinel, thread gain into xavier/kaiming, document residual scaling delta * Address native init review feedback * Format native init * Add config-level initializer sentinel coverage
* preserve generalized component input dtype * preserve dtype in specialized bridge forwards --------- Co-authored-by: jlarson4 <jonahalarson@comcast.net>
* do not overwrite storage dtypes * pipeline fix --------- Co-authored-by: jlarson4 <jonahalarson@comcast.net>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
A complete restructure of our Phase 4 Generation Benchmark, which aims to better judge the quality of model output as part of our tool. It also carries a series of fixes imported from
dev-4.xwhich were effecting the quality of model outputs ondev.Aa well as an additional set of Bug Fixes based on a new crop of issues.
Type of change