groupuser/mistral-1L-tiny
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
Niels Horn 2024-02-27 21:45:50 +00:00 committed by system
parent 5be0accedb
commit 524aad8e8f
No known key found for this signature in database
GPG Key ID: 6A528E38E0733467
1 changed files with 7 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

17
README.md
View File

@ -20,11 +20,9 @@ model-index:
value: 0.5792084706530948 value: 0.5792084706530948
--- ---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
# mistral-1L-tiny # mistral-1L-tiny
A tiny single-layer 35.1M parameter Mistral model, with a hidden size of 512, and an MLP intermediate size of 1024.
This model is trained on the roneneldan/TinyStories dataset. This model is trained on the roneneldan/TinyStories dataset.
It achieves the following results on the evaluation set: It achieves the following results on the evaluation set:
- Loss: 1.6868 - Loss: 1.6868
@ -32,18 +30,17 @@ It achieves the following results on the evaluation set:
## Model description ## Model description
More information needed This work is inspired by the 21M parameter one-layer GPT-Neo of the [Tiny Stories paper](https://arxiv.org/abs/2305.07759).
Results reproduced to acquire high-frequency checkpoints for further analysis.
## Intended uses & limitations ## Intended uses & limitations
More information needed Analysis of feature dynamics and emergence in real-world language models.
## Training and evaluation data
More information needed
## Training procedure ## Training procedure
Trained for 90171 steps, corresponding to ~2 hours on a single H100.
### Training hyperparameters ### Training hyperparameters
The following hyperparameters were used during training: The following hyperparameters were used during training:
@ -57,7 +54,7 @@ The following hyperparameters were used during training:
### Training results ### Training results
Quite consistent English text generation.
### Framework versions ### Framework versions