tapt_ulmfit_reinit_whole_word_5K_no_reinit_classifier-LR_2e-05
This model is a fine-tuned version of microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext on an unknown dataset. It achieves the following results on the evaluation set:
- Loss: 3.2526
- Accuracy: 0.4517
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 3407
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-06 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.06
- num_epochs: 100
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|---|---|---|---|---|
| 10.4971 | 1.0 | 21 | 9.9747 | 0.0130 |
| 9.5453 | 2.0 | 42 | 7.5742 | 0.1006 |
| 7.1693 | 3.0 | 63 | 5.7197 | 0.2509 |
| 5.8951 | 4.0 | 84 | 5.0410 | 0.3039 |
| 5.3672 | 5.0 | 105 | 4.5359 | 0.3297 |
| 5.0224 | 6.0 | 126 | 4.3382 | 0.3462 |
| 4.742 | 7.0 | 147 | 4.1588 | 0.3589 |
| 4.5698 | 8.0 | 168 | 4.0386 | 0.3707 |
| 4.4564 | 9.0 | 189 | 4.0000 | 0.3725 |
| 4.357 | 10.0 | 210 | 3.8163 | 0.3961 |
| 4.2363 | 11.0 | 231 | 3.8212 | 0.3921 |
| 4.1515 | 12.0 | 252 | 3.8091 | 0.3944 |
| 4.0535 | 13.0 | 273 | 3.7407 | 0.4014 |
| 3.9913 | 14.0 | 294 | 3.6963 | 0.4021 |
| 3.9702 | 15.0 | 315 | 3.6459 | 0.4086 |
| 3.9457 | 16.0 | 336 | 3.6291 | 0.4084 |
| 3.8999 | 17.0 | 357 | 3.5868 | 0.4162 |
| 3.8133 | 18.0 | 378 | 3.5709 | 0.4184 |
| 3.7768 | 19.0 | 399 | 3.5593 | 0.4165 |
| 3.7556 | 20.0 | 420 | 3.5804 | 0.4177 |
| 3.6968 | 21.0 | 441 | 3.5838 | 0.4177 |
| 3.7113 | 22.0 | 462 | 3.5509 | 0.4196 |
| 3.6415 | 23.0 | 483 | 3.4941 | 0.4231 |
| 3.6249 | 24.0 | 504 | 3.4634 | 0.4238 |
| 3.6029 | 25.0 | 525 | 3.4496 | 0.4278 |
| 3.5838 | 26.0 | 546 | 3.4745 | 0.4272 |
| 3.5798 | 27.0 | 567 | 3.4518 | 0.4324 |
| 3.5762 | 28.0 | 588 | 3.5176 | 0.4237 |
| 3.5317 | 29.0 | 609 | 3.4472 | 0.4208 |
| 3.5312 | 30.0 | 630 | 3.4399 | 0.4298 |
| 3.484 | 31.0 | 651 | 3.3953 | 0.4348 |
| 3.492 | 32.0 | 672 | 3.4387 | 0.4287 |
| 3.4642 | 33.0 | 693 | 3.4385 | 0.4301 |
| 3.4556 | 34.0 | 714 | 3.4118 | 0.4352 |
| 3.4703 | 35.0 | 735 | 3.4121 | 0.4306 |
| 3.4157 | 36.0 | 756 | 3.3746 | 0.4381 |
| 3.3863 | 37.0 | 777 | 3.3709 | 0.4413 |
| 3.4176 | 38.0 | 798 | 3.3834 | 0.4364 |
| 3.4316 | 39.0 | 819 | 3.3444 | 0.4396 |
| 3.3904 | 40.0 | 840 | 3.3664 | 0.4377 |
| 3.3595 | 41.0 | 861 | 3.2911 | 0.4489 |
| 3.369 | 42.0 | 882 | 3.3726 | 0.4360 |
| 3.3612 | 43.0 | 903 | 3.3574 | 0.4329 |
| 3.3994 | 44.0 | 924 | 3.3158 | 0.4428 |
| 3.3062 | 45.0 | 945 | 3.3189 | 0.4499 |
| 3.2856 | 46.0 | 966 | 3.3371 | 0.4444 |
| 3.314 | 47.0 | 987 | 3.3012 | 0.4494 |
| 3.333 | 48.0 | 1008 | 3.2689 | 0.4522 |
| 3.3043 | 49.0 | 1029 | 3.3684 | 0.4401 |
| 3.2732 | 50.0 | 1050 | 3.3020 | 0.4408 |
| 3.3005 | 51.0 | 1071 | 3.3013 | 0.4443 |
| 3.2666 | 52.0 | 1092 | 3.3370 | 0.4419 |
| 3.2397 | 53.0 | 1113 | 3.3533 | 0.4405 |
| 3.261 | 54.0 | 1134 | 3.3398 | 0.4422 |
| 3.2677 | 55.0 | 1155 | 3.3325 | 0.4405 |
| 3.2501 | 56.0 | 1176 | 3.3436 | 0.4428 |
| 3.2443 | 57.0 | 1197 | 3.3427 | 0.4451 |
| 3.28 | 58.0 | 1218 | 3.3806 | 0.4389 |
| 3.2301 | 59.0 | 1239 | 3.2235 | 0.4572 |
| 3.2526 | 60.0 | 1260 | 3.3543 | 0.4402 |
| 3.2256 | 61.0 | 1281 | 3.3432 | 0.4451 |
| 3.2288 | 62.0 | 1302 | 3.2800 | 0.4516 |
| 3.205 | 63.0 | 1323 | 3.2833 | 0.4542 |
| 3.1952 | 64.0 | 1344 | 3.3317 | 0.4439 |
| 3.2333 | 65.0 | 1365 | 3.3256 | 0.4453 |
| 3.2015 | 66.0 | 1386 | 3.3485 | 0.4472 |
| 3.2215 | 67.0 | 1407 | 3.3037 | 0.4416 |
| 3.1505 | 68.0 | 1428 | 3.3030 | 0.4458 |
| 3.1521 | 69.0 | 1449 | 3.2951 | 0.4491 |
| 3.172 | 70.0 | 1470 | 3.2670 | 0.4511 |
| 3.1717 | 71.0 | 1491 | 3.3420 | 0.4417 |
| 3.1664 | 72.0 | 1512 | 3.3523 | 0.4456 |
| 3.1702 | 73.0 | 1533 | 3.2418 | 0.4536 |
| 3.1604 | 74.0 | 1554 | 3.2931 | 0.4480 |
| 3.1479 | 75.0 | 1575 | 3.2274 | 0.4525 |
| 3.1578 | 76.0 | 1596 | 3.3571 | 0.4469 |
| 3.1335 | 77.0 | 1617 | 3.2580 | 0.4536 |
| 3.155 | 78.0 | 1638 | 3.2328 | 0.4521 |
| 3.1475 | 79.0 | 1659 | 3.2460 | 0.4529 |
| 3.1701 | 80.0 | 1680 | 3.2757 | 0.4450 |
| 3.1547 | 81.0 | 1701 | 3.2507 | 0.4500 |
| 3.1483 | 82.0 | 1722 | 3.2842 | 0.4538 |
| 3.1144 | 83.0 | 1743 | 3.2563 | 0.4519 |
| 3.1554 | 84.0 | 1764 | 3.2697 | 0.4508 |
| 3.1142 | 85.0 | 1785 | 3.3466 | 0.4347 |
| 3.1438 | 86.0 | 1806 | 3.2828 | 0.4481 |
| 3.1466 | 87.0 | 1827 | 3.2621 | 0.4517 |
| 3.1504 | 88.0 | 1848 | 3.2583 | 0.4456 |
| 3.1116 | 89.0 | 1869 | 3.2623 | 0.4538 |
| 3.1362 | 90.0 | 1890 | 3.2623 | 0.4501 |
| 3.1368 | 91.0 | 1911 | 3.2146 | 0.4557 |
| 3.1106 | 92.0 | 1932 | 3.2504 | 0.4546 |
| 3.1341 | 93.0 | 1953 | 3.2531 | 0.4542 |
| 3.1312 | 94.0 | 1974 | 3.3133 | 0.4451 |
| 3.1581 | 95.0 | 1995 | 3.3001 | 0.4465 |
| 3.1451 | 96.0 | 2016 | 3.2344 | 0.4527 |
| 3.1172 | 97.0 | 2037 | 3.2965 | 0.4490 |
| 3.1234 | 98.0 | 2058 | 3.2498 | 0.4553 |
| 3.1064 | 99.0 | 2079 | 3.2326 | 0.4588 |
| 3.1169 | 100.0 | 2100 | 3.2526 | 0.4517 |
Framework versions
- Transformers 4.48.2
- Pytorch 2.4.1+cu121
- Datasets 3.0.2
- Tokenizers 0.21.0
- Downloads last month
- 4