一部の科学コンピューティングプロジェクトではグリッド検索を実行していますが、ファイル名にパラメータを含むログファイルがたくさんあります。
$ ls -l | wc -l
38210
$ ls -l | tail
-rw-rw-r-- 1 qys wheel 1070 Jul 20 15:01 gmm_dim96_aelr0.01_ufactor100_K8_R8_lr1_lambda0_kappa0.25.txt
-rw-rw-r-- 1 qys wheel 1070 Jul 20 15:00 gmm_dim96_aelr0.01_ufactor100_K8_R8_lr1_lambda0_kappa0.5.txt
-rw-rw-r-- 1 qys wheel 1070 Jul 20 15:01 gmm_dim96_aelr0.01_ufactor100_K8_R8_lr1_lambda0_kappa0.75.txt
-rw-rw-r-- 1 qys wheel 1070 Jul 20 15:00 gmm_dim96_aelr0.01_ufactor100_K8_R8_lr1_lambda0_kappa0.txt
-rw-rw-r-- 1 qys wheel 1070 Jul 20 15:01 gmm_dim96_aelr0.01_ufactor100_K8_R8_lr1_lambda0_kappa1.txt
-rw-rw-r-- 1 qys wheel 981 Jul 20 15:06 gmm_dim96_aelr0.01_ufactor100_K8_R8_lr1_lambda1_kappa0.25.txt
-rw-rw-r-- 1 qys wheel 981 Jul 20 15:08 gmm_dim96_aelr0.01_ufactor100_K8_R8_lr1_lambda1_kappa0.5.txt
-rw-rw-r-- 1 qys wheel 981 Jul 20 15:08 gmm_dim96_aelr0.01_ufactor100_K8_R8_lr1_lambda1_kappa0.75.txt
-rw-rw-r-- 1 qys wheel 981 Jul 20 15:06 gmm_dim96_aelr0.01_ufactor100_K8_R8_lr1_lambda1_kappa0.txt
-rw-rw-r-- 1 qys wheel 984 Jul 20 15:08 gmm_dim96_aelr0.01_ufactor100_K8_R8_lr1_lambda1_kappa1.txt
各ファイルは次のようになります。
$ cat gmm_dim96_aelr0.01_ufactor100_K8_R8_lr1_lambda1_kappa1.txt
GMM: epoch 0: train loss 662699.9375, valid accuracy 80.03%
GMM: epoch 1: train loss 662553.75, valid accuracy 79.92%
GMM: epoch 2: train loss 662446.1875, valid accuracy 79.77%
GMM: epoch 3: train loss 662429.5, valid accuracy 79.82%
GMM: epoch 4: train loss 662404.5625, valid accuracy 79.83%
GMM: epoch 5: train loss 662395.0625, valid accuracy 79.85%
GMM: epoch 6: train loss 662381.25, valid accuracy 79.85%
GMM: epoch 7: train loss 662370.25, valid accuracy 79.83%
GMM: epoch 8: train loss 662362.875, valid accuracy 79.86%
GMM: epoch 9: train loss 662353.6875, valid accuracy 79.84%
GMM: epoch 10: train loss 662343.4375, valid accuracy 79.83%
GMM: epoch 11: train loss 662335.0625, valid accuracy 79.83%
GMM: epoch 12: train loss 662325.5, valid accuracy 79.85%
GMM: epoch 13: train loss 662316.6875, valid accuracy 79.87%
GMM: epoch 14: train loss 662307.0625, valid accuracy 79.88%
GMM: epoch 15: train loss 662298.0625, valid accuracy 79.90%
GMM: end: test accuracy 80.38%
精度を評価してパラメータ構成をソートしたいのですが、grep次のエラーが発生します。
$ grep 'GMM: epoch 15: ' gmm_*ufactor100*.txt | sort -k 9 -n -r | head
exec: Failed to execute process '/usr/bin/grep': the total size of the argument list and exported variables (2.1MB) exceeds the OS limit of 1MB.
より大きなサイズのパラメータリストを許可するには？関連がある場合に備えてmacOS Montereyを使用しています。