> I simply don't understand why literally everyone immediately jumps at CNNs, RNNs (transformers et al.) -- they're extremely expensive, slow
Because for text they work a lot better.
HTMs are competitive with other non-NN techniques on small datasets (see [1] you listed) but nothing particularly amazing.
I'd speculate this is because they use bag-of-word variants like LSI and TF-IDF. Prior to Transformers this was a competitive technique and you could get state-of-the-art results on most things using similar techniques.
This representation of data matters a lot. Even just switching to a word embedding representation and using a SVM or something gives a decent gain in most circumstances.
But transformers are much better, particularly on harder tasks (eg question answering on long documents). You can't really see how significant this difference is on these small datasets, but as an example BertGCN is getting over 89% accuracy (HTM in [1] gets 83%).
It's possible (likely!) some of this gain is from the better representation Transformers use, not just the model.
> definitely not usable for SIGINT-sized intel projects
If SIGINT in this context means signal intelligence (on text data) then I assure you that they are being used.
We have seen significantly higher accuracy than that, more than competitive with the transformer approach we’ve been running in parallel. For arbitrary texts we’ve seen accuracies of at least 92%.
> If SIGINT in this context means signal intelligence (on text data) then I assure you that they are being used.
Maybe, I don’t know what projects you’ve been involved with. For the terabit-level pre-sorting of SIGINT data they’re absolutely definitely not used. If at all on the selected information of interest. My information concerns intel actors in Europe.
Because for text they work a lot better.
HTMs are competitive with other non-NN techniques on small datasets (see [1] you listed) but nothing particularly amazing.
I'd speculate this is because they use bag-of-word variants like LSI and TF-IDF. Prior to Transformers this was a competitive technique and you could get state-of-the-art results on most things using similar techniques.
This representation of data matters a lot. Even just switching to a word embedding representation and using a SVM or something gives a decent gain in most circumstances.
But transformers are much better, particularly on harder tasks (eg question answering on long documents). You can't really see how significant this difference is on these small datasets, but as an example BertGCN is getting over 89% accuracy (HTM in [1] gets 83%).
It's possible (likely!) some of this gain is from the better representation Transformers use, not just the model.
> definitely not usable for SIGINT-sized intel projects
If SIGINT in this context means signal intelligence (on text data) then I assure you that they are being used.
[1] https://arxiv.org/pdf/2112.14820.pdf
[2] https://paperswithcode.com/paper/bertgcn-transductive-text-c...