One question, did you try to replicate the other result table (Table 3)?
If I understand correctly, top-2 accuracy would be 1 if you have only 2 classes, but it will differ from "normal" accuracy less and less as the number of classes increases (on average). So this shouldn't change the results for table 3 thaaat much as the datasets have large amounts of classes (see table 1).
In any case, top-2 accuracy of 0.685 for the 20-newsgroups dataset is pretty neat for a method that doesn't even consider characters as characters[1], let alone tokens, n-grams, embeddings and all the nice stuff that those of use working on NLP have been devoting years to.
[1] In my understanding of gzip, it considers only bit sequences, which are not necessarily aligned with words (aka. bytes).
I haven't yet replicated Table 3 because most of those datasets are much larger and it will take awhile to run (they said the YahooAnswers database took them 6 days).
Also, I have only tried the "gzip" row because that is all that is in the github repo they referenced.
Yeah, you're right, the more classes there are, probably the lower the effect this will have.
One question, did you try to replicate the other result table (Table 3)?
If I understand correctly, top-2 accuracy would be 1 if you have only 2 classes, but it will differ from "normal" accuracy less and less as the number of classes increases (on average). So this shouldn't change the results for table 3 thaaat much as the datasets have large amounts of classes (see table 1).
In any case, top-2 accuracy of 0.685 for the 20-newsgroups dataset is pretty neat for a method that doesn't even consider characters as characters[1], let alone tokens, n-grams, embeddings and all the nice stuff that those of use working on NLP have been devoting years to.
[1] In my understanding of gzip, it considers only bit sequences, which are not necessarily aligned with words (aka. bytes).