-
wordfreq is “a Python library for looking up the frequencies of words in many languages, based on many sources of data.” Sadly, it’s now longer going to be updated, as the author writes:
I don’t want to be part of this scene anymore: wordfreq used to be at the intersection of my interests. I was doing corpus linguistics in a way that could also benefit natural language processing tools. The field I know as “natural language processing” is hard to find these days. It’s all being devoured by generative AI. Other techniques still exist but generative AI sucks up all the air in the room and gets all the money. It’s rare to see NLP research that doesn’t have a dependency on closed data controlled by OpenAI and Google, two companies that I already despise. wordfreq was built by collecting a whole lot of text in a lot of languages. That used to be a pretty reasonable thing to do, and not the kind of thing someone would be likely to object to. Now, the text-slurping tools are mostly used for training generative AI, and people are quite rightly on the defensive. If someone is collecting all the text from your books, articles, Web site, or public posts, it’s very likely because they are creating a plagiarism machine that will claim your words as its own. So I don’t want to work on anything that could be confused with generative AI, or that could benefit generative AI. OpenAI and Google can collect their own damn data. I hope they have to pay a very high price for it, and I hope they’re constantly cursing the mess that they made themselves.
(tags: ai language llm nlp openai scraping words genai google)