NLP in Sklearn
This will walk you through how to load a dataset, extract the features (use a pipeline), and get a working model. In general, I discussed a few feature extraction methods in the slides, so let's go through the API docs for those.
CountVectorizer Parameters
Bag-of-words models in scikit-learn are called CountVectorizers. Basically, it will count how many unique words you have in the corpus (all your documents), and then each document will be represented as the count of its unique words.
One of the main parameters you will want to set when using the CountVectorizer is the "ngram_range tuple (min_n, max_n), default=(1, 1)" which will decide what counts as a token, one word, two words, etc. You will also want to set max_features, which decides how many words are actually used in your dictionary. Your corpus may have 100,000 unique words, but it could be that 50,000 are only used one or two times, which means they may not provide very useful features. Max_features allows you to set a cut-off so only the more commonly used words are included in the vocabulary.
TF-IDF
In order to perform TF-IDF, you would want to use the TF-IDFTransformer
In order to use TF-IDF, you would first have to run the count-vectorizer to get all the counts of the words, and then TF-IDF will do the necessary calculations on your dictionary.
Pipelines
Finally, let's take a look at our beloved pipelines. If you read the first link on this page (working with text data), you will get a nice tutorial on how to use a pipeline. They are actually quite simple to set up but do take a small amount of getting used to.
Pipelines don't really have any parameters you need to set, but you will want to learn how to access the different steps of the pipeline if you'd like to try different experiments (grid search over a parameter within a pipeline). You will practice this in the labs.
Like the other subjects on this topic, NLP is an ocean of discussion; you can literally take many courses in NLP and get a Ph.D. on any given topic within it. That being said, I will share a few resources I think are quite valuable for you to get started and go deeper if you are interested.
Course Resources
Finally, given that I am writing this during the recent launch of GPT-3 (June 2020), the world's most giant and amazing language model, I would be remiss if I didn't post a few links to good articles describing it in detail.
GPT-3 Links
- Explaining how transformers work
- GPT-3 Critiques / Discussions: Expectations and learning GPT-3
- GPT-3 examples: MiniMaxir and Gwern
Update Feb 2024 Ok, well, look at that! GPT3 doesn't seem like much of anything these days 😁. I'm leaving the links above for historical reasons, I think it's great to look back and see what people thought of LLMs before they were all the rage. Everyone is familiar with ChatGPT, or GPT3.5, the successor to the GPT3 I mentioned above. GPT3 had 175 billion parameters! The good news is that most advanced LLMs today (2024) are lowering that number dramatically. LLama2 (an open-source model by Meta) has three flavors, a 7Billion, 13Billion, and 70Bilion. We are seeing many models do very well with much smaller parameters.
If you want to try out different LLMs and also see which ones are performing the best, I would strongly suggest you check the chatbot arena a place where you have "head to head" battles between LLMs. They are then ranked by user preference, and you can see all kinds of information. You can also try out one of 35 models in their "direct chat" option. The world of LLMs is moving fast, and I am sure this will be out of date in 6 months!
Summary: Python NLP and Sklearn Resources
- Sklearn provides many options for completing NLP tasks
- CountVectorizer can be used to count the unique words in a corpus
- TF-IDF can be used to extract a feature that represents context
- Pipelines are quite easy to set up in sklearn, and you should be using them!