--- name: spacy-nlp-de6ac2e0c533 description: "Natural language processing with spaCy. Use when: (1) named entity recognition, (2) POS tagging and dependency parsing, (3) text tokenization and linguistic analysis, (4) rule-based pattern matching, (5) custom NER pipelines. NOT for: LLM inference or text generation (use transformers), sentiment analysis at scale (use dedicated models), or machine translation." metadata: { "openclaw": { "emoji": "📝", "requires": { "bins": ["python3"] }, "install": [{ "id": "uv-spacy", "kind": "uv", "package": "spacy" }] } } --- # spaCy NLP Natural language processing using spaCy for tokenization, named entity recognition, dependency parsing, and linguistic analysis. ## When to Use - Named entity recognition (NER) on text - Part-of-speech tagging and morphological analysis - Dependency parsing and syntactic analysis - Rule-based pattern matching in text - Custom NER pipeline creation - Tokenization and sentence segmentation - Lemmatization and linguistic feature extraction ## When NOT to Use - LLM inference or text generation (use transformers/huggingface) - Sentiment analysis at scale (use fine-tuned classifiers) - Machine translation (use dedicated MT models) - Topic modeling (use gensim or sklearn) - Simple regex-only text search (use re module) ## Setup and Model Download ```bash # Download models (run once before using) python3 -m spacy download en_core_web_sm # small, fast, ~12MB python3 -m spacy download en_core_web_md # medium with word vectors, ~40MB python3 -m spacy download en_core_web_trf # transformer-based, most accurate ``` ## Basic Pipeline ```python import spacy # Load a model nlp = spacy.load("en_core_web_sm") # Process text doc = nlp("Apple is looking at buying U.K. startup for $1 billion.") # Token-level attributes for token in doc: print(f"{token.text:12} {token.pos_:6} {token.dep_:10} {token.lemma_}") # Apple PROPN nsubj Apple # is AUX aux be # looking VERB ROOT look # ... ``` ## Named Entity Recognition ```python nlp = spacy.load("en_core_web_sm") doc = nlp("Apple Inc. was founded by Steve Jobs in Cupertino, California in 1976.") # Extract entities for ent in doc.ents: print(f"{ent.text:25} {ent.label_:10} {ent.start_char}-{ent.end_char}") # Apple Inc. ORG 0-10 # Steve Jobs PERSON 26-36 # Cupertino, California GPE 40-61 # 1976 DATE 65-69 # Common entity labels: PERSON, ORG, GPE, DATE, MONEY, PRODUCT, EVENT, LOC # Get explanation of labels print(spacy.explain("GPE")) # "Countries, cities, states" ``` ## Dependency Parsing ```python nlp = spacy.load("en_core_web_sm") doc = nlp("The quick brown fox jumps over the lazy dog.") # Dependency tree for token in doc: print(f"{token.text:10} --{token.dep_:10}--> {token.head.text}") # Noun chunks (base noun phrases) for chunk in doc.noun_chunks: print(f"{chunk.text:25} root={chunk.root.text}, head={chunk.root.head.text}") # The quick brown fox root=fox, head=jumps # the lazy dog root=dog, head=over # Find subject and object of a verb for token in doc: if token.dep_ == "nsubj": print(f"Subject: {token.text} of verb: {token.head.text}") if token.dep_ == "dobj": print(f"Object: {token.text} of verb: {token.head.text}") ``` ## Pattern Matching ```python from spacy.matcher import Matcher, PhraseMatcher nlp = spacy.load("en_core_web_sm") # Token-based pattern matching matcher = Matcher(nlp.vocab) # Pattern: adjective followed by one or more nouns pattern = [{"POS": "ADJ"}, {"POS": "NOUN", "OP": "+"}] matcher.add("ADJ_NOUN", [pattern]) doc = nlp("The bright blue sky and cold winter morning greeted us.") matches = matcher(doc) for match_id, start, end in matches: span = doc[start:end] print(f"Match: {span.text}") # Match: bright blue sky # Match: cold winter morning # Phrase matching (exact phrase lookup, very fast) phrase_matcher = PhraseMatcher(nlp.vocab, attr="LOWER") terms = ["machine learning", "deep learning", "natural language processing"] patterns = [nlp.make_doc(term) for term in terms] phrase_matcher.add("TECH_TERMS", patterns) doc = nlp("This paper covers machine learning and natural language processing.") matches = phrase_matcher(doc) for match_id, start, end in matches: print(f"Found: {doc[start:end].text}") ``` ## Custom Entity Rules ```python from spacy.pipeline import EntityRuler nlp = spacy.load("en_core_web_sm") # Add entity ruler before the NER component ruler = nlp.add_pipe("entity_ruler", before="ner") patterns = [ {"label": "DRUG", "pattern": "aspirin"}, {"label": "DRUG", "pattern": [{"LOWER": "vitamin"}, {"LOWER": "d"}]}, {"label": "DISEASE", "pattern": "diabetes"}, {"label": "DISEASE", "pattern": [{"LOWER": "heart"}, {"LOWER": "disease"}]}, ] ruler.add_patterns(patterns) doc = nlp("The patient takes aspirin daily for heart disease prevention.") for ent in doc.ents: print(f"{ent.text:20} {ent.label_}") # aspirin DRUG # heart disease DISEASE ``` ## Sentence Segmentation and Text Processing ```python nlp = spacy.load("en_core_web_sm") doc = nlp("Dr. Smith went to Washington. He arrived on Monday. It was cold.") # Sentence boundaries for sent in doc.sents: print(f"[{sent.start}:{sent.end}] {sent.text}") # Lemmatization tokens_lemmatized = [token.lemma_ for token in doc if not token.is_stop and not token.is_punct] # Filter by POS nouns = [token.text for token in doc if token.pos_ == "NOUN"] verbs = [token.text for token in doc if token.pos_ == "VERB"] # Similarity (requires md or lg model with vectors) nlp_md = spacy.load("en_core_web_md") doc1 = nlp_md("I like cats") doc2 = nlp_md("I love dogs") print(f"Similarity: {doc1.similarity(doc2):.3f}") ``` ## Best Practices 1. Use `en_core_web_sm` for speed; use `en_core_web_trf` when accuracy matters most. 2. Process text in batches with `nlp.pipe(texts)` for better throughput. 3. Disable unused pipeline components: `nlp.select_pipes(enable=["ner"])`. 4. Use `PhraseMatcher` for exact term lookups; it is much faster than token `Matcher`. 5. Add `EntityRuler` before `ner` to give rule-based patterns priority. 6. Use `doc.to_json()` to serialize processed documents for storage. 7. For large texts, increase `nlp.max_length` or split into paragraphs first. 8. Always download the model before first use: `python3 -m spacy download `. --- Source attribution: beita6969 Original project: https://github.com/beita6969/ScienceClaw/blob/e925f732c7ca435f557e88075bc47a58f1e984f1/skills/spacy-nlp/SKILL.md License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Mega AI Hub. ## Upstream license notice MIT License Copyright (c) 2025 Peter Steinberger Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Mega AI Hub resource metadata Creator / publisher: beita6969 Original project: https://github.com/beita6969/ScienceClaw/blob/e925f732c7ca435f557e88075bc47a58f1e984f1/skills/spacy-nlp/SKILL.md Attribution basis: Source repository record Fields of work: Data & Analytics, Design, Development & Engineering, Education & Research, Marketing Roles: Content Marketer, Data Scientist, ML / AI Engineer, Researcher, UX Designer Specialties: API Tools: Spacy Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: 5d30ed748f8a7f4fdbd5f2512e5eed15e4ec43c34d8f4a9e3ad8d5835c232af3