// article
A pipeline that reads Chamber of Commerce directories with a local 7B-parameter model instead of a CSS selector per site. No API costs, and nothing leaves the machine.
lead-gen-pipeline is a Python tool that extracts business records from Chamber of Commerce directories. Each chamber site is laid out differently (by category, by letter, by page), so brittle CSS selectors don’t work. The pipeline asks a local 7B-parameter model (Qwen2-7B-Instruct, through llama-cpp-python) to read the page and return structured JSON instead.
The trade is straightforward: one LLM call replaces a custom adapter per site. In a development run against the Palo Alto Chamber of Commerce it extracted 296 businesses across 26 categories in about 9 minutes. Nothing leaves the local machine.
Source: github.com/Burton-David/lead-gen-pipeline
Three reasons, in order of how much they actually matter:
0.0 plus a strict prompt produces consistent JSON. The model is good at structured extraction; it doesn’t need to be creative.Chamber URL → find directory links → extract listings → follow pagination → dedup → SQLiteThe pieces that matter here:
llm_processor.py sends preprocessed page text to the model, parses its JSON, and repairs malformed output.chamber_parser.py walks a chamber’s directory pages.crawler.py fetches pages with retry and backoff, identifies itself with its own User-Agent, and honors robots.txt.bulk_database.py normalizes records, drops duplicates, and writes them in batches through SQLAlchemy.Instead of a CSS selector that breaks the moment the chamber redesigns, the pipeline asks the model to find the data. This is the extraction prompt, verbatim:
def _create_business_extraction_prompt(self, page_content: str) -> str:
return f"""Extract business data from this chamber directory page.
JSON only - no extra text.
Page content:
{page_content}
JSON format:
{{
"business_listings": [
{{
"name": "Business Name",
"website": "http://example.com",
"phone": "555-1234",
"email": "contact@example.com",
"address": "123 Main St, City, State",
"industry": "Business Category"
}}
],
"pagination": {{
"next_page_url": null,
"has_more": false
}},
"total_found": 0
}}
Use null for missing fields. Extract all businesses found."""The prompt asks for the next page’s URL in the same response, so pagination falls out of extraction instead of needing its own rules. Generation runs at the configured temperature, which defaults to 0.0:
async def _generate(self, prompt: str) -> str:
return await asyncio.to_thread(
self.backend.generate,
prompt,
max_tokens=self.settings.MAX_TOKENS,
temperature=self.settings.TEMPERATURE,
)llama-cpp-python is synchronous, so the call runs in a thread and the crawler keeps fetching while the model works.
Chamber sites organize their directories by category, by letter, or by page, and the member directory is rarely linked from the same place twice. Rather than hardcode a rule per layout, the first step asks the model which links on the chamber’s main page lead to the directory:
def _create_directory_navigation_prompt(self, page_content: str) -> str:
return f"""\
Analyze this Chamber of Commerce page and find business directory links.
Respond with valid JSON only - no markdown, no explanations.
Page content:
{page_content}
Look for: "Members", "Directory", "Businesses", "Member Directory", etc.
JSON format:
{{
"navigation_links": ["url1", "url2"],
"confidence": 85
}}
Empty array if no directory links found."""Relative links in the answer are joined against the page URL before the crawler follows them.
The LLM returns plausible records; the pipeline has to make them trustworthy.
Keep a record only if it names a business or a website. Every field is stripped, and empty strings become None, so a half-empty row can’t slip through as blanks:
record = {
field: (str(business.get(field) or "").strip() or None)
for field in ("name", "website", "phone", "email", "address", "industry")
}
record["source_url"] = url
if record["name"] or record["website"]:
cleaned.append(record)Dedup on website plus name. Two chambers listing the same business, or one directory listing it under two categories, hash to the same key:
def _create_business_hash(self, business_data: dict[str, Any]) -> str:
"""Create hash for business deduplication."""
website = (business_data.get("website") or "").strip().lower()
company_name = (
(business_data.get("company_name") or business_data.get("name") or "")
.strip()
.lower()
)
hash_input = f"{website}|{company_name}"
return hashlib.md5(hash_input.encode("utf-8")).hexdigest()Normalize phone numbers with phonenumbers. The single-page scraper parses phone text into E.164 and drops anything that isn’t a valid number, rather than hand-formatting digit strings:
number = phonenumbers.parse(phone_text, self.default_region)
if phonenumbers.is_valid_number(number):
return phonenumbers.format_number(number, PhoneNumberFormat.E164)Row-at-a-time inserts to SQLite are the slow way to persist a crawl. bulk_database.py normalizes and deduplicates in memory, then writes in batches of up to 1,000 rows, updating existing leads instead of duplicating them. On a laptop that runs at roughly 1,300 deduplicating upserts per second, which you can reproduce with python scripts/benchmark_bulk_db.py 5000. At the size of one chamber it barely matters; at any larger scale it’s the one change that pays back the most.
The single-page extractor needs no model:
git clone https://github.com/Burton-David/lead-gen-pipeline
cd lead-gen-pipeline
python3 -m venv .venv && source .venv/bin/activate
pip install -e .
lead-gen test https://www.python.orgThe chamber crawl needs the model and a headless browser:
pip install -e ".[llm,browser]"
playwright install chromium
lead-gen setup-llm # downloads the Qwen2-7B-Instruct GGUF (~4 GB)
lead-gen chambers --url https://www.examplechamber.com
lead-gen export -o leads.csvThese numbers come from a single development run against the Palo Alto Chamber of Commerce, made before the codebase was rebuilt. They describe that run, not a benchmark, and the current code hasn’t been re-run live on this path yet.
Per-page timing in that run:
The model was not the bottleneck. Network was. That suggests running chambers in parallel would scale close to linearly, since the model has headroom while the next request is in flight.
A few things worth doing next: