This project is a Python script to scrape Indeed.com job listings for Front-end developer jobs in Lagos, Nigeria.
It loads the first page of results using Selenium, extracts job posting URLs, then fetches & parses each job detail page using requests and BeautifulSoup.
✅ Scrapes job listings from Indeed Nigeria ✅ Searches for a specific job title & location (default: JavaScript Developer, Lagos) ✅ Extracts and prints:
- Job title
- Location
- Company name
- Job posting URL ✅ Configurable: change job title & location by editing the URL in the script
| Component | Technology |
|---|---|
| Language | Python 3 |
| Web Automation | Selenium |
| HTML Parsing | BeautifulSoup4 |
| HTTP Requests | requests |
webscraper.py– main script to scrape job listingswebscraper2.py– alternate version (if applicable)
- Python 3.6+
- Google Chrome
- ChromeDriver (compatible with your Chrome version)
1️⃣ Clone the repository:
git clone https://github.com/notghettolenny/Web-Scraper.git
cd Web-Scraper2️⃣ Install Python dependencies:
pip install selenium beautifulsoup4 requests3️⃣ Download ChromeDriver and note its path.
Edit webscraper.py and replace the driver = webdriver.Chrome(...) line with your local path to chromedriver.
Run the scraper:
python webscraper.pyThe script will: ✅ Open the Indeed search page for JavaScript Developer in Lagos ✅ Collect links to all jobs on the first page ✅ Visit each job’s page & print its details
To change the job title and location:
- Edit the
job_listURL at the top of the script:
job_list = "https://ng.indeed.com/jobs?q=your+job+title&l=your+location"For example:
job_list = "https://ng.indeed.com/jobs?q=data+analyst&l=Abuja"- The script only scrapes the first page of search results.
- Make sure your
chromedriverversion matches your installed Chrome version. - Use responsibly and respect Indeed’s terms of service.
This project is licensed under the MIT License.
Pull requests and suggestions are welcome. Please open an issue first to discuss what you’d like to change.