[Python] 크롤링 Bookmarks

 

 

https://pythondocs.net/selenium/%EC%85%80%EB%A0%88%EB%8B%88%EC%9B%80-%ED%81%AC%EB%A1%A4%EB%9F%AC-%EA%B8%B0%EB%B3%B8-%EC%82%AC%EC%9A%A9%EB%B2%95/

셀레니움 크롤러 기본 사용법 - 뻥뚫리는 파이썬 코드 모음

 

 

 

https://sjwiq200.tistory.com/11

[PYTHON] Selenium 에서 헤더 User-Agent 값 수정하기

 

from selenium import webdriver
import Config

options = webdriver.ChromeOptions()
# options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko")

driver = webdriver.Chrome(executable_path=Config.CONFIG['CHROMEPATH'],options=options)

driver.get('')

 

 

 

 

https://github.com/liamcryan/iherb

GitHub - liamcryan/iherb: Get iherb products and details

 

 

 

 

 

 

 

 

 

 

https://hashcode.co.kr/questions/15732/%EC%9B%B9%EC%82%AC%EC%9D%B4%ED%8A%B8-%ED%81%AC%EB%A1%A4%EB%A7%81%ED%95%98%EB%A0%A4%EB%8A%94%EB%8D%B0-html-error-%EB%95%8C%EB%AC%B8%EC%97%90-%EC%A7%84%ED%96%89%EC%9D%B4-%EC%95%88%EB%90%98%EB%84%A4%EC%9A%94

웹사이트 크롤링하려는데 html error 때문에 진행이 안되네요..

 

import urllib.request
from bs4 import BeautifulSoup

url = 'https://kr.iherb.com/search?kw=21st%20century'
html = urllib.request.urlopen(url).read()
soup = BeautifulSoup(html, 'html.parser')

address = soup.find_all(class_='absolute-link product-link')

for i in address:
    print(i.attrs['href'])
    print()

 

url = 'https://somewhere.com'

request = urllib.request.Request(url)
request.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:99.0) Gecko/20100101 Firefox/99.0')

html = urllib.request.urlopen(request).read()

 

 

 

 

 

 

 

 

 

https://gogl3.github.io/articles/2021-03/webcrawling

Web-crawling using Python