chen*gdeMacBook-Pro:groceryList Mch$
sudo pip3 install beautifulsoup4
The directory '/Users/Mch/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/Users/Mch/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting beautifulsoup4
Could not fetch URL https://pypi.python.org/simple/beautifulsoup4/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590) - skipping
Could not find a version that satisfies the requirement beautifulsoup4 (from versions: )
No matching distribution found for beautifulsoup4
Solution:
* reinstall python？
brew reinstall python
==> Reinstalling python
==> Installing dependencies for python: gdbm
==> Installing python dependency: gdbm
==> Downloading https://homebrew.bintray.com/bottles/gdbm-1.17.el_capitan.bottle.tar.gz
######################################################################## 100.0%
==> Pouring gdbm-1.17.el_capitan.bottle.tar.gz
/usr/local/Cellar/gdbm/1.17: 20 files, 586.2KB
==> Installing python
==> Downloading https://homebrew.bintray.com/bottles/python-3.7.0.el_capitan.bottle.1.tar.gz
######################################################################## 100.0%
==> Pouring python-3.7.0.el_capitan.bottle.1.tar.gz
==> /usr/local/Cellar/python/3.7.0/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/p
==> /usr/local/Cellar/python/3.7.0/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/p
==> /usr/local/Cellar/python/3.7.0/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/p
==> Caveats
Python has been installed as
/usr/local/bin/python3 Unversioned symlinks `python`, `python-config`, `pip` etc. pointing to
`python3`, `python3-config`, `pip3` etc., respectively, have been installed into
/usr/local/opt/python/libexec/bin If you need Homebrew's Python 2.7 run
brew install python@2 Pip, setuptools, and wheel have been installed. To update them run
pip3 install --upgrade pip setuptools wheel You can install Python packages with
pip3 install <package>
They will install into the site-package directory
/usr/local/lib/python3.7/site-packages See: https://docs.brew.sh/Homebrew-and-Python
==> Summary
/usr/local/Cellar/python/3.7.0: 4,787 files, 102MB
==> Caveats
==> python
Python has been installed as
/usr/local/bin/python3 Unversioned symlinks `python`, `python-config`, `pip` etc. pointing to
`python3`, `python3-config`, `pip3` etc., respectively, have been installed into
/usr/local/opt/python/libexec/bin If you need Homebrew's Python 2.7 run
brew install python@2 Pip, setuptools, and wheel have been installed. To update them run
pip3 install --upgrade pip setuptools wheel You can install Python packages with
pip3 install <package>
They will install into the site-package directory
/usr/local/lib/python3.7/site-packages See: https://docs.brew.sh/Homebrew-and-Python
ouput
* install pip3
brew install pip3
Error: No available formula with the name "pip3"
==> Searching for a previously deleted formula (in the last month)...
Error: No previously deleted formula found.
==> Searching for similarly named formulae...
Error: No similarly named formulae found.
==> Searching taps...
==> Searching taps on GitHub...
Warning: Error searching on GitHub: GitHub
The GitHub credentials in the macOS keychain may be invalid.
Clear them with:
printf "protocol=https\nhost=github.com\n" | git credential-osxkeychain erase
Or create a personal access token:
https://github.com/settings/tokens/new?scopes=gist,public_repo&description=Homebrew
and then set the token as: export HOMEBREW_GITHUB_API_TOKEN="your_new_token"
Error: No formulae found in taps.
* Clear The GitHub credentials in the macOS keychain
printf "protocol=https\nhost=github.com\n" | git credential-osxkeychain erase
* create a personal access token
https://github.com/settings/tokens/new?scopes=gist,public_repo&description=Homebrew
Generate token
.btn { position: relative; display: inline-block; padding: 6px 12px; font-size: 14px; font-weight: 600; line-height: 20px; white-space: nowrap; vertical-align: middle; cursor: pointer; -webkit-user-select: none; -moz-user-select: none; -ms-user-select: none; user-select: none; background-repeat: repeat-x; background-position: -1px -1px; background-size: 110% 110%; border: 1px solid rgba(27, 31, 35, 0.2); border-radius: 0.25em; -webkit-appearance: none; -moz-appearance: none; appearance: none }
.btn-primary { color: rgba(255, 255, 255, 1); background-color: rgba(40, 167, 69, 1); background-image: linear-gradient(-180deg, rgba(52, 208, 88, 1) 0, rgba(40, 167, 69, 1) 90%) }
* Click Button "Generate token"
https://github.com/settings/tokens
把这个40位的token粘贴出来
* emacs ~/.bash_profile
echo 'export HOMEBREW_GITHUB_API_TOKEN="your_token"' >> ~/.bash_profile
. !$
* 再试安装beautifulsoup4
pip3 install beautifulsoup4
Collecting beautifulsoup4
Downloading https://files.pythonhosted.org/packages/fe/62/720094d06cb5a92cd4b3aa3a7c678c0bb157526a95c4025d15316d594c4b/beautifulsoup4-4.6.1-py3-none-any.whl (89kB)
100% |████████████████████████████████| 92kB 180kB/s
Installing collected packages: beautifulsoup4
Successfully installed beautifulsoup4-4.6.1
* upgrade pip
sudo curl https://bootstrap.pypa.io/get-pip.py | python3
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/local/bin/pip'
Consider using the `--user` option or check the permissions.
sudo chown -hR `whoami`:staff /usr/local/bin/
output:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1604k 100 1604k 0 0 443k 0 0:00:03 0:00:03 --:--:-- 445k
Collecting pip
Using cached https://files.pythonhosted.org/packages/5f/25/e52d3f31441505a5f3af41213346e5b6c221c9e086a166f3703d2ddaf940/pip-18.0-py2.py3-none-any.whl
Installing collected packages: pip
Found existing installation: pip 18.0
Uninstalling pip-18.0:
Successfully uninstalled pip-18.0
Successfully installed pip-18.0
* 在引用bs4 “frombs4 import BeautifulSoup”时还会报错“ModuleNotFoundError: Nomodule named 'bs4'.” 未找到名为bs4的模块，这时需要在Pycharm上安装bs4模块来解决
按Cmd(win) + , (windows图标键 和 逗号 一起按)或者点 Pycharm => Preferences
windows下 File => settings
* 点击 右边框 左下角 + install
bs 再点击 "Install Package"
这行代码不报错了
from bs4 import BeautifulSoup
* 创建python文件
#-*- coding: UTF-8 -*-
#!/usr/local/bin/python3 from urllib.request import urlopen
from bs4 import BeautifulSoup html = urlopen("http://pythonscraping.com/pages/page1.html")
bsObj = BeautifulSoup(html.read())
print(bsObj.h1)
Run:
/Users/Mch/PycharmProjects/BeautifulSoup/venv/bin/python
/Users/Mch/PycharmProjects/BeautifulSoup/index.py
/Users/Mch/PycharmProjects/BeautifulSoup/index.py:8:
UserWarning: No parser was explicitly specified,
so I'm using the best available HTML parser for this system ("html.parser").
This usually isn't a problem, but if you run this code on another system,
or in a different virtual environment,
it may use a different parser and behave differently. The code that caused this warning is on line 8 of the file
/Users/Mch/PycharmProjects/BeautifulSoup/index.py.
To get rid of this warning,
pass the additional argument 'features="html.parser"' to the BeautifulSoup constructor. bsObj = BeautifulSoup(html.read())
<h1>An Interesting Title</h1> Process finished with exit code 0
* 这里有一个警告️第8行出错
根据提示
To get rid of this warning，pass the additional argument 'features="html.parser"' to the BeautifulSoup constructor.
找到这个构造函数在文件/Users/Mch/PycharmProjects/BeautifulSoup/venv/lib/python3.7/site-packages/bs4/__init__.py
def __init__(self, markup="", features=None, builder=None,
parse_only=None, from_encoding=None, exclude_encodings=None,
**kwargs):
因此在第8行添加参数 'html.parser'
bsObj = BeautifulSoup(html.read(), 'html.parser')
再运行：
/Users/Mch/PycharmProjects/BeautifulSoup/venv/bin/python /Users/Mch/PycharmProjects/BeautifulSoup/index.py
<h1>An Interesting Title</h1>
Process finished with exit code 0
warning没有了
* try ... except... else
#-*- coding: UTF-8 -*-
#!/usr/local/bin/python
from urllib.request import urlopen
from bs4 import BeautifulSoup from urllib.request import HTTPError
# import socket
# socket.setdefaulttimeout(3) try:
# html = urlopen("http://pythonscraping.com/pages/page1.html", None, 3)
# 设置超时时间为3s
html = urlopen("http://nosuchurl", None, 3)
except HTTPError as e:
print(e)
else:
bsObj = BeautifulSoup(html.read(), 'html.parser')
print(bsObj.body.div)
* 可靠的网络连接
#-*- coding: UTF-8 -*-
#!/usr/local/bin/python
from urllib.request import urlopen
from urllib.request import HTTPError, URLError
from bs4 import BeautifulSoup def getTitle(url):
try:
html = urlopen(url, None, 3)
except(HTTPError, URLError) as e:
return None
try:
bsObj = BeautifulSoup(html.read(), "html.parser")
title = bsObj.body.h1
except AttributeError as e:
return None
return title title = getTitle("http://www.pythonscraping.com/pages/page1.html")
if title == None:
print("Title could not be found")
else:
print(title)
Run:
python3 ./index.py
<h1>An Interesting Title</h1>
* python库下载
https://www.lfd.uci.edu/~gohlke/pythonlibs/