[Python from zero to one] 9. Detailed explanation of Selenium basic technology of web crawler (positioning elements, common methods, keyboard and mouse operation)

Welcome everyone to "Python from zero to one", here I will share about 200 Python series articles, take everyone to learn and play, and see the interesting world of Python. All articles will be explained in combination with cases, codes and the author's experience. I really want to share my nearly ten years of programming experience with everyone. I hope it will be helpful to you. Please also Haihan for the inadequacies in the article. The overall framework of the Python series includes 10 basic grammar, 30 web crawlers, 10 visual analysis, 20 machine learning, 20 big data analysis, 30 image recognition, 40 artificial intelligence, 20 Python security, and 10 other skills . Your attention, likes and reposts are your greatest support for Xiuzhang. Knowledge is priceless and people are affectionate. I hope we can all be happy and grow together on the road of life.

The previous article described database operation knowledge, including MySQL installation, SQL statements and Python operation database knowledge, which will lay the foundation for subsequent web crawlers to store in the database. This article introduces Selenium basic technology in detail, involving basic introduction, element positioning, common methods and attributes, mouse operation, keyboard operation and navigation control. Basic articles, I hope to help you.

Article Directory

download link:

Appreciation of the previous article:

The first part of basic grammar

The second part of the web crawler

The author's newly opened "Nazhang AI Security Home" will focus on Python and security technology, mainly sharing Web penetration, system security, artificial intelligence, big data analysis, image recognition, malicious code detection, CVE reproduction, threat intelligence analysis, etc. article. Although the author is a technical novice, he will ensure that every article will be written with great care. I hope that these basic articles will be helpful to you and make progress with everyone on the road to Python and safety.

Selenium is a classic tool for testing Web applications. It runs directly in the browser, as if a real user is operating the browser. It is mainly used for website automation testing, website simulation login, automatic keyboard and mouse operation, and testing. Browser compatibility, testing website functions, etc., can also be used to make simple web crawlers.

This article mainly introduces the Selenium Python API technology, which uses a very intuitive way to access all the functions of Selenium WebDriver, including positioning elements, automatically operating keyboard and mouse, submitting page forms, capturing required information, etc.

1. Getting to know Selenium for the first time

Selenium is an acceptance test tool specially written for Web applications by ThoughtWorks. The API it provides supports multiple languages, including Python, Java, C#, etc. This book mainly introduces Selenium technology under the Python environment. The Python language provides the Selenium extension package, which is an API interface that uses Selenium WebDriver (web page driver) to write functions and verify tests.

Through the Selenium Python API, readers can access all the functions of Selenium WebDriver in an intuitive way. Selenium Python supports a variety of browsers, such as Chrome, Firefox, IE, 360 and other browsers, and also supports PhantomJS special interfaceless browser engine.

Insert picture description here

The Selenium WebDriver API interface provides a strategy for locating elements in web pages (Locate Elements). This book will use Selenium Python to explain the knowledge of web data crawling. This chapter mainly introduces the basic knowledge of Selenium technology. The following chapters will explain how to use Selenium with examples. Positioning webpage elements, automatically crawling, designing crawlers, etc.

Similar to the BeautifulSoup technology, the crawler made by Selenium first analyzes the HTML source code and DOM tree structure of the web page, and then locates the node location of the required information through the methods it provides, and obtains its text content.

At the same time, readers are recommended to read the "Selenium with Python Bindings" open source technical document provided by the official website. This article has also learned a lot of its wonderful knowledge, combined with their own understanding and actual crawler examples to introduce. The following introduces the knowledge of Selenium installation, driver installation, and PhantomJS three parts, let's get started!


1. Install Selenium

Readers can visit the PyPI website to download the Selenium extension package, such as selenium 3.4.3 provided in Figure 2. The corresponding URL is:

  • https://pypi.python.org/pypi/selenium

We click the "Downloads" button to download the Selenium extension package, after decompressing the downloaded file, execute the following command in the decompression directory to install the Selenium package.

C:\selenium\selenium3.4.3> python3 setup.py install

The full name of PyPI is Python Package Index, which is Python's official third-party library warehouse. Everyone can download third-party libraries or upload libraries developed by themselves to PyPI.

Insert picture description here

At the same time, the author recommends that you use the pip tool to install the Selenium library, and PyPI officially recommends using the pip manager to download third-party libraries. Python 3.6 standard library comes with pip, and Python 2.x needs to be installed separately. The previous article introduced the installation process and basic usage of the pip tool. After installing the pip tool, directly call the command to install Selenium:

  • pip install selenium

Call the command "pip install selenium" to install the Selenium package as shown in Figure 3.

Insert picture description here

During the installation process, the percentage of packages related to the installation configuration will be displayed until the "Successfully installed selenium-2.47.1" prompt appears, indicating that the installation is successful, as shown in Figure 4.

Insert picture description here

At this point, the Selenium package has been installed successfully, and then the browser needs to be called to locate or crawl information, and the browser driver needs to be installed during the process of using the browser. The author recommends using Firefox browser, Chrome browser or PhantomJS browser. The following will explain the configuration process of the three browser drivers with examples.


2. Install the browser driver

Selenium needs to install a browser driver to call the browser for automatic crawling or automatic testing. Common browsers include Chrome, Firefox, IE, and PhantomJS. Table 1 is part of the browser driver download page.

Table 1 Browser driver download page

Insert picture description here

Note: After downloading and decompressing the driver, place chromedriver.exe, geckodriver.exe, and Iedriver.exe in the Python installation directory. For example, if the Python installation directory is "C:\python", then place the driver file in this folder ; Then add the Python installation directory to the system environment variable path (Path), open Python IDLE and enter different codes to start different browsers.

  • Firefox browser
    The core code to load the Firefox browser is as follows:
from selenium import webdriver
driver = webdriver.Firefox()
driver.get('http://www.baidu.com/')

The output result is shown in the figure below:

Insert picture description here
  • Chrome browser
    The core code to load Google browser is as follows, and the driver is placed in the chrome browser directory, as shown in the code.
import os 
from selenium import webdriver
chromedriver = "C:\Program Files (x86)\Google\Chrome\Application\chromedriver.exe"  
os.environ["webdriver.chrome.driver"] = chromedriver 
browser = webdriver.Chrome(chromedriver)
browser.get('http://www.baidu.com/')
  • IE browser
    The core code to load Microsoft IE browser is as follows:
from selenium import webdriver
browser = webdriver.Ie()
browser.get('http://www.baidu.com/')

3.Phantomjs

PhantomJS is an open source browser engine (WebKit) with a server-side JavaScript API. It supports various web standards, including DOM tree analysis, CSS selectors, JSON and SVG, etc. PhantomJS is often used for page automation, network monitoring, web page screenshots, and interfaceless testing. Download PhantomJS from the official website http://phantomjs.org/ and decompress it as shown in Figure 5.

Insert picture description here

If an error "Unable to start phantomjs with ghostdriver" is reported when calling, you need to set the path of PhantomJS or configure it in the Scripts directory environment. When Selenium is successfully installed and PhantomJS is downloaded and configured, the following code is the calling method. The executable_path parameter sets the path of PhantomJS.

from selenium import webdriver
driver = webdriver.PhantomJS(executable_path="F:\phantomjs-1.9.1-windows\phantomjs.exe")
driver.get("http://www.baidu.com")
data = driver.title
print(data)

The meaning of the code is:

  • First import the Selenium.webdriver extension package, which provides webdriver implementation methods.
  • Then create a driver instance and call the webdriver.PhantomJS method to configure the path.
  • Open the Baidu webpage through the driver.get("http://www.baidu.com") code. Webdriver will wait for the page elements to load before returning control to the script.
  • Finally, get the title of the article and assign it to the data variable output, the value of which is "just click on Baidu, you will know".

The running result is shown in Figure 6, and Python3 has the same effect.

Insert picture description here

Note that the save_sceenshot() function provided in webdriver can take screenshots of web pages. The code is as follows:

from selenium import webdriver
driver = webdriver.Firefox()
driver.get("http://www.baidu.com")
data = driver.title
driver.save_screenshot('baidu.png')
Insert picture description here

2. Quickly start Selenium analysis

Web pages are usually stored in a document object model tree structure, and these nodes appear in pairs, such as "< html >" corresponding to "</ html >", "< table >" corresponding to "</ table >", "< div >" corresponds to "</ div >" and so on. Selenium technology can determine the location of the current node by locating specific attributes of the node, such as class, id, name, etc., and then obtain information about the relevant web page.

The following code is to locate the Baidu search box and perform automatic search. It serves as our quick start code.

#-*- coding:utf-8 -*-
#By:Eastmount 2021-05-29
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

#启动驱动
driver = webdriver.Firefox()
driver.get("http://www.baidu.com")
assert "百度" in driver.title
print(driver.title)

#查找元素并输入内容
elem = driver.find_element_by_name("wd")
elem.send_keys("数据分析")
elem.send_keys(Keys.RETURN)

#截图并退出
time.sleep(10)
driver.save_screenshot('baidu.png')
driver.close()
driver.quit()

The result of the operation is shown in the figure below. Invoke the Firefox browser and search for the "data analysis" keyword, and finally take a screenshot of the browsed webpage. Therefore, Selenium is often used in the field of automated testing.

Insert picture description here

This part of the code is explained in detail below.

  • from selenium import webdriver
    Import the Selenium.webdriver template, which provides webdriver implementation methods. Currently, browsers that support these methods include Firefox, Chrome, IE, and Remote.
  • from selenium.webdriver.common.keys import Keys
    Import the Keys class, which provides shortcut keys for operating the keyboard, such as the enter key, the space bar, and the ctrl key.
  • driver = webdriver.Firefox()
    Create a Firefox webdriver instance, define the Firefox browser (Firefox) driver, other browsers such as Chrome may also need to set the driver parameters and configuration path.
  • driver.get("http://www.baidu.com")
    Next, open the Baidu URL webpage through the driver.get() function, and webdriver will wait for the webpage elements to load before returning control to the script.
  • assert "Baidu" in driver.title
    Next, use an assert to determine whether the title of the article contains the "Baidu" field. The title of the corresponding crawl is "Baidu, you know", so "Baidu" is included, otherwise an assertion error will appear. Assertions are mainly used to determine whether the result is returned successfully, so as to better perform the next positioning operation.
  • elem = driver.find_element_by_name("wd")
    Webdriver provides many methods like "find_element_by_*" to match the element to be found. For example, the method of using the name attribute to find is find_element_by_name. Here, this method is used to locate the Baidu input box, that is, to review the node whose element name is "wd".

Figure 8 is the feedback result of Baidu homepage review elements. The corresponding attribute name of the input element of the input box is "kw", so the node code for positioning is:

  • driver.find_element_by_id("kw")
Insert picture description here
  • elem.send_keys("Data Analysis")
    The send_keys() method can be used to simulate keyboard operations, which is equivalent to entering the "data analysis" field in the search box.
  • elem.send_keys(Keys.RETURN)
    Call the send_keys() function to input the enter key operation. The Keys class provides common keyboard keys, such as Keys.RETURN for the enter key. But before referencing the Keys class and its methods, you need to import the Keys class first, that is, use the "from selenium.webdriver.common.keys import Keys" code to import.
  • driver.save_screenshot('baidu.png')
    Call the save_screenshot() function to take a screenshot, and save the screenshot locally with the name "baidu.png".
  • driver.close()
    Call the close() method to close the driver.
  • driver.quit()
    Call the quit() method to exit the driver. The difference between it and the close() method is that the quit() method will exit the browser, while the close() method just closes the page, but if only one page is opened, the close() method will also exit the browser.

3. Positioning elements

Selenium Python provides a strategy for locating elements. You can choose the most suitable solution according to the HTML structure of the crawled webpage. Table 8.2 shows the various methods provided by Selenium. When positioning multiple elements, simply add s after the method "element", and these elements will be returned in the form of a list.

Table 2 Selenium element positioning method

Insert picture description here

This section will be explained in conjunction with the following HTML code (blog09.html) about Li Bai's introduction.

<html>
	<head>
		<title>李白简介</title>
	</head>
	<body>
	<p class="title"><b>静夜思</b></p>
	<p class="content">
		窗前明月光,<br />
		疑似地上霜。 <br />
		举头望明月,<br />
		低头思故乡。 <br />
	</p>
	<div class="other" align="left" name="d1" id="nr">
		李白(701年-762年),字太白,号青莲居士,又号“谪仙人”,
		唐代伟大的浪漫主义诗人,被后人誉为“诗仙”,与
	  <a href="http://test.com/dufu" class="poet" id="link" name="dufu">
杜甫</a>
		并称为“李杜”,为了与另两位诗人
	  <a href="http://test.com/lsy" class="poet" id="link" name="lsy">
李商隐</a>、
	  <a href="http://test.com/dumu" class="poet" id="link" name="dumu">
杜牧</a>
即“小李杜”区别,杜甫与李白又合称“大李杜”。
		其人爽朗大方,爱饮酒...
    </div>
	<p class="story">...</p>
</body>
</html>

The webpage opens and runs as shown in Figure 9 below.

Insert picture description here

In the following, we will introduce various element positioning methods in combination with this example, and focus on positioning a single element.


1. Locate elements by ID

This method is to locate the element through the id attribute of the web page tag, and it will return the first element that matches the positioning with the id attribute value. If no element matches the id value, a NoSuchElementException will be returned.
Assuming that the three hyperlinks Du Fu, Li Shangyin, and Du Mu in the page need to be located through the id attribute, the core HTML code is as follows:

Insert picture description here

If you need to get the div layout, use the following code:

  • test_div = driver.find_element_by_id('nr')
  • print(test_div.text)

If written as the following code, it returns the information of the first poet.

  • test_poet = driver.find_element_by_id('link')
  • print(test_poet.text)
  • Du Fu

Among them, test_poet is the obtained value, usually in the form of "<selenium.webdriver...>", and text is to obtain the text content, that is, "Du Fu". If you want to obtain multiple links through the id element, such as the hyperlinks corresponding to the three poets of Du Fu, Li Shangyin, and Du Mu, you need to use:

  • find_elements_by_id()

Note that "elements" means to get multiple values. The three hyperlinks all use the same id name "link". After positioning and obtaining through the find_elements_by_id() function, call the for loop to output the result, as shown below:

#-*- coding:utf-8 -*-
#By:Eastmount 2021-05-29
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

#启动驱动
driver = webdriver.Firefox()
driver.get("file://C:/Users/xiuzhang/Desktop/09.selenium/blog09.html")
print(driver.title)

#查找元素并输入内容
test_div = driver.find_elements_by_id('link')
for t in test_div:
    print(t.text)

The output result is shown in the figure below:

Insert picture description here

2. Locate elements by Name

This method is to locate the element through the name attribute of the web page tag, and it will return the first element that matches the location with the value of the name attribute. If no element matches the name value, a NoSuchElementException will be returned.

The following describes how to locate the three hyperlinks of Du Fu, Li Shangyin, and Du Mu in the page through the name attribute. The HTML source code is as follows:

<div class="other" align="left" name="d1" id="nr">
<a href="http://test.com/dufu" class="poet" id="link" name="dufu">杜甫</a>
	<a href="http://test.com/lsy" class="poet" id="link" name="lsy">李商隐</a>
	<a href="http://test.com/dumu" class="poet" id="link" name=”dumu”>杜牧</a>
</div>

If you need to obtain the three hyperlinks of Du Fu, Li Shangyin, and Du Mu respectively, use the following code:

test_poet1 = driver.find_element_by_name('dufu')
test_poet2 = driver.find_element_by_name('lsy')
test_poet3 = driver.find_element_by_name('dumu')

At this time, the find_elements_by_name() function cannot be called to obtain multiple elements, because the name attributes of the hyperlinks corresponding to the three poets are different, namely "dufu", "lsy", and "dumu". If the name attributes are the same, this method can Get multiple elements of the same name attribute.


3. Locate elements through XPath

XPath is a technology used to locate nodes in XML documents. HTML\XML are all written using the structure of the DOM tree tags of the web page, so the node information can be analyzed through the XPath method. Selenium Python also provides a similar method to track elements in web pages.

The XPath method of locating elements is different from the method of locating elements according to ID or Name attributes. The former is more flexible and convenient. For example, we want to locate the hyperlink information of the third poet "Du Mu" through the ID attribute, but the ID attribute value of the three poets is the same, that is, "link". If there are no other attributes, how can we achieve it? At this point, you can use the XPath method to locate the element. This also reflects an advantage of the XPath method:

  • When there is no suitable ID or Name attribute to locate the element you are looking for, you can use XPath to locate the absolute element (but the author does not recommend locating an absolute element), or locate a relative element with ID or Name attribute.

The XPath method can also locate elements by other attributes besides ID and Name attributes. The complete function is:

  • find_element_by_xpath()
  • find_elements_by_xpath()

Let's start to explain through examples, the HTML code is as follows:

<html>
	<head>
		<title>李白简介</title>
	</head>
	<body>
	<div class="other" align="left" name="d1" id="nr">
		李白(701年-762年),字太白,号青莲居士,又号“谪仙人”,
		唐代伟大的浪漫主义诗人,被后人誉为“诗仙”,与
	  <a href="http://test.com/dufu" class="poet" id="link1" namd="dufu">
杜甫</a>
		并称为“李杜”,为了与另两位诗人
	  <a href="http://test.com/lsy" class="poet" id="link2" namd="lsy">
李商隐</a>、
	  <a href="http://test.com/dumu" class="poet" id="link3" name=”dumu”>
杜牧</a>
即“小李杜”区别,杜甫与李白又合称“大李杜”。
		其人爽朗大方,爱饮酒...
    </div>
</body>
</html>

This div layout may be positioned through the following three XPath methods:

test_div = driver.find_element_by_xpath("/html/body/div[1]")
test_div = driver.find_element_by_xpath("//div[1]")
test_div = driver.find_element_by_xpath("//div[@id='nr']")
  • The first sentence is to use an absolute path to locate the element, starting from the root node of the HTML code, but if the HTML code is slightly changed, the result will be destroyed. At this time, the latter two methods can be used to locate.
  • The second sentence is to get the first div layout element in the HTML code. But if the position of the div node to be crawled is too deep, should we count down from the first div node? Obviously not. At this point, we can locate the desired element by looking for the ID or Name attribute of a nearby element.
  • The third sentence is to call the find_element_by_xpath() method to locate the div layout element whose ID attribute value is "nr". At this time, you can locate the profile information of three famous poets.

The three statements output the content of test_div.text, as shown below:

  • Li Bai (701-762), the word Taibai, was named Qinglianjushi, and also known as "the banished immortal". A great romantic poet in the Tang Dynasty, he was called "the fairy of poetry" by later generations, and he was called "Li Du" together with Du Fu. In order to distinguish them from the other two poets, Li Shangyin and Du Mu, namely "little Li and Du", Du Fu and Li Bai are collectively called "big Li and Du". He is hearty and generous, loves to drink...

If you need to locate the hyperlinked content of the third poet "Du Mu", use the following three methods.

username = driver.find_element_by_xpath("//div[a/@name='dumu']")
username = driver.find_element_by_xpath("//div[@id='nr']/a[3]")
username = driver.find_element_by_xpath("//a[@name='dumu']")
  • The first sentence is to locate a hyperlink a element under the div node, and the name attribute of the a element is "dumu".
  • The second sentence is to locate the "id=nr" div element, and then find its third hyperlink a sub-element.
  • The third sentence is to locate the first hyperlink a element whose name attribute is "dumu".

At the same time, if it is a button control and the name attribute is the same, suppose the HTML code is as follows:

<form id="loginForm">
	<input name="continue" type="submit" value="Login" />
	<input name="continue" type="button" value="Clear" />
</form>

The method to locate the button element with a value of "Clear" is as follows:

clearb = driver.find_element_by_xpath("//input[@name='continue'][@type='button']")
clearb = driver.find_element_by_xpath("//form[@id='loginForm']/input[2]")
  • The first sentence is to locate the input control whose attribute name is "continue" and attribute type is "button".
  • The second sentence is the second input sub-element under the form node of the positioning attribute "id=loginForm".

The XPath positioning method is one of the most commonly used methods for positioning elements. The examples in the following chapters will be used repeatedly. This section only introduces some basic knowledge. For more knowledge, please refer to W3Schools XPath Tutorial, W3C XPath Recommendation or Selenium official Learn from the documentation.


You can use this method when you need to locate the link text (Link Text) in an anchor tag. This method will return the first element that matches the text value of this link. If no element matches the link text, a NoSuchElementException will be thrown. The following describes how to call this method to locate the three hyperlinks of Du Fu, Li Shangyin, and Du Mu on the page, assuming that the HTML source code is as follows:

  • blog09_02.html
<html>
	<body>
      <div class="other" align="left" name="d1" id="nr">
	  <a href="dufu.html" class="poet" id="link" name="dufu">
Dufu</a>
	  <a href="lsy.html" class="poet" id="link" name="lsy">
LiShangYing</a>
	  <a href="dumu.html" class="poet" id="link" name=”dumu”>
DuMu</a>
      </div>
</body>
</html>

If you need to obtain the three hyperlinks of Du Fu, Li Shangyin, and Du Mu respectively, use the following code.

#-*- coding:utf-8 -*-
#By:Eastmount 2021-05-29
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

#启动驱动
driver = webdriver.Firefox()
driver.get("file://C:/Users/xiuzhang/Desktop/09.selenium/blog09_02.html")
print(driver.title)

#分别定位三个超链接
test_poet1 = driver.find_element_by_link_text('Dufu')
print(test_poet1.text)
test_poet2 = driver.find_element_by_link_text('LiShangYing')
print(test_poet2.text)
test_poet3 = driver.find_element_by_link_text('DuMu')
print(test_poet3.text)

#定位超链接部分元素
test_poet4 = driver.find_element_by_partial_link_text('Du')
print(test_poet4.text)

#定位超链接部分元素且定位多个元素
test_poet5 = driver.find_elements_by_partial_link_text('Du')
for t in test_poet5:
    print(t.text)

Among them, the find_element_by_link_text() function uses the link text of the anchor tag for positioning, partial means partial matching, and the method of obtaining multiple elements uses:

  • find_elements_by_partial_link_text()

The code running screenshot is shown in Figure 10, where the address can also be the blog09_02.html file placed in the local Apache server, and the content is the HTML source code above.

  • http://localhost:8080/blog09_02.html
Insert picture description here

5. Locate elements by tag name

This method is to locate the element by the tag name (Tag Name), and it will return the first element that is matched and positioned with the Tag Name. If no elements match, a NoSuchElementException will be returned. Suppose the HTML source code is as follows:

  • blog09_03.html
<html>
	<head>
		<title>李白简介</title>
	</head>
	<body>
      <h1>静夜思</h1>
      <p class='content'>窗前明月光,疑是地上霜。举头望明月,低头思故乡。</p>
</body>
</html>

The method of positioning element h1 and paragraph p is as follows:

  • test1 = driver.find_element_by_tag_name('h1')
  • test2 = driver.find_element_by_tag_name('p')

6. Locate elements by class name

This method is to locate the element by the Class Attribute Name, and it will return the first element that matches the location with the Class Attribute Name. If no elements match, a NoSuchElementException will be returned.

In blog09_03.html code, the method of locating the paragraph p element by the class attribute value is as follows:

  • test1 = driver.find_element_by_class_name('content')

7. Positioning elements through CSS selectors

The method is to locate the element through CSS Selectors, and it will return the first element that matches the CSS selector. If no elements match, a NoSuchElementException will be returned. In blog09_03.html code, the method to locate the paragraph p element through the CSS selector is as follows:

  • test1 = driver.find_element_by_css_selector('p.content')

If there are multiple content tags with the same class, you can use the following methods to locate and obtain:

  • test1 = driver.find_element_by_css_selector(*.content)
  • test2 = driver.find_element_by_css_selector(.content)

The CSS selector positioning method is a more difficult method. It is recommended that readers study it on their own. At the same time, the author recommends that you use common positioning methods such as ID, Name, and XPath.


Four. Common methods and attributes

1. Method of operating elements

After describing the Locate Elements, we need to operate on the objects that have been positioned. The interactive behavior of these operations usually needs to be implemented through the WebElement interface. Common operating element methods are shown in Table 3.

Table 3 Common operating element methods

Insert picture description here

The following author gives an example of automatically logging in to the Baidu homepage to explain commonly used operation element methods, including clear(), send_keys(), click(), submit() and other methods.

First, we open the Baidu homepage through the Firefox browser, find the "login" button, and right-click on the "review element", you can see the HTML source code corresponding to the "login" button on the Baidu homepage as shown in Figure 11.

Insert picture description here

The "Login" button node is actually a hyperlink whose name value is "tj_login". We can locate this node through the following code, and then call the click() function to automatically click on it and jump to the login page.

  • login = driver.find_element_by_name("tj_login")
  • login.click()
Insert picture description here

The new version of Baidu has added the option of "Username Login", we need to further capture the location and click.

Insert picture description here

After clicking the button, the pop-up interface is shown in Figure 13. Next, you need to analyze the HTML source code of the user name and password, and find the node location to realize the automatic login operation.

Insert picture description here

Then review the login page to obtain the "username" and "password" elements, the corresponding HTML core code is as follows:

<input id="TANGRAM__PSP_10__userName" type="text" value="" 
	autocomplete="off" class="pass-text-input pass-text-input-userName"
	name="userName" placeholder="手机/邮箱/用户名"></input>
<input id="TANGRAM__PSP_10__password" type="password" value="" 
	class="pass-text-input pass-text-input-password"  
	name="password" placeholder="密码"></input>

Locate the element through find_element_by_name(), call the function clear() to clear the default content of the input box, such as "please enter the password" and other prompts, and call the send_keys() function to enter the correct user name and password and click login. The core code is as follows:

name = driver.find_element_by_name("userName")
name.send_keys("admin")  
pwd = driver.find_element_by_name("password")
pwd.send_keys("123456")
pwd.send_keys(Keys.RETURN)
Error Prompt
When automatically logging in to the Baidu homepage, the error "selenium.common exceptions ElementNotInteractable Exception: could not be scrolled into view" may be prompted. This is because in some cases, the visibility of the element is hidden or the display attribute is none, we are in There are some hidden elements on the page that are not visible on the page but actually exist. At this time, use is_displayed() to judge and set the time to wait.

Insert picture description here

The complete code is as follows:

#-*- coding:utf-8 -*-
#By:Eastmount CSDN 2021-05-29
import time
from selenium import webdriver  
from selenium.webdriver.common.keys import Keys  
from selenium.webdriver.common.action_chains import ActionChains

#打开浏览器
driver = webdriver.Firefox()  
driver.get("https://www.baidu.com/")
time.sleep(1)

#点击登录链接
logins = driver.find_elements_by_name("tj_login")
for login in logins:
    print(login.text)
    print(login.get_attribute('href'))
    if login.is_displayed():
        login.click()
time.sleep(1)

#通过二次定位寻找用户名登录按钮
uesrlogins = driver.find_elements_by_xpath("//div[@class='tang-pass-footerBar']/p")
for uesrlogin in uesrlogins:
    print(uesrlogin.text)
    if uesrlogin.is_displayed():
        uesrlogin.click()

#输入密码并登陆
name = driver.find_element_by_name("userName")
name.clear
name.send_keys("Eastmount")     
pwd = driver.find_element_by_name("password")
pwd.clear
pwd.send_keys("12345678")

#暂停输入验证码 按回车键登录
time.sleep(5)
pwd.send_keys(Keys.RETURN)
driver.close()         

Note: If you need to enter a verification code during the login process, use the time.sleep(5) pause function. After manually entering the verification code "report", the program will execute the send_keys(Keys.RETURN) function, and enter the enter key to realize Baidu.com automatic log in.

Insert picture description here

Finally, this part of the code will automatically enter the specified user name and password, and then enter the Enter key to achieve the login operation. However, it should be noted that since some pages are dynamically loaded, the nodes may not be captured in actual operation. At the same time, the HTML source code of Baidu web pages will change from time to time, but the knowledge of the principles is more important. I hope readers can master similar analysis methods. When we crawl Weibo, Zhihu, Station B and other cases later, we will also explain the automatic login crawler in detail with examples.

Insert picture description here

2. WebElement common attributes

Commonly used values ​​can be obtained through the WebElement interface, among which common attribute values ​​are shown in the following table.

Table 4 Commonly used attributes

Insert picture description here

This part of the code is as follows:

#-*- coding:utf-8 -*-
#By:Eastmount CSDN 2021-05-29
import time
from selenium import webdriver  
from selenium.webdriver.common.keys import Keys 

driver = webdriver.Firefox()  
driver.get("https://www.baidu.com/")

print(driver.title)
print(driver.current_url)
# 百度一下,你就知道
# https://www.baidu.com/

news = driver.find_element_by_xpath("//div[@id='u1']/a[1]")
print(news.text)
print(news.get_attribute('href'))
print(news.location)
# 新闻
# http://news.baidu.com/
# {'y': 19.0, 'x': 456.0}

The output result is shown in the figure below:

Insert picture description here
  • driver.title is the title of the output webpage "Just click on Baidu, you know", driver.current_url outputs the hyperlink of the current page;
  • Then use the find_element_by_xpath("//div[@id='u1']/a[1]") function to locate the "news" link in the upper right corner of the Baidu homepage;
  • Then call the news.text code to output its content;
  • Finally, the get_attribute('href') function is to get the hyperlink, and news.location is to output its webpage coordinate position.

5. Automated keyboard and mouse operation

Another feature of Selenium technology is that it can automate the operation of the mouse and keyboard, so it is more used in the field of automated testing, which detects the robustness and security of the website by automatically operating the webpage and feedback the results of the response.

1. Keyboard operation

In the Webdriver library provided by Selenium, its subclass Keys provides all keyboard key operations, such as Enter, Tab, and Space, as well as some common key combinations, such as Ctrl+A (select all), Ctrl +C (copy), Ctrl+V (paste), etc. Common keyboard operations are as follows:

  • send_keys(Keys.ENTER): Press the enter key, the most commonly used key operation
  • send_keys(Keys.TAB): Press the Tab tab
  • send_keys(Keys.SPACE): Press Space
  • send_keys(Kyes.ESCAPE): Press the back key Esc
  • send_keys(Keys.BACK_SPACE): Press the delete key BackSpace
  • send_keys(Keys.SHIFT): Press the Shift key
  • send_keys(Keys.CONTROL): Press the Ctrl key
  • send_keys(Keys.CONTROL,'a'): Press the key combination to select all Ctrl+A
  • send_keys(Keys.CONTROL,'c'): Press the key combination to copy Ctrl+C
  • send_keys(Keys.CONTROL,'x'): Press the key combination to cut Ctrl+X
  • send_keys(Keys.CONTROL,'v'): Press the key combination to paste Ctrl+V

Here is a simple example of Baidu automatically searching for "Python" keywords, the code is as follows:

#-*- coding:utf-8 -*-
#By:Eastmount CSDN 2021-05-29
from selenium import webdriver  
from selenium.webdriver.common.keys import Keys 

driver = webdriver.Firefox()  
driver.get("https://www.baidu.com/")
elem = driver.find_element_by_id("kw")
elem.send_keys("Python")
elem.send_keys(Keys.RETURN)

First, you need to locate the HTML source code of the Baidu search box. The analysis result is shown in Figure 14. The HTML tag corresponding to the Baidu search box is input and its ID attribute is "kw", so the positioning code is:

  • driver.find_element_by_id("kw")
Insert picture description here

Then call elem.send_keys("Python") to enter the keyword "Pyhon". The code of elem.send_keys(Keys.RETURN) means to enter the enter key, which is equivalent to clicking the "Baidu Click" button. The feedback result is shown in Figure 15.

Insert picture description here

It can also automatically search for the information of the author "Eastmount", haha~

Insert picture description here

2. Mouse operation

Selenium operation mouse technology is also often used in automated testing. It is located in the ActionChains class. The most commonly used is the click() function, which represents the left-click operation of the mouse. Common mouse operations are as follows:

  • click(): Click the left mouse button once
  • context_click(elem): Right-click the mouse and click the element elem, for example, select "Save As" and other commands in the pop-up shortcut menu
  • double_click(elem): click on the element elem
  • drag_and_drop(source,target): Mouse drag operation. Press the left mouse button under the source element source position, and move to the target element target and release the mouse
  • send_keys(Keys.BACK_SPACE): Press the delete key BackSpace
  • move_to_element(elem): move the mouse cursor to the element elem
  • click_and_hold(elem): Press the left mouse button and hover over the element elem
  • perform(): Perform the storage operation in the ActionChains class, and a dialog box will pop up

The following sample code is to locate Baidu's logo image, and then execute the right mouse button to save as image operation.

Insert picture description here

The pop-up dialog box as shown in the figure below, the new version can also save as a webpage when trying to enter the k key.

Insert picture description here

6. Navigation control

The previous section described the Python operating keyboard and mouse. It is recommended that readers must implement this part of the code by themselves, so as to better apply it to actual projects. This section mainly introduces Selenium's navigation control operations, including page interaction, form operations, and movement between dialogs.

1. Drop-down menu interactive operation

The Baidu search case described above is a process of page interaction, including:

  • Call driver.find_element_by_xpath() to locate the element.
  • Call send_keys(key) to enter keywords or keyboard keys, such as Keys.RETURN.
  • Call the click() function and click the left button, right click "Save as Picture" and so on.

Here we will add an example of the switch drop-down menu for page interaction. After locating the "name" drop-down menu label, we call the SELECT class to select the option, and select_by_visible_text() is used to display the selected menu, and the Form can also be submitted.

from selenium.webdriver.support.ui import Select
name = driver.find_element_by_name('name')
select = Select(name)
select.select_by_index(index)
select.select_by_visible_text("text")
select.select_by_value(value)

If the reader wants to cancel the selected option, use the following code:

from selenium.webdriver.support.ui import Select
name = driver.find_element_by_name('name')
select = Select(name)
all_selected_options = select.all_selected_options

To get all the available options, just call select.options. After the readers fill out the form, they can submit the form through the submit() function, or call the following function to submit the form after finding the submit button.

  • driver.find_element_by_id("submit").click()

2. Dialog box moves between Window and Frame

Websites are usually composed of multiple windows, called multi-frame web applications. WebDriver provides a method switch_to_window to support mobile switching between named windows. such as:

  • driver.switch_to_window("windowName")

Now all operations of the driver will be directed to a specific window. But how can we know the name of the window? You can locate the hyperlink in the HTML source code, or pass a "window handle" to the switch_to_window() method. The common method is to loop through all windows and then obtain the specified handle for positioning. The core code is as follows:

for handle in driver.window_handles:
    driver.switch_to_window(handle)

To switch between frame and frame (Iframe), use the driver.switch_to_frame("frameName") function. For pop-up dialogs, Selenium WebDriver provides built-in support. Through the switch_to_alert() function, the currently opened alert object will be returned, through which you can confirm your consent or objection, and you can also read its content.

  • alert = driver.switch_to_alert()

For more knowledge, readers are recommended to read the official document. Below is the core code to capture the content of the pop-up dialog.

#获取当前窗口句柄
now_handle = driver.current_window_handle 
print(now_handle)

#获取所有窗口句柄
all_handles = driver.window_handles 
for handle in all_handles:
    if handle!=now_handle:
        #输出待选择的窗口句柄
        print(handle)
        driver.switch_to_window(handle)
        time.sleep(1)
        #具体操作
        elem_bt = driver.find_element_by_xpath("...")
        driver.close() #关闭当前窗口

#输出主窗口句柄
print(now_handle)
driver.switch_to_window(now_handle) #返回主窗口

Subsequent examples will also introduce a method of window handle escaping.


Seven. Summary

The method of analyzing and locating nodes in the Selenium library is similar to that of the BeautifulSoup library. They can use similar XPath technology to locate tags, and both have rich operation functions to crawl data. But the difference is:

  • Selenium can easily control the keyboard, mouse, switch dialogs, submit forms, etc. When our target web page needs to be authenticated and logged in before crawling, the crawled data is in the pop-up dialog box or the crawled data is passed through the super When the link jumps to the new form, the advantage of Selenium technology is reflected. It crawls data by controlling the mouse to simulate login or submitting the form, but its disadvantage is that the crawling efficiency is lower and the speed of BeautifulSoup is faster.

Selenium is more widely used in automated testing. It runs directly in the browser (such as Firefox, Chrome, IE, etc.), just like real user operations, it performs various tests on the developed webpages. It is more An indispensable tool in the direction of automated testing. I hope readers can master the crawling method of this technology, especially when the target webpage needs to be verified and logged in.

Download address of all codes in this series:

Thanks to those who are on the way to school, you will live up to the encounter, and don't forget your original intention. This week’s message is emotional~

Insert picture description here

(By: Nazhang House Eastmount 2021-05-29 night in Wuhan https://blog.csdn.net/Eastmount )


references