Day 13 : Web Scraping using python (Corona data)

-
Hello guys,
Previously we learn about libraries and oops concepts of python.

Today we will learn about web scraping.

What is Web Scraping ?


Web Scraping (also termed Screen Scraping, Web Data Extraction, Web Harvesting etc.) is a technique employed to extract large amounts of data from websites whereby the data is extracted and saved to a local file in your computer or to a database in table (spreadsheet) format.

Why Web Scraping?

Web scraping is used to collect large information from websites. But why does someone have to collect such large data from websites? To know about this, let’s look at the applications of web scraping:
  • Price Comparison: Services such as ParseHub use web scraping to collect data from online shopping websites and use it to compare the prices of products.
  • Email address gathering: Many companies that use email as a medium for marketing, use web scraping to collect email ID and then send bulk emails.
  • Social Media Scraping: Web scraping is used to collect data from Social Media websites such as Twitter to find out what’s trending.
  • Research and Development: Web scraping is used to collect a large set of data (Statistics, General Information, Temperature, etc.) from websites, which are analyzed and used to carry out Surveys or for R&D.
  • Job listings: Details regarding job openings, interviews are collected from different websites and then listed in one place so that it is easily accessible to the user.

There are two types of websites
1 Static website
2 Dynamic Website

To scrap static websites we use two libraries
1 beautifulsoup
2 requests

How to use?
install libraries

pip install beautifulsoup4

pip install requests

Let's have some fun

We will scrap https://www.worldometers.info/coronavirus/  this website 


This website gives information about corona


Code:
import requests as rq
from bs4 import BeautifulSoup


data = rq.get("https://www.worldometers.info/coronavirus/")

soup = BeautifulSoup(data.text, 'html.parser')
count = soup.find_all("div", class_="maincounter-number")
print(counts)


Output:

[<div class="maincounter-number">
<span style="color:#aaa">1,361,598 </span>
</div>, <div class="maincounter-number">
<span>76,315</span>
</div>, <div class="maincounter-number" style="color:#8ACA2B ">
<span>293,654</span>
</div>]


So whats going on here,

1. import requests as rq 
The requests module allows you to send HTTP requests using Python.
The HTTP request returns a Response Object with all the response data (content, encoding, status, etc).

2 from bs4 import BeautifulSoup.

Beautiful Soup is a Python library for pulling data out of HTML and XML files. It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree.



3. data = rq.get("https://www.worldometers.info/coronavirus/")

get method Sends a GET request to the specified url and retrun to data


4. if we print(data) that will return <Response [200]>.


5.soup = BeautifulSoup(data.text, 'html.parser')

Beautiful Soup gives us a BeautifulSoup object, which represents the document as a nested data structure

basically returns HTML page.

6. counts = soup.find_all("div", class_="maincounter-number")

Because find_all() is the most popular method in the Beautiful Soup search API, you can use a shortcut for it. If you treat the BeautifulSoup object or a Tag object as though it were a function, then it’s the same as calling find_all() on that object. These two lines of code are equivalent:

find_all(tag,attr) : here we specify div tag and attribute class with value "maincounter-number"

returns list of attributes.


7. print(counts)

Here we print list

Now if we want that actual count not that HTML stuff div then 

for count in counts:
print(count.get_text())


1,362,045 


76,328


293,655



get_text() : return text within HTML tags.

There are such functions like get_link() etc


Quite Fun and Looks like magic😎

Task :
Scrap other such websites




❤❤Quarantine python group link ❤❤

8805271377 WhatsApp

Follow here ❤

@mr._mephisto_ Instagram 

There will be no restrictions just feel free to learn. 

Share and take one more step to share knowledge to others. 

Believe in yourself 🤟 you are awesome. 

Be safe, Be happy😁
Take care of yourself and your family 
Of course watch movies and series🤟😉 

And follow the guidelines of government



Comments

Post a Comment

Popular posts from this blog

Day 16 : Pandas Basics

-
Hey guys, previously we learned about turtle graphics design. Today we will learn about the library pandas which is used for data manipulation and cleaning. What is pandas?  NO not forest or zoo panda. Pandas is a high-level data manipulation tool developed by Wes McKinney. It is built on the Numpy package and its key data structure is called the DataFrame. DataFrames allow you to store and manipulate tabular data in rows of observations and columns of variables. Why Pandas are used? Pandas  is  used   for data manipulation, analysis, and cleaning.  Python pandas  is well suited for different kinds of data, such as Tabular data with heterogeneously-typed columns. Ordered and unordered time series data. How to use Pandas? Pandas is not built-in library so you have to install it. pip install pandas Let's play with pandas How to import? >>> import pandas as pd Same as we import all other libraries Code: >>> dic={...
Read more

News website using Flask and news API - Working with templates : Part 5

-
Image
Hey guys, Today we will display data on the webpage from API. app.py from flask import Flask, render_template,url_for, request, redirect import requests as rq # intialise data of lists. app = Flask(__name__) @app.route('/') def index():               api_key="Your_API_Key"     url="http://newsapi.org/v2/top-headlines?country=in&apiKey="+api_key     data=rq.get(url)     data=data.json()     articles=data["articles"]       return render_template('index.html',articles=articles) if __name__=="__main__":     app.run() render_template -  Generating HTML from within Python is not fun, and actually pretty cumbersome because you have to do the HTML escaping on your own to keep the application secure. Because of that Flask configures the  Jinja2   template engine for you automatically. To render a template you can use the  render_template() ...
Read more

Day 9 : Encapsulation and polymorphism

-
Image
Hello Guys, previously we learn about constructor and inheritance and basics of python. Today we learn about  encapsulation and polymorphism. Encapsulation Encapsulation is   wrapping data and the methods that work on data within one unit.  This puts restrictions on accessing variables and methods directly and can prevent the accidental modification of data. To prevent accidental change, an object’s variable can only be changed by an object’s method. Those type of variables are known as  private varibale . Protected members Protected members (in C++ and JAVA) are those members of the class which cannot be accessed outside the class but can be accessed from within the class and it’s subclasses. Public, private and protected keyword But To accomplish this  in Python, just follow  the convention  by prefixing the name of the member by a  single underscore “_” . Code : class Base:    ...
Read more