Header Ads

Web scraping Python Html Table to JSON / Dict Project with Code

In this post, we'll see how to create Web scraping Python Projects with code. So, we are going to Scrap Covid-19 statistics data and Convert the scraped Html table to python dict/json using Beautifulsoup, List Comprehension, Python Requests and lxml library.   


If you like videos like this consider donating $1, or simply turn off AdBlocker. Either helps me to continue making tutorials.



Code below follows the video to help :
Transcript / Cheat Sheet : 

Step 1: Search for the Required Link:

So searching a relevant link plays a important role in our Web scraping Python Projects as whole code depends on the link to be scraped. If in later stage you decides the change the link then there will lot of mess. 
Best Practices :
  • Authenticity of Link to be scraped.
  • Data you wanna scrap shouldn't be in encrypted form in Inspect Element/ Page Source.
  • Site shouldn't block you for scraping (major issue) : If site blocks you need to find a workaround by modifying the requests with User Agent and Adding Proxy.
Step 2: Analyze the Data :

Once we got required link, next step is to analyze the classes, tags and id's where our data resides. For this post we are scraping the data from Html table, so we are interested in finding the id attached to html table. Later we're going to use this Html table id for filtering out the table from whole page source and Convert the Html Table to JSON / Python Dict using list comprehension and beautifulsoup.

Step 3: Install the required Libraries:

  • pip install requests
  • pip install bs4
  • pip install lxml
Step 3: Get your Hands dirty with Coding : 
import requests
import bs4

page=requests.get("https://covid-19.hackanons.com/test.html")

soup= bs4.BeautifulSoup(page.text,'lxml')

table=soup.find('table',id="main_table_countries_today")

headers=[ heading.text.replace(",Other","") for heading in table.find_all('th')]

table_rows=[ row for row in table.find_all('tr')]

results=[{headers[index]:cell.text for index,cell in enumerate(row.find_all("td")) }for row in table_rows]

for i in results:
    if "Country" in i:
        if i["Country"]=="India":
            print(i)

6 comments:

  1. Need The To Hire A Hacker❓ Then contact PYTHONAX✅

    The really amazing deal about contacting PYTHONAX is that the Hack done by us can’t get traced to you, as every Hacking job we do is strongly protected by our Firewall. It’s like saying if anyone tries to trace the Hack, it will lead them to us and we block whatever actions they are doing.

    We have been Invisible to Authorities for almost a decade now and if you google PYTHONAX, not really about us comes out, you can only see comments made by us or about us.

    Another Amazing thing to you benefit from Hiring our Hackers is that you get a Legit and the best Hacking service, As we provide you with Professional Hackers who have their Hacking Areas of specialization.
    We perform every Hack there is, using special Hacking tools we get from the dark web.

    Some list of Hacking Services we provide are-:
    ▪️Phone Hacking & Cloning ✅
    ▪️Computer Hacking ✅
    ▪️Emails & Social Media Account Hacking✅
    ▪️Recovering Deleted Files✅
    ▪️Tracking & Finding People ✅
    ▪️Hunting Down Scammers✅
    ▪️Hack detecting ✅
    ▪️Stealing/Copying Files & Documents From Restricted Networks and Servers ✅
    ▪️Bitcoin Multiplication✅
    ▪️Binary Option Money Recovery ✅
    ▪️Forex Trading Money Recovery✅
    ▪️IQ Option Money Recovery✅
    And lots more......


    Whatever Hacking service you require, just give us an Email to the Emails Address provided below.
    pythonaxhacks@gmail.com
    pythonaxservices@gmail.com

    PYTHONAX.
    2020 © All Right Reserved.

    ReplyDelete
    Replies
    1. Hello all
      am looking few years that some guys comes into the market
      they called themselves hacker, carder or spammer they rip the
      peoples with different ways and it’s a badly impact to real hacker
      now situation is that peoples doesn’t believe that real hackers and carder scammer exists.
      Anyone want to make deal with me any type am available but first
      I‘ll show the proof that am real then make a deal like

      Available Services

      ..Wire Bank Transfer all over the world

      ..Western Union Transfer all over the world

      ..Credit Cards (USA, UK, AUS, CAN, NZ)

      ..School Grade upgrade / remove Records

      ..Spamming Tool

      ..keyloggers / rats

      ..Social Media recovery

      .. Teaching Hacking / spamming / carding (1/2 hours course)

      discount for re-seller

      Contact: 24/7

      fixitrogers@gmail.com

      Delete
  2. I know a professional hacker named ethicalhackers009@gmail.com who has worked for me this week. He offers very legitimate services such as clearing of bad records online without being traced back to you, He clone/hack mobile phones, hack Facebook account, instagram, WhatsApp, emails, Twitter, bank accounts, Skype, FIXES CREDIT REPORTs, track calls. He also help retrieve accounts that have been taking by hackers. His charges are affordable, reliable and 100% safe. For his job well done this is my own way to show appreciation, Contact him via address below...
    Email...ethicalhackers009@gmail. com
    text or call +16692252253
    whatsapp +12132951376
    http://hackintechnology.com/

    ReplyDelete
  3. Typically the RATES won’t give appeal to on just about any too much taxes repayments, so you are actually consuming it again with the dirt bike pants through not necessarily transforming ones own place a burden on payments. https://royalcbd.com/faq/

    ReplyDelete
  4. HOW I GOT MY LOAN FROM THIS GREAT COMPANY

    Hello my dear people, I am Linda McDonald, currently living in Austin Texas, USA. I am a widow at the moment with three kids and i was stuck in a financial situation in April 2018 and i needed to refinance and pay my bills. I tried seeking loans from various loan firms both private and corporate but never with success, and most banks declined my credit ,do not full prey to those hoodlums at there that call them self-money lender they are all scam , all they want is your money and you well not hear from them again they have done it to me twice before I met Mr. David Wilson the most interesting part of it is that my loan was transfer to me within 74hours so I will advise you to contact Mr. David if you are interested in getting loan and you are sure you can pay him back on time you can contact him via email……… (davidwilsonloancompany4@gmail.com) No credit check, no cosigner with just 2% interest rate and better repayment plans and schedule if you must contact any firm with reference to securing a loan without collateral then contact Mr. David Wilson today for your loan

    They offer all kind of categories of loan they

    Short term loan (5_10years)
    Long term loan (20_40)
    Media term loan (10_20)
    They offer loan like
    Home loan............., Business loan........ Debt loan.......
    Student loan.........., Business startup loan
    Business loan......., Company loan.............. etc
    Email..........( davidwilsonloancompany4@gmail.com)
    When it comes to financial crisis and loan then David Wilson loan financial is the place to go please just tell him I Mrs. Linda McDonald direct you Good Luck.......................

    ReplyDelete
  5. Hello all
    am looking few years that some guys comes into the market
    they called themselves hacker, carder or spammer they rip the
    peoples with different ways and it’s a badly impact to real hacker
    now situation is that peoples doesn’t believe that real hackers and carder scammer exists.
    Anyone want to make deal with me any type am available but first
    I‘ll show the proof that am real then make a deal like

    Available Services

    ..Wire Bank Transfer all over the world

    ..Western Union Transfer all over the world

    ..Credit Cards (USA, UK, AUS, CAN, NZ)

    ..School Grade upgrade / remove Records

    ..Spamming Tool

    ..keyloggers / rats

    ..Social Media recovery

    .. Teaching Hacking / spamming / carding (1/2 hours course)

    discount for re-seller

    Contact: 24/7

    fixitrogers@gmail.com

    ReplyDelete

Powered by Blogger.