Navigation X
ALERT
Click here to register with a few steps and explore all our cool stuff we have to offer!



   716

What language would be the most effective for Scraping?

by JerryKillmer - 06 July, 2024 - 02:18 PM
This post is by a banned member (JerryKillmer) - Unhide
14
Posts
6
Threads
1 Year of service
#1
Hello! Im currently working on a project, this project takes live results in the form of text files and .xlsx as long as some others. Now the issue isn't actually getting those done and read I figured that out but often it requires too much memory or some other issue comes up and my Python script is forced to end early so I've been forced to add a cooldown and make my script work in batches. Any ideas and what language would actually be best to do this with? 

Any help and advice is appreciated (if your not retarded commenting for karma lol)

Cheers!
This post is by a banned member (JerryKillmer) - Unhide
14
Posts
6
Threads
1 Year of service
Bumped #2
This is a bump
This post is by a banned member (UberFuck) - Unhide
UberFuck  
Godlike
1.557
Posts
375
Threads
5 Years of service
#3
Most languages should be fine with scraping data. 20 years ago, Perl and Python were the heavy lifters search engines used for scraping data. Today, Google bots are written in C++. The specific language doesn't really matter if it's written poorly.

Have you tried improving your memory management & garbage collection? Are you using any kind of 3rd party libraries in processing? Beautiful Soup, Numpy, Pandas?
This post is by a banned member (LogKing) - Unhide
LogKing  
Registered
9
Posts
3
Threads
#4
Maybe your script is just written poorly with memory management? 
I personally recommend NodeJS there are many many many libraries for scrapping. Also a tip for automation and scrapping would be the "Puppeteer" Library.

Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
or
Sign in
Already have an account? Sign in here.


Forum Jump:


Users browsing this thread: 1 Guest(s)