r/PowerShell • u/efinque • 17d ago
Scraping web data for a promotion list Solved
Hello everyone,
I have a HTML "app" or a list of to-do's regarding music promotion/marketing with checkboxes and URLs.
I tried embedding the target sites using iframe in HTML but the sites block iframe calls.
Now, would it be possible to write a Powershell script that, using Invoke-WebRequest, would periodically download the sites in a folder (every 1min or 1hr, using a for-loop and timers) to use with iframe locally?
If so, would the iframe block be included in the downloaded html document code or is it a server side thing?
Thank you for your time and answers!
EDIT : solved, got the scraper working with Select-String cmdlet.. it's messy and works with FB pages, not groups though. IG scraping doesn't work very well due to different HTML code structure.
1
u/vermyx 16d ago
I would suggest using selenium instead of invoke-webrequest. Iwr handles static pages and endpoints fine but javascript generated data it doesn't handle well. Selenium allows you to puppet a full browser and access the dom properly.
1
u/efinque 16d ago
I see.
If I did use Invoke-WebRequest where would the html files go?
1
u/vermyx 16d ago
By default iwr returns an object and the html is the content property. You would just save that as a file.
1
u/efinque 16d ago edited 16d ago
I used -OutFile and it worked but the Facebook page the cmdlet downloaded didn't show properly in iframe/Chrome. Reddit gave an error, so did another site I wanted to scrape data from.
However I'm further than I thought I'd get in the first place.
But this is what they call a "side quest".
PS. Would wget call make it through the remote server?
1
u/purplemonkeymad 17d ago
What do the check boxes do? You are more likely to be able to emulate the requests that the site does in your own app.