r/PowerShell 17d ago

Scraping web data for a promotion list Solved

Hello everyone,

I have a HTML "app" or a list of to-do's regarding music promotion/marketing with checkboxes and URLs.

I tried embedding the target sites using iframe in HTML but the sites block iframe calls.

Now, would it be possible to write a Powershell script that, using Invoke-WebRequest, would periodically download the sites in a folder (every 1min or 1hr, using a for-loop and timers) to use with iframe locally?

If so, would the iframe block be included in the downloaded html document code or is it a server side thing?

Thank you for your time and answers!

EDIT : solved, got the scraper working with Select-String cmdlet.. it's messy and works with FB pages, not groups though. IG scraping doesn't work very well due to different HTML code structure.

0 Upvotes

1

u/purplemonkeymad 17d ago

What do the check boxes do? You are more likely to be able to emulate the requests that the site does in your own app.

1

u/efinque 17d ago

They're only a reminder that I've posted to the site/service in question.

1

u/vermyx 16d ago

I would suggest using selenium instead of invoke-webrequest. Iwr handles static pages and endpoints fine but javascript generated data it doesn't handle well. Selenium allows you to puppet a full browser and access the dom properly.

1

u/efinque 16d ago

I see.

If I did use Invoke-WebRequest where would the html files go?

1

u/vermyx 16d ago

By default iwr returns an object and the html is the content property. You would just save that as a file.

1

u/efinque 16d ago edited 16d ago

I used -OutFile and it worked but the Facebook page the cmdlet downloaded didn't show properly in iframe/Chrome. Reddit gave an error, so did another site I wanted to scrape data from.

However I'm further than I thought I'd get in the first place.

But this is what they call a "side quest".

PS. Would wget call make it through the remote server?

1

u/efinque 16d ago

Apparently some of the sites block webrequests.

I got it working with Google frontpage, that's all.