Puppeteer Different Behavior with Browser

By -

I am getting differing results between querySelector commands in the console and the equivalent puppeteer code on nodejs. I'm using headless chromium and puppeteer-extra-plugin-stealth, so it probably isn't captcha or anything. Instead it seems to be some type of ad redirect that is messing everything up.

Using querySelector on the console, I can click on the button I want.

document.querySelectorAll("button[class*=OfferCta__]")[1].click()

This opens a modal on the same page. Sometimes it also opens a new tab with another website.

Unfortunately, when I run the equivalent on puppeteer via nodejs:

    const buttons = await page.$$("button[class*=OfferCta__]");
    await page.waitFor(2000);
    await buttons[1].click();
    await page.waitFor(6000);
    await page.screenshot({ path: "screenshot.png" });

All that shows up is an ad followed by lots of white space and nothing else: enter image description here

The page I'm trying to scrape is https://www.retailmenot.com/view/myntra.com

Edit: Something interesting I discovered in the logs. Every two minutes, puppeteer seems to restart, as seen in the logs. It fails as usual the first few times, but on the fifth restart, ten minutes after I ran the code, it runs correctly, which I denoted with "done retrieving". I have a feeling this means that the success condition, whatever it is, is random?

4:44:33 PM web.1 |  Listening on 5000
4:44:34 PM web.1 |  https://www.retailmenot.com/view/myntra.com
4:44:37 PM web.1 |  done waiting
4:44:43 PM web.1 |  screenshot happening
4:44:43 PM web.1 |  screenshot happened
4:44:43 PM web.1 |  50
4:44:52 PM web.1 |  screenshot happening
4:44:52 PM web.1 |  screenshot happened
4:44:52 PM web.1 |  Error: Error: failed to find element matching selector "div[class^=OfferItemFull__OfferItemWrapper]"
4:44:52 PM web.1 |      at ElementHandle.$eval (/Users/mountaindrake/flock-proj/productPin/node_modules/puppeteer-core/lib/cjs/puppeteer/common/JSHandle.js:647:19)
4:44:52 PM web.1 |      at process._tickCallback (internal/process/next_tick.js:68:7)
4:46:34 PM web.1 |  https://www.retailmenot.com/view/myntra.com
4:46:37 PM web.1 |  done waiting
4:46:43 PM web.1 |  screenshot happening
4:46:43 PM web.1 |  screenshot happened
4:46:43 PM web.1 |  50
4:46:52 PM web.1 |  screenshot happening
4:46:52 PM web.1 |  screenshot happened
4:46:52 PM web.1 |  Error: Error: failed to find element matching selector "div[class^=OfferItemFull__OfferItemWrapper]"
4:46:52 PM web.1 |      at ElementHandle.$eval (/Users/mountaindrake/flock-proj/productPin/node_modules/puppeteer-core/lib/cjs/puppeteer/common/JSHandle.js:647:19)
4:46:52 PM web.1 |      at process._tickCallback (internal/process/next_tick.js:68:7)
4:48:34 PM web.1 |  https://www.retailmenot.com/view/myntra.com
4:48:36 PM web.1 |  done waiting
4:48:43 PM web.1 |  screenshot happening
4:48:43 PM web.1 |  screenshot happened
4:48:43 PM web.1 |  50
4:48:51 PM web.1 |  screenshot happening
4:48:52 PM web.1 |  screenshot happened
4:48:52 PM web.1 |  Error: Error: failed to find element matching selector "div[class^=OfferItemFull__OfferItemWrapper]"
4:48:52 PM web.1 |      at ElementHandle.$eval (/Users/mountaindrake/flock-proj/productPin/node_modules/puppeteer-core/lib/cjs/puppeteer/common/JSHandle.js:647:19)
4:48:52 PM web.1 |      at process._tickCallback (internal/process/next_tick.js:68:7)
4:50:34 PM web.1 |  https://www.retailmenot.com/view/myntra.com
4:50:36 PM web.1 |  done waiting
4:50:43 PM web.1 |  screenshot happening
4:50:43 PM web.1 |  screenshot happened
4:50:43 PM web.1 |  50
4:50:51 PM web.1 |  screenshot happening
4:50:52 PM web.1 |  screenshot happened
4:50:52 PM web.1 |  Error: Error: failed to find element matching selector "div[class^=OfferItemFull__OfferItemWrapper]"
4:50:52 PM web.1 |      at ElementHandle.$eval (/Users/mountaindrake/flock-proj/productPin/node_modules/puppeteer-core/lib/cjs/puppeteer/common/JSHandle.js:647:19)
4:50:52 PM web.1 |      at process._tickCallback (internal/process/next_tick.js:68:7)
4:52:34 PM web.1 |  https://www.retailmenot.com/view/myntra.com
4:52:37 PM web.1 |  done waiting
4:52:43 PM web.1 |  screenshot happening
4:52:43 PM web.1 |  screenshot happened
4:52:43 PM web.1 |  50
4:52:52 PM web.1 |  screenshot happening
4:52:52 PM web.1 |  screenshot happened
4:52:52 PM web.1 |  Error: Error: failed to find element matching selector "div[class^=OfferItemFull__OfferItemWrapper]"
4:52:52 PM web.1 |      at ElementHandle.$eval (/Users/mountaindrake/flock-proj/productPin/node_modules/puppeteer-core/lib/cjs/puppeteer/common/JSHandle.js:647:19)
4:52:52 PM web.1 |      at process._tickCallback (internal/process/next_tick.js:68:7)
4:54:34 PM web.1 |  https://www.retailmenot.com/view/myntra.com
4:54:36 PM web.1 |  done waiting
4:54:42 PM web.1 |  screenshot happening
4:54:43 PM web.1 |  screenshot happened
4:54:43 PM web.1 |  50
4:54:51 PM web.1 |  screenshot happening
4:54:52 PM web.1 |  screenshot happened
4:54:52 PM web.1 |  Error: Error: failed to find element matching selector "div[class^=OfferItemFull__OfferItemWrapper]"
4:54:52 PM web.1 |      at ElementHandle.$eval (/Users/mountaindrake/flock-proj/productPin/node_modules/puppeteer-core/lib/cjs/puppeteer/common/JSHandle.js:647:19)
4:54:52 PM web.1 |      at process._tickCallback (internal/process/next_tick.js:68:7)
4:56:34 PM web.1 |  https://www.retailmenot.com/view/myntra.com
4:56:37 PM web.1 |  done waiting
4:56:43 PM web.1 |  screenshot happening
4:56:43 PM web.1 |  screenshot happened
4:56:43 PM web.1 |  50
4:56:52 PM web.1 |  screenshot happening
4:56:52 PM web.1 |  screenshot happened
4:56:52 PM web.1 |  done retreiving


from Recent Questions - Stack Overflow https://ift.tt/3mpJpnN
https://ift.tt/3jvLuNa

Comments

Post a Comment

Popular posts from this blog

Spring Elasticsearch Operations

By -
Elasticsearch Operations Spring Data Elasticsearch uses two interfaces to define the operations that can be called against an Elasticsearch index. These are ElasticsearchOperations and ReactiveElasticsearchOperations. Whereas the first is used with the classic synchronous implementations, the second one uses reactive infrastructure. The default implementations of the interfaces offer: Read/Write mapping support for domain types. A rich query and criteria api. Resource management and Exception translation. ElasticsearchTemplate The ElasticsearchTemplate is an implementation of the ElasticsearchOperations interface using the Transport Client. @Configuration public class TransportClientConfig extends ElasticsearchConfigurationSupport {   @Bean   public Client elasticsearchClient() throws UnknownHostException {                      Settings settings = Settings.builder().put("cluster.name", "elasticsearch")....
Read more

Network Error and Timeout on Authorize.net JS

By -
i am using authorize.net acceptUI.js in order to create a hosted payment form for credit card acceptance. The problem arising now is that, going on the checkout page, I have the following console message: GET https://js.authorize.net/v3/AcceptUI.js net::ERR_NAME_NOT_RESOLVED Which is weird, cause I cannot ping the domain or anything, as my computer cannot resolve the domain. However, this issue is also reported by individual customers and it seems to be sporadically and not happening the entire time, cause i can see other orders going through. Does anyone know how such an error can be caused, or any ideas how it can be resolved? Thanks for your help EDIT: I just checked the A DNS record for js.authorize.net and am getting the following result only: js.authorize.net. 100 IN CNAME js.authorize.net.cdn.cloudflare.net. Doing the same for js.authorize.net.cdn.cloudflare.net. does not return an A Record, unless I specifically instruct to use 1.1.1.1 as DNS server I...
Read more

Object oriented programming concepts (OOPs)

By -
Object-oriented programming (OOP) is a programming paradigm based on the concept of "objects". Re-usability is the main concept of Oops. There are few major principles of object-oriented paradigm. If any programming Language supports this principles that programming language called Object-oriented programming. Below are object oriented programming concepts : 1. Object        Read More about Object     http://theprogrammersfirst.blogspot.in/2016/07/java-object-concept.html
Read more