Using just HTML, CSS, and JavaScript to create a simple, browsable UI of the NY Times Bestsellers
Sign up for an NYT API key and place your key in the 'fetch' function of the app.js file. Be sure to request the Books API from the list of choices provided by the New York Times.
Install mocha
and nightmare
, and make sure your basic harness is working. Starting on the command line in your repo folder:
npm install --save-dev mocha
npm install --save-dev nightmare
mkdir test
touch test/test.js
Once you have installed the dependencies, you can test this setup by running npm test
on the command line. If your page loads successfully, you should get this result:
> Load a Page
> ✓ should load a web page (12223ms)
> 1 passing (12s)
Utilized the sleek and intuitive front-end framework for faster and easier web development.
Utilized the NYT Books RESTful API to retrieve information about the New York Times Best Sellers Lists.
To use the Books API, you must sign up for an Books API key. Usage is limited to 1,000 requests per day (rate limits are subject to change). Please read and agree to the API Terms of Use and the Attribution Guidelines before you proceed.
A quick introduction to using Nightmare for web UI testing. We are using Mocha as the testing framework.
Nightmare is a high-level browser automation library from Segment.
Here are some helpful documentation for how to use Nightmare:
-
- Set up an instance
- Interact with the page
- Extract from the page
- Cookies
- Proxies
- Promises
- Extending Nightmare
Creates a new instance that can navigate around the web. The available options are documented here, along with the following nightmare-specific options.
Throws an exception if the
.wait()
didn't returntrue
within the set timeframe.const nightmare = Nightmare({ waitTimeout: 1000 // in ms })
Throws an exception if the
.goto()
didn't finish loading within the set timeframe. Note that, even thoughgoto
normally waits for all the resources on a page to load, a timeout exception is only raised if the DOM itself has not yet loaded.const nightmare = Nightmare({ gotoTimeout: 1000 // in ms })
Forces Nightmare to move on if a page transition caused by an action (eg,
.click()
) didn't finish within the set timeframe. IfloadTimeout
is shorter thangotoTimeout
, the exceptions thrown bygotoTimeout
will be suppressed.const nightmare = Nightmare({ loadTimeout: 1000 // in ms })
The maximum amount of time to wait for an
.evaluate()
statement to complete.const nightmare = Nightmare({ executionTimeout: 1000 // in ms })
The default system paths that Electron knows about. Here's a list of available paths: https://github.com/atom/electron/blob/master/docs/api/app.md#appgetpathname
You can overwrite them in Nightmare by doing the following:
const nightmare = Nightmare({ paths: { userData: '/user/data' } })
The command line switches used by the Chrome browser that are also supported by Electron. Here's a list of supported Chrome command line switches: https://github.com/atom/electron/blob/master/docs/api/chrome-command-line-switches.md
const nightmare = Nightmare({ switches: { 'proxy-server': '1.2.3.4:5678', 'ignore-certificate-errors': true } })
The path to the prebuilt Electron binary. This is useful for testing on different versions of Electron. Note that Nightmare only supports the version on which this package depends. Use this option at your own risk.
const nightmare = Nightmare({ electronPath: require('electron') })
A boolean to optionally show the Electron icon in the dock (defaults to
false
). This is useful for testing purposes.const nightmare = Nightmare({ dock: true })
Optionally shows the DevTools in the Electron window using
true
, or use an object hash containingmode: 'detach'
to show in a separate window. The hash gets passed tocontents.openDevTools()
to be handled. This is also useful for testing purposes. Note that this option is honored only ifshow
is set totrue
.const nightmare = Nightmare({ openDevTools: { mode: 'detach' }, show: true })
How long to wait between keystrokes when using
.type()
.const nightmare = Nightmare({ typeInterval: 20 })
How long to wait between checks for the
.wait()
condition to be successful.const nightmare = Nightmare({ pollInterval: 50 //in ms })
Defines the number of times to retry an authentication when set up with
.authenticate()
.const nightmare = Nightmare({ maxAuthRetries: 3 })
A string to determine the client certificate selected by electron. If this options is set, the
select-client-certificate
event will be set to loop through the certificateList and find the first certificate that matchessubjectName
on the electronCertificate Object
.const nightmare = Nightmare({ certificateSubjectName: 'tester' })
Gets the versions for Electron and Chromium.
Sets the
useragent
used by electron.Sets the
user
andpassword
for accessing a web page using basic authentication. Be sure to set it before calling.goto(url)
.Completes any queue operations, disconnect and close the electron process. Note that if you're using promises,
.then()
must be called after.end()
to run the.end()
task. Also note that if using an.end()
callback, the.end()
call is equivalent to calling.end()
followed by.then(fn)
. Consider:nightmare .goto(someUrl) .end(() => 'some value') //prints "some value" .then(console.log)
Clears all queued operations, kills the electron process, and passes error message or 'Nightmare Halted' to an unresolved promise. Done will be called after the process has exited.
Loads the page at
url
. Optionally, aheaders
hash can be supplied to set headers on thegoto
request.When a page load is successful,
goto
returns an object with metadata about the page load, including:url
: The URL that was loadedcode
: The HTTP status code (e.g. 200, 404, 500)method
: The HTTP method used (e.g. "GET", "POST")referrer
: The page that the window was displaying prior to this load or an empty string if this is the first page load.headers
: An object representing the response headers for the request as in{header1-name: header1-value, header2-name: header2-value}
If the page load fails, the error will be an object with the following properties:
message
: A string describing the type of errorcode
: The underlying error code describing what went wrong. Note this is NOT the HTTP status code. For possible values, see https://code.google.com/p/chromium/codesearch#chromium/src/net/base/net_error_list.hdetails
: A string with additional details about the error. This may be null or an empty string.url
: The URL that failed to load
Note that any valid response from a server is considered “successful.” That means things like 404 “not found” errors are successful results for
goto
. Only things that would cause no page to appear in the browser window, such as no server responding at the given address, the server hanging up in the middle of a response, or invalid URLs, are errors.You can also adjust how long
goto
will wait before timing out by setting thegotoTimeout
option on the Nightmare constructor.Goes back to the previous page.
Goes forward to the next page.
Refreshes the current page.
Clicks the
selector
element once.Mousedowns the
selector
element once.Mouseups the
selector
element once.Mouseovers the
selector
element once.Mouseout the
selector
element once.Enters the
text
provided into theselector
element. Empty or falsey values provided fortext
will clear the selector's value..type()
mimics a user typing in a textbox and will emit the proper keyboard events.Key presses can also be fired using Unicode values with
.type()
. For example, if you wanted to fire an enter key press, you would write.type('body', '\u000d')
.If you don't need the keyboard events, consider using
.insert()
instead as it will be faster and more robust.Similar to
.type()
,.insert()
enters thetext
provided into theselector
element. Empty or falsey values provided fortext
will clear the selector's value..insert()
is faster than.type()
but does not trigger the keyboard events.Checks the
selector
checkbox element.Unchecks the
selector
checkbox element.Changes the
selector
dropdown element to the option with attribute [value=option
]Scrolls the page to desired position.
top
andleft
are always relative to the top left corner of the document.Sets the viewport size.
Injects a local
file
onto the current page. The filetype
must be eitherjs
orcss
.Invokes
fn
on the page witharg1, arg2,...
. All theargs
are optional. On completion it returns the return value offn
. Useful for extracting information from the page. Here's an example:const selector = 'h1' nightmare .evaluate(selector => { // now we're executing inside the browser scope. return document.querySelector(selector).innerText }, selector) // <-- that's how you pass parameters from Node scope to browser scope .then(text => { // ... })
Error-first callbacks are supported as a part of
evaluate()
. If the arguments passed are one fewer than the arguments expected for the evaluated function, the evaluation will be passed a callback as the last parameter to the function. For example:const selector = 'h1' nightmare .evaluate((selector, done) => { // now we're executing inside the browser scope. setTimeout( () => done(null, document.querySelector(selector).innerText), 2000 ) }, selector) .then(text => { // ... })
Note that callbacks support only one value argument (eg
function(err, value)
). Ultimately, the callback will get wrapped in a native Promise and only be able to resolve a single value.Promises are also supported as a part of
evaluate()
. If the return value of the function has athen
member,.evaluate()
assumes it is waiting for a promise.Waits for
ms
milliseconds e.g..wait(5000)
.Waits until the element
selector
is present e.g..wait('#pay-button')
.Waits until the
fn
evaluated on the page witharg1, arg2,...
returnstrue
. All theargs
are optional. See.evaluate()
for usage.Adds a header override for all HTTP requests. If
header
is undefined, the header overrides will be reset.Returns whether the selector exists or not on the page.
Returns whether the selector is visible or not.
Captures page events with the callback. You have to call
.on()
before calling.goto()
. Supported events are documented here.This event is triggered if any javascript exception is thrown on the page. But this event is not triggered if the injected javascript code (e.g. via
.evaluate()
) is throwing an exception.Listens for
window.addEventListener('error')
,alert(...)
,prompt(...)
&confirm(...)
.Listens for top-level page errors. This will get triggered when an error is thrown on the page.
Nightmare disables
window.alert
from popping up by default, but you can still listen for the contents of the alert dialog.Nightmare disables
window.prompt
from popping up by default, but you can still listen for the message to come up. If you need to handle the confirmation differently, you'll need to use your own preload script.Nightmare disables
window.confirm
from popping up by default, but you can still listen for the message to come up. If you need to handle the confirmation differently, you'll need to use your own preload script.type
will be eitherlog
,warn
orerror
andarguments
are what gets passed from the console. This event is not triggered if the injected javascript code (e.g. via.evaluate()
) is usingconsole.log
.Similar to
.on()
, but captures page events with the callback one time.Removes a given listener callback for an event.
Takes a screenshot of the current page. Useful for debugging. The output is always a
png
. Both arguments are optional. Ifpath
is provided, it saves the image to the disk. Otherwise it returns aBuffer
of the image data. Ifclip
is provided (as documented here), the image will be clipped to the rectangle.Saves the current page as html as files to disk at the given path. Save type options are here.
Saves a PDF to the specified
path
. Options are here.Returns the title of the current page.
Returns the url of the current page.
Returns the path name of the current page.
Gets a cookie by it's
name
. The url will be the current url.Queries multiple cookies with the
query
object. If aquery.name
is set, it will return the first cookie it finds with that name, otherwise it will query for an array of cookies. If noquery.url
is set, it will use the current url. Here's an example:// get all google cookies that are secure // and have the path `/query` nightmare .goto('http://google.com') .cookies.get({ path: '/query', secure: true }) .then(cookies => { // do something with the cookies })
Available properties are documented here: https://github.com/atom/electron/blob/master/docs/api/session.md#sescookiesgetdetails-callback
Gets all the cookies for the current url. If you'd like get all cookies for all urls, use:
.get({ url: null })
.Sets a cookie's
name
andvalue
. This is the most basic form, and the url will be the current url.Sets a
cookie
. Ifcookie.url
is not set, it will set the cookie on the current url. Here's an example:nightmare .goto('http://google.com') .cookies.set({ name: 'token', value: 'some token', path: '/query', secure: true }) // ... other actions ... .then(() => { // ... })
Available properties are documented here: https://github.com/atom/electron/blob/master/docs/api/session.md#sescookiessetdetails-callback
Sets multiple cookies at once.
cookies
is an array ofcookie
objects. Take a look at the.cookies.set(cookie)
documentation above for a better idea of whatcookie
should look like.Clears a cookie for the current domain. If
name
is not specified, all cookies for the current domain will be cleared.nightmare .goto('http://google.com') .cookies.clear('SomeCookieName') // ... other actions ... .then(() => { // ... })
Clears all cookies for all domains.
nightmare .goto('http://google.com') .cookies.clearAll() // ... other actions ... .then(() => { //... })
Proxies are supported in Nightmare through switches.
If your proxy requires authentication you also need the authentication call.
The following example not only demonstrates how to use proxies, but you can run it to test if your proxy connection is working:
import Nightmare from 'nightmare'; const proxyNightmare = Nightmare({ switches: { 'proxy-server': 'my_proxy_server.example.com:8080' // set the proxy server here ... }, show: true }); proxyNightmare .authentication('proxyUsername', 'proxyPassword') // ... and authenticate here before `goto` .goto('http://www.ipchicken.com') .evaluate(() => { return document.querySelector('b').innerText.replace(/[^\d\.]/g, ''); }) .end() .then((ip) => { // This will log the Proxy's IP console.log('proxy IP:', ip); }); // The rest is just normal Nightmare to get your local IP const regularNightmare = Nightmare({ show: true }); regularNightmare .goto('http://www.ipchicken.com') .evaluate(() => document.querySelector('b').innerText.replace(/[^\d\.]/g, ''); ) .end() .then((ip) => { // This will log the your local IP console.log('local IP:', ip); });
By default, Nightmare uses default native ES6 promises. You can plug in your favorite ES6-style promises library like bluebird or q for convenience!
Here's an example:
var Nightmare = require('nightmare') Nightmare.Promise = require('bluebird') // OR: Nightmare.Promise = require('q').Promise
You can also specify a custom Promise library per-instance with the
Promise
constructor option like so:var Nightmare = require('nightmare') var es6Nightmare = Nightmare() var bluebirdNightmare = Nightmare({ Promise: require('bluebird') }) var es6Promise = es6Nightmare .goto('https://github.com/segmentio/nightmare') .then() var bluebirdPromise = bluebirdNightmare .goto('https://github.com/segmentio/nightmare') .then() es6Promise.isFulfilled() // throws: `TypeError: es6EndPromise.isFulfilled is not a function` bluebirdPromise.isFulfilled() // returns: `true | false`
You can add your own custom actions to the Nightmare prototype. Here's an example:
Nightmare.action('size', function(done) { this.evaluate_now(() => { const w = Math.max( document.documentElement.clientWidth, window.innerWidth || 0 ) const h = Math.max( document.documentElement.clientHeight, window.innerHeight || 0 ) return { height: h, width: w } }, done) }) Nightmare() .goto('http://cnn.com') .size() .then(size => { //... do something with the size information })
Remember, this is attached to the static class
Nightmare
, not the instance.You'll notice we used an internal function
evaluate_now
. This function is different thannightmare.evaluate
because it runs it immediately, whereasnightmare.evaluate
is queued.An easy way to remember: when in doubt, use
evaluate
. If you're creating custom actions, useevaluate_now
. The technical reason is that since our action has already been queued and we're running it now, we shouldn't re-queue the evaluate function.We can also create custom namespaces. We do this internally for
nightmare.cookies.get
andnightmare.cookies.set
. These are useful if you have a bundle of actions you want to expose, but it will clutter up the main nightmare object. Here's an example of that:Nightmare.action('style', { background(done) { this.evaluate_now( () => window.getComputedStyle(document.body, null).backgroundColor, done ) } }) Nightmare() .goto('http://google.com') .style.background() .then(background => { // ... do something interesting with background })
You can also add custom Electron actions. The additional Electron action or namespace actions take
name
,options
,parent
,win
,renderer
, anddone
. Note the Electron action comes first, mirroring how.evaluate()
works. For example:Nightmare.action( 'clearCache', (name, options, parent, win, renderer, done) => { parent.respondTo('clearCache', done => { win.webContents.session.clearCache(done) }) done() }, function(done) { this.child.call('clearCache', done) } ) Nightmare() .clearCache() .goto('http://example.org') //... more actions ... .then(() => { // ... })
...would clear the browser’s cache before navigating to
example.org
.See this document for more details on creating custom actions.
nightmare.use
is useful for reusing a set of tasks on an instance. Check out nightmare-swiftly for some examples.If you need to do something custom when you first load the window environment, you can specify a custom preload script. Here's how you do that:
import path from 'path' const nightmare = Nightmare({ webPreferences: { preload: path.resolve('custom-script.js') //alternative: preload: "absolute/path/to/custom-script.js" } })
The only requirement for that script is that you'll need the following prelude:
window.__nightmare = {} __nightmare.ipc = require('electron').ipcRenderer
To benefit of all of nightmare's feedback from the browser, you can instead copy the contents of nightmare's preload script.
By default nightmare will create an in-memory partition for each instance. This means that any localStorage or cookies or any other form of persistent state will be destroyed when nightmare is ended. If you would like to persist state between instances you can use the webPreferences.partition api in electron.
import Nightmare from 'nightmare'; nightmare = Nightmare(); // non persistent paritition by default yield nightmare .evaluate(() => { window.localStorage.setItem('testing', 'This will not be persisted'); }) .end(); nightmare = Nightmare({ webPreferences: { partition: 'persist: testing' } }); yield nightmare .evaluate(() => { window.localStorage.setItem('testing', 'This is persisted for other instances with the same paritition name'); }) .end();
If you specify a
null
paritition then it will use the electron default behavior (persistent) or any string that starts with'persist:'
will persist under that partition name, any other string will result in in-memory only storage.