Applitools Github Integration | Applitools Documentation

Skip to main content

On this page

The Eyes GitHub integration allows you to benefit from Eyes visual testing using a branch and merge workflow on projects using GitHub.

Eyes supports multiple branches, each of which can consist of multiple test baselines. You can create a new branch based on an existing branch, make independent updates to the baselines in each branch, compare branches to detect conflicts, and merge baselines in branches.

GitHub integrates with Eyes in the following aspects:

note

This article focuses on the modern integration that leverages the Github Application. If you're installing the integration for the first time, continue with this article.

If you're using the legacy integration, see the section at the bottom for information on the legacy integration.

Setting up the Eyes Github Integration [​](/content/docs/eyes/playwright/source-control/github#setting-up-the-eyes-github-integration "Direct link to Setting up the Eyes Github Integration"/index.html)

Eyes/GitHub integration is based on a GitHub App created by Applitools that defines the minimum access that Eyes requires to interact with GitHub on the repositories where Eyes tests are run. You can install multiple GitHub Apps that represent different GitHub Organizations for each team.

To avoid problems during the integration, the user that is running the integration requires the following authorization:

To enable GitHub integration:

  1. Navigate to the Admin Panel

  2. On the Admin panel, click on the Teams tile

  3. Click on the name of the team you want to configure.

  4. On the Integrations tab, click GITHUB.

  5. Depending on your previous configuration, continue with one of the following:

We recommend that you switch to using the GitHub App-based integration; select GitHub App integration, and continue with Installing the GitHub App below.

If you install multiple GitHub installations, the Applitools Org administrator can set Team permissions for each GitHub Installation on the Admin > Account screen. If permission settings have been enabled, any new GitHub installations will need to have permissions granted by the Applitools Org Admin.

Installing the GitHub App [​](/content/docs/eyes/playwright/source-control/github#installing-the-github-app "Direct link to Installing the GitHub App"/index.html)

This integration solution is intended for users that use the GitHub cloud and the Eyes Cloud. If you use a self-hosted GitHub system or have a private cloud or on-prem Eyes system, then you should use the Appltiools GitHub proxy service.

  1. Click Install the App or Install Another App to start the installation process.

  2. The GitHub installation wizard opens in a separate window. Select the GitHub Organization or user name under which you want to install the Applitools GitHub App. Typically, this will be your GitHub Organization and not a specific user.

  3. The GitHub wizard asks you which repositories you want to allow the GitHub App to access and tells you what access type is being granted. We recommend that you allow access to all current and future repositories, since this GitHub App installation serves your entire organization.

If you select only specific repositories at this point, if other repositories need to be added at later point in time, you will need to configure the Applitools GitHub directly on the GitHub site.

  1. Click on the Install button. A confirmation message appears. Click OK to complete the process.

If in the GitHub dialog you see a Install and Request or Request button, instead of Install then there is an issue with your GitHub permissions. Click Cancel to exit from the GitHub installation process. The Installation process has not been completed window will open with a link that you can copy and send to your GitHub organization administrator to complete the integration.

Configuring Github Repositories [​](/content/docs/eyes/playwright/source-control/github#configuring-github-repositories "Direct link to Configuring Github Repositories"/index.html)

  1. Click Manage repositories

  2. Select the GitHub organizations and repositories on which your team runs tests and click the Apply button.

If the list of repositories is large, you can type text in to the filter to display only repositories that include that text.

After you click Apply a list of integrated repositories is displayed in the Github Integration section.

  1. By default, GitHub integration closes a batch automatically once the CI build operation is completed. If your build runs concurrently or consists of multiple tests and you want all of the results to appear together in a single batch, disable automated batch closing and close the batch explicitly when the build is complete in your CI.

To set which repositories should have automated batch closing, select Show batch closing control.

If you disable automated batch closing, then you must notify closing of the batch explicitly, as described in Configuring Your CI.

Uninstalling the GitHub App [​](/content/docs/eyes/playwright/source-control/github#uninstalling-the-github-app "Direct link to Uninstalling the GitHub App"/index.html)

If the GitHub App is installed, and you want to uninstall it:

If a problem occurs during the uninstall procedure, for example you do not have the appropriate permissions on the GitHub site, a message is displayed with information about the problem that you can provide to your GitHub Admin.

Configuring Your CI [​](/content/docs/eyes/playwright/source-control/github#configuring-your-ci "Direct link to Configuring Your CI"/index.html)

Once you have enabled Github integration, you need to set up your CI to configure the Eyes SDK with information related to the GitHub pull request or commit request. To allow the Eyes Batch to integrate data received from the GitHub server, you need to set the following Environment variables:

If you run parallel push and pull requests, you need to disable Automatic Batch Closing, (see Disabling automated batch closing) and notify Eyes when a build is complete.

If Eyes cannot recognize the End of Build event, (for example in your CI system, GitHub cannot recognize identify an End of Build event, or you have multiple concurrent builds), you also need to add a cURL command to notify Eyes when all builds are complete. For details, see Notifying Eyes when a build is complete.

Preparing Your Code For A Commit Push Action [​](/content/docs/eyes/playwright/source-control/github#preparing-your-code-for-a-commit-push-action "Direct link to Preparing Your Code For A Commit Push Action"/index.html)

You need to add an API call to the CI script that will inform Applitools each time there is a commit push action.

curl -L -d '' -X POST "$APPLITOOLS_SERVER_URL/api/externals/github/push?
apiKey=$APPLITOOLS_API_KEY&CommitSha=$APPLITOOLS_BATCH_ID&BranchName=<Branch name>
&ParentBranchName=<Parent branch name>"

Notifying Eyes When A Build Is Complete [​](/content/docs/eyes/playwright/source-control/github#notifying-eyes-when-a-build-is-complete "Direct link to Notifying Eyes When A Build Is Complete"/index.html)

By setting the APPLITOOLS_BATCH_ID environment variable, Eyes puts the results of all tests in a particular CI build in the same Test Manager batch. In some setups, you need to disable this automatic notification and add a command to notify Eyes when a build is complete. Below are the most common setups when this command is needed:

Synchronizing multiple concurrent builds [​](/content/docs/eyes/playwright/source-control/github#synchronizing-multiple-concurrent-builds "Direct link to Synchronizing multiple concurrent builds"/index.html)

  1. In the GitHub integration page of the Test Manager, follow the instructions to disable automatic batch closing described in Configuring GitHub repositories. This is a one-time configuration that needs to be done by each team for each repository that has concurrent builds.

  2. Add an cURL command at the end of the build script to tell Eyes that the build is complete. The following is an example of the required script in a Linux-based system:

curl -X POST "$APPLITOOLS_SERVER_URL/api/externals/github/servers/<github server host>/commit/$APPLITOOLS_BATCH_ID/$APPLITOOLS_BATCH_BUILD_ID/complete?apiKey=$APPLITOOLS_API_KEY"

Where:

Working With The Github Integration [​](/content/docs/eyes/playwright/source-control/github#working-with-the-github-integration "Direct link to Working With The Github Integration"/index.html)

Once you have configured the GitHub repositories, you can use your CI and GitHub in the usual way. A CI job is normally triggered by a push to your repository, and this, in turn, runs your Eyes tests.

The Eyes/GitHub integration is based on the GitHub all commits. When there is an open all commits, Eyes adds the following functionality to your GitHub workflow:

  1. On a branch that corresponds to the repository name and source branch defined by the all commits.
  2. If there is no such branch, Eyes creates a branch and populates it with the latest baseline for that test from the repository name and target branch defined by the all commits
  3. If there is no such baseline, Eyes marks the test as New.

A icon indicates that all of the Eyes tests passed. If you click on the Details link, the browser switches to the Test results screen of the Eyes Test Manager, showing the results of the tests in the most recent build related to that all commits.

If you click Accept on all of the steps of all of the tests, the icon is replaced with the (passed) icon in the GitHub pull panel.

A icon indicates that there are merge conflicts. After you resolve any conflicts, Eyes sends an updated status to GitHub, and the next to the scm/applitools item is replaced with a icon. Click on the Details link to open the Merge branches screen in the Test Manager. See Reviewing Branch Differences for more details.

Merge the GitHub all commits as usual. This merges the Git files and, once this has completed successfully, Eyes is notified that it should merge the Eyes test baselines. Click on the Merge button on the Merge branches screen in the Test Manager (enabled only after all conflicts have been resolved). This operation is independent of any merge process in Git and occurs immediately, even if the GitHub merge has not been done.

By default, code merge is blocked if there are unsaved changes to the checkpoint as the SCM will not be marked as Passed. To enable or disable this setting, contact Customer Support.

Reviewing Branch Differences [​](/content/docs/eyes/playwright/source-control/github#reviewing-branch-differences "Direct link to Reviewing Branch Differences"/index.html)

Merging an Eyes source branch with a target branch results in adding new baselines to the target branch or updating its existing baselines according to baseline changes found in the source branch. Changes can include both updated baseline images and new, changed, or removed annotations (e.g. an added ignore region).

When working on a branch that is a child of another branch (the parent branch) and the child branch doesn't have a baseline, Eyes will use a fallback baseline from the parent branch.

Eyes knows how to compare two baselines and determine whether they can be merged automatically or whether user intervention is required. For example, an ignore region added to a source baseline step that did not change in the target baseline can be automatically added to the target baseline, but if another ignore region was added to the same step in the target baseline, user intervention is required to determine whether to keep only one of the added regions or both of them.

As described above, the item on the GitHub pull request page marked scm/applitools/team-name indicates the merge status of the Eyes baselines in the source branch.

Click on the Details link for this item, to view the Test Manager Merge branches page preloaded with the source and target branches derived from the pull request. If there are no conflicts, then the Test Manager displays a pop-up window to confirm this.

Click Got it! to close the pop-up.

Once you have resolved all conflicts, the merge status of the scm/applitools item in the GitHub pull request page changes to green (passed), and you can proceed to merge the branches, either from within the Test Manager by clicking the Merge button, or via the GitHub pull request page.

Github Proxy service [​](/content/docs/eyes/playwright/source-control/github#github-proxy-service "Direct link to Github Proxy service"/index.html)

Eyes/GitHub integration requires two ways communication between the Eyes server and the GitHub server.

Communication from the Eyes server to the GitHub server is needed during the setup phase, when you select repositories to integrate with. The Eyes server uses the GitHub API to create webhooks in the selected repositories which send all the relevant GitHub events to Eyes. The GitHub API is also used to update the status when an Eyes batch is completed and the status needs to be updated on the GitHub commit or pull request.

Communication from the GitHub server to the Eyes server is needed when events are sent by the webhooks to the Eyes server

If your GitHub server is protected behind a firewall, and you cannot add a custom rule that exposes the relevant GitHub APIs so that Eyes can call them, you can use the Eyes-scm application to create webhooks on the relevant GitHub repositories. Using Eyes-scm, the Eyes server never calls the GitHub APIs directly, all calls are made through the application.

To use Eyes-scm, you need to set up integration and select the required repository in the Eyes-scm interface, not in the Eyes server admin dashboard.

Once integration is complete, the commit or pull request status update flow is also performed by Eyes-scm. The Eyes-scm application constantly polls the Eyes server for status update commands and updates the status when needed on the GitHub server.

Creating the GitHub App for the Eyes integration [​](/content/docs/eyes/playwright/source-control/github#creating-the-github-app-for-the-eyes-integration "Direct link to Creating the GitHub App for the Eyes integration"/index.html)

A GitHub App is an entity created and stored on the GitHub server that defines the access permission for third-party tools. The Applitools Eyes/GitHub integration requires such a GitHub App so that it can receive pull and push request events and update status information on the repositories where the executed test code runs.

  1. On the GitHub server that stores your repositories, click on the option to create a GitHub App.

We recommend you name the app Applitools Eyes Integration, so that other teams in your organization can easily find the app and use it as well.

  1. Enable the Webhook option.

  2. Set the Webhook URL to:

https://<My_Org>.applitools.com/api/externals/github/v2/events?orgId=<orgId>&serverId=<GitHub server host>

where <My_Org> is the name of your Eyes server. This is the value that is typically assigned to the environment variable APPLITOOLS_SERVER_URL, or is passed in the test code to the method Configuration.setServerUrl or Eyes.setServerUrl.

Installing the GitHub App [​](/content/docs/eyes/playwright/source-control/github#installing-the-github-app-1 "Direct link to Installing the GitHub App"/index.html)

Once the GitHub App exists, you need to install it on your GitHub account. The procedure for doing this is described in the GitHub documentation.

All of the Eyes teams in your organization can share a GitHub App, or each team can create its own. The difference is only in the list of repositories configured for the GitHub App. The simplest scheme is that a single GitHub App provides access to all of the repositories used by all of the Eyes teams. Alternatively, each Eyes team can create its own GitHub App that provides Eyes with access to that team's repositories.

Installing the proxy service [​](/content/docs/eyes/playwright/source-control/github#installing-the-proxy-service "Direct link to Installing the proxy service"/index.html)

The Applitools proxy service is a Node.js-based program that runs in the background, continually polling the Eyes server to see if it needs to update the GitHub server with status information. When Eyes indicates that it wants to update the status, the proxy service interacts with GitHub on behalf of Eyes.

Users need to complete a one-time configuration of the proxy service in order to provide access permission and other information that the service needs to implement the Eyes/GitHub interaction. This is done by accessing the proxy server using a browser.

Prerequisites [​](/content/docs/eyes/playwright/source-control/github#prerequisites "Direct link to Prerequisites"/index.html)

Before installing the proxy service, you must have:

One-time setup [​](/content/docs/eyes/playwright/source-control/github#one-time-setup "Direct link to One-time setup"/index.html)

Set up the following environment variables so that they are available to the proxy service.

# set the Eyes server URL to your server
EYES_SERVER_URL=https://myOrg-eyes-api.applitools.com
#set the Eyes organization ID
EYES_ORG_ID=xx1234HxxXv987AB24q1vlF25BxxYaZ1234x5Hk3XaSa111
# set the HTTP port the proxy service should use (optional - default is 4000)
PORT=1234

To install the proxy server run the following from any command shell:

npm i -g @applitools/eyes-scm

Running and configuring the server [​](/content/docs/eyes/playwright/source-control/github#running-and-configuring-the-server "Direct link to Running and configuring the server"/index.html)

Use the following command to start the server:

eyes-scm

Configuring repositories and access credentials [​](/content/docs/eyes/playwright/source-control/github#configuring-repositories-and-access-credentials "Direct link to Configuring repositories and access credentials"/index.html)

The first time your run the server, you need to define the repositories you want to allow Eyes to access and provide the necessary Eyes and GitHub credentials. To do this, start the proxy service and then access it with a browser.

  1. Make sure the proxy service is running as described in One-time setup.

Start a browser on the server where the proxy is running and navigate to the proxy service. The default URL is localhost:4000.

  1. Select Github as the source control management system.

  2. Enter an Eyes API key with Team Admin rights and full privileges.

  3. Click Next

  4. If this is your initial installation, click Add a new server.

If you are reconfiguring an existing installation, then you will see a list of GitHub servers that are currently installed. Either select an existing server from the list or click on Add a new server.

  1. Enter the URL of your GitHub server and the secret entered when the GitHub App was created.

  2. Click Save

  3. In the list of servers, click on the server to configure it.

  4. On the first installation, no repositories will have been enabled, click Manage Repositories to select the repositories that you want Eyes to integrate with.

  5. On the first installation, as repositories have been enable the dialog will ask you to enter the GitHub App ID and the App private key (the entire contents of the downloaded .pem file). Enter these details and click Save.

  6. A screen with a list of all the repositories on the SCM server is displayed. The first time you configure the proxy service, all of the repositories are disabled (grayed out). Enable the repositories that you want Eyes to access.

  7. By default, GitHub integration closes a batch automatically once the CI build operation is completed. If your build runs concurrently or consists of multiple tests and you want all of the results to appear together in a single batch, disable automated batch closing and close the batch explicitly when the build is complete in your CI.

To disable Automated batch closing, clear the checkbox for the that repository.

Disabling automated batch closing [​](/content/docs/eyes/playwright/source-control/github#disabling-automated-batch-closing "Direct link to Disabling automated batch closing"/index.html)

By default, GitHub integration closes a batch automatically once the CI build operation is completed. You many need to disable automatically closing, for example if you have parallel push and pull requests, or run builds concurrently or you have builds that consists of multiple tests and you want all of the results to appear together in a single batch. If you disable automated batch closing, you need to close the batch explicitly when the build is complete in your CI.

To disable Automated batch closing, clear the checkbox for the that repository.

Changing the configuration [​](/content/docs/eyes/playwright/source-control/github#changing-the-configuration "Direct link to Changing the configuration"/index.html)

If you want to enable or remove access to repositories, or to change the Eyes API Key or the GitHub App, then open the browser to the proxy service, as described above, and make the necessary changes.

As part of the GitHub integration, the Eyes server sends batch status information to the GitHub server. Applitools provides two different ways to implement this Eyes/GitHub interaction:

Configuring Direct Eyes/GitHub Integration (Legacy) ​

This section describes how to configure the legacy Eyes/GitHub integration (based on OAuth authorization). This option is only available if it was previously configured by your organization before GitHub App based integration was available. The solution typically requires you to whitelist the Eyes server on the GitHub server, you can avoid this by using the GitHub App based Eyes integration.

Configure The Integration [​](/content/docs/eyes/playwright/source-control/github#configure-the-integration "Direct link to Configure The Integration"/index.html)

You must have Eyes Team admin rights to configure the Eyes/GitHub integration.

The configuration consists of the following steps:

  1. Navigate to the Admin page. Note that this page is only available if you have Admin privileges.

  2. Click on the Teams tile.

  3. Click on the row of the team you want to configure.

  4. Click on the Integrations tab.

  5. Click on the GITHUB row.

  6. Make sure that the Applitools direct integration option is selected.

  7. If your organization uses an Eyes Cloud server and the public github.com cloud, then you can skip this step and continue with step 9 below. If your organization has a GitHub Enterprise server or if you have an on-premise Eyes Server, then this one-time step needs to be completed, and it is then applied to all of the teams in the account. In the box displaying the server names, click on the dropdown to open the server selection menu and select your GitHub server. If your server is not yet on the list, then click on the Add button and add the hostname of your GitHub server and the other details as directed by the wizard.

  8. The GitHub Enterprise server is an on-premise server; therefore, you need to white list the Eyes server. You should white list inbound HTTPS traffic from the Eyes server to the on-premise GitHub Enterprise server in your company's firewall. If you use the public cloud, then you need to white list eyes.applitools.com, and if you use a private cloud, then you need to white list https://myServer.applitools.com If you can not whitelist your server then the recommended solution is to use the Applitools GitHUb proxy server see Using the Applitools GitHub proxy service.

  9. Click the Manage repositories button and in the dialog that opens, select the check boxes of the repositories that you want to activate in the Eyes/GitHub integration. The dialog includes a Filter repository name field that you can use to narrow down the set of repositories that are displayed. If you enter text in the box, only repositories that include that filter text are displayed.

  10. If your build runs as a single instance, you can now proceed to configure the CI - see Configuring your CI.

  11. If your build runs as multiple concurrent instances, then you need to take extra steps to ensure that the test results from all of the builds appear together in a single Test Manager batch. This is required if you have parallel push and pull requests.

    1. Select the Show batch closing control (advanced) check box. The Test Manager adds an another column labeled Automated batch closing that provides a toggle switch for each repository in the list of repositories.

  12. If the toggle switch for a repository is enabled (the default), when a build finishes execution, Eyes closes the batch. If the CI runs the build in multiple concurrent instances, then the test results from the concurrent build may appear in the same batch or in a different batch. In order to ensure that for a given repository build, all of the test results from all of the builds appear in a single batch, disable the toggle for that repository.

  13. If you disable automated batch closing for a repository, you need to ensure that the CI notifies Eyes when all of the concurrent builds are complete so that Eyes can then close the batch.

Previous

CI/CD Configuration

Next

Bitbucket

Documentation Search

Start typing to search across all Applitools documentation.