Lighthouse testing with CI

Brett Weir, February 5, 2023

I really like the Lighthouse service (now PageSpeed Insights) that Google offers, but like anything that has to be done manually, if I need to go to the page to run these reports, I'll never do it. It'd be better if I could run them automatically, and even better if they published somewhere in turn.

In this article, I explore a tool that I've never used before, and package and publish a container based on this exploration. The goal is to show how packaging an application as a container is a lot less arcane and a lot more try-it-and-see.

Discover the tool

I do some searching for "lighthouse ci" to start with and find the lighthouse-ci tool.

It seems good, but it's very GitHub-centric and seems like it adds a lot of stuff I don't really care about. I prefer to implement on top of vanilla tooling when I can. Fewer abstractions, fewer things that can go wrong, and I get a better understanding of what my tools are doing under the hood.

So I hunt for the upstream Lighthouse project instead.

It has a CLI! They instruct me to install it via npm like this:

npm install -g lighthouse

But I want to try it on my local machine before trying to pack up a container, so I use npx instead:

npx lighthouse

So I run it and see this:

$ npx lighthouse
Please provide a url

Specify --help for available options

Mmm, okay, let's do what it says:

$ npx lighthouse --help
lighthouse  
...

The output is actually pretty extensive, but at the very top, I see lighthouse <url>, and that's a pretty great clue of what's needed. So let's try it out:

$ npx lighthouse https://brettops.io/
  LH:ChromeLauncher Waiting for browser. +0ms
  LH:ChromeLauncher Waiting for browser... +1ms
  LH:ChromeLauncher Waiting for browser..... +503ms
  LH:ChromeLauncher Waiting for browser....... +502ms
  LH:ChromeLauncher Waiting for browser......... +501ms
...
...
  LH:status Auditing: Document has a valid `rel=canonical` +1ms
  LH:status Auditing: Structured data is valid +2ms
  LH:status Generating results... +1ms
  LH:Printer html output written to /home/brett/Projects/sites/brettops.io/brettops.io_2023-02-04_15-36-04.report.html +46ms
  LH:CLI Protip: Run lighthouse with `--view` to immediately open the HTML report in your browser +0ms
  LH:ChromeLauncher Killing Chrome instance 25134 +0ms

This single run tells us a lot about what our final deliverable will be.

The first thing that happens is that Chromium opens in my face as I run the tool:

Wasn't expecting Chrome to drop in.

Chromium is the upstream open source project for Google Chrome. This is a pretty solid clue that some version of Chrome is needed. Sure, I could have read that in the docs, but now there's no question.
The second is that we now know what a successful run looks like, and even what the default output is and where it ends up (taken from the above output):
```
...
    LH:Printer html output written to /home/brett/Projects/sites/brettops.io/brettops.io_2023-02-04_15-36-04.report.html +46ms
...
```

Decide how to package Chrome

Since Lighthouse wraps a full Chrome installation, Chrome will need to be available to the job every time it runs. There are two ways to make this happen:

Install it during the job.
Pre-install it in the container used for the job.

Chrome is a large piece of software. The installation takes over 40 seconds on my machine:

# time bash -c 'apt update -y && apt install -y --no-install-recommends chromium'
Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
...
...
update-alternatives: using /usr/bin/chromium to provide /usr/bin/x-www-browser (x-www-browser) in auto mode
update-alternatives: using /usr/bin/chromium to provide /usr/bin/gnome-www-browser (gnome-www-browser) in auto mode
Processing triggers for libc-bin (2.31-13+deb11u5) ...

real    0m43.151s
user    0m16.056s
sys     0m4.042s

If we have to install Chrome for every job, it's going to add this time to every pipeline. The Chrome installation is also fairly static, and isn't something we'll need to modify for any foreseeable reason. Installing at runtime is thus quite wasteful.

Pulling a container will take some time as well, but even if speed wasn't an issue, this installation adds a ton of text to our build logs that is effectively noise. Performing this installation as part of a container build in a separate project keeps our CI output logs focused on what matters, which is the Lighthouse test output.

So let's put this in a container.

Create a blank project

First, create a new, blank project to use. I've created brettops/containers/lighthouse, which I'll clone to get started:

git clone [email protected]:brettops/containers/lighthouse.git
cd lighthouse

We'll be alternating between the following commands:

# build it
docker build -t lighthouse .

# run it
docker run --rm -ti lighthouse

Create a `Dockerfile`

Let's create a Dockerfile:

touch Dockerfile

Pick the base container

Lighthouse is available as a Node module, so we'll need a Node container. I'll check Docker Hub first because containers are searchable there, and I happen to know that there is an official Node container available. We'll use an LTS version for good measure, and base it on the latest Debian, which is bullseye as of writing. I Ctrl + F the page for the term lts:

Finding an appropriate image tag is not an exact science.

I am able to locate an lts-bullseye image via serendipity, which is loosely pinned to allow the container to transparently update without breaking things too much (hopefully).

In the new Dockerfile, add the following:

FROM node:lts-bullseye

Set the default command

The Node container sets CMD to node, which will behave very badly in the context of CI. Instead, I set the CMD instruction to /bin/bash:

CMD ["/bin/bash"]

Install Chromium

Add an apt sandwich to get chromium installed:

RUN apt-get update -y \
    && apt-get install -y --no-install-recommends \
        chromium \
    && rm -rf /var/lib/apt/lists/*

Install the Lighthouse CLI

The next thing to install is the lighthouse CLI itself. First, I'll add an ENV instruction because I like the key package versions to be available if needed. At the time of writing, The latest Lighthouse version is 9.6.8:

ENV LIGHTHOUSE_VERSION="9.6.8"

I'll install the lighthouse command, then run lighthouse --version in the same step. This is a tiny sanity check that I use to ensure early failure detection if installation goes horribly wrong:

RUN npm install -g "lighthouse@${LIGHTHOUSE_VERSION}" \
    && lighthouse --version

Completed `Dockerfile`

Putting it all together, the Dockerfile is:

# Dockerfile
FROM node:lts-bullseye

CMD ["/bin/bash"]

RUN apt-get update -y \
    && apt-get install -y --no-install-recommends \
        chromium \
    && rm -rf /var/lib/apt/lists/*

ENV LIGHTHOUSE_VERSION="9.6.8"

RUN npm install -g "lighthouse@${LIGHTHOUSE_VERSION}" \
    && lighthouse --version

Build the image

Use docker build -t lighthouse . to build the image:

$ docker build -t lighthouse .
Sending build context to Docker daemon  1.092MB
Step 1/5 : FROM node:lts-bullseye
 ---> b450fd0097d2
Step 2/5 : CMD ["/bin/bash"]
 ---> Using cache
 ---> c33857b69b33
Step 3/5 : RUN apt-get update -y     && apt-get install -y --no-install-recommends         chromium     && rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> f4afc61d3da4
Step 4/5 : ENV LIGHTHOUSE_VERSION="9.6.8"
 ---> Running in 26cb6992d0ab
Removing intermediate container 26cb6992d0ab
 ---> 85b5ede9cd5e
Step 5/5 : RUN npm install -g "lighthouse@${LIGHTHOUSE_VERSION}"     && lighthouse --version
 ---> Running in 36867364da0a
npm WARN deprecated [email protected]: We've written a new parser that's 6x faster and is backwards compatible. Please use @formatjs/icu-messageformat-parser

added 131 packages in 8s

15 packages are looking for funding
  run `npm fund` for details
npm notice
npm notice New minor version of npm available! 9.3.1 -> 9.4.1
npm notice Changelog: 
npm notice Run `npm install -g [email protected]` to update!
npm notice
9.6.8
Removing intermediate container 36867364da0a
 ---> f395ef77ec20
Successfully built f395ef77ec20
Successfully tagged lighthouse:latest

Hey hey, it built successfully!

Test out the container

Now we can start a container from our freshly built image:

$ docker run --rm -ti lighthouse
root@34f24530bb7d:/#

Disable error reporting

When I try out lighthouse for the first time, the first thing I get hit with is a prompt about error reporting:

root@34f24530bb7d:/# lighthouse https://brettops.io/
? We're constantly trying to improve Lighthouse and its reliability.
  Learn more: https://github.com/GoogleChrome/lighthouse/blob/master/docs/error-reporting.md
  May we anonymously report runtime exceptions to improve the tool over time?  (y/N) ‣ false

Even though it apparently times out on its own, that's going to eat CI time, so we gotta disable that post-haste.

lighthouse --help dumps a lot of info. By looking through it, I found the --no-enable-error-reporting parameter. Excellent. Not every tool thinks to include such a thing.

The new command looks like this:

lighthouse https://brettops.io/ --no-enable-error-reporting

Sandbox user error

The next hurdle to overcome is this (...you'll be waiting a little while):

root@34f24530bb7d:/# lighthouse https://brettops.io/ --no-enable-error-reporting
  LH:ChromeLauncher Waiting for browser. +0ms
  LH:ChromeLauncher Waiting for browser... +1ms
  LH:ChromeLauncher Waiting for browser..... +504ms
  LH:ChromeLauncher Waiting for browser....... +502ms
...
...
  LH:ChromeLauncher:error connect ECONNREFUSED 127.0.0.1:43865 +2ms
  LH:ChromeLauncher:error Logging contents of /tmp/lighthouse.UFYleiC/chrome-err.log +0ms
  LH:ChromeLauncher:error [142:142:0203/072308.222583:ERROR:zygote_host_impl_linux.cc(100)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
  LH:ChromeLauncher:error  +0ms
Unable to connect to Chrome

You'll notice the following error:

Running as root without --no-sandbox is not supported.

We can attack this from a couple angles:

Remove the sandbox and run as root.
Keep the sandbox and put a user inside the container.

Running as root is probably a bad idea. However, in a CI environment, not having root makes it utterly impossible to configure the environment for a given job.

At this point, it's too early to tell what the right approach is, so we have to try both. I'll try adding a user first as its "probably" less arcane to do so successfully.

Try a regular user

An easy way to force a container to run as a regular user is by adding -u $(id -u):$(id -g) to its run arguments:

docker run --rm -ti -u $(id -u):$(id -g) CONTAINER

In action:

$ docker run --rm -ti -u $(id -u):$(id -g) lighthouse
I have no name!@b02c3ba66118:/$

Woot, new errors! Which means we got past this hurdle. In summary:

Takeaway: Create a user (if needed!)

You may have noticed the following strange message from the previous output:

I have no name!@b02c3ba66118:/$

A user with my own user ID doesn't exist inside the container, which is why I'm seeing this message.

It's fine though. Some things will break when a user doesn't exist, but if the broken things aren't critical to your application, you can survive without one.

Run headless Chrome

The current command is:

lighthouse --no-enable-error-reporting https://brettops.io

Let's see what happens when we run as a regular user:

I have no name!@b02c3ba66118:/$ lighthouse --no-enable-error-reporting https://brettops.io
  LH:ChromeLauncher Waiting for browser. +0ms
  LH:ChromeLauncher Waiting for browser... +0ms
...
...
  LH:ChromeLauncher:error connect ECONNREFUSED 127.0.0.1:35883 +2ms
  LH:ChromeLauncher:error Logging contents of /tmp/lighthouse.PfSs4hy/chrome-err.log +0ms
  LH:ChromeLauncher:error find: '//.config/chromium/Crash Reports/pending/': No such file or directory
  LH:ChromeLauncher:error chrome_crashpad_handler: --database is required
  LH:ChromeLauncher:error Try 'chrome_crashpad_handler --help' for more information.
  LH:ChromeLauncher:error [28:28:0203/073247.702450:ERROR:socket.cc(120)] recvmsg: Connection reset by peer (104)
  LH:ChromeLauncher:error  +0ms
Unable to connect to Chrome

I don't know why exactly this is happening at this point, but my guess, from running it locally and having Chrome pop up in my face, is that Chrome is failing to start because we haven't configured it to run headless. Instead, it's probably waiting for a display to become available, which will never happen, and timing out as a result.

But how to run headless? I'm loving lighthouse --help again because it actually prints examples (wow!) that are useful (wow wow!), including how to pass flags directly to Chrome (😱):

Discovering the --chrome-flags option using lighthouse --help. — Discovering the `--chrome-flags` option using `lighthouse --help`.

So let's add --chrome-flags="--headless". Our command becomes:

lighthouse --no-enable-error-reporting \
  --chrome-flags="--headless" https://brettops.io

Taking it for a spin:

I have no name!@b02c3ba66118:/$ lighthouse --no-enable-error-reporting --chrome-flags="--headless" https://brettops.io
  LH:ChromeLauncher Waiting for browser. +0ms
...
...
  LH:ChromeLauncher:error connect ECONNREFUSED 127.0.0.1:40267 +1ms
  LH:ChromeLauncher:error Logging contents of /tmp/lighthouse.ZKtl7zq/chrome-err.log +0ms
  LH:ChromeLauncher:error find: '//.config/chromium/Crash Reports/pending/': No such file or directory
  LH:ChromeLauncher:error [0203/074354.325439:ERROR:zygote_host_impl_linux.cc(127)] No usable sandbox! If this is a Debian system, please install the chromium-sandbox package to solve this problem. If you want to live dangerously and need an immediate workaround, you can try using --no-sandbox.
  LH:ChromeLauncher:error  +0ms
Unable to connect to Chrome

Different error! Woo! In summary:

Sandbox error (revisited)

In the above error, you'll notice the comment No usable sandbox. This is a stroke of pure fortune. And what's more, the container is Debian, so we're in luck. Let's try it:

RUN apt-get update -y \
    && apt-get install -y --no-install-recommends \
        chromium \
        chromium-sandbox \
    && rm -rf /var/lib/apt/lists/*

Then rebuild our image and re-run:

docker build -t lighthouse .
docker run --rm -ti -u $(id -u):$(id -g) lighthouse

This will give us the same error as before:

I have no name!@b02c3ba66118:/$ lighthouse --no-enable-error-reporting --chrome-flags="--headless" https://brettops.io
  LH:ChromeLauncher Waiting for browser. +0ms
...
...
  LH:ChromeLauncher:error connect ECONNREFUSED 127.0.0.1:40267 +1ms
  LH:ChromeLauncher:error Logging contents of /tmp/lighthouse.ZKtl7zq/chrome-err.log +0ms
  LH:ChromeLauncher:error find: '//.config/chromium/Crash Reports/pending/': No such file or directory
  LH:ChromeLauncher:error [0203/074354.325439:ERROR:zygote_host_impl_linux.cc(127)] No usable sandbox! If this is a Debian system, please install the chromium-sandbox package to solve this problem. If you want to live dangerously and need an immediate workaround, you can try using --no-sandbox.
  LH:ChromeLauncher:error  +0ms
Unable to connect to Chrome

I should have known better. There's no way we're going to get a sandbox working inside a container, because the container is the sandbox, and we're running it with no privileges. We can remove chromium-sandbox because it'll never work, and add the --no-sandbox flag to Chrome like they suggested:

lighthouse --no-enable-error-reporting \
  --chrome-flags="--headless --no-sandbox" https://brettops.io

Then we run it again:

I have no name!@623727cb73be:/$ lighthouse --no-enable-error-reporting --chrome-flags="--headless --no-sandbox" https://brettops.io
  LH:ChromeLauncher Waiting for browser. +0ms
  LH:ChromeLauncher Waiting for browser... +1ms
  LH:ChromeLauncher Waiting for browser..... +504ms
...
...
  LH:status Auditing: Structured data is valid +1ms
  LH:status Generating results... +1ms
  LH:ChromeLauncher Killing Chrome instance 64 +35ms
Runtime error encountered: EACCES: permission denied, open '/brettops.io_2023-02-03_07-59-35.report.html'
Error: EACCES: permission denied, open '/brettops.io_2023-02-03_07-59-35.report.html'

Excellent, we have an exciting new error to work with. In summary:

Try the root user

For good measure, let's start the container as root:

docker run --rm -ti lighthouse

The prompt will once again look like this:

$ docker run --rm -ti lighthouse
root@c22e221331e1:/#

We'll use the latest version of our command that removes the sandbox:

lighthouse --no-enable-error-reporting \
  --chrome-flags="--headless --no-sandbox" https://brettops.io

This will spit out the following:

root@c22e221331e1:/# lighthouse --no-enable-error-reporting --chrome-flags="--headless --no-sandbox" https://brettops.io
  LH:ChromeLauncher Waiting for browser. +0ms
  LH:ChromeLauncher Waiting for browser... +0ms
  LH:ChromeLauncher Waiting for browser..... +504ms
...
...
  LH:status Auditing: Structured data is valid +1ms
  LH:status Generating results... +1ms
  LH:Printer html output written to /brettops.io_2023-02-03_08-38-54.report.html +34ms
  LH:CLI Protip: Run lighthouse with `--view` to immediately open the HTML report in your browser +0ms
  LH:ChromeLauncher Killing Chrome instance 28 +0ms

It works just as it did with a regular user, except now there are no access errors as I'm running as root. 😂

Configure report outputs

Okay, awesome, so our command so far is as follows:

lighthouse --no-enable-error-reporting \
  --chrome-flags="--headless --no-sandbox" https://brettops.io

This command works as root and as a regular user, up to the point where it writes the report to disk, which is where the regular user encounters an access error:

Runtime error encountered: EACCES: permission denied, open '/brettops.io_2023-02-03_07-59-35.report.html'
Error: EACCES: permission denied, open '/brettops.io_2023-02-03_07-59-35.report.html'

This is just lighthouse trying to write to an inaccessible location in the container. We need to provide a means to get data back out of the container. That means mounting the local filesystem inside:

docker run --rm -ti -v $(pwd):/code -w /code lighthouse

-v $(pwd):/code will mount your current directory inside the container at /code.
-w /code will set /code as the default working directory.

This will allow you to run commands as if you were running them in your current directory, and they will work on resources from inside the container.

We can use --output-path to configure where our report is saved:

lighthouse \
  --no-enable-error-reporting \
  --chrome-flags="--headless --no-sandbox" \
  --output-path results.html \
  https://brettops.io

And let's let it run again:

root@c22e221331e1:/# lighthouse --no-enable-error-reporting --chrome-flags="--headless --no-sandbox" --output-path results.html https://brettops.io
  LH:ChromeLauncher Waiting for browser. +0ms
  LH:ChromeLauncher Waiting for browser... +0ms
  LH:ChromeLauncher Waiting for browser..... +504ms
...
...
  LH:status Generating results... +1ms
  LH:Printer html output written to results.html +32ms
  LH:CLI Protip: Run lighthouse with `--view` to immediately open the HTML report in your browser +0ms
  LH:ChromeLauncher Killing Chrome instance 146 +0ms

Outside the container, you can open the results file with your browser of choice (coughUseFirefoxcough):

firefox results.html

Files are owned by the container user

You may be surprised to discover that the results.html file you've generated is owned by root:

$ ls -l
total 612
-rw-rw-r-- 1 brett brett    340 Feb  4 23:40 Dockerfile
-rw-r--r-- 1 root  root  591033 Feb  6 19:44 results.html

This is because the file was created from inside the container, which was running as root.

This is a common point of confusion, and one of the benefits of running the container as a regular user.

To remove the file after you're done, call rm from inside the container, or sudo rm from outside.

And just look at those results:

Lighthouse test results with default settings.

Lighthouse can emulate multiple device configurations, including screen size, throttling, resource availability, etc. It looks like, from our results file, that the default profile is for a mobile device. There appears to be a preset for desktop, so let's try that:

lighthouse \
  --no-enable-error-reporting \
  --chrome-flags="--headless --no-sandbox" \
  --output-path desktop.html \
  --preset desktop \
  https://brettops.io

View the output in our favorite browser from the local machine:

firefox desktop.html

Lighthouse test results with desktop preset. — Lighthouse test results with `desktop` preset.

Publish the container

The easiest way to publish a container is via the container pipeline. I suspect this project will build fine with Kaniko, so I'll try that pipeline variant first.

Create a .gitlab-ci.yml file and add the following:

# .gitlab-ci.yml
include:
  - project: brettops/pipelines/container
    file: kaniko.yml

If your Dockerfile is located at /Dockerfile in the project, no additional configuration is necessary.

Commit the .gitlab-ci.yml and the Dockerfile:

git add .gitlab-ci.yaml Dockerfile
git commit -m "Initial commit"
git push

The project is live, and because we're using ready-made automation, our first ever pipeline is successful, and the container is already published!

You can download the container directly here:

docker pull registry.gitlab.com/brettops/containers/lighthouse

But we're not done yet. How do we know that this container is any good? And how will we be sure that we don't break it in the future?

Validate the container in CI

To demonstrate our pipeline and validate its ongoing fitness, we will add a custom job to run a report and add the pages pipeline to publish the report to GitLab Pages.

Extend the .gitlab-ci.yml file with the pages pipeline:

# .gitlab-ci.yml
include:
  - project: brettops/pipelines/container
    file: kaniko.yml
  - project: brettops/pipelines/pages
    file: include.yml

test-lighthouse:
  stage: deploy
  image: registry.gitlab.com/brettops/containers/lighthouse
  script:
    - mkdir -p public
    - >-
      lighthouse
      --no-enable-error-reporting
      --chrome-flags="--headless --no-sandbox"
      --output-path public/index.html
      --preset desktop
      https://brettops.io/
  artifacts:
    paths:
      - public/

This is a subtle variation of the command that we've already been using. The main difference is that we updated the --output-path to public/index.html. This is so that, after we publish our GitLab Pages site, we'll be able to browse the performance results by navigating to https://brettops.gitlab.io/containers/lighthouse.

Commit the changes:

git add .gitlab-ci.yml
git commit -m "Add test job"
git push

I don't expect this to work on the first try, which it doesn't:

The test-lighthouse job was supposed to come before the pages job. — The `test-lighthouse` job was supposed to come before the `pages` job.

It looks like we need to add an intermediate stage between the default build and deploy job that the pipelines define, so we'll need to add a stages key to this pipeline, and move the job around:

# .gitlab-ci.yml
stages:
  - test
  - build
  - buildtest
  - deploy

include:
  - project: brettops/pipelines/container
    file: kaniko.yml
  - project: brettops/pipelines/pages
    file: include.yml

test-lighthouse:
  stage: buildtest
  image: registry.gitlab.com/brettops/containers/lighthouse
  script:
    - mkdir -p public
    - >-
      lighthouse
      --no-enable-error-reporting
      --chrome-flags="--headless --no-sandbox"
      --output-path public/index.html
      --preset desktop
      https://brettops.io/
  artifacts:
    paths:
      - public/

Commit our fix-up:

git add .gitlab-ci.yml
git commit -m "Fix job ordering"
git push

And it looks like the second time's the charm here.

Conclusion

The pipeline has now passed, and the Lighthouse report is publicly available.

With some modification, this work can be generalized to support deployment in any project, and can also be configured to run on a schedule.