Part 4: Utilities

Introduction

In the previous parts you built the foundation of your source plugin, from now on it’s about adding new functionalities, improving the user experience, and making the plugin better overall.

This part of the tutorial will focus on a subset of the Node API helpers. Feel free to revisit this document as it can also work as standalone instructions for those APIs.

By the end of this part of the tutorial, you will be able to:

• Use the reporter API to output structured terminal messages
• Use the cache API to save artifacts between runs
• Define custom errors for your plugin through reporter.setErrorMap
• Use the pluginOptionsSchema API to verify your plugin’s options

reporter API

You, as a source plugin author, can interact with your users in a variety of ways. You can share information in the README, chat in issues & PRs, create guides, etc. But no matter how active of a maintainer you are, it is your source plugin that will interact the most with your users through the terminal when they run gatsby develop or gatsby build. Gatsby outputs what it does behind the scenes with the goal of giving the users rich information that helps them be more productive with Gatsby itself.

For this, Gatsby internally uses the reporter API and you can use it, too! When adding logs to the terminal be mindful of keeping of them concise, informative, and to a minimum. Most Gatsby plugins don’t even need to add logs (other than errors), so in doubt leave them out at the beginning and reevaluate once you have a bigger user base.

If you need information about an API, always be sure to check Gatsby’s API docs, in this case the reporter API docs. The reporter API is available in all Node APIs and its methods can be used like this:

Here’s a short explanation on the most important methods:

• info: Print a message to the terminal.
• warn: Print a warning message to the terminal.
• error: Print an error message to the terminal.
• panic: Print an error message to the terminal and immediately exit the process.
• panicOnBuild: Print an error message to the terminal and immediately exit the process (only during gatsby build). Most often you should use this over panic as it’ll allow users to debug the error during gatsby develop.
• verbose: Print a message to the terminal that is only visible when the “verbose” flag is enabled (e.g. gatsby build --verbose).
• activityTimer: Print an informational message to the terminal that has a timer attached to it (e.g. how long that step took).
• setErrorMap: Set a custom error map to the reporter. This allows the reporter to extend Gatsby’s internal error map.

You’ll learn how to use panicOnBuild, activityTimer, and setErrorMap in the next three tasks as they are the most relevant and common for plugins.

Task: Output errors

As source plugins interact with third-party APIs they are inherently prone to errors (network issues, API outages, etc.). As explained in Part 2 it’s good practice to gracefully handle all possible error states, as there’s nothing worse than no actionable feedback at all.

In Part 2 you also added the fetchGraphQL function that accesses the GraphQL API inside the api folder. When you send a malformed request to it, it can not only give back data but also errors. Use that information to output it with panicOnBuild.

1. Go to your plugin/src/source-nodes.ts file. Destructure reporter from gatsbyApi at the top of sourceNodes. Add errors to the destructuring statement for the fetchGraphQL response. Also add a conditional statement when errors is truthy directly below fetchGraphQL:

2. Inside the errors conditional, use panicOnBuild to output the error message that you’re receiving from your API. error, panicOnBuild, and panic accept a string, Error or a structured error (you’ll learn about the latter in Task: Define custom errors). Also add an early return to not continue creating nodes.

   Pro tip: Try to rely on TypeScript types to check if the error you’re getting back is a valid Error type. If not, you might need to modify the error a little bit to fit the requirements.

3. To see if it works, add a typo to your GraphQL request inside fetchGraphQL:

   Restart the develop:site script and you should see an error in the terminal:

   Yes, it works 🎉 Don’t forget to remove the typo again. In Task: Define custom errors you’ll learn how to further customize this error.

   Pro tip: This is the error that the GraphQL API directly returns. You can see the same error if you go to http://localhost:4000/graphql (the GraphQL API from api folder) and try the query there. Get yourself familiar with how, when, and in which form your API throws errors.

You’re now surfacing any errors that occur during the data fetching to your users. Use the error, panicOnBuild, and panic methods throughout your code to display all relevant errors to your user.

Please note: If necessary, add context to your error messages instead of only passing the error along. This is especially true if the error could be thrown in multiple places or if the original error is not clear enough.

Task: Show activities and timings

Since your source plugin is retrieving information from a third-party API, that process could take a while, especially if it’s fetching a lot of data. In these instances you can use the activityTimer method to let the user know that something is happening in the background. You shouldn’t add separate activityTimer instances (remember: Keep the output concise and to a minimum) but instead use the setStatus method on activityTimer to update its text.

Now you’ll learn exactly that, by adding an activity for the sourcing and node creation process of your plugin.

1. Create a new activityTimer and store that in a variable (the name doesn’t matter). You’ll use the variable to start, update, and stop the timer. Try to use a text that clearly indicates your plugin name and what it’s doing.

2. Since the sourcingTimer was initialized before the fetchGraphQL call you can also start() it before that call:

   With the start() call the timer is running until you’ll eventually call end().

3. If the sourcing fails, the timer should be stopped. So update the panicOnBuild call to use the sourcingTimer instead:

   Please note: When looking at the TypeScript types for sourcingTimer you’ll notice that it also implements panicOnBuild and panic. You can pass in the same arguments as you’re used to, the activityTimer will handle everything for you. If you had continued to use reporter.panicOnBuild the terminal would output a successful message for the timer which is incorrect of course!

4. Right now the sourcingTimer shows “Sourcing from plugin API” together with a time in seconds. You can use the setStatus() method to append text.

   Pro tip: Adding the information of how many types were created is a really nice debugging help since it’ll allow folks to quickly compare builds and see if e.g. all the information they expect is sourced.

5. Stop the timer once node creation is done:

6. Restart the develop:site script and you should read a new activity in the terminal:

   That’s what we’d expect, nice!

Task: Define custom errors

The error you’ve output to the terminal in Task: Output errors was functional but missing some crucial information. Where is the error coming from? Which part of the plugin? Any suggestions for immediate fixes?

You can use the setErrorMap method to create an error map for your plugin. Gatsby internally uses an error map to benefit from two things (among other additional benefits):

• A user can google the error with its unique ID and hopefully only find resources that are directly related to that error.
• One can track down the location in the source code where an error is thrown quicker through the use of constants and unique IDs.

Both benefits also apply to you when you define a custom error map for your plugin. You’ll see the improvement at the end of this task. As a reminder, this is how an error (for the data fetching step) currently looks like:

Let’s get to work:

1. Open the plugin/src/constants.ts file and add an ERROR_CODES map with its first key GraphQLSourcing:

   The keys of the ERROR_CODES object can be arbitrary, we’d recommend using names that make it easy to differentiate them for their different purposes. The value of each entry is the unique ID. So make sure that each ID only exists once inside ERROR_CODES.

   You don’t need to worry about other plugins or Gatsby itself as the IDs are namespaced with your plugin. If possible, use IDs that are not already used by Gatsby itself.

2. Import ERROR_CODES into plugin/src/on-plugin-init.ts and initialize setErrorMap. Remove the existing info call:

3. You can populate the error map like so:

   • The object’s keys are the unique IDs
   • The object should contain these three keys and values:
     • text: Your text that will be shown to the user for this specific error
     • level: Set this to ERROR
     • category: Set this to THIRD_PARTY

   Add the following to plugin/src/on-plugin-init.ts:

   text has to be a function that returns a string and receives a freeform context. You can pass through any information you’d like in context and then use it to construct the final message that is shown to the user. So the error above requires sourceMessage and graphqlError to work.

   Pro tip: If your text should be a multi-line string, consider using the stripIndent function from common-tags. This way your error will be displayed correctly independent from how you write the string.

4. In this step you’ll invoke the error with the custom ID and pass along the required context information. Open plugin/src/source-nodes.ts and add the ERROR_CODES import and use it inside the existing panicOnBuild call:

5. As in step 3 of Task: Output errors, add a typo to the GraphQL error again (e.g. title2), restart the develop:site script and you should see a new output for the error:

Your error now has a custom ID (#plugin_10000), it prints the name of the plugin (PLUGIN), and you’ve added additional context to the error itself (Sourcing from the GraphQL API failed:). Whenever you need to output an error message, create a new structured error inside setErrorMap and use it to display the best possible error to the user.

cache API

You might already know that Gatsby creates two folders during its build phase: .cache and public. It’s recommended to keep both folders around in between builds as then Gatsby can leverage its caching functionalities the best.

With the cache API you have access to a key-value store to persist data inside the .cache folder. You can use it to persist timestamps, tokens, etc. to use delta updates, or to store the artifacts of time/memory/cpu intensive tasks (e.g. images, big files).

If the term “delta update” doesn’t ring a bell, here’s a short explanation in context of a source plugin: A delta update requires the source plugin to only fetch data from the remote API that is new, or has been changed from a previous state, in contrast to having to fetch all the data again.

Important: Your remote API has to support delta updates, it’s not a universal thing that works everywhere.

Task: Save the timestamp of last sourcing

The most common use case of the cache API is to store timestamps, tokens, etc. and therefore you’ll learn exactly that. Even though the example API this tutorial uses doesn’t support delta updates, you’ll go through all the necessary steps in this task to apply it to your own source plugin.

The cache API is a key-value store, so you save specific data under a key, and then can retrieve a value with a given key.

1. Store the name of the key in an constants to minimize the likelyhood of typos. Open plugin/src/constants.ts and add:

2. Import the new CACHE_KEYS variable into plugin/src/source-nodes.ts and grab the cache API from gatsbyApi inside sourceNodes:

3. Generate a date timestamp just before the fetchGraphQL call and use cache.set to save the timestamp after the successful fetch. This way you’re ensuring accurate timestamps that are only saved when a successful data call happened. The key you use for cache.set has to be unique.

   Date.now() returns the number of milliseconds elapsed since the epoch. If you need another format, read the Date.now() documentation.

   It’s also important to know that cache is shared by all instances of your plugin. So if your user can add your plugin multiple times to gatsby-config (e.g. by setting a different workspaceId in the plugin options), you need to ensure that those cache.set calls use unique keys. In those cases, use a unique instance identifier, for example said plugin options.

4. The next step is to use cache.get to retrieve the timestamp from the cache before the fetchGraphQL call. Since you won’t use it for delta fetching in this tutorial, output the information in the verbose mode of Gatsby with reporter.verbose:

5. Time to test your changes! Restart the develop:site script but this time with the --verbose flag enabled:

   The output got a lot busier! You should see a new log:

   You’re seeing this because on this first run the lastFetchedDate didn’t exist yet and thus undefined is returned from the cache.get call. Stop the development server and restart the script. You now should see a timestamp defined that looks something like this:

   Great, it’s working!

   Pro tip: If you save files with the cache API you can also inspect what is saved by going to the .cache/caches folder and looking for a folder named like your plugin. For this tutorial the path would be site/.cache/caches/plugin.

pluginOptionsSchema API

In most cases a Gatsby source plugin needs one or more options as it interacts with third-party APIs that require customization and authentication (e.g. different endpoints or API keys). You don’t want other users to use your credentials, so you’ll need a way for your users to provide theirs. That’s where plugin options come in!

The Configuring Plugin Usage with Plugin Options guide goes into depth about plugin options and the pluginOptionsSchema API so if you want to know more beyond this guide, be sure to give it a read. But to give a short summary:

• A Gatsby plugin with options included makes those options available in the second argument of Gatsby Node, Browser, and SSR APIs.

  A sample gatsby-config file:

  Usage of that option in a Node API:

• A plugin can optionally define a schema (through the pluginOptionsSchema API) to enforce a type for each option. Gatsby will validate that the options users pass match the schema to help them correctly set up their site.

Here are some tips when working with plugin options:

• Consider the options a public API contract between your users and your plugin. Follow SemVer versioning when adding, modifying, and removing options. For example, if you want to remove an option you should only do this in a major release of your plugin since it’s a breaking change.
• Add as few plugin options as necessary. In the beginning, use safe defaults where possible (to the best of your knowledge) and when users wish to have new options available, seriously contemplate if you should add them. Once added, you’ll need to maintain the option forever (or until you remove it again).
• Use clear names. Naming things is hard, but take extra care in using clear names for your plugin’s options. Best case scenario is that an option is clear enough without any additional description.
• Use “enable” options, not negated options. An example: Instead of using noWarnings: true, use warnings: false. The double negation that would occur with noWarnings: false is really confusing.
• Pass options through. If your source plugin is using a library under the hood (e.g. an SDK from a CMS) that has options on its own, allow people to configure all those options, too. You could collect those under a new key in the options object (e.g. options.sdkOptions) and then refer people to the SDK’s documentation.

Task: Use endpoint option

Instead of hardcoding the URL to the GraphQL API, you’ll use an endpoint option and thus make the URL configurable.

1. The boilerplate you cloned at the beginning already has an endpoint option defined in the site’s gatsby-config. Go to site/gatsby-config.ts to inspect the code:

   So the plugin already receives the GraphQL endpoint through the endpoint option.

2. The IPluginOptionsKeys TypeScript type inside plugin/src/types.ts currently has a generic [key: string]: any catch-all. Replace it with your endpoint option so that both internally and externally the types are correct:

3. Open the file plugin/src/source-nodes.ts and validate that you can access the option. You can do this by accessing the second parameter of the sourceNodes function and console.log it. Use the IPluginOptionsInternal type for it:

4. Restart the develop:site script and you should see an output in the terminal like this:

   Don’t worry about the plugins key there. Since Gatsby’s plugins can also have sub-plugins, the key plugins is added by default to the pluginOptions object. What you really should care about is that the endpoint option successfully comes through.

5. Destructure endpoint from the pluginOptions object and use it in the fetchGraphQL call:

   Once again, restart the develop:site script and you should see the log Sourcing from plugin API - 0.033s - Processing 3 posts and 2 authors to indicate that the sourcing was successful.

You’ve successfully used a plugin option in your source plugin’s source code.

Task: Verify endpoint option

Right now a user could use any serializable value for endpoint like numbers, objects, strings, or booleans. But those are not valid options for endpoint, it should be a string, more specifically a valid URI. Use the pluginOptionsSchema API to validate exactly that.

1. Create a new file called plugin-options-schema.ts inside the plugin’s src folder with the following contents:

   pluginOptionsSchema uses Joi to define a validation schema. Use its API documentation to discover all available options.

2. Define endpoint in the schema. It should be a string (a valid URI), should be required, and have a short description. You can write something like this:

3. Export the pluginOptionsSchema API in the plugin’s gatsby-node file so that it is run:

4. You can test that it works correctly by opening up the site’s gatsby-config and changing the option to an invalid value, e.g. entering a number:

   Restart the develop:site script. The development server should crash and show an error:

   It’s working correctly, great! Go back to site/gatsby-config.ts and use the correct endpoint option again.

Joi as a validation library is really powerful and as shown in Configuring Plugin Usage with Plugin Options you can define a really fine-grained validation schema.

Summary

Awesome! You’ve learned a lot about plugin options.

Take a moment to think back on what you’ve learned so far. Challenge yourself to answer the following questions from memory:

• How can you use the reporter API and what methods does it have?
• What is the difference between the methods panicOnBuild and panic?
• How can you show an activity for e.g. the time it takes to source data?
• What are the advantages of custom errors defined through setErrorMap?
• Which type of storage is the cache API and for which purposes is it useful?
• How can you access plugin options?
• How can you validate that the user input for plugin options is correct?

Key takeaways

• Use the reporter API and its methods to output information (warnings, errors, logs, timers, etc.) to the terminal during gatsby develop and gatsby build. You can also stop the entire Gatsby process on errors if necessary.
• By default, a generic error handler is used. You can use the setErrorMap API to define your own, custom errors to show richer information to your users.
• The cache API gives you access to a key-value store to save data between runs.
• Use the pluginOptionsSchema API to validate the input of your plugin options.

What’s coming next?

In Part 5 you’ll learn all about Incremental Builds and how to best build your API and plugin for it.