Slack Workflows Gotchas: Problems with Programming Custom Slack Bots

I built a Slack bot with a coworker for our internal company hackathon. It was….an experience! Here are some notes that might be useful to other folks.

Our team runs a company-public “help” channel, which is meant for folks outside of our team to ask questions. Unsurprisingly, many of these questions are repetitive, and answering them is a drain on the on-call engineer’s time. I looked back at the last month of questions and found that ~35% of them were good candidates for an automated response. So, I wanted to build a bot that would automatically reply to certain keywords in a specified channel with a canned response.

The first hiccup was: how do we interact with Slack? Slack has a lot of different ways to build automations: there are SDKs for standalone apps that interact with Slack via webhooks, there’s the “Workflow Builder” GUI, there are “workflows”, there are “Slack apps”, there’s Slackbot….

In order to respond to the message_posted event trigger, you need to build a Workflow (not to be confused with the Workflow Builder). Sometimes the docs refer to this as a “Workflow App.”

A Workflow is basically a special app that can be deployed to Slack. You can’t put it in the Slack apps directory. It must be written in Typescript, and you don’t run it as a standalone program.

The entire Workflows developer experience has the vibe of “some folks put a lot of work into this, a while ago.” There are a lot of really cool use-cases for Slack automations, but there are also a bunch of gotchas and rough edges to the API. Here’s some of my notes from working with the Workflows API:

The sample apps are your friend

There’s a robust repo of example projects that are the best point of reference for using the API. While the docs are also useful, I found that the Slack sample apps were the best reference to use.

Nested arrays and objects don’t work.

We ran into all sorts of bugs when trying to store arrays inside of objects or objects inside of arrays, both when interacting with the datastores but also when just trying to pass parameters between functions.

There’s an open bug from 2023 where storing an object containing an array doesn’t work. Hopefully that will get fixed eventually?

No conditionals or early termination

Workflows consist of a series of steps. There’s no way to build in conditional branching logic into a Workflow despite customer demand. Conditionals in Workflow Builder were announced as an upcoming feature back in 2022, but there’s been no news about them since.

There’s also no way to end a workflow early. This becomes a problem if a necessary precondition is not met, because of how errors are handled.

Error handling

If your workflow encounters an error, Slackbot will DM the workflow’s admin. There is no way to mute these messages.

Because of this, it’s really important that you wrap each step in a try/catch if there’s a chance that it will error.

Call the Slack client from a custom step, NOT from a builtin step.

I found that the Slack builtin methods for doing things will throw errors on normal user interactions (e.g. closing a modal window) which will then trigger a DM to the workflow admin. Yikes!

The docs generally advise you to use one of the builtin Slack functions from Schema.slack.functions for interacting with the Slack API, but you can instead use the client param which is available in custom functions to call the Slack API.

For example, the docs might suggest doing this:

MyWorklflow.addStep(
  Schema.slack.functions.SendMessage,
  {
    channel_id: inputs.channel_id,
    message: `Setting topic to ${generateMessageStep.outputs.message}`,
  },
);

But instead, we did this from a custom Workflow step:

let post_args: {
	"channel": string;
  "text": string;
        } = {
          channel: inputs.channel_id,
          text: inputs.message,
        };

await client.chat.postMessage(post_args);

Similarly, I really don’t recommend using the Schema.slack.functions.OpenForm Slack function in your workflows, because it will error if the user closes a modal prematurely.

Instead you can do something like:

export default SlackFunction(
  OpenModal,
  async ({ inputs, client }) => {
    // Open a new modal with the end-user who interacted with the link trigger

    const view: ModalView = {
      "type": "modal",
      
      // Note that this ID can be used for dispatching view submissions and view closed events.
      "callback_id": "first-page",

      // This option is required to be notified when this modal is closed by the user
      "notify_on_close": true,
      
      "title": {
        "type": "plain_text",
        "text": MODAL_TITLE,
      },
      "close": { "type": "plain_text", "text": "Close" },
      "blocks": modalBlocksWithRows(inputs.entries),
    };

    const response = await client.views.open({
      interactivity_pointer: inputs.interactivity.interactivity_pointer,
      view: view,
    });
  }
 );

Not only does calling a custom function let you do bespoke error handling, we also encountered problems where the Slack built-in functions would inexplicably error but our custom functions worked fine. Unclear what was going on there.

Mismatches between CLI help messages and behavior

There are a few places where the CLI help messages provide commands that don’t actually work.

For example, when trying to count the number of entries in a datastore:

$ slack datastore count --help
...
# Count all items in a datastore
$ slack datastore count --datastore tasks

If you try to actually do this, it will error.

$ slack datastore count --datastore response_store
Check .slack/logs/slack-debug-20240514.log for full error logs

🚫 No expression was provided (invalid_datastore_expression)

Suggestion:

Provide a JSON expression in the command arguments
Use slack datastore count --help for an example
Build a new expression using slack datastore count --show --unstable

You can’t delete a Slack workflow’s datastore.

Slack can create a datastore for you, and the CLI provides a nice interface for creating datastores, and adding or removing data. However there is no support for deleting an entire datastore.

To remove a datastore, you must remove the datastore from your manifest and force-deploy, then add the datastore back in and re-deploy.

JSONL bulk put doesn’t work?

Maybe there’s a way to do this that I haven’t been able to figure out. The docs claim that bulk-put is supported for datastores but I wasn’t able to make this work.

The docs suggest running this command like so:

slack datastore bulk-put '{"datastore": "running_datastore"}' —-from-file /path/to/file.jsonl

The --from-file flag is also documented via the help command in CLI:

$ slack datastore bulk-put --help
Create or replace a list of items in a datastore

USAGE
  $ slack datastore bulk-put <expression> [flags]

FLAGS
      --datastore string   the datastore used to store items
      --from-file string   store multiple items from a file of JSON Lines

But, it doesn’t work.

$ slack datastore bulk-put --datastore '{"datastore": "response_store"}' --from-file assets/test.jsonl
Check .slack/logs/slack-debug-20240514.log for full error logs

🚫 No expression was provided (invalid_datastore_expression)

Suggestion:

Provide a JSON expression in the command arguments
Use slack datastore bulk-put --help for an example
Build a new expression using slack datastore bulk-put --show --unstable

Well, that sure was a hackathon.

My coworker and I did eventually get our Slack workflow up and running, and it was even kind of fun! But oof I had trouble figuring out how to get the API working. For being “just” a chat app, Slack is pretty complicated, and unsurprisingly the API surface area is complicated too. Maybe my notes will help you (or, at least, help my future self when I’m debugging this six months from now…).