Create Flow in CLI

Initiate your flow setup in Moose with the moose flow init CLI command. This command structures your project's directory and creates necessary files based on your specified data models.

moose flow init --source <YourSourceDataModel> --destination <YourDestinationDataModel>

This command takes two arguments: --source and --destination, which should each be followed by the name of the source and destination data models you wish to connect through this flow. These data models must exist before you create your data flow. You can find a list of your existing data flows in /datamodels/models.ts.

By way of example:

moose flow init --source UserActivity --destination ParsedActivity

Represents a data flow from a source data model UserActivity and a destination data model ParsedActivity, which themselves were defined in /datamodels/models.ts:

type Key<T extends string | number> = T
 
export interface UserActivity {
    ...
}
 
export interface ParsedActivity {
    ...
}

Ensure your development server is running to keep Moose updated with your data model definitions.

When you run this command, you will create the data flow file and file structure described below.

Directory Structure Post-Initialization

Running the moose flow init command with valid arguments will create a flow.ts file to represent this newly created data flow, in the following directory structure:

In the example above

my-moose-app/flows/UserActivity/ParsedActivity/flow.ts

Where:

The flow.ts file is a TypeScript file that you use to define the transformation needed to get from your source data model to the destination. In the flow.ts file created, Moose provides starter code. This code includes imports for your source and destination data model interfaces and sets up a default function template. The function is pre-configured with the correct argument and return types based on the specified data models:

// Add your models & start the development server to import these types
import { UserActivity } from "/path/to/UserActivity.ts";
import { ParsedActivity } from "/path/to/ParsedActivity.ts";
 
// The 'run' function transforms source data to destination format.
// For more details on how Moose flows work, see: https://docs.moosejs.com
export default function run(event: UserActivity): ParsedActivity | null {
  return null;
}

Develop Transformation Logic

Now that you have your data flow file, flow.ts, you have to define the transformations within it: how you want to change the data in the source data model to get to the output anticipated by the destination data model.

This requires a default export function: The transformation function should be the file's default export, allowing Moose to automatically identify and execute it.

And allows for type definitions: Use TypeScript interfaces generated for your data models to type-check the function's input and output, ensuring they match the source and destination data models, respectively.

Here's an example flow.ts that converts timestamps to UTC:

// Example flow function: Converts local timestamps in UserEvent data to UTC.
// Imports: Source (UserActivity) and Destination (ParsedActivity) data models.
import { UserActivity } from "/path/to/UserActivity.ts";
import { ParsedActivity } from "/path/to/ParsedActivity.ts";
 
// The 'convertUtc' function transforms UserActivity data to ParsedActivity format.
// For more details on how Moose flows work, see: <https://docs.moosejs.com/flows>
export default function convertUtc(event: UserActivity): ParsedActivity {
  // Convert local timestamp to UTC and return new ParsedActivity object.
  return {
    id: event.id, // Retain original event ID.
    userId: "puid" + event.userId, // Example: Prefix user ID.
    activity: event.activity, // Copy activity unchanged.
    timestamp: new Date(event.timestamp.toUTCString()), // Convert timestamp to UTC.
  };
}

Adjust function specifics according to your data transformation needs. This structured approach ensures clarity in your data processing tasks, enabling Moose to efficiently handle and automate data flows.

Validate Your Flow

To test and confirm your flow's logic:

For detailed instructions on data ingestion, consult the Ingesting Data section of the documentation.