Developing your App
Primitive: Streaming Functions
Creating and Using Moose Streaming Functions

Create a Streaming Function in CLI

Initiate your streaming function setup in Moose with the moose function init CLI command. This command structures your project's directory and creates necessary files based on your specified data models.

Terminal
moose functions init --source <YourSourceDataModel> --destination <YourDestinationDataModel>

This command takes two arguments: --source and --destination, which should each be followed by the name of the source and destination data models you wish to connect through this streaming function. These data models must exist before you create your streaming function. You can find a list of your existing data models in /datamodels/models.ts.

By way of example:

Terminal
moose functions init --source UserActivity --destination ParsedActivity

Represents a streaming function from a source data model UserActivity and a destination data model ParsedActivity, which themselves were defined in /datamodels/models.ts:

models.ts
import { Key } from "@514labs/moose-lib";
 
export interface UserActivity {
    ...
}
 
export interface ParsedActivity {
    ...
}

Ensure your development server is running to keep Moose updated with your data model definitions.

When you run this command, you will create the streaming function file and file structure described below.

Directory Structure Post-Initialization

Running the moose function init command with valid arguments will create a .ts file to represent this newly created streaming function, in the following directory structure:

      • UserActivity__ParsedActivity.ts
  • In the example above

    Terminal
    my-moose-app/functions/UserActivity__ParsedActivity.ts

    Where:

    UserActivityis the source data model Moose monitors for new data.
    ParsedActivityis the destination data model where Moose routes the transformed data.

    This TypeScript file is where you define the transformation needed to get from your source data model to the destination. In the file created, Moose provides starter code. This code includes imports for your source and destination data model interfaces and sets up a default function template. The function is pre-configured with the correct argument and return types based on the specified data models:

    UserActivity__ParsedActivity.ts
    // Add your models & start the development server to import these types
    import { UserActivity } from "/path/to/UserActivity";
    import { ParsedActivity } from "/path/to/ParsedActivity";
     
    // The 'run' function transforms source data to destination format.
    // For more details on how Moose streaming functions work, see: https://docs.moosejs.com
    export default function run(event: UserActivity): ParsedActivity | null {
      return null;
    }

    Develop Transformation Logic

    Now that you have your streaming function file, you have to define the transformations within it: how you want to change the data in the source data model to get to the output anticipated by the destination data model.

    This requires a default export function: The transformation function should be the file's default export, allowing Moose to automatically identify and execute it.

    And allows for type definitions: Use TypeScript interfaces generated for your data models to type-check the function's input and output, ensuring they match the source and destination data models, respectively.

    Here's an example streaming function file that converts timestamps to UTC:

    UserActivity__ParsedActivity
    // Example streaming function: Converts local timestamps in UserEvent data to UTC.
    // Imports: Source (UserActivity) and Destination (ParsedActivity) data models.
    import { UserActivity } from "/path/to/UserActivity";
    import { ParsedActivity } from "/path/to/ParsedActivity";
     
    // The 'convertUtc' function transforms UserActivity data to ParsedActivity format.
    export default function convertUtc(
      source: UserActivity,
    ): ParsedActivity[] | ParsedActivity | null {
      // Convert local timestamp to UTC and return new ParsedActivity object.
      return {
        id: source.id, // Retain original event ID.
        userId: "puid" + source.userId, // Example: Prefix user ID.
        activity: source.activity, // Copy activity unchanged.
        timestamp: new Date(source.timestamp), // Convert timestamp to UTC.
      };
    }

    Adjust function specifics according to your data transformation needs. The function's return type can be null (meaning no transformation will take place), a single object, or an array of objects. This structured approach ensures clarity in your data processing tasks, enabling Moose to efficiently handle and automate streaming functions.

    Splitting Data Into Multiple Entries

    To create multiple entries you can return a array of the destination data model. For example using the code above you can return an array with a type of ParsedActivity[]. MooseJS would convert each object inside of the array into a seperate database entry.

    Validate Your Streaming Function

    To test and confirm your streaming function's logic:

  • Send Test Data: Ingest test data into the source data model to trigger the function.
  • Verify Transformation: After the streaming function processes the data, check the destination table in the database to see if the data has been transformed correctly.
  • For detailed instructions on data ingestion, consult the Ingesting Data section of the documentation.

    ℹ️

    Disclaimer: Streaming Functions are in alpha, meaning they can currently be used only in development environments and are not ready for production deployment. Progress is being made to enable the use of streaming functions in production, with updates expected soon.