Ingest Data

Once you've defined a Data Model, MooseJS makes it easy to ingest source data from upstream sources that aligns to the schema of your Data Model. There are two main ways to ingest data:

Ingesting via API Endpoints

As you define data models in your service, MooseJS will automatically generate ingestion points for you to send data to. Ingestion points are HTTP endpoints that you can send data to. The default port is 4000. The standard endpoint for each data model is:

http://localhost:4000/ingest/<DataModelName>

Following is some example code for testing the API endpoints for a Data Model called UserActivity. In the Moose developer console, when you view your Data Model, go to the Setup tab, and you'll find sample code for utilizing the API Endpoints for your specific models.

curl -X POST \
-H "Content-Type: application/json" \
-d '{"eventId": "1234567890", "timestamp": "2019-01-01 00:00:01", "userId": "123456", "activityType": "click"}' \
http://localhost:4000/ingest/UserActivity

Ingesting via MooseJS SDKs

To facilitate the ingestion process, Moose can generate automatically a TypeScript SDK for every data model defined in your project. The SDK provides instant type safety and schema validation, offering real-time feedback on the data you intend to send to Moose during development.

While Moose also provides HTTP endpoints for data ingestion (see above), we recommend utilizing the SDK for more robust data integrity enforcement and reduced runtime errors. We're actively expanding support to include more programming languages and welcome your input on which ones to prioritize.

Getting Started with the SDK

From within your moose project you can generate the SDK for your project by running the following command:

moose sdk generate

This will generate a new directory in your project called sdk that contains the SDK for your project.

You can use this command outside of a project and you can specify the directory you wish the directory to be created in. You can use this functionality for generating the SDK in the project that will call the moose api for example.

Using the SDK to Send Data

To capture data with your data models, import the respective send method from your Moose SDK into your application code. Each data model has a corresponding method that facilitates the transmission of data to the appropriate destination table in your database. The method name follows the format send<DATA_MODEL_NAME>.

For example, if you're sending data for a UserActivity data model, your code snippet will look like this:

import sendUserActivity from "my-moose-project-name-sdk";
...
// YOUR APPLICATION CODE

// WHEREVER YOU NEED TO LOG AN EVENT
sendUserActivity({
  eventId: your_previously_defined_eventId,
  timestamp: new Date(),
  userId: your_previously_defined_userId,
  activity: "Login"
});

// REST OF YOUR CODE...