Android SDK

Using our Android SDK makes it easy to integrate Shen.AI into a new or existing native Android app.

The best way to get started is to download and build the example app - available here on GitHub.

Installing the SDK package#

Download the Shen.AI aar packages (shenai_sdk and shenai_sdk_android) and add them to your project as a dependencies in build.gradle.

dependencies {
    implementation files('./path/to/your/shenai_sdk.aar')
    implementation files('./path/to/your/shenai_sdk_android.aar')
}

Needed permissions#

To use Shen.AI your app needs to have certain permissions. Obviously, permission to use camera but also Internet access to check the license. It is possible to use Shen.AI without Internet connection, but first usage on the device always needs to check the license, so the permission is obligatory. Adding following lines to your AndroidManifest.xml will suffice.

<uses-permission android:name="android.permission.CAMERA" />
<uses-feature android:name="android.hardware.camera" android:required="true" />
<uses-permission android:name="android.permission.INTERNET"/>

Preparing necessary app components#

To use Shen.AI your Android app needs to fulfill two conditions. First you need to have an Android Activity, which you can later attach to Shen.AI handler, as in the example app:

import ai.mxlabs.shenai_sdk_android.ShenAIAndroidSDK

...

private val shenaiSDKHandler = ShenAIAndroidSDK()

...

shenaiSDKHandler.attachToActivity(this)

The attached activity is used for camera initialization and to access app files.

The second condition is to provide an Android Surface into which Shen.AI can render the camera stream along with the measurement information so that you can display it in your app. In example providing the surface to Shen.AI is implemented inside the surfaceCreated callback:

override fun surfaceCreated(holder: SurfaceHolder) {
    renderingSurface = holder!!.surface

    shenaiSDKHandler.provideSurfaceForRendering(renderingSurface)

    ...
}

Initializing the SDK#

Now you should be ready to initialize Shen.AI. You have to provide API_KEY and USER_ID to check the Shen.AI license.

val settings = shenaiSDKHandler.defaultInitializationSettings
var result = shenaiSDKHandler.initialize(API_KEY, USER_ID, settings);

result variable can take values according to enum

enum InitializationResult {
    OK,               // Initialization successful!
    INVALID_API_KEY,  // Invalid api key, check your license
    CONNECTION_ERROR, // Couldn't check the license, check your connection
    INVALID           // Initialization not successful because of internal error
}

You can check if Shen.AI in initialized by calling isInitialized(). You can also deinitialize it, to start a new session with deinitialize().

Starting the measurement#

After initialization Shen.AI is ready to perform the measurement. For that purpose, the user needs to locate their well-lightened face in the middle of the screen. You can provide the feedback for the user about their face position and lighting conditions by using getFaceState() and getLightingConditions() calls. They return the following enums accordingly:

enum FaceState {
    OK,
    NOT_CENTERED,
    TOO_CLOSE,
    TOO_FAR,
    UNSTABLE, // too much head movement
    INVALID,
}

enum LightingConditions {
    OK,
    DARK,
    TOO_DARK,
    TOO_BRIGHT_BACKGROUND,
    TOO_BRIGHT_FACE,
    TOO_BRIGHT,
    INVALID
}

You can check if the conditions are satisfied by calling isReadyForMeasurement(). If the face is located properly you should be able to start the measurement. beginMeasurement() call returns following enum informing if the measurement has successfully started or the reason of failure:

enum BeginMeasurementStatus {
    STARTED,              // Measurement started
    RESUMED,              // Measurement resumed after being paused TODO: should be in the interface?
    NOT_STARTED_CAMERA,   // Measurement not started because of camera problem
    NOT_STARTED_FACE,     // Measurement not started because of wrong face position
    NOT_STARTED_LIGHTING, // Measurement not started because of wrong lighting conditions
    INVALID               // Measurement not started because of internal error
}

You can also check that the measurement has started by calling isMeasuring(). If you want to abort the measurement call abortMeasurement().

During the measurement#

Shen.AI is performing the measurement in the background and rendering the blood flow on the given surface, but you can still provide user feedback about their face position and lighting conditions.

To see what is happening underneath you should be calling getEngineState(). It returns the following enum informing you about the state of the Shen.AI engine and the measurement.

enum EngineState {
    NOT_READY,              // Not ready (generic)
    NOT_READY_FACE,         // Face not detected
    NOT_READY_PARTIAL_FACE, // Face is not fully in the frame
    READY,                  // Face detected in acceptable position, ready to start
    PERFORMING_INITIAL_FIT, // Facial model fitting in progress
    TRACKING_SIGNAL_SHORT,  // Tracking active, signal too short for any conclusions
    TRACKING_BAD_SNR,       // Tracking active, last frames have poor signal quality
    TRACKING_GOOD_SNR,      // Tracking active, last frames have good signal quality
    TRACKING_LOST,          // Tracking lost, trying to relocate
    FINALIZING,             // Measurement has satisfied end conditions and is performing final computations
    SUCCESS,                // Measurement has satisfied end conditions and finished
    FAILURE,                // Measurement has failed and finished
    ENGINE_NOT_INITIALIZED  // Engine not initialized
}

In particular, you should wait for FINALIZING, SUCCESS or FAILURE values, because they indicate that the measurement has finished.

For first 5-10 seconds you can call getHeartRateProgressPercentage() returning float between 0 and 100 informing about progress of heart rate calibration. When the value is 100 you can check current heart rate in real-time using getLatestHeartRate() call. Other metrics, such as HRV, are available after the full measurement is completed.

To check the progress of the whole measurement you can call getMeasurementProgressPercentage() returning float from 0 to 100.

For measured values to be accurate the quality of the extracted blood flow signal is important. You can check how this quality varies calling getCurrentSignalQualityMetric() returning float value representing how good was last 5 seconds of the signal. If the quality is too low, you will see no progress in the measurement completeness.

Measurement completed#

When the Shen.AI engine enters the SUCCESS state it means that the measurement has finished. You can now obtain the results calling getLatestMeasurementSummary(), which returns the following structure:

class MeasurementSummary {
    public float hr_bpm;      // Heart rate in "beats per minute"
    public float hrv_sdnn_ms; // HRV using SDNN metric in ms
    public float br_bpm;      // Breathing rate in "breaths per minute"
}

Testing#

For integration testing purposes we provide also crash() function in the interface. After calling it you should expect the Shen.AI to crash.