Flow Results support for Turn.io
This investment will go towards the development of the necessary API functionality to allow for data captured to be shared off platform.
The larger goal is to facilitate data sharing between Turn.io and the larger technology ecosystem.
Turn.io implements and exposes the WhatsApp Business API, however, the WhatsApp Business API does not expose any functionality for exporting of captured data during a conversation, this needs to be implemented by Turn.io.
Rather than inventing our own proprietary format to address this problem, we want to align with the larger international development community to facilitate interoperability.
The concept of a flow is already implemented as a feature called Threads in the Turn.io application.
Thus the Flow Results support will export the data captured in Turn.io Threads as a Flow Results data package.
Turn.io allows for social impact teams to manage conversations and capture data using the WhatsApp Business API at scale.
Data is already being captured and stored on the Turn.io platform, making the data available as a Flow Results data package allows for interoperability with the larger international development and tech ecosystem.
The work will be added to the Turn.io roadmap and implemented by the Turn.io product team. The team works with weekly iterations and larger feature work is allocated per quarters.
If the grant is approved, the work will be scheduled in the current or next up quarter.
The work will be led by Turn.io’s product manager.
Supporting organisations are some of Turn.io’s key clients, including the WHO, The South African Department of Health, and Praekelt Foundation.
The WhatsApp covid-19 services offered by the WHO and The South African Department of Health are among the largest WhatsApp Business API installations in the world.
Problem statement:
Many social impact organisations are using Turn.io to capture data using Threads, Turn.io’s implementation of the Flow concept.
Currently, Turn.io allows for data to be exported off platform in real-time using webhooks or via a data lake integration to BigQuery.
Both mechanisms are powerful but require a relatively high level of technical expertise to use effectively and there is no easy way to export data captured using Turn.io threads to allow interoperability with the larger international development technology sector.
Objectives:
Allow for data captured with Turn.io threads to be exported as a Flow Results data package by:
- Exposing an HTTP API endpoint per Turn.io Thread which returns the data captured in the given Thread as a paginated Flow Results data package.
- Allowing the UI to download a Flow Results data package as a zip file combining the resource description and the resource data. This will build on the API but not require the Turn.io user to worry about pagination.
Deliverables:
- An open-source Flow Results Data Package library for the Elixir programming language hosted on hex.pm and https://github.com/turnhub licensed under an OSI recognised license.
- Published documentation on Turn.io’s support for Flow Results on https://learn.turn.io
- Published documentation on Turn.io’s API support for Flow Results on https://whatsapp.turn.io/docs
- Support in Turn.io’s Thread builder UI for downloading the captured data as a Flow Results Data Package.
Schedules:
- We expect this work to take 2 to 3 weeks from start to a first beta release exposed as a feature flag in the Turn.io UI.
- Work will start depending on the time of approval and work allocation at the time, and expect it to be scheduled within a calendar quarter.
Risks & Mitigations:
Many of the clients using Turn.io run at significant scale in terms of uniques. One risk is that the approach of generating a ZIP file does not scale for very large initiatives.
If that is the case then the API is a possible fallback and a mitigation is to then restrict the ZIP file based approach to a to be determined maximum number of records.