Graphing on Android: An Evolution from Java to JavaScript

Back in 2014, we began work to allow application builders to define graphical visualizations that would display in their CommCare applications. Since CommCare provides a platform for collecting virtually any kind of data, the scope of visualizations that users might want to create was theoretically infinite. To ground this work, we focused on two specific use cases for health care in clinical settings: growth charts, which are commonly used in pediatrics, and the partograph, a graphical record of vital measurements tracking childbirth. These two visualizations would add value to existing projects, and they were complex enough visualizations that supporting them would also mean supporting many simpler use cases.

The first implementation of the graphing functionality integrated CommCare with a third-party, open source Android library. Some time later, we revisited graphing in the interest of making the generated graphs look more professional. We ended up introducing JavaScript and WebViews to our previously fully-native Android client. This approach has enabled us to support higher-quality visualizations, although it has had some performance costs.

Requirements

From the two main use cases – growth charts and partographs – we broke out more granular requirements:

• Line graphs
• Bubble graphs – we chose to replace the partograph’s unique bar graph, which uses differently-colored bars to indicate contraction strength, with a bubble chart, a more standard way to depict three dimensions of data
• Multiple series
• Multiple y axes
• Shading and color to highlight critical areas
• Text annotations to label critical areas

Left image: A sample growth chart from the CDC, where a clinician would record an infant girl’s length and weight over time, comparing her growth to reference lines that describe infants at the 50th, 75th, etc. percentile. Right image: A completed partograph. The top three graphs contain the most critical information: fetal heart rate, cervical dilation, and a shaded bar graph that depicts contraction frequency and strength. Image credit: WHO.

Besides the concrete requirements for the two main use cases, we considered how graphing functionality might evolve over time. In particular, adding interactivity in the future sounded appealing, perhaps a graph that would allow a user to add or correct data and then redraw itself immediately. While this wasn’t a hard requirement, it was a consideration in the early stages of this initiative.

Initial Implementation (Java)

The first implementation of a partograph in CommCare.

The set of required graphing functionality was too large to justify implementing it from scratch, particularly for a research and development effort. Instead, the implementation was a matter of evaluating third-party libraries and integrating one with CommCare.

This was a fairly quick search. No more than half a dozen Android libraries looked like plausible candidates, and our fairly lengthy list of requirements narrowed the candidates down fast. The most limiting of the concrete requirements turned out to be the need for a bubble graph, a somewhat uncommon chart type. While evaluating the handful of libraries we’d found, AChartEngine emerged as the most fully-featured.

The implementation was intended to be only loosely coupled with AChartEngine, in case we needed to change libraries at some point. So it defined a set of classes to model graphing concepts, and it then integrated these classes with AChartEngine. This implementation covered the hard requirements, but thinking about interactivity was pushed into the future, since we didn’t have a concrete use for it yet and hadn’t found a library that supported it well.

Adoption

The first implementation held up reasonably well for about a year. We added functionality as requests came in, but none required major code restructuring or, the major risk, switching libraries. Graphing-related changes during this time consisted of:

• Adding time-based data, needed by health projects to show change in vitals or other patient data over time.
• Fixing minor visual issues, which involved us forking AChartEngine.
• Adding bar graphs, which was inspired by a desire to see report data on mobile devices. The HQ server team completed an initiative to make mobile worker performance data available on devices, and the new graphing code allowed applications to create bar graphs showing community health workers their performance relative to their peers. This kind of feedback has the potential to improve mobile worker performance, an exciting area for future work.

In the summer of 2015 we started to seriously consider replacing AChartEngine, largely for visual reasons. In particular, while it had been technically easy to implement bar graphs, they were prone to having text labels that overlapped or spilled into the data area of the graph, making the data difficult to interpret. We decided to revisit graphing on Android and evaluate the current state of the art.

Re-implementation (Javascript), or, Margins are Hard

Our second round of evaluating graphing libraries started much as the first one did, with searches for Android-based libraries. We found that while Android graphing libraries had gotten more slick, there still weren’t many options that supported CommCare’s wide array of features. So this time around, we also considered the approach of using a JavaScript graphing library to build an HTML visualization and then using a WebView to display this visualization in CommCare. JavaScript had a much larger and more mature ecosystem of graphing libraries from which to choose.

An early graph of mobile worker performance data.

JavaScript had its drawbacks. Introducing JavaScript (and a bit of CSS and HTML) into CommCare’s pure Java codebase would add technical complexity. However, since Dimagi also builds a web-based application, the team has substantial JavaScript expertise. At a mobile-only company, this would be a bigger risk. Also, a JavaScript-based solution was likely to be slow – by definition, using a WebView adds another layer to the Android app and won’t perform as fast as native code.

So the major tradeoff between native Android library and JavaScript library plus WebView ended up being more mature and professional-looking JavaScript functionality versus the performance hit that comes with using WebViews. Ultimately, JavaScript won out.

While JavaScript has quite a few graphing libraries, most of them, like most of the Android libraries, weren’t fully-featured enough for our use. Most apps that display graphs are displaying a specific graph configuration and a known set of data. CommCare, as an app-building platform, is one level more abstract; we were creating a graphing platform where users could configure their own graphs, with their own data, and expect that the resulting graph would look reasonable. After limiting the candidate libraries to those that supported the necessary functionality, the remaining decision-making mostly came down to the details of how well the library adjusted to different data sets, and specifically to the minor-sounding detail of how well the library handled margins and axis labels.

Creating a graphing library that properly handles labels, without long labels overlapping each other or running out of the bounds of the view, is a challenging problem, and not many libraries handle it well. Some deal with this problem by having the calling code to specify exact margins, which would be fine if we were displaying known data, but doesn’t work well with dynamic labels. We could have exposed that configuration to end users, but creating a graph already involved a lot of configuration, without users needing to worry about making pixel-level visual tweaks to their graphs.

There were three major contenders to replace AChartEngine. D3 was reliable, high-quality, and gives the developer complete control over the visual output, but it didn’t offer pre-canned visualizations, so it could be time-consuming to implement. Google Charts handled dynamic labels and margins well, but its license prohibited offline use, which made it a non-starter for CommCare. Vega looked interesting and powerful, but its data model didn’t map quite as well to CommCare’s data models, so it would be a higher level of effort to implement.

D3 worked the best for our needs once we added one more library, C3, which added a layer on top of D3 to provide standard visualizations like line charts and bar charts. C3 took a JSON object of options as input and fed it to D3 to generate reasonable-looking graphs consisting of SVG elements, with the visual details handled pretty well, including calculating appropriate spacing so that text labels didn’t overlap.

Integrating with C3 primarily meant writing Android code that translated CommCare graphing configuration into that options object. For functionality that C3 didn’t natively support, like different shapes for points, we added JavaScript to manipulate the final SVG object – a bit hacky, but effective.

Present and Future State

The partograph, after re-implementing graphing.

The list of specific improvements added by the re-implementation is fairly short,but makes a non-trivial difference in graphs’ look and feel. The new implementation added opacity, which makes a surprising amount of difference in making something look professional, and indeed handled issues of overlap much better, also a big win for professionalism. As a minor benefit, the new version of graphing supports a little bit of interactivity, in the form of tooltips.

Having direct access to the SVG elements in the graphs allows for hacky but fine-grained control when needed. This enabled us to support a feature where bar graphs highlight a single bar in a series with a unique color – not a typical feature of bar graphs, and not one C3 is likely to support any time soon, but a win for worker performance reports, where it’s beneficial to make the user’s own data visually different from that of their peers.

Unfortunately, performance has indeed degraded: graphs can now take seconds to load, which feels like eternity for users. Investigating this didn’t yield much low-hanging fruit: we minified the web files and added UI elements so it’s apparent to the user that the graph is loading, not broken. We’ve done some experimenting with rendering in canvas instead of SVG and with rendering the graphs in the background as PNGs, but this hasn’t been fruitful. It’s also difficult to share code between graphs when there are several on one screen, since the underlying CommCare code means that each graph is rendered as its own WebView. The next step in improving graphing performance will likely involve non-trivial code restructuring.

From a technical complexity perspective, the JavaScript approach has been mostly positive. It worked out well when we needed to print graphs, since the printing code already accepted an HTML template as input. Debugging in Chrome is relatively smooth, but the interaction between android and JavaScript isn’t always easy to understand. The commcare-android codebase, once wholly Java, is now about 10% JavaScript.

The new language breakdown of our commcare-android repository.

The Next Frontier

Beyond performance, the next frontier in graphing in CommCare is likely figuring out how to fit it into a good product experience. Graphing is still part of research and development, available to a limited number of projects, and even if it were in the main product today, configuring a graph while building an application in CommCareHQ is prohibitively complex. In an ideal world, a user could add a growth chart or a worker performance report with minimal configuration, yet an advanced user would still have the power to put together a visualization as complex as the partograph.