MS Pic2Card

Overview

Pic2Card is developed to solve a challenging problem of generating usable UI template from Image. This conversion process involves identifying different elements, collecting the elements’ attributes and details about the layout. It detects and extract the element details from the Image by using both neural network logic and rule-based systems. To plot the card-based UI, the output is created in the form of JSON that adhere to Adaptive Card’s schema. Our object detection model performs with a mAP score of 60%, and overall code generation quality is measured with a BLEU score of 90%.

Challenge

Card-based UI is becoming a preferred format for most front-end applications. But a product owner must wait till a UI developer develops an initial version of the screens to get a feel of the app. And this wait-time directly impacts the release cycle since it takes some effort to accommodate each correction/feedback from the Product owner. Pic2Card OSS to solve the challenging problem of generating a usable UI template from Image. The other challenge we faced during implementation is with the data, not having adequate data, data with all permutations, and normalized data set for our ML model to learn

Solution

We used a combination of computer vision, deep learning, and heuristic models to understand the objects and formatting ​This conversion process involves identifying different elements and collecting the elements’ attributes and details about the layout. It detects and extracts the element details from the Image by using both neural network logic and rule-based systems. To plot the card-based UI, the output is created in the form of JSON that adhere to Adaptive Card’s schema. When we understood Data is the real bottleneck to solving the problem, we produced synthetic Data to avoid bias, and achieve a large dataset for training.​

Outcome

We achieved creating adaptive cards by plotting the card-based UI, the output is created in the form of JSON that adhere to Adaptive Card’s schema. Pic2card pipeline implementation is available atRepo