Try : Insurtech, Application Development

AgriTech(1)

Augmented Reality(21)

Clean Tech(9)

Customer Journey(17)

Design(45)

Solar Industry(8)

User Experience(68)

Edtech(10)

Events(34)

HR Tech(3)

Interviews(10)

Life@mantra(11)

Logistics(5)

Manufacturing(3)

Strategy(18)

Testing(9)

Android(48)

Backend(32)

Dev Ops(11)

Enterprise Solution(33)

Technology Modernization(9)

Frontend(29)

iOS(43)

Javascript(15)

AI in Insurance(38)

Insurtech(66)

Product Innovation(58)

Solutions(22)

E-health(12)

HealthTech(24)

mHealth(5)

Telehealth Care(4)

Telemedicine(5)

Artificial Intelligence(152)

Bitcoin(8)

Blockchain(19)

Cognitive Computing(8)

Computer Vision(8)

Data Science(23)

FinTech(51)

Banking(7)

Intelligent Automation(27)

Machine Learning(48)

Natural Language Processing(14)

expand Menu Filters

Pushing the Envelope on ICR Accuracy in Hand-written Forms

5 minutes, 6 seconds read

The need for and consequently the number of solutions for reading hand-written forms in an automated manner has been on a rise for as long as one could remember. Almost all businesses to varying degrees utilize paper-based forms that are filled by customers by hand. Most if not all of these businesses convert this handwritten information into the digital format. Depending on the technological sophistication or the size of the business this digitization might be done manually by one or more data entry specialists or through an automated solution. 

It’s easy to see how the manual route may not be an ideal solution for medium or large-sized business. Some of the apparent drawbacks of manual document processing are:

  1. The cost of having data entry specialists quickly add up as more documents need to be digitized necessitating adding more resources.
  2. Manual data entry is a slow process.
  3. Manual data entry is error-prone and requires a quality inspection which is costly and not fail-proof.

Many businesses have realized this and have transitioned to some form of a partially or fully automated solution to this problem. However, it’s not all rosy for these businesses either. The problems these businesses face is primarily related to the accuracy of the current solutions in the market. 

Shortcomings of Existing Hand-written Document Processing Solutions

The industry average for ICR (Intelligent Character Recognition) accuracy at the character level is about 70% and it will drop significantly if measured at word level which is what matters at the end. Such automation may allow for reducing the number of data entry personnel but with such a low level of accuracy, there will be a need for increased quality check resources, which are often more expensive than data entry resources hence diluting the cost-benefit of automation. Moreover, since the quality check is a slower process than data entry, this kind of automation doesn’t even address the speed problem.

Some of the reasons that result in a low level of accuracy among existing document processing solutions are:

  • Poor form design
  • User input not in line with the format
  • Noisy images
  • Misaligned documents
  • Low-quality scanning of documents
  • Spelling mistakes by the user
  • Overwriting/corrections by user

While we may not have control over some of the above factors such as form design and user input, we can definitely improvise the data extraction models to account for the other factors such as image noise, misalignments, spelling mistakes etc.

Our ICR Solution

The Document Parser solution in FlowMagic provides an intuitive user interface where data can be extracted from any standard form in three easy steps:

Step 1:   The user annotates the form (this is a one-time exercise for each new form) using an easy and intuitive UI. During annotation, each input field can optionally be labelled as mandatory. The user can specify the datatype for each field as alphabets, numeric or checkbox and also set the context for the field e.g. Name, PAN, City, Car Make, Date etc. Once done, the saved template can be used repeatedly for reading forms of the same type as long as there are no changes in the form design. In case of a change, the saved template can be easily modified. 

Step 2:   The user uploads one or more forms and chooses the corresponding template (from previous annotations). The system automatically extracts data from the forms.

Step 3:  The system exports the output in CSV, XML or JSON as desired by the user. If any field was marked as mandatory during annotation, the system also outputs a list of all mandatory fields that are blank.

Salient features of ICR Document Parser

  1. The standard form being annotated can be any number of pages. The input form need not have the same number of pages. If there is a mismatch between the pages in the input form and the template, the system does a matching and runs the data extraction on matching pages only. This also means that the input form need not be sorted correctly.
  2. The system can read handwritten as well as printed forms.
  3. The system corrects for minor misalignments during scanning of documents or documents scanned in the wrong orientation.
  4. The system has inbuilt dictionaries for various contexts such as Name, Cities, States, Countries, PAN, Profession, Marital Status, Relationship, Amount, Car Make, Date, Gender.
  5. The various data types supported by the system are alphabets, numeric, alphanumeric, checkboxes and special characters.
  6. The system corrects user errors or scanning issues by performing data type and dictionary checks (see examples below).
  7. The system checks for mandatory fields to make sure the form is completely filled.

Examples of Data Read/Corrections Made by an ICR

Benefits of ICR

Flexibility – you can annotate a wide variety of forms with complex inputs and data formats using the multiple data types and contexts built into the system.

Speed – Both annotation and data extraction are very user-friendly and fast. The system can extract data from a five-page form in under 30 seconds.

Scalability – The system is highly extensible and once set up for one type of form can easily be scaled for multiple forms or to process documents in bulk of the same format.

Accuracy – The character level accuracy of our model is over 90%. Word level accuracy depends on the form design and quality but in general, varies between 75% and 85%.

Workflow

ICR (Intelligent Character Recognizer) workflow

No matter what solution you use, you can always benefit from these best practices for form design to improve the accuracy of your ICR:

  1. Have all instructions in bold at the top of the form.
  2. Instruct the user to write clearly in block letters as the form will be processed by a machine.
  3. Provide examples of how to enter data wherever there is a scope for confusion.
  4. Instead of providing a free form space for data entry, it provides a clearly marked space with a specific location to enter each character.
  5. The overall space should be large enough to contain the requisite data to avoid user writing outside of this space.
  6. Have enough separation between the space for two fields to avoid overlap.

To learn more about how FlowMagic can improve the accuracy and speed of your document digitization/Intelligent Character Recognition (ICR) or discuss your broader AI goals, please get in touch with us at hello@mantralabsglobal.com

Cancel

Knowledge thats worth delivered in your inbox

Machines That Make Up Facts? Stopping AI Hallucinations with Reliable Systems

There was a time when people truly believed that humans only used 10% of their brains, so much so that it fueled Hollywood Movies and self-help personas promising untapped genius. The truth? Neuroscientists have long debunked this myth, proving that nearly all parts of our brain are active, even when we’re at rest. Now, imagine AI doing the same, providing information that is untrue, except unlike us, it doesn’t have a moment of self-doubt. That’s the bizarre and sometimes dangerous world of AI hallucinations.

AI hallucinations aren’t just funny errors; they’re a real and growing issue in AI-generated misinformation. So why do they happen, and how do we build reliable AI systems that don’t confidently mislead us? Let’s dive in.

Why Do AI Hallucinations Happen?

AI hallucinations happen when models generate errors due to incomplete, biased, or conflicting data. Other reasons include:

  • Human oversight: AI mirrors human biases and errors in training data, leading to AI’s false information
  • Lack of reasoning: Unlike humans, AI doesn’t “think” critically—it generates predictions based on patterns.

But beyond these, what if AI is too creative for its own good?

‘Creativity Gone Rogue’: When AI’s Imagination Runs Wild

AI doesn’t dream, but sometimes it gets ‘too creative’—spinning plausible-sounding stories that are basically AI-generated fake data with zero factual basis. Take the case of Meta’s Galactica, an AI model designed to generate scientific papers. It confidently fabricated entire studies with fake references, leading Meta to shut it down in three days.

This raises the question: Should AI be designed to be ‘less creative’ when AI trustworthiness matters?

The Overconfidence Problem

Ever heard the phrase, “Be confident, but not overconfident”? AI definitely hasn’t.

AI hallucinations happen because AI lacks self-doubt. When it doesn’t know something, it doesn’t hesitate—it just generates the most statistically probable answer. In one bizarre case, ChatGPT falsely accused a law professor of sexual harassment and even cited fake legal documents as proof.

Take the now-infamous case of Google’s Bard, which confidently claimed that the James Webb Space Telescope took the first-ever image of an exoplanet, a factually incorrect statement that went viral before Google had to step in and correct it.

There are more such multiple instances where AI hallucinations have led to Human hallucinations. Here are a few instances we faced.

When we tried the prompt of “Padmavaat according to the description of Malik Muhammad Jayasi-the writer ”

When we tried the prompt of “monkey to man evolution”

Now, if this is making you question your AI’s ability to get things right, then you should probably start looking have a checklist to check if your AI is reliable.

Before diving into solutions. Question your AI. If it can do these, maybe these will solve a bit of issues:

  • Can AI recognize its own mistakes?
  • What would “self-awareness” look like in AI without consciousness?
  • Are there techniques to make AI second-guess itself?
  • Can AI “consult an expert” before answering?

That might be just a checklist, but here are the strategies that make AI more reliable:

Strategies for Building Reliable AI

1. Neurosymbolic AI

It is a hybrid approach combining symbolic reasoning (logical rules) with deep learning to improve factual accuracy. IBM is pioneering this approach to build trustworthy AI systems that reason more like humans. For example, RAAPID’s solutions utilize this approach to transform clinical data into compliant, profitable risk adjustment, improving contextual understanding and reducing misdiagnoses.

2. Human-in-the-Loop Verification

Instead of random checks, AI can be trained to request human validation in critical areas. Companies like OpenAI and Google DeepMind are implementing real-time feedback loops where AI flags uncertain responses for review. A notable AI hallucination prevention use case is in medical AI, where human radiologists verify AI-detected anomalies in scans, improving diagnostic accuracy.

3. Truth Scoring Mechanism

IBM’s FactSheets AI assigns credibility scores to AI-generated content, ensuring more fact-based responses. This approach is already being used in financial risk assessment models, where AI outputs are ranked by reliability before human analysts review them.

4. AI ‘Memory’ for Context Awareness

Retrieval-Augmented Generation (RAG) allows AI to access verified sources before responding. This method is already being used by platforms like Bing AI, which cites sources instead of generating standalone answers. In legal tech, RAG-based models ensure AI-generated contracts reference actual legal precedents, reducing AI accuracy problems.

5. Red Teaming & Adversarial Testing

Companies like OpenAI and Google regularly use “red teaming”—pitting AI against expert testers who try to break its logic and expose weaknesses. This helps fine-tune AI models before public release. A practical AI reliability example is cybersecurity AI, where red teams simulate hacking attempts to uncover vulnerabilities before systems go live 

The Future: AI That Knows When to Say, “I Don’t Know”

One of the most important steps toward reliable AI is training models to recognize uncertainty. Instead of making up answers, AI should be able to respond with “I’m unsure” or direct users to validated sources. Google DeepMind’s Socratic AI model is experimenting with ways to embed self-doubt into AI.

Conclusion:

AI hallucinations aren’t just quirky mistakes—they’re a major roadblock in creating trustworthy AI systems. By blending techniques like neurosymbolic AI, human-in-the-loop verification, and retrieval-augmented generation, we can push AI toward greater accuracy and reliability.

But here’s the big question: Should AI always strive to be 100% factual, or does some level of ‘creative hallucination’ have its place? After all, some of the best innovations come from thinking outside the box—even if that box is built from AI-generated data and machine learning algorithms.

At Mantra Labs, we specialize in data-driven AI solutions designed to minimize hallucinations and maximize trust. Whether you’re developing AI-powered products or enhancing decision-making with machine learning, our expertise ensures your models provide accurate information, making life easier for humans

Cancel

Knowledge thats worth delivered in your inbox

Loading More Posts ...
Go Top
ml floating chatbot