Online Reporter
As the agrarian world continues to change and sub-Saharan African embraces Artificial Intelligence (AI), the agribusiness sector is now forced to analyse the status and the quantitative and qualitative quality of their databases.
This status quo analysis posteriorly forces industry players in the public and private sectors to improve their data quality.
The types of data available which need to be optimally processed to get the best results from AI are structured data, unstructured data, semi-structured data, labelled data, unlabelled data, real-time (streaming) data, synthetic data and human feedback data.
Structured data is information that is highly organised and stored in fixed formats. Examples of these are databases (tables, records, fields etc.), financial and accounting data, transaction records, sensor readings (e.g., rainfall, soil moisture). This type of data is used for forecasting and prediction, risk scoring and optimisation and planning.
Unstructured data is the type that does not follow a predefined format. Examples include text (documents, emails and social media posts), images (photos, satellite imagery), audio (speech, call recordings), video (surveillance, drone footage). This data is usually used for natural language processing, image and video recognition and speech-to-text and voice assistants.
Semi-structured data is that which has some structure but not rigidly tabular. Examples are JSON, XML files, system logs, GPS and telemetry data. Usual AI uses these are in Web and mobile analytics and IoT and tracking systems.
Data that includes correct answers or tag is referred to as labelled data. Examples include images labelled “healthy crop” / “diseased crop” and messages labelled “spam” / “not spam”. Typical AI uses of these are in supervised machine learning and classification and regression models.
Unlabelled data is without predefined labels. Examples are Raw customer behaviour data and Unannotated satellite images. Typical AI uses include Clustering, Pattern discovery and Anomaly detection.
Real-time (streaming) data is generated continuously and immediately. Examples of these are weather feeds, financial market data and live sensor data from machines. This data is utilised for in real-time monitoring, predictive maintenance and early warning systems.
Synthetic data is artificially generated data used to supplement or replace real data. Examples of this are simulated images and modelled climate scenarios. Training models where real data is limited and Privacy-preserving AI development are the applications of synthetic data.
Lastly there is human feedback data, which is created from human evaluation and correction. Examples of this are ratings of AI outputs, expert annotations and reinforcement feedback. Model refinement and accuracy and safety improvement are where human feedback data are normally used.
The challenges confronting sub-Saharan Africa are the lacking of data categorisation in weak data collection frameworks, poor data standardisation, fragmented data silos, limited metadata and taxonomy design, low data quality and validation controls, skills deficit in data management, weak legal and governance frameworks and donor-driven data fragmentation.
Weak data collection frameworks are evident in that: data is collected inconsistently across ministries, companies, NGOs and regions; there is a heavy reliance on manual, paper-based systems and surveys are infrequent and not standardised. The result of these are incomplete datasets, time lags of months or years and poor interoperability.
Related stories:
AI reliance for sub – Saharan agriculture – Part 1
Poor data standardisation means that there are no common standards for: data formats, naming conventions, geographic identifiers and industry classifications. The effect of this is that same data cannot be merged or compared, there is high duplication and conflicting statistics across institutions.
Fragmented data silos results in data being owned and locked within government departments, donor-funded projects, parastatals and private companies. This status quo results in no single source of truth, AI and analytics models cannot scale and decision-making is based on partial views.
Limited metadata and taxonomy design means that data is stored without clear metadata which present challenges such as unavailability of data dictionaries, no classification schemas and no version control. Due to these challenges data becomes unusable over time, new users cannot interpret legacy datasets and there are high error rates in analysis.
Another issue is low data quality and validation controls which results in minimal validation at data entry, there is no automated error detection and poor handling of missing values. This results in inaccurate reporting and policy and investment decisions based on flawed data.
There is also skills deficit in data management translating to a shortage of data engineers, data architects, data governance professionals and training focused on data collection, not data structuring. The impact of this is that Databases are badly designed and Analytics systems fail after pilot stages.
Weak legal and governance frameworks result in inconsistent or poorly enforced data governance laws, inter-agency data-sharing rules and data ownership definitions. This unfortunately leads to fear of sharing data, duplication of effort and underutilisation of national datasets.
Donor-driven data fragmentation leads to each donor introducing its own reporting formats, indicators and platforms. This creates parallel systems, a lack of long-term data continuity and high post-project data loss.
All these challenges if unaddressed affect AI, policy, and business as AI models fail due to dirty, unlabelled, and inconsistent data, Governments cannot implement evidence-based policy, businesses struggle with market intelligence and the health, infrastructure and agriculture planning remain reactive.
The writer is Engineer Tapuwa Justice Mashangwa, GCEO Emerald Investments, CEO DataFarm, CEO Emerald Agribusiness and CEO TranslateZW. He can be contacted on +263771641714 and email: [email protected] or [email protected].
