Define a use case and a baseline
NIST highlights the importance of evaluating industrial AI in the context of its system and users. For a project, we propose defining a task, a comparison baseline and an evaluation criterion. In a hypothetical carbonization example, a model might flag temperature curves for review. A specialist should investigate whether the difference comes from the process, the raw material or a measurement problem.
Prepare the data before the model
We recommend checking timestamps, units, quality and instrumentation changes. The team needs to understand records of normal and abnormal conditions without assuming that every alert represents a real failure. A pilot can separate development and evaluation periods rather than measuring results only against examples already used. Recording assumptions helps distinguish consistent performance from a coincidence within the available data set and supports a more useful review.
Maintain review and operational limits
AI outputs can be wrong and need a review process suited to their use. We recommend displaying the data associated with a recommendation and recording the user's decision. Analytical models do not replace safety protections or interlocks. Process changes may alter performance, so the project should plan monitoring, criteria for suspending recommendations and a return to the operation's approved procedure when required.
What to assess for your project
- Choose a question and a comparable baseline.
- Assess data quality and context.
- Measure errors in a defined pilot.
- Define human review and limits of use.
COM3
How COM3 can help
COM3 can help assess opportunities for AI in industrial processes, organize data collection and integrate analysis into SCADA and dashboards. The scope can start with a decision support pilot, including documented evaluation, participation from the operations team and clear criteria for deciding whether to continue implementation.
Topics in this article
Further reading and references
Technical sources consulted for the fundamentals of this content. Project recommendations are an editorial synthesis by COM3.