Understanding the Different Levels of MES Validation: From ERP to Process Control

Why MES Validation Can’t Be Viewed in Isolation?

Manufacturing Execution Systems (MES) are the backbone of today's manufacturing facilities, seamlessly bridging the gap between the enterprise planning systems and production floor. MES sits within a wider Manufacturing Technology (MTS) solution that connects Level 4 ERP systems that run business processes with Level 2 control systems that perform real-time manufacturing activities.  

However, many organizations still consider MES, as a single entity, and only validate the application. This short-sighted perspective ignores an important fact: MES is not a stand-alone entity. It is integrated with a highly connected digital ecosystem, through the continuous exchange of data with upstream and downstream systems and is coupled to the ISA-95 hierarchy.  

Validation is often done in "islands", which can have serious repercussions. However, in regulated sectors, like pharmaceuticals, where traceability and audit requirements are paramount, fragmented validation strategies can result in compliance gaps. Meanwhile, lack of integrity in interdependent systems can lead to issues such as batch records, regulatory reporting, and more - all with the potential of risk that can only be found during an audit or operational issue.  This is what is driving leading companies to embrace a more holistic approach. They're not just validating MES, they're validating the interactions between the data in ERP, MES and control systems.  

This greater vision is the basis of how validation is considered at Arbour Group. Arbour's end-to-end manufacturing system validation (ESV) provides a way to maintain compliance, ensure data integrity, and to maintain system performance throughout the entire technology stack - not just within individual systems.

Understanding the ISA-95 Framework (The Foundation of System-Level Validation)

The ISA-95 standard is an industry specification for the architecture of industrial automation systems and is the only way to validate MES- and all of manufacturing in general- effectively. ISA-95 offers a clear model of how the various layers of technology work together in a manufacturing organization which is a key element in the development of system-level validation strategies.

ISA-95, at a high-level, breaks the systems into hierarchical levels, according to their function:

Level 4 - Enterprise Systems (ERP):

Organizes business planning, logistics and enterprise-wide data including production scheduling, inventory and financials. 

Level 3 - Manufacturing Operations (MES):

Responsible for the execution of production such as work order, electronic batch records, quality management and performance tracking.

Level 2 - Process Control Systems (SCADA, PLC, DCS):

Monitors and controls physical equipment and processes on the shop floor, in real time. 

They are not separate layers but are tightly integrated and exchange data continuously. For instance, ERP (Level 4) generates production orders, which are then carried out and monitored in MES (Level 3), and with the help of accurate, real-time data from the control system (Level 2). Production data flows in reverse through the stack, to be reported, released and then used for decisions.  The interconnected structure is the reason why ISA-95 is significant for validation.

It is a very important principle to remember:

“Validation should be done on the flow of data and process-not solely the boundaries of the individual data systems.” 

If this is not the case, organizations may end up validating one layer at a time, missing any interfaces, dependencies, or data transformations that span them. Common challenges such as compliance problems, data inconsistencies, and operational risks often occur at these integration points.  

With the guidance of ISA-95, companies can implement a validation strategy that matches the way their systems are utilized- providing a way to validate each individual system and validating the interactions between them.

Understanding Validation Requirements Across Level 4, Level 3, and Level 2 Systems 

Level 4 Validation: ERP Systems in Manufacturing

ERP systems (Level 4) are the highest-level systems used to control the overall processes in the enterprise which are involved in production. This includes production planning, inventory management, batch record integration and regulatory reporting. ERP systems don't directly manage manufacturing processes, but they play a key role in feeding the data into systems downstream. 

When it comes to validation, the main concern is on data integrity and governance. This includes ALCOA+ principles (A- Accurate, L- Legible, C- Contemporaneous, O- Original and A- Accurate, and (+)- Complete, Consistent, Enduring, and Available), user access, user security, and confirmation of integration points with MES for seamless data transfer. As ERP is usually the system of record, any inaccuracies here ripple throughout the manufacturing ecosystem. 

Typical problems are poor or poorly maintained master data that impact MES execution and reporting, as well as poor or missing audit trails that can lead to problems during inspection for compliance.

Key Insight:

ERP validation is more about the accuracy of enterprise information and its traceability and governance across the enterprise than real-time process control. 

Level 3 Validation: MES as the Operational Core

MES (Level 3) is the middle ground in manufacturing processes, which provides a link between enterprise-level instructions and issues on the shop floor and transform them into executable workflows. This includes electronic batch records (EBR), work instructions, production execution, quality monitoring and more, and is one of the most critical systems in regulated environments. 

The level of validation focuses much on the accuracy and compliance of workflow. This includes compliance with 21CFR Part 11 and Annex 11 criteria, handling exceptions and deviations correctly, and depicting the configuration of the system in the correct way to validate the process (but not over-customizing).

One of the main areas of potential integration issues is related to MES and ERP systems, which can be connected together. Common issues include bad business logic between ERP and MES, mismatching data mapping, and batch records or reporting inaccuracies as a result. 

Key Insight:

MES validation is the layer on the manufacturing stack that links business systems and shop floor execution, making it the most critical layer.

Level 2 Validation: Process Control Systems (SCADA, PLC, DCS)

Process control systems like SCADA, PLCs and DCS are used on Level 2 to directly control and monitor manufacturing equipment. These systems are used for automation, real-time data acquisition from sensors and control of production parameters, playing a crucial role in production activity.

At this level, validation is geared toward ensuring reliable and accurate operation at the source. This involves control logic programming, alarm management and ensuring the data is accurate as it is generated. This information is essential to MES and even minor inaccuracies can lead to greater compliance or quality issues. 

Possible risks are equipment failures or signal failures (e.g. incorrect signal received) resulting in inaccuracies in the MES records and lack of traceability from the activities on the machine to activities on higher level systems making it difficult to conduct traceability audits or investigations.

Key Insight:

Validation at Level 2, real time, ensures that data fed into MES is accurate, reliable and trustworthy for all upstream systems.

Closing the Integration Gap with a Risk-Based, End-to-End Validation Strategy

One of the most common mistakes in manufacturing validation is to validate systems individually. Validation per module may be done on a single ERP system, a single MES system, or a single control system, but the linkages between them are not necessarily validated or tested. The zone in between where systems share key information is where the greatest threats often lie.

Missed are the data exchanges and technical connections that make production a seamless process. These include APIs, middleware, and systems integrations that carry instructions and information between the ISA-95 levels. These connections, if not validated, can lead to unforeseen risks in otherwise secure systems.

The impact in the real world could be significant. Batch record discrepancies, data integrity problems, or compliance issues, none of which would be identified as a problem with system functionality, but rather a failure in the system communicating to one another, can arise for a company during a regulatory audit. Critical integration points need to be validated are:

• ERP → MES: transfer of work orders and master data
• MES → control systems: execution instructions and parameters
• Control systems → MES: real-time production and process data
• MES → ERP: production reporting and release data

Many go so far as to specify a strategy for validation with the same result, that cross-system interaction effects are not covered.  In response, more and more organizations are using a risk-based validation approach, based on GAMP 5 principles. This way, validation will target the most important aspects. The ISA-95 stack has varying degree of risks:

• Level 4 (ERP): business and data governance risk
• Level 3 (MES): compliance and operational risk
• Level 2 (Control Systems): safety and process risk

For this reason, some systems and functions do not need a complete level of validation. For instance, priority should be given to critical data interfaces and integrations over less critical data such as standard reports. The idea is to put effort into that which could be lost.

Effective organizations take this as the basis and adopt an end-to-end validation strategy that truly mirrors how systems are used. This includes:

1. Using ISA-95 as a reference, design a system architecture for mapping.
2. Determining the most important data flows between systems.
3. Specifying the responsibility for validating across IT and OT teams.
4. Validating the integrations and interfaces along with core systems.
5. Ensuring the ongoing validation lifecycle, with evolving systems.

Best practices further reinforce this strategy, including implementing traceability matrices across systems, cross functional teams, and a clear integration testing process. 

The integration gap must be addressed by changing the mindset from validation of systems to validation of the entire manufacturing ecosystem as one integrated system.

Arbour Group’s Approach: Ecosystem Validation That Follows the Data 

Validation does not end at MES- and that is where many organizations fail. At Arbour Group, the solution isn't just about the individual system; it's about the entire manufacturing ecosystem validation - knowing that ERP, MES and control systems will work not only separately but together and in compliance.

The expertise of cross-system validation is what makes Arbour Group special. The method focuses not just on application-level checks, but also on integration, data flow and system interaction, which can be the source of compliance risks. In addition to its in-depth knowledge of the regulated environment, Arbour validates each layer of the ISA-95 stack, and each connection between the layers.

This end-to-end capability covers:

• Validation of ERP data to guarantee data accuracy and governance enterprise wide.
• MES validation: ensuring compliance and reliability in manufacturing processes.
• The need for automation and validation of control systems for accuracy at source. Need for automation and validation of control systems to ensure accuracy at the source.
• Data flow and validation across all systems to ensure data integrity.

Finally, validation needs to be done after the data, not the system silos. The interoperability of systems is as important as their interoperability in terms of the interactions they have with one another. 

The point to remember here is that to achieve a truly compliant and reliable manufacturing environment, all layers of the ERP system, MES system, and all control systems need to be synchronized and to be able to support accurate and audit-traceable operations all the way through the technology stack as well as all the connections between the layers.

Key Insight:

“Validating your MES without validating its integrations is like testing a bridge without checking its supports.”