如何确认数据迁移过程的数据完整性问题

恨我不成钢

        Introduction简介Due to the dynamic nature oftechnological changes and dataintegrity requirements companies are faced withthe challenge of frequentlymigrating data from legacy systems. ..       
                        Introduction简介


Due to the dynamic nature oftechnological changes and dataintegrity requirements companies are faced withthe challenge of frequentlymigrating data from legacy systems.
基于不断变化的技术新变革和数据完整性新需求，公司常常面对着如何从旧有系统系统中迁移数据的挑战。


Data migration can be very challengingand can create dataintegrity issues if not executed with an adequate strategy.
如果没有一个合适的指导策略，数据迁移可能会存在风险，并且引起数据完整性问题。


One of the biggest challenges is thatcompanies fail to understandthe magnitude and impact of migrating data fromlegacy systems whilemaintaining data integrity.
其中一个最大最普遍的问题就是在评估数据迁移行动中的数据完整性风险时，受影响公司常常没有正确评估从旧有系统迁移数据的体量和其造成的影响。


Prior to the execution of projects thatrequire data migrationcompanies need to understand the risk impact of thelegacy data that requiresmigration.
在实施数据迁移项目前，公司应该充分理解评估从旧有系统迁移数据引起的关联影响。


Based on the risk impact of the legacydata companies need todefine a data migration strategy that enables compliancewith data integrity regulatoryrequirements.
基于旧有数据迁移的风险评估，公司需要定义数据迁移策略来确保行动全程的数据完整性合规性。


This article will discuss dataintegrity strategies during thedata migration of legacy systems.
本文将主要讨论旧有系统数据迁移过程中数据完整性策略：


Data Migration Challenges
数据迁移过程中的风险挑战


The drivers for migrating data fromlegacy systems may bedifferent from one company to the other but they typicallyinclude thefollowing:
驱使公司进行旧有系统数据迁移的原因可能各不相同，但典型理由如下:


·        Obsoletetechnology
·        技术更新换代
·        Newregulatory requirements that legacy systems are unable to meet
·        新法律法规导致旧有系统无法满足合规性需求
·        Vendor nolonger supporting the technology
·        供应商不再提供技术支持（例如： 美东时间4月8日夜晚12点之后，微软就停止支持Windows XP）
·        Vendor goesout of business
·        供应商破产
·        Vendor istaken over by another business
·        供应商业务被第三方收购
·        Businessdecision to replace legacy system
·        高层商业决定，决定旧有系统淘汰


There are several challenges that needto be well understood andmanaged to reduce or eliminate any data integritycompliance risk.
以下有关数据迁移的事项，需要特别注意以减少或者消除数据完整性合规风险


Some of these challenges include thefollowing:
包括如下:


·        Clearunderstanding of which data must be migrated
·        清晰了解哪些数据需要被迁移 （例如master file， metadata，audit trail record, etc.）
·        Clearunderstanding of the risk impact of the data
·        清晰了解这些数据的风险影响（例如Impact in business/function/process）
·        Ensuringdata integrity during the migration process
·        在整个迁移过程中确保数据完整性（例如 test in test environment withlive data firstly, parallel run, go-liveperiod ）
·        Defining anadequate data migration strategy
·        定义一个合适的数据迁移策略
·        Defining anadequate data migration verification strategy
·        定义一个合适的数据迁移验证策略（Data availability test, Data Integritytest）


Another challenge is developing a clearunderstanding of the typeof data that will be migrated including the followingattributes:
另一方面，我们需要从以下方面对所迁移数据的数据类型进行评估：


·        Legacy dataformat
·        旧有数据的格式（Read-only or editable ? Open data orEncrypted Data?）
·        New systemdata format
·        新系统要求数据的格式
·        Legacy datasize
·        旧有数据的大小容量
·        Legacy dataregulatory impact
·        旧有数据在法规上影响（GxP， Privacy Information，海关，SoX美国业务，GDPR欧盟业务）
·        Legacy dataretention period  
·        旧有数据的保存年限（药品的有效期， GDP的定义）


One of the biggest issues with datamigration is that companiesnormally don’t understand the data that needs to bemigrated.  Thiscreates the challenge that very often companies end upmigrating data that doesnot need to be migrated to the new environment. Thisissue is driven by thelack of understanding of the record retentionrequirements of the legacy data.Data migration should only include data that iscurrently under recordretention requirements that must be migrated to the newsystem. Migrating onlyrelevant data enables efficiency and cost control duringdata migrationprojects.
数据迁移问题里最大的一个挑战就是公司实际上一般不理解（哪些）数据需要迁移。所以公司最后经常把旧系统的所有数据迁移到新系统来完成项目（举例：而这样操作的问题是：并不是所有数据都需要迁入新系统；迁入过多无用数据导致 -1.迁移后数据完整性测试任务量增加， 2. 新系统因为数据过多导致运行效率降低和备份时间过长）。这个（简单粗暴迁移所有数据）问题的产生源自对旧有数据的数据保存要求没正确的认知。数据迁移应当仅限于数据保存期的数据方才被要求迁移入新系统。这样仅迁移相关数据的做法有助于帮助数据迁移项目实现高效率和合理成本控制。


The risk impact of the legacy data needto be very well understoodprior to data migration . Data that has direct impacton Critical ProcessParameters and Critical Quality Attributes is consideredcritical data that hasa high-risk impact. The risk impact of the legacy dataisa critical input intothe data migration strategy and planning.
在实施数据迁移之前，旧有数据的风险应该被充分理解评估。直接影响CPP（关键工艺参数）和CQA（关键质量属性）的数据将被定义为高风险。旧有系统的风险评估报告将作为一个重要参考引入数据迁移的策略和计划的制定。（举例：在准备 Data Migration Plan之前，应完成Blue Prints ofProcess Flow, High RiskField Assessment according to CPP and CQA.）


Maintaining data integrity during datamigration projects iscritical from a compliance and business perspective. Dataintegrity is acritical regulatory requirement that needs to be part of the datamigrationstrategy.  The data migration strategy needs to be documentedalong withdeliverables required to execute the data migration. The datamigrationstrategy needs to ensure that the data will not be altered or lostduring thedata migration activities. The data migration strategy need toinclude alertsand notification of errors and failures during the data migrationprocess.
从合规及商业角度来看，在整个数据迁移项目中维持数据完整性都是十分重要的。数据完整性是合规性要求的重要一环，需要在数据迁移策略中得以体现。数据迁移战略需要与执行数据迁移所需的可交付成果一起记录。数据迁移策略需要确保在数据迁移活动期间数据不会被改变或丢失。（举例：为防止潜在的data loss，在新系统go alive后，试运行三个月之前，旧有系统可以并行运行，将来也可以冻结使用而非直接删除）数据迁移策略需要在数据迁移过程中包含警报和错误和失败通知功能。


To ensure that the goals of the datamigration strategy areproperly executed companies need to develop a solid datamigration verificationprocess. The data migration verification activities areintended to demonstrateand provide documented evidence that data integrity wasmaintained during thedata migration process.
为确保数据迁移策略的目标得到正确执行，公司需要设计一个可靠的数据迁移验证流程。数据迁移验证活动旨在展示数据迁移过程中保持数据完整性的文件证据。（举例：完整的证明文件链Data Migration Plan，Impact and RiskAssessment， Re-Qualification test ProtocolandReport，etc）


Data formats are a critical attributethat needs to be very wellunderstood prior to data migration and included as aninput in the strategy.Compatibility issues related to data format are a commonproblem oftenoverlooked by companies prior to data migration.  
在数据迁移之前，数据的格式问题也是一个需要提前考虑的问题。在很多公司的数据迁移项目，兼容性问题常常被忽视。（举例：如果单纯的GMP Manufacturing – QA、生产部和QC 作为Process Owner 可以发起数据迁移，但是新系统可行性论证最好有system owner IT，工程部和 technicalspecialist from vendor 参与）


The size of the legacy data it isanother critical attribute thatneeds to be well understood prior to datamigration. The size of the legacydata can be impacted by any data conversionthat need to occur during the migrationprocess. When migrating data to a cloudenvironment companies need to clearlyunderstand the legacy data sizerequirements and future disk and memory spaceneeded to manage the data duringits lifecycle.
旧有数据的大小是数据迁移之前需要充分理解的另一个关键属性。旧有数据的大小可能会受到迁移过程中需要发生的任何数据转换的影响。同时将数据迁移到云环境时，公司需要清楚地了解传统数据大小要求以及在其生命周期中管理数据所需的未来磁盘和内存空间。（简单来说：新系统的设计不能简单考虑目前，需要考虑未来5-10年的data growth。同时基于商业数据敏感及客户数据私隐的要求，cloud service在医药企业并不非常普及，医药企业并不倾向于委托第三方存储数据，ERP等软件的运营也由企业自己承担，但是作为IT技术先行者，银行业和保险业已经逐渐普及Cloud Service，甚至是取代ERP的SaaS - Softwareas a Service）有时被作为“即需即用软件”【好处在于避免了频繁系统升级对客户带来的Re-validation 工作，Service供应商中央一次升级完成】）。


In summary, one of the biggestchallenges with data migrationprojects is the lack of adequate understanding ofthis task, includingplanning, strategy and understanding of the critical dataattributes that needto be well managed and understood.
总之，数据迁移项目面临的最大挑战之一是缺乏对此任务的充分理解，包括需要妥善管理和理解的关键数据属性的规划，策略和理解。


The next section of this article willprovide strategies and ideasabout managing and controlling the data integrityrisk associated with datamigration projects.
本文的下一部分将提供有关管理和控制与数据迁移项目相关的数据完整性风险的策略和理念。

Data Migration Strategies &Solutions
数据迁移策略与方案


The migration strategy and planning canbe initiated oncecompanies develop an understanding of the data that needs tobe migrated, therisk and regulatory impact.
当公司充分了解需要迁移的数据，风险和监管影响，就可以启动迁移策略和规划的制定。


During the strategy and planningcompanies need to evaluate theiroptions prior to data migration which includesthe following:
在策略和规划期间，公司需要在数据迁移实施之前评估他们的选择，其中包括以下内容


·        Archiving归档
·        Keepinglegacy data read-only将旧有数据转为“只读性”
·        Hybridapproach纸质和电子并行
·        Fullmigration整体迁移


Data archiving is the process of movingdata that is no longeractively used to a separate storage location for longterm retention. Beforeconsidering archiving, companies need to consider thefollowing factors:
数据归档是将不再主动使用的数据移动到单独的存储位置进行长期保留的过程。在考虑归档之前，公司需要考虑以下因素：


·        How oftenthe company need to access the data
·        公司访问（旧有）数据的频率有多高（因为Archived Data 需要re-install 才能被访问，太高访问频率则不建议归档）
·        Criticalityof the data
·        数据的重要性
·        High amountof data manipulation required to move to the newsolution
·        迁移到新系统所需要的大量数据操作（因为归档的数据越多，则迁移的数据越少，数据迁移的实施与验证越容易）
·        Projectbudget constraints
·        项目预算限制
·        Regulatoryimpact
·        法规要求（参考Data Retention Period in Company Policyand Regulatory Requirement）
·        Data mustbe retrievable and accessible during retention period
·        数据保留期间，数据必需可以被访问（因为Archived Data 需要re-install 才能被访问，这个re-stall的动作和archiving data的数据完整性必需被验证）


Keeping the data read-only alsorequires consideration of thefollowing factors:
保持（迁移）数据只读性主要基于以下考虑：


·        Costassociated with maintenance and licenses
·        （旧系统）维持和授权费用（因为冻结旧有系统，保持数据只读性之后，maintenanceand licenses cost 大大减少）
·        Limitingaccess to data
·        限制数据的访问
·        No datachanges
·        防止数据篡改（如果冻结退休旧有系统，只读性数据可以保护这部分数据的数据完整性）
·        Data recordretention requirements
·        数据保留的要求


Archiving and making data read-only areoptions that should beconsider as part of the strategy and planning phaseduring data migrationprojects.
在数据迁移项目中，将数据归档和数据转换为只读应作为战略和计划阶段的一部分。


Part of the data migration strategy andplanning requiresdocumented deliverables that are intended to define anddocument the datamigration strategy and related verification.
数据迁移策略和规划的一部分需要记录的可交付成果，旨在定义和记录数据迁移策略和相关验证。


For data migration projects companiesneed to create and executethe following deliverables:
可交付成果的举例如下：


·        Data riskand impact assessment数据风险及影响评估
·        DataMigration Plan数据迁移计划
·        DataMigration Protocol数据迁移方案
·        DataMigration Summary Report数据迁移总结报告


The data risk impact assessment is acritical activity that needsto be documented very early prior to any datamigration activities. The datarisk impact assessment is intended to provide therisk impact and level of thedata that needs to be migrated. The data riskimpact assessment is a key inputto the overall migration strategy and plan.
数据风险及影响评估是一项重要的活动，需要在任何数据迁移活动之前尽早实施。数据风险影响评估旨在提供需要迁移的数据的风险影响和级别。数据风险影响评估是整体迁移策略和计划的重要内容。


Once the data risk and impactassessment is completed then theData Migration Plan. The Data Migration Plan isintended to define and documentthe overall migration strategy and deliverables.
一旦数据风险和影响评估完成，那么数据迁移计划。数据迁移计划旨在定义和记录整个迁移策略和可交付成果。


The Data Migration Plan should includethe following information:
数据迁移计划应包含以下信息：


·        Purpose目的
·        Scope范围
·        Roles andResponsibilities角色与责任
·        MigrationStrategy迁移策略
·        Deliverables可交付文件
·        DataMigration Verification Strategy数据迁移验证策略
·        Data RiskImpact Assessment数据风险及影响评估
·        IdentifyData to be Migrated将被迁移数据
·        MigrationTools迁移工具（Technical Partby IT or Vendor）
·        SamplingStrategy取样策略（Technical Partby IT or Vendor）
·        AcceptanceCriteria验收标准
·        DeviationHandling偏差处理


The Data Migration Plan is a keydeliverable of the project thatmust be approved prior to initiating anymigration activities. The DataMigration Plan requires Quality review andapproval. The plan needs to definethe sampling strategy which should be drivenby the risk impact of the data.100% sampling is not feasible and adequatetherefore standards such as ANSI/AQLshould be used to define the risk basedsampling strategy. The data migrationverification strategy need to provideevidence that the migration wassuccessfully completed and data integrity wasmaintained. The data migrationverification strategy should define any tools thatwill be used for themigration activities. Tool or techniques that can be usedfor data migrationverification include source vs target data integrity checksthat can beperformed using the following tools:
数据迁移计划是该项目的关键交付项目，必须在启动任何迁移活动之前获得批准。数据迁移计划要求质量审核和批准。该计划应基于数据的风险影响确定抽样策略。 100％抽样是不可行和充分的，因此应使用ANSI / AQL等标准来定义基于风险的抽样策略。（释义：ANSI：美国国家标准化组织，是一个核准多种行业标准的组织。SQL：结构化查询语言，是与关系型数据库进行通信的标准语言）数据迁移验证策略需要提供证据证明迁移已成功完成并保持了数据完整性。数据迁移验证策略应定义将用于迁移活动的任何工具。可以用于数据迁移验证的工具或技术包括可以使用以下工具执行的源数据完整性检查和目标数据完整性检查：（迁移中和迁移后数据完整性验证的软件工具）

·        Cryptographichash function加密哈希函数
·        CheckSum总和检查码


Cryptographic hash function is amathematical function that takesan input of any size and returns as an outputan alpha numeric digest of afixed size. Any alterations to the inputs willdrastically change the digest.Cryptographic hash functions can be used tocompute a digest of a data set atthe source and then compute a digest at thetarget.
加密哈希函数是一种数学函数，它接受任意大小的输入并将固定大小的字母数字摘要作为输出返回。对输入的任何更改都会彻底改变数据摘要。（运用这个进行加密，确保数据迁移过程中，数据没有丢失或者被篡改， Checksum 也是相同目的，不同技术实现手段）加密散列函数可用于计算源数据集的摘要，然后计算目标处的摘要。


The Data Migration Protocol is acritical deliverable that isintended to provide documented evidence thatmigration activities weresuccessfully executed and that they meet theestablished acceptance criteria.The Data Migration Protocol is an executabledocument that need to include howdeviations and failures will be managed duringthe execution.
数据迁移方案是一项重要的交付项目，旨在提供书面证据证明迁移活动已成功执行并且符合既定的接受标准。数据迁移方案是一个可执行文件，需要包括在执行过程中如何管理偏差和失败。


The Data Migration Protocol shouldinclude the followinginformation:
数据迁移方案应包含以下信息：


·        Purpose目的
·        Scope范围
·        Roles &Responsibilities角色与职责
·        DataSampling Approach Strategy 数据抽样策略
·        MigrationVerification Tools迁移验证工具
·        MigrationVerification Strategy 迁移验证策略
·        Deviations偏差
·        AcceptanceCriteria验收标准


Once the Data Migration Protocols isexecuted a Data MigrationSummary Report need to be created. The Data Migration SummaryReport isintended to summarize the results and deviation found during theexecution ofthe protocol.
当执行数据迁移方案被执行完成，就需要创建数据迁移总结报告。数据迁移总结报告旨在总结执行方案期间发现的结果和偏差。


The Data Migration Summary Reportshould include the followinginformation:
数据迁移总结报告应包含以下信息：


·        Purpose目的
·        Scope范围
·        Roles &Responsibilities角色与职责
·        MigrationVerification Results迁移验证结果
·        MigrationVerification Summary迁移验证总结
·        DeviationSummary偏差总结
·        AcceptanceCriteria Summary验收总结


These deliverables and relatedactivities provide a structured andconsistent approach to manage and controldata migration projects. Thesedeliverables are also intended to provideobjective documented evidence thatdata integrity is maintained during datamigration projects.
这些可交付成果和相关活动为管理和控制数据迁移项目提供了结构化且一致的方法。这些可交付成果还旨在提供客观的书面证据，用来证明数据迁移项目期间数据完整性得到维护。

Summary总结


Data migrations require adequatestrategic planning to reduce thebusiness and compliance risk. The DataMigration Plan is a critical deliverablethat needs to be created to documentthe migration strategy.
数据迁移需要进行充分的战略规划以降低业务和合规风险。数据迁移计划（DMP）是需要创建的关键交付物，以记录迁移策略。


The data risk impact must be assessedand documented and it is aninput into the migration strategy and plan.
（迁移）数据风险及影响必需经过评估和文件记录，它将被用于数据迁移策略和计划之中。


In summary, data migration is acritical activity that must bewell managed and controlled to maintain theintegrity of the data that need tobe migrated.
总而言之，数据迁移是一个极为重要的行为，需要妥善管理和控制来保证迁移数据的数据完整性


翻译及中文解读：


胡大伟，现供职于香港幸福制药 IT GxP 系统专家。主要负责自动化生产系统搭建，IT项目实施，数据完整性风险以及GxP, FDA,EU PIC/S迎检培训。曾参与 SAP（GSK Asia葛兰史素克亚太），CERP（TCS 塔塔信息咨询公司），WMS（自动物流货仓管理系统香港澳美制药），MES（制造企业生产过程执行管理系统香港幸福制药）多个项目
With practical experience, working knowledge andinternational exposure in IT System Implementation, IT Risk Compliance and Management,Information Security Management, Data Migration, Data Integrity, Regular Audit,GxP,, PIC/S areas. An expert on computerized system such as ERP, MES, WMS, EMS,LIMS.       

洁521

占领沙发
这是翻译法规么？还是个人理解

baiyi1495

学习了     

诗的远方

谢谢分享     

syhorchid

谢谢分享

kakashi713

洁521 发表于 2018-7-18 11:40
占领沙发
这是翻译法规么？还是个人理解

同问啊 这么高大尚的内容

小王yoo

看不太明白