Validation Methodology

Data Validation

Data validation occurs at three stages and includes both automatic and manual controls.

1. Data input. The spreadsheet is designed with input controls and provides guidance to the user. Input controls and guidance include:
   • Password protection for non-editable cells and embedded equations
   • Colour differentiation of cells
   • Messages and error alerts associated with certain data elements.
2. Pre-upload to SBS. Submissions were manually reviewed for the following:
   • Presence of company name; where it was missing the company name was inserted for identification purposes
   • 'Reasonableness' of submitted data based on knowledge of reporting companies and crosschecks between data fields
   • Explanatory comments.

After the above review, follow-up was conducted on anomalies identified and explanatory comments. A hard copy of the submission, together with the accompanying e-mail (where appropriate), was printed and kept in CAPP's records.

3. Upload to SBS The SBS automated many of the core processes associated with data validation based on defined rules. The SBS checked if defined causal relationships were fulfilled, and where a rule was violated, the system 'flagged' the results. The following six processes were supported by the system:

• Collect Data: process determined on upload to the SBS whether the spreadsheet was the current version of the spreadsheet supplied by CAPP. The analysis resulted in the spreadsheet being uploaded or an exception report produced to explain why the spreadsheet was rejected.
• Validate Data: process validated company-specific data against a series of defined rules. Process outputs were 'flagged' reports and validated data sets.

'Flagging' of questionable entries required manual examination of the submission to determine the appropriate follow-up. A 'flag' did not mean that the data was unacceptable or did not qualify to move on to subsequent SBS processes, but acted as notice of potential errors. When potential errors and omissions were identified, the reporting company was contacted and steps taken to resolve the concerns.

Occasionally, other CAPP personnel or industry contacts with knowledge of the reporting company's operations were consulted for their views on the reported data.

The rules used in the Validate Data process focused on data completeness and reasonableness. The function of each rule is described below:

• Gross error: identified if production was reported, but direct and indirect greenhouse gas emissions were not
• Unit error changes: identified reporting unit inconsistencies for data elements, e.g., annual totals instead of daily totals
• Percent value changes: compared percent change between current year submissions against the previous year's data, and analyzed significant changes
• Eligibility criteria: determined whether data sets will be included or excluded from normalized indicator calculations based on specified criteria.

• Aggregate Data: process occurred after data sets have been manually validated through application of the Percent Value Changes test. At this point data were considered appropriate for aggregation and benchmarking purposes. Process output tallies data for all reporting companies and produces industry aggregate totals.
• Calculate Indicators: process occurred only when a company-specific data set met the eligibility criteria for inclusion in the normalized indicator calculation, e.g., a data set where all values included in the numerator and denominator of a calculated indicator were submitted and validated. This relationship often resulted in a difference between the number of companies included in the industry aggregate count and the normalized indicator count. Process output calculated normalized indicators.
• Generate Reports: process occurred at any point in the data management procedure. The process inputs were report requirements and/or query criteria. The process outputs were various data reports for internal use only.
• Amend Data: process resulted in an updated submission and a record of changes made whenever a submission amendment was required. Presently, all amendments are made in the company-specific Spreadsheet and then re-uploaded to the SBS with accompanying explanatory notes.

Report Preparation

The final output of the benchmarking process was the Benchmarking Report. Report preparation involved three main processes:

1. Examination of the aggregate absolute values and normalized performance indicators to identify trends and to explain anomalies;
2. Interpretation of data in the context of company-specific issues and comparison to other relevant published and publicly available data sources; and
3. Review of the benchmarking report or specific portions of the report by relevant professionals, CAPP managers, the Executive team and the Stewardship Committee to provide perspectives based on current work being done by industry and multi-stakeholder groups.