One stage stop to know all about BW Extractors-Part2
This blog focuses on customer generated extractors behavior (EX: CO-PA and FI-SL).
Along with that ,this blog also explains on data recovery methods incase of lost delta.
Please refer "One stage stop to know all about BW Extractors-Part1" to get an idea on BW content extractors.
Note: With regards to Generic extraction, only helpful links are provided.
Controlling is broken down into following sub modules:
CO-PA:
Profitability analysis allows Management to review information with respect to the company's profit or contribution margin by business segment.
It can be obtained by the following methods:
How the data Extraction happens?
When the data is requested from SAP BW, the extractor determines which data source the data is to be read from. This depends on the
Once an Info-Package is executed, the SAP BW Staging Engine calls the CO-PA transaction data interface. CO-PA extraction program for the SAP BW uses the same replication method as the update program for CO-PA updating summarization levels. On the BW side, only data that is "at least 30 minutes old" is received .This is to secure data integrity.Because the time stamps from different application servers can be slightly different.
This retention period of 30 minutes is often described as a "security delta/Safety delta" The system only extracts data that is at least 30 Min Old.
Account-Based Analysis
For account-based CO-PA extraction, only Full Update from summarization levels is supported for releases up to and including Release PI2001.1.
In this case we can carry out delta using pseudo delta technique. Here we need to do selective full load based on some selection conditions (Fiscal period) and then we need to selectively drop the requests for the last period and reload the data that have changed.
From Release PI2001.2, the delta method can also be used.
Initialization: The initialization must be performed from a summarization level.
Delta update: Delta will be read from line items.
During the delta load controlling area, fiscal period fields should be mandatory.
Note: If the data needs to be read from a summarization level, then the level must also contain all the characteristics that are to be extracted using the Data Source (entry * in maintenance transaction KEDV). Furthermore, the summarization must have status ACTIVE.
Account based CO-PA is part of the CO module. This means the data which is posted in account based CO-PA is always in synchronize with the CO-module (CCA, OPA, PA, PS etc).
The CO tables are COEP, COBK (for line items) COSS and COSP (for the totals).
Cost-Based Analysis:
In the case of costing-based CO-PA, data can only be read from a summarization level if no characteristics of the line item are selected apart from the Record Currency (REC_WAERS) field, which is always selected.
An extraction from the segment level, that is, from the combination of the tables CE3XXXX / CE4XXXX (where XXXX stands for the operating concern), is only performed for Full Updates if no line item characteristics are selected (as with summarization levels).
Initialization: There are two possible sources for the initialization of the delta method. One is from Summarization levels (if no characteristics of the line item are selected) and the other one is from line item level.
In case of Summarization level, it will also record the time when the data was last updated / built.
If it is not possible to read data from a summarization level, data is read from line items instead.
Delta update: Data is always read from line items.
Costing Based CO-PA data is statistical data. This means that the update of CO-PA is not always equal to what is stored in the CO modules or in finance. The cost element is also not always updated and there are also more key-figures used to store info about the type of costs or revenues.
Understanding various tables(CE1/CE2/CE3/CE4) that are involved in co-pa extraction, please read BW data extraction .
CO-PA Delta Mode:
Extraction is based on Timestamp.
When data is extracted from CO-PA, a "safety delta" of half an hour is used with the initialization and the delta upload. This always ensures that only records that are already half an hour old since the start of the upload are loaded into SAP BW. Half an hour was chosen as the safety delta to overcome any time differences between the clocks on the different application servers.
Please check the below links for more information:
Profitability analysis
FI-SL:
There are two types of ledgers in the FI-SL System:
Standard Ledger: Delivered by SAP, Ex: General Ledger Accounting (FI-GL)
Special Purpose Ledgers: These will be designed as per business needs (User defined,Ex:FI-SL)
The FI-SL Data Source can supply the data both at totals record level and also at line item level
How the data extraction happens?
Prerequisite:
Since FI-SL is a generating application, the Data Source, transfer structure and assignment of the DataSource to the InfoSource must be created manually.
FI-SL line items:
Line item Data Source provides actual data at line item level.
Full and Delta mode: FI-SL line items can be extracted both in full and delta upload mode. The time stamp (TIMSTAMP field in the extract structure) is used to identify the delta load, which is supplied from the CPUDT and CPUTM fields in the line items table. It uses safety delta concept set to one hour. This means that posted line items can be loaded into BW after an hour.
Constraint:The extract structure does not contain the BALANCE field. Refer note 577644 to find out alternative ways to populate this field.
FI-SL Totals Records:
This DataSource can provide both actual and plan data at totals record level
Full update: The full update DataSource can be used to determine the balance carry forward, since the line items DataSource does not supply this.
Usually Plan data will be transferred using the totals datasource in full update mode.
Delta Update: The delta method can only be used for actual data with the selection (0VTYPE = 010). The Delta method is based on Delta queue technology. That means after initialization during updating, the relevant data is posted to the Delta queue.
Before running the delta, please check the restrictions in the below link
Delta-Special Ledger
We will go for Generic extraction:
Solution:
Make the QM status red, delete the request from all targets
Re-schedule the load this time it will prompt a window as shown below
Click on request again, it will recover the failed request
Senario2: Everyday delta was running fine but you find suddenly delta is missing for certain period (the reason may be anything),
Solution:
1. Reload data from the PSA
2. Reload data from an ODS Object or an Info Cube (in a layered
Architecture, EDW approach)
Applicable to Logistics:
Please refer "One stage stop to know all about BW Extractors-Part1" to get an idea on Logistics extraction .
Option 1 and 2 are not applicable, the only choice is to extract the data from sources system
Check this OSS notes: 691721: Restoring lost data from a delta request
Here again we have one more constraint
As explained in the above OSS, because of huge data we can't bear the downtime due to re-initialization, we have a workaround here
1. in BW,transfer the existing target contents to an external source using open hub services
2. Then selectively fill the setup tables for the missing data for the respective duration.
3. And run initialization, schedule V3 jobs to enable delta postings
(Here we have a drawback, since we are deleting setup tables and refilling them using selections, setup tables won't contain entire historical data)
Check this interesting document:
How to minimize downtime for delta initilization
Further-602260: Procedure for reconstructing data for BW
In case of ODS
You can go for repair full load(739863: Repairing data in BW)
Senario3:
Accidentally good data was deleted consequently all the data which was loaded later was deleted.(assuming no further data mart, no aggregates to avoid the complexity, if considered the solution is more dynamic and situational)
Check these links for more intricate details to handle the above situation.
Along with that ,this blog also explains on data recovery methods incase of lost delta.
Please refer "One stage stop to know all about BW Extractors-Part1" to get an idea on BW content extractors.
Note: With regards to Generic extraction, only helpful links are provided.
Application specific-customer generated extractors:
Controlling:Controlling is broken down into following sub modules:
- Cost Element Accounting
- Cost Center Accounting
- Internal Orders
- Activity-Based Costing ( ABC)
- Product Cost Controlling
- Profitability Analysis
- Profit Center Accounting
CO-PA:
Profitability analysis allows Management to review information with respect to the company's profit or contribution margin by business segment.
It can be obtained by the following methods:
- Account-Based Analysis
- Cost-Based Analysis
How the data Extraction happens?
When the data is requested from SAP BW, the extractor determines which data source the data is to be read from. This depends on the
- Update mode (full, initialization of the delta method, or delta update)
- On the definition of the DataSource (line item characteristics (apart from field REC_WAERS) or calculated key figures)
- On the available summarization levels.
Once an Info-Package is executed, the SAP BW Staging Engine calls the CO-PA transaction data interface. CO-PA extraction program for the SAP BW uses the same replication method as the update program for CO-PA updating summarization levels. On the BW side, only data that is "at least 30 minutes old" is received .This is to secure data integrity.Because the time stamps from different application servers can be slightly different.
This retention period of 30 minutes is often described as a "security delta/Safety delta" The system only extracts data that is at least 30 Min Old.
Account-Based Analysis
For account-based CO-PA extraction, only Full Update from summarization levels is supported for releases up to and including Release PI2001.1.
In this case we can carry out delta using pseudo delta technique. Here we need to do selective full load based on some selection conditions (Fiscal period) and then we need to selectively drop the requests for the last period and reload the data that have changed.
From Release PI2001.2, the delta method can also be used.
Initialization: The initialization must be performed from a summarization level.
Delta update: Delta will be read from line items.
During the delta load controlling area, fiscal period fields should be mandatory.
Note: If the data needs to be read from a summarization level, then the level must also contain all the characteristics that are to be extracted using the Data Source (entry * in maintenance transaction KEDV). Furthermore, the summarization must have status ACTIVE.
Account based CO-PA is part of the CO module. This means the data which is posted in account based CO-PA is always in synchronize with the CO-module (CCA, OPA, PA, PS etc).
The CO tables are COEP, COBK (for line items) COSS and COSP (for the totals).
Cost-Based Analysis:
In the case of costing-based CO-PA, data can only be read from a summarization level if no characteristics of the line item are selected apart from the Record Currency (REC_WAERS) field, which is always selected.
An extraction from the segment level, that is, from the combination of the tables CE3XXXX / CE4XXXX (where XXXX stands for the operating concern), is only performed for Full Updates if no line item characteristics are selected (as with summarization levels).
Initialization: There are two possible sources for the initialization of the delta method. One is from Summarization levels (if no characteristics of the line item are selected) and the other one is from line item level.
In case of Summarization level, it will also record the time when the data was last updated / built.
If it is not possible to read data from a summarization level, data is read from line items instead.
Delta update: Data is always read from line items.
Costing Based CO-PA data is statistical data. This means that the update of CO-PA is not always equal to what is stored in the CO modules or in finance. The cost element is also not always updated and there are also more key-figures used to store info about the type of costs or revenues.
Understanding various tables(CE1/CE2/CE3/CE4) that are involved in co-pa extraction, please read BW data extraction .
CO-PA Delta Mode:
Extraction is based on Timestamp.
When data is extracted from CO-PA, a "safety delta" of half an hour is used with the initialization and the delta upload. This always ensures that only records that are already half an hour old since the start of the upload are loaded into SAP BW. Half an hour was chosen as the safety delta to overcome any time differences between the clocks on the different application servers.
Please check the below links for more information:
Profitability analysis
FI-SL:
There are two types of ledgers in the FI-SL System:
Standard Ledger: Delivered by SAP, Ex: General Ledger Accounting (FI-GL)
Special Purpose Ledgers: These will be designed as per business needs (User defined,Ex:FI-SL)
The FI-SL Data Source can supply the data both at totals record level and also at line item level
How the data extraction happens?
Prerequisite:
Since FI-SL is a generating application, the Data Source, transfer structure and assignment of the DataSource to the InfoSource must be created manually.
FI-SL line items:
Line item Data Source provides actual data at line item level.
Full and Delta mode: FI-SL line items can be extracted both in full and delta upload mode. The time stamp (TIMSTAMP field in the extract structure) is used to identify the delta load, which is supplied from the CPUDT and CPUTM fields in the line items table. It uses safety delta concept set to one hour. This means that posted line items can be loaded into BW after an hour.
Constraint:The extract structure does not contain the BALANCE field. Refer note 577644 to find out alternative ways to populate this field.
FI-SL Totals Records:
This DataSource can provide both actual and plan data at totals record level
Full update: The full update DataSource can be used to determine the balance carry forward, since the line items DataSource does not supply this.
Usually Plan data will be transferred using the totals datasource in full update mode.
Delta Update: The delta method can only be used for actual data with the selection (0VTYPE = 010). The Delta method is based on Delta queue technology. That means after initialization during updating, the relevant data is posted to the Delta queue.
Before running the delta, please check the restrictions in the below link
Delta-Special Ledger
Part3: Cross application -Generic extractors
When none of the SAP- predefined extractors meeting the business demand, then the choice is to go for Generic extractionWe will go for Generic extraction:
- When Business content does not include a data source for your application.
- Business content requires additional enhancements that need data that is not supplied by SAP BW.
- The application does not features it's own generic data extraction method
- When the requirement demands to use your own programs to fill your tables in SAP Systems
Data recovery:
Scenario 1: The last run delta was failed(Not applicable to ALE based datasources)Solution:
Make the QM status red, delete the request from all targets
Re-schedule the load this time it will prompt a window as shown below
Click on request again, it will recover the failed request
Senario2: Everyday delta was running fine but you find suddenly delta is missing for certain period (the reason may be anything),
Solution:
1. Reload data from the PSA
2. Reload data from an ODS Object or an Info Cube (in a layered
Architecture, EDW approach)
Applicable to Logistics:
Please refer "One stage stop to know all about BW Extractors-Part1" to get an idea on Logistics extraction .
Option 1 and 2 are not applicable, the only choice is to extract the data from sources system
Check this OSS notes: 691721: Restoring lost data from a delta request
Here again we have one more constraint
As explained in the above OSS, because of huge data we can't bear the downtime due to re-initialization, we have a workaround here
1. in BW,transfer the existing target contents to an external source using open hub services
2. Then selectively fill the setup tables for the missing data for the respective duration.
3. And run initialization, schedule V3 jobs to enable delta postings
(Here we have a drawback, since we are deleting setup tables and refilling them using selections, setup tables won't contain entire historical data)
Check this interesting document:
How to minimize downtime for delta initilization
Further-602260: Procedure for reconstructing data for BW
In case of ODS
You can go for repair full load(739863: Repairing data in BW)
Senario3:
Accidentally good data was deleted consequently all the data which was loaded later was deleted.(assuming no further data mart, no aggregates to avoid the complexity, if considered the solution is more dynamic and situational)
Check these links for more intricate details to handle the above situation.
5) 'Save' process variant and go back, 'Change selections' is used to schedule the process chain, we will revisit this again later
6) Press 'Continue'
7) Select the InfoPackage which is used to load data in
