Reprocess Billing Reports

CloudCheckr ingests your billing data to build reports on cost and credits. The billing report is a standard text file (in CSV format, many GBs in size) that is typically updated every 4-6 hours by AWS, and is stored in a user-designated S3 bucket. When CloudCheckr ingests those files, they are copied into tables in a relational database in which ad hoc queries may be run. CloudCheckr also writes numerous optimized formats that allow you to run faster queries and see summarized data.

Detailed Billing Data in CloudCheckr

CloudCheckr processes the billing data in two separate stages. One stage builds the Summary data, the other builds the Detailed data. The Summary data is used for:

  • All reports within the Cost > AWS Billing > Summary section of the application.
  • Updating saved filters created in the Cost > AWS Billing > Custom Reporting > Advanced Grouping and Advanced Filtering reports.
  • Any emailed billing reports such as the Daily Summary, Billing Alerts, and Saved Filter emails.
  • Summary and Details reports under Cost > AWS Billing > Spend Analysis.

The detailed data is used for:

  • Populating data used in the Invoice Generator.
  • Data for ad hoc (non-saved filter) queries run against the Cost > AWS Billing > Custom Reporting > Advanced Grouping and Advanced Filtering reports.

Within the AWS Billing > Partner Tools menu of CloudCheckr, you are able to make changes to the cost data within CloudCheckr.  This can be by adding custom charges, enabling payee support fees, assigning credits, etc.  When you make adjustments to your billing data, especially if done in the middle of a month, those adjustments typically require the month to be reprocessed.  If you would like to change data from historic months, those months will need to be reprocessed as well. CloudCheckr has multiple jobs that check for AWS updating the billing reports. If it finds an updated file, that file is automatically processed. For the current month, AWS is writing that file every few hours, so CloudCheckr reprocesses the file every few hours. For historical months, CloudCheckr looks at those files every day to see if any have been updated again by AWS, but if they have not been updated since the last successful ingestion, it does not reprocess the older files.

When you create a Saved Filter in the Advanced Grouping or Advanced Filtering report, CloudCheckr needs to go back through all historical months of billing data and cache the data for the query. This is done automatically each time you save a filter. You do not need to tell CloudCheckr to reprocess old months, but you will need to wait until it finishes processing the historic months to get the advantage of querying the cached data. The Historical reprocessing goes back through the most recent months first, so if you are running a query on the data that is already reprocessed, it will use the Saved Filter. If the month is not yet reprocessed, it will have to query the raw data (the detailed data described above). When you create any of the various Custom Charges in CloudCheckr, you need to reprocess the months for which you would like the custom charges to apply.  The only exception to this is if you add a Custom Charge and only want it to apply moving forward.  In this scenario you won't need to reprocess any months. 

If you add a custom charge in the middle of a month, the custom charge will not automatically be applied against the already-processed data.  It will only be applied against future data for that month.  When the month is finalized by Amazon (typically between the 3rd-5th of the following month), CloudCheckr will fully reprocess that month's data, so the custom charge will be applied against the full month before you run your final billing reports/invoices. Types of charges that you will need to reprocess historical months:

  • Account Families
  • Assigned Credits
  • Configuring a Custom Cost for your account
  • Custom Credit Memos
  • Custom Billing Charges
  • Custom Usage Rates
  • Payee Support Charges
  • Reserved Instance Mapping
  • Tag Mapping
  • Cost Allocator (Tag Splitting)

How To Set Months to Reprocess

CloudCheckr gives you the ability to choose which months you would like to have data reprocessed within the Cost > AWS Partner Tools > Reprocess Billing Report screen.You'll find a list of all historic months that billing data is available within your account, ordered from the current month to the oldest, with the following data points for each:

  • Summary State - shows whether the summary data for the month has finished processing or is currently running.
  • Summary Data Start - shows the last time the summary update began in the account.
  • Summary Data End - shows the last time the summary update completed in the account.
  • Detailed State - shows whether the detailed data for the month has finished processing or is currently running.
  • Detailed Data Start - shows the last time the detailed update began in the account.
  • Detailed Data End - shows the last time the detailed update completed in the account.

To set any month to reprocess click the Reprocess button to the right of the month.  You can set as many months as you'd like to reload. Next, if you want the reload to start immediately, click the Run Billing Update Now button. This will start the reload process in the back end. If you don't click the Run Billing Update Now, CloudCheckr will still schedule your reload to occur within a short amount of time. Once the reloading is in progress, you can refresh the Reload DBR page to see when the update begins, and again when the End time populates. The Detailed update will run once the Summary Data completes. 

Checking on a Reload Job's Progress

When a month's reload is in progress, you'll see a message of Running under the status box. You can get additional detail on the reload job progress if you click the clock icon. You'll be able to see details such as when the job started, and at what stage the job is at. For large billing months, it is not uncommon for there to be a most recently updated time in the past—this is because the update is occurring in real time and the job progress will reflect new info once the particular step has finished. 

Locking a Billing Month

After you have created your billing customizations for a specific month in List Cost, it is recommended best practice to lock your billing month so that no further changes can be made. This ensures that any subsequent reloads of the month will not change the data for the month-in-question, which could cause confusion for your end customers. To lock a month, simply click the Lock button. Only months that are closed by Amazon will show as available to be locked. Once locked, the Reprocess option will not be unavailable. You can click the Unlock button if you want to make additional changes. 

To Summarize

You will need to reload a month's data whenever you make custom changes to the billing data. The fastest way to kick off a reload is to click Reprocess for the month in question, then click Run Billing Update Now. You'll know your reload is done when both the Summary State and Detailed State say Processed and you see a recent timestamp for both processes.

How did we do?