Wednesday 13 February 2013

The Art of Testing


Seismic processing is often described as more of an art than a science. 

I suspect this is because, in practice, the Earth is a complex place. Fundamentally, seismic processing assumes a flat earth made up of homogeneous layers – sometimes a single layer – that have constant properties.  If it were truly that simple, the entire endeavour would be completely automated.

It is the variability of seismic data that makes processing a challenge and at the heart of this is testing and applying a processing sequence.

For the new processor, testing is probably the biggest challenge.  It’s not just what to test and how to test it, but also keeping track of what you have tested, what worked well, and then applying it to all of the data. 

Experts make this look easy but that’s largely because they have developed habits over a number of years that help them to avoid making mistakes. Rest assured that most of these habits only came about because they made the mistakes in the first place!

The aim of this blog post is to help new processors develop good habits when it comes to testing, so as to avoid making the mistakes in the first place.

What kind of mistakes are we talking about?  Here are some of the classics:
  • creating a spectacular test section, then overwriting the processing flow so you have no idea how you did it
  • using the wrong (or different) input files for the tests, with different pre-processing applied so that it is not a fair comparison
  • not being able to explain at a later stage why you chose a particular approach
  • putting the test sequence into “production” processing, and getting a different result

In my experience, testing is seldom well-documented in a final processing report. Many reports don’t discuss the things that were tried which didn’t work; this can be very important information at a later date. They can also describe what tests were applied, but not what issue was trying to be addressed and why a given result was chosen.

Here are my top ten testing suggestions, to help you become a better processor:

  1. Pick your test line/area carefully
There is always a push to use a test-line that is in the main area of interest, e.g. the well location or the target area.  From a processing flow point of view you ideally want a line that has the greatest geological variation, or the worst noise contamination.  Try and balance these needs if you can; it’s better to test on more lines than be unpleasantly surprised by the results!

  1. Keep a Testing Log
Create a log of the tests you have run; this can be a test file, a Word Document, even a hand written journal, but it needs to be done.  Treat each test as an experiment, and log the objective, method, results and conclusion.  Put the science back in to the art of testing.
Example of a basic Testing Log

  1. Do not re-use processing flows
Create a new processing flow for each test, and preserve the original.  The flow – and any log file – is a vital part of the “audit trail” that you will need if you have to back-track.  Recording the name of each testing flow in your testing log will save a huge amount of time and effort if things go wrong.

  1. Number your tests
Give each test a unique number.  I tend to use double digits 00, 01, 02 – I’ve never been on a project yet where I needed 99 different tests!  Record this number in the Testing Log.  Tests are not just processing flows, but can include things like a mute you have designed (and tested the application of) or a velocity function you picked.

  1. Number each test panel
I extend the numbering system so that each sub-test has a unique number as well.  The “00” sub-test is always “nothing applied”, and then all the other parameter variations have a separate number.  So for example, if my test 04 was “deconvolution before stack” my test numbers might be:

04_00: nothing applied
04_01: 160ms operator, 24ms gap
04_02: 200ms operator, 24ms gap
04_03: 240ms operator, 24ms gap
etc.

This allows you to extend the testing sequence as much or as little as you need.

  1. Use these numbers in your processing flows
Make the start of each testing flow the unique test number; the use of “double digits” will help to keep these in order when you list them.  Some systems let you run “panel tests” with repeat copies of data, but that’s not always applicable for every test. Some of your tests might end up with a test and sub-test number like this:  08_02_radon200.job

Because you have a Testing Log, you won’t need to try and think up (or remember) long and convoluted processing flow names that describe the test or the parameters. 

  1. Use the same numbering for input and output datasets
If you extend this numbering technique to the output datasets from each test, then you can easily trace an output file back to where it was created.  The datasets have a link to both the test number, and the processing flow, which is simple and easy to remember.  What was the output from 08_02_radon200.job?  Why it was 08_02_radon200.cdps!
This greatly reduces the chance of a labelling error or mistake when reviewing results and making decisions. 

  1. Number your metadata files
Metadata is “data about data”. In the context of testing, it’s all the “other” files and information you create, like mutes, velocities, horizons, filters, design gates, and picks.  Number these too! If 08_02_radon200.job uses the mute you picked as test number 05, then the mute file should start with 05 in the name. Applying the wrong “metadata” is one of the main causes of incorrect processing. 

  1. Create a flow diagram showing how tests are related
Simple test numbering really makes this easy; but it is useful as a check to make sure that parameters can actually be compared, especially when looking from one technique to another.

Example flow diagram for demultiple processing routes

A flow diagram is useful for spotting omissions or errors, as well as explaining what you did, and why.

  1. Extend tests until you find the ‘end members’

To be sure that you are not looking at a “local minima” when testing – especially with things like demultiple, filters, gain control and so on – it is a good idea to push the testing of the parameters so that the results look terrible.  These “end member” tests are quickly discarded, but show you have covered the full spectrum of possibilities.  Test panels that show artefacts, or over processed data help to assure that the overall sequence will be effective.




By: Guy Maslen


3 comments:

  1. Yes unfortunately tests are often filed in the rubbish bin every Friday.
    Some clients require detailed processing tests to be submitted, not just a final report.
    Although these are typically only a comparison with some parameter being X, Y or Z.
    Occasionally it might be algorithm A versus B.

    ReplyDelete
  2. Thank you for the detailed information. I think it will be very helpful to start with to new processors like me.

    Also, I have a question about your post. Is not the same thing the processing flow and testing? from the post it appears to be there is a difference but I could not understand it well.

    ReplyDelete
    Replies
    1. Hi -

      Generally speaking a processing flow comprises one or more processing steps.

      Each processing step will require one or more "modules" in the software package. For example, deconvolution is usually a single "module", but if you wanted to apply deconvolution in the tau-P domain you might need to use forward and reverse transforms as well and so on.

      There tends to be three types of processing flow we use:

      - testing flows
      - production flows
      - QC/QA flows

      A "testing flow" is usually set up to test a single processing step, often on a subset of the data to be processed. In some software or situations you can have a single flow that tests multiple parameters (by creating repeat copies of the data, but in other cases you might need a different flow for each parameter variation.

      The key thing is that you only test one processing step, and usually you are exploring parameter variations for that processing stage.

      A "production flow" typically includes multiple processing steps, each one of which has been tested independently. This is largely about saving disc space, reducing the number of data files and minimizing processing time - input/output takes time, especially on large projects.

      Typically we run tests on the subset of data until we can create a production flow, and then run all the data in the survey through that production flow sequence.

      We might also break up production flows when there is a need for QC, or manual analysis (velocities, first breaks, horizons, mutes etc)

      The final type of flow (QA/QC) is used to check data or investigate problems; this is typically based around display data or trace headers in some way.

      Hope this helps.

      Guy

      Delete

Total Pageviews