首頁 探索 說明
登入
Shellmiao
/
FateWithShellmiaoComment
1
0
複刻 0
Files 問題管理 0 合併請求 0 Wiki
分支: master
分支列表 標籤列表
FateWithShel...
/
Fate
/
doc
/
api
/
fate_test.md

fate_test.md 30 KB
永久連結 文件歷史 原始文件

FATE Test

A collection of useful tools to running FATE's test.

Testsuite

Testsuite is used for running a collection of jobs in sequence. Data used for jobs could be uploaded before jobs are submitted and, optionally, be cleaned after jobs finish. This tool is useful for FATE's release test.

command options

fate_test suite --help

  1. include:

    fate_test suite -i <path1 contains *testsuite.json>

    will run testsuites in path1

    1. exclude: 
    fate_test suite -i <path1 contains *testsuite.json> -e <path2 to exclude> -e <path3 to exclude> ...

    will run testsuites in path1 but not in path2 and path3

  2. glob:

    fate_test suite -i <path1 contains *testsuite.json> -g "hetero*"

    will run testsuites in sub directory start with hetero of path1

    1. replace: 
    fate_test suite -i <path1 contains *testsuite.json> -r '{"maxIter": 5}'

    will find all key-value pair with key "maxIter" in data conf or conf or dsl and replace the value with 5

  3. timeout:

    fate_test suite -i <path1 contains *testsuite.json> -m 3600

    will run testsuites in path1 and timeout when job does not finish within 3600s; if tasks need more time, use a larger threshold

    1. task-cores 
    fate_test suite -i <path1 contains *testsuite.json> -p 4

    will run testsuites in path1 with EGGROLL "task-cores" set to 4; only effective for DSL conf

  4. update-job-parameters

    fate_test suite -i <path1 contains *testsuite.json> -uj {}

    will run testsuites in path1 with respective job parameters set to provided values

    1. update-component-parameters 
    fate_test suite -i <path1 contains *testsuite.json> -uc {}

    will run testsuites in path1 with respective component parameters set to provided values

  5. skip-dsl-jobs:

    fate_test suite -i <path1 contains *testsuite.json> --skip-dsl-jobs

    will run testsuites in path1 but skip all tasks in testsuites. It's would be useful when only pipeline tasks needed.

    1. skip-pipeline-jobs: 
    fate_test suite -i <path1 contains *testsuite.json> --skip-pipeline-jobs

    will run testsuites in path1 but skip all pipeline tasks in testsuites. It's would be useful when only dsl tasks needed.

  6. skip-data:

    fate_test suite -i <path1 contains *testsuite.json> --skip-data

    will run testsuites in path1 without uploading data specified in testsuite.json.

    1. data only: 
    fate_test suite -i <path1 contains *testsuite.json> --data-only

    will only upload data specified in testsuite.json without running jobs

  7. disable-clean-data:

    fate_test suite -i <path1 contains *testsuite.json> --disable-clean-data

    will run testsuites in path1 without removing data from storage after tasks finish

    1. enable-clean-data: 
    fate_test suite -i <path1 contains *testsuite.json> --enable-clean-data

    will remove data from storage after finishing running testsuites

  8. yes:

    fate_test suite -i <path1 contains *testsuite.json> --yes

    will run testsuites in path1 directly, skipping double check

    testsuite configuration

    Configuration of jobs should be specified in a testsuite whose file name ends with "*testsuite.json". For testsuite examples, please refer dsl examples and pipeline examples.

    A testsuite includes the following elements:

    • data: list of local data to be uploaded before running FATE jobs

      • file: path to original data file to be uploaded, should be relative to testsuite or FATE installation path
      • head: whether file includes header
      • partition: number of partition for data storage
      • table_name: table name in storage
      • namespace: table namespace in storage
      • role: which role to upload the data, as specified in fate_test.config; naming format is: "{role_type}_{role_index}", index starts at 0 
    "data": [
    {
        "file": "examples/data/motor_hetero_host.csv",
        "head": 1,
        "partition": 8,
        "table_name": "motor_hetero_host",
        "namespace": "experiment",
        "role": "host_0"
    }
]

    • tasks: includes arbitrary number of jobs with paths to corresponding dsl and conf files

      • job: name of job to be run, must be unique within each group list

        • conf: path to conf filw, should be relative to testsuite
        • dsl: path to dsl file, should be relative to testsuite 

      "tasks": {
     "cv": {
         "conf": "hetero_lr_cv_conf.json",
         "dsl": "hetero_lr_cv_dsl.json"
     },
     "early-stop": {
         "conf": "hetero_lr_early_stop_conf.json",
         "dsl": "hetero_lr_early_stop_dsl.json"
     }
}

      • pipeline_tasks: includes arbitrary number of pipeline jobs with paths to corresponding python script

      • job: name of job to be run, must be unique within each group list

        • script: path to pipeline script, should be relative to testsuite 

      "pipeline_tasks": {
     "cv": {
         "script": "pipeline-hetero-lr-cv.py"
     },
     "normal": {
         "script": "pipeline-hetero-lr-early-stop.py"
     }
}
      • model_deps(deps): model to be used for prediction task 
      "tasks": {
     "cv": {
         "conf": "hetero_lr_cv_conf.json",
         "dsl": "hetero_lr_cv_dsl.json"
     },
     "normal": {
         "conf": "hetero_lr_normal_conf.json",
         "dsl": "hetero_lr_normal_dsl.json"
     },
     "predict": {
     "conf": "hetero-lr-normal-predict-conf.json",
     "dsl": "hetero-lr-normal-predict-dsl.json",
     "deps": "normal"
     }
}
      • data_deps: component output data from previous task to be used as designated input for current task(only used for dsl tasks) 
      "tasks": {
"column-expand": {
    "conf": "./test_column_expand_job_conf.json",
    "dsl": "./test_column_expand_job_dsl.json"
},
"column-expand-train": {
    "conf": "./test_column_expand_train_job_conf.json",
    "dsl": "./test_column_expand_train_job_dsl.json",
    "data_deps": {
        "column-expand": {
            "guest_0": {
                "reader_0": "column_expand_0"
            }
        }
    }
}

      }

      • cache_deps: component output cache from previous task to be used as designated cache loader input for current task(only used for intersect dsl tasks) 
      "tasks": {
"intersect-cache": {
    "conf": "./test_intersect_cache_job_conf.json",
    "dsl": "./test_intersect_cache_job_dsl.json"
},
"intersect-cache-loader": {
    "conf": "./test_intersect_cache_loader_job_conf.json",
    "dsl": "./test_intersect_cache_loader_job_dsl.json",
    "cache_deps": {
        "intersect-cache"
    }
}

      }

      • model_loader_deps: component output model from previous task to be used as designated model loader input for current task(only used for dsl tasks) 
      "tasks": {
"hetero-lr": {
    "conf": "./test_hetero_lr_job_conf.json",
    "dsl": "./test_hetero_lr_job_dsl.json"
},
"hetero-lr-model-loader": {
    "conf": "./test_hetero_lr_model_loader_job_conf.json",
    "dsl": "./test_hetero_lr_model_loader_job_dsl.json",
    "model_loader_deps": {
        "hetero-lr"
    }
}

      }

Benchmark Quality

Benchmark-quality is used for comparing modeling quality between FATE and other machine learning systems. Benchmark produces a metrics comparison summary for each benchmark job group.

Benchmark can also compare metrics of different models from the same script/PipeLine job. Please refer to the script writing guide below for instructions.

fate_test benchmark-quality -i examples/benchmark_quality/hetero_linear_regression

|----------------------------------------------------------------------|
|                             Data Summary                             |
|-------+--------------------------------------------------------------|
|  Data |                         Information                          |
|-------+--------------------------------------------------------------|
| train | {'guest': 'motor_hetero_guest', 'host': 'motor_hetero_host'} |
|  test | {'guest': 'motor_hetero_guest', 'host': 'motor_hetero_host'} |
|-------+--------------------------------------------------------------|


|-------------------------------------------------------------------------------------------------------------------------------------|
|                                                           Metrics Summary                                                           |
|-------------------------------------------+-------------------------+--------------------+---------------------+--------------------|
|                 Model Name                | root_mean_squared_error |      r2_score      |  mean_squared_error | explained_variance |
|-------------------------------------------+-------------------------+--------------------+---------------------+--------------------|
| local-hetero_linear_regression-regression |    0.312552080517407    | 0.9040310440206087 | 0.09768880303575968 | 0.9040312584426697 |
|  FATE-hetero_linear_regression-regression |    0.3139977881119483   | 0.9031411831961411 | 0.09859461093919598 | 0.903146386539082  |
|-------------------------------------------+-------------------------+--------------------+---------------------+--------------------|
|-------------------------------------|
|            Match Results            |
|-------------------------+-----------|
|          Metric         | All Match |
| root_mean_squared_error |    True   |
|         r2_score        |    True   |
|    mean_squared_error   |    True   |
|    explained_variance   |    True   |
|-------------------------+-----------|


|-------------------------------------------------------------------------------------|
|                             FATE Script Metrics Summary                             |
|--------------------+---------------------+--------------------+---------------------|
| Script Model Name  |         min         |        max         |         mean        |
|--------------------+---------------------+--------------------+---------------------|
|  linr_train-FATE   | -1.5305666678748353 | 1.4968292506353484 | 0.03948016870496807 |
| linr_validate-FATE | -1.5305666678748353 | 1.4968292506353484 | 0.03948016870496807 |
|--------------------+---------------------+--------------------+---------------------|
|---------------------------------------|
|   FATE Script Metrics Match Results   |
|----------------+----------------------|
|     Metric     |      All Match       |
|----------------+----------------------|
|      min       |         True         |
|      max       |         True         |
|      mean      |         True         |
|----------------+----------------------|

command options

use the following command to show help message

fate_test benchmark-quality --help

  1. include:

    fate_test benchmark-quality -i <path1 contains *benchmark.json>

    will run benchmark testsuites in path1

    1. exclude: 
    fate_test benchmark-quality -i <path1 contains *benchmark.json> -e <path2 to exclude> -e <path3 to exclude> ...

    will run benchmark testsuites in path1 but not in path2 and path3

  2. glob:

    fate_test benchmark-quality -i <path1 contains *benchmark.json> -g "hetero*"

    will run benchmark testsuites in sub directory start with hetero of path1

    1. tol: 
    fate_test benchmark-quality -i <path1 contains *benchmark.json> -t 1e-3

    will run benchmark testsuites in path1 with absolute tolerance of difference between metrics set to 0.001. If absolute difference between metrics is smaller than tol, then metrics are considered almost equal. Check benchmark testsuite writing guide on setting alternative tolerance.

  3. storage-tag

    fate_test performance -i <path1 contains *benchmark.json> -s test

    will run benchmark testsuites in path1 with performance stored under provided tag for future comparison; note that FATE-Test always records the most recent run for each tag; if the same tag is used more than once, only metrics from the latest job is kept

    1. history-tag 


    fate_test performance -i <path1 contains *benchmark.json> -v test1 -v test2

    will run benchmark testsuites in path1 with performance compared to history jobs under provided tag(s)

  4. skip-data:

    fate_test benchmark-quality -i <path1 contains *benchmark.json> --skip-data

    will run benchmark testsuites in path1 without uploading data specified in benchmark.json.

    1. disable-clean-data: 
    fate_test suite -i <path1 contains *benchmark.json> --disable-clean-data

    will run benchmark testsuites in path1 without removing data from storage after tasks finish

  5. enable-clean-data:

    fate_test suite -i <path1 contains *benchmark.json> --enable-clean-data

    will remove data from storage after finishing running benchmark testsuites

    1. yes: bash fate_test benchmark-quality -i <path1 contains *benchmark.json> --yes will run benchmark testsuites in path1 directly, skipping double check

benchmark job configuration

Configuration of jobs should be specified in a benchmark testsuite whose file name ends with "*benchmark.json". For benchmark testsuite example, please refer here.

A benchmark testsuite includes the following elements:

  • data: list of local data to be uploaded before running FATE jobs

    • file: path to original data file to be uploaded, should be relative to testsuite or FATE installation path
    • head: whether file includes header
    • partition: number of partition for data storage
    • table_name: table name in storage
    • namespace: table namespace in storage
    • role: which role to upload the data, as specified in fate_test.config; naming format is: "{role_type}_{role_index}", index starts at 0 

    "data": [
    {
        "file": "examples/data/motor_hetero_host.csv",
        "head": 1,
        "partition": 8,
        "table_name": "motor_hetero_host",
        "namespace": "experiment",
        "role": "host_0"
    }
]

    • job group: each group includes arbitrary number of jobs with paths to corresponding script and configuration

      • job: name of job to be run, must be unique within each group list

        • script: path to testing script, should be relative to testsuite
        • conf: path to job configuration file for script, should be relative to testsuite 

      "local": {
     "script": "./local-linr.py",
     "conf": "./linr_config.yaml"
}
      • compare_setting: additional setting for quality metrics comparison, currently only takes relative_tol

      If metrics a and b satisfy abs(a-b) <= max(relative_tol * max(abs(a), abs(b)), absolute_tol) (from math module), they are considered almost equal. In the below example, metrics from "local" and "FATE" jobs are considered almost equal if their relative difference is smaller than 0.05 * max(abs(local_metric), abs(pipeline_metric).

      "linear_regression-regression": {
"local": {
    "script": "./local-linr.py",
    "conf": "./linr_config.yaml"
},
"FATE": {
    "script": "./fate-linr.py",
    "conf": "./linr_config.yaml"
},
"compare_setting": {
    "relative_tol": 0.01
}
}

testing script

All job scripts need to have Main function as an entry point for executing jobs; scripts should return two dictionaries: first with data information key-value pairs: {data_type}: {data_name_dictionary}; the second contains {metric_name}: {metric_value} key-value pairs for metric comparison.

By default, the final data summary shows the output from the job named "FATE"; if no such job exists, data information returned by the first job is shown. For clear presentation, we suggest that user follow this general guideline for data set naming. In the case of multi-host task, consider numbering host as such:

{'guest': 'default_credit_homo_guest',
 'host_1': 'default_credit_homo_host_1',
 'host_2': 'default_credit_homo_host_2'}

Returned quality metrics of the same key are to be compared. Note that only real-value metrics can be compared.

To compare metrics of different models from the same script, metrics of each model need to be wrapped into dictionary in the same format as the general metric output above.

In the returned dictionary of script, use reserved key script_metrics to indicate the collection of metrics to be compared.

  • FATE script: Main should have three inputs:
    • config: job configuration, JobConfig object loaded from "fate_test_config.yaml"
    • param: job parameter setting, dictionary loaded from "conf" file specified in benchmark testsuite
    • namespace: namespace suffix, user-given namespace or generated timestamp string when using namespace-mangling
  • non-FATE script: Main should have one or two inputs:
    • param: job parameter setting, dictionary loaded from "conf" file specified in benchmark testsuite
    • (optional) config: job configuration, JobConfig object loaded from "fate_test_config.yaml"

Note that Main in FATE & non-FATE scripts can also be set to take zero input argument.

Benchmark Performance

Performance sub-command is used to test efficiency of designated FATE jobs.

Example tests may be found here.

command options

fate_test performance --help

  1. job-type:

    fate_test performance -t intersect

    will run testsuites from intersect sub-directory (set in config) in the default performance directory; note that only one of task and include is needed

    1. include: 
    fate_test performance -i <path1 contains *testsuite.json>; note that only one of ``task`` and ``include`` needs to be specified.

    will run testsuites in path1. Note that only one of task and include needs to be specified; when both are given, path from include takes priority.

  2. replace:

    fate_test performance -i <path1 contains *testsuite.json> -r '{"maxIter": 5}'

    will find all key-value pair with key "maxIter" in data conf or conf or dsl and replace the value with 5

    1. timeout: 
    fate_test performance -i <path1 contains *testsuite.json> -m 3600

    will run testsuites in path1 and timeout when job does not finish within 3600s; if tasks need more time, use a larger threshold

  3. max-iter:

    fate_test performance -i <path1 contains *testsuite.json> -e 5

    will run testsuites in path1 with all values to key "max_iter" set to 5

    1. max-depth 
    fate_test performance -i <path1 contains *testsuite.json> -d 4

    will run testsuites in path1 with all values to key "max_depth" set to 4

  4. num-trees

    fate_test performance -i <path1 contains *testsuite.json> -nt 5

    will run testsuites in path1 with all values to key "num_trees" set to 5

    1. task-cores 
    fate_test performance -i <path1 contains *testsuite.json> -p 4

    will run testsuites in path1 with EGGROLL "task_cores" set to 4

  5. update-job-parameters

    fate_test performance -i <path1 contains *testsuite.json> -uj {}

    will run testsuites in path1 with respective job parameters set to provided values

    1. update-component-parameters 
    fate_test performance -i <path1 contains *testsuite.json> -uc {}

    will run testsuites in path1 with respective component parameters set to provided values

  6. storage-tag

    fate_test performance -i <path1 contains *testsuite.json> -s test

    will run testsuites in path1 with performance time stored under provided tag for future comparison; note that FATE-Test always records the most recent run for each tag; if the same tag is used more than once, only performance from the latest job is kept

    1. history-tag 
    fate_test performance -i <path1 contains *testsuite.json> -v test1 -v test2

    will run testsuites in path1 with performance time compared to history jobs under provided tag(s)

  7. skip-data:

    fate_test performance -i <path1 contains *testsuite.json> --skip-data

    will run testsuites in path1 without uploading data specified in testsuite.json.

    1. disable-clean-data: 
    fate_test performance -i <path1 contains *testsuite.json> --disable-clean-data

    will run testsuites in path1 without removing data from storage after tasks finish

  8. yes:

    fate_test performance -i <path1 contains *testsuite.json> --yes

    will run testsuites in path1 directly, skipping double check

    data

    Data sub-command is used for upload, delete, and generate dataset.

    data command options

    fate_test data --help

  9. include:

    fate_test data [upload|delete] -i <path1 contains *testsuite.json | *benchmark.json>

    will upload/delete dataset in testsuites in path1

    1. exclude: 
    fate_test data [upload|delete] -i <path1 contains *testsuite.json | *benchmark.json> -e <path2 to exclude> -e <path3 to exclude> ...

    will upload/delete dataset in testsuites in path1 but not in path2 and path3

  10. glob:

    fate_test data [upload|delete] -i <path1 contains \*testsuite.json | \*benchmark.json> -g "hetero*"

    will upload/delete dataset in testsuites in sub directory start with hetero of path1

    1. upload example data:
    fate_test data upload -t [min_test|all_examples]

    will upload dataset for min_test or all examples of fate. Once command is executed successfully, you are expected to see the following feedback which showing the table information for you:

    [2020-06-12 14:19:39]uploading @examples/data/breast_hetero_guest.csv >> experiment.breast_hetero_guest
[2020-06-12 14:19:39]upload done @examples/data/breast_hetero_guest.csv >> experiment.breast_hetero_guest, job_id=2020061214193960279930
[2020-06-12 14:19:42]2020061214193960279930 success, elapse: 0:00:02
[2020-06-12 14:19:42] check_data_out {'data': {'count': 569, 'namespace': 'experiment', 'partition': 16, 'table_name': 'breast_hetero_guest'}, 'retcode': 0, 'retmsg': 'success'}

    Note: uploading configurations are min_test_config and all_examples,

      user can add more data by modifying them or check out the example data's name and namespace.
    1. download mnist data:
    fate_test data download -t mnist -o ${mnist_data_dir}

    -t: if not specified, default is "mnist" -o: directory of download data, default is "examples/data"

generate command options

fate_test data --help

  1. include:

    fate_test data generate -i <path1 contains *testsuite.json | *benchmark.json>

    will generate dataset in testsuites in path1; note that only one of type and include is needed

    1. host-data-type: 
    fate_test suite -i <path1 contains *testsuite.json | *benchmark.json> -ht {tag-value | dense | tag }

    will generate dataset in testsuites path1 where host data are of selected format

  2. sparsity:

    fate_test suite -i <path1 contains *testsuite.json | *benchmark.json> -s 0.2

    will generate dataset in testsuites in path1 with sparsity at 0.1; useful for tag-formatted data

    1. encryption-type: 
    fate_test data generate -i <path1 contains *testsuite.json | *benchmark.json> -p {sha256 | md5}

    will generate dataset in testsuites in path1 with hash id using SHA256 method

  3. match-rate:

    fate_test data generate -i <path1 contains *testsuite.json | *benchmark.json> -m 1.0

    will generate dataset in testsuites in path1 where generated host and guest data have intersection rate of 1.0

    1. guest-data-size: 
    fate_test data generate -i <path1 contains *testsuite.json | *benchmark.json> -ng 10000

    will generate dataset in testsuites path1 where guest data each have 10000 entries

  4. host-data-size:

    fate_test data generate -i <path1 contains *testsuite.json | *benchmark.json> -nh 10000

    will generate dataset in testsuites path1 where host data have 10000 entries

    1. guest-feature-num: 
    fate_test data generate -i <path1 contains *testsuite.json | *benchmark.json> -fg 20

    will generate dataset in testsuites path1 where guest data have 20 features

  5. host-feature-num:

    fate_test data generate -i <path1 contains *testsuite.json | *benchmark.json> -fh 200

    will generate dataset in testsuites path1 where host data have 200 features

    1. output-path: 
    fate_test data generate -i <path1 contains *testsuite.json | *benchmark.json> -o <path2>

    will generate dataset in testsuites path1 and write file to path2

  6. force:

    fate_test data generate -i <path1 contains *testsuite.json | *benchmark.json> -o <path2> --force

    will generate dataset in testsuites path1 and write file to path2; will overwrite existing file(s) if designated file name found under path2

    1. split-host: 
    fate_test data generate -i <path1 contains *testsuite.json | *benchmark.json> -nh 10000 --split-host

    will generate dataset in testsuites path1; 10000 entries will be divided equally among all host data sets

  7. upload-data

    fate_test data generate  -i <path1 contains *testsuite.json | *benchmark.json> --upload-data

    will generate dataset in testsuites path1 and upload generated data for all parties to FATE

    1. remove-data 
    fate_test data generate -i <path1 contains *testsuite.json | *benchmark.json> --upload-data --remove-data

    (effective with upload-data set to True) will delete generated data after generate and upload dataset in testsuites path1

  8. use-local-data

    fate_test data generate -i <path1 contains *testsuite.json | *benchmark.json> --upload-data  --use-local-data

    (effective with upload-data set to True) will generate dataset in testsuites path1 and upload data from local server; use this option if flow and data storage are deployed to the same server

    MPC Operation Test

    op-test sub-command is used to test efficiency and accuracy of secure multiparty computation protocols like Paillier and SPDZ.

    1. paillier: 
    fate_test op-test paillier

    will generate performance and accuracy table for Paillier Protocol, including encryption，decryption, addition and addition and scalar multiplication.

  9. spdz

    fate_test op-test spdz

    will generate performance and accuracy table for SPDZ Protocol, including encryption，decryption, addition and addition and scalar multiplication.

    Convert tools

    convert sub-command is used to convert pipeline to dsl.

    1. pipeline-to-dsl
    fate_test convert pipeline-to-dsl -i ${your pipeline file}

    will generate dsl & conf under the same folder of your pipeline file, before executing the command, make sure the running data of pipeline is already uploaded.

  10. pipeline-testsuite-to-dsl-testsuite

    fate_test convert pipeline-testsuite-to-dsl-testsuite -i {your pipeline testsuite folder}

    will transform pipeline testsuite to dsl testsuite under the {your pipeline testsuite folder}/dsl_testsuite