FateWithShellmiaoComment/Fate/doc/tutorial/fate_test_tutorial.md

FATE Test Tutorial

A collection of useful tools to running FATE tests and :file_folder:examples.

quick start

1. install

   pip install -e python/fate_test
   

   1. edit default fate_test_config.yaml
      

   # edit priority config file with system default editor
   # filling some field according to comments
   fate_test config edit
   

2. configure FATE-Pipeline and FATE-Flow Commandline server setting

   # configure FATE-Pipeline server setting
   pipeline init --port 9380 --ip 127.0.0.1
   # configure FATE-Flow Commandline server setting
   flow init --port 9380 --ip 127.0.0.1
   

   1. run some fate_test suite
      

   fate_test suite -i <path contains *testsuite.json>
   

3. run some fate_test benchmark

   fate_test benchmark-quality -i <path contains *benchmark.json>
   

   1. useful logs or exception will be saved to logs dir with namespace shown in last step

   command types

   • suite: used for running testsuites, collection of FATE jobs
     

   fate_test suite -i <path contains *testsuite.json>
   

   • data: used for upload, delete, and generate dataset

   • upload/delete data command:

     fate_test data [upload|delete] -i <path1 contains *testsuite.json | *benchmark.json>
   

   • upload example data of min_test/all_examples command:

     fate_test data upload -t min_test
     fate_test data upload -t all_examples
   

   • generate data command:
     

     fate_test data generate -i <path1 contains *testsuite.json | *benchmark.json>
   

   • benchmark-quality: used for comparing modeling quality between FATE and other machine learning systems, as specified in benchmark job configuration
     

   fate_test bq -i <path contains *benchmark.json>
   

   • benchmark-performance: used for checking FATE algorithm performance; user should first generate and upload data before running performance testsuite

   fate_test data generate -i <path contains *benchmark.json> -ng 10000 -fg 10 -fh 10 -m 1.0 --upload-data
   fate_test performance -i <path contains *benchmark.json> --skip-data
   

   • op-test: used for testing FATE's mpc protocol.

   fate_test op-test paillier
   fate_test op-test spdz
   

   • convert: used for converting pipeline to dsl&conf.

   fate_test convert pipeline-to-dsl -i ${your pipeline file}
   fate_test convert pipeline-testsuite-to-dsl-testsuite -i {your pipeline testsuite folder}
   

   Usage