iGIS平台算法测试情况一览
=================================
说明
------------------------
1. 当开发者提交算法源码并完成自测时，可以申请进行交叉验证，开发人员和测试人员不可为同一人
#. 当前状态包括 [ 提交 | 自测通过 | 测试通过 | 测试未通过 | 返工提交 ]
#. 进行交叉测试，包括返工的后续测试过程中，需保留测试验收通过或者未通过的测试报告，即当前一览表中仅保留最新的测试情况，历史测试记录需详细记录在备注中。
#. 测试报告应按要求的详细规范填写 模板链接及详细要求
BUG
----------------------
* Oozie版本库老旧，Oozie的sharelib中hive相关库的版本为1.2.1,存在一些关键词敏感的问题，例如table、name等保留字在sql语句中无法使用，被识别为保留字
* 20190730，ML_WF多起始点运行未考虑Start节点的父亲依赖，会报空指针异常，已定位待处理
* 20190731，特征离散，不保留原列信息，列顺序被修改，被处理的列，默认置后，如果重新组织列顺序，则浪费资源；使用select(StartCol,Cols)，将列顺序固定
* 20190801，保留列和附加列实现有所不同，部分算子实现了通过T/F来决定是否保留列，实际上附加列的功能实现更符合用户预期，需要进行算子核查和修改，待处理
* 20190801，Hive表存稀疏向量格式如下，{"type":0,"size":3,"indices":[1],"values":[1.0]}；后端节点无法复用数据，未有解决方案
历史问题修订
---------------------------------
20190723 问题1：
^^^^^^^^^^^^^^^^^^^^^^^
使用spark算子处理之后，展示结果是否需要保留原表数据，或者部分保留原表数据。
.. code::
例：col1,col2,label------>newCol1,newCol2（舍弃原表，展示结果全部为新表,例读数据表，PCA）
_____col1,col2,label------>col1,col2,label,newCol1,newCol2,newLabel（完全保留原表）
_____col1,col2,label------>newCol1,newCol2,Label（算子仅处理col1,col2，部分保留原表）
解决方法：前端增加单选框，增加标志位，来完成第二第三种的区别，20190725问题解决
20190823 问题2：
^^^^^^^^^^^^^^^^^^^^^^^
Hive版本问题，在一个session中如果使用两句Create table语句，会报unable to move hdfs resource的问题，无法创建Hive表，必须要重启spark master节点
解决方法：使用kudu
测试情况一览
----------------------------
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| 模块名称 | 算法名称 | 开发人员 | 开发完成时间 | 状态 | 算法源码链接 | 测试人员 | 测试完成时间 | 测试结果 | 测试报告 | 备注 |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| **数据预处理** | | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| **采样与过滤** | 随机采样 | 石冶 | 2020-03-03 | 提交 | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 加权采样 | 石冶 | 2020-03-03 | 自测通过 | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 过滤与映射 | 石冶 | 2020-03-03 | **测试通过** | | 易晓博 | 2020-03-04 | 测试通过 | | 如果一次性测试通过可不填 |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 分层采样 | 石冶 | 2020-03-03 | 测试未通过 | | 易晓博 | 2020-03-04 | 测试未通过 | | 2020-03-04，第一次测试未通过，需返工 |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| **数据合并** | JOIN | 石冶 | 2020-03-03 | 返工提交 | | | | | | 2020-03-04，第一次测试未通过，需返工，测试人员：易晓博；测试报告： |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 合并列 | 石冶 | 2020-03-03 | 自测通过 | | | | | | 2020-03-04，第一次测试未通过，需返工，测试人员：易晓博；测试报告： |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 合并行 | 石冶 | 2020-03-03 | 测试通过 | | 易晓博 | 2020-03-04 | 测试通过 | | 2020-03-04，第一次测试未通过，需返工，测试人员：易晓博；测试报告： |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 类型转换 | 石冶 | 2020-03-03 | 测试未通过 | | 易晓博 | 2020-03-04 | 测试未通过 | | 2020-03-04，第一次测试未通过，需返工，测试人员：易晓博；测试报告：
2020-03-05，第二次测试未通过，需返工 |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 增加序列号 | 石冶 | 2020-03-03 | 返工提交 | | | | | | 2020-03-04，第一次测试未通过，需返工，测试人员：易晓博；测试报告：
2020-03-05，第二次测试未通过，需返工，测试人员：易晓博；测试报告： |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 拆分 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 缺失值填充 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 归一化 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 标准化 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | KVtoTable | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | TabletoKV | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| **特征工程** | | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| **特征变换** | 奇异值分解 SVD | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 主成分分析 PCA | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 特征尺度变换 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 特征异常平滑 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | Zscore平滑 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 百分位平滑 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 阈值平滑 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 异常检测模块 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 箱型图 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 特征离散 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| **特征重要性评估** | 随机森林-特征重要性 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | GBDT-特征重要性 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| **特征选择** | 特征编码 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | one-hot编码 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| **统计分析** | | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 数据视图 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 协方差 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 经验概率密度图 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 全表统计 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 卡方拟合性检验 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 卡方独立性检验 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 箱线图 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 散点图 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 相关系数矩阵 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 双样本T检验 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 单样本T检验 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 正态检验 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 洛伦兹曲线 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 百分位 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 皮尔森系数 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 直方图 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 离散值特征分析 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| **机器学习** | | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 预测 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| **二分类** | 逻辑回归二分类 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | GBDT二分类 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | AdaBoost二分类 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 线性支持向量机 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| **多分类** | K近邻 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | AdaBoost多分类 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 随机森林 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 朴素贝叶斯 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 逻辑回归多分类 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| **聚类** | K均值 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | DBSCAN | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| **回归** | GBDT回归 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 线性回归 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | AdaBoost回归 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| **关联推荐** | 协同过滤 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| **模型评估** | 二分类模型评估 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 回归模型评估 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 聚类模型评估 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 混淆矩阵 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| | 多分类评估 | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+
| **GIS算法** | | | | | | | | | | |
+--------------------+---------------------+----------+--------------+--------------+--------------+----------+--------------+------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------+