FengZiYjun
95a72f06b9
* AutoPadder will not pad when dtype is None
* add ignore_type in DataSet.apply
6 years ago
FengZiYjun
3e01b14249
add ignore_type in DataSet.add_field
6 years ago
FengZiYjun
864c2238f8
添加FieldArray对list of np.array的支持
6 years ago
FengZiYjun
b93ca9bb30
* FieldArray添加对list of np.array的支持
* 添加测试:FieldArray的初始化
6 years ago
FengZiYjun
e4f997d52a
refactor type system in FieldArray:
* 重构dtype的检测代码,在FieldArray的初始化和append两处,达到更好的代码复用
* 类型检测的责任完全落在FieldArray,DataSet与之配合
测试:
* 整理dtype相关的测试代码
* 给所有tutorial添加测试
其他:
* 完善一个完整的Conll dataset loader
* 升级POS tag model训练脚本
6 years ago
yh
3e33a23042
修改Padder的测试用例
6 years ago
yh
8091a734ee
1. 将pad的功能从FieldArray中剥离出来,使用Padder完成各种padding操作。
2. FieldArray默认使用AutoPadder, AutoPadder的行为与之前不使用padder是一致的的
3. 为了解决二维padding的问题,引入了EngChar2dPadder用于对character进行padding
4. 增加一份padding的tutorial。
6 years ago
FengZiYjun
6129a31c1d
* fix tests
* clean up unused codes
7 years ago
FengZiYjun
f26f11608b
* 更新教程,放在在./tutorial
* remove unused codes in metrics.py
* add tests for DataSet
* add tests for FieldArray
* add tests for metrics.py
* fix predictor, add tests for predictor
* fix bucket sampler, add tests for bucket sampler
7 years ago
FengZiYjun
27833d06ae
FieldArray only check type when is_input or is_target is set.
7 years ago
FengZiYjun
661780b975
Improve FieldArray. Support nested list and a list of np.array
7 years ago
FengZiYjun
6839bb91cc
Add auto type detection/conversion in FieldArray
* In init, detect content type to be Python int, float, or str.
* In append(), check type consistence.
* In init & append(), int will be cast into float if they occur together.
* Map Python type into numpy dtype
* Raise error if type detection fails.
7 years ago
FengZiYjun
3d66975091
* refine code comments
* refine code style
* set up unit tests for Batch, DataSet, FieldArray
* remove a lot of out-of-date unit tests, to get testing passed
7 years ago
FengZiYjun
837bef47dc
* add unit tests for instance, vocabulary
* remove and fix other unit tests
* add more code comments
7 years ago