Merge pull request #33 from jkkummerfeld/data-development

Data development

Author: Jonathan Kummerfeld

LICENSE.md

@@ -7,4 +7,4 @@ Files                          | License
 `systems/baseline-template`    | [Apache License 2.0](./systems/baseline-template/LICENSE.txt)
 `systems/sequence-to-sequence` | [Apache License 2.0](./systems/sequence-to-sequence/LICENSE.txt)
 `data/advising*`               | [CC-BY-4.0](./data/advising-LICENSE.txt)
-
+`data/wikisql*`                | [BSD](./data/wikisql-LICENSE.txt)

README.md

@@ -17,6 +17,7 @@ We have separate files describing the [datasets](./data/), [systems](./systems/)
 
 Version | Description
 ------- | -------------
+3       | Data fixes and addition of data from Spider and WikiSQL
 2       | Data with fixes for variables incorrectly defined in questions
 1       | Data used in the ACL 2018 paper
 
@@ -26,7 +27,7 @@ If you use this data in your work, please cite our ACL paper _and_ the appropria
 For example, in your paper you could write (using the BibTeX below):
 
 ```
-In this work, we use version 2 of the modified SQL datasets from \citet{data-advising}, based on \citet{data-academic,data-atis-original,data-geography-original,data-atis-geography-scholar,data-imdb-yelp,data-restaurants-logic,data-restaurants-original,data-restaurants}
+In this work, we use version 3 of the modified SQL datasets from \citet{data-advising}, based on \citet{data-academic,data-atis-original,data-geography-original,data-atis-geography-scholar,data-imdb-yelp,data-restaurants-logic,data-restaurants-original,data-restaurants,data-spider,data-wikisql}
 ```
 
 If you are only using one dataset, here are example citation commands:
@@ -39,8 +40,10 @@ ATIS         | `\citet{data-advising,data-atis-original,data-atis-geography-scho
 Geography    | `\citet{data-advising,data-geography-original,data-atis-geography-scholar}`
 Restaurants  | `\citet{data-advising,data-restaurants-logic,data-restaurants-original,data-restaurants}`
 Scholar      | `\citet{data-advising,data-atis-geography-scholar}`
+Spider       | `\citet{data-advising,data-spider}`
 IMDB         | `\citet{data-advising,data-imdb-yelp}`
 Yelp         | `\citet{data-advising,data-imdb-yelp}`
+WikiSQL      | `\citet{data-advising,data-wikisql}`
 
 ```TeX
 @InProceedings{data-sql-advising,
@@ -140,6 +143,24 @@ Yelp         | `\citet{data-advising,data-imdb-yelp}`
   pages     = {59--76},
   url       = {https://doi.org/10.1007/978-3-642-45260-4_5},
 }
+
+@InProceedings{data-spider,
+  author    = {Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev},
+  title     = {Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task},
+  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
+  year      = {2018},
+  location  = {Brussels, Belgium},
+  pages     = {3911--3921},
+  url       = {http://aclweb.org/anthology/D18-1425},
+}
+
+@article{data-wikisql,
+  author    = {Victor Zhong, Caiming Xiong, and Richard Socher},
+  title     = {Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning},
+  year      = {2017},
+  journal   = {CoRR},
+  volume    = {abs/1709.00103},
+}
 ```
 
 # Contributions
@@ -153,4 +174,5 @@ For some ideas of issues to address, see our list of [known issues](./known-issu
 
 # Acknowledgments
 
-This material is based in part upon work supported by IBM under contract 4915012629. Any opinions, findings, conclusions or recommendations expressed are those of the authors and do not necessarily reflect the views of IBM.
+This material is based in part upon work supported by IBM under contract 4915012629.
+Any opinions, findings, conclusions or recommendations expressed are those of the authors and do not necessarily reflect the views of IBM.

data/READ-history.md

@@ -11,7 +11,9 @@ geography    | [Iyer et al., 2017](http://aclweb.org/anthology/P/P17/P17-1089.pd
 imdb         | [Yaghmazadeh et al., 2017](http://doi.org/10.1145/3133887)                  | [UT](https://drive.google.com/drive/folders/0B-2uoWxAwJGKY09kaEtTZU1nTWM)
 restaurants  | [Popescu et al., 2003](https://doi.org/10.1007/978-3-642-45260-4_5)         | [Trento](https://ikernels-portal.disi.unitn.it/repository/semmap/)
 scholar      | [Iyer et al., 2017](http://aclweb.org/anthology/P/P17/P17-1089.pdf)         | [UW](https://github.com/sriniiyer/nl2sql/tree/master/data)
+spider       | [Yu et al., 2018)](http://aclweb.org/anthology/D18-1425)                    | [Yale](https://yale-lily.github.io/spider)
 yelp         | [Yaghmazadeh et al., 2017](http://doi.org/10.1145/3133887)                  | [UT](https://drive.google.com/drive/folders/0B-2uoWxAwJGKY09kaEtTZU1nTWM)
+wikisql      | [Zhong et al., 2017](https://arxiv.org/pdf/1709.00103.pdf)                  | [Salesforce](https://github.com/salesforce/WikiSQL)
 
 ## academic
 
@@ -56,6 +58,11 @@ We have corrected some minor issues in the data:
 
 Constructed at UW in 2017
 
+## spider
+
+1. Combination of data from this repository (1,659 queries) and new data (8,034 queries) across a large set of tabe=les.
+2. SQL canonicalised and variables detected automatically by us
+
 ## yelp and imdb
 
 Constructed at UT Austin in 2017
@@ -68,3 +75,8 @@ Note - in the imdb dataset there are some cases where multiple SQL queries are p
 "select actor_0.nationality from actor as actor_0 where actor_0.name = \" Ben Affleck \" "
 ```
 
+## wikiSQL
+
+1. Data collected by Salesforce
+2. SQL converted to our format and duplicate queries detected by us
+

data/README.md

@@ -14,13 +14,14 @@ For each dataset we provide:
 - `*-fields.txt`, a list of fields in the database
 - `*-schema.csv`, key information about each database field
 
-Four of the databases are not included in the repository either for size or licensing reasons.
+Some of the databases are not included in the repository either for size or licensing reasons.
 They can be found as follows:
 
 Dataset  | Database
 -------- | ----------
 Academic (MAS), IMDB, Yelp | [https://drive.google.com/drive/folders/0B-2uoWxAwJGKY09kaEtTZU1nTWM](https://drive.google.com/drive/folders/0B-2uoWxAwJGKY09kaEtTZU1nTWM)
 Scholar  | [https://drive.google.com/file/d/0Bw5kFkY8RRXYRXdYYlhfdXRlTVk](https://drive.google.com/file/d/0Bw5kFkY8RRXYRXdYYlhfdXRlTVk)
+Spider   | [https://yale-lily.github.io/spider](https://yale-lily.github.io/spider)
 
 For more information about the sources of data see the [READ-history.md](./READ-history.md) file.
 
@@ -50,6 +51,23 @@ variables/example        | string            | An example value that could fill
 variables/name           | string            | The variable name
 variables/type           | string            | Dataset specific type
 
+For WikiSQL and Spider we have a few additional fields:
+
+Symbol             | Type              | Meaning
+------------------ | ----------------- | -----------------------------
+sentences/original | string            | The question from the original dataset, before our simple tokenisation
+sentences/database | string            | The name of the database this question is for [Spider]
+sentences/table-id | string            | The name of the table this question is for [WikiSQL]
+sql-original       | list of strings   | The query from the original dataset, before our canonicalisation
+
+Also, there are a few caveats:
+
+- There is no query split.
+- For Spider the test set is currently not available here (it is being kept secret by the original creators).
+- We did automatic variable identification and spot checked some of it (if you find issues, let us know!).
+- We modified our canonicalisation script to convert the data and spot checked the output (again, if you find issues, let us know!).
+- For WikiSQL we substituted in the actual field names, which contain all sorts of characters, so the SQL is not always valid. This substitution also means the 'sql-original' column numbers may be incorrect for some of the examples as we merged based on the final SQL.
+
 Example:
 
 ```

data/advising-schema.csv

@@ -1,4 +1,4 @@
-Table Name, Field Name, Is Primary Key, Is Foreign Key, Type
+Table Name, Field Name, Type, Is Foreign Key, Is Primary Key
 AREA, course_id,  int(11),      YES,   -,  NULL,     -
 AREA, area,       varchar(30),  YES,   -,  NULL,     -
 -, -, -, -, -, -, -

data/advising.json

@@ -4186,7 +4186,7 @@
             }
         ],
         "sql": [
-            "SELECT DISTINCT COURSEalias0.NAME , COURSEalias0.NUMBER FROM COURSE AS COURSEalias0 WHERE COURSEalias0.DEPARTMENT = \"department0\" AND COURSEalias0.HAS_LAB = \"N\" AND ;"
+            "SELECT DISTINCT COURSEalias0.NAME , COURSEalias0.NUMBER FROM COURSE AS COURSEalias0 WHERE COURSEalias0.DEPARTMENT = \"department0\" AND COURSEalias0.HAS_LAB = \"N\" ;"
         ],
         "variables": [
             {
@@ -15067,7 +15067,7 @@
             }
         ],
         "sql": [
-            "SELECT DISTINCT COURSEalias0.NAME , COURSEalias0.NUMBER FROM COURSE AS COURSEalias0 WHERE COURSEalias0.DEPARTMENT = \"department0\" COURSEalias0.CREDITS = credit0 ;"
+            "SELECT DISTINCT COURSEalias0.NAME , COURSEalias0.NUMBER FROM COURSE AS COURSEalias0 WHERE COURSEalias0.DEPARTMENT = \"department0\" AND COURSEalias0.CREDITS = credit0 ;"
         ],
         "variables": [
             {
@@ -25745,7 +25745,7 @@
             }
         ],
         "sql": [
-            "SELECT DISTINCT COURSEalias0.NAME , COURSEalias0.NUMBER , PROGRAM_COURSEalias0.WORKLOAD FROM COURSE AS COURSEalias0 , PROGRAM_COURSE AS PROGRAM_COURSEalias0 WHERE COURSEalias0.DEPARTMENT LIKE \"%department0%\" AND PROGRAM_COURSEalias0.COURSE_ID = COURSEalias0.COURSE_ID AND PROGRAM_COURSEalias0.WORKLOAD < 3 ; ;"
+            "SELECT DISTINCT COURSEalias0.NAME , COURSEalias0.NUMBER , PROGRAM_COURSEalias0.WORKLOAD FROM COURSE AS COURSEalias0 , PROGRAM_COURSE AS PROGRAM_COURSEalias0 WHERE COURSEalias0.DEPARTMENT LIKE \"%department0%\" AND PROGRAM_COURSEalias0.COURSE_ID = COURSEalias0.COURSE_ID AND PROGRAM_COURSEalias0.WORKLOAD < 3 ;"
         ],
         "variables": [
             {