Join "optional" and "extra_info" rules in grammar

docs/typing_syntax.md

@@ -7,24 +7,24 @@
 > Several features are experimental and included to make adoption of docstub easier.
 > Long-term, some of these might be discouraged or removed as docstub matures.
 
-Docstub defines its own [grammar](../src/docstub/doctype.lark) to parse and transform type information in docstrings into valid type annotations.
+Docstub defines its own [grammar](../src/docstub/doctype.lark) to parse and transform type information in docstrings (doctypes) into valid Python type expressions.
 This grammar fully supports [Python's conventional typing syntax](https://typing.python.org/en/latest/index.html).
-So any type annotation that is valid in Python, can be used in a docstrings as is.
+So any type expression that is valid in Python, can be used in a docstrings as is.
 In addition, docstub extends this syntax with several "natural language" expressions that are commonly used in the scientific Python ecosystem.
 
-Docstrings are expected to follow the NumPyDoc style:
+Docstrings should follow a form that is inspired by the NumPyDoc style:
 ```
 Section name
 ------------
-name : annotation, optional, extra_info
+name : doctype, optional_info
   Description.
 ```
 
-- `name` might be the name of a parameter or attribute.
-  Other sections like "Returns" or "Yields" are supported.
-- `annotation` the actual type information that will be transformed into the type annotation.
-- `optional` and `extra_info` can be appended to provide additional information.
-  Their presence and content doesn't currently affect the resulting type annotation.
+- `name` might be the name of a parameter, attribute or similar.
+- `doctype` the actual type information that will be transformed into a Python type.
+- `optional_info` is optional and captures anything after the first comma (that is not inside a type expression).
+  It is useful to provide additional information for readers.
+  Its presence and content doesn't currently affect the resulting type annotation.
 
 
 ## Unions

docs/user_guide.md

@@ -93,21 +93,21 @@ def example_metric(
 There are several interesting things to note here:
 
 - Many existing conventions that the scientific Python ecosystem uses, will work out of the box.
-  In this case, docstub knew how to translate `array-like`, `array of dtype uint8` into a valid type annotation in the stub file.
-  In a similar manner, `or` can be used as a "natural language" alternative to `|`.
+  In this case, docstub knew how to translate `array-like`, `array of dtype uint8` into a valid Python type for the stub file.
+  In a similar manner, `or` can be used as a "natural language" alternative to `|` to form unions.
   You can find more details in [Typing syntax in docstrings](typing_syntax.md).
 
-- Optional arguments that default to `None` are recognized and a `| None` is appended automatically if the type doesn't include it already.
+- Optional arguments that default to `None` are recognized and a `| None` is appended automatically.
   The `optional` or `default = ...` part don't influence the annotation.
 
 - Referencing the `float` and `Iterable` types worked out of the box.
-  All builtin types as well as types from the standard libraries `typing` and `collections.abc` module can be used.
+  All builtin types as well as types from the standard libraries `typing` and `collections.abc` module can be used like this.
   Necessary imports will be added automatically to the stub file.
 
 
-## Using types & nicknames
+## Referencing types & nicknames
 
-To translate a type from a docstring into a valid type annotation, docstub needs to know where that type originates from and how to import it.
+To translate a type from a docstring into a valid type annotation, docstub needs to know where names in that type are defined from where to import them.
 Out of the box, docstub will know about builtin types such as `int` or `bool` that don't need an import, and types in `typing`, `collections.abc` from Python's standard library.
 It will source these from the Python environment it is installed in.
 In addition to that, docstub will collect all types in the package directory you are running it on.

src/docstub/_docstrings.py

@@ -137,16 +137,19 @@ def as_generator(cls, *, yield_types, receive_types=(), return_types=()):
         generator = cls(value=value, imports=imports)
         return generator
 
-    def as_optional(self):
-        """Return optional version of this annotation by appending `| None`.
+    def as_union_with_none(self):
+        """Return a union with `| None` of the current annotation.
+
+        .. note::
+            Doesn't check for `| None` or `Optional[...]` being present.
 
         Returns
         -------
-        optional : Annotation
+        union : Annotation
 
         Examples
         --------
-        >>> Annotation(value="int").as_optional()
+        >>> Annotation(value="int").as_union_with_none()
         Annotation(value='int | None', imports=frozenset())
         """
         # TODO account for `| None` or `Optional` already being included?
@@ -471,7 +474,7 @@ def optional(self, tree):
         logger.debug("dropping optional / default info")
         return lark.Discard
 
-    def extra_info(self, tree):
+    def optional_info(self, tree):
         """
         Parameters
         ----------
@@ -481,7 +484,7 @@ def extra_info(self, tree):
         -------
         out : lark.visitors._DiscardType
         """
-        logger.debug("dropping extra info")
+        logger.debug("dropping optional info")
         return lark.Discard
 
     def __default__(self, data, children, meta):

src/docstub/_stubs.py

@@ -537,7 +537,7 @@ def leave_Param(self, original_node, updated_node):
             pytype = pytypes.parameters.get(name)
             if pytype:
                 if defaults_to_none:
-                    pytype = pytype.as_optional()
+                    pytype = pytype.as_union_with_none()
                 annotation_value = pytype.value
 
                 if original_node.annotation is None:

src/docstub/doctype.lark

@@ -12,7 +12,7 @@
 // The basic structure of a full docstring annotation as it comes after the
 // `name : `. It includes additional meta information that is optional and
 // currently ignored.
-?annotation_with_meta: type ("," optional)? ("," extra_info)?
+?annotation_with_meta: type ("," optional_info)?
 
 
 // A type annotation. Can range from a simple qualified name to a complex
@@ -132,14 +132,10 @@ shape: "(" dim ",)"
 ?dim: INT | ELLIPSES | NAME ("=" INT)?
 
 
-// Optional information about a parameter has a default value, added after the
-// docstring annotation. Currently dropped during transformation.
-optional:  "optional" | "default" ("=" | ":")? literal_item
-
-
-// Extra meta information added after the docstring annotation.
-// Currently dropped during transformation.
-extra_info: /[^\r\n]+/
+// Optional meta information added after the docstring annotation, e.g.
+// "optional" or "in range (0, 10), default: 3". Information is dropped and not
+// used to generate stubs.
+optional_info: /[^\r\n]+/
 
 // A simple name. Can start with a number or character. Can be delimited by "_"
 // or "-" but not by ".".

tests/test_docstrings.py

@@ -136,21 +136,29 @@ def test_literals(self, doctype, expected):
     @pytest.mark.parametrize(
         ("doctype", "expected"),
         [
-            ("int, optional", "int"),
-            ("int | None, optional", "int | None"),
-            ("int, default -1", "int"),
-            ("int, default = 1", "int"),
-            ("int, default: 0", "int"),
-            ("float, default: 1.0", "float"),
-            ("{'a', 'b'}, default : 'a'", "Literal['a', 'b']"),
+            ("int", "int"),
+            ("int | None", "int | None"),
+            ("tuple of (int, float)", "tuple[int, float]"),
+            ("{'a', 'b'}", "Literal['a', 'b']"),
         ],
     )
-    @pytest.mark.parametrize("extra_info", [None, "int", ", extra, info"])
-    def test_optional_extra_info(self, doctype, expected, extra_info):
-        if extra_info:
-            doctype = f"{doctype}, {extra_info}"
+    @pytest.mark.parametrize(
+        "optional_info",
+        [
+            "",
+            ", optional",
+            ", default -1",
+            ", default: -1",
+            ", default = 1",
+            ", in range (0, 1), optional",
+            ", optional, in range [0, 1]",
+            ", see parameter `image`, optional",
+        ],
+    )
+    def test_optional_info(self, doctype, expected, optional_info):
+        doctype_with_optional = doctype + optional_info
         transformer = DoctypeTransformer()
-        annotation, _ = transformer.doctype_to_annotation(doctype)
+        annotation, _ = transformer.doctype_to_annotation(doctype_with_optional)
         assert annotation.value == expected
 
     @pytest.mark.parametrize(