How to add a reference in a Sphinx custom directive?

I'm creating a custom directive to display the list of all the available components in pydata-sphinx theme. I try to avoid using the raw directive so I'm building a custom one to remain compatible with the other builders.

Here is the important part of the code:

"""A directive to generate the list of all the built-in components.

Read the content of the component folder and generate a list of all the components.
This list will display some information about the component and a link to the
GitHub file.
"""
from docutils import nodes
from sphinx.application import Sphinx
from sphinx.util.docutils import SphinxDirective


class ComponentListDirective(SphinxDirective):
    """A directive to generate the list of all the built-in components."""

    # ...

    def run(self) -> List[nodes.Node]:
        """Create the list."""
        
        # ... 
        # `component` is a list of pathlib Path
        # `url` is a list of string 
        # `docs` is a list of string

        # build the list of all the components
        items = []
        for component, url, doc in zip(components, urls, docs):
            items.append(nodes.list_item(
                "",
                nodes.reference("", component.name, refuri=url), #this line is the source of the issue
                nodes.Text(f": {doc}")
            ))

        return [nodes.bullet_list("", *items)]

When I try to execute the previous code in my sphinx build I get the following error:

Exception occurred:
  File "/home/borntobealive/libs/pydata-sphinx-theme/.nox/docs/lib/python3.10/site-packages/sphinx/writers/html5.py", line 225, in visit_reference
    assert len(node) == 1 and isinstance(node[0], nodes.image)
AssertionError

This assertion is performed by sphinx if the parent node is not a TextELement. So I tried to wrap things in a Text node:

nodes.Text(nodes.reference("", component.name, refuri=url))

But the I only get the __repr__ of the reference not a real link (I think it's because Text nodes only accept strings)

So I also tried using a TextElement:

nodes.TextElement("", "", nodes.reference("", component.name, refuri=url))

which also raised an error:

Exception occurred:
  File "/home/borntobealive/libs/pydata-sphinx-theme/.nox/docs/lib/python3.10/site-packages/docutils/nodes.py", line 2040, in unknown_departure
    raise NotImplementedError(
NotImplementedError: <class 'types.BootstrapHTML5Translator'> departing unknown node type: TextElement

Does someone know how I should add the link at the start of the bullet list ? If you miss some context, you can find the complete code of the directive here (<100 lines)