UseLazyLoadingProxies with EF core loads whole list when adding a new child entity

I'm using UseLazyLoadingProxies with EF core, postgresql and DDD architecture. I have Parent and Child objects lets say they look something like this:

public class Parent 
{
    public int Id { get; private set;}
    public string Name {get; private set}
    public virtual ICollection<Child> Children {get; private set}

    public void AddChild(Child child)
    {
        Children.Add(child);
    }
}

public class Child 
{
    public int Id { get; private set;}
    public string Name {get; private set}
}

when I use parent.Add(child) proxy does its thing and pulls whole list of child elements before adding new child in the list, is there a way to work around it? because child elements can be 50 000 and right now its pulling all the rows from database for no reason other than to add a new child entity